Automations

Automations are data related tasks you can schedule on a regular basis (e.g. every night, every hour).

Here is a list of common automations:

  • Send an email with a dashboard attached as a pdf file
  • Aggregate purchases data per customer and store the result in a table of the Serenytics internal Datawarehouse
  • Fill in the cache of a dashboard to reduce the load time for a user
  • Send an email alert if the sales of the current day on a given marketplace are below a threshold
  • Chain several automations

There are three types of automations:

  • ETL Steps : used to load data in a datasource, transform it and store the result in an output datasource.
  • Tasks : library of frequently used jobs (e.g. send an email, load a dashboard cache...)
  • Python scripts : to execute any Python code.

On an automation page:

  • The first tab lets you configure the automation.
  • The Execution tab lets you run the script, or schedule it.
  • The Usage tab shows you how your automation is used.
  • The Wehhook tab lets you configure an URL you can call to trigger the execution of the automation.

Task

A task is the simplest type of automation to configure. For example, the task to send a dashboard in an email can be configured with a simple form: email subject, content, recipients list and the dashboard to attach as pdf.

Available tasks are:

  • Workflow / chain scripts: to chain several automations.
  • Emailing / Send email with dashboard attached as PDF: to send a dashboard as a pdf file in an email to a list of recipients.
  • Admin / Reload users meta-data: lets you load meta-data per login to filter dashboards according to the logged user. (see filter data with login)
  • Cache management / Refresh dashboards cache : to load the cache of a list of dashboards.
  • Data import / Synchronize a data source into a Serenytics storage: to copy all the rows from any datasource into a Serenytics storage.
  • Predictive / Linear regression: to compute a linear regression on a time serie.

ETL Steps

An ETL step takes data from an input data-source, apply transformations and store the result in a table of the internal datawarehouse.

Before using an ETL step, you must have created the output storage table. To achieve that, use the Data menu and create a new datasource of type Storage (in Serenytics's Datawarehouse tab).

Possible transformations are:

  • Select columns or formulas from the input data source
  • Apply filters
  • Aggregate data according to dimensions
  • Rename columns (by clicking on the columns header in the preview)

In the configuration tab, the displayed table is just a preview. The full computation is not computed. To actually execute store the result in the output table, you must use the Execution tab (either by clicking on Run Now, or by scheduling the automation).

Warning

If your input data source is a Serenytics storage, the computation is optimized to handle large datasets (e.g. hundreds of millions of rows). Because of this optimization, if you aggregate by time dimensions, the preview format of the time dimensions might be different between the preview and the actual result.

Warning

If the input data source is a Serenytics storage, there is no limit on the number of rows you can manipulate with an ETL step. For any other type of data source, the limit is 1.000.000 rows.

Append mode

In default mode, the execution of an ETL step overwrites all the existing rows contained in the output source (and replaces them by the result of its query). When the append mode is activated, the result of the ETL query is appended to the existing rows of the output data source.

Warning

The output source must already have its columns set (e.g. created by a first ETL step without append mode). The output columns (names and types) of the ETL step in append mode must be identical to the existing columns of the current output source. Otherwise, the ETL step will fail.

Another way to run an ETL step in append mode is to call it using a Python script. If you run an ETL step script and specify the append parameter in the script, it will overwrite the append mode defined in the ETL step parameters.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
import serenytics
client = serenytics.Client()
MY_SCRIPT_UUID = 'XXXXX'

script = client.get_script_by_uuid(uuid=MY_SCRIPT_UUID)

params = {
  'append': True,
}
script.run(params=params, async=False)

Python scripts

Python scripts let you write Python code and execute it directly on the Serenytics platform. The main use case is to retrieve data from a configured data source and to execute actions according to this data. For example, you can apply advanced computation in Python/Pandas and store the results in a Serenytics storage. You can also send alert emails according to advanced computation on your data. You can also load data from any system and store it into a Serenytics storage.

Input and output datasources of a Python script

For a Python script, it is important to specify its input and output datasources in the Usage tab. When you do this, the script will appear correctly in your dataflow and removing the input or output datasource will be forbidden as this script uses it.

Detailed documentation is available in the developer section: Developer doc.

Run-as-user field for automations

When an automation is run, it must decide which user credentials will be used. This is very important as a script using credentials from a user with large permissions will have access to many datasources.

If the automation is run from the Studio interface by clicking on 'Run Now' button, the automation will be run using the credentials of the connected user.

If the automation is run automatically from a scheduling, the automation uses the credentials of the automation field run-as-user. When a user creates a new automation, he becomes the default value for run-as-user. If this field is not set (i.e. the user has been deleted), the script uses the credentials of the first admin user of the organization.

This field can be modified from the Advanced tab in the script configuration (only admin users can modify this field).