Skip to content


Automations are data related tasks you can schedule on a regular basis (e.g. every night, every hour).

Here is a list of common automations:

  • Send an email with a dashboard attached as a pdf file
  • Aggregate purchases data per customer and store the result in a table of the Serenytics internal Datawarehouse
  • Fill in the cache of a dashboard to reduce the load time for a user
  • Send an email alert if the sales of the current day on a given marketplace are below a threshold
  • Chain several automations

There are three types of automations:

  • ETL Steps : used to load data in a datasource, transform it and store the result in an output datasource.
  • Tasks : library of frequently used jobs (e.g. send an email, load a dashboard cache...)
  • Python scripts : to execute any Python code.

On an automation page:

  • The first tab lets you configure the automation.
  • The Execution tab lets you run the script, or schedule it.
  • The Usage tab shows you how your automation is used.
  • The Wehhook tab lets you configure an URL you can call to trigger the execution of the automation.


A task is the simplest type of automation to configure. For example, the task to send a dashboard in an email can be configured with a simple form: email subject, content, recipients list and the dashboard to attach as pdf.

Available tasks are:

  • Workflow / chain scripts: to chain several automations.
  • Emailing / Send email with dashboard attached as PDF: to send a dashboard as a pdf file in an email to a list of recipients.
  • Admin / Reload users meta-data: lets you load meta-data per login to filter dashboards according to the logged user. (see filter data with login)
  • Cache management / Refresh dashboards cache : to load the cache of a list of dashboards.
  • Data import / Synchronize a data source into a Serenytics storage: to copy all the rows from any datasource into a Serenytics storage.
  • Predictive / Linear regression: to compute a linear regression on a time serie.

ETL Steps

An ETL step takes data from an input data-source, apply transformations and store the result in a table of the internal datawarehouse.

Before using an ETL step, you must have created the output storage table. To achieve that, use the Data menu and create a new datasource of type Storage (in Serenytics's Datawarehouse tab).

Possible transformations are:

  • Select columns or formulas from the input data source
  • Apply filters
  • Aggregate data according to dimensions
  • Rename columns (by clicking on the columns header in the preview)

In the configuration tab, the displayed table is just a preview. The full computation is not computed. To actually execute store the result in the output table, you must use the Execution tab (either by clicking on Run Now, or by scheduling the automation).


If your input data source is a Serenytics storage, the computation is optimized to handle large datasets (e.g. hundreds of millions of rows). Because of this optimization, if you aggregate by time dimensions, the preview format of the time dimensions might be different between the preview and the actual result.


If the input data source is a Serenytics storage, there is no limit on the number of rows you can manipulate with an ETL step. For any other type of data source, the limit is 1.000.000 rows.

Append mode

In default mode, the execution of an ETL step overwrites all the existing rows contained in the output source (and replaces them by the result of its query). When the append mode is activated, the result of the ETL query is appended to the existing rows of the output data source.


The output source must already have its columns set (e.g. created by a first ETL step without append mode). The output columns (names and types) of the ETL step in append mode must be identical to the existing columns of the current output source. Otherwise, the ETL step will fail.

Another way to run an ETL step in append mode is to call it using a Python script. If you run an ETL step script and specify the append parameter in the script, it will overwrite the append mode defined in the ETL step parameters.

import serenytics
client = serenytics.Client()

script = client.get_script_by_uuid(uuid=MY_SCRIPT_UUID)

params = {
  'append': True,
}, async_=False)

Remove duplicates

If you enable the Remove duplicates option, an extra step will be added to the processing to delete duplicate rows from the output. This step is applied after the transformation.

The first parameter to choose is the Unique column. This is the column that must contain only unique values in the output table. Very often, this is an ID such as customer_id or product_id.

If you run the ETL step, the output will not contain any duplicate of this value. For example, if you have 5 rows having the value customer_id=128 in the input, only one of these rows will exist in the output. And 4 rows will be dropped.

The next parameters allow you to choose which rows must be dropped in case of duplicates.

If you keep the Default mode value in the Duplicate to keep's mode option, you have no control on this choice.

If you choose Max of a column, the algorithm will sort the duplicates rows and keep the row having the maximum value. The sort column can be selected in the sort column parameter.

And selecting the Min of a column value will do the opposite.

Duplicate are removed after the transformation step

An ETL step will first store the result of your transformation (groupby, select, filter) in the output table. And then, it will remove the duplicates. If your transformation requires the input to have no duplicate, you need to add an extra ETL step (and an extra table) to be run before your transformation step. This is particularly important if you have groupby and $$count$$ in your transformation as the $$count$$ result will include all the duplicates.

Python scripts

Python scripts let you write Python code and execute it directly on the Serenytics platform. The main use case is to retrieve data from a configured data source and to execute actions according to this data. For example, you can apply advanced computation in Python/Pandas and store the results in a Serenytics storage. You can also send alert emails according to advanced computation on your data. You can also load data from any system and store it into a Serenytics storage.

A Python script can be triggered by a button in a dashboard (using an action widget). The script can also read parameters defined by the dashboard viewer in input fields. This way, you can create low-code data applications.

Input and output datasources of a Python script

For a Python script, it is important to specify its input and output datasources in the Usage tab. When you do this, the script will appear correctly in your dataflow and removing the input or output datasource will be forbidden as this script uses it.

Detailed documentation is available in the developer section: Developer doc.

Run-as-user field for automations

When an automation is run, it must decide which user credentials will be used. This is very important as a script using credentials from a user with large permissions will have access to many datasources.

If the automation is run from the Studio interface by clicking on 'Run Now' button, the automation will be run using the credentials of the connected user.

If the automation is run automatically from a scheduling, the automation uses the credentials of the automation field run-as-user. When a user creates a new automation, he becomes the default value for run-as-user. If this field is not set (i.e. the user has been deleted), the script uses the credentials of the first admin user of the organization.

This field can be modified from the Advanced tab in the script configuration (only admin users can modify this field).

Allow all users to run this script field

In some cases, you want to allow all the logged user of your organization to run a script. This is only possible if you enable this option (by default, only studio users who are authorized on the automation's folder can run it). For example, this is required if you add a button in a dashboard triggering this script (using the function btn_run_automation).

Credentials and Allow all users to run this script option

When this option is activated, the automation will always be run with the credentials of the user defined in the field run-as-user. If you are using a Python script, you can retrieve the user who triggered the script in the dict provided by client.script_args in the field triggered_by_user.


When a mutex is configured for a set of scripts, it ensures these scripts will never run in parallel. This is particularly useful when your scripts are doing reads and writes on the same data and are triggered from dashboards buttons (in this case, you have no control over their executions' order).

When several scripts share the same mutex name, if they are run at the same time, one of them will run first, and the others will wait. Once the first one is finished, the second one will run and so on.

An easy way to test this option is to create two Python scripts. The first one executes a time.sleep(10) and the second one is a simple print("hello world"). When both scripts are executed simultaneously, the second script will be executed almost immediately. If you define a mutex for them and trigger them at the same time (starting with the waiting script), the second one will be executed once the first one is finished (so it will be in waiting status for at least 10 seconds).