Storage data source

A Storage data source is a table within the Serenytics internal data-warehouse (i.e. AWS Redshift). It can be used to store and merge data from different systems (e.g. the CRM, the marketing automation system, the purchases system...). It can also be used to store the results of data processings involving these merged datasets.

When you create a new storage, it is empty by default. There are several ways to load data in a storage:

  • Using an ETL step

    ETL steps are jobs you can create and schedule from the Automation menu. An ETL takes an input data source of any type, transforms its data, and stores the resulting dataset in a storage. This is useful to prepare your data.

  • Importing a CSV file from a (S)FTP Server

    You can create a FTP Job Import from the Automation menu. This lets you load a CSV file from a FTP or SFTP server in a storage.

  • Using the Serenytics Python client to load a file or any Pandas dataframe

    You can create Python scripts from the Automation menu. In these scripts, the serenytics module lets you load a file into a storage. You can also load a Pandas dataframe. This is useful when you need to achieve advanced computation and use the resulting storage in a dashboard (or as a source of another etl step/python script). You can also create a python script to load data from a source if Serenytics does not provide a plug&play connector ( e.g. for non standard REST API). You just need to obtain your data in Python and then store the result in a storage. Then, you can use this data in your dashboards.

  • Pushing messages

    You can also push JSON messages into a storage calling the Serenytics REST API. If a field previously used (i.e. it exists in the current storage model) is not provided by a JSON message, it is set to a default value in the added row. If a field exists in a new JSON message but not in the current model, it will be added to your storage (and all existing rows will have the NULL value for this field).

Tip: how to push a message in a Storage with curl

Use: curl -d '{"key":"msg1", "key2":"msg2"}' -H "Content-Type: application/json" -X POST STORAGE_URL

where you need to replace STORAGE_URL by the actual push url of your data source. It is available in the configuration page of your storage, in the field Url. It looks like

Of course, any similar HTTP POST done in any language will work.