On-premises Docker-compose Install guide¶

In this guide, you will learn how to install the Serenytics data-platform on your own server.

The Serenytics platform is a set of Docker containers. In this documentation, you will learn how to deploy and run those containers on a machine using docker-compose. If you have a Kubernetes cluster, please follow this dedicated Helm/k8s install documentation.

This guide only explains how to install Serenytics on a single machine. For multi-machines architecture, please contact our support.

Prerequisites¶

You need a server with an Ubuntu 18.04 LTS - 64 bits OS (similar Linux distributions should work).
You need an admin (sudo) account on this machine.
The recommended machine is highly related to your Serenytics needs. As a minimum, we recommend a quad-core machine, with CPU frequency above 3GHz, 16Gb of RAM and 100Gb of disk.
You need to have obtained a dedicated DockerHub login/password from Serenytics to be able to download the Serenytics images. These identifiers are to be kept secret.
If you want to enable SSL (strongly encouraged), you also need a private key and an SSL certificate. If you don't have it, the instructions below will guide through the setup of a default self-signed certificate. If you don't enable the SSL, your passwords and data won't be encrypted. That's only acceptable if you use test data or work in a strictly controlled environment and know precisely what you are doing.

Step 1: Download the Serenytics CLI¶

The install is done through the Serenytics CLI (Command Line Interface).

The first step is to download this CLI on your machine:

curl -L "https://s3-us-west-2.amazonaws.com/assets.serenytics.com/install/serenytics_onpremises_cli.py" -o serenytics_onpremises_cli.py && chmod u+x serenytics_onpremises_cli.py

This step downloads the file serenytics_onpremises_cli.py which is our CLI. This CLI has commands to install Serenytics, but also to restart it and a few other ones used to administrate the platform.

Step 2: Download the config files¶

Before launching the install, you also need to download the default configuration file.

For non SSL install¶

Run the command below to download the serenytics.yml file (that's the docker configuration as a docker-compose file).

curl -L "https://s3-us-west-2.amazonaws.com/assets.serenytics.com/install/serenytics.yml" -o serenytics.yml

For SSL install¶

SSL - step 2.a¶

Run the command below to download the serenytics.yml file (that's the docker configuration as a docker-compose file with SSL enabled).

curl -L "https://s3-us-west-2.amazonaws.com/assets.serenytics.com/install/serenytics-with-ssl.yml" -o serenytics.yml

SSL - step 2.b¶

You also need to download a specific config file to configure the Serenytics web server and proxy (using Nginx) with SSL:

curl -L "https://s3-us-west-2.amazonaws.com/assets.serenytics.com/install/nginx-custom.conf" -o nginx-custom.conf

This file customizes NGinx to enable SSL. If your a web administrator, you can later open and check the configuration.

SSL - step 2.c¶

You need to drop your certificate file and your private key in a folder (you can specify this folder later to Serenytics).

If you do not have it, run:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt

This will create a private key and a certificate in the default folder /etc/ssl.

Step 3: Run the installation¶

Run the command ./serenytics_onpremises_cli.py install

The CLI will ask you a few questions. Your answers will be stored in a .env file. During the setup, you can skip answers and modify this .env file later.

The install will have the following steps:

It installs Docker on the machine (if not already installed).
It asks for the credentials you obtained from Serenytics to get the Serenytics images.
It asks for the dedicated version name to install (leave empty in most cases)
It asks for the version number to install (asks the Serenytics support if you're not sure). Current default version is 4717.
It asks for the IP of the machine. In Serenytics, the frontend (in the browser) calls the Serenytics API provided by the backend. This IP will be used in the frontend to query the API. You can provide an URL (e.g. serenytics.mydomain.com) or an IP.
It asks for the PostGreSQL database password (for the db storing the Serenytics objects such as dashboards, datasources, automations...).
It asks for the datawarehouse password (by default a local PostGreSQL database).
It asks for a secret (used to generate authentication tokens). You can enter any text of your choice.
It asks for SMTP settings (required if you want the platform to be able to send emails). Remember that you can edit any setting later.
It asks for a Mapbox key. Such a key is mandatory to show maps background (obtained from Mapbox).

Then, the CLI will download the docker images and run it. This can take a few minutes.

If you need to edit the .env file manually, you need to restart the Serenytics platform with ./serenytics_onpremises_cli.py start once you have saved your modifications.

SSL - specify the folder for your certificate and private key¶

In the .env file, you need to define :

SSL_PEM_PATH: the path to your certificate file
SSL_PRIVATE_KEY_PATH: the path to your private key

And add these three lines into your .env file:

SSL_PEM_PATH=/etc/ssl/certs/nginx-selfsigned.crt SSL_PRIVATE_KEY_PATH=/etc/ssl/private/nginx-selfsigned.key IP_ADDRESS=YOUR_IP_OR_DOMAIN

You need to replace YOUR_IP_OR_DOMAIN by either the server IP or URL (without the http part, e.g. serenytics.mydomain.com).

Once you have configured these options, restart the Serenytics platform with:

./serenytics_onpremises_cli.py start

Step 4: Firewall configuration¶

By default, everything must be closed on your server (except ssh access to administrate it).

For a non SSL install¶

You need to open the following ports from the outside world to this machine:

port 80: Serenytics embeds a web server running on this port. To enable a user to get the Serenytics frontEnd in its browser, this port must be open.
port 4000: This port is used by the Serenytics backend to provide the API. To enable the Serenytics frontend app to query the API, this port must be open.

For an SSL install¶

You need to open the following ports from the outside to this machine:

port 80: In SSL mode, queries to this port will be redirected to https on port 443.
port 443: In an SSL install, all http queries are run to this port. The Nginx then redirects the queries either to the frontEnd web server or to the backend server (if this is an API query).

In an SSL install, do not open the port 4000

If you open the port 4000, the api will be callable without SSL and that would be a security breach.

Step 5: Test your installation¶

Open the page http://machine_IP, http://my_url (with https if you activated SSL). You should see the Serenytics login page.

Limitation of self-signed certificates

If you used a self-signed certificate, your browser will certainly refuse to open this page. You need to accept the certificate (the steps to do that depends on your OS and browsers, this is beyond the goal of this guide). For the same security reasons, some Serenytics features won't work. Those are the features that request our backend to call our API. For example, generating a PDF file from a dashboard will not work with a self-signed certificate.

Click on create account to create your first account.

Then follow these 3 steps:

As a very first installation test, create a new datasource (using an XLS file). The preview must display its content.
Then create a dashboard with a first widget using this datasource.
Then, create an automation of type Python script. Run it. It should display "Hello world" in its output.
Then, create an automation of type 'Send dashbaord as PDF'. Choose the dashboard you just created, enter your email and execute it. You should receive the corresponding PDF in your inbox a few minutes later (this only works if you have configured an SMTP server and you are not using a self-signed certificate).

Troubleshooting¶

The CLI command ./serenytics_onpremises_cli.py status checks that all the services are running.

If you run a docker ps, you should see all the Serenytics containers running. You can run a docker logs --follow CONTAINER_ID to see a container's output.

If you have access to the frontend, open the browser console and if all the calls to the Serenytics API are failing, this means the firewall is blocking the API.

Custom setup¶

Change the Serenytics PostGreSQL server¶

With the default setup described above, the objects (e.g users, dashboards...) are stored in a PostGreSQL database which is hosted in a dedicated Docker container. This container runs on the machine where is done the install.

It's a good practice to have this PostGreSQL server on a different server. It's up to you to decide how you run your PostGreSQL server. It can be executed straight in another server, it can be run in a container in another machine, or you can use a managed service (e.g. RDS from AWS).

The only constraint is that the version must be PostGreSQL >= 9.6.

The first step is to run your PostGreSQL server and get its credentials. Of course, you need to configure your network in such a way that this server is reachable by the server where Serenytics is running (this is usually the main source of issues).

The second step is to stop the container running the default PostGreSQL database. To do that, execute a docker ps to get the list of running containers; note the CONTAINER ID of the container named root_postgres_1 and execute the command docker stop the_container_id.

Then, you need to modify the serenytics.yml file (i.e. the docker compose file) to remove the default PostGreSQL from the configuration (otherwise, it will be restarted next start you restart Serenytics).

To achieve that:

remove the postgres service (i.e. remove all the lines for this service).
remove the lines where this service is used (in the depends_on sections and links sections).

Then, you need to specify the credentials of your SQL server in this serenytics.yml file. To do that, provide the new values for the variables:

DB_NAME
DB_HOST
DB_LOGIN
DB_PASSWORD
DB_PORT
In the default configuration, DB_NAME, DB_HOST and DB_LOGIN are written in the serenytics.yml file. You can edit them directly.
DB_PASSWORD is provided with the docker-compose syntax: DB_PASSWORD=${POSTGRES_PASSWORD}. This will use the environment variable named POSTGRES_PASSWORD. It can be defined either in the shell or in the .env file. Note that if you defined it both in the shell (e.g. with export POSTGRES_PASSWORD=mypassword) and in the .env file, the value defined in the shell will be used. But for clarity, you should delete the default value written by the installer in the .env file.
DB_PORT is not written in the default configuration as it uses the default PostGreSQL port (5432). If you need to modify this port, just add this variable in the configuration file with the correct value.

When you have modified the serenytics.yml file, restart the Serenytics server with the CLI command: ./serenytics_onpremises_cli.py start. At the first run, our engine will create all the tables required by Serenytics in the provided database.

Credentials must be provided to 3 services in the docker-compose file

In the serenytics.yml file, there are 3 services that need to access the PostGreSQL database (backend, worker, worker-script). You need to provide the credentials to those 3 services.

Troubleshooting¶

To check that the SQL server can be reached from the machine hosting Serenytics:

install psql apt install postgresql-client postgresql-client-common
try to connect to the server using the PSQL syntax: psql -p 25060 -U dbuser -d mydbname -h myserver.mydomain.com

If you can't connect, it means you have a network or credentials issue to reach this server.

Change the PostGreSQL datawarehouse¶

By default, the datawarehouse is also a PostGreSQL server. The way to change it is very similar to changing the PostGreSQL server used for Serenytics objects (as described in the section above).

For the datawarehouse, the steps are:

stop the container named root_datawarehouse_1
disable the service named datawarehouse in the serenytics.yml file (i.e. remove the lines for this service and disable all references to it).
Provide the credentials for your custom datawarehouse (must be a PostGreSQL >= 9.6) with the variables:
- DATAWAREHOUSE_SMALL_ACCOUNTS_ADMIN_LOGIN
- DATAWAREHOUSE_SMALL_ACCOUNTS_ADMIN_PASSWORD (again, this uses a env variable)
- DATAWAREHOUSE_SMALL_ACCOUNTS_HOST
- DATAWAREHOUSE_SMALL_ACCOUNTS_PORT
- DATAWAREHOUSE_SMALL_ACCOUNTS_ADMIN_DB

The provided user must have rights to create databases in the server.

The Serenytics engine creates one database per Serenytics account. Note that this database creation only happens when the user creates its first Storage datasource (i.e. a table in the Serenytics internal datawarehouse). If you do not need this feature (e.g. if you only use Dataviz features), you can just drop the datawarehouse service, and all the other features will work.

Enable Keycloak SSO¶

You can enable SSO in Serenytics by using a Keycloak identity manager.

The first setup steps are :

In Serenytics, you must have created an organization that will be linked to a KeyCloak realm where your users are managed.
Then add the Serenytics client to your KeyCloack realm. The JSON-exported config is available here: https://s3.us-west-2.amazonaws.com/assets.serenytics.com/install/keycloak-SSO/serenytics-client-v1/serenytics.json
In this client in Keycloak, in its client-scopes, click on serenytics-dedicated, and in the mapper, edit the claim named serenyics_organization_uuid. You must fill it with the uuid of your Serenytics organization.

Then you can start to add the roles attached to the Serenytics client to your users (e.g. serenytics-admin, serenytics-business_analyst, serenytics-studio, serenytics-viewer). You only need to specify one role. In case you add several Serenytics roles to a user (ag viewer and admin), Serenytics will consider the role with the highest privileges.

In your Serenytics install, you must:

In the backend container:
- define an envvar ENABLE_KEYCLOAK (to true)
- define KEYCLOAK_PUBLIC_KEY_FILEPATH that must point to the public key of your Realm (for the RSA256 algorithm).
- ensure this container path is available (usually using a volume redirection)
In the frontend container:
- define the envvar USE_KEYCLOAK_AUTH to true
- define the envvar KEYCLOAK_URL

Then restart your Serenytics containers.

When trying to access Serenytics for the first time, you need to open the route /studio/MY_REALM_NAME/first_access. When accessing this page, your browser will store locally your REALM_NAME. Then you will be automatically redirected to KeyCloak for authentication.

A few points:

Users existing in Keycloak, but not in Serenytics will be created in Serenytics at their first login.
Clicking on log out in Serenytics will log the user out of Keycloak. And he will be redirected to its keycloak login page. The Realm name will be kept in the browser. In case you need to change your realm, you must delete the local storage of your browser.
If the category of a user (admin/viewer/business analyst/studio) is modified in Keycloak, it will be modified the next time this user accesses Serenytics.
A user with no Serenytics role in KeyCloak won't be able to access Serenytics (even if he had such an access previously).

Run the backend Serenytics CLI¶

The Serenytics backend has a CLI that allows you to execute several commands to administrate your objects.

To run it:

identify your backend container ID. To do that, run a docker ps command and pick the container id named xxx_backend.
run a shell in this container with: docker exec -it MY_BACKEND_CONTAINER /bin/bash
run the cli with python manage.pyc

Then you can get the list of all commands with: python manage.pyc --help. Here are a few examples:

You can create an organization with : python manage.pyc create_organization --orga_name=test_orga.
You can list all organizations with: python manage.pyc list_organizations.

You can get help on any command, for example: python manage.pyc create_organization --help.

Clone Organizations between several Serenytics instances

The cli can be used to clone a full organization to another one. The command name is clone_organization. It contains production in its name because its initial target was to get objects from our prod env. But it can be used to get objects from any instance of Serenytics.

Here is an example of this command: python manage.py clone_organization --platform-url https://9fds7feerf1f654.ngrok.app/api/ --api-key-of-source-account API_KEY_OF_AN_ADMIN_USER_IN_SOURCE_ORGANIZATION -u demo@serenytics.com

With this command, all the objects from the organization of the admin user having the given API-KEY, from the instance https://9fds7feerf1f654.ngrok.app/api/ will be cloned into the local server in the organization named demo@serenytics.com. You can use that to create a template organization and then clone it when you get a new customer, or to move an organization from a DEV server to a PROD server.

Note that the platform-url must be the API url, with a slash at the end. To get this url, open a Serenytics page, and check the requests to the backend API in the inspector.

Custom Log level and format¶

By default, containers will log their output in their sdtout/stderr in console format. You can access this through the command: docker logs --follow CONTAINER_ID

You can set the log level by passing the variable LOG_LEVEL to your containers.

In some cases, you want to monitor this log in an external tool. We use Datadog but any tool that loads JSON log will be compatible. To enable the log to be in JSON format, you must pass the LOG_HANDLER=JSON to your containers.