HumanSignals is an open-source product analytics suite, built for engineers

  • Automatically track every event on your website or app
  • Understand your users and how to improve your product
  • Deploy on your own infrastructure to keep control of your data.

Getting started for free

Follow our User Guide for on-premise deploys.

Features

We bring all the tools into one place to give you everything you need to build better products:

  • Event-based Analytics on users or groups - capture your product's usage data to see which users are doing what in your application
  • Product data visualizations graphs, funnels, cohorts, paths, retention, and dashboards
  • Complete control over your data - host it yourself on any infrastructure
  • Session recording to watch videos of your users' behavior, with fine-grained privacy controls
  • Automatically capture clicks and pageviews to analyze what your users are doing without pushing events manually
  • Feature flags to understand the impact of new features before rolling them out more widely
  • Heatmaps to understand how users interact with your product with the HumanSignals Toolbar
  • Automated Analysis to find correlations between successful users and their behaviors or attributes
  • Plays nicely with data warehouses import events or user data from your warehouse by writing a simple transformation plugin, and export data with pre-built plugins - such as BigQuery, Redshift, Snowflake and S3
  • Infinitely extensible use custom plugins to extend HumanSignals and integrate with any service or tool
  • And much much more...

Requirements

  • You have deployed a Linux Debian/Ubuntu Virtual Machine.
    • An instance with 4 vCPU and 8GB of RAM can handle approximately 100k events spread over a month
    • We highly recommend an instance with at least 6 vCPU and 12GB of RAM to handle any surges in event volume
    • You will also need at least 100 GB of SSD disk
  • You have set up a user on Virtual Machine.
    • The user must have sudo privileges without a password
  • You have set up an A record to connect a custom domain to your instance.
    • HumanSignals will automatically create an TLS certificate for your domain using LetsEncrypt
  • You have installed on your PC.
    • Git
    • Ansible

Installation

Download this repository using Git

$ git clone https://github.com/ivolga-tech/humansignals-devops.git
$ cd humansignals-devops/ansible

Create a hosts file with your data

Use this command to create a hosts file with your credentails such as destination IP and user with sudo privileges.

$ VM_IP=<destination IP> VM_USER=<sudo user> envsubst < hosts.example > hosts

Set your domain

  1. Open the group_vars/humansignals.yml
  2. Change the server.host value

Certificate configuration

If you want to use Let's Encrypt certificate:

  1. Open the group_vars/humansignals.yml
  2. Change the production value to true

Clickhouse configuration

If you want to use external ClickHouse cluster:

  1. Open the group_vars/humansignals.yml
  2. Change the clickhouse.internal value to false
  3. Set all other clickhouse.* values

ClickHouse version must be >=21.6.0 and <22.4.0. Humansignals may not work correctly with another version.

Install Ansible dependencies

$ ansible-galaxy install -r requirements.yml

Run Ansible Playbook

$ ansible-playbook -i hosts main.yml

Now you have to wait ~30 minutes.

Check your HumanSignals installation

  1. Open the web browser
  2. Enter your domain or use the default
  3. Congratulations! 🎉🎉🎉

Verify Installation

In this section we check that some of the required steps have been correctly installed and configured.

Check pods

Go to the server where are HumanSignals is deployed and run this command:

$ kubectl -n humansignals get pods

You will see something like this:

pods

All pods should be running or completed.

Check ingress

Go to your web browser and enter your domain.

ingress

If you do not have real domain, you need to edit your hosts file:

$ echo "<VM_IP> <HS_DOMAIN>" | sudo tee -a /etc/hosts

hosts

Troubleshooting

If something went wrong, this section covers a number of steps you can take to debug issues.

Check pods

You can use this command to check that all of our pods are running.

$ kubectl -n humansignals get pods

pods

Check logs

You can use this command to check the logs of our pods.

$ kubectl -n humansignals logs <pod_name> | less

The best way to start your research is with the web container, which runs all database migrations and gives you an error if any of them fail.

Frequently Asked Questions

Are the errors I'm seeing important?

Here are some examples of log spam that currently exists in our app and is safe to ignore:

The following messages in the ClickHouse pod happen when ClickHouse reshuffles how it consumes from the topics. So, anytime ClickHouse or Kafka restarts we'll get a bit of noise and the following log entries are safe to ignore:

<Error> TCPHandler: Code: 60, e.displayText() = DB::Exception: Table humansignals.sharded_events doesn't exist.
...
<Warning> StorageKafka (kafka_session_recording_events): Can't get assignment. It can be caused by some issue with consumer group (not enough partitions?). Will keep trying.

The following error is produced by some low-priority celery tasks and we haven't seen any actual impact so can safely be ignored.

TooManyConnections: too many connections
  File "humansignals/celery.py",
  ...
  File "clickhouse_pool/pool.py", line 102, in pull
    raise TooManyConnections("too many connections")

How do I see logs for a pod?

  1. Find the name of the pod you want to get logs on:
$ kubectl -n humansignals get pods

This command will list all running pods. If you want app/plugin server logs, for example, look for a pod that has a name starting with humansignals-plugins. This will be something like humansignals-plugins-b7759745d-kwb7b.

  1. Get the logs for that pod using the name from the previous step:
$ kubectl -n humansignals logs humansignals-plugins-b7759745d-kwb7b

How do I connect to the web server's shell?

HumanSignals is built on Django, which comes with some useful utilities. One of them is a Python shell. You can connect to it like so:

$ kubectl -n humansignals exec -it deploy/humansignals-web -c humansignals-web -- python manage.py shell_plus

In a moment you should see the shell load and finally a message like this appear:

Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>

That means you can now type Python code and run it with HumanSignals context (such as models) already loaded!

For example, to see the number of users in your instance run:

User.objects.count()

How do I connect to PostgreSQL?

  1. Find out your PostgreSQL password from the web pod:
$ kubectl -n humansignals exec -it deploy/humansignals-web -c humansignals-web -- sh -c 'echo The PostgreSQL password is: $POSTHOG_DB_PASSWORD'
  1. Connect to the humansignals database:

You're connecting to your production database, proceed with caution!

$ kubectl -n humansignals exec -it sts/humansignals-humansignals-postgresql -- psql -d humansignals -U postgres

PostgreSQL will ask you for the password. Use the value you found out in step 1.

Now you can run SQL queries! Just remember that an SQL query needs to be terminated with a semicolon ; to run.

How do I connect to ClickHouse?

  1. Find out your ClickHouse user and password from the web pod:
$ kubectl -n humansignals exec -it deploy/humansignals-web -c humansignals-web -- sh -c 'echo -e "User: $CLICKHOUSE_USER\nPassword: $CLICKHOUSE_PASSWORD"'
  1. Connect to the humansignals database:

You're connecting to your production database, proceed with caution!

$ kubectl -n humansignals exec -it sts/chi-humansignals-humansignals-0-0 -- clickhouse-client -d humansignals --user <user_from_step_1> --password <password_from_step_1>

How do I restart all pods for a service?

Important: Not all services can be safely restarted this way. It is safe to do this for the app/plugin server. If you have any doubts, ask someone from the HumanSignals team.

  1. Terminate all running pods for the service:
$ kubectl -n humansignals scale --replicas 0 deploy/humansignals-plugins
  1. Start new pods for the service:
$ kubectl -n humansignals scale --replicas 1 deploy/humansignals-plugins