Running synapse on Kubernetes

  1. Overview
    1. About this Tutorial
  2. Bootstrap
    1. Upgrade the host
    2. Firewall configuration
    3. Kubernetes installation
      1. Create a directory for local (stateful) storage
  3. Create a synapse Docker Image
    1. Building the image
    2. Extracting the Configuration & Generating a Signing Key
  4. Configuring synapse
    1. Main Configuration File
    2. Configuring the logger
    3. Create a Certificate
  5. Running it
    1. Requirements
      1. Data tree
    2. Edit and apply the configuration
  6. Rejoice!

Overview

Matrix is intriguing to me from both a community and technological aspect, but for this purpose, I wanted to run my own community for friends and family. What this entry goes into is the configuration, from setup of the initial digital ocean droplet/VM to the deployment of synapse on Kubernetes. TURN setup will be covered in a future article; so if you’re looking for that, come back for updates! :D

What does the Matrix protocol provide? A very high level overview compared to IRC:

  • Offline chat logs (sync on connect, like slack/discord/others)
  • Voice/Video chat (this involves the aforementioned TURN server)
  • End-to-end encryption of channels and private messages
  • User accounts & roles (with multiple back-ends to integrate with your existing environments)
  • Federation with other servers

And a lot more. Matrix and synapse are quite the bears to get going primarily because of all the features it supports.

Synapse is the premier software that powers Matrix protocol servers. It is primarily written in python and uses YAML configuration, as well as a database and file storage. In this article we will account for most of the basic storage areas as well as touch on network management a little.

About this Tutorial

I expect this to change over time and I will revise it or rewrite it as needed. This document is current as of Aug 6, 2020.

Bootstrap

Digital Ocean will be providing our VM today for our experiment. Some of the context around installation will be related to that. Rest assured, most of this article will be applicable to anyone looking to set up synapse in Kubernetes, no specific cloud technology is used.

Our installation is a Ubuntu 20.04 with DO’s basic plan $40/mo option, which is currently rated at 8GB ram, 4 vcpu, and 160GB storage.

My host will be named int throughout the tutorial.

This is a hobby server, and while appropriate measures are taken to ensure security, uptime and scalability are kind of thrown out the door. If want to scale synapse, that’s going to take more effort.

Upgrade the host

Get all the latest dependencies:

apt update && apt dist-upgrade -y && reboot

Firewall configuration

For this tutorial, matrix only needs port 443 ingress; egress should be unlimited. We have disabled traefik in our example to enable our load balancer and hard-code the IP.

Kubernetes installation

We are going to use k3s to set up our installation of Kubernetes since it is great for single-node installations. To do this, you can follow the instructions on the website; or simply:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik" sh -

After a minute or two, you should be able to k3s kubectl get nodes to see the “cluster” in action. Here’s what ours looks like:

root@int:~# curl -sfL https://get.k3s.io | sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.18.6+k3s1 as release
[INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v1.18.6+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/rancher/k3s/releases/download/v1.18.6+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
root@int:~# k3s kubectl get nodes
NAME   STATUS   ROLES    AGE   VERSION
int    Ready    master   28s   v1.18.6+k3s1

Create a directory for local (stateful) storage

Call it /data, for the purposes of lining up with our manifests.

Create a synapse Docker Image

First order of business is to create a docker image. We can do that by starting with an ubuntu image; there are good packages for it provided by the matrix.org folks that we can leverage.

In this image we do a few small things:

  1. First, we set the timezone and set the apt frontend to not prompt us during installation. We are going to muck with most of the configuration anyway, and the tzinfo package will prompt mid-build (and lock it up) if this is not set.
  2. Second, we get our keyring from matrix.org and install it and the package we need to run synapse.
  3. Finally, we set the command to not daemonize so it occupies PID 1, and point it at our configuration, which will be in /data/synapse/homeserver.yaml.
  4. We do not copy any data into the image.
FROM ubuntu:latest

ENV TZ=America/Los_Angeles DEBIAN_FRONTEND=noninteractive
RUN apt-get update -qq && apt-get install -y lsb-release wget apt-transport-https

RUN wget -O /usr/share/keyrings/matrix-org-archive-keyring.gpg https://packages.matrix.org/debian/matrix-org-archive-keyring.gpg
RUN echo "deb [signed-by=/usr/share/keyrings/matrix-org-archive-keyring.gpg] https://packages.matrix.org/debian/ $(lsb_release -cs) main" | \
    tee /etc/apt/sources.list.d/matrix-org.list
RUN apt-get update -qq && apt-get install -y matrix-synapse-py3

CMD ["synctl", "--no-daemonize", "start", "/data/synapse/homeserver.yaml" ]

Building the image

Place the contents of the Dockerfile in a file in an empty directory. Since setting up a registry is beyond the scope for this article, we’ll assume our images are to be published to Docker Hub. Let’s call this image synapse, but the full name will be dependent on what account you use on Docker Hub.

docker build -t erikh/synapse . && docker push erikh/synapse:latest

Extracting the Configuration & Generating a Signing Key

So, one thing that would be incredibly useful to you, is to have the latest annotated configuration file available for reading. Especially later; I will give you an abridged configuration file, you should be aware of all the options.

Additionally, you need to generate a fresh signing key for your chat server so it can perform encrypted operations with the clients. Please note that one will be in the image, as this is public (we just published it to docker hub!) you will not want to use it.

To do all this, we’ll bind mount /tmp/config in the container to /tmp/config outside of it, and then run our command that points everything at that directory. The configuration and signing key will be in /tmp/config on your host.

I know this command is a mouthful; but what it’s doing here is running our image, running the python in the virtualenv provided by the package we installed, then running our key/config generation tool to populate the directory. Since the directory is bind-mounted, we’ll see the changes on the host.

docker run \
  -v /tmp/config:/tmp/config \
  -it erikh/synapse \
  /opt/venvs/matrix-synapse/bin/python \
    -m synapse.app.homeserver \
    --keys-directory /tmp/config --generate-keys \
    --generate-config -H chat.foo.com \
    --report-stats no \
    -c /tmp/config/homeserver.yaml

You should have, among other things, a homeserver.yaml and a <domain>.signing.key in the newly created /tmp/config directory in your current working directory. This is your primary configuration and signing key. Copy the key for later use.

Configuring synapse

Synapse has a LOT of configuration knobs and we’ll only be covering a small amount here; the ones that pertain to our infrastructure.

Here, we’re calling our server chat.foo.com – users will point their Element clients at this endpoint to talk to others. It is not a federated server that talks to matrix.org or other matrix servers.

We’re also using Postgres and Redis here to solve some of the state management that is required by synapse.

Main Configuration File

This is only a very small part of the synapse configuration; please review the example set above you extracted from the image for a LOT more to configure!

Also, be sure to edit the parts that say <shared secret>, as they need generated passwords. For each field, try something like apg to generate it:

apt-get install apg -y
apg -a 1 -m 64

If you make any adjustments to the listeners section, the Kubernetes service load balancer manifest will also need to be edited. Same goes for changing any of the paths, with regard to storage.

server_name: "chat.foo.com"
public_baseurl: "https://chat.foo.com/"
use_presence: true
default_room_version: "6"
limit_profile_requests_to_users_who_share_rooms: true
pid_file: /var/run/homeserver.pid
# NOTE: this is your TCP listener configuration. You'll want to
# read the docs if you modify this portion in particular.
listeners:
  - port: 8448
    type: http
    tls: true
    x_forwarded: true
    resources:
      - names: [client]
tls_certificate_path: "/data/synapse/tls/fullchain.pem"
tls_private_key_path: "/data/synapse/tls/privkey.pem"
acme:
  enabled: false
send_federation: false
federation_ip_range_blacklist:
  - "0.0.0.0/0"
database:
  name: psycopg2
  args:
    user: synapse
    password: synapse
    database: synapse
    host: postgres
    # connection pool parameters
    cp_min: 5
    cp_max: 10
# NOTE: this is a separate configuration file; more on this below
log_config: "/data/synapse/log.config"
media_store_path: "/data/synapse/media"
max_upload_size: 100M
max_image_pixels: 32M
enable_registration: true
session_lifetime: 2w
auto_join_rooms:
  - "#general:chat.foo.com"
# NOTE: generate these with a tool like `apg`
macaroon_secret_key: "<shared secret>"
form_secret: "<shared secret>"
signing_key_path: "/data/synapse/chat.foo.com.signing.key"
key_refresh_interval: 1d
trusted_key_servers: []
push:
  include_content: true
encryption_enabled_by_default_for_room_type: all
enable_group_creation: true
group_creation_prefix: "community/"
user_directory:
  enabled: true
  search_all_users: false
redis:
  enabled: true
  host: redis
  port: 6379
url_preview_enabled: true
max_spider_size: 10M
url_preview_accept_language:
  - en-US
url_preview_ip_range_blacklist:
  - "127.0.0.0/8"
  - "10.0.0.0/8"
  - "172.16.0.0/12"
  - "192.168.0.0/16"
  - "100.64.0.0/10"
  - "169.254.0.0/16"
  - "::1/128"
  - "fe80::/64"
  - "fc00::/7"
password_config:
  pepper: "<shared secret>"
  policy:
    enabled: true
    minimum_length: 8
    require_digit: true
    require_symbol: true
    require_lowercase: true
    require_uppercase: true
report_stats: false

Configuring the logger

We primarily want the logger to spit to stdout, saving our storage for other things. This configuration gets us mostly there.

version: 1
formatters:
  precise:
    format: "%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s"
filters:
  context:
    (): synapse.logging.context.LoggingContextFilter
    request: ""
handlers:
  console:
    class: logging.StreamHandler
    formatter: precise
    filters: [context]
loggers:
  synapse.storage.SQL:
    # beware: increasing this to DEBUG will make synapse log sensitive
    # information such as access tokens.
    level: INFO
root:
  level: INFO
  handlers: [console]
disable_existing_loggers: false

Create a Certificate

If you have your own certs that were generated in some other way, just put them in /data/synapse/tls.

We’ll use certbot to generate our cert, and copy our files directly into the /data/synapse/tls directory so they can be referenced. A better implementation would be to embed these as disposable secrets.

Be sure to enable port 80 in your firewall!

apt install certbot -y
certbot certonly -d chat.foo.com --standalone
cd /etc/letsencrypt/live/chat.foo.com
cp fullchain.pem privkey.pem /data/synapse/tls

Running it

Once all the files are in place, we can launch our set of services. I’ll spare you the line-by-line; you should just be able to kubectl apply -f this YAML with one edit: to the public IP of the load balancer at the bottom of the text.

Requirements

By this point, you should have:

  • A docker image
  • Configuration created
  • Signing key generated & copied
  • Certificates generated
  • Firewall accepting port 443 (https) on the local IP.

Data tree

Your /data tree should look something like:

/data/postgres
/data/synapse
/data/synapse/homeserver.yaml
/data/synapse/log.config
/data/synapse/chat.foo.com.signing.key
/data/synapse/tls
/data/synapse/tls/fullchain.pem
/data/synapse/tls/privkey.pem

Edit and apply the configuration

You should not apply this verbatim. Edit it!

  • Node int should be changed to your Node name
  • IP in load balancer should be changed
  • Images should be changed to use something other than mine! :D

The configuration creates a namespace called matrix and stuffs a number of details (pods, services, etc) underneath it. It also creates PVs for the mountpoints of /data/synapse and /data/postgres.

You can continue launching this likely get a working result! :D You can find a gist here for downloading.

Rejoice!

Assuming you’ve accomplished all of the above; you should see this at the https:// URL of your matrix server:

matrix server!

Congratulations! Point your Element client at it and have a blast!