matrix-docker-ansible-deploy/docs/maintenance-synapse.md

106 lines
7.4 KiB
Markdown
Raw Permalink Normal View History

# Synapse maintenance
This document shows you how to perform various maintenance tasks related to the Synapse chat server.
Table of contents:
- [Purging old data with the Purge History API](#purging-old-data-with-the-purge-history-api), for when you wish to delete in-use (but old) data from the Synapse database
- [Compressing state with rust-synapse-compress-state](#compressing-state-with-rust-synapse-compress-state)
- [Browse and manipulate the database](#browse-and-manipulate-the-database), for when you really need to take matters into your own hands
- [Make Synapse faster](#make-synapse-faster)
## Purging old data with the Purge History API
You can use the **[Purge History API](https://github.com/element-hq/synapse/blob/master/docs/admin_api/purge_history_api.md)** to delete old messages on a per-room basis. **This is destructive** (especially for non-federated rooms), because it means **people will no longer have access to history past a certain point**.
To make use of this Synapse Admin API, **you'll need an admin access token** first. Refer to the documentation on [how to obtain an access token](obtaining-access-tokens.md).
Synapse's Admin API is not exposed to the internet by default, following [official Synapse reverse-proxying recommendations](https://github.com/element-hq/synapse/blob/master/docs/reverse_proxy.md#synapse-administration-endpoints). To expose it you will need to add `matrix_synapse_container_labels_public_client_synapse_admin_api_enabled: true` to your `vars.yml` file.
2021-05-08 20:07:54 +00:00
Follow the [Purge History API](https://github.com/element-hq/synapse/blob/master/docs/admin_api/purge_history_api.md) documentation page for the actual purging instructions.
After deleting data, you may wish to run a [`FULL` Postgres `VACUUM`](./maintenance-postgres.md#vacuuming-postgresql).
## Compressing state with rust-synapse-compress-state
2021-05-08 20:07:54 +00:00
[rust-synapse-compress-state](https://github.com/matrix-org/rust-synapse-compress-state) can be used to optimize some `_state` tables used by Synapse. If your server participates in large rooms this is the most effective way to reduce the size of your database.
**Note**: besides running the `rust-synapse-compress-state` tool manually, you can also enable its `synapse-auto-compressor` tool by [Setting up synapse-auto-compressor](configuring-playbook-synapse-auto-compressor.md). The automatic tool will run on a schedule every day and you won't have to compress state manually ever again.
`rust-synapse-compress-state` should be safe to use (even when Synapse is running), but it's always a good idea to [make Postgres backups](./maintenance-postgres.md#backing-up-postgresql) first.
To ask the playbook to run rust-synapse-compress-state, execute:
```
ansible-playbook -i inventory/hosts setup.yml --tags=rust-synapse-compress-state
```
By default, all rooms with more than `100000` state group rows will be compressed.
If you need to adjust this, pass: `--extra-vars='matrix_synapse_rust_synapse_compress_state_min_state_groups_required=SOME_NUMBER_HERE'` to the command above.
After state compression, you may wish to run a [`FULL` Postgres `VACUUM`](./maintenance-postgres.md#vacuuming-postgresql).
## Browse and manipulate the database
When the [Synapse Admin API](https://github.com/element-hq/synapse/tree/master/docs/admin_api) and the other tools do not provide a more convenient way, having a look at synapse's postgresql database can satisfy a lot of admins' needs.
2021-05-08 20:07:54 +00:00
Editing the database manually is not recommended or supported by the Synapse developers. If you are going to do so you should [make a database backup](./maintenance-postgres.md#backing-up-postgresql).
First, set up an SSH tunnel to your matrix server (skip if it is your local machine):
```
# you may replace 1799 with an arbitrary port unbound on both machines
ssh -L 1799:localhost:1799 matrix.DOMAIN
```
2020-05-11 08:09:22 +00:00
Then start up an ephemeral [adminer](https://www.adminer.org/) container on the Matrix server, connecting it to the `matrix` network and linking the postgresql container:
```
docker run --rm --publish 1799:8080 --link matrix-postgres --net matrix adminer
```
You should then be able to browse the adminer database administration GUI at http://localhost:1799/ after entering your DB credentials (found in the `host_vars` or on the server in `{{matrix_synapse_config_dir_path}}/homeserver.yaml` under `database.args`)
⚠️ Be **very careful** with this, there is **no undo** for impromptu DB operations.
## Make Synapse faster
2021-04-22 11:22:07 +00:00
Synapse's presence feature which tracks which users are online and which are offline can use a lot of processing power. You can disable presence by adding `matrix_synapse_presence_enabled: false` to your `vars.yml` file.
If you have enough compute resources (CPU & RAM), you can make Synapse better use of them by [enabling load-balancing with workers](configuring-playbook-synapse.md#load-balancing-with-workers).
[Tuning your PostgreSQL database](maintenance-postgres.md#tuning-postgresql) could also improve Synapse performance. The playbook tunes the integrated Postgres database automatically, but based on your needs you may wish to adjust tuning variables manually. If you're using an [external Postgres database](configuring-playbook-external-postgres.md), you will aslo need to tune Postgres manually.
Increase Synapse caches and enable cache-autotuning by default (#3017) * Modify Synapse Cache Factor to use Auto Tune Synapse has the ability to as it calls in its config auto tune caches. This ability lets us set very high cache factors and then instead limit our resource use. Defaults for this commit are 1/10th of what Element apparently runs for EMS stuff and matrix.org on Cache Factor and upstream documentation defaults for auto tune. * Add vars to Synapse main.yml to control cache related config This commit adds various cache related vars to main.yml for Synapse. Some are auto tune and some are just adding explicit ways to control upstream vars. * Updated Auto Tune figures Autotuned figures have been bumped in consultation with other community members as to a reasonable level. Please note these defaults are more on the one of each workers side than they are on the monolith Side. * Fix YML Error The playbook is not happy with the previous state of this patch so this commit hopefully fixes it * Add to_json to various Synapse tuning related configs * Fix incorrect indication in homeserver.yaml.j2 * Minor cleanups * Synapse Cache Autotuning Documentation * Upgrade Synapse Cache Autotune to auto configure memory use * Update Synapse Tuning docs to reflect automatic memory use configuration * Fix Linting errors in synapses main.yml * Rename variables for consistency (matrix_synapse_caches_autotuning_* -> matrix_synapse_cache_autotuning_*) * Remove FIX ME comment about Synapse's `cache_autotuning` `docs/maintenance-synapse.md` and `roles/custom/matrix-synapse/defaults/main.yml` already contains documentation about these variables and the default values we set. * Improve "Tuning caches and cache autotuning" documentation for Synapse * Announce larger Synapse caches and cache auto-tuning --------- Co-authored-by: Slavi Pantaleev <slavi@devture.com>
2024-02-14 10:02:06 +00:00
### Tuning caches and cache autotuning
Tuning Synapse's cache factor is useful for performance increases but also as part of controlling Synapse's memory use. Use the variable `matrix_synapse_caches_global_factor` to set the cache factor as part of this process.
**The playbook defaults the global cache factor to a large value** (e.g. `10`). A smaller value (e.g. `0.5`) will decrease the amount used for caches, but will [not necessarily decrease RAM usage as a whole](https://github.com/matrix-org/synapse/issues/3939).
Tuning the cache factor is useful only to a limited degree (as its crude to do in isolation) and therefore users who are tuning their cache factor should likely look into tuning autotune variables as well (see below).
Cache autotuning is **enabled by default** and controlled via the following variables:
Fix Synapse cache auto-tuning variables to use bytes, not KB Fixup for https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3017 This reverts 1cd82cf06823fb7680 and also multiplies results by `1024` so as to pass bytes to Synapse, not KB (as done before). 1cd82cf06823fb7680 was correctly documenting what we were doing (passing KB values), but that's incorrect. Synapse's Config Conventions (https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#config-conventions) are supposed to clear it up, but they don't currently state what happens when you pass a plain number (without a unit suffix). Thankfully, the source code tells us: https://github.com/element-hq/synapse/blob/bc1db16086d0718c9c0bb61b32b825ba62049bb0/synapse/config/_base.py#L181-L206 > If an integer is provided it is treated as bytes and is unchanged. > > String byte sizes can have a suffix of ... > No suffix is understood as a plain byte count. We were previously passing strings, but that has been improved in 3d73ec887aec27483. Regardless, non-suffixed values seem to be treated as bytes by Synapse, so this patch changes the variables to use bytes. Moreover, we're moving from `matrix_synapse_memtotal_kb` to `matrix_synapse_cache_size_calculations_memtotal_bytes` as working with the base unit everywhere is preferrable. Here, we also introduce 2 new variables to allow for the caps to be tweaked: - `matrix_synapse_cache_size_calculations_max_cache_memory_usage_cap_bytes` - `matrix_synapse_cache_size_calculations_target_cache_memory_usage_cap_bytes`
2024-02-14 11:39:40 +00:00
- `matrix_synapse_cache_autotuning_max_cache_memory_usage` - defaults to 1/8 of total RAM with a cap of 2GB; values are specified in bytes
- `matrix_synapse_cache_autotuning_target_cache_memory_usage` - defaults to 1/16 of total RAM with a cap of 1GB; values are specified in bytes
Increase Synapse caches and enable cache-autotuning by default (#3017) * Modify Synapse Cache Factor to use Auto Tune Synapse has the ability to as it calls in its config auto tune caches. This ability lets us set very high cache factors and then instead limit our resource use. Defaults for this commit are 1/10th of what Element apparently runs for EMS stuff and matrix.org on Cache Factor and upstream documentation defaults for auto tune. * Add vars to Synapse main.yml to control cache related config This commit adds various cache related vars to main.yml for Synapse. Some are auto tune and some are just adding explicit ways to control upstream vars. * Updated Auto Tune figures Autotuned figures have been bumped in consultation with other community members as to a reasonable level. Please note these defaults are more on the one of each workers side than they are on the monolith Side. * Fix YML Error The playbook is not happy with the previous state of this patch so this commit hopefully fixes it * Add to_json to various Synapse tuning related configs * Fix incorrect indication in homeserver.yaml.j2 * Minor cleanups * Synapse Cache Autotuning Documentation * Upgrade Synapse Cache Autotune to auto configure memory use * Update Synapse Tuning docs to reflect automatic memory use configuration * Fix Linting errors in synapses main.yml * Rename variables for consistency (matrix_synapse_caches_autotuning_* -> matrix_synapse_cache_autotuning_*) * Remove FIX ME comment about Synapse's `cache_autotuning` `docs/maintenance-synapse.md` and `roles/custom/matrix-synapse/defaults/main.yml` already contains documentation about these variables and the default values we set. * Improve "Tuning caches and cache autotuning" documentation for Synapse * Announce larger Synapse caches and cache auto-tuning --------- Co-authored-by: Slavi Pantaleev <slavi@devture.com>
2024-02-14 10:02:06 +00:00
- `matrix_synapse_cache_autotuning_min_cache_ttl` - defaults to `30s`
You can **learn more about cache-autotuning and the global cache factor settings** in the [Synapse's documentation on caches and associated values](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#caches-and-associated-values).
To **disable cache auto-tuning**, unset all values:
```yml
matrix_synapse_cache_autotuning_max_cache_memory_usage: ''
matrix_synapse_cache_autotuning_target_cache_memory_usage: ''
matrix_synapse_cache_autotuning_min_cache_ttl: ''
```
Users who wish to lower Synapse's RAM footprint should look into lowering the global cache factor and tweaking the autotune variables (or disabling auto-tuning). If your cache factor is too low for a given auto tune setting your caches will not reach autotune thresholds and autotune won't be able to do its job. Therefore, when auto-tuning is enabled (which it is by default), it's recommended to have your cache factor be large.
See also [How do I optimize this setup for a low-power server?](faq.md#how-do-i-optimize-this-setup-for-a-low-power-server).