Upgrade Postgres (v16.0-5 -> v16.0-6) - adds auto-tuning

CHANGELOG.md

@@ -1,3 +1,16 @@
+# 2023-10-18
+
+## Postgres parameters are automatically tuned now
+
+The playbook has provided some hints about [Tuning PostgreSQL](docs/maintenance-postgres.md#tuning-postgresql) for quite a while now.
+
+From now on, the [Postgres Ansible role](https://github.com/devture/com.devture.ansible.role.postgres) automatically tunes your Postgres configuration with the same [calculation logic](https://github.com/le0pard/pgtune/blob/master/src/features/configuration/configurationSlice.js) that powers https://pgtune.leopard.in.ua/.
+
+Our [Tuning PostgreSQL](docs/maintenance-postgres.md#tuning-postgresql) documentation page has details about how you can turn auto-tuning off or adjust the automatically-determined Postgres configuration parameters manually.
+
+People who [enable load-balancing with Synapse workers](docs/configuring-playbook-synapse.md#load-balancing-with-workers) no longer need to increase the maximum number of Postgres connections manually (previously done via `devture_postgres_process_extra_arguments`). There's a new variable (`devture_postgres_max_connections`) for controlling this number and the playbook automatically raises its value from `200` to `500` for setups which enable workers.
+
+
 # 2023-08-31
 
 ## SchildiChat support

docs/configuring-playbook-synapse.md

@@ -34,13 +34,7 @@ We support a few configuration presets (`matrix_synapse_workers_preset: one-of-e
 
 If you'd like more customization power, you can start with one of the presets and tweak various `matrix_synapse_workers_*_count` variables manually.
 
-If you increase worker counts too much, you may need to increase the maximum number of Postgres connections too (example):
-
-```yaml
-devture_postgres_process_extra_arguments: [
-  "-c 'max_connections=200'"
-]
-```
+When Synapse workers are enabled, the integrated [Postgres database is tuned](maintenance-postgres.md#tuning-postgresql), so that the maximum number of Postgres connections are increased from `200` to `500`. If you need to decrease or increase the number of maximum Postgres connections further, use the `devture_postgres_max_connections` variable.
 
 In case any problems occur, make sure to have a look at the [list of synapse issues about workers](https://github.com/matrix-org/synapse/issues?q=workers+in%3Atitle) and your `journalctl --unit 'matrix-*'`.
 

docs/maintenance-postgres.md

@@ -106,63 +106,15 @@ Example: `--extra-vars="postgres_dump_name=matrix-postgres-dump.sql"`
 
 ## Tuning PostgreSQL
 
-PostgreSQL can be tuned to make it run faster. This is done by passing extra arguments to Postgres with the `devture_postgres_process_extra_arguments` variable. You should use a website like https://pgtune.leopard.in.ua/ or information from https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server to determine what Postgres settings you should change.
+PostgreSQL can be [tuned](https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server) to make it run faster. This is done by passing extra arguments to the Postgres process.
 
-**Note**: the configuration generator at https://pgtune.leopard.in.ua/ adds spaces around the `=` sign, which is invalid. You'll need to remove it manually (`max_connections = 300` -> `max_connections=300`)
+The [Postgres Ansible role](https://github.com/devture/com.devture.ansible.role.postgres) **already does some tuning by default**, which matches the [tuning logic](https://github.com/le0pard/pgtune/blob/master/src/features/configuration/configurationSlice.js) done by websites like https://pgtune.leopard.in.ua/.
+You can manually influence some of the tuning variables . These parameters (variables) are injected via the `devture_postgres_postgres_process_extra_arguments_auto` variable.
 
-### Here are some examples:
+Most users should be fine with the automatically-done tuning. However, you may wish to:
 
-These are not recommended values and they may not work well for you. This is just to give you an idea of some of the options that can be set. If you are an experienced PostgreSQL admin feel free to update this documentation with better examples.
+- **adjust the automatically-deterimned tuning parameters manually**: change the values for the tuning variables defined in the Postgres role's [default configuration file](https://github.com/devture/com.devture.ansible.role.postgres/blob/main/defaults/main.yml) (see `devture_postgres_max_connections`, `devture_postgres_data_storage` etc). These variables are ultimately passed to Postgres via a `devture_postgres_postgres_process_extra_arguments_auto` variable
 
-Here is an example config for a small 2 core server with 4GB of RAM and SSD storage:
-```
-devture_postgres_process_extra_arguments: [
-  "-c shared_buffers=128MB",
-  "-c effective_cache_size=2304MB",
-  "-c effective_io_concurrency=100",
-  "-c random_page_cost=2.0",
-  "-c min_wal_size=500MB",
-]
-```
+- **turn automatically-performed tuning off**: override it like this: `devture_postgres_postgres_process_extra_arguments_auto: []`
 
-Here is an example config for a 4 core server with 8GB of RAM on a Virtual Private Server (VPS); the paramters have been configured using https://pgtune.leopard.in.ua with the following setup: PostgreSQL version 12, OS Type: Linux, DB Type: Mixed type of application, Data Storage: SSD storage:
-```
-devture_postgres_process_extra_arguments: [
-  "-c max_connections=100",
-  "-c shared_buffers=2GB",
-  "-c effective_cache_size=6GB",
-  "-c maintenance_work_mem=512MB",
-  "-c checkpoint_completion_target=0.9",
-  "-c wal_buffers=16MB",
-  "-c default_statistics_target=100",
-  "-c random_page_cost=1.1",
-  "-c effective_io_concurrency=200",
-  "-c work_mem=5242kB",
-  "-c min_wal_size=1GB",
-  "-c max_wal_size=4GB",
-  "-c max_worker_processes=4",
-  "-c max_parallel_workers_per_gather=2",
-  "-c max_parallel_workers=4",
-  "-c max_parallel_maintenance_workers=2",
-]
-```
-
-Here is an example config for a large 6 core server with 24GB of RAM:
-```
-devture_postgres_process_extra_arguments: [
-  "-c max_connections=40",
-  "-c shared_buffers=1536MB",
-  "-c checkpoint_completion_target=0.7",
-  "-c wal_buffers=16MB",
-  "-c default_statistics_target=100",
-  "-c random_page_cost=1.1",
-  "-c effective_io_concurrency=100",
-  "-c work_mem=2621kB",
-  "-c min_wal_size=1GB",
-  "-c max_wal_size=4GB",
-  "-c max_worker_processes=6",
-  "-c max_parallel_workers_per_gather=3",
-  "-c max_parallel_workers=6",
-  "-c max_parallel_maintenance_workers=3",
-]
-```
+- **add additional tuning parameters**: define your additional Postgres configuration parameters in `devture_postgres_postgres_process_extra_arguments_custom`. See `devture_postgres_postgres_process_extra_arguments_auto` defined in the Postgres role's [default configuration file](https://github.com/devture/com.devture.ansible.role.postgres/blob/main/defaults/main.yml) for inspiration

docs/maintenance-synapse.md

@@ -72,8 +72,10 @@ You should then be able to browse the adminer database administration GUI at htt
 
 Synapse's presence feature which tracks which users are online and which are offline can use a lot of processing power. You can disable presence by adding `matrix_synapse_presence_enabled: false` to your `vars.yml` file.
 
+If you have enough compute resources (CPU & RAM), you can make Synapse better use of them by [enabling load-balancing with workers](configuring-playbook-synapse.md#load-balancing-with-workers).
+
 Tuning Synapse's cache factor can help reduce RAM usage. [See the upstream documentation](https://github.com/matrix-org/synapse#help-synapse-is-slow-and-eats-all-my-ram-cpu) for more information on what value to set the cache factor to. Use the variable `matrix_synapse_caches_global_factor` to set the cache factor.
 
-Tuning your PostgreSQL database will also make Synapse run significantly faster. See [maintenance-postgres.md##tuning-postgresql](maintenance-postgres.md##tuning-postgresql).
+[Tuning your PostgreSQL database](maintenance-postgres.md#tuning-postgresql) could also improve Synapse performance. The playbook tunes the integrated Postgres database automatically, but based on your needs you may wish to adjust tuning variables manually. If you're using an [external Postgres database](configuring-playbook-external-postgres.md), you will aslo need to tune Postgres manually.
 
 See also [How do I optimize this setup for a low-power server?](faq.md#how-do-i-optimize-this-setup-for-a-low-power-server).

group_vars/matrix_servers

@@ -2964,6 +2964,8 @@ devture_postgres_db_name: matrix
 
 devture_postgres_systemd_services_to_stop_for_maintenance_list_auto: "{{ devture_systemd_service_manager_services_list_auto | map(attribute='name') | reject('equalto', (devture_postgres_identifier + '.service')) }}"
 
+devture_postgres_max_connections: "{{ 500 if matrix_synapse_workers_enabled else 200 }}"
+
 devture_postgres_managed_databases_auto: |
   {{
     ([{

requirements.yml

@@ -16,7 +16,7 @@
 - src: git+https://github.com/devture/com.devture.ansible.role.playbook_state_preserver.git
   version: ff2fd42e1c1a9e28e3312bbd725395f9c2fc7f16
 - src: git+https://github.com/devture/com.devture.ansible.role.postgres.git
-  version: v16.0-5
+  version: v16.0-6
 - src: git+https://github.com/devture/com.devture.ansible.role.postgres_backup.git
   version: a0cc7c1c696872ba8880d9c5e5a54098de825030
 - src: git+https://github.com/devture/com.devture.ansible.role.systemd_docker_base.git