From 038d046612f1d225c8aecfe15a43882c8be94cd6 Mon Sep 17 00:00:00 2001 From: Suguru Hirahara Date: Thu, 30 Jan 2025 13:29:38 +0900 Subject: [PATCH 01/10] Update docs/maintenance-and-troubleshooting.md: tidy up Signed-off-by: Suguru Hirahara --- docs/maintenance-and-troubleshooting.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/maintenance-and-troubleshooting.md b/docs/maintenance-and-troubleshooting.md index fa38df727..ed164997e 100644 --- a/docs/maintenance-and-troubleshooting.md +++ b/docs/maintenance-and-troubleshooting.md @@ -14,14 +14,14 @@ sudo systemctl status matrix-synapse Docker containers that the playbook configures are supervised by [systemd](https://wiki.archlinux.org/title/Systemd) and their logs are configured to go to [systemd-journald](https://wiki.archlinux.org/title/Systemd/Journal). -To prevent double-logging, Docker logging is disabled by explicitly passing `--log-driver=none` to all containers. Due to this, you **cannot** view logs using `docker logs`. - -To view systemd-journald logs using [journalctl](https://man.archlinux.org/man/journalctl.1), run a command like this: +To view systemd-journald logs using [journalctl](https://man.archlinux.org/man/journalctl.1), log in to the server with SSH and run a command like this: ```sh sudo journalctl -fu matrix-synapse ``` +**Note**: to prevent double-logging, Docker logging is disabled by explicitly passing `--log-driver=none` to all containers. Due to this, you **cannot** view logs using `docker logs`. + ## How to check if services work The playbook can perform a check to ensure that you've configured things correctly and that services are running. @@ -40,7 +40,7 @@ Besides this self-check, you can also check whether your server federates with t ## Remove unused Docker data -You can free some disk space from Docker, see [docker system prune](https://docs.docker.com/engine/reference/commandline/system_prune/) for more information. +You can free some disk space from Docker by removing its unused data. See [docker system prune](https://docs.docker.com/engine/reference/commandline/system_prune/) for more information. ```sh ansible-playbook -i inventory/hosts setup.yml --tags=run-docker-prune From 61c9d4c55c49966b4b8bcbc684a1118c24b95fda Mon Sep 17 00:00:00 2001 From: Suguru Hirahara Date: Sat, 1 Feb 2025 15:48:59 +0900 Subject: [PATCH 02/10] Update docs/faq.md and docs/maintenance-and-troubleshooting.md: adopt the common description Signed-off-by: Suguru Hirahara --- docs/faq.md | 2 +- docs/maintenance-and-troubleshooting.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index c41c7fe11..bd537aa69 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -419,7 +419,7 @@ Available service names can be seen by doing `ls /etc/systemd/system/matrix*.ser Some services also log to files in `/matrix/*/data/..`, but we're slowly moving away from that. -We also disable Docker logging, so you can't use `docker logs matrix-*` either. We do this to prevent useless double (or even triple) logging and to avoid having to rotate log files. +To prevent double-logging, Docker logging is disabled by explicitly passing `--log-driver=none` to all containers. Due to this, you **cannot** view logs using `docker logs matrix-*`. We just simply delegate logging to journald and it takes care of persistence and expiring old data. diff --git a/docs/maintenance-and-troubleshooting.md b/docs/maintenance-and-troubleshooting.md index ed164997e..e6439a3bb 100644 --- a/docs/maintenance-and-troubleshooting.md +++ b/docs/maintenance-and-troubleshooting.md @@ -20,7 +20,7 @@ To view systemd-journald logs using [journalctl](https://man.archlinux.org/man/j sudo journalctl -fu matrix-synapse ``` -**Note**: to prevent double-logging, Docker logging is disabled by explicitly passing `--log-driver=none` to all containers. Due to this, you **cannot** view logs using `docker logs`. +**Note**: to prevent double-logging, Docker logging is disabled by explicitly passing `--log-driver=none` to all containers. Due to this, you **cannot** view logs using `docker logs matrix-*`. ## How to check if services work From ecdf370cb77553568bf708a51adf544af56859c8 Mon Sep 17 00:00:00 2001 From: Suguru Hirahara Date: Sat, 1 Feb 2025 16:19:10 +0900 Subject: [PATCH 03/10] Update docs/faq.md and docs/maintenance-and-troubleshooting.md: move the troubleshooting section from the FAQ page to the dedicated page As the theme deserves the dedicated page and we already have it, it seems sensible to move the topic from the general FAQ page. Signed-off-by: Suguru Hirahara --- docs/faq.md | 33 +------------------------ docs/maintenance-and-troubleshooting.md | 25 +++++++++++++++++-- 2 files changed, 24 insertions(+), 34 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index bd537aa69..502e40b0d 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -407,40 +407,9 @@ If you're running Ansible from within a container (one of the possibilities we l ### I get "Error response from daemon: configured logging driver does not support reading" when I do `docker logs matrix-synapse`. -See [How can I see the logs?](#how-can-i-see-the-logs). - -### How can I see the logs? - -We utilize [systemd/journald](https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html#Description) for logging. - -To see logs for Synapse, run `journalctl -fu matrix-synapse.service`. You may wish to see the [manual page for journalctl](https://www.commandlinux.com/man-page/man1/journalctl.1.html). - -Available service names can be seen by doing `ls /etc/systemd/system/matrix*.service` on the server. - -Some services also log to files in `/matrix/*/data/..`, but we're slowly moving away from that. - To prevent double-logging, Docker logging is disabled by explicitly passing `--log-driver=none` to all containers. Due to this, you **cannot** view logs using `docker logs matrix-*`. -We just simply delegate logging to journald and it takes care of persistence and expiring old data. - -Also see: [How long do systemd/journald logs persist for?](#how-long-do-systemdjournald-logs-persist-for) - -### How long do systemd/journald logs persist for? - -On some distros, the journald logs are just in-memory and not persisted to disk. - -Consult (and feel free to adjust) your distro's journald logging configuration in `/etc/systemd/journald.conf`. - -To enable persistence and put some limits on how large the journal log files can become, adjust your configuration like this: - -```ini -[Journal] -RuntimeMaxUse=200M -SystemMaxUse=1G -RateLimitInterval=0 -RateLimitBurst=0 -Storage=persistent -``` +See [this section](maintenance-and-troubleshooting.md#how-to-see-the-logs) on the page for maintenance and troubleshooting for more details to see the logs. ## Maintenance diff --git a/docs/maintenance-and-troubleshooting.md b/docs/maintenance-and-troubleshooting.md index e6439a3bb..377eca51a 100644 --- a/docs/maintenance-and-troubleshooting.md +++ b/docs/maintenance-and-troubleshooting.md @@ -12,15 +12,36 @@ sudo systemctl status matrix-synapse Active: active (running) since Sun 2024-01-14 09:13:06 UTC; 1h 31min ago ``` +## How to see the logs + Docker containers that the playbook configures are supervised by [systemd](https://wiki.archlinux.org/title/Systemd) and their logs are configured to go to [systemd-journald](https://wiki.archlinux.org/title/Systemd/Journal). -To view systemd-journald logs using [journalctl](https://man.archlinux.org/man/journalctl.1), log in to the server with SSH and run a command like this: +For example, you can find the logs of `matrix-synapse` in `systemd-journald` by logging in to the server with SSH and running the command as below: ```sh sudo journalctl -fu matrix-synapse ``` -**Note**: to prevent double-logging, Docker logging is disabled by explicitly passing `--log-driver=none` to all containers. Due to this, you **cannot** view logs using `docker logs matrix-*`. +Available service names can be seen by doing `ls /etc/systemd/system/matrix*.service` on the server. Some services also log to files in `/matrix/*/data/..`, but we're slowly moving away from that. + +We just simply delegate logging to journald and it takes care of persistence and expiring old data. + +### Enable systemd/journald logs persistence + +On some distros, the journald logs are just in-memory and not persisted to disk. + +Consult (and feel free to adjust) your distro's journald logging configuration in `/etc/systemd/journald.conf`. + +To enable persistence and put some limits on how large the journal log files can become, adjust your configuration like this: + +```ini +[Journal] +RuntimeMaxUse=200M +SystemMaxUse=1G +RateLimitInterval=0 +RateLimitBurst=0 +Storage=persistent +``` ## How to check if services work From bd794e8c2c39020724474dacf25b65f0280ca934 Mon Sep 17 00:00:00 2001 From: Suguru Hirahara Date: Sat, 1 Feb 2025 16:21:21 +0900 Subject: [PATCH 04/10] Update maintenance-and-troubleshooting.md: create sections "Maintenance" and "Troubleshooting" Signed-off-by: Suguru Hirahara --- docs/maintenance-and-troubleshooting.md | 40 ++++++++++++++----------- 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/docs/maintenance-and-troubleshooting.md b/docs/maintenance-and-troubleshooting.md index 377eca51a..460144183 100644 --- a/docs/maintenance-and-troubleshooting.md +++ b/docs/maintenance-and-troubleshooting.md @@ -1,6 +1,24 @@ # Maintenance and Troubleshooting -## How to see the current status of your services +## Maintenance + +### Remove unused Docker data + +You can free some disk space from Docker by removing its unused data. See [docker system prune](https://docs.docker.com/engine/reference/commandline/system_prune/) for more information. + +```sh +ansible-playbook -i inventory/hosts setup.yml --tags=run-docker-prune +``` + +The shortcut command with `just` program is also available: `just run-tags run-docker-prune` + +### Postgres + +See the dedicated [PostgreSQL Maintenance](maintenance-postgres.md) documentation page. + +## Troubleshooting + +### How to see the current status of your services You can check the status of your services by using `systemctl status`. Example: @@ -12,7 +30,7 @@ sudo systemctl status matrix-synapse Active: active (running) since Sun 2024-01-14 09:13:06 UTC; 1h 31min ago ``` -## How to see the logs +### How to see the logs Docker containers that the playbook configures are supervised by [systemd](https://wiki.archlinux.org/title/Systemd) and their logs are configured to go to [systemd-journald](https://wiki.archlinux.org/title/Systemd/Journal). @@ -26,7 +44,7 @@ Available service names can be seen by doing `ls /etc/systemd/system/matrix*.ser We just simply delegate logging to journald and it takes care of persistence and expiring old data. -### Enable systemd/journald logs persistence +#### Enable systemd/journald logs persistence On some distros, the journald logs are just in-memory and not persisted to disk. @@ -43,7 +61,7 @@ RateLimitBurst=0 Storage=persistent ``` -## How to check if services work +### How to check if services work The playbook can perform a check to ensure that you've configured things correctly and that services are running. @@ -58,17 +76,3 @@ The shortcut command with `just` program is also available: `just run-tags self- If it's all green, everything is probably running correctly. Besides this self-check, you can also check whether your server federates with the Matrix network by using the [Federation Tester](https://federationtester.matrix.org/) against your base domain (`example.com`), not the `matrix.example.com` subdomain. - -## Remove unused Docker data - -You can free some disk space from Docker by removing its unused data. See [docker system prune](https://docs.docker.com/engine/reference/commandline/system_prune/) for more information. - -```sh -ansible-playbook -i inventory/hosts setup.yml --tags=run-docker-prune -``` - -The shortcut command with `just` program is also available: `just run-tags run-docker-prune` - -## Postgres - -See the dedicated [PostgreSQL Maintenance](maintenance-postgres.md) documentation page. From 1be1a5e3970828d4edcdfbb53b9423c99066bbe2 Mon Sep 17 00:00:00 2001 From: Suguru Hirahara Date: Sat, 1 Feb 2025 16:30:46 +0900 Subject: [PATCH 05/10] Update docs/faq.md and docs/maintenance-and-troubleshooting.md: move entries which are instruction and how-to, rather than questions Signed-off-by: Suguru Hirahara --- docs/faq.md | 23 ----------------------- docs/maintenance-and-troubleshooting.md | 23 +++++++++++++++++++++++ 2 files changed, 23 insertions(+), 23 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index 502e40b0d..9effaa235 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -425,35 +425,12 @@ If you have an existing installation done using this Ansible playbook, you can e If your previous installation is done in some other way (not using this Ansible playbook), see [I installed Synapse some other way. Can I migrate such a setup to the playbook?](#i-installed-synapse-some-other-way-can-i-migrate-such-a-setup-to-the-playbook). -### How do I back up the data on my server? - -We haven't documented this properly yet, but the general advice is to: - -- back up Postgres by making a database dump. See [Backing up PostgreSQL](maintenance-postgres.md#backing-up-postgresql) - -- back up all `/matrix` files, except for `/matrix/postgres/data` (you already have a dump) and `/matrix/postgres/data-auto-upgrade-backup` (this directory may exist and contain your old data if you've [performed a major Postgres upgrade](maintenance-postgres.md#upgrading-postgresql)). - -You can later restore these by: - -- Restoring the `/matrix` directory and files on the new server manually -- Following the instruction described on [Installing a server into which you'll import old data](installing.md#installing-a-server-into-which-youll-import-old-data) - -If your server's IP address has changed, you may need to [set up DNS](configuring-dns.md) again. - ### What is this `/matrix/postgres/data-auto-upgrade-backup` directory that is taking up so much space? When you [perform a major Postgres upgrade](maintenance-postgres.md#upgrading-postgresql), we save the the old data files in `/matrix/postgres/data-auto-upgrade-backup`, just so you could easily restore them should something have gone wrong. After verifying that everything still works after the Postgres upgrade, you can safely delete `/matrix/postgres/data-auto-upgrade-backup` -### How do I debug or force SSL certificate renewal? - -SSL certificates are managed automatically by the [Traefik](https://doc.traefik.io/traefik/) reverse-proxy server. - -If you're having trouble with SSL certificate renewal, check the Traefik logs (`journalctl -fu matrix-traefik`). - -If you're [using your own webserver](configuring-playbook-own-webserver.md) instead of the integrated one (Traefik), you should investigate in another way. - ## Miscellaneous ### I would like to see this favorite service of mine integrated and become available on my Matrix server. How can I request it? diff --git a/docs/maintenance-and-troubleshooting.md b/docs/maintenance-and-troubleshooting.md index 460144183..7cf08e2eb 100644 --- a/docs/maintenance-and-troubleshooting.md +++ b/docs/maintenance-and-troubleshooting.md @@ -2,6 +2,21 @@ ## Maintenance +### How to back up the data on your server + +We haven't documented this properly yet, but the general advice is to: + +- back up Postgres by making a database dump. See [Backing up PostgreSQL](maintenance-postgres.md#backing-up-postgresql) + +- back up all `/matrix` files, except for `/matrix/postgres/data` (you already have a dump) and `/matrix/postgres/data-auto-upgrade-backup` (this directory may exist and contain your old data if you've [performed a major Postgres upgrade](maintenance-postgres.md#upgrading-postgresql)). + +You can later restore these by: + +- Restoring the `/matrix` directory and files on the new server manually +- Following the instruction described on [Installing a server into which you'll import old data](installing.md#installing-a-server-into-which-youll-import-old-data) + +If your server's IP address has changed, you may need to [set up DNS](configuring-dns.md) again. + ### Remove unused Docker data You can free some disk space from Docker by removing its unused data. See [docker system prune](https://docs.docker.com/engine/reference/commandline/system_prune/) for more information. @@ -76,3 +91,11 @@ The shortcut command with `just` program is also available: `just run-tags self- If it's all green, everything is probably running correctly. Besides this self-check, you can also check whether your server federates with the Matrix network by using the [Federation Tester](https://federationtester.matrix.org/) against your base domain (`example.com`), not the `matrix.example.com` subdomain. + +### How to debug or force SSL certificate renewal + +SSL certificates are managed automatically by the [Traefik](https://doc.traefik.io/traefik/) reverse-proxy server. + +If you're having trouble with SSL certificate renewal, check the Traefik logs (`journalctl -fu matrix-traefik`). + +If you're [using your own webserver](configuring-playbook-own-webserver.md) instead of the integrated one (Traefik), you should investigate in another way. From e36848e9e6605b720e0a9a935761e1ad929427d4 Mon Sep 17 00:00:00 2001 From: Suguru Hirahara Date: Sat, 1 Feb 2025 16:41:19 +0900 Subject: [PATCH 06/10] Update docs/faq.md: merge the sections "Troubleshooting" and "Maintenance" Signed-off-by: Suguru Hirahara --- docs/faq.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index 9effaa235..e48dba495 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -403,15 +403,7 @@ It can perform a local connection instead. Just set `ansible_connection=local` a If you're running Ansible from within a container (one of the possibilities we list on our [dedicated Ansible documentation page](ansible.md)), then using `ansible_connection=local` is not possible. -## Troubleshooting - -### I get "Error response from daemon: configured logging driver does not support reading" when I do `docker logs matrix-synapse`. - -To prevent double-logging, Docker logging is disabled by explicitly passing `--log-driver=none` to all containers. Due to this, you **cannot** view logs using `docker logs matrix-*`. - -See [this section](maintenance-and-troubleshooting.md#how-to-see-the-logs) on the page for maintenance and troubleshooting for more details to see the logs. - -## Maintenance +## Maintenance and Troubleshooting ### Do I need to do anything to keep my Matrix server updated? @@ -431,6 +423,12 @@ When you [perform a major Postgres upgrade](maintenance-postgres.md#upgrading-po After verifying that everything still works after the Postgres upgrade, you can safely delete `/matrix/postgres/data-auto-upgrade-backup` +### I get "Error response from daemon: configured logging driver does not support reading" when I do `docker logs matrix-synapse`. + +To prevent double-logging, Docker logging is disabled by explicitly passing `--log-driver=none` to all containers. Due to this, you **cannot** view logs using `docker logs matrix-*`. + +See [this section](maintenance-and-troubleshooting.md#how-to-see-the-logs) on the page for maintenance and troubleshooting for more details to see the logs. + ## Miscellaneous ### I would like to see this favorite service of mine integrated and become available on my Matrix server. How can I request it? From 05fb62e525c3abf86285dfc28c9c56af9dbc9274 Mon Sep 17 00:00:00 2001 From: Suguru Hirahara Date: Sat, 1 Feb 2025 16:43:06 +0900 Subject: [PATCH 07/10] Update docs/faq.md: change the entry for troubleshooting into a question Signed-off-by: Suguru Hirahara --- docs/faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/faq.md b/docs/faq.md index e48dba495..313e29a29 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -423,7 +423,7 @@ When you [perform a major Postgres upgrade](maintenance-postgres.md#upgrading-po After verifying that everything still works after the Postgres upgrade, you can safely delete `/matrix/postgres/data-auto-upgrade-backup` -### I get "Error response from daemon: configured logging driver does not support reading" when I do `docker logs matrix-synapse`. +### I get "Error response from daemon: configured logging driver does not support reading" when I run `docker logs matrix-synapse`. Why? To prevent double-logging, Docker logging is disabled by explicitly passing `--log-driver=none` to all containers. Due to this, you **cannot** view logs using `docker logs matrix-*`. From 46a821af75da64a269d9770b619f72156c468252 Mon Sep 17 00:00:00 2001 From: Suguru Hirahara Date: Sat, 1 Feb 2025 16:47:28 +0900 Subject: [PATCH 08/10] Update docs/faq.md: remove an emphasis from the entry for troubleshooting Signed-off-by: Suguru Hirahara --- docs/faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/faq.md b/docs/faq.md index 313e29a29..7a74d4043 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -425,7 +425,7 @@ After verifying that everything still works after the Postgres upgrade, you can ### I get "Error response from daemon: configured logging driver does not support reading" when I run `docker logs matrix-synapse`. Why? -To prevent double-logging, Docker logging is disabled by explicitly passing `--log-driver=none` to all containers. Due to this, you **cannot** view logs using `docker logs matrix-*`. +To prevent double-logging, Docker logging is disabled by explicitly passing `--log-driver=none` to all containers. Due to this, you cannot view logs using `docker logs matrix-*`. See [this section](maintenance-and-troubleshooting.md#how-to-see-the-logs) on the page for maintenance and troubleshooting for more details to see the logs. From 0b1ee94b0046015235f095fd3355e4760a41695a Mon Sep 17 00:00:00 2001 From: Suguru Hirahara Date: Sun, 2 Feb 2025 03:48:48 +0900 Subject: [PATCH 09/10] Update docs/maintenance-and-troubleshooting.md: add the link to Synapse maintenance documentation page Signed-off-by: Suguru Hirahara --- docs/maintenance-and-troubleshooting.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/maintenance-and-troubleshooting.md b/docs/maintenance-and-troubleshooting.md index 7cf08e2eb..99205697e 100644 --- a/docs/maintenance-and-troubleshooting.md +++ b/docs/maintenance-and-troubleshooting.md @@ -29,7 +29,11 @@ The shortcut command with `just` program is also available: `just run-tags run-d ### Postgres -See the dedicated [PostgreSQL Maintenance](maintenance-postgres.md) documentation page. +See the dedicated [PostgreSQL maintenance](maintenance-postgres.md) documentation page. + +### Synapse + +See the dedicated [Synapse maintenance](maintenance-synapse.md) documentation page. ## Troubleshooting From 143f8a541548d68b68b5bd8f3455689e478b8682 Mon Sep 17 00:00:00 2001 From: Suguru Hirahara Date: Tue, 4 Feb 2025 17:38:01 +0900 Subject: [PATCH 10/10] Update docs/faq.md: add the link to maintenance-and-troubleshooting.md Signed-off-by: Suguru Hirahara --- docs/faq.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/faq.md b/docs/faq.md index 7a74d4043..df8f74e86 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -405,6 +405,8 @@ If you're running Ansible from within a container (one of the possibilities we l ## Maintenance and Troubleshooting +💡 Also see this page for generic information about maintaining the services and troubleshooting: [Maintenance and Troubleshooting](maintenance-and-troubleshooting.md) + ### Do I need to do anything to keep my Matrix server updated? Yes. We don't update anything for you automatically.