matrix-docker-ansible-deploy/docs/configuring-playbook-prometheus-nginxlog.md

60 lines
4.2 KiB
Markdown
Raw Normal View History

# Enabling metrics and graphs for NginX logs (optional)
It can be useful to have some (visual) insight into [nginx](https://nginx.org/) logs.
This adds [prometheus-nginxlog-exporter](https://github.com/martin-helmich/prometheus-nginxlog-exporter/) to your Matrix deployment.
Remove matrix-homeserver-proxy role in favor of the new internal Traefik entrypoint This was meant to serve as an intermediary for services needing to reach the homeserver. It was used like that for a while in this `bye-bye-nginx-proxy` branch, but was never actually public. It has recently been superseded by homeserver-like services injecting themselves into a new internal Traefik entrypoint (see `matrix_playbook_internal_matrix_client_api_traefik_entrypoint_*`), so `matrix-homeserver-proxy` is no longer necessary. --- This is probably a good moment to share some benchmarks and reasons for going with the internal Traefik entrypoint as opposed to this nginx service. 1. (1400 rps) Directly to Synapse (`ab -n 1000 -c 100 http://matrix-synapse:8008/_matrix/client/versions` 2. (~900 rps) Via `matrix-homeserver-proxy` (nginx) proxying to Synapse (`ab -n 1000 -c 100 http://matrix-homeserver-proxy:8008/_matrix/client/versions`) 3. (~1200 rps) Via the new internal entrypoint of Traefik (`matrix-internal-matrix-client-api`) proxying to Synapse (`ab -n 1000 -c 100 http://matrix-traefik:8008/_matrix/client/versions`) Besides Traefik being quicker for some reason, there are also other benefits to not having this `matrix-homeserver-proxy` component: - we can reuse what we have in terms of labels. Services can register a few extra labels on the new Traefik entrypoint - we don't need services (like `matrix-media-repo`) to inject custom nginx configs into `matrix-homeserver-proxy`. They just need to register labels, like they do already. - Traefik seems faster than nginx on this benchmark for some reason, which is a nice bonus - no need to run one extra container (`matrix-homeserver-proxy`) and execute one extra Ansible role - no need to maintain a setup where some people run the `matrix-homeserver-proxy` component (because they have route-stealing services like `matrix-media-repo` enabled) and others run an optimized setup without this component and everything needs to be rewired to talk to the homeserver directly. Now, everyone can go through Traefik and we can all run an identical setup Downsides of the new Traefik entrypoint setup are that: - all addon services that need to talk to the homeserver now depend on Traefik - people running their own Traefik setup will be inconvenienced - they need to manage one additional entrypoint
2024-01-14 08:53:14 +00:00
It will collect access logs from various nginx reverse-proxies which may be used internally (e.g. `matrix-synapse-reverse-proxy-companion`, if Synapse workers are enabled) and will make them available at a Prometheus-compatible `/metrics` endpoint.
**Note**: nginx is only used internally by this Ansible playbook. With Traefik being our default reverse-proxy, collecting nginx metrics is less relevant.
To make use of this, you need to install [Prometheus](./configuring-playbook-prometheus-grafana.md) either via the playbook or externally. When using an external Prometheus, configuration adjustments are necessary - see [Save metrics on an external Prometheus server](#save-metrics-on-an-external-prometheus-server).
If your setup includes [Grafana](./configuring-playbook-prometheus-grafana.md), a dedicated `NGINX PROXY` Grafana dashboard will be created.
Update docs: misc edits for consistency (#3911) * Add a warning sign to "Warning" labels Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-matrix-registration.md Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/maintenance-and-troubleshooting.md: remove a section for ma1sd As the project has not updated since several years, it does not seem to be reasonable to pick it up specially on the document. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Common header for sections about adjusting the playbook configuration Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-dendrite.md: fix links to dendrite.yaml.j2 Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-bridge-mautrix-signal.md: remove a note added by a commit to remove signalgo The note has been added with 2f6525ccb3666e0ec8f295e8eeffd78bac15a23e, apparently copied from docs/configuring-playbook-bridge-mautrix-signalgo.md Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-bridge-mautrix-wsproxy.md: fix the anchor link text to mautrix-imessage documentation Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-etherpad.md: add a note about the component being managed externally Refer docs/configuring-playbook-backup-borg.md Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-jitsi.md: use the common label for warning messages Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-ldap-auth.md: unrecommend using ma1sd for authentication Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-appservice-double-puppet.md: remove a duplicate anchor link Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs for old mautrix bridges for Facebook and Instagram: remove anchor links to the deleted files Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-bridge-wechat.md: use common descriptions Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-bridge-matrix-bridge-sms.md: create a section for the prerequisite Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/maintenance-and-troubleshooting.md: use the common header text Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Use common descriptions for adding the configuration Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-bridge-mautrix-telegram.md: small edits - Add a section for a Telegram API key - Add a section for instruction about Appservice Double Puppet or Shared Secret Auth Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs for Draupnir and Mjolnir: replace colons with periods Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-rageshake.md: adopt the common instruction Based on docs/configuring-playbook-sygnal.md regarding the notification about necessity of the service. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add a note about the components managed externally Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> --------- Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> Co-authored-by: Suguru Hirahara <acioustick@noreply.codeberg.org>
2025-01-04 12:52:58 +00:00
## Adjusting the playbook configuration
Add the following configuration to your `inventory/host_vars/matrix.example.com/vars.yml` file:
```yaml
matrix_prometheus_nginxlog_exporter_enabled: true
```
## Installing
Edit descriptions about installation of components (#3842) * Replace installation command shortcut for the "just" program with the most conservative raw ansible-playbook command This commit replaces installation command shortcut ("recipe") for the "just" program with the raw ansible-playbook command, so that the shortcut will be added to it later. The command is so conservative that failure of the command will mean something is clearly broken. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add comments about using setup-all instead of install-all Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add description about shortcut command with the "just" program to the ansible-playbook command with "setup-all" and "start" tags It also explains difference between "just install-all" and "just setup-all" recipes. The explanation is based on docs/playbook-tags.md Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update raw ansible-playbook command to have it do what "just install-all" or "just setup-all" does Since "just install-all" or "just setup-all" invokes "ensure-matrix-users-created" as well, it needs adding to the raw ansible-playbook command. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Remove "ensure-matrix-users-created" from the raw ansible-playbook command which does not need it Also: update the "just" recipes accordingly. "just install-all" and "just setup-all" run "ensure-matrix-users-created" tag as well, therefore they need to be replaced with "run-tags" recipes to skip "ensure-matrix-users-created" Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-etherpad.md: add ensure-matrix-users-created to the raw ansible-playbook Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add description about "ensure-matrix-users-created" and create a list with description about shortcut commands with "just" This commit also fixes list item capitalization and punctuation. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add notes bullet lists Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-matrix-corporal.md and docs/configuring-playbook-email2matrix.md: adopt common instructions Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Replace "run the installation command" with "run the playbook with tags" Now that shortcut commands for the "just" program are displayed along with the existing "installation command", this commit replaces "run the installation command" with "run the playbook with tags" in order to prevent misunderstanding and confusion. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add notes about changing passwords of users specified on vars.yml Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-synapse-admin.md: add the playbook command and just recipes Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Remove redundant blank lines Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-alertmanager-receiver.md: remove the direction to proceed to Usage Such a kind of direction is not used on other documentation, so it should be fine to just remove it. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/importing-synapse-media-store.md: code block for ansible-playbook Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> --------- Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> Co-authored-by: Suguru Hirahara <acioustick@noreply.codeberg.org>
2024-12-01 07:42:30 +00:00
After configuring the playbook, run it with [playbook tags](playbook-tags.md) as below:
<!-- NOTE: let this conservative command run (instead of install-all) to make it clear that failure of the command means something is clearly broken. -->
```sh
ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start
```
The shortcut commands with the [`just` program](just.md) are also available: `just install-all` or `just setup-all`
Edit descriptions about installation of components (#3842) * Replace installation command shortcut for the "just" program with the most conservative raw ansible-playbook command This commit replaces installation command shortcut ("recipe") for the "just" program with the raw ansible-playbook command, so that the shortcut will be added to it later. The command is so conservative that failure of the command will mean something is clearly broken. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add comments about using setup-all instead of install-all Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add description about shortcut command with the "just" program to the ansible-playbook command with "setup-all" and "start" tags It also explains difference between "just install-all" and "just setup-all" recipes. The explanation is based on docs/playbook-tags.md Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update raw ansible-playbook command to have it do what "just install-all" or "just setup-all" does Since "just install-all" or "just setup-all" invokes "ensure-matrix-users-created" as well, it needs adding to the raw ansible-playbook command. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Remove "ensure-matrix-users-created" from the raw ansible-playbook command which does not need it Also: update the "just" recipes accordingly. "just install-all" and "just setup-all" run "ensure-matrix-users-created" tag as well, therefore they need to be replaced with "run-tags" recipes to skip "ensure-matrix-users-created" Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-etherpad.md: add ensure-matrix-users-created to the raw ansible-playbook Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add description about "ensure-matrix-users-created" and create a list with description about shortcut commands with "just" This commit also fixes list item capitalization and punctuation. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add notes bullet lists Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-matrix-corporal.md and docs/configuring-playbook-email2matrix.md: adopt common instructions Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Replace "run the installation command" with "run the playbook with tags" Now that shortcut commands for the "just" program are displayed along with the existing "installation command", this commit replaces "run the installation command" with "run the playbook with tags" in order to prevent misunderstanding and confusion. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add notes about changing passwords of users specified on vars.yml Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-synapse-admin.md: add the playbook command and just recipes Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Remove redundant blank lines Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-alertmanager-receiver.md: remove the direction to proceed to Usage Such a kind of direction is not used on other documentation, so it should be fine to just remove it. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/importing-synapse-media-store.md: code block for ansible-playbook Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> --------- Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> Co-authored-by: Suguru Hirahara <acioustick@noreply.codeberg.org>
2024-12-01 07:42:30 +00:00
`just install-all` is useful for maintaining your setup quickly ([2x-5x faster](../CHANGELOG.md#2x-5x-performance-improvements-in-playbook-runtime) than `just setup-all`) when its components remain unchanged. If you adjust your `vars.yml` to remove other components, you'd need to run `just setup-all`, or these components will still remain installed. Note these shortcuts run the `ensure-matrix-users-created` tag too.
## Docker Image Compatibility
At the moment of writing only images for `amd64` and `arm64` architectures are available. The playbook currently does not support [self-building](./self-building.md) a container image on other architectures. You can however use a custom-build image by setting:
```yaml
matrix_prometheus_nginxlog_exporter_docker_image_arch_check_enabled: false
matrix_prometheus_nginxlog_exporter_docker_image: path/to/docker/image:tag
```
## Security and privacy
Metrics and resulting graphs can contain a lot of information. NginX logs contain information like IP address, URLs, UserAgents and more. This information can reveal usage patterns and could be considered Personally Identifiable Information (PII). Think about this before enabling (anonymous) access. Please make sure you change the default Grafana password.
## Save metrics on an external Prometheus server
The playbook will automatically integrate the metrics into the [Prometheus](./configuring-playbook-prometheus-grafana.md) server provided with this playbook (if enabled). In such cases, the metrics endpoint is not exposed publicly - it's only available on the container network.
When using an external Prometheus server, you'll need to expose metrics publicly. See [Collecting metrics to an external Prometheus server](./configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server).
You can either use `matrix_prometheus_nginxlog_exporter_metrics_proxying_enabled: true` to expose just this one service, or `matrix_metrics_exposure_enabled: true` to expose all services.
Whichever way you go with, this service will expose its metrics endpoint **without password-protection** at `https://matrix.example.com/metrics/nginxlog` by default.
For password-protection, use (`matrix_metrics_exposure_http_basic_auth_enabled` and `matrix_metrics_exposure_http_basic_auth_users`) or (`matrix_prometheus_nginxlog_exporter_container_labels_metrics_middleware_basic_auth_enabled` and `matrix_prometheus_nginxlog_exporter_container_labels_metrics_middleware_basic_auth_users`).