ansol
/
matrix-docker-ansible-deploy
mirror of https://github.com/spantaleev/matrix-docker-ansible-deploy.git
7
0
Fork 0
Code Issues Projects Releases Wiki Activity

Added matrix-media-repo role

This commit is contained in:
Michael Hollister 2023-07-12 01:09:27 -05:00
parent 3d321142d6
commit 78bd1dbd1b
17 changed files with 2486 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
README.md
View File

@ -92,6 +92,7 @@ Use alternative file storage to the default `media_store` folder.
| ---- | -------- | ----------- | ------------- |
| [Goofys](https://github.com/kahing/goofys) | x | [Amazon S3](https://aws.amazon.com/s3/) (or other S3-compatible object store) storage for Synapse's content repository (`media_store`) files | [Link](docs/configuring-playbook-s3-goofys.md) |
| [synapse-s3-storage-provider](https://github.com/matrix-org/synapse-s3-storage-provider) | x | [Amazon S3](https://aws.amazon.com/s3/) (or other S3-compatible object store) storage for Synapse's content repository (`media_store`) files | [Link](docs/configuring-playbook-s3.md) |
| [matrix-media-repo](https://github.com/turt2live/matrix-media-repo) | x | matrix-media-repo is a highly customizable multi-domain media repository for Matrix. Intended for medium to large deployments, this media repo de-duplicates media while being fully compliant with the specification. | [Link](docs/configuring-playbook-media-repo.md) |
### Bridges

106
docs/configuring-playbook-matrix-media-repo.md Normal file
View File

@ -0,0 +1,106 @@
# Setting up matrix-media-repo (optional)
matrix-media-repo is a highly customizable multi-domain media repository for Matrix. Intended for medium to large environments consisting of several homeservers, this media repo de-duplicates media (including remote media) while being fully compliant with the specification.
Smaller/individual homeservers can still make use of this project's features, though it may be difficult to set up or have higher than expected resource consumption - please do your research before deploying this as this project may not be useful for your environment.
More documentation about the project can be found at: https://docs.t2bot.io/matrix-media-repo/
## Quickstart
Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file:
```yaml
matrix_media_repo_enabled: true
# (optional) Turned off by default
# matrix_media_repo_metrics_enabled: true
```
The repo is pre-configured for integrating with the postgres database, NGINX proxy and prometheus/grafana (if metrics enabled) from this playbook for all the available homeserver roles. When the media repo is enabled, other media store roles should be disabled (if using Synapse with other media store roles).
By default, the media-repo will use the local filesystem for data storage. Additional options include `s3` and `IPFS` (experimental). Access token caching is also enabled by default since the logout endpoints are proxied through the media repo.
## Configuring the media-repo
Additional common configuration options:
```yaml
# The postgres database pooling options
# The maximum number of connects to hold open. More of these allow for more concurrent
# processes to happen.
matrix_media_repo_database_max_connections: 25
# The maximum number of connects to leave idle. More of these reduces the time it takes
# to serve requests in low-traffic scenarios.
matrix_media_repo_database_max_idle_connections: 5
# These users have full access to the administrative functions of the media repository.
# See https://github.com/turt2live/matrix-media-repo/blob/release-v1.2.8/docs/admin.md for information on what these people can do. They must belong to one of the
# configured homeservers above.
matrix_media_repo_admins:
admins: []
# admins:
# - "@your_username:example.org"
# Datastores are places where media should be persisted. This isn't dedicated for just uploads:
# thumbnails and other misc data is also stored in these places. The media repo, when looking
# for a datastore to use, will always use the smallest datastore first.
matrix_media_repo_datastores:
datastores:
- type: file
enabled: true # Enable this to set up data storage.
# Datastores can be split into many areas when handling uploads. Media is still de-duplicated
# across all datastores (local content which duplicates remote content will re-use the remote
# content's location). This option is useful if your datastore is becoming very large, or if
# you want faster storage for a particular kind of media.
#
# The kinds available are:
# thumbnails - Used to store thumbnails of media (local and remote).
# remote_media - Original copies of remote media (servers not configured by this repo).
# local_media - Original uploads for local media.
# archives - Archives of content (GDPR and similar requests).
forKinds: ["thumbnails", "remote_media", "local_media", "archives"]
opts:
path: /data/media
- type: s3
enabled: false # Enable this to set up s3 uploads
forKinds: ["thumbnails", "remote_media", "local_media", "archives"]
opts:
# The s3 uploader needs a temporary location to buffer files to reduce memory usage on
# small file uploads. If the file size is unknown, the file is written to this location
# before being uploaded to s3 (then the file is deleted). If you aren't concerned about
# memory usage, set this to an empty string.
tempPath: "/tmp/mediarepo_s3_upload"
endpoint: sfo2.digitaloceanspaces.com
accessKeyId: ""
accessSecret: ""
ssl: true
bucketName: "your-media-bucket"
# An optional region for where this S3 endpoint is located. Typically not needed, though
# some providers will need this (like Scaleway). Uncomment to use.
#region: "sfo2"
# An optional storage class for tuning how the media is stored at s3.
# See https://aws.amazon.com/s3/storage-classes/ for details; uncomment to use.
#storageClass: STANDARD
# The media repo does support an IPFS datastore, but only if the IPFS feature is enabled. If
# the feature is not enabled, this will not work. Note that IPFS support is experimental at
# the moment and not recommended for general use.
#
# NOTE: Everything you upload to IPFS will be publicly accessible, even when the media repo
# puts authentication on the download endpoints. Only use this option for cases where you
# expect your media to be publicly accessible.
- type: ipfs
enabled: false # Enable this to use IPFS support
forKinds: ["local_media"]
# The IPFS datastore currently has no options. It will use the daemon or HTTP API configured
# in the IPFS section of your main config.
opts: {}
```
Full list of configuration options with documentation can be found in `roles/custom/matrix-media-repo/templates/defaults/main.yml`

1
docs/configuring-playbook-prometheus-grafana.md
View File

@ -83,6 +83,7 @@ Name | Description
`matrix_bridge_hookshot_metrics_proxying_enabled`|Set this to `true` to expose the [Hookshot](configuring-playbook-bridge-hookshot.md) metrics on `https://matrix.DOMAIN/metrics/hookshot` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`)
`matrix_SERVICE_metrics_proxying_enabled`|Various other services/roles may provide similar `_metrics_enabled` and `_metrics_proxying_enabled` variables for exposing their metrics. Refer to each role for details. Only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`
`matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks`|Add nginx `location` blocks to this list if you'd like to expose additional exporters manually (see below)
`matrix_media_repo_metrics_enabled`|Set this to `true` to make media-repo expose metrics (locally, on the container network)
Example for how to make use of `matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks` for exposing additional metrics locations:
```nginx

3
docs/configuring-playbook-s3.md
View File

@ -9,7 +9,7 @@ First, [choose an Object Storage provider](#choosing-an-object-storage-provider)
Then, [create the S3 bucket](#bucket-creation-and-security-configuration).
Finally, [set up S3 storage for Synapse](#setting-up) (with [Goofys](configuring-playbook-s3-goofys.md) or [synapse-s3-storage-provider](configuring-playbook-synapse-s3-storage-provider.md)).
Finally, [set up S3 storage for Synapse](#setting-up) (with [Goofys](configuring-playbook-s3-goofys.md), [synapse-s3-storage-provider](configuring-playbook-synapse-s3-storage-provider.md), or use s3 datastore with the [matrix-media-repo](https://docs.t2bot.io/matrix-media-repo/configuration/s3-datastore.html)).
## Choosing an Object Storage provider
@ -105,3 +105,4 @@ To set up Synapse to store files in S3, follow the instructions for the method o
- using [synapse-s3-storage-provider](configuring-playbook-synapse-s3-storage-provider.md) (recommended)
- using [Goofys to mount the S3 store to the local filesystem](configuring-playbook-s3-goofys.md)
- using [matrix-media-repo](https://docs.t2bot.io/matrix-media-repo/configuration/s3-datastore.html)

67
group_vars/matrix_servers
View File

@ -326,6 +326,8 @@ devture_systemd_service_manager_services_list_auto: |
+
([{'name': 'matrix-ma1sd.service', 'priority': 2000, 'groups': ['matrix', 'ma1sd']}] if matrix_ma1sd_enabled else [])
+
([{'name': 'matrix-media-repo.service', 'priority': 4000, 'groups': ['matrix', 'media_store']}] if matrix_media_repo_enabled else [])
+
([{'name': 'matrix-mailer.service', 'priority': 2000, 'groups': ['matrix', 'mailer']}] if matrix_mailer_enabled else [])
+
([{'name': 'matrix-nginx-proxy.service', 'priority': 3000, 'groups': ['matrix', 'nginx', 'reverse-proxies']}] if matrix_nginx_proxy_enabled else [])
@ -395,7 +397,6 @@ devture_systemd_service_manager_services_list_auto: |
########################################################################
######################################################################
#
# com.devture.ansible.role.playbook_state_preserver
@ -418,7 +419,6 @@ devture_playbook_state_preserver_commit_hash_preservation_dst: "{{ matrix_base_d
######################################################################
######################################################################
#
# matrix-base
@ -2557,6 +2557,38 @@ matrix_ma1sd_database_password: "{{ '%s' | format(matrix_homeserver_generic_secr
#
######################################################################
######################################################################
#
# matrix-media-repo
#
######################################################################
matrix_media_repo_enabled: false
matrix_media_repo_identifier: matrix-media-repo
matrix_media_repo_container_network: "{{ matrix_docker_network }}"
matrix_media_repo_container_labels_traefik_enabled: false
matrix_media_repo_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}"
matrix_media_repo_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}"
matrix_media_repo_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}"
matrix_media_repo_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}"
matrix_media_repo_database_username: matrix_media_repo
matrix_media_repo_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'mediarepo.db', rounds=655555) | to_uuid }}"
matrix_media_repo_database_name: matrix_media_repo
matrix_media_repo_systemd_required_services_list: |
{{
(['docker.service'])
+
([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled else [])
}}
######################################################################
#
# /matrix-media-repo
#
######################################################################
######################################################################
#
@ -2638,6 +2670,10 @@ matrix_nginx_proxy_proxy_matrix_identity_api_enabled: "{{ matrix_ma1sd_enabled }
matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container: "matrix-ma1sd:{{ matrix_ma1sd_container_port }}"
matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container: "127.0.0.1:{{ matrix_ma1sd_container_port }}"
matrix_nginx_proxy_proxy_media_repo_enabled: "{{ matrix_media_repo_enabled }}"
matrix_nginx_proxy_proxy_media_repo_addr_with_container: "matrix-media-repo:{{ matrix_media_repo_port }}"
matrix_nginx_proxy_proxy_media_repo_addr_sans_container: "127.0.0.1:{{ matrix_media_repo_port }}"
# By default, we do TLS termination for the Matrix Federation API (port 8448) at matrix-nginx-proxy.
# Unless this is handled there OR Synapse's federation listener port is disabled, we'll reverse-proxy.
matrix_nginx_proxy_proxy_matrix_federation_api_enabled: |-
@ -2696,6 +2732,8 @@ matrix_nginx_proxy_systemd_wanted_services_list: |
+
(['matrix-ma1sd.service'] if matrix_ma1sd_enabled else [])
+
(['matrix-media-repo.service'] if matrix_media_repo_enabled else [])
+
(['matrix-client-cinny.service'] if matrix_client_cinny_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] else [])
+
(['matrix-bot-buscarron.service'] if matrix_bot_buscarron_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] else [])
@ -3050,6 +3088,12 @@ devture_postgres_managed_databases_auto: |
'username': prometheus_postgres_exporter_database_username,
'password': prometheus_postgres_exporter_database_password,
}] if (prometheus_postgres_exporter_enabled and prometheus_postgres_exporter_database_hostname == devture_postgres_connection_hostname) else [])
+
([{
'name': matrix_media_repo_database_name,
'username': matrix_media_repo_database_username,
'password': matrix_media_repo_database_password,
}] if (matrix_media_repo_enabled and matrix_media_repo_database_hostname == devture_postgres_connection_hostname) else [])
}}
@ -3424,6 +3468,9 @@ matrix_synapse_redis_password: "{{ redis_connection_password if redis_enabled el
matrix_synapse_container_extra_arguments_auto: "{{ matrix_homeserver_container_extra_arguments_auto }}"
matrix_synapse_app_service_config_files_auto: "{{ matrix_homeserver_app_service_config_files_auto }}"
# Disable creation of media repository Synapse worker when using media-repo
matrix_synapse_ext_media_repo_enabled: "{{ matrix_media_repo_enabled }}"
######################################################################
#
# /matrix-synapse
@ -3653,6 +3700,8 @@ prometheus_container_additional_networks: |
([matrix_hookshot_container_network] if matrix_prometheus_services_connect_scraper_hookshot_enabled and matrix_hookshot_container_network != prometheus_container_network else [])
+
([matrix_prometheus_nginxlog_exporter_container_network] if matrix_prometheus_services_connect_scraper_nginxlog_enabled and matrix_prometheus_nginxlog_exporter_container_network != prometheus_container_network else [])
+
([matrix_media_repo_container_network] if matrix_prometheus_services_connect_scraper_media_repo_enabled and matrix_media_repo_container_network != prometheus_container_network else [])
) | unique
}}
@ -3678,6 +3727,8 @@ prometheus_config_scrape_configs_auto: |
(matrix_prometheus_services_connect_scraper_hookshot_scrape_configs if matrix_prometheus_services_connect_scraper_hookshot_enabled else [])
+
(matrix_prometheus_services_connect_scraper_nginxlog_scrape_configs if matrix_prometheus_services_connect_scraper_nginxlog_enabled else [])
+
(matrix_prometheus_services_connect_scraper_media_repo_scrape_configs if matrix_prometheus_services_connect_scraper_media_repo_enabled else [])
}}
######################################################################
@ -3713,6 +3764,9 @@ matrix_prometheus_services_connect_scraper_hookshot_static_configs_target: "{{ m
matrix_prometheus_services_connect_scraper_nginxlog_enabled: "{{ matrix_prometheus_nginxlog_exporter_enabled }}"
matrix_prometheus_services_connect_scraper_nginxlog_static_configs_target: "{{ matrix_prometheus_nginxlog_exporter_container_hostname }}:{{ matrix_prometheus_nginxlog_exporter_container_metrics_port | string }}"
matrix_prometheus_services_connect_scraper_media_repo_enabled: "{{ matrix_media_repo_enabled and matrix_media_repo_metrics_enabled }}"
matrix_prometheus_services_connect_scraper_media_repo_static_configs_target: "{{ matrix_media_repo_identifier }}:{{ matrix_media_repo_metrics_port }}"
######################################################################
#
# /matrix-prometheus-services-connect
@ -3777,6 +3831,8 @@ grafana_dashboard_download_urls: |
(prometheus_postgres_exporter_dashboard_urls if prometheus_postgres_exporter_enabled else [])
+
(matrix_prometheus_nginxlog_exporter_dashboard_urls if matrix_prometheus_nginxlog_exporter_enabled else [])
+
(matrix_media_repo_dashboard_urls if matrix_media_repo_metrics_enabled else [])
}}
grafana_provisioning_dashboard_template_files: |
@ -3785,6 +3841,11 @@ grafana_provisioning_dashboard_template_files: |
'path': 'roles/custom/matrix-prometheus-nginxlog-exporter/templates/grafana/nginx-proxy.json',
'name': 'nginx-proxy.json',
}] if matrix_prometheus_nginxlog_exporter_enabled else [])
+
([{
'path': 'roles/custom/matrix-media-repo/templates/grafana/media-repo.json',
'name': 'media-repo.json',
}] if matrix_media_repo_metrics_enabled else [])
}}
grafana_default_home_dashboard_path: |-
@ -3803,7 +3864,6 @@ grafana_default_home_dashboard_path: |-
######################################################################
######################################################################
#
# matrix-registration
@ -3853,7 +3913,6 @@ matrix_registration_database_password: "{{ '%s' | format(matrix_homeserver_gener
######################################################################
######################################################################
#
# matrix-sliding-sync

681
roles/custom/matrix-media-repo/defaults/main.yml Normal file
View File

@ -0,0 +1,681 @@
---
# matrix-media-repo is a highly customizable multi-domain media repository for Matrix.
# Intended for medium to large environments consisting of several homeservers, this
# media repo de-duplicates media (including remote media) while being fully compliant
# with the specification.
# See: https://github.com/turt2live/matrix-media-repo
matrix_media_repo_enabled: true
matrix_media_repo_container_image_self_build: false
matrix_media_repo_container_image_self_build_repo: "https://github.com/turt2live/matrix-media-repo.git"
matrix_media_repo_docker_image_path: "turt2live/matrix-media-repo"
matrix_media_repo_docker_image: "{{ matrix_media_repo_docker_image_name_prefix }}{{ matrix_media_repo_docker_image_path }}:{{ matrix_media_repo_docker_image_tag }}"
matrix_media_repo_docker_image_name_prefix: "{{ 'localhost/' if matrix_media_repo_container_image_self_build else matrix_container_global_registry_prefix }}"
matrix_media_repo_docker_image_tag: "v1.2.13"
matrix_media_repo_docker_image_force_pull: "{{ matrix_media_repo_docker_image.endswith(':latest') }}"
matrix_media_repo_base_path: "{{ matrix_base_data_path }}/media-repo"
matrix_media_repo_docker_src_files_path: "{{ matrix_media_repo_base_path }}/docker-src"
# List of systemd services that matrix-conduit.service depends on
matrix_media_repo_systemd_required_services_list: ["docker.service"]
# List of systemd services that matrix-conduit.service wants
matrix_media_repo_systemd_wanted_services_list: []
# The base container network. It will be auto-created by this role if it doesn't exist already.
matrix_media_repo_container_network: "{{ matrix_docker_network }}"
# A list of additional container networks that the container would be connected to.
# The role does not create these networks, so make sure they already exist.
# Use this to expose this container to another reverse proxy, which runs in a different container network.
matrix_media_repo_container_additional_networks: []
# Extra arguments for the Docker container
matrix_media_repo_container_extra_arguments: []
# matrix_media_repo_dashboard_urls contains a list of URLs with Grafana dashboard definitions.
# If the Grafana role is enabled, these dashboards will be downloaded.
matrix_media_repo_dashboard_urls:
- https://raw.githubusercontent.com/spantaleev/matrix-docker-ansible-deploy/master/roles/custom/matrix-media-repo/templates/grafana/media-repo.json
# *****************************************************************************
# Configuration File Settings
# *****************************************************************************
# General repo configuration
matrix_media_repo_bind_address: '0.0.0.0'
matrix_media_repo_port: 8000
# Where to store the logs, relative to where the repo is started from. Logs will be automatically
# rotated every day and held for 14 days. To disable the repo logging to files, set this to
# "-" (including quotation marks).
#
# Note: to change the log directory you'll have to restart the repository. This setting cannot be
# live reloaded.
matrix_media_repo_log_directory: logs
# Set to true to enable color coding in your logs. Note that this may cause escape sequences to
# appear in logs which render them unreadable, which is why colors are disabled by default.
matrix_media_repo_log_colors: false
# Set to true to enable JSON logging for consumption by things like logstash. Note that this is
# incompatible with the log color option and will always render without colors.
matrix_media_repo_json_logs: false
# The log level to log at. Note that this will need to be at least "info" to receive support.
#
# Values (in increasing spam): panic | fatal | error | warn | info | debug | trace
matrix_media_repo_log_level: "info"
# If true, the media repo will accept any X-Forwarded-For header without validation. In most cases
# this option should be left as "false". Note that the media repo already expects an X-Forwarded-For
# header, but validates it to ensure the IP being given makes sense.
matrix_media_repo_trust_any_forwarded_address: false
# If false, the media repo will not use the X-Forwarded-Host header commonly added by reverse proxies.
# Typically this should remain as true, though in some circumstances it may need to be disabled.
# See https://github.com/turt2live/matrix-media-repo/issues/202 for more information.
matrix_media_repo_use_forwarded_host: true
# Options for dealing with federation
# On a per-host basis, the number of consecutive failures in calling the host before the
# media repo will back off. This defaults to 20 if not given. Note that 404 errors from
# the remote server do not count towards this.
matrix_media_repo_federation_backoff_at: 20
# The database configuration for the media repository
# Do NOT put your homeserver's existing database credentials here. Create a new database and
# user instead. Using the same server is fine, just not the same username and database.
matrix_media_repo_database_username: "matrix_media_repo"
matrix_media_repo_database_password: "your_password"
matrix_media_repo_database_hostname: "matrix-postgres"
matrix_media_repo_database_port: 5432
matrix_media_repo_database_name: "matrix_media_repo"
# Currently only "postgres" is supported.
matrix_media_repo_database_postgres: "postgres://{{ matrix_media_repo_database_username }}:{{ matrix_media_repo_database_password }}@{{ matrix_media_repo_database_hostname }}:{{ matrix_media_repo_database_port }}/{{ matrix_media_repo_database_name }}?sslmode=disable"
# The database pooling options
# The maximum number of connects to hold open. More of these allow for more concurrent
# processes to happen.
matrix_media_repo_database_max_connections: 25
# The maximum number of connects to leave idle. More of these reduces the time it takes
# to serve requests in low-traffic scenarios.
matrix_media_repo_database_max_idle_connections: 5
# The configuration for the homeservers this media repository is known to control. Servers
# not listed here will not be able to upload media.
matrix_media_repo_homeservers:
homeservers:
# This should match the server_name of your homeserver, and the Host header
# provided to the media repo.
- name: "{{ matrix_server_fqn_matrix }}"
# The base URL to where the homeserver can actually be reached
csApi: "https://{{ matrix_server_fqn_matrix }}/"
# The number of consecutive failures in calling this homeserver before the
# media repository will start backing off. This defaults to 10 if not given.
backoffAt: 10
# The kind of admin API the homeserver supports. If set to "matrix",
# the media repo will use the Synapse-defined endpoints under the
# unstable client-server API. When this is "synapse", the new /_synapse
# endpoints will be used instead. Unknown values are treated as the
# default, "matrix".
adminApiKind: "matrix"
# Options for controlling how access tokens work with the media repo. It is recommended that if
# you are going to use these options that the `/logout` and `/logout/all` client-server endpoints
# be proxied through this process. They will also be called on the homeserver, and the response
# sent straight through the client - they are simply used to invalidate the cache faster for
# a particular user. Without these, the access tokens might still work for a short period of time
# after the user has already invalidated them.
#
# This will also cache errors from the homeserver.
#
# Note that when this config block is used outside of a per-domain config, all hosts will be
# subject to the same cache. This also means that application services on limited homeservers
# could be authorized on the wrong domain.
#
# ***************************************************************************
# * IT IS HIGHLY RECOMMENDED TO USE PER-DOMAIN CONFIGS WITH THIS FEATURE. *
# ***************************************************************************
matrix_media_repo_access_tokens:
accessTokens:
# The maximum time a cached access token will be considered valid. Set to zero (the default)
# to disable the cache and constantly hit the homeserver. This is recommended to be set to
# 43200 (12 hours) on servers with the logout endpoints proxied through the media repo, and
# zero for servers who do not proxy the endpoints through.
maxCacheTimeSeconds: 43200
# Whether or not to use the `appservices` config option below. If disabled (the default),
# the regular access token cache will be used for each user, potentially leading to high
# memory usage.
useLocalAppserviceConfig: false
# The application services (and their namespaces) registered on the homeserver. Only used
# if `useLocalAppserviceConfig` is enabled (recommended).
#
# Usually the appservice will provide you with these config details - they'll just need
# translating from the appservice registration to here. Note that this does not require
# all options from the registration, and only requires the bare minimum required to run
# the media repo.
# appservices:
# - id: Name_of_appservice_for_your_reference
# asToken: Secret_token_for_appservices_to_use
# senderUserId: "@_example_bridge:yourdomain.com"
# userNamespaces:
# - regex: "@_example_bridge_.+:yourdomain.com"
# # A note about regexes: it is best to suffix *all* namespaces with the homeserver
# # domain users are valid for, as otherwise the appservice can use any user with
# # any domain name it feels like, even if that domain is not configured with the
# # media repo. This will lead to inaccurate reporting in the case of the media
# # repo, and potentially leading to media being considered "remote".
# These users have full access to the administrative functions of the media repository.
# See docs/admin.md for information on what these people can do. They must belong to one of the
# configured homeservers above.
matrix_media_repo_admins:
admins: []
# admins:
# - "@your_username:example.org"
# Shared secret auth is useful for applications building on top of the media repository, such
# as a management interface. The `token` provided here is treated as a repository administrator
# when shared secret auth is enabled: if the `token` is used in place of an access token, the'
# request will be authorized. This is not limited to any particular domain, giving applications
# the ability to use it on any configured hostname.
# Set this to true to enable shared secret auth.
matrix_media_repo_shared_secret_auth_enabled: false
# Use a secure value here to prevent unauthorized access to the media repository.
matrix_media_repo_shared_secret_auth_token: "PutSomeRandomSecureValueHere"
# Datastores are places where media should be persisted. This isn't dedicated for just uploads:
# thumbnails and other misc data is also stored in these places. The media repo, when looking
# for a datastore to use, will always use the smallest datastore first.
matrix_media_repo_datastores:
datastores:
- type: file
enabled: true # Enable this to set up data storage.
# Datastores can be split into many areas when handling uploads. Media is still de-duplicated
# across all datastores (local content which duplicates remote content will re-use the remote
# content's location). This option is useful if your datastore is becoming very large, or if
# you want faster storage for a particular kind of media.
#
# The kinds available are:
# thumbnails - Used to store thumbnails of media (local and remote).
# remote_media - Original copies of remote media (servers not configured by this repo).
# local_media - Original uploads for local media.
# archives - Archives of content (GDPR and similar requests).
forKinds: ["thumbnails", "remote_media", "local_media", "archives"]
opts:
path: /data/media
- type: s3
enabled: false # Enable this to set up s3 uploads
forKinds: ["thumbnails", "remote_media", "local_media", "archives"]
opts:
# The s3 uploader needs a temporary location to buffer files to reduce memory usage on
# small file uploads. If the file size is unknown, the file is written to this location
# before being uploaded to s3 (then the file is deleted). If you aren't concerned about
# memory usage, set this to an empty string.
tempPath: "/tmp/mediarepo_s3_upload"
endpoint: sfo2.digitaloceanspaces.com
accessKeyId: ""
accessSecret: ""
ssl: true
bucketName: "your-media-bucket"
# An optional region for where this S3 endpoint is located. Typically not needed, though
# some providers will need this (like Scaleway). Uncomment to use.
# region: "sfo2"
# An optional storage class for tuning how the media is stored at s3.
# See https://aws.amazon.com/s3/storage-classes/ for details; uncomment to use.
# storageClass: STANDARD
# The media repo does support an IPFS datastore, but only if the IPFS feature is enabled. If
# the feature is not enabled, this will not work. Note that IPFS support is experimental at
# the moment and not recommended for general use.
#
# NOTE: Everything you upload to IPFS will be publicly accessible, even when the media repo
# puts authentication on the download endpoints. Only use this option for cases where you
# expect your media to be publicly accessible.
- type: ipfs
enabled: false # Enable this to use IPFS support
forKinds: ["local_media"]
# The IPFS datastore currently has no options. It will use the daemon or HTTP API configured
# in the IPFS section of your main config.
opts: {}
# Options for controlling archives. Archives are exports of a particular user's content for
# the purpose of GDPR or moving media to a different server.
# Whether archiving is enabled or not. Default enabled.
matrix_media_repo_archiving_enabled: true
# If true, users can request a copy of their own data. By default, only repository administrators
# can request a copy.
# This includes the ability for homeserver admins to request a copy of their own server's
# data, as known to the repo.
matrix_media_repo_archiving_self_service: false
# The number of bytes to target per archive before breaking up the files. This is independent
# of any file upload limits and will require a similar amount of memory when performing an export.
# The file size is also a target, not a guarantee - it is possible to have files that are smaller
# or larger than the target. This is recommended to be approximately double the size of your
# file upload limit, provided there is enough memory available for the demand of exporting.
matrix_media_repo_archiving_target_bytes_per_part: 209715200 # 200mb default
# The file upload settings for the media repository
matrix_media_repo_uploads:
uploads:
# The maximum individual file size a user can upload.
maxBytes: 104857600 # 100MB default, 0 to disable
# The minimum number of bytes to let people upload. This is recommended to be non-zero to
# ensure that the "cost" of running the media repo is worthwhile - small file uploads tend
# to waste more CPU and database resources than small files, thus a default of 100 bytes
# is applied here as an approximate break-even point.
minBytes: 100 # 100 bytes by default
# The number of bytes to claim as the maximum size for uploads for the limits API. If this
# is not provided then the maxBytes setting will be used instead. This is useful to provide
# if the media repo's settings and the reverse proxy do not match for maximum request size.
# This is purely for informational reasons and does not actually limit any functionality.
# Set this to -1 to indicate that there is no limit. Zero will force the use of maxBytes.
reportedMaxBytes: 0
# Options for limiting how much content a user can upload. Quotas are applied to content
# associated with a user regardless of de-duplication. Quotas which affect remote servers
# or users will not take effect. When a user exceeds their quota they will be unable to
# upload any more media.
quotas:
# Whether or not quotas are enabled/enforced. Note that even when disabled the media repo
# will track how much media a user has uploaded. This is disabled by default.
enabled: false
# The quota rules that affect users. The first rule to match the uploader will take effect.
# An implied rule which matches all users and has no quota is always last in this list,
# meaning that if no rules are supplied then users will be able to upload anything. Similarly,
# if no rules match a user then the implied rule will match, allowing the user to have no
# quota. The quota will let the user upload to 1 media past their quota, meaning that from
# a statistics perspective the user might exceed their quota however only by a small amount.
users:
- glob: "@*:*" # Affect all users. Use asterisks (*) to match any character.
maxBytes: 53687063712 # 50GB default, 0 to disable
# Settings related to downloading files from the media repository
# The maximum number of bytes to download from other servers
matrix_media_repo_downloads_max_bytes: 104857600 # 100MB default, 0 to disable
# The number of workers to use when downloading remote media. Raise this number if remote
# media is downloading slowly or timing out.
#
# Maximum memory usage = numWorkers multiplied by the maximum download size
# Average memory usage is dependent on how many concurrent downloads your users are doing.
matrix_media_repo_downloads_num_workers: 10
# How long, in minutes, to cache errors related to downloading remote media. Once this time
# has passed, the media is able to be re-requested.
matrix_media_repo_downloads_failure_cache_minutes: 5
# The cache control settings for downloads. This can help speed up downloads for users by
# keeping popular media in the cache. This cache is also used for thumbnails.
matrix_media_repo_downloads_cache_enabled: true
# The maximum size of cache to have. Higher numbers are better.
matrix_media_repo_downloads_cache_max_size_bytes: 1048576000 # 1GB default
# The maximum file size to cache. This should normally be the same size as your maximum
# upload size.
matrix_media_repo_downloads_cache_max_file_size_bytes: 104857600 # 100MB default
# The number of minutes to track how many downloads a file gets
matrix_media_repo_downloads_cache_tracked_minutes: 30
# The number of downloads a file must receive in the window above (trackedMinutes) in
# order to be cached.
matrix_media_repo_downloads_cache_min_downloads: 5
# The minimum amount of time an item should remain in the cache. This prevents the cache
# from cycling out the file if it needs more room during this time. Note that the media
# repo regularly cleans out media which is past this point from the cache, so this number
# may need increasing depending on your use case. If the maxSizeBytes is reached for the
# media repo, and some cached items are still under this timer, new items will not be able
# to enter the cache. When this happens, consider raising maxSizeBytes or lowering this
# timer.
matrix_media_repo_downloads_cache_min_cache_time_seconds: 300
# The minimum amount of time an item should remain outside the cache once it is removed.
matrix_media_repo_downloads_cache_min_evicted_time_seconds: 60
# How many days after a piece of remote content is downloaded before it expires. It can be
# re-downloaded on demand, this just helps free up space in your datastore. Set to zero or
# negative to disable. Defaults to disabled.
matrix_media_repo_downloads_expire_after_days: 0
# URL Preview settings
matrix_media_repo_url_previews:
urlPreviews:
enabled: true # If enabled, the preview_url routes will be accessible
maxPageSizeBytes: 10485760 # 10MB default, 0 to disable
# If true, the media repository will try to provide previews for URLs with invalid or unsafe
# certificates. If false (the default), the media repo will fail requests to said URLs.
previewUnsafeCertificates: false
# Note: URL previews are limited to a given number of words, which are then limited to a number
# of characters, taking off the last word if it needs to. This also applies for the title.
numWords: 50 # The number of words to include in a preview (maximum)
maxLength: 200 # The maximum number of characters for a description
numTitleWords: 30 # The maximum number of words to include in a preview's title
maxTitleLength: 150 # The maximum number of characters for a title
# The mime types to preview when OpenGraph previews cannot be rendered. OpenGraph previews are
# calculated on anything matching "text/*". To have a thumbnail in the preview the URL must be
# an image and the image's type must be allowed by the thumbnailer.
filePreviewTypes:
- "image/*"
# The number of workers to use when generating url previews. Raise this number if url
# previews are slow or timing out.
#
# Maximum memory usage = numWorkers multiplied by the maximum page size
# Average memory usage is dependent on how many concurrent urls your users are previewing.
numWorkers: 10
# Either allowedNetworks or disallowedNetworks must be provided. If both are provided, they
# will be merged. URL previews will be disabled if neither is supplied. Each entry must be
# a CIDR range.
disallowedNetworks:
- "127.0.0.1/8"
- "10.0.0.0/8"
- "172.16.0.0/12"
- "192.168.0.0/16"
- "100.64.0.0/10"
- "169.254.0.0/16"
- '::1/128'
- 'fe80::/64'
- 'fc00::/7'
allowedNetworks:
# "Everything". The blacklist will help limit this.
# This is the default value for this field.
- "0.0.0.0/0"
# How many days after a preview is generated before it expires and is deleted. The preview
# can be regenerated safely - this just helps free up some space in your database. Set to
# zero or negative to disable. Defaults to disabled.
expireAfterDays: 0
# The default Accept-Language header to supply when generating URL previews when one isn't
# supplied by the client.
# Reference: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Language
defaultLanguage: "en-US,en"
# When true, oEmbed previews will be enabled. Typically these kinds of previews are used for
# sites that do not support OpenGraph or page scraping, such as Twitter. For information on
# specifying providers for oEmbed, including your own, see the following documentation:
# https://docs.t2bot.io/matrix-media-repo/url-previews/oembed.html
# Defaults to disabled.
oEmbed: false
# The thumbnail configuration for the media repository.
matrix_media_repo_thumbnails:
thumbnails:
# The maximum number of bytes an image can be before the thumbnailer refuses.
maxSourceBytes: 10485760 # 10MB default, 0 to disable
# The maximum number of pixels an image can have before the thumbnailer refuses. Note that
# this only applies to image types: file types like audio and video are affected solely by
# the maxSourceBytes.
maxPixels: 32000000 # 32M default
# The number of workers to use when generating thumbnails. Raise this number if thumbnails
# are slow to generate or timing out.
#
# Maximum memory usage = numWorkers multiplied by the maximum image source size
# Average memory usage is dependent on how many thumbnails are being generated by your users
numWorkers: 100
# All thumbnails are generated into one of the sizes listed here. The first size is used as
# the default for when no width or height is requested. The media repository will return
# either an exact match or the next largest size of thumbnail.
sizes:
- width: 32
height: 32
- width: 96
height: 96
- width: 320
height: 240
- width: 640
height: 480
- width: 768 # This size is primarily used for audio thumbnailing.
height: 240
- width: 800
height: 600
# To allow for thumbnails to be any size, not just in the sizes specified above, set this to
# true (default false). When enabled, whatever size requested by the client will be generated
# up to a maximum of the largest possible dimensions in the `sizes` list. For best results,
# specify only one size in the `sizes` list when this option is enabled.
dynamicSizing: false
# The content types to thumbnail when requested. Types that are not supported by the media repo
# will not be thumbnailed (adding application/json here won't work). Clients may still not request
# thumbnails for these types - this won't make clients automatically thumbnail these file types.
types:
- "image/jpeg"
- "image/jpg"
- "image/png"
- "image/apng"
- "image/gif"
- "image/heif"
- "image/webp"
# - "image/svg+xml" # Be sure to have ImageMagick installed to thumbnail SVG files
- "audio/mpeg"
- "audio/ogg"
- "audio/wav"
- "audio/flac"
# - "video/mp4" # Be sure to have ffmpeg installed to thumbnail video files
# Animated thumbnails can be CPU intensive to generate. To disable the generation of animated
# thumbnails, set this to false. If disabled, regular thumbnails will be returned.
allowAnimated: true
# Default to animated thumbnails, if available
defaultAnimated: false
# The maximum file size to thumbnail when a capable animated thumbnail is requested. If the image
# is larger than this, the thumbnail will be generated as a static image.
maxAnimateSizeBytes: 10485760 # 10MB default, 0 to disable
# On a scale of 0 (start of animation) to 1 (end of animation), where should the thumbnailer try
# and thumbnail animated content? Defaults to 0.5 (middle of animation).
stillFrame: 0.5
# How many days after a thumbnail is generated before it expires and is deleted. The thumbnail
# can be regenerated safely - this just helps free up some space in your datastores. Set to
# zero or negative to disable. Defaults to disabled.
expireAfterDays: 0
# Controls for the rate limit functionality
# Set this to false if rate limiting is handled at a higher level or you don't want it enabled.
matrix_media_repo_rate_limit_enabled: true
# The number of requests per second before an IP will be rate limited. Must be a whole number.
matrix_media_repo_rate_limit_requests_per_second: 1
# The number of requests an IP can send at once before the rate limit is actually considered.
matrix_media_repo_rate_limit_burst: 10
# Identicons are generated avatars for a given username. Some clients use these to give users a
# default avatar after signing up. Identicons are not part of the official matrix spec, therefore
# this feature is completely optional.
matrix_media_repo_identicons_enabled: true
# The quarantine media settings.
# If true, when a thumbnail of quarantined media is requested an image will be returned. If no
# image is given in the thumbnailPath below then a generated image will be provided. This does
# not affect regular downloads of files.
matrix_media_repo_quarantine_replace_thumbnails: true
# If true, when media which has been quarantined is requested an image will be returned. If
# no image is given in the thumbnailPath below then a generated image will be provided. This
# will replace media which is not an image (ie: quarantining a PDF will replace the PDF with
# an image).
matrix_media_repo_quarantine_replace_downloads: false
# If provided, the given image will be returned as a thumbnail for media that is quarantined.
matrix_media_repo_quarantine_thumbnail_path: ""
# If true, administrators of the configured homeservers may quarantine media for their server
# only. Global administrators can quarantine any media (local or remote) regardless of this
# flag.
matrix_media_repo_quarantine_allow_local_admins: true
# The various timeouts that the media repo will use.
# The maximum amount of time the media repo should spend trying to fetch a resource that is
# being previewed.
matrix_media_repo_timeouts_url_preview_timeout_seconds: 10
# The maximum amount of time the media repo will spend making remote requests to other repos
# or homeservers. This is primarily used to download media.
matrix_media_repo_timeouts_federation_timeout_seconds: 120
# The maximum amount of time the media repo will spend talking to your configured homeservers.
# This is usually used to verify a user's identity.
matrix_media_repo_timeouts_client_server_timeout_seconds: 30
# Prometheus metrics configuration
# For an example Grafana dashboard, import the following JSON:
# https://github.com/turt2live/matrix-media-repo/blob/master/docs/grafana.json
# If true, the bindAddress and port below will serve GET /metrics for Prometheus to scrape.
matrix_media_repo_metrics_enabled: false
# The address to listen on. Typically "127.0.0.1" or "0.0.0.0" for all interfaces.
matrix_media_repo_metrics_bind_address: "0.0.0.0"
# The port to listen on. Cannot be the same as the general web server port.
matrix_media_repo_metrics_port: 9000
# Plugins are optional pieces of the media repo used to extend the functionality offered.
# Currently there are only antispam plugins, but in future there should be more options.
# Plugins are not supported on per-domain paths and are instead repo-wide. For more
# information on writing plugins, please visit #matrix-media-repo:t2bot.io on Matrix.
matrix_media_repo_plugins:
plugins: []
# An example OCR plugin to block images with certain text. Note that the Docker image
# for the media repo automatically ships this at /plugins/plugin_antispam_ocr
# - exec: /plugins/plugin_antispam_ocr
# config:
# # The URL to your OCR server (https://github.com/otiai10/ocrserver)
# ocrServer: "http://localhost:8080"
# # The keywords to scan for. The image must contain at least one of the keywords
# # from each list to qualify for spam.
# keywordGroups:
# - - elon
# - musk
# - elonmusk
# - - bitcoin
# # The minimum (and maximum) sizes of images to process.
# minSizeBytes: 20000
# maxSizeBytes: 200000
# # The types of files to process
# types: ["image/png", "image/jpeg", "image/jpg"]
# # The user ID regex to check against
# userIds: "@telegram_.*"
# # How much of the image's height, starting from the top, to consider before
# # discarding the rest. Set to 1.0 to consider the whole image.
# percentageOfHeight: 0.35
# Options for controlling various MSCs/unstable features of the media repo
# Sections of this config might disappear or be added over time. By default all
# features are disabled in here and must be explicitly enabled to be used.
matrix_media_repo_feature_support:
featureSupport:
# MSC2248 - Blurhash
MSC2448:
# Whether or not this MSC is enabled for use in the media repo
enabled: false
# Maximum dimensions for converting a blurhash to an image. When no width and
# height options are supplied, the default will be half these values.
maxWidth: 1024
maxHeight: 1024
# Thumbnail size in pixels to use to generate the blurhash string
thumbWidth: 64
thumbHeight: 64
# The X and Y components to use. Higher numbers blur less, lower numbers blur more.
xComponents: 4
yComponents: 3
# The amount of contrast to apply when converting a blurhash to an image. Lower values
# make the effect more subtle, larger values make it stronger.
punch: 1
# IPFS Support
# This is currently experimental and might not work at all.
IPFS:
# Whether or not IPFS support is enabled for use in the media repo.
enabled: false
# Options for the built in IPFS daemon
builtInDaemon:
# Enable this to spawn an in-process IPFS node to use instead of a localhost
# HTTP agent. If this is disabled, the media repo will assume you have an HTTP
# IPFS agent running and accessible. Defaults to using a daemon (true).
enabled: true
# If the Daemon is enabled, set this to the location where the IPFS files should
# be stored. If you're using Docker, this should be something like "/data/ipfs"
# so it can be mapped to a volume.
repoPath: "./ipfs"
# Support for redis as a cache mechanism
#
# Note: Enabling Redis support will mean that the existing cache mechanism will do nothing.
# It can be safely disabled once Redis support is enabled.
#
# See docs/redis.md for more information on how this works and how to set it up.
redis:
# Whether or not use Redis instead of in-process caching.
enabled: false
# The Redis shards that should be used by the media repo in the ring. The names of the
# shards are for your reference and have no bearing on the connection, but must be unique.
shards:
- name: "server1"
addr: ":7000"
- name: "server2"
addr: ":7001"
- name: "server3"
addr: ":7002"
# Optional sentry (https://sentry.io/) configuration for the media repo
# Whether or not to set up error reporting. Defaults to off.
matrix_media_repo_sentry_enabled: false
# Get this value from the setup instructions in Sentry
matrix_media_repo_sentry_dsn: "https://examplePublicKey@ingest.sentry.io/0"
# Optional environment flag. Defaults to an empty string.
matrix_media_repo_sentry_environment: ""
# Whether or not to turn on sentry's built in debugging. This will increase log output.
matrix_media_repo_sentry_debug: false

17
roles/custom/matrix-media-repo/tasks/main.yml Normal file
View File

@ -0,0 +1,17 @@
---
- tags:
- setup-all
- setup-media-repo
- install-all
- install-media-repo
block:
- when: matrix_media_repo_enabled | bool
ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml"
- tags:
- setup-all
- setup-media-repo
block:
- when: not matrix_media_repo_enabled | bool
ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml"

74
roles/custom/matrix-media-repo/tasks/setup_install.yml Normal file
View File

@ -0,0 +1,74 @@
---
- name: Ensure media-repo paths exist
ansible.builtin.file:
path: "{{ item.path }}"
state: directory
mode: 0750
owner: "{{ matrix_user_username }}"
group: "{{ matrix_user_groupname }}"
with_items:
- path: "{{ matrix_media_repo_base_path }}"
when: true
- path: "{{ matrix_media_repo_docker_src_files_path }}"
when: "{{ matrix_media_repo_container_image_self_build }}"
when: "item.when | bool"
- name: Ensure media-repo configuration installed
ansible.builtin.template:
src: "{{ role_path }}/templates/media-repo/media-repo.yaml.j2"
dest: "{{ matrix_media_repo_base_path }}/media-repo.yaml"
mode: 0640
owner: "{{ matrix_user_username }}"
group: "{{ matrix_user_groupname }}"
- name: Ensure media-repo Docker image is pulled
community.docker.docker_image:
name: "{{ matrix_media_repo_docker_image }}"
source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}"
force_source: "{{ matrix_media_repo_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}"
force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_media_repo_docker_image_force_pull }}"
when: "not matrix_media_repo_container_image_self_build | bool"
register: result
retries: "{{ devture_playbook_help_container_retries_count }}"
delay: "{{ devture_playbook_help_container_retries_delay }}"
until: result is not failed
- when: "matrix_media_repo_container_image_self_build | bool"
block:
- name: Ensure media-repo repository is present on self-build
ansible.builtin.git:
repo: "{{ matrix_media_repo_container_image_self_build_repo }}"
dest: "{{ matrix_media_repo_docker_src_files_path }}"
version: "{{ matrix_media_repo_docker_image.split(':')[1] }}"
force: "yes"
become: true
become_user: "{{ matrix_user_username }}"
register: matrix_media_repo_git_pull_results
- name: Check if media-repo Docker image exists
ansible.builtin.command: "{{ devture_systemd_docker_base_host_command_docker }} images --quiet --filter 'reference={{ matrix_media_repo_docker_image }}'"
register: matrix_media_repo_docker_image_check_result
changed_when: false
# Invoking the `docker build` command here, instead of calling the `docker_image` Ansible module,
# because the latter does not support BuildKit.
# See: https://github.com/ansible-collections/community.general/issues/514
- name: Ensure media-repo Docker image is built
ansible.builtin.command:
cmd: "{{ devture_systemd_docker_base_host_command_docker }} build -t {{ matrix_media_repo_docker_image }} {{ matrix_media_repo_docker_src_files_path }}"
environment:
DOCKER_BUILDKIT: 1
changed_when: true
when: "matrix_media_repo_git_pull_results.changed | bool or matrix_media_repo_docker_image_check_result.stdout == ''"
- name: Ensure media-repo container network is created
community.general.docker_network:
name: "{{ matrix_media_repo_container_network }}"
driver: bridge
- name: Ensure matrix-media-repo.service installed
ansible.builtin.template:
src: "{{ role_path }}/templates/media-repo/systemd/matrix-media-repo.service.j2"
dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-media-repo.service"
mode: 0640

19
roles/custom/matrix-media-repo/tasks/setup_uninstall.yml Normal file
View File

@ -0,0 +1,19 @@
---
- name: Check existence of matrix-media-repo service
ansible.builtin.stat:
path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-media-repo.service"
register: matrix_media_repo_service_stat
- when: matrix_media_repo_service_stat.stat.exists | bool
block:
- name: Ensure matrix-media-repo is stopped
ansible.builtin.systemd:
name: matrix-media-repo
state: stopped
daemon_reload: true
- name: Ensure matrix-media-repo.service doesn't exist
ansible.builtin.file:
path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-media-repo.service"
state: absent

991
roles/custom/matrix-media-repo/templates/grafana/media-repo.json Normal file
View File

@ -0,0 +1,991 @@
{
"__inputs": [
{
"name": "DS_PROMETHEUS",
"label": "Prometheus",
"description": "",
"type": "datasource",
"pluginId": "prometheus",
"pluginName": "Prometheus"
}
],
"__elements": {},
"__requires": [
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "9.3.1"
},
{
"type": "datasource",
"id": "prometheus",
"name": "Prometheus",
"version": "1.0.0"
},
{
"type": "panel",
"id": "timeseries",
"name": "Time series",
"version": ""
}
],
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
}
]
},
"description": "",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": 9,
"links": [],
"liveNow": false,
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "hertz"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 0
},
"id": 2,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "9.5.3",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(media_http_requests_total[2m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "{{ '{{host}}: {{method}} {{action}}' }}",
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(media_invalid_http_requests_total[2m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "{{ 'Invalid Host: {{method}} {{action}}' }}",
"refId": "B"
}
],
"title": "HTTP Requsts",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "hertz"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 0
},
"id": 3,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "9.5.3",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(media_http_responses_total[2m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "{{ '{{host}}: {{method}} {{action}} {{statusCode}}' }}",
"refId": "A"
}
],
"title": "HTTP Responses",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "bytes"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 9
},
"id": 8,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "9.5.3",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "go_memstats_alloc_bytes",
"format": "time_series",
"interval": "",
"intervalFactor": 1,
"legendFormat": "memory usage (alloc)",
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "go_memstats_sys_bytes",
"interval": "",
"legendFormat": "memory usage (sys)",
"refId": "C"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "go_memstats_heap_alloc_bytes",
"interval": "",
"legendFormat": "heap usage (alloc)",
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "go_memstats_heap_idle_bytes",
"interval": "",
"legendFormat": "heap usage (idle)",
"refId": "D"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "go_memstats_heap_inuse_bytes",
"interval": "",
"legendFormat": "heap usage (used)",
"refId": "E"
}
],
"title": "Memory Usage",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "bytes"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 9
},
"id": 4,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "9.5.3",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "media_cache_num_bytes_used",
"format": "time_series",
"interval": "",
"intervalFactor": 1,
"legendFormat": "{{ 'size of cache: {{cache}}' }}",
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "media_cache_num_live_bytes_used",
"interval": "",
"legendFormat": "{{ 'live size of cache: {{cache}}' }}",
"refId": "C"
}
],
"title": "Cache Size (Bytes)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 18
},
"id": 9,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "9.5.3",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "media_cache_num_items",
"format": "time_series",
"interval": "",
"intervalFactor": 1,
"legendFormat": "{{ 'items in cache: {{cache}}' }}",
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "media_cache_num_live_items",
"interval": "",
"legendFormat": "{{ 'live items in cache: {{cache}}' }}",
"refId": "C"
}
],
"title": "Cache Size (# of items)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "hertz"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 18
},
"id": 5,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "9.5.3",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(media_cache_hits_total[2m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "{{ 'hits in {{cache}}' }}",
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(media_cache_misses_total[2m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "{{ 'misses in {{cache}}' }}",
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(media_cache_evictions_total[2m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "{{ 'evictions due to {{reason}} in {{cache}}' }}",
"refId": "C"
}
],
"title": "Cache Operations",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "hertz"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 27
},
"id": 6,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "9.5.3",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(media_thumbnails_generated_total[2m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "{{ '{{origin}} {{width}}x{{height}} {{method}} animated={{animated}}' }}",
"refId": "A"
}
],
"title": "Thumbnail Generation",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "hertz"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 27
},
"id": 7,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "9.5.3",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(media_downloaded_total[2m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "{{ 'downloads from {{origin}}' }}",
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(media_url_previews_generated_total[2m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "{{ 'preview with engine: {{type}}' }}",
"refId": "B"
}
],
"title": "Resource Handling",
"type": "timeseries"
}
],
"refresh": "1m",
"schemaVersion": 38,
"style": "dark",
"tags": [],
"templating": {
"list": [
{
"current": {
"selected": true,
"text": "Prometheus",
"value": "Prometheus"
},
"hide": 0,
"includeAll": false,
"multi": false,
"name": "DS_PROMETHEUS",
"options": [],
"query": "prometheus",
"queryValue": "",
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"type": "datasource"
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "",
"title": "Media Repo Dashboard",
"uid": "xJUZ3xfmk",
"version": 2,
"weekStart": ""
}

359
roles/custom/matrix-media-repo/templates/media-repo/media-repo.yaml.j2 Normal file
View File

@ -0,0 +1,359 @@
# General repo configuration
repo:
bindAddress: {{ matrix_media_repo_bind_address }}
port: {{ matrix_media_repo_port }}
# Where to store the logs, relative to where the repo is started from. Logs will be automatically
# rotated every day and held for 14 days. To disable the repo logging to files, set this to
# "-" (including quotation marks).
#
# Note: to change the log directory you'll have to restart the repository. This setting cannot be
# live reloaded.
logDirectory: {{ matrix_media_repo_log_directory }}
# Set to true to enable color coding in your logs. Note that this may cause escape sequences to
# appear in logs which render them unreadable, which is why colors are disabled by default.
logColors: {{ matrix_media_repo_log_colors }}
# Set to true to enable JSON logging for consumption by things like logstash. Note that this is
# incompatible with the log color option and will always render without colors.
jsonLogs: {{ matrix_media_repo_json_logs }}
# The log level to log at. Note that this will need to be at least "info" to receive support.
#
# Values (in increasing spam): panic | fatal | error | warn | info | debug | trace
logLevel: {{ matrix_media_repo_log_level }}
# If true, the media repo will accept any X-Forwarded-For header without validation. In most cases
# this option should be left as "false". Note that the media repo already expects an X-Forwarded-For
# header, but validates it to ensure the IP being given makes sense.
trustAnyForwardedAddress: {{ matrix_media_repo_trust_any_forwarded_address }}
# If false, the media repo will not use the X-Forwarded-Host header commonly added by reverse proxies.
# Typically this should remain as true, though in some circumstances it may need to be disabled.
# See https://github.com/turt2live/matrix-media-repo/issues/202 for more information.
useForwardedHost: {{ matrix_media_repo_use_forwarded_host }}
# Options for dealing with federation
federation:
# On a per-host basis, the number of consecutive failures in calling the host before the
# media repo will back off. This defaults to 20 if not given. Note that 404 errors from
# the remote server do not count towards this.
backoffAt: {{ matrix_media_repo_federation_backoff_at }}
# The database configuration for the media repository
# Do NOT put your homeserver's existing database credentials here. Create a new database and
# user instead. Using the same server is fine, just not the same username and database.
database:
# Currently only "postgres" is supported.
postgres: {{ matrix_media_repo_database_postgres }}
# The database pooling options
pool:
# The maximum number of connects to hold open. More of these allow for more concurrent
# processes to happen.
maxConnections: {{ matrix_media_repo_database_max_connections }}
# The maximum number of connects to leave idle. More of these reduces the time it takes
# to serve requests in low-traffic scenarios.
maxIdleConnections: {{ matrix_media_repo_database_max_idle_connections }}
# The configuration for the homeservers this media repository is known to control. Servers
# not listed here will not be able to upload media.
{#
homeservers:
- name: example.org # This should match the server_name of your homeserver, and the Host header
# provided to the media repo.
csApi: "https://example.org/" # The base URL to where the homeserver can actually be reached
backoffAt: 10 # The number of consecutive failures in calling this homeserver before the
# media repository will start backing off. This defaults to 10 if not given.
adminApiKind: "matrix" # The kind of admin API the homeserver supports. If set to "matrix",
# the media repo will use the Synapse-defined endpoints under the
# unstable client-server API. When this is "synapse", the new /_synapse
# endpoints will be used instead. Unknown values are treated as the
# default, "matrix".
#}
{{ matrix_media_repo_homeservers | to_nice_yaml(indent=2, sort_keys=false) }}
# Options for controlling how access tokens work with the media repo. It is recommended that if
# you are going to use these options that the `/logout` and `/logout/all` client-server endpoints
# be proxied through this process. They will also be called on the homeserver, and the response
# sent straight through the client - they are simply used to invalidate the cache faster for
# a particular user. Without these, the access tokens might still work for a short period of time
# after the user has already invalidated them.
#
# This will also cache errors from the homeserver.
#
# Note that when this config block is used outside of a per-domain config, all hosts will be
# subject to the same cache. This also means that application services on limited homeservers
# could be authorized on the wrong domain.
#
# ***************************************************************************
# * IT IS HIGHLY RECOMMENDED TO USE PER-DOMAIN CONFIGS WITH THIS FEATURE. *
# ***************************************************************************
{{ matrix_media_repo_access_tokens | to_nice_yaml(indent=2, sort_keys=false) }}
# These users have full access to the administrative functions of the media repository.
# See docs/admin.md for information on what these people can do. They must belong to one of the
# configured homeservers above.
{{ matrix_media_repo_admins | to_nice_yaml(indent=2, sort_keys=false) }}
# Shared secret auth is useful for applications building on top of the media repository, such
# as a management interface. The `token` provided here is treated as a repository administrator
# when shared secret auth is enabled: if the `token` is used in place of an access token, the'
# request will be authorized. This is not limited to any particular domain, giving applications
# the ability to use it on any configured hostname.
sharedSecretAuth:
# Set this to true to enable shared secret auth.
enabled: {{ matrix_media_repo_shared_secret_auth_enabled }}
# Use a secure value here to prevent unauthorized access to the media repository.
token: {{ matrix_media_repo_shared_secret_auth_token }}
# Datastores are places where media should be persisted. This isn't dedicated for just uploads:
# thumbnails and other misc data is also stored in these places. The media repo, when looking
# for a datastore to use, will always use the smallest datastore first.
{#
datastores:
- type: file
enabled: false # Enable this to set up data storage.
# Datastores can be split into many areas when handling uploads. Media is still de-duplicated
# across all datastores (local content which duplicates remote content will re-use the remote
# content's location). This option is useful if your datastore is becoming very large, or if
# you want faster storage for a particular kind of media.
#
# The kinds available are:
# thumbnails - Used to store thumbnails of media (local and remote).
# remote_media - Original copies of remote media (servers not configured by this repo).
# local_media - Original uploads for local media.
# archives - Archives of content (GDPR and similar requests).
forKinds: ["thumbnails"]
opts:
path: /var/matrix/media
- type: s3
enabled: false # Enable this to set up s3 uploads
forKinds: ["thumbnails", "remote_media", "local_media", "archives"]
opts:
# The s3 uploader needs a temporary location to buffer files to reduce memory usage on
# small file uploads. If the file size is unknown, the file is written to this location
# before being uploaded to s3 (then the file is deleted). If you aren't concerned about
# memory usage, set this to an empty string.
tempPath: "/tmp/mediarepo_s3_upload"
endpoint: sfo2.digitaloceanspaces.com
accessKeyId: ""
accessSecret: ""
ssl: true
bucketName: "your-media-bucket"
# An optional region for where this S3 endpoint is located. Typically not needed, though
# some providers will need this (like Scaleway). Uncomment to use.
#region: "sfo2"
# The media repo does support an IPFS datastore, but only if the IPFS feature is enabled. If
# the feature is not enabled, this will not work. Note that IPFS support is experimental at
# the moment and not recommended for general use.
#
# NOTE: Everything you upload to IPFS will be publicly accessible, even when the media repo
# puts authentication on the download endpoints. Only use this option for cases where you
# expect your media to be publicly accessible.
- type: ipfs
enabled: false # Enable this to use IPFS support
forKinds: ["local_media"]
# The IPFS datastore currently has no options. It will use the daemon or HTTP API configured
# in the IPFS section of your main config.
opts: {}
#}
{{ matrix_media_repo_datastores | to_nice_yaml(indent=2, sort_keys=false) }}
# Options for controlling archives. Archives are exports of a particular user's content for
# the purpose of GDPR or moving media to a different server.
archiving:
# Whether archiving is enabled or not. Default enabled.
enabled: {{ matrix_media_repo_archiving_enabled }}
# If true, users can request a copy of their own data. By default, only repository administrators
# can request a copy.
# This includes the ability for homeserver admins to request a copy of their own server's
# data, as known to the repo.
selfService: {{ matrix_media_repo_archiving_self_service }}
# The number of bytes to target per archive before breaking up the files. This is independent
# of any file upload limits and will require a similar amount of memory when performing an export.
# The file size is also a target, not a guarantee - it is possible to have files that are smaller
# or larger than the target. This is recommended to be approximately double the size of your
# file upload limit, provided there is enough memory available for the demand of exporting.
targetBytesPerPart: {{ matrix_media_repo_archiving_target_bytes_per_part }} # 200mb default
# The file upload settings for the media repository
{{ matrix_media_repo_uploads | to_nice_yaml(indent=2, sort_keys=false) }}
# Settings related to downloading files from the media repository
downloads:
# The maximum number of bytes to download from other servers
maxBytes: {{ matrix_media_repo_downloads_max_bytes }} # 100MB default, 0 to disable
# The number of workers to use when downloading remote media. Raise this number if remote
# media is downloading slowly or timing out.
#
# Maximum memory usage = numWorkers multiplied by the maximum download size
# Average memory usage is dependent on how many concurrent downloads your users are doing.
numWorkers: {{ matrix_media_repo_downloads_num_workers }}
# How long, in minutes, to cache errors related to downloading remote media. Once this time
# has passed, the media is able to be re-requested.
failureCacheMinutes: {{ matrix_media_repo_downloads_failure_cache_minutes }}
# The cache control settings for downloads. This can help speed up downloads for users by
# keeping popular media in the cache. This cache is also used for thumbnails.
cache:
enabled: {{ matrix_media_repo_downloads_cache_enabled }}
# The maximum size of cache to have. Higher numbers are better.
maxSizeBytes: {{ matrix_media_repo_downloads_cache_max_size_bytes }} # 1GB default
# The maximum file size to cache. This should normally be the same size as your maximum
# upload size.
maxFileSizeBytes: {{ matrix_media_repo_downloads_cache_max_file_size_bytes }} # 100MB default
# The number of minutes to track how many downloads a file gets
trackedMinutes: {{ matrix_media_repo_downloads_cache_tracked_minutes }}
# The number of downloads a file must receive in the window above (trackedMinutes) in
# order to be cached.
minDownloads: {{ matrix_media_repo_downloads_cache_min_downloads }}
# The minimum amount of time an item should remain in the cache. This prevents the cache
# from cycling out the file if it needs more room during this time. Note that the media
# repo regularly cleans out media which is past this point from the cache, so this number
# may need increasing depending on your use case. If the maxSizeBytes is reached for the
# media repo, and some cached items are still under this timer, new items will not be able
# to enter the cache. When this happens, consider raising maxSizeBytes or lowering this
# timer.
minCacheTimeSeconds: {{ matrix_media_repo_downloads_cache_min_cache_time_seconds }}
# The minimum amount of time an item should remain outside the cache once it is removed.
minEvictedTimeSeconds: {{ matrix_media_repo_downloads_cache_min_evicted_time_seconds }}
# How many days after a piece of remote content is downloaded before it expires. It can be
# re-downloaded on demand, this just helps free up space in your datastore. Set to zero or
# negative to disable. Defaults to disabled.
expireAfterDays: {{ matrix_media_repo_downloads_expire_after_days }}
# URL Preview settings
{{ matrix_media_repo_url_previews | to_nice_yaml(indent=2) }}
# The thumbnail configuration for the media repository.
{{ matrix_media_repo_thumbnails | to_nice_yaml(indent=2) }}
# Controls for the rate limit functionality
rateLimit:
# Set this to false if rate limiting is handled at a higher level or you don't want it enabled.
enabled: {{ matrix_media_repo_rate_limit_enabled }}
# The number of requests per second before an IP will be rate limited. Must be a whole number.
requestsPerSecond: {{ matrix_media_repo_rate_limit_requests_per_second }}
# The number of requests an IP can send at once before the rate limit is actually considered.
burst: {{ matrix_media_repo_rate_limit_burst }}
# Identicons are generated avatars for a given username. Some clients use these to give users a
# default avatar after signing up. Identicons are not part of the official matrix spec, therefore
# this feature is completely optional.
identicons:
enabled: {{ matrix_media_repo_identicons_enabled }}
# The quarantine media settings.
quarantine:
# If true, when a thumbnail of quarantined media is requested an image will be returned. If no
# image is given in the thumbnailPath below then a generated image will be provided. This does
# not affect regular downloads of files.
replaceThumbnails: {{ matrix_media_repo_quarantine_replace_thumbnails }}
# If true, when media which has been quarantined is requested an image will be returned. If
# no image is given in the thumbnailPath below then a generated image will be provided. This
# will replace media which is not an image (ie: quarantining a PDF will replace the PDF with
# an image).
replaceDownloads: {{ matrix_media_repo_quarantine_replace_downloads }}
# If provided, the given image will be returned as a thumbnail for media that is quarantined.
#thumbnailPath: "/path/to/thumbnail.png"
thumbnailPath: {{ "" if matrix_media_repo_quarantine_thumbnail_path == "" else matrix_media_repo_quarantine_thumbnail_path }}
# If true, administrators of the configured homeservers may quarantine media for their server
# only. Global administrators can quarantine any media (local or remote) regardless of this
# flag.
allowLocalAdmins: {{ matrix_media_repo_quarantine_allow_local_admins }}
# The various timeouts that the media repo will use.
timeouts:
# The maximum amount of time the media repo should spend trying to fetch a resource that is
# being previewed.
urlPreviewTimeoutSeconds: {{ matrix_media_repo_timeouts_url_preview_timeout_seconds }}
# The maximum amount of time the media repo will spend making remote requests to other repos
# or homeservers. This is primarily used to download media.
federationTimeoutSeconds: {{ matrix_media_repo_timeouts_federation_timeout_seconds }}
# The maximum amount of time the media repo will spend talking to your configured homeservers.
# This is usually used to verify a user's identity.
clientServerTimeoutSeconds: {{ matrix_media_repo_timeouts_client_server_timeout_seconds }}
# Prometheus metrics configuration
# For an example Grafana dashboard, import the following JSON:
# https://github.com/turt2live/matrix-media-repo/blob/master/docs/grafana.json
metrics:
# If true, the bindAddress and port below will serve GET /metrics for Prometheus to scrape.
enabled: {{ matrix_media_repo_metrics_enabled }}
# The address to listen on. Typically "127.0.0.1" or "0.0.0.0" for all interfaces.
bindAddress: {{ matrix_media_repo_metrics_bind_address }}
# The port to listen on. Cannot be the same as the general web server port.
port: {{ matrix_media_repo_metrics_port }}
# Plugins are optional pieces of the media repo used to extend the functionality offered.
# Currently there are only antispam plugins, but in future there should be more options.
# Plugins are not supported on per-domain paths and are instead repo-wide. For more
# information on writing plugins, please visit #matrix-media-repo:t2bot.io on Matrix.
{{ matrix_media_repo_plugins | to_nice_yaml(indent=2) }}
# An example OCR plugin to block images with certain text. Note that the Docker image
# for the media repo automatically ships this at /plugins/plugin_antispam_ocr
# - exec: /plugins/plugin_antispam_ocr
# config:
# # The URL to your OCR server (https://github.com/otiai10/ocrserver)
# ocrServer: "http://localhost:8080"
# # The keywords to scan for. The image must contain at least one of the keywords
# # from each list to qualify for spam.
# keywordGroups:
# - - elon
# - musk
# - elonmusk
# - - bitcoin
# # The minimum (and maximum) sizes of images to process.
# minSizeBytes: 20000
# maxSizeBytes: 200000
# # The types of files to process
# types: ["image/png", "image/jpeg", "image/jpg"]
# # The user ID regex to check against
# userIds: "@telegram_.*"
# # How much of the image's height, starting from the top, to consider before
# # discarding the rest. Set to 1.0 to consider the whole image.
# percentageOfHeight: 0.35
# Options for controlling various MSCs/unstable features of the media repo
# Sections of this config might disappear or be added over time. By default all
# features are disabled in here and must be explicitly enabled to be used.
{{ matrix_media_repo_feature_support | to_nice_yaml(indent=2) }}
# Optional sentry (https://sentry.io/) configuration for the media repo
sentry:
# Whether or not to set up error reporting. Defaults to off.
enabled: {{ matrix_media_repo_sentry_enabled }}
# Get this value from the setup instructions in Sentry
dsn: {{ matrix_media_repo_sentry_dsn }}
# Optional environment flag. Defaults to an empty string.
environment: {{ "" if matrix_media_repo_sentry_environment == "" else matrix_media_repo_sentry_environment }}
# Whether or not to turn on sentry's built in debugging. This will increase log output.
debug: {{ matrix_media_repo_sentry_debug }}

52
roles/custom/matrix-media-repo/templates/media-repo/systemd/matrix-media-repo.service.j2 Normal file
View File

@ -0,0 +1,52 @@
#jinja2: lstrip_blocks: "True"
[Unit]
Description=Matrix media-repo
{% for service in matrix_media_repo_systemd_required_services_list %}
Requires={{ service }}
After={{ service }}
{% endfor %}
{% for service in matrix_media_repo_systemd_wanted_services_list %}
Wants={{ service }}
{% endfor %}
DefaultDependencies=no
[Service]
Type=simple
Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}"
ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-media-repo 2>/dev/null || true'
ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-media-repo 2>/dev/null || true'
ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \
--rm \
--name=matrix-media-repo \
--log-driver=none \
--user={{ matrix_user_uid }}:{{ matrix_user_gid }} \
--cap-drop=ALL \
--network={{ matrix_docker_network }} \
-p {{ matrix_media_repo_port }}:{{ matrix_media_repo_port }} \
{% if matrix_media_repo_metrics_enabled %}
-p {{ matrix_media_repo_metrics_port }}:{{ matrix_media_repo_metrics_port }} \
{% endif %}
-v {{ matrix_media_repo_base_path }}:/data:z \
--workdir='/data' \
--entrypoint='media_repo' \
{% for arg in matrix_media_repo_container_extra_arguments %}
{{ arg }} \
{% endfor %}
{{ matrix_media_repo_docker_image }}
{% for network in matrix_media_repo_container_additional_networks %}
ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-media-repo
{% endfor %}
ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-media-repo
ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-media-repo 2>/dev/null || true'
ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-media-repo 2>/dev/null || true'
ExecReload={{ devture_systemd_docker_base_host_command_docker }} exec matrix-media-repo /bin/sh -c 'kill -HUP 1'
Restart=always
RestartSec=30
SyslogIdentifier=matrix-media-repo
[Install]
WantedBy=multi-user.target

5
roles/custom/matrix-nginx-proxy/defaults/main.yml
View File

@ -333,6 +333,11 @@ matrix_nginx_proxy_proxy_matrix_identity_api_enabled: false
matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container: "matrix-ma1sd:{{ matrix_ma1sd_container_port }}"
matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container: "127.0.0.1:{{ matrix_ma1sd_container_port }}"
# Controls whether proxying for the media repo (`/_matrix/media`) should be done (on the matrix domain)
matrix_nginx_proxy_proxy_media_repo_enabled: false
matrix_nginx_proxy_proxy_media_repo_addr_with_container: "matrix-media-repo:{{ matrix_media_repo_port }}"
matrix_nginx_proxy_proxy_media_repo_addr_sans_container: "127.0.0.1:{{ matrix_media_repo_port }}"
# The addresses where the Matrix Client API is.
# Certain extensions (like matrix-corporal) may override this in order to capture all traffic.
matrix_nginx_proxy_proxy_matrix_client_api_addr_with_container: "matrix-nginx-proxy:12080"

90
roles/custom/matrix-nginx-proxy/templates/nginx/conf.d/matrix-domain.conf.j2
View File

@ -94,6 +94,96 @@
}
{% endif %}
{% if matrix_nginx_proxy_proxy_media_repo_enabled %}
# Redirect all media endpoints to the media-repo
location ^~ /_matrix/media {
{% if matrix_nginx_proxy_enabled %}
{# Use the embedded DNS resolver in Docker containers to discover the service #}
resolver {{ matrix_nginx_proxy_http_level_resolver }} valid=5s;
set $backend "{{ matrix_nginx_proxy_proxy_media_repo_addr_with_container }}";
proxy_pass http://$backend;
{% else %}
{# Generic configuration for use outside of our container setup #}
proxy_pass http://{{ matrix_nginx_proxy_proxy_media_repo_addr_sans_container }};
{% endif %}
# Make sure this matches your homeserver in media-repo.yaml
# You may have to manually specify it if using delegation or the
# incoming Host doesn't match.
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
}
# Redirect other endpoints registered by the media-repo to its container
# /_matrix/client/r0/logout
# /_matrix/client/r0/logout/all
location ^~ /_matrix/client/(r0|v1|v3|unstable)/(logout|logout/all) {
{% if matrix_nginx_proxy_enabled %}
{# Use the embedded DNS resolver in Docker containers to discover the service #}
resolver {{ matrix_nginx_proxy_http_level_resolver }} valid=5s;
set $backend "{{ matrix_nginx_proxy_proxy_media_repo_addr_with_container }}";
proxy_pass http://$backend;
{% else %}
{# Generic configuration for use outside of our container setup #}
proxy_pass http://{{ matrix_nginx_proxy_proxy_media_repo_addr_sans_container }};
{% endif %}
# Make sure this matches your homeserver in media-repo.yaml
# You may have to manually specify it if using delegation or the
# incoming Host doesn't match.
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
}
# Redirect other endpoints registered by the media-repo to its container
# /_matrix/client/r0/admin/purge_media_cache
# /_matrix/client/r0/admin/quarantine_media/{roomId:[^/]+}
location ^~ /_matrix/client/(r0|v1|v3|unstable)/admin/(purge_media_cache|quarantine_media/.*) {
{% if matrix_nginx_proxy_enabled %}
{# Use the embedded DNS resolver in Docker containers to discover the service #}
resolver {{ matrix_nginx_proxy_http_level_resolver }} valid=5s;
set $backend "{{ matrix_nginx_proxy_proxy_media_repo_addr_with_container }}";
proxy_pass http://$backend;
{% else %}
{# Generic configuration for use outside of our container setup #}
proxy_pass http://{{ matrix_nginx_proxy_proxy_media_repo_addr_sans_container }};
{% endif %}
# Make sure this matches your homeserver in media-repo.yaml
# You may have to manually specify it if using delegation or the
# incoming Host doesn't match.
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
}
# Redirect other endpoints registered by the media-repo to its container
location ^~ /_matrix/client/unstable/io.t2bot.media {
{% if matrix_nginx_proxy_enabled %}
{# Use the embedded DNS resolver in Docker containers to discover the service #}
resolver {{ matrix_nginx_proxy_http_level_resolver }} valid=5s;
set $backend "{{ matrix_nginx_proxy_proxy_media_repo_addr_with_container }}";
proxy_pass http://$backend;
{% else %}
{# Generic configuration for use outside of our container setup #}
proxy_pass http://{{ matrix_nginx_proxy_proxy_media_repo_addr_sans_container }};
{% endif %}
# Make sure this matches your homeserver in media-repo.yaml
# You may have to manually specify it if using delegation or the
# incoming Host doesn't match.
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
}
{% endif %}
{% if matrix_nginx_proxy_proxy_matrix_user_directory_search_enabled %}
location ^~ /_matrix/client/r0/user_directory/search {
{% if matrix_nginx_proxy_enabled %}

20
roles/custom/matrix-prometheus-services-connect/defaults/main.yml
View File

@ -142,3 +142,23 @@ matrix_prometheus_services_connect_scraper_nginxlog_scrape_configs: |
'static_configs': matrix_prometheus_services_connect_scraper_nginxlog_static_configs,
}]
}}
# Controls whether media-repo shall be scraped
matrix_prometheus_services_connect_scraper_media_repo_enabled: false
matrix_prometheus_services_connect_scraper_media_repo_job_name: media-repo
matrix_prometheus_services_connect_scraper_media_repo_metrics_path: /metrics
matrix_prometheus_services_connect_scraper_media_repo_scrape_interval: 15s
matrix_prometheus_services_connect_scraper_media_repo_scrape_timeout: 15s
matrix_prometheus_services_connect_scraper_media_repo_static_configs: "{{ [{'targets': [matrix_prometheus_services_connect_scraper_media_repo_static_configs_target]}] }}"
matrix_prometheus_services_connect_scraper_media_repo_static_configs_target: ''
# The final scrape config for the media-repo scraper
matrix_prometheus_services_connect_scraper_media_repo_scrape_configs: |
{{
[{
'job_name': matrix_prometheus_services_connect_scraper_media_repo_job_name,
'metrics_path': matrix_prometheus_services_connect_scraper_media_repo_metrics_path,
'scrape_interval': matrix_prometheus_services_connect_scraper_media_repo_scrape_interval,
'scrape_timeout': matrix_prometheus_services_connect_scraper_media_repo_scrape_timeout,
'static_configs': matrix_prometheus_services_connect_scraper_media_repo_static_configs,
}]
}}

6
roles/custom/matrix-synapse/defaults/main.yml
View File

@ -633,14 +633,14 @@ matrix_synapse_workers_federation_sender_workers_metrics_range_start: 19400
# Adjusting this value manually is generally not necessary.
matrix_synapse_federation_sender_instances: []
matrix_synapse_workers_media_repository_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['media_repository_workers_count'] }}"
matrix_synapse_workers_media_repository_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['media_repository_workers_count'] if not matrix_synapse_ext_media_repo_enabled else 0 }}"
matrix_synapse_workers_media_repository_workers_port_range_start: 18551
matrix_synapse_workers_media_repository_workers_metrics_range_start: 19551
# matrix_synapse_enable_media_repo controls if the main Synapse process should serve media repository endpoints or if it should be left to media_repository workers (see `matrix_synapse_workers_media_repository_workers_count`).
# This is enabled if workers are disabled, or if they are enabled, but there are no media repository workers.
# Adjusting this value manually is generally not necessary.
matrix_synapse_enable_media_repo: "{{ not matrix_synapse_workers_enabled or (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'media_repository') | list | length == 0) }}"
matrix_synapse_enable_media_repo: "{{ not matrix_synapse_ext_media_repo_enabled and (not matrix_synapse_workers_enabled or (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'media_repository') | list | length == 0)) }}"
# matrix_synapse_media_instance_running_background_jobs populates the `media_instance_running_background_jobs` Synapse configuration used when Synapse workers are in use (`matrix_synapse_workers_enabled`).
# `media_instance_running_background_jobs` is meant to point to a single media-repository worker, which is dedicated to running background tasks that maintain the media repository.
@ -901,6 +901,8 @@ matrix_synapse_ext_synapse_s3_storage_provider_update_db_day_count: 0
# This is a systemd timer OnCalendar definition. Learn more here: https://man.archlinux.org/man/systemd.time.7#CALENDAR_EVENTS
matrix_synapse_ext_synapse_s3_storage_provider_periodic_migration_schedule: '*-*-* 05:00:00'
matrix_synapse_ext_media_repo_enabled: false
matrix_s3_media_store_enabled: false
matrix_s3_media_store_custom_endpoint_enabled: false
matrix_s3_goofys_docker_image: "{{ matrix_s3_goofys_docker_image_name_prefix }}ewoutp/goofys:latest"

1
setup.yml
View File

@ -113,6 +113,7 @@
- galaxy/ntfy
- custom/matrix-nginx-proxy
- custom/matrix-coturn
- custom/matrix-media-repo
- role: galaxy/auxiliary