Encrypt the agent to Zabbix channel with TLS (PSK or certificate)

README.md

@@ -535,5 +535,6 @@ API query:
 ## Additional chapters
 - [**Adding custom plugins**](documentation/adding_custom_plugins.md)
 - [**Configuration file**](documentation/configuration_file.md)
+- [**Encrypting the connection to Zabbix**](TLS_ENCRYPTION.md)
 - [**Metrics**](documentation/metrics.md)
 - [**Tools**](documentation/tools.md)

TLS_ENCRYPTION.md

@@ -0,0 +1,144 @@
+# Encrypting the connection to Zabbix (mamonsu 3.5.17.1)
+
+This document describes what changed compared to upstream **3.5.17**.
+
+## Why
+
+In 3.5.17 `ZbxSender` sends metrics to the Zabbix server over a plain TCP socket:
+
+```python
+sock = socket.socket()
+sock.connect((self.host, self.port))
+sock.sendall(packet)
+```
+
+When a host in Zabbix is set to **Connections from host: PSK**, the server rejects such connections and mamonsu cannot deliver a single metric, even though the stock `zabbix_sender` works with the same PSK settings.
+
+Starting with 3.5.17.1 mamonsu encrypts the connection itself, with no external processes and no additional Python packages.
+
+## What changed
+
+| File | Change |
+| --- | --- |
+| `mamonsu/lib/senders/tls.py` | new module: the TLS-PSK and TLS-cert transports |
+| `mamonsu/lib/senders/zbx.py` | reads the new settings and picks the transport in `_connect()` |
+| `mamonsu/lib/config.py` | defaults for the `tls_*` parameters |
+| `mamonsu/lib/parser.py`, `mamonsu/lib/runner.py` | the `--zabbix-tls-*` command line options |
+| `packaging/conf/example_linux.conf` | commented example of the settings |
+| `documentation/configuration_file.md` | description of the parameters |
+| `tests/unit/` | tests that need neither docker nor a Zabbix server |
+
+The wire format and the queue logic are untouched: `_send_data()` still sends `ZBXD\x01` + length + JSON. The only difference is where the socket comes from.
+
+## Compatibility
+
+**Without the new parameters the behaviour is exactly that of 3.5.17.** An existing `agent.conf` needs no changes: `tls_connect` defaults to `unencrypted`, and in that case the code path is the previous one.
+
+If the settings are wrong (unknown mode, missing identity or key file, unreadable file), the sender is **disabled with an error in the log** instead of falling back to an unencrypted connection: mamonsu never quietly sends metrics in the clear.
+
+## Configuring PSK
+
+In `/etc/mamonsu/agent.conf`:
+
+```ini
+[zabbix]
+address = zabbix-5
+port = 10051
+client = db2
+
+tls_connect = psk
+tls_psk_identity = PSK DB2
+tls_psk_file = /etc/zabbix/zabbix_agentd.psk
+```
+
+`tls_psk_identity` and `tls_psk_file` have to match what is configured for this host in Zabbix (and the `TLSPSKIdentity` / `TLSPSKFile` of the Zabbix agent, if one runs on the same host).
+
+How it works:
+
+* on Python **3.13 and newer** — the standard `ssl` module (`SSLContext.set_psk_client_callback`);
+* on Python **3.7 to 3.12** — a TLS client of our own through `ctypes` to the system libssl. Only the public OpenSSL API is used and no CPython internals are touched, so the same code works on Astra Linux 1.7.6 (OpenSSL 1.1.1, `libssl.so.1.1`) and Astra Linux 1.8 (OpenSSL 3.x, `libssl.so.3`).
+
+A PSK connection is negotiated as **TLS 1.2**: TLS 1.3 carries the PSK through a different mechanism (`psk_use_session`), which is not implemented yet. Every Zabbix version that supports encryption accepts TLS 1.2.
+
+## Configuring certificates
+
+```ini
+[zabbix]
+tls_connect = cert
+tls_ca_file = /etc/zabbix/ca.crt
+tls_cert_file = /etc/zabbix/mamonsu.crt
+tls_key_file = /etc/zabbix/mamonsu.key
+# tls_crl_file = /etc/zabbix/ca.crl
+# tls_server_cert_issuer = CN=Zabbix CA,O=Company
+# tls_server_cert_subject = CN=zabbix server,O=Company
+```
+
+The server is validated the way the Zabbix agent does it: the chain is verified against the CA file, while the host name is **not** matched against the certificate — the server is pinned by its issuer and subject instead. Escaped commas in a DN are honoured (`CN=Company\, Inc`), the order of the attributes does not matter, and both short (`CN`, `O`, `OU`, `C`, ...) and long (`commonName`) names are accepted.
+
+The `cert` mode works on any Python 3.x and uses the standard library only.
+
+## Command line
+
+Any of the settings except the cipher ones (`tls_cipher_psk`, `tls_cipher_cert`) can be overridden without touching the config file, which is convenient for checking a setup:
+
+```bash
+mamonsu -c /etc/mamonsu/agent.conf \
+        --zabbix-tls-connect psk \
+        --zabbix-tls-psk-identity 'PSK DB2' \
+        --zabbix-tls-psk-file /etc/zabbix/zabbix_agentd.psk
+```
+
+The full list: `--zabbix-tls-connect`, `--zabbix-tls-psk-identity`, `--zabbix-tls-psk-file`, `--zabbix-tls-ca-file`, `--zabbix-tls-crl-file`, `--zabbix-tls-cert-file`, `--zabbix-tls-key-file`, `--zabbix-tls-server-cert-issuer`, `--zabbix-tls-server-cert-subject`. Only the options actually passed are overridden, the rest come from the config file. They work both for the daemon and for `mamonsu upload`.
+
+## Security
+
+* the PSK is read from the file once at startup and is kept in the memory of the process only;
+* the content of the PSK file never reaches the log or the text of an error (there is a test for that);
+* the PSK is not duplicated in `agent.conf`, which only holds the path to the file and the identity;
+* the file has to be readable by the user mamonsu runs as (`mamonsu` in the systemd unit).
+
+## Testing
+
+The tests live in `tests/unit` and need neither docker nor Zabbix:
+
+```bash
+python -m pytest tests/unit                    # with pytest
+python3 tests/unit/test_tls_openssl.py         # without it, e.g. on the monitored host
+python3 tests/unit/test_zbx_sender_socket.py
+```
+
+What they cover:
+
+* a real TLS handshake against `openssl s_server`, both PSK and certificates, with data going both ways;
+* a wrong PSK, an unknown CA and a certificate subject that does not match are all rejected;
+* parsing of the PSK file, and the absence of the secret in error messages;
+* parsing and comparison of distinguished names;
+* the unencrypted path: the frame is byte for byte the previous one and `failed: N` is still detected;
+* the choice of the transport and the command line overrides.
+
+Verified on Linux with Python 3.12 and OpenSSL 3.0.13 (the ctypes path) and on Windows with Python 3.12 (the cert mode on the standard library; the PSK tests are skipped there). The `psk` mode has also been run in a pilot on Astra Linux 1.7.6 with Python 3.7 and OpenSSL 1.1.1, the oldest combination the ctypes path is meant to cover, with metrics reaching a Zabbix 7.4 server.
+
+The reference check with the stock utility:
+
+```bash
+zabbix_sender -c /etc/zabbix/zabbix_agent2.conf -z zabbix-5 -p 10051 -s db2 -k 'pgsql.ping[]' -o 1
+# processed: 1; failed: 0; total: 1
+```
+
+After the settings are in place, mamonsu has to reach the same result without allowing `No encryption` on the Zabbix side.
+
+## Known limitations
+
+* TLS 1.3 with a PSK is not supported: the connection is pinned to TLS 1.2;
+* the PSK file is read once at startup, so the agent has to be restarted after the key is changed;
+* on Windows the `psk` mode requires Python 3.13 or newer, as there is no system libssl there;
+* the `cert` mode has only been tested against `openssl s_server`, not against a production Zabbix server.
+
+## Building
+
+The version comes from `mamonsu/__init__.py`, and the same value is set in `packaging/debian/changelog` and `packaging/rpm/SPECS/mamonsu.spec` — `3.5.17.1`.
+
+```bash
+make -f Makefile.pkg deb
+make -f Makefile.pkg rpm
+```

documentation/configuration_file.md

@@ -99,6 +99,53 @@ The [zabbix] section provides connection settings for the Zabbix server and can
 
         Default: 15
 
+**tls_connect**  
+        How _mamonsu_ connects to the Zabbix server: `unencrypted` for a plain TCP connection, `psk` for a TLS connection with a pre-shared key, or `cert` for a TLS connection with certificates. The value has to match the _Connections from host_ setting of this host in Zabbix.
+
+        A PSK connection is negotiated as TLS 1.2, which Zabbix accepts in every version that supports encryption; a certificate connection uses TLS 1.2 or newer. PSK needs no additional Python package: on Python 3.13 and newer the handshake uses the standard `ssl` module, on older versions it goes through the system libssl (OpenSSL 1.1.1 or 3.x) directly.
+
+        Default: unencrypted
+
+**tls_psk_identity**  
+        The PSK identity string, exactly as configured for this host in Zabbix. Required when tls_connect is psk.
+
+**tls_psk_file**  
+        Path to the file with the pre-shared key, a single line of hexadecimal digits — the same file the Zabbix agent uses in its TLSPSKFile parameter. The file is read once at startup and must be readable by the user _mamonsu_ runs as; its content is never written to the mamonsu log. Required when tls_connect is psk.
+
+**tls_cipher_psk**  
+        OpenSSL cipher string used to select the PSK cipher suites, for the rare case when the Zabbix server is restricted to a specific one.
+
+        Default: PSK
+
+**tls_ca_file**  
+        Path to the top-level CA certificate that signed the certificate of the Zabbix server. Required when tls_connect is cert.
+
+**tls_cert_file**  
+        Path to the certificate _mamonsu_ presents to the Zabbix server. Required when tls_connect is cert.
+
+**tls_key_file**  
+        Path to the private key of that certificate. Required when tls_connect is cert.
+
+**tls_crl_file**  
+        Path to a certificate revocation list. When set, the certificate of the Zabbix server is checked against it.
+
+**tls_server_cert_issuer**  
+        Expected issuer of the server certificate, for example `CN=Zabbix CA,O=Company`. The attributes listed here must all be present in the certificate; their order does not matter, and a comma inside a value is escaped with a backslash. As in the Zabbix agent, the host name is not matched against the certificate — the issuer and subject are what identify the server.
+
+**tls_server_cert_subject**  
+        Expected subject of the server certificate, in the same format as tls_server_cert_issuer.
+
+**tls_cipher_cert**  
+        OpenSSL cipher string used to select the certificate cipher suites, for the rare case when the Zabbix server is restricted to a specific one.
+
+<p>&nbsp;</p>
+
+All of the tls_* parameters except the cipher ones (tls_cipher_psk, tls_cipher_cert) can be overridden from the command line, which is convenient for checking a setting without editing the config file:
+
+```bash
+mamonsu -c /etc/mamonsu/agent.conf --zabbix-tls-connect psk         --zabbix-tls-psk-identity 'PSK 001' --zabbix-tls-psk-file /etc/zabbix/zabbix_agentd.psk
+```
+
 <p>&nbsp;</p>
 
 **[agent]**  

mamonsu/__init__.py

@@ -1,7 +1,7 @@
 __author__ = 'Dmitry Vasilyev'
 __author_email__ = 'info@postgrespro.ru'
 __description__ = 'Monitoring agent for PostgreSQL'
-__version__ = '3.5.17'
+__version__ = '3.5.17.1'
 __licence__ = 'BSD'
 
 __url__ = 'https://github.com/postgrespro/mamonsu'

mamonsu/lib/config.py

@@ -56,6 +56,18 @@ def __init__(self, cfg_file=None, plugin_directories=None):
         config.set('zabbix', 'port', str(10051))
         config.set('zabbix', 'timeout', str(15))
         config.set('zabbix', 're_send', str(False))
+        # unencrypted keeps the plain TCP connection used before 3.5.18
+        config.set('zabbix', 'tls_connect', 'unencrypted')
+        config.set('zabbix', 'tls_psk_identity', str(None))
+        config.set('zabbix', 'tls_psk_file', str(None))
+        config.set('zabbix', 'tls_cipher_psk', str(None))
+        config.set('zabbix', 'tls_cipher_cert', str(None))
+        config.set('zabbix', 'tls_ca_file', str(None))
+        config.set('zabbix', 'tls_crl_file', str(None))
+        config.set('zabbix', 'tls_cert_file', str(None))
+        config.set('zabbix', 'tls_key_file', str(None))
+        config.set('zabbix', 'tls_server_cert_issuer', str(None))
+        config.set('zabbix', 'tls_server_cert_subject', str(None))
 
         config.add_section('metric_log')
         config.set('metric_log', 'enabled', str(False))

mamonsu/lib/parser.py

@@ -15,6 +15,19 @@
     usage_msg += """    -d                 daemonize
 """
 
+usage_msg += """    Encryption of the connection to the Zabbix server, overrides the
+    [zabbix] section of the config file:
+        --zabbix-tls-connect              <unencrypted|psk|cert>
+        --zabbix-tls-psk-identity         <identity>
+        --zabbix-tls-psk-file             <file>
+        --zabbix-tls-ca-file              <file>
+        --zabbix-tls-crl-file             <file>
+        --zabbix-tls-cert-file            <file>
+        --zabbix-tls-key-file             <file>
+        --zabbix-tls-server-cert-issuer   <distinguished name>
+        --zabbix-tls-server-cert-subject  <distinguished name>
+"""
+
 usage_msg += """        --version      prints version information, then exits
         --help         shows this help message, then exits
 
@@ -296,6 +309,19 @@ def parse_args():
     parser.add_option('--zabbix-file', dest='zabbix_file', default='/var/log/mamonsu/localhost.log')
     # log level to send metrics
     parser.add_option('--zabbix-log-level', dest='zabbix_log_level', default='INFO')
+    # encryption of the connection to the Zabbix server, overrides [zabbix] of the
+    # config file; without these options the settings from the config file are used
+    parser.add_option('--zabbix-tls-connect', dest='zabbix_tls_connect', default=None)
+    parser.add_option('--zabbix-tls-psk-identity', dest='zabbix_tls_psk_identity', default=None)
+    parser.add_option('--zabbix-tls-psk-file', dest='zabbix_tls_psk_file', default=None)
+    parser.add_option('--zabbix-tls-ca-file', dest='zabbix_tls_ca_file', default=None)
+    parser.add_option('--zabbix-tls-crl-file', dest='zabbix_tls_crl_file', default=None)
+    parser.add_option('--zabbix-tls-cert-file', dest='zabbix_tls_cert_file', default=None)
+    parser.add_option('--zabbix-tls-key-file', dest='zabbix_tls_key_file', default=None)
+    parser.add_option(
+        '--zabbix-tls-server-cert-issuer', dest='zabbix_tls_server_cert_issuer', default=None)
+    parser.add_option(
+        '--zabbix-tls-server-cert-subject', dest='zabbix_tls_server_cert_subject', default=None)
 
     # check unknown options
     args, commands = parser.parse_args()

mamonsu/lib/runner.py

@@ -19,6 +19,23 @@
     from mamonsu.plugins.system.linux.scripts import Scripts
 
 
+def apply_zabbix_tls_args(cfg, args):
+    """Let the command line override the TLS settings of the config file."""
+    options = (
+        ('tls_connect', args.zabbix_tls_connect),
+        ('tls_psk_identity', args.zabbix_tls_psk_identity),
+        ('tls_psk_file', args.zabbix_tls_psk_file),
+        ('tls_ca_file', args.zabbix_tls_ca_file),
+        ('tls_crl_file', args.zabbix_tls_crl_file),
+        ('tls_cert_file', args.zabbix_tls_cert_file),
+        ('tls_key_file', args.zabbix_tls_key_file),
+        ('tls_server_cert_issuer', args.zabbix_tls_server_cert_issuer),
+        ('tls_server_cert_subject', args.zabbix_tls_server_cert_subject))
+    for key, value in options:
+        if value is not None:
+            cfg.config.set('zabbix', key, value)
+
+
 def start():
     def quit_handler(_signo=None, _stack_frame=None):
         logging.info("Bye bye!")
@@ -67,6 +84,7 @@ def quit_handler(_signo=None, _stack_frame=None):
             cfg.config.set('zabbix', 'port', args.zabbix_port)
             cfg.config.set('zabbix', 'client', args.zabbix_client)
             cfg.config.set('log', 'level', args.zabbix_log_level)
+            apply_zabbix_tls_args(cfg, args)
 
             supervisor = Supervisor(cfg)
             supervisor.send_file_zabbix(cfg, args.zabbix_file)
@@ -186,6 +204,7 @@ def quit_handler(_signo=None, _stack_frame=None):
     if len(commands) > 0:
         print_total_help()
     cfg = Config(args.config_file, args.plugins_dirs)
+    apply_zabbix_tls_args(cfg, args)
 
     # simple daemon
     if args.daemon: