feat(roles/postgresql_server)!: review against the PostgreSQL source

CHANGELOG.md

@@ -10,6 +10,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Breaking Changes
 
+* **role:postgresql_server**: Databases are created with the `C.UTF-8` locale instead of `en_US.UTF-8`, which failed on hosts with only the minimal glibc language pack. `C.UTF-8` sorts text by code point instead of by language rules. Existing databases are unaffected. To create new databases as before, set `lc_collate: 'en_US.UTF-8'` and `lc_ctype: 'en_US.UTF-8'` on their entries in `postgresql_server__databases__*_var`.
+* **role:postgresql_server, playbook:setup_mastodon**: `postgresql_server__version` is mandatory, and PostgreSQL is only installed from the PostgreSQL Yum Repository instead of optionally from the distribution packages. Set the major version in your inventory, for example `postgresql_server__version: '18'`. On a host that already runs PostgreSQL from the distribution packages, the run aborts before it installs anything; move the data to the PostgreSQL Yum Repository release first, as described in the role README.
+* **role:postgresql_server**: `postgresql_server__pg_hba_entries` is replaced by `postgresql_server__pg_hba_local_entries__host_var` / `__group_var` for `local` records and `postgresql_server__pg_hba_host_entries__host_var` / `__group_var` for all `host*` records. Both are merged with the role defaults instead of replacing them, and an entry with `state: 'absent'` removes a default. Move your `local` entries to the first variable without their `type` key, and all other entries to the second, where `address` is mandatory. Entries from the inventory are written before the role defaults. The run aborts as long as the old variable is set.
 * **role:icinga2_agent**: If the agent cannot get a PKI ticket from the Icinga2 master, the run aborts and names the cause and the fix. Until now the role set the agent up anyway, which also replaced the signed certificate of an agent that was already connected with an unsigned one. If you sign agent certificates on the master by hand, or do not set `icinga2_agent__icinga2_api_user_login`, set `icinga2_agent__skip_pki_ticket: true`.
 * **role:dnf_makecache**: `dnf_makecache__service_enabled` and `dnf_makecache__service_state` are gone; remove them from your inventory. The role only manages `dnf-makecache.timer` now, since `dnf-makecache.service` cannot be enabled at boot and only runs when the timer triggers it. `dnf_makecache__service_enabled` never had an effect, but a run against an unchanged host reported a change for it. A host that set `dnf_makecache__service_state: 'started'` no longer runs `dnf makecache` on every run of the role. Use `dnf_makecache__timer_enabled` and `dnf_makecache__timer_state` for the periodic cache refresh.
 * **role:kibana**: The session cookie always carries the `Secure` flag, also behind a reverse proxy that terminates TLS, where Kibana left the flag off. A Kibana that browsers reach over plain HTTP no longer logs anyone in until `kibana__xpack_security_secure_cookies: false` is set. Remove `xpack.security.secureCookies` from `kibana__raw` if you set it there.
@@ -35,6 +38,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+* **role:postgresql_server**: A changed `pg_hba.conf` or `postgresql_server__conf_password_encryption` reloads PostgreSQL instead of restarting it, so open connections are kept; the first run after the update still restarts it once.
+* **role:postgresql_server**: The configuration is checked before PostgreSQL is restarted, so a broken setting in `postgresql.conf` or `pg_hba.conf` aborts the run with the error message instead of leaving PostgreSQL down after the restart.
 * **role:repo_postgresql**: The PostgreSQL version repositories take precedence over the distribution's packages of the same name, so on RHEL 10 an install or update no longer switches a PostgreSQL server from the PGDG build to the AppStream build, which uses a different file layout.
 * **role:apache_httpd**: Responses of type `text/markdown` are compressed like HTML, so the Markdown versions of pages that CMSs such as Grav hand to AI agents no longer go out uncompressed.
 * **role:grav**: The README lists setting `session.secure` as a manual step behind a reverse proxy that terminates TLS, where Grav sends its session cookies without the `Secure` flag.
@@ -45,6 +50,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+* **role:postgresql_server**: `postgresql-dump` also saves the roles and tablespaces (`globals.sql.gz`), without which a restored database fails on every object owner, no longer skips the `postgres` database and databases whose name starts with `template`, and uses the client tools of the installed PostgreSQL version.
+* **role:postgresql_server**: Clients connecting to `localhost` are accepted over IPv6 as well, where they were rejected with `no pg_hba.conf entry for host "::1"`.
 * **role:postfix**: `postfix__compatibility_level` takes effect on Debian and Ubuntu as well, and defaults to the level the distribution ships, so Debian 13 and Ubuntu 26.04 run at `3.9` instead of `3.6` ([#364](https://github.com/Linuxfabrik/lfops/issues/364)).
 * **role:mariadb_server**: On RHEL 10 with a `selinux-policy-targeted` older than `42.1.18-4.el10_2.3`, MariaDB runs confined in `mysqld_t` again, so web applications such as WordPress or Nextcloud reach its socket. Until now it ran in `initrc_t` there, and PHP-FPM failed to connect until the SELinux policy was updated and MariaDB restarted.
 * **role:apache_httpd**: The role hands the content of the document root to the web server user, but leaves the directory itself to the httpd package, whose tmpfiles rule resets it to `root` on every boot and after some package installations. A second run on a fresh host no longer reports the ownership as changed.

COMPATIBILITY.md

@@ -122,7 +122,7 @@ Which Ansible role is proven to run on which OS?
 | podman_containers                     |        |        |  (x)   |   x    |   (x)   |           |           |           |                                              |
 | policycoreutils                       |        |        |   x    |   x    |    x    |           |           |           | Fedora 35                                    |
 | postfix                               |   x    |   x    |   x    |   x    |    x    |    (x)    |    (x)    |    (x)    | Fedora 35                                    |
-| postgresql_server                     |        |        |   x    |  (x)   |   (x)   |           |           |           |                                              |
+| postgresql_server                     |        |        |   x    |   x    |    x    |           |           |           |                                              |
 | proxysql                              |        |        |   x    |  (x)   |   (x)   |           |           |           |                                              |
 | python                                |   x    |   x    |   x    |   x    |    x    |    (x)    |    (x)    |    (x)    | Windows                                      |
 | python_venv                           |   x    |   x    |   x    |   x    |    x    |    (x)    |    (x)    |    (x)    | Fedora 35                                    |

extensions/molecule/postgresql_server/converge.yml

@@ -0,0 +1,2 @@
+- name: 'Converge postgresql_server playbook'
+  ansible.builtin.import_playbook: 'linuxfabrik.lfops.postgresql_server'

extensions/molecule/postgresql_server/inventory/group_vars/systems_under_test.yml

@@ -0,0 +1,31 @@
+# Variables the playbook under test needs, applied to every system under test.
+#
+# Where a value is compared in verify.yml, it deliberately differs from the role default, so the
+# check can tell "the running server uses what the inventory asked for" apart from "the running
+# server happens to use the packaged default".
+postgresql_server__version: '18'
+
+postgresql_server__conf_listen_addresses:
+  - '*'
+postgresql_server__conf_max_connections: 150
+
+postgresql_server__databases__group_var:
+  - name: 'app'
+    owner: 'app'
+  # the dump used to skip every database whose name starts with "template"
+  - name: 'template_app'
+    owner: 'app'
+
+# Replaces the catch-all default for Unix-socket connections in place, and adds a more specific
+# entry that only works if the role renders it before the defaults.
+postgresql_server__pg_hba_local_entries__group_var:
+  - database: 'all'
+    user: 'all'
+    auth_method: 'reject'
+  - database: 'app'
+    user: 'app'
+    auth_method: 'scram-sha-256'
+
+postgresql_server__users__group_var:
+  - username: 'app'
+    password: 'linuxfabrik'

extensions/molecule/postgresql_server/inventory/hosts.yml

@@ -0,0 +1,15 @@
+# yamllint disable rule:empty-values
+
+# postgresql_server targets 'lfops_postgresql_server' (see playbooks/postgresql_server.yml: hosts).
+#
+# Only Red Hat-family hosts run: the role installs PostgreSQL from the PostgreSQL Yum Repository
+# and has no Debian support (see COMPATIBILITY.md).
+lfops_postgresql_server:
+  children:
+    systems_under_test:
+
+systems_under_test:
+  hosts:
+    rocky8-vm:
+    rocky9-vm:
+    rocky10-vm:

extensions/molecule/postgresql_server/molecule.yml

@@ -0,0 +1 @@
+# Molecule scenario marker

extensions/molecule/postgresql_server/verify.yml

@@ -0,0 +1,198 @@
+# verify.yml runs after converge and again after the idempotence step. It asserts the observable
+# end state, not that individual tasks ran: every setting is read back from the running server,
+# and pg_hba.conf is proven by connections that have to succeed or be rejected.
+- name: 'Verify postgresql_server is running and uses the configured values'
+  hosts: 'systems_under_test'
+  gather_facts: true
+  become: true
+
+  vars:
+
+    __molecule__postgresql_bin_dir: '/usr/pgsql-{{ postgresql_server__version }}/bin'
+    __molecule__postgresql_service_name: 'postgresql-{{ postgresql_server__version }}.service'
+    __molecule__psql: '{{ __molecule__postgresql_bin_dir }}/psql --no-align --tuples-only --no-psqlrc'
+    __molecule__dump_directory: '/backup/postgresql-dump'
+
+  tasks:
+
+    - name: 'Get the service facts'
+      ansible.builtin.service_facts:  # yamllint disable-line rule:empty-values
+
+    - name: 'Assert the PostgreSQL service is enabled and running'
+      ansible.builtin.assert:
+        that:
+          - 'ansible_facts["services"][__molecule__postgresql_service_name]["state"] == "running"'
+          - 'ansible_facts["services"][__molecule__postgresql_service_name]["status"] == "enabled"'
+        fail_msg: '{{ __molecule__postgresql_service_name }} is {{ ansible_facts["services"][__molecule__postgresql_service_name] | d("absent") }}'
+
+    # Ask the running server what it applied, which proves the chain from group_vars through
+    # conf.d/z00-linuxfabrik.conf and the include_dir line to the server.
+    - name: 'psql --command="SHOW ..."'
+      ansible.builtin.command: '{{ __molecule__psql }} --command="SHOW {{ item["setting"] }}"'
+      become: true
+      become_user: 'postgres'
+      register: '__molecule__postgresql_show_result'
+      changed_when: false
+      loop:
+        - setting: 'listen_addresses'
+          expected: '*'
+        - setting: 'max_connections'
+          expected: '{{ postgresql_server__conf_max_connections | string }}'
+      loop_control:
+        label: '{{ item["setting"] }}'
+
+    - name: 'Assert the running server applied the configured values'
+      ansible.builtin.assert:
+        that: 'item["stdout"] == item["item"]["expected"]'
+        fail_msg: '{{ item["item"]["setting"] }} is {{ item["stdout"] }}, expected {{ item["item"]["expected"] }}'
+      loop: '{{ __molecule__postgresql_show_result["results"] }}'
+      loop_control:
+        label: '{{ item["item"]["setting"] }}'
+
+    # The databases in the inventory set no locale, so they get the role default.
+    - name: 'psql --command="SELECT datcollate, datctype FROM pg_database WHERE datname = ''app''"'
+      ansible.builtin.command: '{{ __molecule__psql }} --field-separator=" " --command="SELECT datcollate, datctype FROM pg_database WHERE datname = ''app''"'
+      become: true
+      become_user: 'postgres'
+      register: '__molecule__postgresql_locale_result'
+      changed_when: false
+
+    - name: 'Assert the database was created with the C.UTF-8 locale'
+      ansible.builtin.assert:
+        that: '__molecule__postgresql_locale_result["stdout"] == "C.UTF-8 C.UTF-8"'
+        fail_msg: 'datcollate and datctype of app are {{ __molecule__postgresql_locale_result["stdout"] }}'
+
+    - name: 'psql --command="SELECT ... FROM pg_hba_file_rules WHERE error IS NOT NULL"'
+      ansible.builtin.command: '{{ __molecule__psql }} --command="SELECT line_number, error FROM pg_hba_file_rules WHERE error IS NOT NULL"'
+      become: true
+      become_user: 'postgres'
+      register: '__molecule__postgresql_hba_errors_result'
+      changed_when: false
+
+    - name: 'Assert pg_hba.conf has no errors'
+      ansible.builtin.assert:
+        that: '__molecule__postgresql_hba_errors_result["stdout"] | length == 0'
+        fail_msg: 'pg_hba.conf contains errors: {{ __molecule__postgresql_hba_errors_result["stdout"] }}'
+
+    # The IPv4 and IPv6 loopback defaults. `localhost` resolves to ::1 first, and libpq does not
+    # fall back to 127.0.0.1 when the server rejects the connection on ::1.
+    - name: 'psql over TCP as app'
+      ansible.builtin.command: '{{ __molecule__psql }} "host={{ item }} user=app dbname=app" --command="SELECT inet_client_addr()"'
+      environment:
+        PGPASSWORD: 'linuxfabrik'
+      register: '__molecule__postgresql_tcp_result'
+      changed_when: false
+      loop: '{{ ["127.0.0.1", "localhost"] + (__molecule__has_ipv6_loopback | ternary(["::1"], [])) }}'
+      vars:
+        # Gate on the loopback interface, not on ansible_facts['all_ipv6_addresses']: that fact
+        # lists the real addresses only and never contains ::1.
+        __molecule__has_ipv6_loopback: '{{ "::1" in (ansible_facts["lo"]["ipv6"] | d([]) | map(attribute="address") | list) }}'
+
+    - name: 'Assert app connects over TCP from the expected address'
+      ansible.builtin.assert:
+        that: 'item["stdout"] in (item["item"] == "localhost") | ternary(["127.0.0.1", "::1"], [item["item"]])'
+        fail_msg: 'connecting to {{ item["item"] }} came from {{ item["stdout"] }}'
+      loop: '{{ __molecule__postgresql_tcp_result["results"] }}'
+      loop_control:
+        label: '{{ item["item"] }}'
+
+    # The inventory replaces the `local all all` default with `reject` and adds `local app app`.
+    # The connection to app only works if the added entry is rendered before the defaults, the
+    # one to postgres is only rejected if the default was replaced.
+    - name: 'psql over the Unix socket as app to the database app'
+      ansible.builtin.command: '{{ __molecule__psql }} "user=app dbname=app" --command="SELECT current_user"'
+      environment:
+        PGPASSWORD: 'linuxfabrik'
+      register: '__molecule__postgresql_socket_app_result'
+      changed_when: false
+
+    - name: 'psql over the Unix socket as app to the database postgres'
+      ansible.builtin.command: '{{ __molecule__psql }} "user=app dbname=postgres" --command="SELECT current_user"'
+      environment:
+        PGPASSWORD: 'linuxfabrik'
+      register: '__molecule__postgresql_socket_postgres_result'
+      changed_when: false
+      failed_when: false
+
+    - name: 'Assert the local pg_hba entries are applied in the expected order'
+      ansible.builtin.assert:
+        that:
+          - '__molecule__postgresql_socket_app_result["stdout"] == "app"'
+          - '__molecule__postgresql_socket_postgres_result["rc"] != 0'
+          - '"rejects connection" in __molecule__postgresql_socket_postgres_result["stderr"]'
+        fail_msg: 'socket to postgres as app: rc={{ __molecule__postgresql_socket_postgres_result["rc"] }} {{ __molecule__postgresql_socket_postgres_result["stderr"] }}'
+
+    # postgresql-dump: dump a database with content and restore it, with the tools the script is
+    # expected to use.
+    - name: 'Create a table with a row in the database app'
+      ansible.builtin.command: >-
+        {{ __molecule__psql }} "user=app dbname=app"
+        --command="CREATE TABLE IF NOT EXISTS molecule_marker (id int)"
+        --command="INSERT INTO molecule_marker SELECT 1 WHERE NOT EXISTS (SELECT 1 FROM molecule_marker)"
+      environment:
+        PGPASSWORD: 'linuxfabrik'
+      changed_when: false
+
+    - name: 'systemctl start postgresql-dump.service'
+      ansible.builtin.systemd:
+        name: 'postgresql-dump.service'
+        state: 'started'
+      changed_when: false
+
+    - name: 'stat {{ __molecule__dump_directory }}/*.sql.gz'
+      ansible.builtin.stat:
+        path: '{{ __molecule__dump_directory }}/{{ item }}'
+      register: '__molecule__postgresql_dump_stat_result'
+      loop:
+        - 'dump-app.sql.gz'
+        - 'dump-postgres.sql.gz'
+        - 'dump-template1.sql.gz'
+        - 'dump-template_app.sql.gz'
+        - 'globals.sql.gz'
+
+    - name: 'Assert every database and the globals were dumped'
+      ansible.builtin.assert:
+        that: 'item["stat"]["exists"]'
+        fail_msg: '{{ item["item"] }} is missing'
+      loop: '{{ __molecule__postgresql_dump_stat_result["results"] }}'
+      loop_control:
+        label: '{{ item["item"] }}'
+
+    - name: 'zcat globals.sql.gz; zcat dump-app.sql.gz'
+      ansible.builtin.command: 'zcat {{ __molecule__dump_directory }}/{{ item }}'
+      register: '__molecule__postgresql_dump_content_result'
+      changed_when: false
+      loop:
+        - 'globals.sql.gz'
+        - 'dump-app.sql.gz'
+
+    - name: 'Assert the dumps contain the role and were made with the matching pg_dump'
+      ansible.builtin.assert:
+        that:
+          - '__molecule__postgresql_dump_content_result["results"][0]["stdout"] is search("CREATE ROLE app;")'
+          - '__molecule__postgresql_dump_content_result["results"][1]["stdout"] is search("Dumped from database version " ~ __molecule__server_major ~ "\.")'
+          - '__molecule__postgresql_dump_content_result["results"][1]["stdout"] is search("Dumped by pg_dump version " ~ __molecule__server_major ~ "\.")'
+        fail_msg: 'unexpected dump content, expected PostgreSQL {{ __molecule__server_major }}'
+      vars:
+        __molecule__server_major: '{{
+            __molecule__postgresql_dump_content_result["results"][1]["stdout"]
+            | regex_search("Dumped from database version (\d+)", "\1") | first
+          }}'
+
+    # The dumps are readable by root only, so zcat runs as root and only psql as postgres.
+    - name: 'Restore dump-app.sql.gz into the database app_restore'
+      ansible.builtin.shell:
+        cmd: >-
+          set -o pipefail;
+          runuser --user=postgres -- {{ __molecule__psql }} --command="DROP DATABASE IF EXISTS app_restore" &&
+          runuser --user=postgres -- {{ __molecule__psql }} --command="CREATE DATABASE app_restore OWNER app" &&
+          zcat {{ __molecule__dump_directory }}/dump-app.sql.gz | runuser --user=postgres -- {{ __molecule__psql }} --quiet --dbname=app_restore &&
+          runuser --user=postgres -- {{ __molecule__psql }} --dbname=app_restore --command="SELECT tableowner || ':' || (SELECT count(*) FROM molecule_marker) FROM pg_tables WHERE tablename = 'molecule_marker'"
+        executable: '/bin/bash'
+      register: '__molecule__postgresql_restore_result'
+      changed_when: false
+
+    - name: 'Assert the restored table has its owner and its row'
+      ansible.builtin.assert:
+        that: '__molecule__postgresql_restore_result["stdout_lines"][-1] == "app:1"'
+        fail_msg: 'restore returned {{ __molecule__postgresql_restore_result["stdout"] }} {{ __molecule__postgresql_restore_result["stderr"] }}'