From a40cfef8c3d59f7eba663973f730b3370562e727 Mon Sep 17 00:00:00 2001 From: hotragn Date: Thu, 27 Aug 2026 20:14:48 -0400 Subject: [PATCH] Initialise embedded PostgreSQL where a platform volume lets it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `EMBEDDED_POSTGRES=on` could not create its cluster on a platform whose volume is an ext4 mount, which is most of them, and it failed differently depending on which of two paths the volume was mounted at. Neither worked, and the two suggestions in this repo disagreed: docs/deployment.md said /var/lib/postgresql/data, the Dockerfile comment said /var/lib/postgresql. Mounted at the parent, the mount arrives owned by root, `data` is not in it, and the image's build-time chown is underneath the mount. `initdb` runs as `postgres` by then and cannot create the directory. `postgres-init` is a oneshot whose `up` runs as root, so it creates and chowns it first. That also fixes plain `docker run -v openbot-data:/var/lib/postgresql`, which relied on the hidden chown. Mounted on the data directory itself, the mount holds a `lost+found` and `initdb` refuses a directory with anything in it. The documented mount is now the parent, leaving `data` a subdirectory — what initdb's own hint asks for and what the Dockerfile already said. A Docker named volume at the old path arrives empty rather than with a lost+found, so a deployment already working that way keeps working and needs no change; this is why the data directory is not moved into a subdirectory of itself, which would have looked for a cluster somewhere that deployment has none and initialised a fresh one over the top. The failure also said nothing useful. `api` waits on `postgres` and `migrate`, so neither started, the container came up regardless, the platform reported the deploy a success, and the URL served a persistent 502 with the reason only in the container log. A data directory holding no cluster and not empty is now refused with a sentence naming the mount to use instead. Reported by Jerel Velarde in #269, with the logs for both mount paths, which is what made the two failure modes separable. Closes #269 Verification is honest about its limits: no Docker on this machine, so the image was not built or booted. `sh -n` is clean, and the decision logic was exercised against fixtures for all four states — no volume, volume at the parent, volume on the data directory, and an existing cluster — with the last confirming an initialised cluster is still never touched. CI's image job boots the no-volume path; neither it nor I cover a real platform volume. --- CHANGELOG.md | 29 +++++++++++++++++++++++++++++ docker/s6/scripts/postgres-init.sh | 24 ++++++++++++++++++++++++ docs/deployment.md | 14 +++++++++++--- 3 files changed, 64 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3400b3e7..3d7584fc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,35 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged. ## Unreleased +### Embedded PostgreSQL initialises on a platform volume, and says so when it cannot + +`EMBEDDED_POSTGRES=on` could not create its cluster on a platform whose persistent volume is an ext4 +mount — Railway, and by the same mechanism most others — and it failed differently depending on where +the volume was mounted. Neither of the two paths this repo suggested worked, and the two suggestions +disagreed with each other: `docs/deployment.md` said `/var/lib/postgresql/data`, the `Dockerfile` +comment said `/var/lib/postgresql`. + +Mounted at the parent, the mount arrives owned by root, `data` is not in it, and the image's +build-time `chown` is hidden underneath — so `initdb`, which has already dropped to the `postgres` +user, cannot create the directory. `postgres-init` now creates and chowns it first, as root, which is +the only step in a position to. This also fixes the plain +`docker run -v openbot-data:/var/lib/postgresql` case, which relied entirely on that hidden chown. + +Mounted directly on the data directory, the mount arrives holding a `lost+found`, and `initdb` will +not initialise into a directory with anything in it. **The documented mount is now the parent, +`/var/lib/postgresql`**, which leaves `data` an ordinary subdirectory — what PostgreSQL's own hint +asks for, and what the `Dockerfile` already said. A volume already mounted at +`/var/lib/postgresql/data` and working — a Docker named volume, which arrives empty rather than with a +`lost+found` — keeps working and needs no change. + +**The failure said nothing useful.** `api` waits on `postgres` and `migrate`, so neither started, the +container came up anyway, the platform reported the deploy a success, and the public URL served a +persistent 502 with the real reason visible only in the container log. A data directory that holds no +cluster and is not empty is now refused with a sentence naming the mount to use instead. + +Reported by [@jerelvelarde](https://github.com/CopilotKit/OpenBot/issues/269) with the container logs +for both mount paths, which is what made the two failure modes separable. + ### The framework Bot answers on 5.6-tier models, and can be told how hard to think Pointing `BOT_MODEL` at a `gpt-5.6-*` model gave a Bot that started, reported healthy, and then said diff --git a/docker/s6/scripts/postgres-init.sh b/docker/s6/scripts/postgres-init.sh index 69bbeb14..ba9a1ffe 100755 --- a/docker/s6/scripts/postgres-init.sh +++ b/docker/s6/scripts/postgres-init.sh @@ -9,6 +9,30 @@ set -eu DATA=/var/lib/postgresql/data if [ ! -s "$DATA/PG_VERSION" ]; then + # Created and owned here, as root, because this is the only step in a position to do it. + # + # The image creates and chowns this at build time, and a volume mounted over /var/lib/postgresql at + # run time hides that completely: the mount arrives owned by root, `data` is not in it, and nothing + # else in the image puts either back. There is no `fix-attrs.d` under docker/s6, so the built-in + # s6-overlay service of that name has nothing to act on. `postgres-init` is a oneshot whose `up` + # runs as root, so it can, and `initdb` a line below cannot — it has already dropped to `postgres`. + mkdir -p "$DATA" + chown postgres:postgres "$DATA" + + # A volume mounted directly AT the data directory, rather than at its parent, arrives holding + # `lost+found` on any platform whose volume is an ext4 mount — which is most of them. `initdb` + # refuses a directory with anything in it, and its own hint says to use a subdirectory instead. + # + # Said here, naming this image's answer, rather than left to that hint. The failure is otherwise a + # generic message about mount points in a container log, while `api` never starts because it depends + # on `postgres` and `migrate`, the platform reports the deploy a success, and the public URL serves + # a 502 with nothing on it to explain why. + if [ -n "$(ls -A "$DATA" 2>/dev/null)" ]; then + echo "postgres-init: $DATA holds no cluster and is not empty, so initdb cannot use it." >&2 + echo "postgres-init: mount the volume at /var/lib/postgresql rather than at $DATA. A volume mounted directly on the data directory arrives with a lost+found in it, and PostgreSQL will not initialise into that." >&2 + exit 1 + fi + s6-setuidgid postgres /usr/lib/postgresql/16/bin/initdb -D "$DATA" -A trust -U openbot >/dev/null s6-setuidgid postgres /usr/lib/postgresql/16/bin/pg_ctl -D "$DATA" -o "-c listen_addresses=127.0.0.1" -w start >/dev/null s6-setuidgid postgres /usr/lib/postgresql/16/bin/createdb -U openbot openbot diff --git a/docs/deployment.md b/docs/deployment.md index da5b23ee..e8e64447 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -11,7 +11,7 @@ docker run -p 3001:3001 --env-file .env openbot # Or one inside the container. Nothing else to provision. docker run -p 3001:3001 --env-file .env \ - -e EMBEDDED_POSTGRES=on -v openbot-data:/var/lib/postgresql/data openbot + -e EMBEDDED_POSTGRES=on -v openbot-data:/var/lib/postgresql openbot ``` ## What is in the image, and what is not @@ -24,8 +24,16 @@ process beside it. the database and the `vector` extension the first time, and runs the migrations on every start. It listens on loopback only and is never published, so there is no password to manage. -Give it a volume at `/var/lib/postgresql/data`. Without one, a redeploy takes the audit trail with -it, and the audit trail is the product. Platforms that offer no persistent volume are the ones to +Give it a volume at `/var/lib/postgresql` — the parent, not the data directory itself. Without one, +a redeploy takes the audit trail with it, and the audit trail is the product. + +**Mount the parent, not `/var/lib/postgresql/data`.** On any platform whose volume is an ext4 mount, +and that is most of them, the mount arrives holding a `lost+found` directory. `initdb` will not +initialise into a directory that has anything in it, so mounting it directly on the data directory +leaves the cluster uncreated — and because `api` waits on `postgres` and `migrate`, the container +starts, the platform reports the deploy a success, and the URL serves a 502. Mounting the parent +leaves `data` as an ordinary subdirectory, which is what PostgreSQL asks for. A Docker named volume +works either way; a platform volume does not. Platforms that offer no persistent volume are the ones to point at a managed database instead: set `DATABASE_URL` and leave `EMBEDDED_POSTGRES` off. The `vector` extension must be enabled there; RDS, Cloud SQL and Azure Database all support it, none enable it for you.