diff --git a/CHANGELOG.md b/CHANGELOG.md index f3338284..95f9be68 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,34 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged. ## Unreleased +### Embedded PostgreSQL initialises on a platform volume, and says so when it cannot + +`EMBEDDED_POSTGRES=on` could not create its cluster on a platform whose persistent volume is an ext4 +mount — Railway, and by the same mechanism most others — and it failed differently depending on where +the volume was mounted. Neither of the two paths this repo suggested worked, and the two suggestions +disagreed with each other: `docs/deployment.md` said `/var/lib/postgresql/data`, the `Dockerfile` +comment said `/var/lib/postgresql`. + +Mounted at the parent, the mount arrives owned by root, `data` is not in it, and the image's +build-time `chown` is hidden underneath — so `initdb`, which has already dropped to the `postgres` +user, cannot create the directory. `postgres-init` now creates and chowns it first, as root, which is +the only step in a position to. This also fixes the plain +`docker run -v openbot-data:/var/lib/postgresql` case, which relied entirely on that hidden chown. + +Mounted directly on the data directory, the mount arrives holding a `lost+found`, and `initdb` will +not initialise into a directory with anything in it. **The documented mount is now the parent, +`/var/lib/postgresql`**, which leaves `data` an ordinary subdirectory — what PostgreSQL's own hint +asks for, and what the `Dockerfile` already said. A volume already mounted at +`/var/lib/postgresql/data` and working — a Docker named volume, which arrives empty rather than with a +`lost+found` — keeps working and needs no change. + +**The failure said nothing useful.** `api` waits on `postgres` and `migrate`, so neither started, the +container came up anyway, the platform reported the deploy a success, and the public URL served a +persistent 502 with the real reason visible only in the container log. A data directory that holds no +cluster and is not empty is now refused with a sentence naming the mount to use instead. + +Reported by [@jerelvelarde](https://github.com/CopilotKit/OpenBot/issues/269) with the container logs +for both mount paths, which is what made the two failure modes separable. ### A sign-in a site opens in a new window is shown, and can be clicked A Bot's browser was bound to the page it launched with, and to nothing the site opened afterwards. diff --git a/docker/s6/scripts/postgres-init.sh b/docker/s6/scripts/postgres-init.sh index 69bbeb14..ba9a1ffe 100755 --- a/docker/s6/scripts/postgres-init.sh +++ b/docker/s6/scripts/postgres-init.sh @@ -9,6 +9,30 @@ set -eu DATA=/var/lib/postgresql/data if [ ! -s "$DATA/PG_VERSION" ]; then + # Created and owned here, as root, because this is the only step in a position to do it. + # + # The image creates and chowns this at build time, and a volume mounted over /var/lib/postgresql at + # run time hides that completely: the mount arrives owned by root, `data` is not in it, and nothing + # else in the image puts either back. There is no `fix-attrs.d` under docker/s6, so the built-in + # s6-overlay service of that name has nothing to act on. `postgres-init` is a oneshot whose `up` + # runs as root, so it can, and `initdb` a line below cannot — it has already dropped to `postgres`. + mkdir -p "$DATA" + chown postgres:postgres "$DATA" + + # A volume mounted directly AT the data directory, rather than at its parent, arrives holding + # `lost+found` on any platform whose volume is an ext4 mount — which is most of them. `initdb` + # refuses a directory with anything in it, and its own hint says to use a subdirectory instead. + # + # Said here, naming this image's answer, rather than left to that hint. The failure is otherwise a + # generic message about mount points in a container log, while `api` never starts because it depends + # on `postgres` and `migrate`, the platform reports the deploy a success, and the public URL serves + # a 502 with nothing on it to explain why. + if [ -n "$(ls -A "$DATA" 2>/dev/null)" ]; then + echo "postgres-init: $DATA holds no cluster and is not empty, so initdb cannot use it." >&2 + echo "postgres-init: mount the volume at /var/lib/postgresql rather than at $DATA. A volume mounted directly on the data directory arrives with a lost+found in it, and PostgreSQL will not initialise into that." >&2 + exit 1 + fi + s6-setuidgid postgres /usr/lib/postgresql/16/bin/initdb -D "$DATA" -A trust -U openbot >/dev/null s6-setuidgid postgres /usr/lib/postgresql/16/bin/pg_ctl -D "$DATA" -o "-c listen_addresses=127.0.0.1" -w start >/dev/null s6-setuidgid postgres /usr/lib/postgresql/16/bin/createdb -U openbot openbot diff --git a/docs/deployment.md b/docs/deployment.md index 827731a9..b47462e4 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -11,7 +11,7 @@ docker run -p 3001:3001 --env-file .env openbot # Or one inside the container. Nothing else to provision. docker run -p 3001:3001 --env-file .env \ - -e EMBEDDED_POSTGRES=on -v openbot-data:/var/lib/postgresql/data openbot + -e EMBEDDED_POSTGRES=on -v openbot-data:/var/lib/postgresql openbot ``` ## What is in the image, and what is not @@ -24,8 +24,16 @@ process beside it. the database and the `vector` extension the first time, and runs the migrations on every start. It listens on loopback only and is never published, so there is no password to manage. -Give it a volume at `/var/lib/postgresql/data`. Without one, a redeploy takes the audit trail with -it, and the audit trail is the product. Platforms that offer no persistent volume are the ones to +Give it a volume at `/var/lib/postgresql` — the parent, not the data directory itself. Without one, +a redeploy takes the audit trail with it, and the audit trail is the product. + +**Mount the parent, not `/var/lib/postgresql/data`.** On any platform whose volume is an ext4 mount, +and that is most of them, the mount arrives holding a `lost+found` directory. `initdb` will not +initialise into a directory that has anything in it, so mounting it directly on the data directory +leaves the cluster uncreated — and because `api` waits on `postgres` and `migrate`, the container +starts, the platform reports the deploy a success, and the URL serves a 502. Mounting the parent +leaves `data` as an ordinary subdirectory, which is what PostgreSQL asks for. A Docker named volume +works either way; a platform volume does not. Platforms that offer no persistent volume are the ones to point at a managed database instead: set `DATABASE_URL` and leave `EMBEDDED_POSTGRES` off. The `vector` extension must be enabled there; RDS, Cloud SQL and Azure Database all support it, none enable it for you.