Skip to content

feat(durable): add Postgres storage backend - #2

Closed
rgarcia wants to merge 7 commits into
mainfrom
hypeship/durable-postgres-storage
Closed

rgarcia wants to merge 7 commits into
mainfrom
hypeship/durable-postgres-storage

Conversation

@rgarcia

@rgarcia rgarcia commented Oct 2, 2026

Copy link
Copy Markdown

Summary

Adds PostgresStorage to @earendil-works/pi-durable/storage/postgres, a fourth storage backend beside memory, SQLite, and JSONL.

import pg from "pg";
import { PostgresStorage } from "@earendil-works/pi-durable/storage/postgres";

const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
const harness = await Harness.open(await PostgresStorage.open(pool, { session: "support-bot" }), { models, registry }, context);

Design

  • No runtime dependency. The storage takes a caller-owned pool through a small structural interface (query, connect, and release/on/off on a client). A pg.Pool satisfies it as is; pg and @types/pg are dev dependencies for tests only.
  • Port of SqliteStorage. Same tables, same commit logic, same error messages, Postgres SQL. Records are json (not jsonb) so the stored text round-trips exactly. Indexed strings stay JSON-encoded, as in SQLite, because drivers send UTF-8 and would replace lone surrogates.
  • Many Sessions per schema. Every table is keyed by session_id first. The schema defaults to pi_durable, can be overridden with schema, and is created and migrated on first open under a transaction-scoped advisory lock.
  • Single owner, enforced. SQLite and JSONL leave this to the caller; a shared database needs it:
    • open() takes a session-level advisory lock on a client it keeps checked out until close(), and rejects if another process holds the Session.
    • open() also writes a fresh owner token. Every commit reads the Session row FOR UPDATE and rejects if the token changed, so a process whose lock connection silently died cannot write after another process takes over.
    • An error on the lock client marks the storage lost; later operations reject.
  • document() reads in a REPEATABLE READ READ ONLY transaction so the record and its revisions come from one snapshot. close() waits for admitted operations before releasing the lock. The pool needs at least two connections.

Testing

  • test/postgres-storage.test.ts runs the shared storage conformance suite twice (fresh storage, and reopen after every commit) plus ownership and migration tests: Session isolation, second-open rejection, lock-connection loss via pg_terminate_backend, commit fencing after takeover, ID and sequence persistence, concurrent first opens of a fresh schema, and newer-schema rejection. These are skipped unless PI_DURABLE_POSTGRES_URL is set, so CI without Postgres is unaffected.
  • Locally against Postgres 17: all 909 durable tests pass with PI_DURABLE_POSTGRES_URL set (908 passed, 1 pre-existing skip).
  • Ran examples 13-recovery and 31-reload-and-restart with their SQLite storage swapped for PostgresStorage; output matched SQLite exactly (task resumes from its checkpoint after reopen; reload and restart behave the same).
  • npm run build, root tsc --noEmit, biome, and the pre-commit checks pass. The root tsc run also confirms pg.Pool is assignable to the pool interface.
  • Not done: no benchmarks against SQLite yet.

rgarcia and others added 7 commits October 2, 2026 00:41
PostgresStorage stores Sessions in a Postgres schema over a caller-owned pool such as pg.Pool, without adding a runtime dependency. Each Session holds an advisory lock for its lifetime, and every commit checks the owner token written by the latest open.
Each record used to cost three round trips (id check, id claim, insert) and
each document action read its record twice. A commit now checks and claims
ids as sets, inserts each table's rows in one multi-row statement, reads the
documents it touches once, and clears and adds revisions together, all as
data-modifying CTEs of the statement that advances the session. Conversation
records are cached per storage, which is the Session's only writer.

Admitting one input goes from 22 queries to 6, creating a conversation from
37 to 13, and a plain model turn from 61 to 28. PostgresStorage.open takes
migrate: false for deployments that migrate in an explicit step.
One statement reads the session row, the batch's claimed ids, and its
documents as JSON; one data-modifying CTE statement writes the batch. Every
CTE is gated on the session still carrying this owner at the sequence the
read observed, and the final UPDATE advances it, so a takeover by another
process writes nothing and is reported as before. A single statement is
atomic, so no transaction brackets the commit.
The storage is the Session's only writer, so a record it has read stays
valid until it writes that record. Tasks and submissions by id and request,
each conversation's newest entries and head markers, current document scans
and addresses, and the materialized current content of documents are cached
and kept current by every commit. Cached values are JSON-encoded so callers
get detached objects, as from the database. The delta version check joins
the commit's pre-read. A plain turn is now 9 round trips, from 61.
…hema function

A plpgsql function created by schema version 2 locks the session row,
checks ownership at the expected sequence, validates the batch against the
stored state, writes every table, and advances the session. Each rule
raises the adapter's message with its own SQLSTATE, which the adapter maps
back to its errors. Records and content travel inside the batch as JSON
text and are cast on insert, because Postgres rejects lone-surrogate
escapes in any function that decodes string values. Historical document
reads are one statement, and a request key with no submission is cached.
…rom memory (#4)

feat(durable): Postgres storage at one round trip per commit, reads from memory
@rgarcia

rgarcia commented Nov 3, 2026

Copy link
Copy Markdown
Author

Superseded by #5, which carries the Postgres storage adapter rebuilt on the grouped-commit contract (and includes #4, which merged into this branch). Retargeted #5 to main.

@rgarcia rgarcia closed this Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant