Thread reads over References and lookup by Message-ID, scoped by a ref (CL-7447)

ARCHITECTURE.md

@@ -110,6 +110,7 @@ mounting more than one passes the same function to each.
 | --- | --- |
 | `mount.ts` | HTTP surface. Parsing, validation, status codes, SSE. No SQL. |
 | `read.ts` | List (cached columns, no `raw`) and detail (frame-decoded) projection, keyset paging, snippets on detail. |
+| `thread.ts` | The conversation under one entity ref: keyset-paged oldest-first, parents resolved by RFC 5256 References linking, plus the msg-id lookup. |
 | `mutations.ts` | Read/unread, archive, trash, restore, bulk, enrich, assign. |
 | `write.ts` | `writeMailboxMessage` / `deliverInboxItems` — the host-facing write API. |
 | `persist.ts` | The transport dual-write wrapper and the `authorizeSender` seam. |
@@ -354,7 +355,61 @@ stays on `principal_mail`, matching the list's `ORDER BY` and its row-value
 cursor seek exactly, so the default (highest-traffic) page remains a
 single-table index scan that stops at `limit + 1` rows. The triage indexes and
 the three partial view indexes (`unread`, `archived_at`, `trashed_at`) live on
-`mailbox`.
+`mailbox`. The thread read adds three more on `principal_mail`:
+`(tenant_id, principal_id, message_id)` — not unique, since a msg-id is the
+*sender's* identifier and nothing stops two delivered frames carrying the same
+one — a GIN index on `refs`, the only kind that can serve the `refs @> …`
+containment filter the ref scope is expressed as, and
+`(tenant_id, principal_id, created_at, id)` matching `readMailboxThread`'s own
+oldest-first `ORDER BY` verbatim. That last one covers the same three leading
+columns the list path's own keyset index does, in the opposite direction; a
+backward scan of the list's index already serves the thread query, but a
+dedicated index removes the dependence on the planner choosing to scan the
+other one in reverse. Whichever index a page's plan uses, a ref whose messages
+cluster at one end of the principal's own `created_at` history while a page
+seeks from the other end still costs a `Filter` proportional to how much
+*unrelated* history sits between them — no index shape fixes that; only
+clustering by ref would, and this package deliberately holds no opinion on
+physical row order. See "What the split costs, measured" below.
+
+### Thread reads
+
+`readMailboxThread(db, scope, { ref, cursor?, limit? })` answers the
+conversation under one entity ref: oldest first, keyset-paged on
+`(created_at, id)`, scoped to `(tenant_id, principal_id)` and filtered by
+jsonb containment on `refs`.
+
+**Parents are resolved by RFC 5256 References linking, never by subject.** For
+each message the candidate ancestors are its `In-Reply-To` followed by its
+`References` chain walked newest-to-oldest, and the first candidate present in
+*this* mailbox under *this* ref wins. An ancestor that is not present yields
+`parentId: null` — a message whose parent lives in another principal's mailbox,
+or under a different ref, is a root of what this reader can see, and inventing
+a node for it would be a lie about the conversation.
+
+**`parentId` chains are acyclic.** RFC 5256 step 1.B calls out that nothing
+stops a delivered frame's `In-Reply-To`/`References` from naming a msg-id that,
+directly or through further ancestors, points back at the frame itself. Before
+a page is projected, the candidate-parent graph is walked (breadth-first,
+beyond the page itself when a chain reaches further) and every edge that would
+close a loop is cut: the LATER-created message in the cycle (ties broken by
+id) becomes a root instead, deterministically — the cut depends only on the
+cycle's own membership, never on which page or cursor triggered the read.
+
+The ancestor lookup spans the whole ref-scoped set rather than the current
+page, so a chain crossing a page boundary cannot report a parent on one page
+and `null` on another. It costs one query per hop of the ancestry graph — a
+msg-id map over the ids referenced so far, served by
+`principal_mail_tenant_id_principal_id_message_id_idx` — capped defensively at
+`MAX_THREAD_ANCESTRY_NODES` so a pathological reference graph degrades a
+resolved parent to `null` rather than reading an unbounded number of rows.
+
+The whole module runs on the list path and never selects `raw`. That is what
+the cached `message_id`, `in_reply_to` and `references` columns exist for: a
+thread is read on every conversation open, and decoding one MIME frame per row
+would make the cache pointless. `readMailboxMessageByMessageId(db, scope,
+messageId)` is the same posture — a scoped lookup on the list projection,
+oldest match winning, `null` when this mailbox holds no such message.
 
 ### What the split costs, measured
 
@@ -412,6 +467,16 @@ every boot of every replica.
   host's columns through our codec. The expectation is derived from the drizzle
   table objects, so it cannot drift; a rejected boot rolls the ledger row back
   with it.
+- **A migration can also run that same check early**, via
+  `Migration.assertColumnsBeforeStatement`. `0003_mail_references` sets it: a
+  host whose `principal_mail` predates this package leaves `refs` missing (it
+  is only ever declared inline in `0001`'s `CREATE TABLE`, which no-ops against
+  a pre-existing table), and without the early check the first statement to
+  notice would be `0003`'s `CREATE INDEX ... USING gin ("refs")` — a raw
+  Postgres "column \"refs\" does not exist" instead of the named
+  `SchemaTypeMismatchError` diagnostic. The knob only changes when the runner
+  calls the check, never `Migration.statements`, so it cannot change
+  `migrationChecksum`.
 
 **Everything lands in the `mailbox` schema, fully qualified.** Nothing resolves
 through `search_path`, so the host's own setting cannot redirect or shadow

CHANGELOG.md

@@ -41,6 +41,53 @@ always called out under their own heading.
   thread reader can fetch a principal's own sent copies (`"outbound"`) or
   both directions together (`"all"`) alongside the existing inbox-only
   default.
+- **Thread reads over `References`, and lookup by `Message-ID`.**
+  `readMailboxThread(db, scope, { ref, cursor?, limit? })` returns the
+  principal's messages carrying a `refs` entry equal to `ref`, oldest
+  first, keyset-paged on `(created_at, id)`. Each message projects `id`,
+  `messageId`, `inReplyTo`, `references`, `fromAddress`, `subject`,
+  `createdAt`, its read/archived state, and `parentId` — resolved by
+  **RFC 5256 References linking** (`In-Reply-To` first, then the
+  `References` chain newest-to-oldest) across the whole ref-scoped set,
+  never by subject grouping. **`parentId` chains are acyclic** — RFC 5256
+  step 1.B calls out that a delivered frame's `In-Reply-To`/`References`
+  can name a msg-id that, directly or through further ancestors, points
+  back at the frame itself; a cycle among the resolved candidate parents
+  is detected and cut (the LATER-created message in the cycle, ties broken
+  by id, becomes a root) before a page is projected, so a client's
+  ancestry walk always terminates. A parent that is not in this mailbox
+  under this ref yields `parentId: null` rather than a fabricated node,
+  and the ancestor lookup spans the whole ref-scoped set rather than the
+  current page, so a chain crossing a page boundary keeps a stable
+  `parentId`. Cursors are bound to the ref that minted them; paging one
+  into another ref is a `RangeError`, as is a malformed cursor or a limit
+  outside `1..200`. `readMailboxMessageByMessageId(db, scope, messageId)`
+  looks one message up by its `Message-ID`, scoped to
+  `(tenantId, principalId)`, oldest match winning since nothing makes a
+  msg-id unique.
+  `principal_mail` gains a cached `references` column, populated on the
+  write and transport-persist paths; migration `0003_mail_references`
+  adds it, backfills it from each existing row's `raw` — unfolding the
+  `References:` continuation lines RFC 2822 line limits force, with the
+  same `bytea`-level header slicing and NUL stripping `0002` uses — and
+  creates the three indexes the reads need:
+  `(tenant_id, principal_id, message_id)`, a GIN index on `refs`, and
+  `(tenant_id, principal_id, created_at, id)` matching `readMailboxThread`'s
+  own oldest-first `ORDER BY` (the list path's index over the same three
+  columns, in the opposite direction, already served this query via a
+  backward scan; this one lets the thread path's plan match its `ORDER BY`
+  directly). Both surfaces run on the list projection and never load `raw`.
+  A host whose own `principal_mail` predates this package (and so is
+  missing `refs` — see `ARCHITECTURE.md`'s "Migrations" section) now fails
+  `0003` with the named `SchemaTypeMismatchError` diagnostic rather than a
+  raw Postgres "column \"refs\" does not exist" at the `CREATE INDEX`
+  statement that needs it.
+  Exported alongside them: `MailboxThreadMessageSchema`,
+  `MailboxThreadResponseSchema`, `canonicalMailboxThreadRef`,
+  `encodeMailboxThreadCursor`, `decodeMailboxThreadCursor`,
+  `DEFAULT_MAILBOX_THREAD_LIMIT`, `MAX_MAILBOX_THREAD_LIMIT`, and the
+  `MailboxThreadScope` / `MailboxThreadArgs` / `MailboxThreadMessage` /
+  `MailboxThreadPage` / `MailboxThreadCursor` types.
 
 - **Threading headers on the frame and in the list projection.**
   `buildMailFrame` accepts `references` — the thread's ancestry, oldest

README.md

@@ -70,6 +70,39 @@ publish bus events only after commit, one per row actually written. See
 including `writeMailboxMessage`'s caller-supplied `messageId`, `direction`,
 and default `messageKey`.
 
+## Thread reads
+
+```ts
+import {
+  readMailboxThread,
+  readMailboxMessageByMessageId,
+} from "@corbits/mailbox";
+
+// The conversation under one entity ref, oldest first, keyset-paged.
+const page = await readMailboxThread(
+  db,
+  { tenantId, principalId },
+  { ref: { kind: "workbench", id: "wb-1" }, limit: 50 },
+);
+// page.items: { id, messageId, inReplyTo?, references, fromAddress, subject?,
+//               createdAt, read, archived, parentId }
+// page.nextCursor: pass back as `cursor` for the next page.
+
+// One message by its Message-ID, scoped to this mailbox.
+const message = await readMailboxMessageByMessageId(
+  db,
+  { tenantId, principalId },
+  "<child@acme.example>",
+);
+```
+
+`parentId` is resolved by RFC 5256 References linking — `In-Reply-To` first,
+then the `References` chain newest-to-oldest — across the whole ref-scoped set,
+not just the current page. It is `null`, never fabricated, when the nearest
+ancestor is not in this mailbox under this ref. Subjects are never used to
+group. A cursor is bound to the ref that minted it; paging it into a different
+ref is a `RangeError`, as is a malformed cursor or an out-of-range limit.
+
 ## Working on it
 
 ```sh

src/index.ts

@@ -123,6 +123,27 @@ export type {
   MailboxPage,
 } from "./read.js";
 
+// Thread reads: the conversation under one entity ref, parents resolved by
+// RFC 5256 References linking (never by subject), and the msg-id lookup.
+export {
+  readMailboxThread,
+  readMailboxMessageByMessageId,
+  canonicalMailboxThreadRef,
+  encodeMailboxThreadCursor,
+  decodeMailboxThreadCursor,
+  MailboxThreadMessageSchema,
+  MailboxThreadResponseSchema,
+  DEFAULT_MAILBOX_THREAD_LIMIT,
+  MAX_MAILBOX_THREAD_LIMIT,
+} from "./thread.js";
+export type {
+  MailboxThreadScope,
+  MailboxThreadArgs,
+  MailboxThreadMessage,
+  MailboxThreadPage,
+  MailboxThreadCursor,
+} from "./thread.js";
+
 export {
   markMailboxMessageRead,
   markMailboxMessageUnread,

src/migrations.test.ts

@@ -84,6 +84,7 @@ describe("runMailboxMigrations", () => {
         "message_key",
         "principal_id",
         "raw",
+        "references",
         "refs",
         "subject",
         "tenant_id",
@@ -112,12 +113,17 @@ describe("runMailboxMigrations", () => {
             WHERE schemaname = 'mailbox' AND tablename = 'principal_mail'
             ORDER BY indexname`,
       );
-      // The mail plane keeps exactly two access paths: the dedupe constraint
-      // and the keyset the default page seeks on.
+      // The mail plane keeps exactly five access paths: the dedupe constraint,
+      // the keyset the default page seeks on, and the three the thread read
+      // adds — the msg-id lookup, the GIN index serving the `refs` containment
+      // filter, and the thread's own oldest-first keyset.
       // `schema-ddl-parity.test.ts` holds schema.ts to this same list.
       expect(mailIndexes.map((i) => i.indexname)).toEqual([
         "principal_mail_pkey",
+        "principal_mail_refs_idx",
+        "principal_mail_tenant_id_principal_id_created_at_id_asc_idx",
         "principal_mail_tenant_id_principal_id_created_at_id_idx",
+        "principal_mail_tenant_id_principal_id_message_id_idx",
         "principal_mail_tenant_id_principal_id_message_key_idx",
       ]);
 
@@ -385,6 +391,7 @@ describe("runMailboxMigrations", () => {
       expect(ledger.map((r) => r.id)).toEqual([
         "0001_principal_mailbox",
         "0002_mail_threading_headers",
+        "0003_mail_references",
       ]);
 
       const rows = await db.execute<{
@@ -401,6 +408,77 @@ describe("runMailboxMigrations", () => {
     });
   });
 
+  test("0003 backfills the References chain, unfolding continuation lines", async () => {
+    // `References` is the header that FOLDS: RFC 2822 caps a line at 78
+    // characters, so a real chain of more than a couple of ids arrives split
+    // across continuation lines. A backfill anchored to one line would cache
+    // only the first fragment, and every older message would then link to the
+    // wrong ancestor — worse than linking to none.
+    await fromEmpty(async ({ db }) => {
+      await runMailboxMigrations(db);
+      await db.execute(
+        sql`ALTER TABLE "mailbox"."principal_mail" DROP COLUMN "references"`,
+      );
+      await db.execute(
+        sql`DELETE FROM "mailbox"."corbits_mailbox_migrations"
+            WHERE "id" = '0003_mail_references'`,
+      );
+      await seedScope(db, "acme", "user-1");
+
+      const enc = new TextEncoder();
+      const folded = enc.encode(
+        "From: bot@acme.example\r\n" +
+          "Message-ID: <child@acme.example>\r\n" +
+          "References: <root@acme.example>\r\n" +
+          "\t<middle@acme.example>\r\n" +
+          " <parent@acme.example>\r\n" +
+          "\r\nBody\r\n",
+      );
+      const none = enc.encode(
+        "From: bot@acme.example\r\nSubject: no chain\r\n\r\nBody\r\n",
+      );
+      // The body says `References:` at the start of a line; the header slice
+      // must not reach it, and a NUL after it must not abort the UPDATE.
+      const decoy = Uint8Array.from([
+        ...enc.encode(
+          "From: bot@acme.example\r\nMessage-ID: <decoy@acme.example>\r\n" +
+            "\r\nReferences: <fake@acme.example>\r\n",
+        ),
+        0x00,
+        0x41,
+      ]);
+      for (const [key, raw] of [
+        ["refs-folded", folded],
+        ["refs-none", none],
+        ["refs-decoy", decoy],
+      ] as const) {
+        await db.execute(sql`
+          INSERT INTO "mailbox"."principal_mail"
+            ("tenant_id","principal_id","address","direction","raw","message_key")
+          VALUES ('acme','user-1','user-1@acme.example','inbound',
+                  ${Buffer.from(raw)}, ${key})
+        `);
+      }
+
+      await runMailboxMigrations(db);
+
+      const rows = await db.execute<{
+        message_key: string;
+        references: string[] | null;
+      }>(sql`SELECT "message_key", "references"
+             FROM "mailbox"."principal_mail" ORDER BY "message_key"`);
+      expect(rows.map((r) => [r.message_key, r.references])).toEqual([
+        ["refs-decoy", null],
+        ["refs-folded", [
+          "<root@acme.example>",
+          "<middle@acme.example>",
+          "<parent@acme.example>",
+        ]],
+        ["refs-none", null],
+      ]);
+    });
+  });
+
   test("0002 backfill agrees with the runtime decoder on non-bracketed and multi-id In-Reply-To", async () => {
     // Characterization of the shared rule (see persist.ts): the FIRST
     // bracketed msg-id if present, else NULL. `parseMsgIdList` is what the
@@ -476,6 +554,7 @@ describe("runMailboxMigrations", () => {
       expect(rows.map((r) => [r.id, r.count])).toEqual([
         ["0001_principal_mailbox", "1"],
         ["0002_mail_threading_headers", "1"],
+        ["0003_mail_references", "1"],
       ]);
     });
   });
@@ -682,6 +761,7 @@ describe("runMailboxMigrations under concurrent cold start", () => {
     expect(ledger.map((r) => r.id)).toEqual([
       "0001_principal_mailbox",
       "0002_mail_threading_headers",
+      "0003_mail_references",
     ]);
   });