Skip to content

Thread reads over References and lookup by Message-ID, scoped by a ref (CL-7447) - #14

Merged
TheGreatAxios merged 6 commits into
mainfrom
cl-7447-thread-read
Sep 5, 2026
Merged

Thread reads over References and lookup by Message-ID, scoped by a ref (CL-7447)#14
TheGreatAxios merged 6 commits into
mainfrom
cl-7447-thread-read

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

readMailboxThread(db, scope, { ref, cursor?, limit? }) returns the
principal's messages carrying a refs entry equal to ref, oldest first and
keyset-paged on (created_at, id). Each message projects id, messageId,
inReplyTo, references, fromAddress, subject, createdAt, its
read/archived state, and parentId.

Parents come from RFC 5256 References linking, never from subjects.
In-Reply-To first, then the References chain newest-to-oldest; the first
candidate present in this mailbox under this ref wins. An absent ancestor
yields parentId: null rather than a fabricated node. The ancestor lookup
spans the whole ref-scoped set rather than the current page, so a chain
crossing a page boundary cannot report a parent on one page and null on the
next — it costs one extra query per read.

readMailboxMessageByMessageId(db, scope, messageId) looks one message up by
its Message-ID, scoped to (tenantId, principalId), oldest match winning
since a msg-id is the sender's identifier and nothing makes it unique.

Both run on the list projection and never load raw — which is what the
cached threading columns are for.

Migration 0003_mail_references

Adds the cached references column (jsonb, oldest first, NULL rather than
[] for no chain), backfills it from each existing row's frame, and creates
the two indexes below.

The backfill uses the same bytea-level header slicing and NUL stripping
0002 uses — so a body line beginning References: is never mistaken for a
header, and a legacy frame with a NUL byte anywhere cannot abort the UPDATE
(and with it every boot, forever). One thing 0002 did not need: References
folds, so the header section is unfolded before matching; a regex anchored to
one line would have cached only the first fragment and linked older messages
to the wrong ancestor.

EXPLAIN

EXPLAIN (ANALYZE, BUFFERS) on 200 000 inbound rows for one principal spread
across 500 refs (400 rows in the queried ref):

Query Plan
thread page (ref filter + keyset, LIMIT 51) Bitmap Index Scan on principal_mail_refs_idx → top-N heapsort, 4.1 ms, 683 buffers
ancestor map (message_id IN (…), ref-scoped) Index Scan using principal_mail_tenant_id_principal_id_message_id_idx, 0.07 ms, 15 buffers
readMailboxMessageByMessageId Index Scan using principal_mail_tenant_id_principal_id_message_id_idx, 0.02 ms, 4 buffers

Both new indexes are chosen by the planner; neither read degrades to a
sequential scan.

Tests

src/thread.test.ts (13): two replies with different In-Reply-To resolve to
different parents; References fallback picks the newest present ancestor; a
parent outside the principal's mailbox is null; a parent outside the ref is
null; the refs filter excludes other workbenches; a chain spanning a page
boundary keeps parentId identical to the unpaged read; the projection
carries the threading headers and state; cursor minted for another ref,
malformed cursor and out-of-range limit are all RangeError; msg-id lookup
respects scope. Plus 0003 backfills the References chain, unfolding continuation lines in src/migrations.test.ts, and the DDL-parity check now
canonicalizes the index access method so a GIN index is compared as one.

Full suite 372 pass / 0 fail, typecheck clean, build clean, reference-host
acceptance 20 pass.

Fixes CL-7447
Part of CL-7445

Pins the two properties a thread read has to get right: ancestry is never
fabricated, and it never changes when the reader turns the page. Also covers
the cached `references` column, the 0003 backfill's unfolding of folded
`References:` continuation lines, and the msg-id lookup's scope.
`readMailboxThread` returns the conversation under one entity ref, oldest
first and keyset-paged, with `parentId` resolved by RFC 5256 References
linking across the whole ref-scoped set — never by subject, and never
fabricated when the ancestor is absent. `readMailboxMessageByMessageId`
looks one message up by its sender-minted id, scoped to the mailbox.

Both run on the list projection and never load `raw`, which is what the
cached threading columns exist for. Migration 0003 adds the third of them,
`references`, backfilling it from each legacy row's frame on the same
NUL-safe terms 0002 uses plus the unfolding a folded header needs, and
creates the two access paths the reads need: (tenant_id, principal_id,
message_id) and a GIN index serving the refs containment filter.
…447)

Folds critique coverage (self-reference, mutual cycle, out-of-order
ancestor, duplicate Message-ID tie-break, cross-tenant scoping, exact
ref match, malformed references) into src/thread.test.ts, and adds an
EXPLAIN test pinning that a large, time-clustered ref still pages via
an Index Scan with a Limit rather than a full sort. Updates the
migration-ledger index-name assertions for the new keyset index ahead
of the implementation that adds it.
…r (CL-7447)

Three fixes surfaced by critique:

- readMailboxThread resolved parentId per row from an ancestor map, but
  never checked whether the resolved chain looped back on itself (RFC
  5256 step 1.B). A self-reference was already excluded, but a mutual
  or longer cycle among delivered frames' In-Reply-To/References would
  resolve every member to a non-null parent forever. Ancestors are now
  fetched breadth-first (beyond the page itself, when a chain reaches
  further) into a small graph, and every edge that would close a loop
  is cut before a page is projected: the later-created message in the
  cycle (ties broken by id) becomes a root. A defensive node cap bounds
  the walk against a pathological reference graph.

- Added `principal_mail_tenant_id_principal_id_created_at_id_asc_idx`
  in the (still unshipped) 0003_mail_references migration, matching
  readMailboxThread's own oldest-first ORDER BY verbatim rather than
  depending on the planner reversing the list path's DESC index.
  schema.ts mirrors it for schema-ddl-parity.

- Migration.assertColumnsBeforeStatement lets a migration call
  assertExpectedColumnTypes mid-run; 0003 uses it ahead of its GIN
  index so a host whose principal_mail predates this package (missing
  `refs`) fails with the named SchemaTypeMismatchError instead of a
  raw Postgres "column \"refs\" does not exist".

Also replaces thread.ts's per-row references-column warn with the
same once-per-read dropped/report pattern read.ts uses for `refs`.
@TheGreatAxios
TheGreatAxios merged commit 118a2cd into main Sep 5, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant