Thread reads over References and lookup by Message-ID, scoped by a ref (CL-7447) - #14
Merged
Conversation
Pins the two properties a thread read has to get right: ancestry is never fabricated, and it never changes when the reader turns the page. Also covers the cached `references` column, the 0003 backfill's unfolding of folded `References:` continuation lines, and the msg-id lookup's scope.
`readMailboxThread` returns the conversation under one entity ref, oldest first and keyset-paged, with `parentId` resolved by RFC 5256 References linking across the whole ref-scoped set — never by subject, and never fabricated when the ancestor is absent. `readMailboxMessageByMessageId` looks one message up by its sender-minted id, scoped to the mailbox. Both run on the list projection and never load `raw`, which is what the cached threading columns exist for. Migration 0003 adds the third of them, `references`, backfilling it from each legacy row's frame on the same NUL-safe terms 0002 uses plus the unfolding a folded header needs, and creates the two access paths the reads need: (tenant_id, principal_id, message_id) and a GIN index serving the refs containment filter.
…447) Folds critique coverage (self-reference, mutual cycle, out-of-order ancestor, duplicate Message-ID tie-break, cross-tenant scoping, exact ref match, malformed references) into src/thread.test.ts, and adds an EXPLAIN test pinning that a large, time-clustered ref still pages via an Index Scan with a Limit rather than a full sort. Updates the migration-ledger index-name assertions for the new keyset index ahead of the implementation that adds it.
…r (CL-7447) Three fixes surfaced by critique: - readMailboxThread resolved parentId per row from an ancestor map, but never checked whether the resolved chain looped back on itself (RFC 5256 step 1.B). A self-reference was already excluded, but a mutual or longer cycle among delivered frames' In-Reply-To/References would resolve every member to a non-null parent forever. Ancestors are now fetched breadth-first (beyond the page itself, when a chain reaches further) into a small graph, and every edge that would close a loop is cut before a page is projected: the later-created message in the cycle (ties broken by id) becomes a root. A defensive node cap bounds the walk against a pathological reference graph. - Added `principal_mail_tenant_id_principal_id_created_at_id_asc_idx` in the (still unshipped) 0003_mail_references migration, matching readMailboxThread's own oldest-first ORDER BY verbatim rather than depending on the planner reversing the list path's DESC index. schema.ts mirrors it for schema-ddl-parity. - Migration.assertColumnsBeforeStatement lets a migration call assertExpectedColumnTypes mid-run; 0003 uses it ahead of its GIN index so a host whose principal_mail predates this package (missing `refs`) fails with the named SchemaTypeMismatchError instead of a raw Postgres "column \"refs\" does not exist". Also replaces thread.ts's per-row references-column warn with the same once-per-read dropped/report pattern read.ts uses for `refs`.
TheGreatAxios
force-pushed
the
cl-7447-thread-read
branch
from
September 5, 2026 01:52
206324d to
0e1bbaa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
readMailboxThread(db, scope, { ref, cursor?, limit? })returns theprincipal's messages carrying a
refsentry equal toref, oldest first andkeyset-paged on
(created_at, id). Each message projectsid,messageId,inReplyTo,references,fromAddress,subject,createdAt, itsread/archived state, and
parentId.Parents come from RFC 5256 References linking, never from subjects.
In-Reply-Tofirst, then theReferenceschain newest-to-oldest; the firstcandidate present in this mailbox under this ref wins. An absent ancestor
yields
parentId: nullrather than a fabricated node. The ancestor lookupspans the whole ref-scoped set rather than the current page, so a chain
crossing a page boundary cannot report a parent on one page and
nullon thenext — it costs one extra query per read.
readMailboxMessageByMessageId(db, scope, messageId)looks one message up byits
Message-ID, scoped to(tenantId, principalId), oldest match winningsince a msg-id is the sender's identifier and nothing makes it unique.
Both run on the list projection and never load
raw— which is what thecached threading columns are for.
Migration
0003_mail_referencesAdds the cached
referencescolumn (jsonb, oldest first,NULLrather than[]for no chain), backfills it from each existing row's frame, and createsthe two indexes below.
The backfill uses the same
bytea-level header slicing and NUL stripping0002uses — so a body line beginningReferences:is never mistaken for aheader, and a legacy frame with a NUL byte anywhere cannot abort the UPDATE
(and with it every boot, forever). One thing
0002did not need:Referencesfolds, so the header section is unfolded before matching; a regex anchored to
one line would have cached only the first fragment and linked older messages
to the wrong ancestor.
EXPLAIN
EXPLAIN (ANALYZE, BUFFERS)on 200 000 inbound rows for one principal spreadacross 500 refs (400 rows in the queried ref):
LIMIT 51)principal_mail_refs_idx→ top-N heapsort, 4.1 ms, 683 buffersmessage_id IN (…), ref-scoped)principal_mail_tenant_id_principal_id_message_id_idx, 0.07 ms, 15 buffersreadMailboxMessageByMessageIdprincipal_mail_tenant_id_principal_id_message_id_idx, 0.02 ms, 4 buffersBoth new indexes are chosen by the planner; neither read degrades to a
sequential scan.
Tests
src/thread.test.ts(13): two replies with differentIn-Reply-Toresolve todifferent parents;
Referencesfallback picks the newest present ancestor; aparent outside the principal's mailbox is
null; a parent outside the ref isnull; the refs filter excludes other workbenches; a chain spanning a pageboundary keeps
parentIdidentical to the unpaged read; the projectioncarries the threading headers and state; cursor minted for another ref,
malformed cursor and out-of-range limit are all
RangeError; msg-id lookuprespects scope. Plus
0003 backfills the References chain, unfolding continuation linesinsrc/migrations.test.ts, and the DDL-parity check nowcanonicalizes the index access method so a GIN index is compared as one.
Full suite 372 pass / 0 fail, typecheck clean, build clean, reference-host
acceptance 20 pass.
Fixes CL-7447
Part of CL-7445