From 18608e50a8c77288f6c235f006de87193b9bea52 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 21 Oct 2026 01:32:54 +0000 Subject: [PATCH 1/9] fix(sqlite): fix a statement's size by its shape, not its query MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Durable Object SQLite caps a statement at 100 bound parameters and 100 KB of text. The shared declaration promised contentByType and contentFieldByType at 64 keys, but each arm and entry bound its own parameters and text, so a query within the cap failed on DO; so did a plain filter.content with a few deep paths, which no cap bounds. Rather than lower the caps for one engine, every caller-sized part of a query is now data: contentFieldByType's entries bind as one list each, and all content conditions (filter.content, contentPresent, every contentByType arm) bind as one table of scoped rows whose paths a recursive query walks. The spec states the rule once, in adapters.md § Statement size, and the 32-segment path cap now rests on the content nesting cap rather than SQLite's join limit. Closes #462. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Mf3p8BVtrQJLZx1JFCHchC --- .changeset/fixed-statement-size.md | 6 + .changeset/scoped-query-reach.md | 2 +- docs/spec/adapters.md | 13 +- docs/spec/data-model.md | 2 +- packages/core/src/query-validation.ts | 8 +- .../tests/record.test.ts | 77 ++++- .../tests/record.test.ts | 5 +- packages/sqlite-shared/src/query.ts | 290 ++++++++---------- packages/sqlite-shared/src/record-adapter.ts | 5 +- 9 files changed, 223 insertions(+), 185 deletions(-) create mode 100644 .changeset/fixed-statement-size.md diff --git a/.changeset/fixed-statement-size.md b/.changeset/fixed-statement-size.md new file mode 100644 index 00000000..2c614eb3 --- /dev/null +++ b/.changeset/fixed-statement-size.md @@ -0,0 +1,6 @@ +--- +'@haverstack/record-adapter-do-sqlite': minor +'@haverstack/record-adapter-sqlite': patch +--- + +A SQLite statement's text and bound parameters no longer grow with a query's size. Content conditions — every `filter.content` and `filter.contentPresent` key and every `filter.contentByType` arm — bind as one table of rows whose paths are walked by a recursive query, and `sort.contentFieldByType`'s entries bind as one list, so Durable Object SQLite answers queries at the declared `contentByType` and `contentFieldByType` caps, and content filters of any size, instead of failing with `too many SQL variables` or `statement too long`. diff --git a/.changeset/scoped-query-reach.md b/.changeset/scoped-query-reach.md index 7452ca55..fc04028f 100644 --- a/.changeset/scoped-query-reach.md +++ b/.changeset/scoped-query-reach.md @@ -7,4 +7,4 @@ '@haverstack/conformance-fixtures': patch --- -State what a scoped requester may do as a `Reach` per verb — `create`, `read`, `update`, `delete` — and answer every grant- and permission-based gate from it: a type fence for a delegated principal, and within it the types reached whole, the types reached where the subject authored, and the permission-element sets that carry the verb, a `write` beside its grantee's `read` or `anyone`. `get()`, refusals and the change feed ask the read reach of one record, write gates ask the update or delete reach, `create()` asks the create reach of the record it would make, and `query()` hands the read reach to the adapter's new `queryReadable(query, reach)`, which matches it as one more condition — so pages, cursors and whether a cursor comes back are decided among readable records alone, and a cursor never names a record the requester cannot read. A requester's grants, rosters and reaches are each read at most once per operation, and a subscription holds them until a `_grant` or `_group` event clears them. `asEntity()`/`asActor()` refuse with `ScopeUnavailableError` — replacing `RelayScopeError` — over a stack whose adapter lacks `queryReadable()` or relays changes. `MemoryAdapter`, the SQLite adapters and `LocalAdapter` implement it; `inReach()` in `@haverstack/core/adapter` is the reference predicate, and the conformance suite holds adapters to it. The SQLite adapters bind every list a query carries — a filter's type, app and author lists, its tags, and every list in a reach — as one JSON parameter, so neither a reach nor a filter list of any length runs into Durable Object SQLite's 100-parameter cap; per-Type arms stay bounded by the declared `contentByType` and `contentFieldByType` caps. `MemoryAdapter`'s cursor names the last record a page held rather than an offset, so a page resumes after it however the matched set moved, as the SQL adapters' cursors do. +State what a scoped requester may do as a `Reach` per verb — `create`, `read`, `update`, `delete` — and answer every grant- and permission-based gate from it: a type fence for a delegated principal, and within it the types reached whole, the types reached where the subject authored, and the permission-element sets that carry the verb, a `write` beside its grantee's `read` or `anyone`. `get()`, refusals and the change feed ask the read reach of one record, write gates ask the update or delete reach, `create()` asks the create reach of the record it would make, and `query()` hands the read reach to the adapter's new `queryReadable(query, reach)`, which matches it as one more condition — so pages, cursors and whether a cursor comes back are decided among readable records alone, and a cursor never names a record the requester cannot read. A requester's grants, rosters and reaches are each read at most once per operation, and a subscription holds them until a `_grant` or `_group` event clears them. `asEntity()`/`asActor()` refuse with `ScopeUnavailableError` — replacing `RelayScopeError` — over a stack whose adapter lacks `queryReadable()` or relays changes. `MemoryAdapter`, the SQLite adapters and `LocalAdapter` implement it; `inReach()` in `@haverstack/core/adapter` is the reference predicate, and the conformance suite holds adapters to it. The SQLite adapters bind every list a query carries — a filter's type, app and author lists, its tags, and every list in a reach — as one JSON parameter, so neither a reach nor a filter list of any length runs into Durable Object SQLite's 100-parameter cap. `MemoryAdapter`'s cursor names the last record a page held rather than an offset, so a page resumes after it however the matched set moved, as the SQL adapters' cursors do. diff --git a/docs/spec/adapters.md b/docs/spec/adapters.md index 81f0b34e..c790dd3b 100644 --- a/docs/spec/adapters.md +++ b/docs/spec/adapters.md @@ -69,7 +69,7 @@ type Reach = { The shape and its reading are [Access control § Reach](./access-control.md#reach)'s, over TypeIds where that section's are over families: `Stack` resolves each family to its registered versions before the call, the resolution [`filter.baseId`](./data-model.md#filter) gets, so an adapter needs no family concept. The reach is one more condition of the query, beside its own `filter`, so it narrows and never widens. Every rule behind it is resolved before the adapter sees it, so the adapter applies none of access control: it matches a description, the way it matches a `filter`. `inReach()` in `@haverstack/core/adapter` is the reference reading, and the [conformance suite](#conformance) holds an adapter to it. -**Every local adapter implements it.** It is optional only so `APIAdapter` can leave it out: a stack whose adapter lacks it [cannot be scoped at all](./access-control.md#enforcement-stackasentity), which a server-backed stack never needs. A reach is sized by the requester's grants and rosters, not by anything the adapter controls, so **a list in it never costs a bound parameter per entry** — the rule a SQLite adapter holds every list a query carries to, a `filter`'s and a reach's alike: each binds as one JSON parameter read through `json_each`, the permission sets and a tag filter's all-of list included, since Durable Object SQLite caps a statement at 100 parameters. What is not a list, the per-Type arms of `contentByType` and `contentFieldByType`, is bounded instead by the cap the adapter [declares](#adapter-capabilities) for it. It matches each permission element through an index on the element's identity ending in `record_id`, the same semi-join shape as the association filters. +**Every local adapter implements it.** It is optional only so `APIAdapter` can leave it out: a stack whose adapter lacks it [cannot be scoped at all](./access-control.md#enforcement-stackasentity), which a server-backed stack never needs. A reach is sized by the requester's grants and rosters, not by anything the adapter controls, so a SQLite adapter binds each list in it as one parameter, the permission sets included — the rule it holds every part of a query to; see [Statement size](#statement-size). It matches each permission element through an index on the element's identity ending in `record_id`, the same semi-join shape as the association filters. ## Package naming convention @@ -172,6 +172,15 @@ This is why a SQLite-backed adapter's write is unindex, write, re-index rather t A missing row is not only a stale read. `deleteUnreferencedAttachmentRecords()` decides whether a file is still referenced by looking for one, so a `file-ref` field with no row makes a referenced file look unreferenced, and the attachment's metadata Records are purged rather than refused. A writer that touches `content` without rewriting the row is the one way to reach that. +### Statement size + +**A statement's text and its bound parameters are fixed by which keys a query sets, never by how many entries they hold.** Durable Object SQLite caps a statement at 100 parameters and 100 KB of text, and a query's size is not the adapter's to choose: a reach is sized by the requester's grants and rosters, a filter by whoever wrote it. A statement that grew with either would fail on one engine for a query another answers, and a declared [cap](#adapter-capabilities) could not promise anything. Every caller-sized part of a query is therefore data, bound as one JSON parameter and read through `json_each`: + +- **A list** — a filter's type, app and author lists, its tags, a reach's type lists and permission sets, and the per-Type entries of `sort.contentFieldByType`, each one row. A single value still binds as itself, so an index on its column keeps yielding rows in order. +- **The content conditions** — every `filter.content` and `filter.contentPresent` key, and every key of every `filter.contentByType` arm, as one table of rows, each naming its path, what it asks, and the Type it is scoped to (none for `filter.content`). A record matches when every row scoped to nothing or to its own Type holds. A row's path is walked by a recursive query that matches one segment per step, so neither the number of conditions nor a path's length adds text. + +What remains binds a fixed count: a scalar filter's value, a date bound, a cursor's position, the page's row budget. Since a statement's size is set by its shape and not by its query, every engine running the shared query builder honors the same capability declaration. + ## Adapter capabilities Adapters expose a capabilities object so apps can check what's supported before relying on a feature: @@ -208,7 +217,7 @@ One type, `StackCapabilities`, names both sides: an adapter declares it as `capa **`filter.contentPresent` sits beside the ladder rather than on it.** It gates `filter.contentPresent`, and is required `true` for local adapters on the same reasoning: asking whether a path holds a value is the same walk with a weaker predicate. It is not a rung because a server promising to match a content value has not thereby promised to answer whether one is there at all. It is meaningful only above `'none'` — presence travels in the `POST /records/query` body, which a server reaching no content does not expose — and a presence path is itself gated by the rung, so a multi-segment one needs `'path'`. -**`filter.contentByType` and `sort.contentFieldByType` are caps, not flags.** Each is the most keys the [per-Type shape](./data-model.md#content-by-type) may hold, and `0` declines it. A local adapter must honor both, on the reasoning that makes `'path'` required: an arm is the content filter it already runs, scoped to one `typeId`. The first-party ones declare `64`, far above any family's version count and inside SQLite's limits on expression depth and compound statements. A server may declare any cap, `0` included, and a collection then narrows the versions it cannot reach rather than failing the query — see [Data model § Content queries across versions](./data-model.md#content-queries-across-versions). Both are meaningful only above `filter.content: 'none'`, since they travel in the `POST /records/query` body. +**`filter.contentByType` and `sort.contentFieldByType` are caps, not flags.** Each is the most keys the [per-Type shape](./data-model.md#content-by-type) may hold, and `0` declines it. A local adapter must honor both, on the reasoning that makes `'path'` required: an arm is the content filter it already runs, scoped to one `typeId`. The first-party ones declare `64`, far above any family's version count; a per-Type key costs a SQLite adapter no statement text or parameter of its own (see [Statement size](#statement-size)), so the cap is the same on every engine. A server may declare any cap, `0` included, and a collection then narrows the versions it cannot reach rather than failing the query — see [Data model § Content queries across versions](./data-model.md#content-queries-across-versions). Both are meaningful only above `filter.content: 'none'`, since they travel in the `POST /records/query` body. **`filter.search` has no local-required rule.** A full-text index is a different mechanism from field matching, and neither implies the other, so a local adapter may legitimately decline it (see the JSON adapter note below). diff --git a/docs/spec/data-model.md b/docs/spec/data-model.md index cd99f229..d67d40c3 100644 --- a/docs/spec/data-model.md +++ b/docs/spec/data-model.md @@ -622,7 +622,7 @@ The motivating query is a listing: an article's `publishedAt` is optional, so dr A path that descends through a scalar, or through a field that isn't there, reaches no value and matches nothing — it is not an error. A path may be at most **32 segments**; longer is `StackBadRequestError` (400), as is an empty segment (`a..b`) or a segment containing a reserved character. Those are structural faults in the request, refused rather than answered with an empty result, on the same reasoning as a malformed cursor. -**A segment is a value, never syntax.** An adapter resolving a path MUST carry each segment as a bound parameter matched against a key, rather than assembling it into a path expression — so no field name a write would accept can be reinterpreted as syntax, and no key can make the statement itself malformed. The 32-segment cap is what keeps that statement inside the engine's own limits: a SQLite adapter walks a segment with two `json_each` joins, and 32 segments is the longest path that fits SQLite's 64-table join limit. The cap and the generated shape move together — a longer path is refused because it could not be executed, not merely because it is unusual. +**A segment is a value, never syntax.** An adapter resolving a path MUST carry each segment as a bound value matched against a key, rather than assembling it into a path expression — so no field name a write would accept can be reinterpreted as syntax, and no key can make the statement itself malformed. The 32-segment cap is the [nesting cap](#types) read from the other side: each segment descends one level, so a longer path could reach no value a write accepts, and is refused as a fault in the request rather than answered with a match that cannot exist. **Multi-segment keys need the `'path'` rung of `filter.content`**, where a single-segment key needs only `'field'` — see [Capability-gated filters](#capability-gated-filters), and [Adapter capabilities](./adapters.md#adapter-capabilities) for why the reach is one ordered value rather than a flag per rung. diff --git a/packages/core/src/query-validation.ts b/packages/core/src/query-validation.ts index 1fe0143f..79462506 100644 --- a/packages/core/src/query-validation.ts +++ b/packages/core/src/query-validation.ts @@ -167,7 +167,7 @@ export function filtersContent(features: Pick): boo return features.filter.content !== 'none'; } -/** The longest path both SQLite engines can execute — see the spec link below. */ +/** The deepest content a write accepts — see the spec link below. */ const MAX_CONTENT_PATH_SEGMENTS = 32; /** @@ -178,9 +178,9 @@ const MAX_CONTENT_PATH_SEGMENTS = 32; */ export function parseContentFilterKey(key: string): string[] { const segments = key.split('.'); - // A SQLite adapter spends two json_each joins per segment against a - // 64-table join limit, so a path past the cap is one the engine could - // not execute. See docs/spec/data-model.md § Nested content paths. + // Each segment descends one level of content nested at most 32 deep, so + // a longer path could reach no stored value. See docs/spec/data-model.md + // § Nested content paths. if (segments.length > MAX_CONTENT_PATH_SEGMENTS) { throw new StackBadRequestError( `Invalid content filter path "${key}": at most ${MAX_CONTENT_PATH_SEGMENTS} segments.`, diff --git a/packages/record-adapter-do-sqlite/tests/record.test.ts b/packages/record-adapter-do-sqlite/tests/record.test.ts index 10945f61..541ce554 100644 --- a/packages/record-adapter-do-sqlite/tests/record.test.ts +++ b/packages/record-adapter-do-sqlite/tests/record.test.ts @@ -333,9 +333,8 @@ describe('records — queries', () => { expect(result.records.map((r) => r.id)).toEqual(['r1']); }); - // The segment cap is sized against SQLite's join limit, and this build - // is where a difference in that limit would surface. The longest legal - // path has to execute, not merely be accepted. + // This build is where a difference in SQLite's limits would surface. The + // longest legal path has to execute, not merely be accepted. test('a path at the segment cap executes; one past it is refused', async () => { const stub = getStub(); await stub.createRecord(makeRecord({ id: 'r1', content: { text: 'shallow' } })); @@ -552,7 +551,7 @@ describe('commitMigration', () => { // A reach is sized by the requester's grants and rosters, and this engine // caps a statement at 100 bound parameters, so a reach binds each of its -// lists as one. See docs/spec/adapters.md § Querying within a reach. +// lists as one. See docs/spec/adapters.md § Statement size. test('queryReadable answers a reach far past the parameter cap', async () => { const stub = getStub(); const shared = makeRecord({ @@ -598,3 +597,73 @@ test('a tag filter far past the parameter cap', async () => { const result = await stub.queryRecords({ filter: { tags: labels } }); expect(result.records.map((r) => r.id)).toEqual([tagged.id]); }); + +// The caps are the shared SQLite declaration, so they have to hold on the +// engine with the tightest statement limits, at full size. +describe('per-Type keys at the declared cap', () => { + const typeIds = (cap: number) => + Array.from({ length: cap }, (_, i) => (i === 0 ? NOTE_TYPE_V1 : `com.example/t${i}@1`)); + + test('filter.contentByType', async () => { + const stub = getStub(); + const cap = (await stub.getCapabilities()).filter.contentByType; + await stub.createRecord(makeRecord({ id: 'r1', content: { text: 'a', meta: { n: 1 } } })); + await stub.createRecord(makeRecord({ id: 'r2', content: { text: 'b', meta: { n: 2 } } })); + const contentByType = Object.fromEntries( + typeIds(cap).map((typeId) => [ + typeId, + { content: { 'meta.n': 1, 'meta.missing': null }, contentPresent: ['text'] }, + ]), + ); + const result = await stub.queryRecords({ filter: { contentByType } }); + expect(result.records.map((r) => r.id)).toEqual(['r1']); + }); + + test('sort.contentFieldByType', async () => { + const stub = getStub(); + const cap = (await stub.getCapabilities()).sort.contentFieldByType; + const other = 'com.example/t1@1'; + await stub.saveType({ + id: NOTE_TYPE_V1, + baseId: 'com.example.test/note', + version: 1, + name: 'Note', + schema: { text: { kind: 'string' }, n: { kind: 'number' } }, + schemaHash: 'note-hash', + createdAt: new Date(), + }); + await stub.createRecord(makeRecord({ id: 'r1', content: { text: 'a', n: 3 } })); + await stub.createRecord(makeRecord({ id: 'r2', content: { text: 'b', n: 1 } })); + await stub.createRecord(makeRecord({ id: 'r3', typeId: other, content: { text: 'c' } })); + await stub.createRecord(makeRecord({ id: 'r4', content: { text: 'd' } })); + const contentFieldByType = Object.fromEntries( + typeIds(cap).map((typeId, i) => [typeId, i % 2 === 0 ? { field: 'n' } : { value: 2 }]), + ); + const result = await stub.queryRecords({ + sort: { contentFieldByType, direction: 'asc' }, + limit: 2, + }); + expect(result.records.map((r) => r.id)).toEqual(['r2', 'r3']); + const rest = await stub.queryRecords({ + sort: { contentFieldByType, direction: 'asc' }, + cursor: result.cursor!, + }); + expect(rest.records.map((r) => r.id)).toEqual(['r1', 'r4']); + }); +}); + +// Nothing caps how many keys filter.content holds, so neither they nor their +// paths can cost statement text or a parameter each. See +// docs/spec/adapters.md § Statement size. +test('content conditions far past the statement limits execute', async () => { + const stub = getStub(); + await stub.createRecord(makeRecord({ id: 'r1', content: { text: 'a', n: 1 } })); + await stub.createRecord(makeRecord({ id: 'r2', content: { text: 'b', n: 2 } })); + const prefix = Array.from({ length: 31 }, (_, i) => `s${i}`).join('.'); + const content = Object.fromEntries( + Array.from({ length: 8 }, (_, i) => [`${prefix}.k${i}`, null] as [string, unknown]), + ); + content.n = 1; + const result = await stub.queryRecords({ filter: { content } }); + expect(result.records.map((r) => r.id)).toEqual(['r1']); +}); diff --git a/packages/record-adapter-sqlite/tests/record.test.ts b/packages/record-adapter-sqlite/tests/record.test.ts index 681274ac..761cbb07 100644 --- a/packages/record-adapter-sqlite/tests/record.test.ts +++ b/packages/record-adapter-sqlite/tests/record.test.ts @@ -1049,9 +1049,8 @@ describe('records — queries', () => { } }); - // The segment cap is what keeps the generated join list inside SQLite's - // 64-table limit, so the longest legal path has to be executable, not - // merely accepted. See docs/spec/data-model.md § Nested content paths. + // The longest legal path has to be executable, not merely accepted. See + // docs/spec/data-model.md § Nested content paths. test('a path at the segment cap executes; one past it is refused', async () => { const adapter = await initAdapter(); const deepest = Array.from({ length: 32 }, (_, i) => `s${i}`).join('.'); diff --git a/packages/sqlite-shared/src/query.ts b/packages/sqlite-shared/src/query.ts index 5b51663d..3812ff5f 100644 --- a/packages/sqlite-shared/src/query.ts +++ b/packages/sqlite-shared/src/query.ts @@ -75,56 +75,40 @@ const spread = (alias: string): string => `ELSE json_array(${alias}.value) END`; /** - * The members of a json_each row that is an object, and none for a row - * that is not — a path descending through a scalar reaches no value rather - * than reaching an error. + * The members of a walk row that is an object, and none for a row that is + * not — a path descending through a scalar reaches no value rather than + * reaching an error. */ const members = (alias: string): string => - `CASE WHEN ${alias}.type = 'object' THEN ${alias}.value ELSE '{}' END`; + `CASE WHEN ${alias}.t = 'object' THEN ${alias}.v ELSE '{}' END`; /** - * True when some value reached by `segments` satisfies `leaf`, applied to - * the alias holding that value. Each segment is a bound parameter matched - * against json_each's `key`, never interpolated into a path expression, so - * no key can be read as syntax. See docs/spec/data-model.md § Filter. + * The content conditions a statement tests, one row each: the Type it is + * scoped to (`null` for every record), its path segments, its op and value. + * They bind as one parameter, materialized once ahead of the statement so a + * walk reads columns rather than re-parsing the list per record. */ -const contentPathExists = ( - segments: string[], - leaf: (alias: string) => string, - params: unknown[], - nextAlias: () => string, -): string => { - // The walk is a flat join list, not a subquery per segment: both SQLite - // builds cap expression-tree depth far below the segment cap, and - // nesting spends that budget while a join list spends the join budget - // the cap is actually sized against. - let key = nextAlias(); - let value = nextAlias(); - const from = [`json_each(r.content) AS ${key}`, `json_each(${spread(key)}) AS ${value}`]; - const where = [`${key}.key = ?`]; - params.push(segments[0]); - - for (const segment of segments.slice(1)) { - key = nextAlias(); - from.push(`json_each(${members(value)}) AS ${key}`); - value = nextAlias(); - from.push(`json_each(${spread(key)}) AS ${value}`); - where.push(`${key}.key = ?`); - params.push(segment); - } - - where.push(leaf(value)); - return `EXISTS (SELECT 1 FROM ${from.join(', ')} WHERE ${where.join(' AND ')})`; -}; +const CONTENT_CONDITIONS = + "WITH cond AS MATERIALIZED (SELECT json_extract(value, '$[0]') AS scope, " + + "json_extract(value, '$[1]') AS path, json_array_length(value, '$[1]') AS depth, " + + "json_extract(value, '$[2]') AS op, json_extract(value, '$[3]') AS val FROM json_each(?)) "; /** - * A scalar filter value never matches an object or array stored at the - * path: json_each exposes those as their JSON text, which would compare - * equal to a string spelling the same document while every non-SQL adapter - * compares the value itself. + * Whether the `cond` row `c` holds for `r`, walked as data by a recursive + * CTE: each step matches one segment against json_each's `key`, so no key + * is read as syntax and the statement's text is the same for any path. + * `'absent'` holds where `'present'` does not; an `'eq'` value never + * matches an object or array, which json_each exposes as JSON text. See + * docs/spec/data-model.md § Filter. */ -const equalsLeaf = (alias: string): string => - `${alias}.type NOT IN ('object', 'array') AND ${alias}.value = ?`; +const HOLDS = + "((c.op = 'absent') != EXISTS (WITH RECURSIVE w(d, v, t) AS (" + + "SELECT 0, r.content, 'object' UNION ALL " + + `SELECT w.d + 1, e.value, e.type FROM w JOIN json_each(${members('w')}) m ` + + `JOIN json_each(${spread('m')}) e ` + + "WHERE w.d < c.depth AND m.key = json_extract(c.path, '$[' || w.d || ']')) " + + "SELECT 1 FROM w WHERE w.d = c.depth AND CASE c.op WHEN 'eq' THEN " + + "w.t NOT IN ('object', 'array') AND w.v = c.val ELSE w.v IS NOT NULL END))"; /** The join alias the content index is read through. */ const SORT_ALIAS = 'cs'; @@ -152,57 +136,26 @@ const contentCursor = (cursor: DecodedCursor, field: string) => { }; /** - * A `content` and `contentPresent` predicate as conditions on `r`, binding - * in textual order. A dot-separated path is matched element-wise through - * arrays, and a `null` value means "no value at the path, or a value that - * is null" (docs/spec/data-model.md § Filter): the second arm below, "some - * value there is null" OR "nothing is there at all". + * A `content` and `contentPresent` predicate as `cond` rows scoped to + * `scope`. A `null` value means "no value at the path, or a value that is + * null" (docs/spec/data-model.md § Filter): the path holds no non-null + * value. Presence is the question an exact-match value cannot ask. */ -const contentConditions = ( - predicate: ContentArm, - params: unknown[], - nextAlias: () => string, -): string[] => { - const conditions: string[] = []; - for (const [key, value] of Object.entries(predicate.content ?? {})) { - const segments = parseContentFilterKey(key); - // undefined is the same absence null names — JSON drops it on the - // way over the wire, so treating it as a value would make an - // in-process query and its wire equivalent disagree. - if (value === null || value === undefined) { - const isNull = contentPathExists(segments, (a) => `${a}.value IS NULL`, params, nextAlias); - const anyValue = contentPathExists( - segments, - (a) => `${a}.value IS NOT NULL`, - params, - nextAlias, - ); - conditions.push(`(${isNull} OR NOT ${anyValue})`); - } else { - const match = contentPathExists(segments, equalsLeaf, params, nextAlias); - params.push(value); - conditions.push(match); - } - } - // Presence — the question an exact-match value cannot ask: a path holds - // a value when at least one non-null value is reachable at it. - for (const key of predicate.contentPresent ?? []) { - conditions.push( - contentPathExists( - parseContentFilterKey(key), - (a) => `${a}.value IS NOT NULL`, - params, - nextAlias, - ), - ); - } - return conditions; -}; +const contentRows = (scope: string | null, predicate: ContentArm): unknown[][] => [ + ...Object.entries(predicate.content ?? {}).map(([key, value]) => + // undefined is the same absence null names — JSON drops it on the way + // over the wire, so treating it as a value would make an in-process + // query and its wire equivalent disagree. + value === null || value === undefined + ? [scope, parseContentFilterKey(key), 'absent'] + : [scope, parseContentFilterKey(key), 'eq', value], + ), + ...(predicate.contentPresent ?? []).map((key) => [scope, parseContentFilterKey(key), 'present']), +]; /** * Values as the condition matching any of them. A list binds as one JSON - * parameter, since it is caller- or reach-sized and a statement's parameter - * count is capped (at 100 on Durable Object SQLite); one value binds as + * parameter (docs/spec/adapters.md § Statement size); one value binds as * itself, so an index on the column still yields rows in its order. An * empty list matches nothing. */ @@ -255,20 +208,18 @@ const reachConditions = (reach: Reach, params: unknown[]): string[] => { return conditions; }; +/** A statement's conditions on `r`, their parameters, and its `cond` rows. */ +type Conditions = { conditions: string[]; params: unknown[]; content: unknown[][] }; + /** * Every condition on the records row itself — everything a query asks * except where in the ordering to resume, which is per-partition and gets * appended by the caller. */ -const recordConditions = ( - query: StackQuery, - reach: Reach | undefined, -): { conditions: string[]; params: unknown[] } => { +const recordConditions = (query: StackQuery, reach: Reach | undefined): Conditions => { const conditions: string[] = ["r.id != '_config'"]; const params: unknown[] = []; const f = query.filter ?? {}; - let aliasSeq = 0; - const nextContentAlias = () => `cp${aliasSeq++}`; /** * A filter that accepts one value or a list of them, as the one `IN` that @@ -407,18 +358,22 @@ const recordConditions = ( conditions.push(hasAssociation('relationship', clauses)); } - if (f.content || f.contentPresent?.length) { - conditions.push(...contentConditions(f, params, nextContentAlias)); + // An arm is the content filter scoped to one Type; types partition + // records, so a record meets at most the arm for its own type + // (docs/spec/data-model.md § Content by type). + const content = contentRows(null, f); + if (f.contentByType) { + const arms = Object.entries(f.contentByType); + const typeIds = arms.map(([typeId]) => typeId); + conditions.push(inList('r.type_id', typeIds, params)); + for (const [typeId, arm] of arms) content.push(...contentRows(typeId, arm)); } - // One arm per Type, each pinned to its type_id; types partition records, - // so no record meets two arms (docs/spec/data-model.md § Content by type). - if (f.contentByType) { - const arms = Object.entries(f.contentByType).map(([typeId, arm]) => { - params.push(typeId); - return ['r.type_id = ?', ...contentConditions(arm, params, nextContentAlias)].join(' AND '); - }); - conditions.push(`(${arms.map((arm) => `(${arm})`).join(' OR ')})`); + if (content.length > 0) { + conditions.push( + 'NOT EXISTS (SELECT 1 FROM cond c WHERE (c.scope IS NULL OR c.scope = r.type_id) ' + + `AND NOT ${HOLDS})`, + ); } if (f.search) { @@ -434,7 +389,7 @@ const recordConditions = ( } } - return { conditions, params }; + return { conditions, params, content }; }; /** @@ -466,16 +421,23 @@ const presentCursorCondition = ( ); }; +/** + * One statement reading `r` from `from`, whose own parameters bind ahead of + * the conditions', with the `cond` table first when it tests content. + */ const statement = ( - select: string, from: string, - conditions: string[], - params: unknown[], - tail = '', -): QueryStatement => ({ - sql: `SELECT ${select} FROM ${from} WHERE ${conditions.join(' AND ')}${tail}`, - params, -}); + { conditions, params, content }: Conditions, + tail: string, + fromParams: unknown[] = [], +): QueryStatement => { + const select = `SELECT r.* FROM ${from} WHERE ${conditions.join(' AND ')}${tail}`; + if (content.length === 0) return { sql: select, params: [...fromParams, ...params] }; + return { + sql: CONTENT_CONDITIONS + select, + params: [JSON.stringify(content), ...fromParams, ...params], + }; +}; /** * Compile a query into the statements that answer it, read in order and @@ -493,23 +455,15 @@ export const buildQueryPlan = (query: StackQuery, reach?: Reach): QueryStatement const contentField = query.sort?.contentField; if (contentField === undefined) { - const { conditions, params } = recordConditions(query, reach); + const where = recordConditions(query, reach); const field = getSortField(query); const col = getSortColumn(field); if (decoded) { const cursor = nativeCursor(decoded, field); - conditions.push(`(r.${col} ${op} ? OR (r.${col} = ? AND r.id ${op} ?))`); - params.push(cursor.value, cursor.value, cursor.id); + where.conditions.push(`(r.${col} ${op} ? OR (r.${col} = ? AND r.id ${op} ?))`); + where.params.push(cursor.value, cursor.value, cursor.id); } - return [ - statement( - 'r.*', - 'records r', - conditions, - params, - ` ORDER BY r.${col} ${dir}, r.id ${dir} LIMIT ?`, - ), - ]; + return [statement('records r', where, ` ORDER BY r.${col} ${dir}, r.id ${dir} LIMIT ?`)]; } const cursor = decoded ? contentCursor(decoded, contentField) : null; @@ -518,34 +472,34 @@ export const buildQueryPlan = (query: StackQuery, reach?: Reach): QueryStatement // Records holding a value at the field, in that value's order. A cursor // sitting in the absent partition has already passed every one of them. if (cursor?.kind !== 'absent') { - const { conditions, params } = recordConditions(query, reach); + const where = recordConditions(query, reach); const from = `content_index ${SORT_ALIAS} JOIN records r ON r.id = ${SORT_ALIAS}.record_id`; - conditions.push(`${SORT_ALIAS}.field = ?`); - params.push(contentField); - if (cursor) conditions.push(presentCursorCondition(cursor, op, params)); + where.conditions.push(`${SORT_ALIAS}.field = ?`); + where.params.push(contentField); + if (cursor) where.conditions.push(presentCursorCondition(cursor, op, where.params)); // Ordered exactly as idx_content_index_sort stores it, so the page is // an ordered walk of the index rather than a sort of the matches. const order = `${SORT_ALIAS}.value_rank ${dir}, ${SORT_ALIAS}.num_value ${dir}, ` + `${SORT_ALIAS}.text_key ${dir}, ${SORT_ALIAS}.text_value ${dir}, ` + `${SORT_ALIAS}.record_id ${dir}`; - pages.push(statement('r.*', from, conditions, params, ` ORDER BY ${order} LIMIT ?`)); + pages.push(statement(from, where, ` ORDER BY ${order} LIMIT ?`)); } // Records holding nothing there, which trail the others whichever way // the sort runs (docs/spec/data-model.md § Sorting by a content field). { - const { conditions, params } = recordConditions(query, reach); - conditions.push( + const where = recordConditions(query, reach); + where.conditions.push( `NOT EXISTS (SELECT 1 FROM content_index ${SORT_ALIAS} ` + `WHERE ${SORT_ALIAS}.record_id = r.id AND ${SORT_ALIAS}.field = ?)`, ); - params.push(contentField); + where.params.push(contentField); if (cursor?.kind === 'absent') { - conditions.push(`r.id ${op} ?`); - params.push(cursor.id); + where.conditions.push(`r.id ${op} ?`); + where.params.push(cursor.id); } - pages.push(statement('r.*', 'records r', conditions, params, ` ORDER BY r.id ${dir} LIMIT ?`)); + pages.push(statement('records r', where, ` ORDER BY r.id ${dir} LIMIT ?`)); } return pages; @@ -554,8 +508,9 @@ export const buildQueryPlan = (query: StackQuery, reach?: Reach): QueryStatement /** * A sort naming its value per Type, read through a derived table in place * of the index: each Type's own index rows, and its constant in the index's - * columns, so one cursor condition and one ORDER BY serve both. See - * docs/spec/data-model.md § Sorting by a content field. + * columns, so one cursor condition and one ORDER BY serve both. Each kind + * of entry is a list of rows, bound as one parameter like any other list. + * See docs/spec/data-model.md § Sorting by a content field. */ const byTypePlan = ( query: StackQuery, @@ -577,8 +532,7 @@ const byTypePlan = ( if (entry) constants.push([typeId, entry]); } const pages: QueryStatement[] = []; - const fieldOf = (alias: string, record: string) => - fields.map(() => `(${record}.type_id = ? AND ${alias}.field = ?)`).join(' OR '); + const at = (i: number) => `json_extract(e.value, '$[${i}]')`; if (cursor?.kind !== 'absent' && fields.length + constants.length > 0) { const sources: string[] = []; @@ -587,22 +541,29 @@ const byTypePlan = ( sources.push( 'SELECT c.record_id AS record_id, c.value_rank AS value_rank, c.num_value AS num_value, ' + 'c.text_key AS text_key, c.text_value AS text_value ' + - `FROM content_index c JOIN records t ON t.id = c.record_id WHERE ${fieldOf('c', 't')}`, + `FROM json_each(?) e JOIN content_index c ON c.field = ${at(1)} ` + + `JOIN records t ON t.id = c.record_id AND t.type_id = ${at(0)}`, ); - for (const [typeId, field] of fields) sourceParams.push(typeId, field); + sourceParams.push(JSON.stringify(fields)); } - for (const [typeId, entry] of constants) { + if (constants.length > 0) { sources.push( - 'SELECT t.id AS record_id, ? AS value_rank, ? AS num_value, ? AS text_key, ' + - '? AS text_value FROM records t WHERE t.type_id = ?', + `SELECT t.id AS record_id, ${at(1)} AS value_rank, ${at(2)} AS num_value, ` + + `${at(3)} AS text_key, ${at(4)} AS text_value ` + + `FROM json_each(?) e JOIN records t ON t.type_id = ${at(0)}`, ); sourceParams.push( - ...(entry.kind === 'num' ? [0, entry.num, null, null] : [1, null, entry.key, entry.text]), - typeId, + JSON.stringify( + constants.map(([typeId, entry]) => + entry.kind === 'num' + ? [typeId, 0, entry.num, null, null] + : [typeId, 1, null, entry.key, entry.text], + ), + ), ); } - const { conditions, params } = recordConditions(query, reach); - if (cursor) conditions.push(presentCursorCondition(cursor, op, params)); + const where = recordConditions(query, reach); + if (cursor) where.conditions.push(presentCursorCondition(cursor, op, where.params)); const from = `(${sources.join(' UNION ALL ')}) ${SORT_ALIAS} ` + `JOIN records r ON r.id = ${SORT_ALIAS}.record_id`; @@ -610,37 +571,30 @@ const byTypePlan = ( `${SORT_ALIAS}.value_rank ${dir}, ${SORT_ALIAS}.num_value ${dir}, ` + `${SORT_ALIAS}.text_key ${dir}, ${SORT_ALIAS}.text_value ${dir}, ` + `${SORT_ALIAS}.record_id ${dir}`; - pages.push( - statement( - 'r.*', - from, - conditions, - [...sourceParams, ...params], - ` ORDER BY ${order} LIMIT ?`, - ), - ); + pages.push(statement(from, where, ` ORDER BY ${order} LIMIT ?`, sourceParams)); } // Records holding nothing to order by: a Type's field with no value, a // `null` constant, or a Type the map does not name. { - const { conditions, params } = recordConditions(query, reach); + const where = recordConditions(query, reach); if (fields.length > 0) { - conditions.push( - `NOT EXISTS (SELECT 1 FROM content_index ${SORT_ALIAS} ` + - `WHERE ${SORT_ALIAS}.record_id = r.id AND (${fieldOf(SORT_ALIAS, 'r')}))`, + where.conditions.push( + `NOT EXISTS (SELECT 1 FROM json_each(?) e JOIN content_index ${SORT_ALIAS} ` + + `ON ${SORT_ALIAS}.record_id = r.id AND ${SORT_ALIAS}.field = ${at(1)} ` + + `WHERE ${at(0)} = r.type_id)`, ); - for (const [typeId, field] of fields) params.push(typeId, field); + where.params.push(JSON.stringify(fields)); } if (constants.length > 0) { const typeIds = constants.map(([typeId]) => typeId); - conditions.push(`NOT (${inList('r.type_id', typeIds, params)})`); + where.conditions.push(`NOT (${inList('r.type_id', typeIds, where.params)})`); } if (cursor?.kind === 'absent') { - conditions.push(`r.id ${op} ?`); - params.push(cursor.id); + where.conditions.push(`r.id ${op} ?`); + where.params.push(cursor.id); } - pages.push(statement('r.*', 'records r', conditions, params, ` ORDER BY r.id ${dir} LIMIT ?`)); + pages.push(statement('records r', where, ` ORDER BY r.id ${dir} LIMIT ?`)); } return pages; }; diff --git a/packages/sqlite-shared/src/record-adapter.ts b/packages/sqlite-shared/src/record-adapter.ts index ba00f6fa..f051a560 100644 --- a/packages/sqlite-shared/src/record-adapter.ts +++ b/packages/sqlite-shared/src/record-adapter.ts @@ -53,8 +53,9 @@ export const SQLITE_RECORD_CAPABILITIES: StackCapabilities = { filter: { content: 'path', contentPresent: true, - // Inside SQLite's limits on expression depth and compound selects; see - // docs/spec/adapters.md § Adapter capabilities. + // A per-Type key costs no statement text or parameter of its own, so + // every engine honors the same cap; see docs/spec/adapters.md § Statement + // size. contentByType: 64, search: true, }, From 9214bb3f2cc10bfd7b80e755fe4fd8c122635210 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 21 Oct 2026 01:52:45 +0000 Subject: [PATCH 2/9] fix(sqlite): a null filter matches a path reaching a null beside a value MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The null op held only where the path reached no non-null value, which made it the exact complement of contentPresent. The spec has them overlap on a multi-valued path: [null, 'x'] satisfies both. It now holds where the walk reaches a null, or reaches nothing non-null, and the conformance suite pins the overlap for every adapter. Also from review: the content path cap reads the nesting constant rather than restating 32, and § Statement size says a statement never grows with a query's entries, naming the single-value binding and contentFieldByType's two lists. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Mf3p8BVtrQJLZx1JFCHchC --- .changeset/fixed-statement-size.md | 4 ++- docs/spec/adapters.md | 4 +-- packages/adapter-conformance/src/record.ts | 18 +++++++++++ packages/core/src/query-validation.ts | 14 ++++----- packages/core/src/validate.ts | 3 +- packages/sqlite-shared/src/query.ts | 35 +++++++++++++--------- 6 files changed, 52 insertions(+), 26 deletions(-) diff --git a/.changeset/fixed-statement-size.md b/.changeset/fixed-statement-size.md index 2c614eb3..578730ca 100644 --- a/.changeset/fixed-statement-size.md +++ b/.changeset/fixed-statement-size.md @@ -1,6 +1,8 @@ --- '@haverstack/record-adapter-do-sqlite': minor '@haverstack/record-adapter-sqlite': patch +'@haverstack/adapter-conformance': minor +'@haverstack/core': patch --- -A SQLite statement's text and bound parameters no longer grow with a query's size. Content conditions — every `filter.content` and `filter.contentPresent` key and every `filter.contentByType` arm — bind as one table of rows whose paths are walked by a recursive query, and `sort.contentFieldByType`'s entries bind as one list, so Durable Object SQLite answers queries at the declared `contentByType` and `contentFieldByType` caps, and content filters of any size, instead of failing with `too many SQL variables` or `statement too long`. +A SQLite statement's text and bound parameters no longer grow with a query's size. Content conditions — every `filter.content` and `filter.contentPresent` key and every `filter.contentByType` arm — bind as one table of rows whose paths are walked by a recursive query, and `sort.contentFieldByType`'s entries bind as one list, so Durable Object SQLite answers queries at the declared `contentByType` and `contentFieldByType` caps, and content filters of any size, instead of failing with `too many SQL variables` or `statement too long`. The conformance suite now holds every adapter to a multi-valued path satisfying both `contentPresent` and a `null` content filter, and the content filter path cap is the content nesting cap itself rather than a second constant. diff --git a/docs/spec/adapters.md b/docs/spec/adapters.md index c790dd3b..c9d40637 100644 --- a/docs/spec/adapters.md +++ b/docs/spec/adapters.md @@ -174,9 +174,9 @@ A missing row is not only a stale read. `deleteUnreferencedAttachmentRecords()` ### Statement size -**A statement's text and its bound parameters are fixed by which keys a query sets, never by how many entries they hold.** Durable Object SQLite caps a statement at 100 parameters and 100 KB of text, and a query's size is not the adapter's to choose: a reach is sized by the requester's grants and rosters, a filter by whoever wrote it. A statement that grew with either would fail on one engine for a query another answers, and a declared [cap](#adapter-capabilities) could not promise anything. Every caller-sized part of a query is therefore data, bound as one JSON parameter and read through `json_each`: +**A statement's text and its bound parameters never grow with how many entries a query's keys hold.** Durable Object SQLite caps a statement at 100 parameters and 100 KB of text, and a query's size is not the adapter's to choose: a reach is sized by the requester's grants and rosters, a filter by whoever wrote it. A statement that grew with either would fail on one engine for a query another answers, and a declared [cap](#adapter-capabilities) could not promise anything. Every caller-sized part of a query is therefore data, bound as one JSON parameter and read through `json_each`: -- **A list** — a filter's type, app and author lists, its tags, a reach's type lists and permission sets, and the per-Type entries of `sort.contentFieldByType`, each one row. A single value still binds as itself, so an index on its column keeps yielding rows in order. +- **A list** — a filter's type, app and author lists, its tags, a reach's type lists and permission sets, and the per-Type entries of `sort.contentFieldByType`, which bind as two lists of rows, one for the Types ordered by a field and one for the Types ordered by a constant. A list of one value binds as that value instead, so an index on its column keeps yielding rows in order; that is a smaller statement, never a larger one. - **The content conditions** — every `filter.content` and `filter.contentPresent` key, and every key of every `filter.contentByType` arm, as one table of rows, each naming its path, what it asks, and the Type it is scoped to (none for `filter.content`). A record matches when every row scoped to nothing or to its own Type holds. A row's path is walked by a recursive query that matches one segment per step, so neither the number of conditions nor a path's length adds text. What remains binds a fixed count: a scalar filter's value, a date bound, a cursor's position, the page's row budget. Since a statement's size is set by its shape and not by its query, every engine running the shared query builder honors the same capability declaration. diff --git a/packages/adapter-conformance/src/record.ts b/packages/adapter-conformance/src/record.ts index e5c269f1..1cfda362 100644 --- a/packages/adapter-conformance/src/record.ts +++ b/packages/adapter-conformance/src/record.ts @@ -1284,6 +1284,24 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt }); } + if (capabilities.filter.contentPresent && capabilities.filter.content === 'path') { + // Both are element-wise, so a path reaching a null and a value + // satisfies each. See docs/spec/data-model.md § Filter. + test('a multi-valued path satisfies both contentPresent and a null filter', async () => { + const mixed = makeRecord({ + content: { title: 'x', tags: [null, 'a'], emails: [{ value: null }, { value: 1 }] }, + }); + await adapter.createRecord(mixed); + + for (const path of ['tags', 'emails.value']) { + const present = await adapter.queryRecords({ filter: { contentPresent: [path] } }); + expect(present.records.map((r) => r.id)).toEqual([mixed.id]); + const nulls = await adapter.queryRecords({ filter: { content: { [path]: null } } }); + expect(nulls.records.map((r) => r.id)).toEqual([mixed.id]); + } + }); + } + if (capabilities.filter.content === 'path') { test('an array is spread element-wise when filtering a nested path', async () => { const match = makeRecord({ diff --git a/packages/core/src/query-validation.ts b/packages/core/src/query-validation.ts index 79462506..b249a315 100644 --- a/packages/core/src/query-validation.ts +++ b/packages/core/src/query-validation.ts @@ -19,6 +19,7 @@ import { ARGUMENTS_INVALID, StackBadRequestError, StackValidationError } from './errors.js'; import { familyIdProblem, parseTypeId } from './schema.js'; import { targetErrors } from './associations/validation.js'; +import { MAX_VALIDATION_DEPTH } from './validate.js'; import { CONTENT_SEGMENT_METACHARACTERS, SEGMENT_METACHARACTER_RE, @@ -167,9 +168,6 @@ export function filtersContent(features: Pick): boo return features.filter.content !== 'none'; } -/** The deepest content a write accepts — see the spec link below. */ -const MAX_CONTENT_PATH_SEGMENTS = 32; - /** * Split a content filter key into path segments. Write-time validation * keeps stored field names free of the path metacharacters, but a filter @@ -178,12 +176,12 @@ const MAX_CONTENT_PATH_SEGMENTS = 32; */ export function parseContentFilterKey(key: string): string[] { const segments = key.split('.'); - // Each segment descends one level of content nested at most 32 deep, so - // a longer path could reach no stored value. See docs/spec/data-model.md - // § Nested content paths. - if (segments.length > MAX_CONTENT_PATH_SEGMENTS) { + // Each segment descends one level of content, so a path deeper than + // content may nest could reach no stored value. See + // docs/spec/data-model.md § Nested content paths. + if (segments.length > MAX_VALIDATION_DEPTH) { throw new StackBadRequestError( - `Invalid content filter path "${key}": at most ${MAX_CONTENT_PATH_SEGMENTS} segments.`, + `Invalid content filter path "${key}": at most ${MAX_VALIDATION_DEPTH} segments.`, ); } for (const segment of segments) { diff --git a/packages/core/src/validate.ts b/packages/core/src/validate.ts index 3f260f04..ff8830aa 100644 --- a/packages/core/src/validate.ts +++ b/packages/core/src/validate.ts @@ -12,7 +12,8 @@ import type { TypeSchema, FieldDef, ScalarFieldKind } from './types/index.js'; -const MAX_VALIDATION_DEPTH = 32; +/** How deep content may nest; a content filter path is capped by it too. */ +export const MAX_VALIDATION_DEPTH = 32; /** fileId format: SHA-256 hex, lowercase — matches blob-adapter-disk's assertFileId(). */ const FILE_ID_RE = /^[0-9a-f]{64}$/; diff --git a/packages/sqlite-shared/src/query.ts b/packages/sqlite-shared/src/query.ts index 3812ff5f..609b5d31 100644 --- a/packages/sqlite-shared/src/query.ts +++ b/packages/sqlite-shared/src/query.ts @@ -94,21 +94,28 @@ const CONTENT_CONDITIONS = "json_extract(value, '$[2]') AS op, json_extract(value, '$[3]') AS val FROM json_each(?)) "; /** - * Whether the `cond` row `c` holds for `r`, walked as data by a recursive - * CTE: each step matches one segment against json_each's `key`, so no key - * is read as syntax and the statement's text is the same for any path. - * `'absent'` holds where `'present'` does not; an `'eq'` value never - * matches an object or array, which json_each exposes as JSON text. See - * docs/spec/data-model.md § Filter. + * Whether the walk of the `cond` row `c`'s path through `r` reaches a value + * satisfying `leaf`. The walk is data, a recursive CTE matching one segment + * per step against json_each's `key`, so no key is read as syntax and the + * statement's text is the same for any path. */ -const HOLDS = - "((c.op = 'absent') != EXISTS (WITH RECURSIVE w(d, v, t) AS (" + - "SELECT 0, r.content, 'object' UNION ALL " + +const reaches = (leaf: string): string => + "EXISTS (WITH RECURSIVE w(d, v, t) AS (SELECT 0, r.content, 'object' UNION ALL " + `SELECT w.d + 1, e.value, e.type FROM w JOIN json_each(${members('w')}) m ` + `JOIN json_each(${spread('m')}) e ` + "WHERE w.d < c.depth AND m.key = json_extract(c.path, '$[' || w.d || ']')) " + - "SELECT 1 FROM w WHERE w.d = c.depth AND CASE c.op WHEN 'eq' THEN " + - "w.t NOT IN ('object', 'array') AND w.v = c.val ELSE w.v IS NOT NULL END))"; + `SELECT 1 FROM w WHERE w.d = c.depth AND ${leaf})`; + +/** + * Whether the `cond` row `c` holds for `r`. Each op is element-wise, so a + * path reaching a null and a value meets both `'null'` and `'present'`; an + * `'eq'` value never matches an object or array, which json_each exposes as + * JSON text. See docs/spec/data-model.md § Filter. + */ +const LEAF = + "CASE c.op WHEN 'eq' THEN w.t NOT IN ('object', 'array') AND w.v = c.val " + + "WHEN 'present' THEN w.v IS NOT NULL ELSE w.t = 'null' END"; +const HOLDS = `(${reaches(LEAF)} OR (c.op = 'null' AND NOT ${reaches('w.v IS NOT NULL')}))`; /** The join alias the content index is read through. */ const SORT_ALIAS = 'cs'; @@ -138,8 +145,8 @@ const contentCursor = (cursor: DecodedCursor, field: string) => { /** * A `content` and `contentPresent` predicate as `cond` rows scoped to * `scope`. A `null` value means "no value at the path, or a value that is - * null" (docs/spec/data-model.md § Filter): the path holds no non-null - * value. Presence is the question an exact-match value cannot ask. + * null" (docs/spec/data-model.md § Filter). Presence is the question an + * exact-match value cannot ask. */ const contentRows = (scope: string | null, predicate: ContentArm): unknown[][] => [ ...Object.entries(predicate.content ?? {}).map(([key, value]) => @@ -147,7 +154,7 @@ const contentRows = (scope: string | null, predicate: ContentArm): unknown[][] = // over the wire, so treating it as a value would make an in-process // query and its wire equivalent disagree. value === null || value === undefined - ? [scope, parseContentFilterKey(key), 'absent'] + ? [scope, parseContentFilterKey(key), 'null'] : [scope, parseContentFilterKey(key), 'eq', value], ), ...(predicate.contentPresent ?? []).map((key) => [scope, parseContentFilterKey(key), 'present']), From e1dd51e63d8ae06ffac78478c382de073535e180 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 21 Oct 2026 17:10:18 +0000 Subject: [PATCH 3/9] fix(sqlite): one rows primitive, one walk, and equality by JSON type Review found the SQL leaf drifting from matchesContent() again: a filter value of true matched a stored 1, because spread() rewrapped a boolean from json_each's 0/1 value. Booleans now keep their type through the walk, and an equals condition compares kind as well as value, as === does; the conformance suite pins it. Every caller-sized part of a query is now one primitive, rows(): a JSON array of tuples read as named columns. Lists, permission sets, contentFieldByType's two tables and the content conditions all use it. Each condition walks its path once and gathers what its op asks, which also halves the isNull cost. Ops are equals, present and isNull. The path cap is derived from the nesting cap instead of restating it: a top-level field sits at depth 0, so 33 segments reach the deepest value a write accepts. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Mf3p8BVtrQJLZx1JFCHchC --- .changeset/fixed-statement-size.md | 8 +- docs/spec/adapters.md | 8 +- docs/spec/data-model.md | 8 +- packages/adapter-conformance/src/record.ts | 18 ++ packages/core/src/content-match.ts | 4 +- packages/core/src/query-validation.ts | 17 +- packages/core/src/validate.ts | 2 +- .../tests/record.test.ts | 6 +- .../tests/record.test.ts | 20 +- packages/sqlite-shared/src/query.ts | 201 ++++++++++-------- 10 files changed, 176 insertions(+), 116 deletions(-) diff --git a/.changeset/fixed-statement-size.md b/.changeset/fixed-statement-size.md index 578730ca..874e86e0 100644 --- a/.changeset/fixed-statement-size.md +++ b/.changeset/fixed-statement-size.md @@ -1,8 +1,10 @@ --- '@haverstack/record-adapter-do-sqlite': minor -'@haverstack/record-adapter-sqlite': patch +'@haverstack/record-adapter-sqlite': minor '@haverstack/adapter-conformance': minor -'@haverstack/core': patch +'@haverstack/core': minor --- -A SQLite statement's text and bound parameters no longer grow with a query's size. Content conditions — every `filter.content` and `filter.contentPresent` key and every `filter.contentByType` arm — bind as one table of rows whose paths are walked by a recursive query, and `sort.contentFieldByType`'s entries bind as one list, so Durable Object SQLite answers queries at the declared `contentByType` and `contentFieldByType` caps, and content filters of any size, instead of failing with `too many SQL variables` or `statement too long`. The conformance suite now holds every adapter to a multi-valued path satisfying both `contentPresent` and a `null` content filter, and the content filter path cap is the content nesting cap itself rather than a second constant. +A SQLite statement's text and bound parameters no longer grow with a query's size. Every caller-sized part of a query binds as one table of rows: lists, permission sets, `sort.contentFieldByType`'s entries, and the content conditions — every `filter.content` and `filter.contentPresent` key and every `filter.contentByType` arm — whose paths one recursive walk per condition resolves. Durable Object SQLite now answers queries at the declared `contentByType` and `contentFieldByType` caps, and content filters of any size, instead of failing with `too many SQL variables` or `statement too long`. + +A content filter value now matches only a value of its own JSON type in the SQLite adapters, as `===` does: `true` no longer matches `1`. A content filter path may be 33 segments, the depth that reaches the deepest value a write accepts. The conformance suite holds every adapter to both, and to a multi-valued path satisfying both `contentPresent` and a `null` filter. diff --git a/docs/spec/adapters.md b/docs/spec/adapters.md index c9d40637..9e9ad58d 100644 --- a/docs/spec/adapters.md +++ b/docs/spec/adapters.md @@ -174,10 +174,12 @@ A missing row is not only a stale read. `deleteUnreferencedAttachmentRecords()` ### Statement size -**A statement's text and its bound parameters never grow with how many entries a query's keys hold.** Durable Object SQLite caps a statement at 100 parameters and 100 KB of text, and a query's size is not the adapter's to choose: a reach is sized by the requester's grants and rosters, a filter by whoever wrote it. A statement that grew with either would fail on one engine for a query another answers, and a declared [cap](#adapter-capabilities) could not promise anything. Every caller-sized part of a query is therefore data, bound as one JSON parameter and read through `json_each`: +**A statement's text and its bound parameters never grow with how many entries a query's keys hold.** Durable Object SQLite caps a statement at 100 parameters and 100 KB of text, and a query's size is not the adapter's to choose: a reach is sized by the requester's grants and rosters, a filter by whoever wrote it. A statement that grew with either would fail on one engine for a query another answers, and a declared [cap](#adapter-capabilities) could not promise anything. **Every caller-sized part of a query is a table of rows**, bound as one JSON array of tuples and read through `json_each` as named columns: -- **A list** — a filter's type, app and author lists, its tags, a reach's type lists and permission sets, and the per-Type entries of `sort.contentFieldByType`, which bind as two lists of rows, one for the Types ordered by a field and one for the Types ordered by a constant. A list of one value binds as that value instead, so an index on its column keeps yielding rows in order; that is a smaller statement, never a larger one. -- **The content conditions** — every `filter.content` and `filter.contentPresent` key, and every key of every `filter.contentByType` arm, as one table of rows, each naming its path, what it asks, and the Type it is scoped to (none for `filter.content`). A record matches when every row scoped to nothing or to its own Type holds. A row's path is walked by a recursive query that matches one segment per step, so neither the number of conditions nor a path's length adds text. +- **A list is rows of one column** — a filter's type, app and author lists and its tags, and a reach's type lists. A list of one value binds as that value instead, so an index on its column keeps yielding rows in order; that is a smaller statement, never a larger one. +- **A reach's permission sets** are rows of one element each, naming the set it belongs to and that set's size. +- **`sort.contentFieldByType`** is two tables, the Types ordered by a field and the Types ordered by a constant. +- **The content conditions** — every `filter.content` and `filter.contentPresent` key, and every key of every `filter.contentByType` arm — are one table, each row naming its path, what it asks, and the Type it is scoped to (none for `filter.content`). A record matches when every row scoped to nothing or to its own Type holds. One recursive walk per row matches a segment per step and gathers what the row asks of the values it reaches, so neither the number of conditions nor a path's length adds text. What remains binds a fixed count: a scalar filter's value, a date bound, a cursor's position, the page's row budget. Since a statement's size is set by its shape and not by its query, every engine running the shared query builder honors the same capability declaration. diff --git a/docs/spec/data-model.md b/docs/spec/data-model.md index d67d40c3..50558d14 100644 --- a/docs/spec/data-model.md +++ b/docs/spec/data-model.md @@ -454,7 +454,7 @@ A record at an older version that no passed migration carries to the handle's is **A subscription covers the family.** A change's `record`, when present, is migrated to the handle's version; the change's own `typeId` stays the stored one, as `RecordChange.typeId` documents. A change whose record has `deletedAt` set carries neither `record` nor `misfit`, on either surface, and neither does a change that [migrates the record out](#type-migrations) of the family, which the filter matches by its previous `typeId`. The change filter takes every key but `typeId` and `baseId`. -**A content filter or sort spans the versions the family's migrations can carry it to**, and reaches no further: see [Content queries across versions](#content-queries-across-versions). The keys are typed against the handle's version: `filter.content` and `filter.contentPresent` take the content's paths (`ContentPathOf`), through declared objects and arrays, with any suffix below an `open` object or array, and a value of the field's type as a write takes it (an array's element type, since matching is element-wise) or `null`. A path reaching an object takes only `null`, since an adapter compares the value a path reaches and an object never equals one; a value below an `open` node is any scalar. Typing values as writes take them means an enum value the handle does not list is filtered for with a cast, the price of refusing a misspelled one; `sort.contentField` takes only a top-level scalar (`SortableFieldOf`). Path typing stops six segments deep and accepts any suffix below that, a limit of the compiler, not of the [32-segment path](#nested-content-paths). `query()` takes neither `typeId` nor `presentAt`, which would change which records a page holds or the shape they arrive in, and drops either if an untyped caller passes it; `subscribe()` drops a filter's `typeId` the same way. +**A content filter or sort spans the versions the family's migrations can carry it to**, and reaches no further: see [Content queries across versions](#content-queries-across-versions). The keys are typed against the handle's version: `filter.content` and `filter.contentPresent` take the content's paths (`ContentPathOf`), through declared objects and arrays, with any suffix below an `open` object or array, and a value of the field's type as a write takes it (an array's element type, since matching is element-wise) or `null`. A path reaching an object takes only `null`, since an adapter compares the value a path reaches and an object never equals one; a value below an `open` node is any scalar. Typing values as writes take them means an enum value the handle does not list is filtered for with a cast, the price of refusing a misspelled one; `sort.contentField` takes only a top-level scalar (`SortableFieldOf`). Path typing stops six segments deep and accepts any suffix below that, a limit of the compiler, not of the [33-segment path](#nested-content-paths). `query()` takes neither `typeId` nor `presentAt`, which would change which records a page holds or the shape they arrive in, and drops either if an untyped caller passes it; `subscribe()` drops a filter's `typeId` the same way. **Every verb that returns a record returns it as `get()` would.** The write already returned the record, so narrowing it costs no round trip, and a `restoreVersion()` to a pre-migration snapshot reads back migrated, like any older record. A write is checked against the read it already makes, before anything lands: a record outside the family is refused with `StackNotFoundError`, the answer `get()` gives, and so is a `restoreVersion()` to a snapshot taken before the record migrated into the family, which is not a version of a record in the collection. A content write to a record stored at a newer version is refused with `MisfitError` before anything is written, since its patch would be validated against a schema the handle does not know. Any other write that leaves a misfit lands, then throws `MisfitError` carrying the stored record, so the app knows the write landed. A deleted record, or a stale `ifVersion`, is refused first, as the untyped write refuses it. @@ -620,9 +620,11 @@ The motivating query is a listing: an article's `publishedAt` is optional, so dr **An array along the path is matched element-wise.** `contact@1` stores `emails` as an array of `{ value, label }`, and the question that motivates paths at all — which contact holds this address — is that shape. Traversal and containment are therefore one feature rather than two: a segment applies to each element of an array it meets, and the final comparison does the same, so `{ content: { tags: 'starred' } }` matches a record whose `tags` array contains `'starred'`. A record storing a scalar where another stores an array is matched by the same filter either way. -A path that descends through a scalar, or through a field that isn't there, reaches no value and matches nothing — it is not an error. A path may be at most **32 segments**; longer is `StackBadRequestError` (400), as is an empty segment (`a..b`) or a segment containing a reserved character. Those are structural faults in the request, refused rather than answered with an empty result, on the same reasoning as a malformed cursor. +**A value matches a value of its own JSON type.** `{ content: { flag: true } }` matches `true` and never `1`, and `1` never matches `'1'` — the comparison JavaScript's `===` makes, which is the reference reading every adapter is held to. A number is one type, however an engine stores it. An object or array never equals a filter value, so a path reaching one matches only `null` or presence. -**A segment is a value, never syntax.** An adapter resolving a path MUST carry each segment as a bound value matched against a key, rather than assembling it into a path expression — so no field name a write would accept can be reinterpreted as syntax, and no key can make the statement itself malformed. The 32-segment cap is the [nesting cap](#types) read from the other side: each segment descends one level, so a longer path could reach no value a write accepts, and is refused as a fault in the request rather than answered with a match that cannot exist. +A path that descends through a scalar, or through a field that isn't there, reaches no value and matches nothing — it is not an error. A path may be at most **33 segments**; longer is `StackBadRequestError` (400), as is an empty segment (`a..b`) or a segment containing a reserved character. Those are structural faults in the request, refused rather than answered with an empty result, on the same reasoning as a malformed cursor. + +**A segment is a value, never syntax.** An adapter resolving a path MUST carry each segment as a bound value matched against a key, rather than assembling it into a path expression — so no field name a write would accept can be reinterpreted as syntax, and no key can make the statement itself malformed. The 33-segment cap is the [nesting cap](#types) read from the other side: a top-level field is reached by one segment and each further segment descends one level, so 33 reach the deepest value a write accepts and a longer path reaches none — it is refused as a fault in the request rather than answered with a match that cannot exist. **Multi-segment keys need the `'path'` rung of `filter.content`**, where a single-segment key needs only `'field'` — see [Capability-gated filters](#capability-gated-filters), and [Adapter capabilities](./adapters.md#adapter-capabilities) for why the reach is one ordered value rather than a flag per rung. diff --git a/packages/adapter-conformance/src/record.ts b/packages/adapter-conformance/src/record.ts index 1cfda362..68c868ed 100644 --- a/packages/adapter-conformance/src/record.ts +++ b/packages/adapter-conformance/src/record.ts @@ -1284,6 +1284,24 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt }); } + // A value matches as `===` compares it: its JSON type as well as + // its value. See docs/spec/data-model.md § Filter. + test('a content filter value matches its own JSON type only', async () => { + const bool = makeRecord({ content: { title: 'b', flag: true } }); + const num = makeRecord({ content: { title: 'n', flag: 1 } }); + const text = makeRecord({ content: { title: 't', flag: '1' } }); + for (const record of [bool, num, text]) await adapter.createRecord(record); + + for (const [value, record] of [ + [true, bool], + [1, num], + ['1', text], + ] as const) { + const result = await adapter.queryRecords({ filter: { content: { flag: value } } }); + expect(result.records.map((r) => r.id)).toEqual([record.id]); + } + }); + if (capabilities.filter.contentPresent && capabilities.filter.content === 'path') { // Both are element-wise, so a path reaching a null and a value // satisfies each. See docs/spec/data-model.md § Filter. diff --git a/packages/core/src/content-match.ts b/packages/core/src/content-match.ts index 5b2857b6..2ddd0b41 100644 --- a/packages/core/src/content-match.ts +++ b/packages/core/src/content-match.ts @@ -17,8 +17,8 @@ const spreadValue = (value: unknown): unknown[] => (Array.isArray(value) ? value /** * Every value reachable by walking `segments`, spreading arrays at each - * step — sqlite-shared's contentPathExists() walk, so a filter answers the - * same against either. Empty means the path reaches nothing, which a `null` + * step — the walk sqlite-shared runs in SQL, so a filter answers the same + * against either. Empty means the path reaches nothing, which a `null` * filter value matches. */ export const valuesAtContentPath = ( diff --git a/packages/core/src/query-validation.ts b/packages/core/src/query-validation.ts index b249a315..1a9fa3b3 100644 --- a/packages/core/src/query-validation.ts +++ b/packages/core/src/query-validation.ts @@ -19,9 +19,9 @@ import { ARGUMENTS_INVALID, StackBadRequestError, StackValidationError } from './errors.js'; import { familyIdProblem, parseTypeId } from './schema.js'; import { targetErrors } from './associations/validation.js'; -import { MAX_VALIDATION_DEPTH } from './validate.js'; import { CONTENT_SEGMENT_METACHARACTERS, + MAX_VALIDATION_DEPTH, SEGMENT_METACHARACTER_RE, isPlainObject, } from './validate.js'; @@ -168,6 +168,14 @@ export function filtersContent(features: Pick): boo return features.filter.content !== 'none'; } +/** + * The longest path that can reach a stored value: a top-level field sits at + * depth 0 and each segment descends one level, so a path past this reaches + * nothing a write accepts. See docs/spec/data-model.md § Nested content + * paths. + */ +const MAX_CONTENT_PATH_SEGMENTS = MAX_VALIDATION_DEPTH + 1; + /** * Split a content filter key into path segments. Write-time validation * keeps stored field names free of the path metacharacters, but a filter @@ -176,12 +184,9 @@ export function filtersContent(features: Pick): boo */ export function parseContentFilterKey(key: string): string[] { const segments = key.split('.'); - // Each segment descends one level of content, so a path deeper than - // content may nest could reach no stored value. See - // docs/spec/data-model.md § Nested content paths. - if (segments.length > MAX_VALIDATION_DEPTH) { + if (segments.length > MAX_CONTENT_PATH_SEGMENTS) { throw new StackBadRequestError( - `Invalid content filter path "${key}": at most ${MAX_VALIDATION_DEPTH} segments.`, + `Invalid content filter path "${key}": at most ${MAX_CONTENT_PATH_SEGMENTS} segments.`, ); } for (const segment of segments) { diff --git a/packages/core/src/validate.ts b/packages/core/src/validate.ts index ff8830aa..9618e844 100644 --- a/packages/core/src/validate.ts +++ b/packages/core/src/validate.ts @@ -12,7 +12,7 @@ import type { TypeSchema, FieldDef, ScalarFieldKind } from './types/index.js'; -/** How deep content may nest; a content filter path is capped by it too. */ +/** How deep content may nest below a top-level field, which is depth 0. */ export const MAX_VALIDATION_DEPTH = 32; /** fileId format: SHA-256 hex, lowercase — matches blob-adapter-disk's assertFileId(). */ diff --git a/packages/record-adapter-do-sqlite/tests/record.test.ts b/packages/record-adapter-do-sqlite/tests/record.test.ts index 541ce554..83b0ef83 100644 --- a/packages/record-adapter-do-sqlite/tests/record.test.ts +++ b/packages/record-adapter-do-sqlite/tests/record.test.ts @@ -338,14 +338,14 @@ describe('records — queries', () => { test('a path at the segment cap executes; one past it is refused', async () => { const stub = getStub(); await stub.createRecord(makeRecord({ id: 'r1', content: { text: 'shallow' } })); - const deepest = Array.from({ length: 32 }, (_, i) => `s${i}`).join('.'); + const deepest = Array.from({ length: 33 }, (_, i) => `s${i}`).join('.'); const miss = await stub.queryRecords({ filter: { content: { [deepest]: 'x' } } }); expect(miss.records).toEqual([]); const absent = await stub.queryRecords({ filter: { content: { [deepest]: null } } }); expect(absent.records.map((r) => r.id)).toEqual(['r1']); const err = await stub - .queryRecords({ filter: { content: { [`${deepest}.s32`]: 'x' } } }) + .queryRecords({ filter: { content: { [`${deepest}.s33`]: 'x' } } }) .catch((e: unknown) => e); expect((err as { code?: string }).code).toBe('bad_request'); }); @@ -659,7 +659,7 @@ test('content conditions far past the statement limits execute', async () => { const stub = getStub(); await stub.createRecord(makeRecord({ id: 'r1', content: { text: 'a', n: 1 } })); await stub.createRecord(makeRecord({ id: 'r2', content: { text: 'b', n: 2 } })); - const prefix = Array.from({ length: 31 }, (_, i) => `s${i}`).join('.'); + const prefix = Array.from({ length: 32 }, (_, i) => `s${i}`).join('.'); const content = Object.fromEntries( Array.from({ length: 8 }, (_, i) => [`${prefix}.k${i}`, null] as [string, unknown]), ); diff --git a/packages/record-adapter-sqlite/tests/record.test.ts b/packages/record-adapter-sqlite/tests/record.test.ts index 761cbb07..a68bed18 100644 --- a/packages/record-adapter-sqlite/tests/record.test.ts +++ b/packages/record-adapter-sqlite/tests/record.test.ts @@ -1049,21 +1049,27 @@ describe('records — queries', () => { } }); - // The longest legal path has to be executable, not merely accepted. See - // docs/spec/data-model.md § Nested content paths. - test('a path at the segment cap executes; one past it is refused', async () => { - const adapter = await initAdapter(); - const deepest = Array.from({ length: 32 }, (_, i) => `s${i}`).join('.'); + // The longest legal path reaches the deepest value a write accepts, and + // has to execute, not merely be accepted. See docs/spec/data-model.md + // § Nested content paths. + test('a path at the segment cap reaches the deepest value; one past it is refused', async () => { + const adapter = await initAdapter(); + const segments = Array.from({ length: 33 }, (_, i) => `s${i}`); + const deepest = segments.join('.'); + const nested = segments.reduceRight((value, key) => ({ [key]: value }), 'x'); await adapter.createRecord(makeRecord({ id: 'r1', content: { text: 'shallow' } })); + await adapter.createRecord( + makeRecord({ id: 'r2', content: { text: 'deep', ...(nested as object) } }), + ); await expect( adapter.queryRecords({ filter: { content: { [deepest]: 'x' } } }), - ).resolves.toMatchObject({ records: [] }); + ).resolves.toMatchObject({ records: [{ id: 'r2' }] }); await expect( adapter.queryRecords({ filter: { content: { [deepest]: null } } }), ).resolves.toMatchObject({ records: [{ id: 'r1' }] }); await expect( - adapter.queryRecords({ filter: { content: { [`${deepest}.s32`]: 'x' } } }), + adapter.queryRecords({ filter: { content: { [`${deepest}.s33`]: 'x' } } }), ).rejects.toThrow(StackBadRequestError); }); diff --git a/packages/sqlite-shared/src/query.ts b/packages/sqlite-shared/src/query.ts index 609b5d31..d8215ccf 100644 --- a/packages/sqlite-shared/src/query.ts +++ b/packages/sqlite-shared/src/query.ts @@ -66,12 +66,14 @@ const sqlDirection = (query: StackQuery): 'ASC' | 'DESC' => { /** * Normalize a json_each row's value so one array element and one bare * value are walked by the same clause: an array stands for its elements, - * anything else for itself. See docs/spec/data-model.md § Nested content - * paths. + * anything else for itself. A boolean is rewrapped from its type, since + * json_each's `value` reads it as 0 or 1. See docs/spec/data-model.md + * § Nested content paths. */ const spread = (alias: string): string => - `CASE WHEN ${alias}.type = 'array' THEN ${alias}.value ` + - `WHEN ${alias}.type = 'object' THEN json_array(json(${alias}.value)) ` + + `CASE ${alias}.type WHEN 'array' THEN ${alias}.value ` + + `WHEN 'object' THEN json_array(json(${alias}.value)) ` + + `WHEN 'true' THEN '[true]' WHEN 'false' THEN '[false]' ` + `ELSE json_array(${alias}.value) END`; /** @@ -83,39 +85,46 @@ const members = (alias: string): string => `CASE WHEN ${alias}.t = 'object' THEN ${alias}.v ELSE '{}' END`; /** - * The content conditions a statement tests, one row each: the Type it is - * scoped to (`null` for every record), its path segments, its op and value. - * They bind as one parameter, materialized once ahead of the statement so a - * walk reads columns rather than re-parsing the list per record. + * Tuples as a table with `columns`, named in tuple order: the one shape a + * caller-sized part of a query takes, bound as a single JSON parameter so + * the statement never grows with it. See docs/spec/adapters.md § Statement + * size. */ -const CONTENT_CONDITIONS = - "WITH cond AS MATERIALIZED (SELECT json_extract(value, '$[0]') AS scope, " + - "json_extract(value, '$[1]') AS path, json_array_length(value, '$[1]') AS depth, " + - "json_extract(value, '$[2]') AS op, json_extract(value, '$[3]') AS val FROM json_each(?)) "; +const rows = (columns: readonly string[], tuples: unknown[][], params: unknown[]): string => { + params.push(JSON.stringify(tuples)); + const named = columns.map((column, i) => `json_extract(value, '$[${i}]') AS ${column}`); + return `(SELECT ${named.join(', ')} FROM json_each(?))`; +}; + +/** The columns of a `cond` row; see contentRows(). */ +const CONDITION_COLUMNS = ['scope', 'path', 'depth', 'op', 'val', 'kind']; /** - * Whether the walk of the `cond` row `c`'s path through `r` reaches a value - * satisfying `leaf`. The walk is data, a recursive CTE matching one segment - * per step against json_each's `key`, so no key is read as syntax and the - * statement's text is the same for any path. + * A JSON value's type as json_each reports it, with integer and real as one + * number: what an `equals` condition compares besides the value itself, as + * `===` does, so `true` never matches `1` and `'1'` never matches either. */ -const reaches = (leaf: string): string => - "EXISTS (WITH RECURSIVE w(d, v, t) AS (SELECT 0, r.content, 'object' UNION ALL " + - `SELECT w.d + 1, e.value, e.type FROM w JOIN json_each(${members('w')}) m ` + - `JOIN json_each(${spread('m')}) e ` + - "WHERE w.d < c.depth AND m.key = json_extract(c.path, '$[' || w.d || ']')) " + - `SELECT 1 FROM w WHERE w.d = c.depth AND ${leaf})`; +const kindOf = (type: string): string => + `CASE ${type} WHEN 'integer' THEN 'number' WHEN 'real' THEN 'number' ELSE ${type} END`; /** - * Whether the `cond` row `c` holds for `r`. Each op is element-wise, so a - * path reaching a null and a value meets both `'null'` and `'present'`; an - * `'eq'` value never matches an object or array, which json_each exposes as - * JSON text. See docs/spec/data-model.md § Filter. + * Whether the `cond` row `c` holds for `r`. The path is walked as data, a + * recursive CTE matching one segment per step against json_each's `key`, so + * no key is read as syntax and the text is the same for any path; one walk + * gathers what every op asks of the values it reaches. Each op is + * element-wise, so a path reaching a null and a value meets both `isNull` + * and `present`. See docs/spec/data-model.md § Filter. */ -const LEAF = - "CASE c.op WHEN 'eq' THEN w.t NOT IN ('object', 'array') AND w.v = c.val " + - "WHEN 'present' THEN w.v IS NOT NULL ELSE w.t = 'null' END"; -const HOLDS = `(${reaches(LEAF)} OR (c.op = 'null' AND NOT ${reaches('w.v IS NOT NULL')}))`; +const HOLDS = + "(WITH RECURSIVE w(d, v, t) AS (SELECT 0, r.content, 'object' UNION ALL " + + `SELECT w.d + 1, e.value, e.type FROM w JOIN json_each(${members('w')}) m ` + + `JOIN json_each(${spread('m')}) e ` + + "WHERE w.d < c.depth AND m.key = json_extract(c.path, '$[' || w.d || ']')) " + + 'SELECT CASE c.op ' + + `WHEN 'equals' THEN IFNULL(MAX(${kindOf('w.t')} = c.kind AND w.v = c.val), 0) ` + + "WHEN 'present' THEN IFNULL(MAX(w.t != 'null'), 0) " + + "ELSE IFNULL(MAX(w.t = 'null'), 0) OR NOT IFNULL(MAX(w.t != 'null'), 0) END " + + 'FROM w WHERE w.d = c.depth)'; /** The join alias the content index is read through. */ const SORT_ALIAS = 'cs'; @@ -142,56 +151,70 @@ const contentCursor = (cursor: DecodedCursor, field: string) => { return cursor; }; +/** A filter value's kindOf(), or null for one no stored scalar can equal. */ +const valueKind = (value: unknown): string | null => { + if (typeof value === 'boolean') return `${value}`; + if (typeof value === 'number') return 'number'; + return typeof value === 'string' ? 'text' : null; +}; + /** * A `content` and `contentPresent` predicate as `cond` rows scoped to - * `scope`. A `null` value means "no value at the path, or a value that is - * null" (docs/spec/data-model.md § Filter). Presence is the question an - * exact-match value cannot ask. + * `scope`: path segments, their count, the op, and for `equals` the value + * and its kindOf() (none for an object or array, which `===` never + * matches). A `null` value means "no value at the path, or a value that is + * null" (docs/spec/data-model.md § Filter). */ -const contentRows = (scope: string | null, predicate: ContentArm): unknown[][] => [ - ...Object.entries(predicate.content ?? {}).map(([key, value]) => - // undefined is the same absence null names — JSON drops it on the way - // over the wire, so treating it as a value would make an in-process - // query and its wire equivalent disagree. - value === null || value === undefined - ? [scope, parseContentFilterKey(key), 'null'] - : [scope, parseContentFilterKey(key), 'eq', value], - ), - ...(predicate.contentPresent ?? []).map((key) => [scope, parseContentFilterKey(key), 'present']), -]; +const contentRows = (scope: string | null, predicate: ContentArm): unknown[][] => { + const row = (key: string, op: string, value: unknown = null) => { + const path = parseContentFilterKey(key); + return [scope, path, path.length, op, value, valueKind(value)]; + }; + return [ + ...Object.entries(predicate.content ?? {}).map(([key, value]) => + // undefined is the same absence null names — JSON drops it on the way + // over the wire, so treating it as a value would make an in-process + // query and its wire equivalent disagree. + value === null || value === undefined ? row(key, 'isNull') : row(key, 'equals', value), + ), + ...(predicate.contentPresent ?? []).map((key) => row(key, 'present')), + ]; +}; /** - * Values as the condition matching any of them. A list binds as one JSON - * parameter (docs/spec/adapters.md § Statement size); one value binds as - * itself, so an index on the column still yields rows in its order. An - * empty list matches nothing. + * Values as the condition matching any of them: rows() of one column, or + * one value bound as itself, so an index on the column still yields rows in + * its order. An empty list matches nothing. */ const inList = (column: string, values: readonly string[], params: unknown[]): string => { if (values.length === 1) { params.push(values[0]); return `${column} = ?`; } - params.push(JSON.stringify(values)); - return `${column} IN (SELECT value FROM json_each(?))`; + const tuples = values.map((value) => [value]); + const table = rows(['value'], tuples, params); + return `${column} IN (SELECT value FROM ${table})`; }; /** * Whether the record carries every element of one of `sets`, matched on the * permissions table's key — the identity editAccess() removes by — through - * idx_permissions_element. The sets travel as one JSON parameter, each row - * naming its set and that set's size; an empty set matches nothing. + * idx_permissions_element. Each element is a row naming its set and that + * set's size; an empty set matches nothing. */ const carriesPermissions = (sets: AuthorityAssociation[][], params: unknown[]): string => { - const rows = sets.flatMap((set, i) => - set.map((p) => [i, set.length, p.kind, p.label, ...permissionKeyColumns(p)]), + const elements = rows( + ['set_index', 'set_size', 'kind', 'label', 'grantee_kind', 'grantee_id', 'role'], + sets.flatMap((set, i) => + set.map((p) => [i, set.length, p.kind, p.label, ...permissionKeyColumns(p)]), + ), + params, ); - params.push(JSON.stringify(rows)); - const at = (i: number) => `json_extract(e.value, '$[${i}]')`; return ( - `r.id IN (SELECT p.record_id FROM json_each(?) e JOIN permissions p ` + - `ON p.kind = ${at(2)} AND p.label = ${at(3)} AND p.grantee_kind = ${at(4)} ` + - `AND p.grantee_id = ${at(5)} AND p.role = ${at(6)} ` + - `GROUP BY p.record_id, ${at(0)} HAVING COUNT(*) = MAX(${at(1)}))` + `r.id IN (SELECT p.record_id FROM ${elements} e JOIN permissions p ` + + 'ON p.kind = e.kind AND p.label = e.label AND p.grantee_kind = e.grantee_kind ' + + 'AND p.grantee_id = e.grantee_id AND p.role = e.role ' + + 'GROUP BY p.record_id, e.set_index HAVING COUNT(*) = MAX(e.set_size))' ); }; @@ -430,7 +453,9 @@ const presentCursorCondition = ( /** * One statement reading `r` from `from`, whose own parameters bind ahead of - * the conditions', with the `cond` table first when it tests content. + * the conditions'. When it tests content, the `cond` rows come first, + * materialized once so a walk reads columns rather than re-parsing JSON + * per record. */ const statement = ( from: string, @@ -440,9 +465,11 @@ const statement = ( ): QueryStatement => { const select = `SELECT r.* FROM ${from} WHERE ${conditions.join(' AND ')}${tail}`; if (content.length === 0) return { sql: select, params: [...fromParams, ...params] }; + const head: unknown[] = []; + const cond = rows(CONDITION_COLUMNS, content, head); return { - sql: CONTENT_CONDITIONS + select, - params: [JSON.stringify(content), ...fromParams, ...params], + sql: `WITH cond AS MATERIALIZED ${cond} ${select}`, + params: [...head, ...fromParams, ...params], }; }; @@ -515,9 +542,8 @@ export const buildQueryPlan = (query: StackQuery, reach?: Reach): QueryStatement /** * A sort naming its value per Type, read through a derived table in place * of the index: each Type's own index rows, and its constant in the index's - * columns, so one cursor condition and one ORDER BY serve both. Each kind - * of entry is a list of rows, bound as one parameter like any other list. - * See docs/spec/data-model.md § Sorting by a content field. + * columns, so one cursor condition and one ORDER BY serve both. Fields and + * constants are each rows(). See docs/spec/data-model.md § Sorting by a content field. */ const byTypePlan = ( query: StackQuery, @@ -539,34 +565,33 @@ const byTypePlan = ( if (entry) constants.push([typeId, entry]); } const pages: QueryStatement[] = []; - const at = (i: number) => `json_extract(e.value, '$[${i}]')`; if (cursor?.kind !== 'absent' && fields.length + constants.length > 0) { const sources: string[] = []; const sourceParams: unknown[] = []; if (fields.length > 0) { + const byField = rows(['type_id', 'field'], fields, sourceParams); sources.push( - 'SELECT c.record_id AS record_id, c.value_rank AS value_rank, c.num_value AS num_value, ' + - 'c.text_key AS text_key, c.text_value AS text_value ' + - `FROM json_each(?) e JOIN content_index c ON c.field = ${at(1)} ` + - `JOIN records t ON t.id = c.record_id AND t.type_id = ${at(0)}`, + 'SELECT ci.record_id AS record_id, ci.value_rank AS value_rank, ' + + 'ci.num_value AS num_value, ci.text_key AS text_key, ci.text_value AS text_value ' + + `FROM ${byField} f JOIN content_index ci ON ci.field = f.field ` + + 'JOIN records t ON t.id = ci.record_id AND t.type_id = f.type_id', ); - sourceParams.push(JSON.stringify(fields)); } if (constants.length > 0) { - sources.push( - `SELECT t.id AS record_id, ${at(1)} AS value_rank, ${at(2)} AS num_value, ` + - `${at(3)} AS text_key, ${at(4)} AS text_value ` + - `FROM json_each(?) e JOIN records t ON t.type_id = ${at(0)}`, - ); - sourceParams.push( - JSON.stringify( - constants.map(([typeId, entry]) => - entry.kind === 'num' - ? [typeId, 0, entry.num, null, null] - : [typeId, 1, null, entry.key, entry.text], - ), + const byConstant = rows( + ['type_id', 'value_rank', 'num_value', 'text_key', 'text_value'], + constants.map(([typeId, entry]) => + entry.kind === 'num' + ? [typeId, 0, entry.num, null, null] + : [typeId, 1, null, entry.key, entry.text], ), + sourceParams, + ); + sources.push( + 'SELECT t.id AS record_id, k.value_rank AS value_rank, k.num_value AS num_value, ' + + 'k.text_key AS text_key, k.text_value AS text_value ' + + `FROM ${byConstant} k JOIN records t ON t.type_id = k.type_id`, ); } const where = recordConditions(query, reach); @@ -586,12 +611,12 @@ const byTypePlan = ( { const where = recordConditions(query, reach); if (fields.length > 0) { + const byField = rows(['type_id', 'field'], fields, where.params); where.conditions.push( - `NOT EXISTS (SELECT 1 FROM json_each(?) e JOIN content_index ${SORT_ALIAS} ` + - `ON ${SORT_ALIAS}.record_id = r.id AND ${SORT_ALIAS}.field = ${at(1)} ` + - `WHERE ${at(0)} = r.type_id)`, + `NOT EXISTS (SELECT 1 FROM ${byField} f JOIN content_index ${SORT_ALIAS} ` + + `ON ${SORT_ALIAS}.record_id = r.id AND ${SORT_ALIAS}.field = f.field ` + + 'WHERE f.type_id = r.type_id)', ); - where.params.push(JSON.stringify(fields)); } if (constants.length > 0) { const typeIds = constants.map(([typeId]) => typeId); From 391e6eb7ad61d6a1c3a4a6eaead5c5ee485e4a83 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 21 Oct 2026 17:15:36 +0000 Subject: [PATCH 4/9] feat(core,wire-types): a per-Type cap states a real limit or none Once a per-Type key cost a SQLite statement no text or parameter of its own, the 64 local adapters declared had no cause left. The caps are now number | null: null states no limit, 0 still declines, and an absent or malformed cap still reads as 0. Local adapters declare null; a cap is for a server bounding its own cost. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Mf3p8BVtrQJLZx1JFCHchC --- .changeset/fixed-statement-size.md | 5 +++- docs/spec/adapters.md | 8 +++--- docs/spec/data-model.md | 4 +-- packages/adapter-conformance/src/record.ts | 9 ++++--- packages/core/src/collection.ts | 5 ++-- packages/core/src/query-validation.ts | 4 +-- packages/core/src/testing.ts | 4 +-- packages/core/src/types/capabilities.ts | 17 +++++++----- .../core/tests/query-capabilities.test.ts | 14 ++++++++++ .../tests/record.test.ts | 22 +++++++-------- packages/sqlite-shared/src/record-adapter.ts | 7 +++-- packages/wire-types/src/discovery.ts | 27 +++++++++++++------ packages/wire-types/tests/discovery.test.ts | 24 ++++++++++++----- 13 files changed, 97 insertions(+), 53 deletions(-) diff --git a/.changeset/fixed-statement-size.md b/.changeset/fixed-statement-size.md index 874e86e0..c7e665c1 100644 --- a/.changeset/fixed-statement-size.md +++ b/.changeset/fixed-statement-size.md @@ -3,8 +3,11 @@ '@haverstack/record-adapter-sqlite': minor '@haverstack/adapter-conformance': minor '@haverstack/core': minor +'@haverstack/wire-types': minor --- -A SQLite statement's text and bound parameters no longer grow with a query's size. Every caller-sized part of a query binds as one table of rows: lists, permission sets, `sort.contentFieldByType`'s entries, and the content conditions — every `filter.content` and `filter.contentPresent` key and every `filter.contentByType` arm — whose paths one recursive walk per condition resolves. Durable Object SQLite now answers queries at the declared `contentByType` and `contentFieldByType` caps, and content filters of any size, instead of failing with `too many SQL variables` or `statement too long`. +A SQLite statement's text and bound parameters no longer grow with a query's size. Every caller-sized part of a query binds as one table of rows: lists, permission sets, `sort.contentFieldByType`'s entries, and the content conditions — every `filter.content` and `filter.contentPresent` key and every `filter.contentByType` arm — whose paths one recursive walk per condition resolves. Durable Object SQLite no longer fails a query with `too many SQL variables` or `statement too long`, however many per-Type keys or content conditions it carries. + +The per-Type caps `filter.contentByType` and `sort.contentFieldByType` are now `number | null`, `null` stating no limit. `MemoryAdapter` and the SQLite adapters declare `null`, since nothing bounds the count locally; a cap is for a server bounding its own cost. `normalizeCapabilities()` reads a stated `null` as no limit, and an absent or malformed cap as `0`, as before. A content filter value now matches only a value of its own JSON type in the SQLite adapters, as `===` does: `true` no longer matches `1`. A content filter path may be 33 segments, the depth that reaches the deepest value a write accepts. The conformance suite holds every adapter to both, and to a multi-valued path satisfying both `contentPresent` and a `null` filter. diff --git a/docs/spec/adapters.md b/docs/spec/adapters.md index 9e9ad58d..aef95843 100644 --- a/docs/spec/adapters.md +++ b/docs/spec/adapters.md @@ -192,13 +192,13 @@ type StackCapabilities = { filter: { content: 'none' | 'field' | 'path'; // how far a content filter key may reach contentPresent: boolean; // filter.contentPresent is honored - contentByType: number; // most arms filter.contentByType may hold; 0 = not honored + contentByType: number | null; // most arms filter.contentByType may hold; null = no limit, 0 = not honored search: boolean; // filter.search is honored }; sort: { fields: ('createdAt' | 'updatedAt' | 'version')[]; // native columns this adapter can order by contentField: boolean; // sort.contentField is honored - contentFieldByType: number; // most keys sort.contentFieldByType may hold; 0 = not honored + contentFieldByType: number | null; // most keys sort.contentFieldByType may hold; null = no limit, 0 = not honored }; limits: { attachmentBytes: number | null; // upload size ceiling, or null = client can't pre-check @@ -219,7 +219,7 @@ One type, `StackCapabilities`, names both sides: an adapter declares it as `capa **`filter.contentPresent` sits beside the ladder rather than on it.** It gates `filter.contentPresent`, and is required `true` for local adapters on the same reasoning: asking whether a path holds a value is the same walk with a weaker predicate. It is not a rung because a server promising to match a content value has not thereby promised to answer whether one is there at all. It is meaningful only above `'none'` — presence travels in the `POST /records/query` body, which a server reaching no content does not expose — and a presence path is itself gated by the rung, so a multi-segment one needs `'path'`. -**`filter.contentByType` and `sort.contentFieldByType` are caps, not flags.** Each is the most keys the [per-Type shape](./data-model.md#content-by-type) may hold, and `0` declines it. A local adapter must honor both, on the reasoning that makes `'path'` required: an arm is the content filter it already runs, scoped to one `typeId`. The first-party ones declare `64`, far above any family's version count; a per-Type key costs a SQLite adapter no statement text or parameter of its own (see [Statement size](#statement-size)), so the cap is the same on every engine. A server may declare any cap, `0` included, and a collection then narrows the versions it cannot reach rather than failing the query — see [Data model § Content queries across versions](./data-model.md#content-queries-across-versions). Both are meaningful only above `filter.content: 'none'`, since they travel in the `POST /records/query` body. +**`filter.contentByType` and `sort.contentFieldByType` are caps, not flags.** Each is the most keys the [per-Type shape](./data-model.md#content-by-type) may hold: `null` states no limit, and `0` declines the shape. **A cap states a real limit or none at all.** A local adapter declares `null` for both, on the reasoning that makes `'path'` required: an arm is the content filter it already runs, scoped to one `typeId`, and costs a SQLite adapter no statement text or parameter of its own (see [Statement size](#statement-size)), so nothing bounds the count. A cap is for a server, which bounds what a request may cost it — see [Wire format § Bounding query cost](./wire-format.md#bounding-query-cost). A server may declare any cap, `0` and `null` included, and a collection narrows the versions past it rather than failing the query — see [Data model § Content queries across versions](./data-model.md#content-queries-across-versions). Both are meaningful only above `filter.content: 'none'`, since they travel in the `POST /records/query` body. **`filter.search` has no local-required rule.** A full-text index is a different mechanism from field matching, and neither implies the other, so a local adapter may legitimately decline it (see the JSON adapter note below). @@ -227,7 +227,7 @@ One type, `StackCapabilities`, names both sides: an adapter declares it as `capa **Absent, malformed or unrecognized reads as the least capable value it could stand for.** One rule, for every entry: `'none'` for the reach, `false` for a flag, `0` for a cap, nothing for `sort.fields`, `null` for a limit. A discovery response that omits a group, omits `capabilities` entirely, or names a reach this client has never heard of describes a server this client will not send that query to — silence is never a claim, in either direction, and an unrecognized rung places nowhere on a ladder so it can only be read as its bottom. `normalizeCapabilities()` in `@haverstack/wire-types` applies the rule, so a client reads a foreign discovery response through one function rather than defaulting keys at each use. -Two things this does not refuse. A query naming no sort asks for the server's own default order rather than a stated one, so it claims nothing that has to be declared and runs against an empty `sort.fields`. And a `null` limit is permissive by design: it says this client cannot pre-check, not that nothing is enforced — the server's own ceiling still answers with a `413`. +Two things this does not refuse. A query naming no sort asks for the server's own default order rather than a stated one, so it claims nothing that has to be declared and runs against an empty `sort.fields`. And `null` is permissive where a server states it: a `null` limit says this client cannot pre-check, not that nothing is enforced — the server's own ceiling still answers with a `413` — and a `null` cap says the server takes any number of keys. Only a stated `null` reads this way; an absent cap is still `0`. **Per-adapter notes:** diff --git a/docs/spec/data-model.md b/docs/spec/data-model.md index 50558d14..12633edc 100644 --- a/docs/spec/data-model.md +++ b/docs/spec/data-model.md @@ -655,7 +655,7 @@ By default, `query()` (like `get()`) returns Records exactly as stored — see [ - **Every arm is pinned to a `typeId`**, and a filter carrying `contentByType` names at least one. An empty map is refused with `StackBadRequestError` (400), so the key cannot encode to a filter that matches everything. - **An arm is flat**: `content` and `contentPresent`, nothing else, and no nesting. An arm naming neither matches every Record at its `typeId`, which is what `typeId` alone matches. - **It never sits beside a top-level `content` or `contentPresent`**: a filter with both is refused with 400, since which one applies to a Record would be ambiguous. Every other key applies to every arm, and a `typeId` beside it intersects with its keys. -- **It is bounded** by the adapter's `filter.contentByType`, the most arms it takes; a filter with more is refused with 400 naming that capability (see [Capability-gated filters](#capability-gated-filters)). +- **It is bounded** by the adapter's `filter.contentByType`, the most arms it takes, where it states a limit; a filter with more is refused with 400 naming that capability (see [Capability-gated filters](#capability-gated-filters)). It discloses nothing new: each arm is a query the caller could already run, its `typeId` with its content filter, and the permission check applies per Record as for any query. Types partition Records, so no Record is matched by two arms, and the cost of _n_ arms stays within what _n_ queries would cost — what [Wire format § Bounding query cost](./wire-format.md#bounding-query-cost) already leaves to the server. **It is not a first step toward general OR.** A disjunction across Types is the only one a migration chain needs, and match-any forms stay refused for the reason [above](#filter): a filter that can encode to nothing can silently widen a query when it crosses the wire. @@ -754,7 +754,7 @@ Locale-correct ordering is out of reach for a stored key regardless of effort, b **`contentPresent` needs `filter.contentPresent` on top of a content reach**, a separate flag on the same reasoning the reach ladder rests on: a server promising to match a content value has not thereby promised to answer whether one is there, and reading it as such would hand a client the unfiltered superset that ignoring the filter produces. A multi-segment presence path needs the `'path'` rung too, exactly as a filter key does. -**`contentByType` and `contentFieldByType` need their own capabilities**, `filter.contentByType` and `sort.contentFieldByType`: each is the most keys the adapter takes, and `0` declines the shape. An arm's keys are gated as the same keys at the top of a filter are, so a multi-segment path in an arm needs `'path'`, and an arm's `contentPresent` needs `filter.contentPresent`. +**`contentByType` and `contentFieldByType` need their own capabilities**, `filter.contentByType` and `sort.contentFieldByType`: each is the most keys the adapter takes, `null` for no limit, and `0` declines the shape. An arm's keys are gated as the same keys at the top of a filter are, so a multi-segment path in an arm needs `'path'`, and an arm's `contentPresent` needs `filter.contentPresent`. **A sort is gated the same way.** `sort.contentField` needs the adapter's `sort.contentField`, and a native `sort.field` must appear in its `sort.fields`; a sort an adapter hasn't declared throws `StackBadRequestError` rather than being answered in some other order — which a caller reading one bounded page has no way to notice. `sort.contentField` and `filter.content` are independent: a server may order by a content field without offering to filter on one, or the reverse. diff --git a/packages/adapter-conformance/src/record.ts b/packages/adapter-conformance/src/record.ts index 68c868ed..26c83cb5 100644 --- a/packages/adapter-conformance/src/record.ts +++ b/packages/adapter-conformance/src/record.ts @@ -52,6 +52,9 @@ export interface RecordAdapterConformanceOptions { capabilities: StackCapabilities; } +/** Whether a per-Type cap admits `keys` keys; `null` admits any number. */ +const takes = (cap: number | null, keys: number): boolean => cap === null || cap >= keys; + export function runRecordAdapterConformance(options: RecordAdapterConformanceOptions): void { const { name, open, close, capabilities } = options; @@ -89,7 +92,7 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt capabilities.filter.contentByType, capabilities.sort.contentFieldByType, ]) { - expect(Number.isInteger(cap) && cap >= 0).toBe(true); + expect(cap === null || (Number.isInteger(cap) && cap >= 0)).toBe(true); } }); @@ -1363,7 +1366,7 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt // ----------------------------------------------------------------- // Content by type — capability-gated on filter.contentByType // ----------------------------------------------------------------- - if (capabilities.filter.contentByType >= 2) { + if (takes(capabilities.filter.contentByType, 2)) { describe('content by type', () => { const OTHER = 'org.haverstack.conformance/other@1'; const THIRD = 'org.haverstack.conformance/third@1'; @@ -1442,7 +1445,7 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt // ----------------------------------------------------------------- // Sort by content field per type — gated on sort.contentFieldByType // ----------------------------------------------------------------- - if (capabilities.sort.contentFieldByType >= 3) { + if (takes(capabilities.sort.contentFieldByType, 3)) { describe('sort by content field per type', () => { const OTHER = 'org.haverstack.conformance/other@1'; const THIRD = 'org.haverstack.conformance/third@1'; diff --git a/packages/core/src/collection.ts b/packages/core/src/collection.ts index 5b1489a7..60ba4f1b 100644 --- a/packages/core/src/collection.ts +++ b/packages/core/src/collection.ts @@ -499,8 +499,9 @@ export const createCollection = ( ? Infinity : capabilities.filter.content === 'none' ? 0 - : capabilities.filter.contentByType; - const sortCap = sortField === undefined ? Infinity : capabilities.sort.contentFieldByType; + : (capabilities.filter.contentByType ?? Infinity); + const sortCap = + sortField === undefined ? Infinity : (capabilities.sort.contentFieldByType ?? Infinity); const cap = Math.min(filterCap, sortCap); const sortKind = (() => { const def = sortField === undefined ? undefined : handle.schema[sortField]; diff --git a/packages/core/src/query-validation.ts b/packages/core/src/query-validation.ts index 1a9fa3b3..e5d57a55 100644 --- a/packages/core/src/query-validation.ts +++ b/packages/core/src/query-validation.ts @@ -72,7 +72,7 @@ export function assertQueryCapabilities( 'filter.content', ); } - if (arms.length > contentByType) { + if (contentByType !== null && arms.length > contentByType) { throw new StackBadRequestError( contentByType === 0 ? 'Query uses filter.contentByType, but this adapter does not declare the ' + @@ -329,7 +329,7 @@ export function assertSortCapability( if (sort.contentFieldByType !== undefined) { const keys = Object.keys(sort.contentFieldByType).length; const cap = capabilities.sort.contentFieldByType; - if (keys > cap) { + if (cap !== null && keys > cap) { throw new StackBadRequestError( cap === 0 ? 'Query uses sort.contentFieldByType, but this adapter does not declare the ' + diff --git a/packages/core/src/testing.ts b/packages/core/src/testing.ts index b0daefa5..3658b723 100644 --- a/packages/core/src/testing.ts +++ b/packages/core/src/testing.ts @@ -83,13 +83,13 @@ export class MemoryAdapter implements StackAdapter { filter: { content: 'path', contentPresent: true, - contentByType: 64, + contentByType: null, search: false, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 64, + contentFieldByType: null, }, limits: { attachmentBytes: null, diff --git a/packages/core/src/types/capabilities.ts b/packages/core/src/types/capabilities.ts index 407c894d..d109f3aa 100644 --- a/packages/core/src/types/capabilities.ts +++ b/packages/core/src/types/capabilities.ts @@ -40,12 +40,12 @@ export type StackCapabilities = { */ contentPresent: boolean; /** - * The most arms `filter.contentByType` may hold; `0` declines it. A - * collection narrows the versions it cannot reach rather than failing. - * Local adapters must honor it. See docs/spec/adapters.md § Adapter - * capabilities. + * The most arms `filter.contentByType` may hold: `null` for no limit, + * `0` to decline it. A collection narrows the versions past a limit + * rather than failing. Local adapters declare `null`. See + * docs/spec/adapters.md § Adapter capabilities. */ - contentByType: number; + contentByType: number | null; /** * Whether `filter.search` is honored. Not a rung on the ladder: a * full-text index is a different mechanism from field matching, and @@ -62,8 +62,11 @@ export type StackCapabilities = { * scalar. Independent of `filter.content`. */ contentField: boolean; - /** The most keys `sort.contentFieldByType` may hold; `0` declines it. */ - contentFieldByType: number; + /** + * The most keys `sort.contentFieldByType` may hold: `null` for no limit, + * `0` to decline it, as `filter.contentByType`. + */ + contentFieldByType: number | null; }; /** * Ceilings a client can check before spending a request; the server's diff --git a/packages/core/tests/query-capabilities.test.ts b/packages/core/tests/query-capabilities.test.ts index 69cfea36..c04f1cac 100644 --- a/packages/core/tests/query-capabilities.test.ts +++ b/packages/core/tests/query-capabilities.test.ts @@ -161,6 +161,20 @@ describe('a refused query names the capability it needed', () => { ).not.toThrow(); }); + test('a null per-Type cap admits any number of keys', () => { + const typeIds = Array.from({ length: 500 }, (_, i) => `com.example/x@${i + 1}`); + const filter = { contentByType: Object.fromEntries(typeIds.map((id) => [id, {}])) }; + const sort = { + contentFieldByType: Object.fromEntries(typeIds.map((id) => [id, { field: 'a' }])), + }; + const unbounded = without({ + filter: { contentByType: null }, + sort: { contentFieldByType: null }, + }); + expect(() => assertQueryCapabilities(filter, unbounded)).not.toThrow(); + expect(() => assertSortCapability(sort, unbounded)).not.toThrow(); + }); + test('sort.fields', () => { expect( refusalFor(() => diff --git a/packages/record-adapter-do-sqlite/tests/record.test.ts b/packages/record-adapter-do-sqlite/tests/record.test.ts index 83b0ef83..d527a918 100644 --- a/packages/record-adapter-do-sqlite/tests/record.test.ts +++ b/packages/record-adapter-do-sqlite/tests/record.test.ts @@ -113,13 +113,13 @@ describe('construction', () => { filter: { content: 'path', contentPresent: true, - contentByType: 64, + contentByType: null, search: true, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 64, + contentFieldByType: null, }, limits: { attachmentBytes: null, @@ -598,19 +598,20 @@ test('a tag filter far past the parameter cap', async () => { expect(result.records.map((r) => r.id)).toEqual([tagged.id]); }); -// The caps are the shared SQLite declaration, so they have to hold on the -// engine with the tightest statement limits, at full size. -describe('per-Type keys at the declared cap', () => { - const typeIds = (cap: number) => - Array.from({ length: cap }, (_, i) => (i === 0 ? NOTE_TYPE_V1 : `com.example/t${i}@1`)); +// The shared SQLite declaration sets no per-Type cap, so it has to hold on +// the engine with the tightest statement limits, far past any count a +// statement could have afforded. See docs/spec/adapters.md § Statement size. +describe('per-Type keys with no declared cap', () => { + const KEYS = 200; + const typeIds = (count: number) => + Array.from({ length: count }, (_, i) => (i === 0 ? NOTE_TYPE_V1 : `com.example/t${i}@1`)); test('filter.contentByType', async () => { const stub = getStub(); - const cap = (await stub.getCapabilities()).filter.contentByType; await stub.createRecord(makeRecord({ id: 'r1', content: { text: 'a', meta: { n: 1 } } })); await stub.createRecord(makeRecord({ id: 'r2', content: { text: 'b', meta: { n: 2 } } })); const contentByType = Object.fromEntries( - typeIds(cap).map((typeId) => [ + typeIds(KEYS).map((typeId) => [ typeId, { content: { 'meta.n': 1, 'meta.missing': null }, contentPresent: ['text'] }, ]), @@ -621,7 +622,6 @@ describe('per-Type keys at the declared cap', () => { test('sort.contentFieldByType', async () => { const stub = getStub(); - const cap = (await stub.getCapabilities()).sort.contentFieldByType; const other = 'com.example/t1@1'; await stub.saveType({ id: NOTE_TYPE_V1, @@ -637,7 +637,7 @@ describe('per-Type keys at the declared cap', () => { await stub.createRecord(makeRecord({ id: 'r3', typeId: other, content: { text: 'c' } })); await stub.createRecord(makeRecord({ id: 'r4', content: { text: 'd' } })); const contentFieldByType = Object.fromEntries( - typeIds(cap).map((typeId, i) => [typeId, i % 2 === 0 ? { field: 'n' } : { value: 2 }]), + typeIds(KEYS).map((typeId, i) => [typeId, i % 2 === 0 ? { field: 'n' } : { value: 2 }]), ); const result = await stub.queryRecords({ sort: { contentFieldByType, direction: 'asc' }, diff --git a/packages/sqlite-shared/src/record-adapter.ts b/packages/sqlite-shared/src/record-adapter.ts index f051a560..747a8207 100644 --- a/packages/sqlite-shared/src/record-adapter.ts +++ b/packages/sqlite-shared/src/record-adapter.ts @@ -54,15 +54,14 @@ export const SQLITE_RECORD_CAPABILITIES: StackCapabilities = { content: 'path', contentPresent: true, // A per-Type key costs no statement text or parameter of its own, so - // every engine honors the same cap; see docs/spec/adapters.md § Statement - // size. - contentByType: 64, + // nothing bounds it; see docs/spec/adapters.md § Statement size. + contentByType: null, search: true, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 64, + contentFieldByType: null, }, limits: { attachmentBytes: null, diff --git a/packages/wire-types/src/discovery.ts b/packages/wire-types/src/discovery.ts index d1fc4822..f982f145 100644 --- a/packages/wire-types/src/discovery.ts +++ b/packages/wire-types/src/discovery.ts @@ -39,8 +39,13 @@ export type DiscoveryResponse = { * checks rather than casts. See docs/spec/wire-format.md § Discovery. */ export type DiscoveryCapabilities = { - filter?: { content?: string; contentPresent?: boolean; contentByType?: number; search?: boolean }; - sort?: { fields?: string[]; contentField?: boolean; contentFieldByType?: number }; + filter?: { + content?: string; + contentPresent?: boolean; + contentByType?: number | null; + search?: boolean; + }; + sort?: { fields?: string[]; contentField?: boolean; contentFieldByType?: number | null }; limits?: { attachmentBytes?: number | null; contentBytes?: number | null }; }; @@ -50,8 +55,9 @@ const SORT_FIELDS: ReadonlySet = new Set(NATIVE_SORT_FIELDS); /** * Resolves anything absent, malformed or unrecognized to the least capable * value it could stand for — one rule, applied once, rather than a default - * per key at each call site. A `null` limit is the one permissive reading, - * since the server's own ceiling still answers 413. See + * per key at each call site. `null` is the permissive reading, and only + * where it is stated: a limit the client cannot pre-check, whose server + * still answers 413, or a cap with no limit. See * docs/spec/adapters.md § Adapter capabilities. */ export function normalizeCapabilities( @@ -65,7 +71,7 @@ export function normalizeCapabilities( : 'none'; // The per-Type shapes travel in the POST /records/query body, which a // server reaching no content does not serve. - const cap = (value: unknown): number => (reach === 'none' ? 0 : capOrZero(value)); + const cap = (value: unknown): number | null => (reach === 'none' ? 0 : capOrZero(value)); return { filter: { content: reach, @@ -85,9 +91,14 @@ export function normalizeCapabilities( }; } -/** A cap on keys, or 0 — declined — for anything but a non-negative integer. */ -const capOrZero = (value: unknown): number => - typeof value === 'number' && Number.isSafeInteger(value) && value > 0 ? value : 0; +/** + * A cap on keys, `null` — no limit — where the server says so, or 0 — + * declined — for anything else that isn't a non-negative integer. + */ +const capOrZero = (value: unknown): number | null => { + if (value === null) return null; + return typeof value === 'number' && Number.isSafeInteger(value) && value > 0 ? value : 0; +}; /** A ceiling this client can compare against, or null for "cannot pre-check". */ const finiteOrNull = (value: unknown): number | null => diff --git a/packages/wire-types/tests/discovery.test.ts b/packages/wire-types/tests/discovery.test.ts index 54583d2d..c3c47742 100644 --- a/packages/wire-types/tests/discovery.test.ts +++ b/packages/wire-types/tests/discovery.test.ts @@ -82,8 +82,8 @@ describe('normalizeCapabilities', () => { ).toEqual([...NATIVE_SORT_FIELDS]); }); - it('reads a per-Type cap that is not a positive integer as 0', () => { - for (const cap of [-1, 2.5, '16', true, null, Number.POSITIVE_INFINITY]) { + it('reads a per-Type cap that is not a positive integer or null as 0', () => { + for (const cap of [-1, 2.5, '16', true, undefined, Number.POSITIVE_INFINITY]) { const read = normalizeCapabilities({ filter: { content: 'path', contentByType: cap as number }, sort: { contentFieldByType: cap as number }, @@ -92,14 +92,24 @@ describe('normalizeCapabilities', () => { } }); + it('reads a stated null per-Type cap as no limit', () => { + const read = normalizeCapabilities({ + filter: { content: 'path', contentByType: null }, + sort: { contentFieldByType: null }, + }); + expect([read.filter.contentByType, read.sort.contentFieldByType]).toEqual([null, null]); + }); + // Both travel in the POST /records/query body, which a server reaching no // content does not serve, so a cap it declares is one it cannot honor. it('reads a per-Type cap as 0 beside a reach of none', () => { - const read = normalizeCapabilities({ - filter: { content: 'none', contentByType: 16 }, - sort: { contentFieldByType: 16 }, - }); - expect([read.filter.contentByType, read.sort.contentFieldByType]).toEqual([0, 0]); + for (const cap of [16, null]) { + const read = normalizeCapabilities({ + filter: { content: 'none', contentByType: cap }, + sort: { contentFieldByType: cap }, + }); + expect([read.filter.contentByType, read.sort.contentFieldByType]).toEqual([0, 0]); + } }); it('reads an absent capabilities object as declaring nothing', () => { From 82d24bb30ca98c07a8b12d5ce5eef3828cd436ed Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 21 Oct 2026 17:44:01 +0000 Subject: [PATCH 5/9] refactor(core,wire-types): one reading of a per-Type cap admitsKeys(cap, keys) is now the only place a cap is read; query validation, collection narrowing and the conformance suite all call it rather than spelling null-means-no-limit three ways. readCap replaces capOrZero, whose name stopped describing it, and a discovery fixture pins a stated null cap on the wire. Also: the data-model sentence on object and array values now says what matchesContent() does, normalizeCapabilities() says which null readings need stating, and contentByType's type list says why it stays. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Mf3p8BVtrQJLZx1JFCHchC --- .changeset/fixed-statement-size.md | 3 ++- docs/spec/data-model.md | 2 +- packages/adapter-conformance/src/record.ts | 9 +++---- .../conformance-fixtures/src/discovery.ts | 24 +++++++++++++++++++ packages/core/src/adapter/index.ts | 1 + packages/core/src/collection.ts | 16 ++++++------- packages/core/src/query-validation.ts | 13 ++++++++-- packages/sqlite-shared/src/query.ts | 2 ++ packages/wire-types/src/discovery.ts | 21 ++++++++-------- 9 files changed, 63 insertions(+), 28 deletions(-) diff --git a/.changeset/fixed-statement-size.md b/.changeset/fixed-statement-size.md index c7e665c1..8185535a 100644 --- a/.changeset/fixed-statement-size.md +++ b/.changeset/fixed-statement-size.md @@ -4,10 +4,11 @@ '@haverstack/adapter-conformance': minor '@haverstack/core': minor '@haverstack/wire-types': minor +'@haverstack/conformance-fixtures': minor --- A SQLite statement's text and bound parameters no longer grow with a query's size. Every caller-sized part of a query binds as one table of rows: lists, permission sets, `sort.contentFieldByType`'s entries, and the content conditions — every `filter.content` and `filter.contentPresent` key and every `filter.contentByType` arm — whose paths one recursive walk per condition resolves. Durable Object SQLite no longer fails a query with `too many SQL variables` or `statement too long`, however many per-Type keys or content conditions it carries. -The per-Type caps `filter.contentByType` and `sort.contentFieldByType` are now `number | null`, `null` stating no limit. `MemoryAdapter` and the SQLite adapters declare `null`, since nothing bounds the count locally; a cap is for a server bounding its own cost. `normalizeCapabilities()` reads a stated `null` as no limit, and an absent or malformed cap as `0`, as before. +The per-Type caps `filter.contentByType` and `sort.contentFieldByType` are now `number | null`, `null` stating no limit. `MemoryAdapter` and the SQLite adapters declare `null`, since nothing bounds the count locally; a cap is for a server bounding its own cost. `normalizeCapabilities()` reads a stated `null` as no limit, and an absent or malformed cap as `0`, as before; a discovery fixture pins the stated `null`. `admitsKeys(cap, keys)` in `@haverstack/core/adapter` is the one reading of a cap. A content filter value now matches only a value of its own JSON type in the SQLite adapters, as `===` does: `true` no longer matches `1`. A content filter path may be 33 segments, the depth that reaches the deepest value a write accepts. The conformance suite holds every adapter to both, and to a multi-valued path satisfying both `contentPresent` and a `null` filter. diff --git a/docs/spec/data-model.md b/docs/spec/data-model.md index 12633edc..4e3a74d7 100644 --- a/docs/spec/data-model.md +++ b/docs/spec/data-model.md @@ -620,7 +620,7 @@ The motivating query is a listing: an article's `publishedAt` is optional, so dr **An array along the path is matched element-wise.** `contact@1` stores `emails` as an array of `{ value, label }`, and the question that motivates paths at all — which contact holds this address — is that shape. Traversal and containment are therefore one feature rather than two: a segment applies to each element of an array it meets, and the final comparison does the same, so `{ content: { tags: 'starred' } }` matches a record whose `tags` array contains `'starred'`. A record storing a scalar where another stores an array is matched by the same filter either way. -**A value matches a value of its own JSON type.** `{ content: { flag: true } }` matches `true` and never `1`, and `1` never matches `'1'` — the comparison JavaScript's `===` makes, which is the reference reading every adapter is held to. A number is one type, however an engine stores it. An object or array never equals a filter value, so a path reaching one matches only `null` or presence. +**A value matches a value of its own JSON type.** `{ content: { flag: true } }` matches `true` and never `1`, and `1` never matches `'1'` — the comparison JavaScript's `===` makes, which is the reference reading every adapter is held to. A number is one type, however an engine stores it. A filter value that is an object or array never matches anything: a path is compared by the scalars it reaches, an array reached at its end standing for its elements. A path that descends through a scalar, or through a field that isn't there, reaches no value and matches nothing — it is not an error. A path may be at most **33 segments**; longer is `StackBadRequestError` (400), as is an empty segment (`a..b`) or a segment containing a reserved character. Those are structural faults in the request, refused rather than answered with an empty result, on the same reasoning as a malformed cursor. diff --git a/packages/adapter-conformance/src/record.ts b/packages/adapter-conformance/src/record.ts index 26c83cb5..bc2828c8 100644 --- a/packages/adapter-conformance/src/record.ts +++ b/packages/adapter-conformance/src/record.ts @@ -29,7 +29,7 @@ import type { StackQuery, StackRecord, } from '@haverstack/core'; -import { ElementSetMovedError, inReach } from '@haverstack/core/adapter'; +import { ElementSetMovedError, admitsKeys, inReach } from '@haverstack/core/adapter'; import type { Reach, StackRecordAdapter } from '@haverstack/core/adapter'; type PermissionElement = Extract; @@ -52,9 +52,6 @@ export interface RecordAdapterConformanceOptions { capabilities: StackCapabilities; } -/** Whether a per-Type cap admits `keys` keys; `null` admits any number. */ -const takes = (cap: number | null, keys: number): boolean => cap === null || cap >= keys; - export function runRecordAdapterConformance(options: RecordAdapterConformanceOptions): void { const { name, open, close, capabilities } = options; @@ -1366,7 +1363,7 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt // ----------------------------------------------------------------- // Content by type — capability-gated on filter.contentByType // ----------------------------------------------------------------- - if (takes(capabilities.filter.contentByType, 2)) { + if (admitsKeys(capabilities.filter.contentByType, 2)) { describe('content by type', () => { const OTHER = 'org.haverstack.conformance/other@1'; const THIRD = 'org.haverstack.conformance/third@1'; @@ -1445,7 +1442,7 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt // ----------------------------------------------------------------- // Sort by content field per type — gated on sort.contentFieldByType // ----------------------------------------------------------------- - if (takes(capabilities.sort.contentFieldByType, 3)) { + if (admitsKeys(capabilities.sort.contentFieldByType, 3)) { describe('sort by content field per type', () => { const OTHER = 'org.haverstack.conformance/other@1'; const THIRD = 'org.haverstack.conformance/third@1'; diff --git a/packages/conformance-fixtures/src/discovery.ts b/packages/conformance-fixtures/src/discovery.ts index 61af2eec..9eafd153 100644 --- a/packages/conformance-fixtures/src/discovery.ts +++ b/packages/conformance-fixtures/src/discovery.ts @@ -36,6 +36,30 @@ export const discoveryFixtures: ConformanceFixture }, }, }, + { + name: 'discovery-declares-per-type-shapes-without-a-limit', + description: + 'A per-Type cap of null states no limit: the server takes filter.contentByType and ' + + 'sort.contentFieldByType with any number of keys, and a client sends them unrefused. ' + + 'Only a stated null reads this way — an absent cap declines the shape, as 0 does. See ' + + 'docs/spec/adapters.md § Adapter capabilities.', + method: 'GET', + path: '/.well-known/stack', + responseStatus: 200, + responseBody: { + version: WIRE_PROTOCOL_VERSION, + entityId: 'did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK', + capabilities: { + filter: { content: 'path', contentPresent: true, contentByType: null, search: true }, + sort: { + fields: ['createdAt', 'updatedAt', 'version'], + contentField: true, + contentFieldByType: null, + }, + limits: { attachmentBytes: null, contentBytes: null }, + }, + }, + }, { name: 'discovery-advertises-install-requests', description: diff --git a/packages/core/src/adapter/index.ts b/packages/core/src/adapter/index.ts index dc54cdc2..7dfab385 100644 --- a/packages/core/src/adapter/index.ts +++ b/packages/core/src/adapter/index.ts @@ -27,6 +27,7 @@ export { combineAdapters } from './combine.js'; export { inReach } from './reach.js'; export type { ReachedRecord } from './reach.js'; export { + admitsKeys, assertQueryCapabilities, assertSortCapability, assertValidSort, diff --git a/packages/core/src/collection.ts b/packages/core/src/collection.ts index 60ba4f1b..4d53aa8f 100644 --- a/packages/core/src/collection.ts +++ b/packages/core/src/collection.ts @@ -17,6 +17,7 @@ import { StackBadRequestError, StackMigrationError, StackPermissionError } from './errors.js'; import { baseIdOf, parseTypeId } from './schema.js'; +import { admitsKeys, filtersContent } from './query-validation.js'; import { StoredVersionError, WRITE_EXPECTATION } from './stack/write-expectation.js'; import type { ExpectingClient, WriteExpectation } from './stack/write-expectation.js'; import type { MigrationRegistry } from './stack/migrations.js'; @@ -496,13 +497,12 @@ export const createCollection = ( // the older a version, the fewer apps still write it. const filterCap = predicate === undefined - ? Infinity - : capabilities.filter.content === 'none' - ? 0 - : (capabilities.filter.contentByType ?? Infinity); - const sortCap = - sortField === undefined ? Infinity : (capabilities.sort.contentFieldByType ?? Infinity); - const cap = Math.min(filterCap, sortCap); + ? null + : filtersContent(capabilities) + ? capabilities.filter.contentByType + : 0; + const sortCap = sortField === undefined ? null : capabilities.sort.contentFieldByType; + const admits = (keys: number) => admitsKeys(filterCap, keys) && admitsKeys(sortCap, keys); const sortKind = (() => { const def = sortField === undefined ? undefined : handle.schema[sortField]; return def && def.kind !== 'array' && def.kind !== 'object' ? def.kind : undefined; @@ -543,7 +543,7 @@ export const createCollection = ( if ( translated === UNTRANSLATABLE || key === UNTRANSLATABLE || - Object.keys(arms).length >= cap + !admits(Object.keys(arms).length + 1) ) { narrowed.push(type.id); continue; diff --git a/packages/core/src/query-validation.ts b/packages/core/src/query-validation.ts index e5d57a55..39dbe5bd 100644 --- a/packages/core/src/query-validation.ts +++ b/packages/core/src/query-validation.ts @@ -72,7 +72,7 @@ export function assertQueryCapabilities( 'filter.content', ); } - if (contentByType !== null && arms.length > contentByType) { + if (!admitsKeys(contentByType, arms.length)) { throw new StackBadRequestError( contentByType === 0 ? 'Query uses filter.contentByType, but this adapter does not declare the ' + @@ -168,6 +168,15 @@ export function filtersContent(features: Pick): boo return features.filter.content !== 'none'; } +/** + * Whether a per-Type cap admits a shape of `keys` keys: `null` states no + * limit, and `0` admits none. See docs/spec/adapters.md § Adapter + * capabilities. + */ +export function admitsKeys(cap: number | null, keys: number): boolean { + return cap === null || keys <= cap; +} + /** * The longest path that can reach a stored value: a top-level field sits at * depth 0 and each segment descends one level, so a path past this reaches @@ -329,7 +338,7 @@ export function assertSortCapability( if (sort.contentFieldByType !== undefined) { const keys = Object.keys(sort.contentFieldByType).length; const cap = capabilities.sort.contentFieldByType; - if (cap !== null && keys > cap) { + if (!admitsKeys(cap, keys)) { throw new StackBadRequestError( cap === 0 ? 'Query uses sort.contentFieldByType, but this adapter does not declare the ' + diff --git a/packages/sqlite-shared/src/query.ts b/packages/sqlite-shared/src/query.ts index d8215ccf..24b42f08 100644 --- a/packages/sqlite-shared/src/query.ts +++ b/packages/sqlite-shared/src/query.ts @@ -394,6 +394,8 @@ const recordConditions = (query: StackQuery, reach: Reach | undefined): Conditio const content = contentRows(null, f); if (f.contentByType) { const arms = Object.entries(f.contentByType); + // The Types themselves bound the result too: an empty arm adds no + // condition row, yet still admits only its own Type. const typeIds = arms.map(([typeId]) => typeId); conditions.push(inList('r.type_id', typeIds, params)); for (const [typeId, arm] of arms) content.push(...contentRows(typeId, arm)); diff --git a/packages/wire-types/src/discovery.ts b/packages/wire-types/src/discovery.ts index f982f145..d6907b8b 100644 --- a/packages/wire-types/src/discovery.ts +++ b/packages/wire-types/src/discovery.ts @@ -55,9 +55,10 @@ const SORT_FIELDS: ReadonlySet = new Set(NATIVE_SORT_FIELDS); /** * Resolves anything absent, malformed or unrecognized to the least capable * value it could stand for — one rule, applied once, rather than a default - * per key at each call site. `null` is the permissive reading, and only - * where it is stated: a limit the client cannot pre-check, whose server - * still answers 413, or a cap with no limit. See + * per key at each call site. `null` reads permissively for both: a limit + * the client cannot pre-check, absent or stated, since a limit gates + * nothing and its server still answers 413; and a cap with no limit, only + * where stated, since an absent cap gates a shape and reads as 0. See * docs/spec/adapters.md § Adapter capabilities. */ export function normalizeCapabilities( @@ -71,7 +72,7 @@ export function normalizeCapabilities( : 'none'; // The per-Type shapes travel in the POST /records/query body, which a // server reaching no content does not serve. - const cap = (value: unknown): number | null => (reach === 'none' ? 0 : capOrZero(value)); + const cap = (value: unknown): number | null => (reach === 'none' ? 0 : readCap(value)); return { filter: { content: reach, @@ -85,23 +86,23 @@ export function normalizeCapabilities( contentFieldByType: cap(sort?.contentFieldByType), }, limits: { - attachmentBytes: finiteOrNull(limits?.attachmentBytes), - contentBytes: finiteOrNull(limits?.contentBytes), + attachmentBytes: readLimit(limits?.attachmentBytes), + contentBytes: readLimit(limits?.contentBytes), }, }; } /** - * A cap on keys, `null` — no limit — where the server says so, or 0 — - * declined — for anything else that isn't a non-negative integer. + * A cap on keys: a positive integer as stated, `null` — no limit — where + * the server says so, and 0 — declined — for anything else. */ -const capOrZero = (value: unknown): number | null => { +const readCap = (value: unknown): number | null => { if (value === null) return null; return typeof value === 'number' && Number.isSafeInteger(value) && value > 0 ? value : 0; }; /** A ceiling this client can compare against, or null for "cannot pre-check". */ -const finiteOrNull = (value: unknown): number | null => +const readLimit = (value: unknown): number | null => typeof value === 'number' && Number.isFinite(value) ? value : null; /** Splits a MAJOR.MINOR protocol version. Returns null if it isn't one. */ From 9b7b7b1a5a727fb9089cb01e5f9a6252b69ca1f9 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 21 Oct 2026 20:55:11 +0000 Subject: [PATCH 6/9] refactor(sqlite,core): primitives own their policy Review found the same rule spelled more than one way again, both times because a primitive returned a fragment and left its policy to callers. rows() now registers a named, materialized table on the statement it builds, so every caller-sized table is parsed once per statement and binds first. The absent partition of a by-Type sort no longer re-parses its JSON per record, and statement() loses its parameter-ordering logic. perTypeCap(capabilities, shape) applies the reach rule wherever a cap is used; normalizeCapabilities() reads a cap as stated, collection narrowing and both capability asserts go through it, and assertSortCapability() now refuses a by-Type sort against a reach of 'none' as the spec says. null states no limit on a cap and a limit alike; only absence reads differently, by the least-capable rule. Also: holds() names the aliases it reads. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Mf3p8BVtrQJLZx1JFCHchC --- .changeset/fixed-statement-size.md | 2 +- docs/spec/adapters.md | 4 +- packages/adapter-conformance/src/record.ts | 6 +- packages/core/src/adapter/index.ts | 2 + packages/core/src/collection.ts | 11 +- packages/core/src/query-validation.ts | 63 ++++--- packages/core/src/types/capabilities.ts | 6 +- .../core/tests/query-capabilities.test.ts | 17 ++ packages/sqlite-shared/src/query.ts | 167 +++++++++--------- packages/wire-types/src/discovery.ts | 23 ++- packages/wire-types/tests/discovery.test.ts | 8 +- 11 files changed, 168 insertions(+), 141 deletions(-) diff --git a/.changeset/fixed-statement-size.md b/.changeset/fixed-statement-size.md index 8185535a..6fa3fc5e 100644 --- a/.changeset/fixed-statement-size.md +++ b/.changeset/fixed-statement-size.md @@ -9,6 +9,6 @@ A SQLite statement's text and bound parameters no longer grow with a query's size. Every caller-sized part of a query binds as one table of rows: lists, permission sets, `sort.contentFieldByType`'s entries, and the content conditions — every `filter.content` and `filter.contentPresent` key and every `filter.contentByType` arm — whose paths one recursive walk per condition resolves. Durable Object SQLite no longer fails a query with `too many SQL variables` or `statement too long`, however many per-Type keys or content conditions it carries. -The per-Type caps `filter.contentByType` and `sort.contentFieldByType` are now `number | null`, `null` stating no limit. `MemoryAdapter` and the SQLite adapters declare `null`, since nothing bounds the count locally; a cap is for a server bounding its own cost. `normalizeCapabilities()` reads a stated `null` as no limit, and an absent or malformed cap as `0`, as before; a discovery fixture pins the stated `null`. `admitsKeys(cap, keys)` in `@haverstack/core/adapter` is the one reading of a cap. +The per-Type caps `filter.contentByType` and `sort.contentFieldByType` are now `number | null`, `null` stating no limit. `MemoryAdapter` and the SQLite adapters declare `null`, since nothing bounds the count locally; a cap is for a server bounding its own cost. `null` means the same on a cap and a limit: no limit stated, with the server's own bound still answering. `normalizeCapabilities()` reads a cap as stated — `null` as no limit, absent or malformed as `0` — and no longer zeroes one beside `filter.content: 'none'`; `perTypeCap(capabilities, shape)` in `@haverstack/core/adapter` applies that reach wherever a cap is used, and `admitsKeys(cap, keys)` compares against it. `assertSortCapability()` now refuses `sort.contentFieldByType` against a reach of `'none'`, as the spec already required. A discovery fixture pins a stated `null` cap. A content filter value now matches only a value of its own JSON type in the SQLite adapters, as `===` does: `true` no longer matches `1`. A content filter path may be 33 segments, the depth that reaches the deepest value a write accepts. The conformance suite holds every adapter to both, and to a multi-valued path satisfying both `contentPresent` and a `null` filter. diff --git a/docs/spec/adapters.md b/docs/spec/adapters.md index aef95843..553874fb 100644 --- a/docs/spec/adapters.md +++ b/docs/spec/adapters.md @@ -219,7 +219,7 @@ One type, `StackCapabilities`, names both sides: an adapter declares it as `capa **`filter.contentPresent` sits beside the ladder rather than on it.** It gates `filter.contentPresent`, and is required `true` for local adapters on the same reasoning: asking whether a path holds a value is the same walk with a weaker predicate. It is not a rung because a server promising to match a content value has not thereby promised to answer whether one is there at all. It is meaningful only above `'none'` — presence travels in the `POST /records/query` body, which a server reaching no content does not expose — and a presence path is itself gated by the rung, so a multi-segment one needs `'path'`. -**`filter.contentByType` and `sort.contentFieldByType` are caps, not flags.** Each is the most keys the [per-Type shape](./data-model.md#content-by-type) may hold: `null` states no limit, and `0` declines the shape. **A cap states a real limit or none at all.** A local adapter declares `null` for both, on the reasoning that makes `'path'` required: an arm is the content filter it already runs, scoped to one `typeId`, and costs a SQLite adapter no statement text or parameter of its own (see [Statement size](#statement-size)), so nothing bounds the count. A cap is for a server, which bounds what a request may cost it — see [Wire format § Bounding query cost](./wire-format.md#bounding-query-cost). A server may declare any cap, `0` and `null` included, and a collection narrows the versions past it rather than failing the query — see [Data model § Content queries across versions](./data-model.md#content-queries-across-versions). Both are meaningful only above `filter.content: 'none'`, since they travel in the `POST /records/query` body. +**`filter.contentByType` and `sort.contentFieldByType` are caps, not flags.** Each is the most keys the [per-Type shape](./data-model.md#content-by-type) may hold: `null` states no limit, and `0` declines the shape. **A cap states a real limit or none at all.** A local adapter declares `null` for both, on the reasoning that makes `'path'` required: an arm is the content filter it already runs, scoped to one `typeId`, and costs a SQLite adapter no statement text or parameter of its own (see [Statement size](#statement-size)), so nothing bounds the count. A cap is for a server, which bounds what a request may cost it — see [Wire format § Bounding query cost](./wire-format.md#bounding-query-cost). A server may declare any cap, `0` and `null` included, and a collection narrows the versions past it rather than failing the query — see [Data model § Content queries across versions](./data-model.md#content-queries-across-versions). Both are meaningful only above `filter.content: 'none'`, since they travel in the `POST /records/query` body: a cap is read as declared, and `perTypeCap()` in `@haverstack/core/adapter` applies the reach where it is used, so every reader holds a shape to the same cap. `admitsKeys()` beside it is the one comparison. **`filter.search` has no local-required rule.** A full-text index is a different mechanism from field matching, and neither implies the other, so a local adapter may legitimately decline it (see the JSON adapter note below). @@ -227,7 +227,7 @@ One type, `StackCapabilities`, names both sides: an adapter declares it as `capa **Absent, malformed or unrecognized reads as the least capable value it could stand for.** One rule, for every entry: `'none'` for the reach, `false` for a flag, `0` for a cap, nothing for `sort.fields`, `null` for a limit. A discovery response that omits a group, omits `capabilities` entirely, or names a reach this client has never heard of describes a server this client will not send that query to — silence is never a claim, in either direction, and an unrecognized rung places nowhere on a ladder so it can only be read as its bottom. `normalizeCapabilities()` in `@haverstack/wire-types` applies the rule, so a client reads a foreign discovery response through one function rather than defaulting keys at each use. -Two things this does not refuse. A query naming no sort asks for the server's own default order rather than a stated one, so it claims nothing that has to be declared and runs against an empty `sort.fields`. And `null` is permissive where a server states it: a `null` limit says this client cannot pre-check, not that nothing is enforced — the server's own ceiling still answers with a `413` — and a `null` cap says the server takes any number of keys. Only a stated `null` reads this way; an absent cap is still `0`. +Two things this does not refuse. A query naming no sort asks for the server's own default order rather than a stated one, so it claims nothing that has to be declared and runs against an empty `sort.fields`. And `null` states no limit, on a cap and a limit alike: the client checks nothing before sending, and the server's own bound still answers — `413` for a request's size, `503` for its [cost](./wire-format.md#bounding-query-cost). Only absence reads differently between the two, and by the rule above rather than an exception to it: the least a cap could stand for is `0`, which declines a shape, and the least a limit could stand for is `null`, since a limit gates nothing. **Per-adapter notes:** diff --git a/packages/adapter-conformance/src/record.ts b/packages/adapter-conformance/src/record.ts index bc2828c8..bf0ffbd5 100644 --- a/packages/adapter-conformance/src/record.ts +++ b/packages/adapter-conformance/src/record.ts @@ -29,7 +29,7 @@ import type { StackQuery, StackRecord, } from '@haverstack/core'; -import { ElementSetMovedError, admitsKeys, inReach } from '@haverstack/core/adapter'; +import { ElementSetMovedError, admitsKeys, inReach, perTypeCap } from '@haverstack/core/adapter'; import type { Reach, StackRecordAdapter } from '@haverstack/core/adapter'; type PermissionElement = Extract; @@ -1363,7 +1363,7 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt // ----------------------------------------------------------------- // Content by type — capability-gated on filter.contentByType // ----------------------------------------------------------------- - if (admitsKeys(capabilities.filter.contentByType, 2)) { + if (admitsKeys(perTypeCap(capabilities, 'filter.contentByType'), 2)) { describe('content by type', () => { const OTHER = 'org.haverstack.conformance/other@1'; const THIRD = 'org.haverstack.conformance/third@1'; @@ -1442,7 +1442,7 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt // ----------------------------------------------------------------- // Sort by content field per type — gated on sort.contentFieldByType // ----------------------------------------------------------------- - if (admitsKeys(capabilities.sort.contentFieldByType, 3)) { + if (admitsKeys(perTypeCap(capabilities, 'sort.contentFieldByType'), 3)) { describe('sort by content field per type', () => { const OTHER = 'org.haverstack.conformance/other@1'; const THIRD = 'org.haverstack.conformance/third@1'; diff --git a/packages/core/src/adapter/index.ts b/packages/core/src/adapter/index.ts index 7dfab385..139f0ab6 100644 --- a/packages/core/src/adapter/index.ts +++ b/packages/core/src/adapter/index.ts @@ -36,7 +36,9 @@ export { assertValidVersionsQuery, filtersContent, parseContentFilterKey, + perTypeCap, } from '../query-validation.js'; +export type { PerTypeShape } from '../query-validation.js'; export { contentSortEntry, contentSortKey, diff --git a/packages/core/src/collection.ts b/packages/core/src/collection.ts index 4d53aa8f..f75afa64 100644 --- a/packages/core/src/collection.ts +++ b/packages/core/src/collection.ts @@ -17,7 +17,7 @@ import { StackBadRequestError, StackMigrationError, StackPermissionError } from './errors.js'; import { baseIdOf, parseTypeId } from './schema.js'; -import { admitsKeys, filtersContent } from './query-validation.js'; +import { admitsKeys, perTypeCap } from './query-validation.js'; import { StoredVersionError, WRITE_EXPECTATION } from './stack/write-expectation.js'; import type { ExpectingClient, WriteExpectation } from './stack/write-expectation.js'; import type { MigrationRegistry } from './stack/migrations.js'; @@ -496,12 +496,9 @@ export const createCollection = ( // A version past the adapter's cap is narrowed, newest kept first: // the older a version, the fewer apps still write it. const filterCap = - predicate === undefined - ? null - : filtersContent(capabilities) - ? capabilities.filter.contentByType - : 0; - const sortCap = sortField === undefined ? null : capabilities.sort.contentFieldByType; + predicate === undefined ? null : perTypeCap(capabilities, 'filter.contentByType'); + const sortCap = + sortField === undefined ? null : perTypeCap(capabilities, 'sort.contentFieldByType'); const admits = (keys: number) => admitsKeys(filterCap, keys) && admitsKeys(sortCap, keys); const sortKind = (() => { const def = sortField === undefined ? undefined : handle.schema[sortField]; diff --git a/packages/core/src/query-validation.ts b/packages/core/src/query-validation.ts index 39dbe5bd..b072ae87 100644 --- a/packages/core/src/query-validation.ts +++ b/packages/core/src/query-validation.ts @@ -43,9 +43,9 @@ import type { */ export function assertQueryCapabilities( filter: RecordFilter | undefined, - capabilities: Pick, + capabilities: StackCapabilities, ): void { - const { content: reach, contentPresent, contentByType, search } = capabilities.filter; + const { content: reach, contentPresent, search } = capabilities.filter; if (filter?.search && !search) { throw new StackBadRequestError( 'Query uses filter.search, but this adapter does not declare the filter.search capability.', @@ -72,16 +72,7 @@ export function assertQueryCapabilities( 'filter.content', ); } - if (!admitsKeys(contentByType, arms.length)) { - throw new StackBadRequestError( - contentByType === 0 - ? 'Query uses filter.contentByType, but this adapter does not declare the ' + - 'filter.contentByType capability.' - : `Query uses ${arms.length} filter.contentByType arms, but this adapter takes at ` + - `most ${contentByType}.`, - 'filter.contentByType', - ); - } + if (arms.length > 0) assertAdmitted(capabilities, 'filter.contentByType', arms.length, 'arms'); if (present.length > 0 && !contentPresent) { throw new StackBadRequestError( 'Query uses filter.contentPresent, but this adapter does not declare the ' + @@ -168,15 +159,44 @@ export function filtersContent(features: Pick): boo return features.filter.content !== 'none'; } +/** The query shapes a per-Type cap governs, each named for its capability. */ +export type PerTypeShape = 'filter.contentByType' | 'sort.contentFieldByType'; + /** - * Whether a per-Type cap admits a shape of `keys` keys: `null` states no - * limit, and `0` admits none. See docs/spec/adapters.md § Adapter - * capabilities. + * The cap an adapter holds a per-Type shape to: as declared, except `0` + * below `filter.content: 'none'`, since both shapes travel in the + * POST /records/query body a server reaching no content does not serve. + * See docs/spec/adapters.md § Adapter capabilities. */ +export function perTypeCap(capabilities: StackCapabilities, shape: PerTypeShape): number | null { + if (!filtersContent(capabilities)) return 0; + return shape === 'filter.contentByType' + ? capabilities.filter.contentByType + : capabilities.sort.contentFieldByType; +} + +/** Whether `cap`, a perTypeCap(), admits `keys` keys: `null` states no limit. */ export function admitsKeys(cap: number | null, keys: number): boolean { return cap === null || keys <= cap; } +/** Refuses a per-Type shape of `keys` keys its perTypeCap() does not admit. */ +function assertAdmitted( + capabilities: StackCapabilities, + shape: PerTypeShape, + keys: number, + unit: 'arms' | 'keys', +): void { + const cap = perTypeCap(capabilities, shape); + if (admitsKeys(cap, keys)) return; + throw new StackBadRequestError( + cap === 0 + ? `Query uses ${shape}, but this adapter does not declare the ${shape} capability.` + : `Query uses ${keys} ${shape} ${unit}, but this adapter takes at most ${cap}.`, + shape, + ); +} + /** * The longest path that can reach a stored value: a top-level field sits at * depth 0 and each segment descends one level, so a path past this reaches @@ -332,21 +352,12 @@ export function normalizeSort( */ export function assertSortCapability( sort: QuerySort | undefined, - capabilities: Pick, + capabilities: StackCapabilities, ): void { if (!sort) return; if (sort.contentFieldByType !== undefined) { const keys = Object.keys(sort.contentFieldByType).length; - const cap = capabilities.sort.contentFieldByType; - if (!admitsKeys(cap, keys)) { - throw new StackBadRequestError( - cap === 0 - ? 'Query uses sort.contentFieldByType, but this adapter does not declare the ' + - 'sort.contentFieldByType capability.' - : `Query uses ${keys} sort.contentFieldByType keys, but this adapter takes at most ${cap}.`, - 'sort.contentFieldByType', - ); - } + assertAdmitted(capabilities, 'sort.contentFieldByType', keys, 'keys'); return; } if (sort.contentField !== undefined) { diff --git a/packages/core/src/types/capabilities.ts b/packages/core/src/types/capabilities.ts index d109f3aa..94ad7a1c 100644 --- a/packages/core/src/types/capabilities.ts +++ b/packages/core/src/types/capabilities.ts @@ -70,9 +70,9 @@ export type StackCapabilities = { }; /** * Ceilings a client can check before spending a request; the server's - * own limit stays authoritative. `null` means this client cannot - * pre-check, not that nothing is enforced — local adapters declare it. - * See docs/spec/adapters.md § Adapter capabilities. + * own limit stays authoritative. `null` states no limit, as on a cap — + * nothing to pre-check, not nothing enforced — and local adapters + * declare it. See docs/spec/adapters.md § Adapter capabilities. */ limits: { /** Maximum attachment upload size in bytes. */ diff --git a/packages/core/tests/query-capabilities.test.ts b/packages/core/tests/query-capabilities.test.ts index c04f1cac..7527b3c3 100644 --- a/packages/core/tests/query-capabilities.test.ts +++ b/packages/core/tests/query-capabilities.test.ts @@ -15,6 +15,7 @@ import { assertSortCapability, assertValidContentByType, assertValidSort, + perTypeCap, } from '../src/query-validation.js'; import { StackBadRequestError } from '../src/errors.js'; import type { @@ -161,6 +162,22 @@ describe('a refused query names the capability it needed', () => { ).not.toThrow(); }); + // Both shapes travel in the POST /records/query body, which a reach of + // 'none' does not serve, so a cap declared beside it admits nothing. + test('a per-Type cap admits nothing beside a reach of none', () => { + const none = without({ + filter: { content: 'none', contentByType: null }, + sort: { contentFieldByType: 16 }, + }); + expect(perTypeCap(none, 'filter.contentByType')).toBe(0); + expect(perTypeCap(none, 'sort.contentFieldByType')).toBe(0); + expect( + refusalFor(() => + assertSortCapability({ contentFieldByType: { 'com.example/x@1': { field: 'a' } } }, none), + ), + ).toBe('sort.contentFieldByType'); + }); + test('a null per-Type cap admits any number of keys', () => { const typeIds = Array.from({ length: 500 }, (_, i) => `com.example/x@${i + 1}`); const filter = { contentByType: Object.fromEntries(typeIds.map((id) => [id, {}])) }; diff --git a/packages/sqlite-shared/src/query.ts b/packages/sqlite-shared/src/query.ts index 24b42f08..973c07cc 100644 --- a/packages/sqlite-shared/src/query.ts +++ b/packages/sqlite-shared/src/query.ts @@ -85,18 +85,40 @@ const members = (alias: string): string => `CASE WHEN ${alias}.t = 'object' THEN ${alias}.v ELSE '{}' END`; /** - * Tuples as a table with `columns`, named in tuple order: the one shape a - * caller-sized part of a query takes, bound as a single JSON parameter so - * the statement never grows with it. See docs/spec/adapters.md § Statement - * size. + * One statement as it is built: its tables, then its conditions on `r` and + * the parameters its text binds, in textual order. Every table's parameter + * binds ahead of the text's, since the tables lead the statement. */ -const rows = (columns: readonly string[], tuples: unknown[][], params: unknown[]): string => { - params.push(JSON.stringify(tuples)); +type Statement = { + tables: string[]; + tableParams: unknown[]; + conditions: string[]; + params: unknown[]; +}; + +const emptyStatement = (): Statement => ({ + tables: [], + tableParams: [], + conditions: ["r.id != '_config'"], + params: [], +}); + +/** + * Tuples as a table of `s` with `columns`, named in tuple order: the one + * shape a caller-sized part of a query takes. It binds as one JSON + * parameter, so the statement never grows with it, and is materialized + * once, so nothing that reads it per record re-parses the JSON. Returns the + * table's name. See docs/spec/adapters.md § Statement size. + */ +const rows = (s: Statement, columns: readonly string[], tuples: unknown[][]): string => { + const name = `t${s.tables.length}`; const named = columns.map((column, i) => `json_extract(value, '$[${i}]') AS ${column}`); - return `(SELECT ${named.join(', ')} FROM json_each(?))`; + s.tables.push(`${name} AS MATERIALIZED (SELECT ${named.join(', ')} FROM json_each(?))`); + s.tableParams.push(JSON.stringify(tuples)); + return name; }; -/** The columns of a `cond` row; see contentRows(). */ +/** The columns of a content condition; see contentRows(). */ const CONDITION_COLUMNS = ['scope', 'path', 'depth', 'op', 'val', 'kind']; /** @@ -108,23 +130,23 @@ const kindOf = (type: string): string => `CASE ${type} WHEN 'integer' THEN 'number' WHEN 'real' THEN 'number' ELSE ${type} END`; /** - * Whether the `cond` row `c` holds for `r`. The path is walked as data, a - * recursive CTE matching one segment per step against json_each's `key`, so - * no key is read as syntax and the text is the same for any path; one walk - * gathers what every op asks of the values it reaches. Each op is - * element-wise, so a path reaching a null and a value meets both `isNull` - * and `present`. See docs/spec/data-model.md § Filter. + * Whether the condition row `c` holds for the record row `r`. The path is + * walked as data, a recursive CTE matching one segment per step against + * json_each's `key`, so no key is read as syntax and the text is the same + * for any path; one walk gathers what every op asks of the values it + * reaches. Each op is element-wise, so a path reaching a null and a value + * meets both `isNull` and `present`. See docs/spec/data-model.md § Filter. */ -const HOLDS = - "(WITH RECURSIVE w(d, v, t) AS (SELECT 0, r.content, 'object' UNION ALL " + +const holds = (c: string, r: string): string => + `(WITH RECURSIVE w(d, v, t) AS (SELECT 0, ${r}.content, 'object' UNION ALL ` + `SELECT w.d + 1, e.value, e.type FROM w JOIN json_each(${members('w')}) m ` + `JOIN json_each(${spread('m')}) e ` + - "WHERE w.d < c.depth AND m.key = json_extract(c.path, '$[' || w.d || ']')) " + - 'SELECT CASE c.op ' + - `WHEN 'equals' THEN IFNULL(MAX(${kindOf('w.t')} = c.kind AND w.v = c.val), 0) ` + + `WHERE w.d < ${c}.depth AND m.key = json_extract(${c}.path, '$[' || w.d || ']')) ` + + `SELECT CASE ${c}.op ` + + `WHEN 'equals' THEN IFNULL(MAX(${kindOf('w.t')} = ${c}.kind AND w.v = ${c}.val), 0) ` + "WHEN 'present' THEN IFNULL(MAX(w.t != 'null'), 0) " + "ELSE IFNULL(MAX(w.t = 'null'), 0) OR NOT IFNULL(MAX(w.t != 'null'), 0) END " + - 'FROM w WHERE w.d = c.depth)'; + `FROM w WHERE w.d = ${c}.depth)`; /** The join alias the content index is read through. */ const SORT_ALIAS = 'cs'; @@ -159,7 +181,7 @@ const valueKind = (value: unknown): string | null => { }; /** - * A `content` and `contentPresent` predicate as `cond` rows scoped to + * A `content` and `contentPresent` predicate as condition rows scoped to * `scope`: path segments, their count, the op, and for `equals` the value * and its kindOf() (none for an object or array, which `===` never * matches). A `null` value means "no value at the path, or a value that is @@ -186,13 +208,16 @@ const contentRows = (scope: string | null, predicate: ContentArm): unknown[][] = * one value bound as itself, so an index on the column still yields rows in * its order. An empty list matches nothing. */ -const inList = (column: string, values: readonly string[], params: unknown[]): string => { +const inList = (s: Statement, column: string, values: readonly string[]): string => { if (values.length === 1) { - params.push(values[0]); + s.params.push(values[0]); return `${column} = ?`; } - const tuples = values.map((value) => [value]); - const table = rows(['value'], tuples, params); + const table = rows( + s, + ['value'], + values.map((value) => [value]), + ); return `${column} IN (SELECT value FROM ${table})`; }; @@ -202,13 +227,13 @@ const inList = (column: string, values: readonly string[], params: unknown[]): s * idx_permissions_element. Each element is a row naming its set and that * set's size; an empty set matches nothing. */ -const carriesPermissions = (sets: AuthorityAssociation[][], params: unknown[]): string => { +const carriesPermissions = (s: Statement, sets: AuthorityAssociation[][]): string => { const elements = rows( + s, ['set_index', 'set_size', 'kind', 'label', 'grantee_kind', 'grantee_id', 'role'], sets.flatMap((set, i) => set.map((p) => [i, set.length, p.kind, p.label, ...permissionKeyColumns(p)]), ), - params, ); return ( `r.id IN (SELECT p.record_id FROM ${elements} e JOIN permissions p ` + @@ -222,33 +247,28 @@ const carriesPermissions = (sets: AuthorityAssociation[][], params: unknown[]): * The reach as conditions on `r`: inside the fence, and inside `within` by * one of its arms. See docs/spec/access-control.md § Reach. */ -const reachConditions = (reach: Reach, params: unknown[]): string[] => { - const conditions: string[] = []; - if (reach.fence) conditions.push(inList('r.type_id', reach.fence, params)); +const reachConditions = (s: Statement, reach: Reach): void => { + if (reach.fence) s.conditions.push(inList(s, 'r.type_id', reach.fence)); const within = reach.within; if (within) { - const arms = [inList('r.type_id', within.types, params)]; + const arms = [inList(s, 'r.type_id', within.types)]; if (within.authored) { - params.push(within.authored.subjectId); - arms.push(`(r.entity_id = ? AND ${inList('r.type_id', within.authored.types, params)})`); + s.params.push(within.authored.subjectId); + arms.push(`(r.entity_id = ? AND ${inList(s, 'r.type_id', within.authored.types)})`); } - arms.push(carriesPermissions(within.permissions, params)); - conditions.push(`(${arms.join(' OR ')})`); + arms.push(carriesPermissions(s, within.permissions)); + s.conditions.push(`(${arms.join(' OR ')})`); } - return conditions; }; -/** A statement's conditions on `r`, their parameters, and its `cond` rows. */ -type Conditions = { conditions: string[]; params: unknown[]; content: unknown[][] }; - /** * Every condition on the records row itself — everything a query asks * except where in the ordering to resume, which is per-partition and gets * appended by the caller. */ -const recordConditions = (query: StackQuery, reach: Reach | undefined): Conditions => { - const conditions: string[] = ["r.id != '_config'"]; - const params: unknown[] = []; +const recordConditions = (query: StackQuery, reach: Reach | undefined): Statement => { + const s = emptyStatement(); + const { conditions, params } = s; const f = query.filter ?? {}; /** @@ -257,9 +277,7 @@ const recordConditions = (query: StackQuery, reach: Reach | undefined): Conditio */ const inFilter = (column: string, value: string | readonly string[] | undefined): void => { if (value === undefined) return; - conditions.push( - inList(`r.${column}`, Array.isArray(value) ? value : [value as string], params), - ); + conditions.push(inList(s, `r.${column}`, Array.isArray(value) ? value : [value as string])); }; /** One side of a date range, as the open-ended comparison it is. */ @@ -285,7 +303,7 @@ const recordConditions = (query: StackQuery, reach: Reach | undefined): Conditio conditions.push('r.unlisted_at IS NULL'); } - if (reach) conditions.push(...reachConditions(reach, params)); + if (reach) reachConditions(s, reach); inFilter('type_id', f.typeId); @@ -326,7 +344,7 @@ const recordConditions = (query: StackQuery, reach: Reach | undefined): Conditio // counted against the distinct tags asked for, the list bound as one. if (f.tags?.length) { const tags = [...new Set(f.tags)]; - const label = inList('a.label', tags, params); + const label = inList(s, 'a.label', tags); conditions.push( `r.id IN (SELECT a.record_id FROM associations a WHERE a.kind = 'tag' AND ${label} ` + `GROUP BY a.record_id HAVING COUNT(DISTINCT a.label) = ?)`, @@ -397,14 +415,15 @@ const recordConditions = (query: StackQuery, reach: Reach | undefined): Conditio // The Types themselves bound the result too: an empty arm adds no // condition row, yet still admits only its own Type. const typeIds = arms.map(([typeId]) => typeId); - conditions.push(inList('r.type_id', typeIds, params)); + conditions.push(inList(s, 'r.type_id', typeIds)); for (const [typeId, arm] of arms) content.push(...contentRows(typeId, arm)); } if (content.length > 0) { + const table = rows(s, CONDITION_COLUMNS, content); conditions.push( - 'NOT EXISTS (SELECT 1 FROM cond c WHERE (c.scope IS NULL OR c.scope = r.type_id) ' + - `AND NOT ${HOLDS})`, + `NOT EXISTS (SELECT 1 FROM ${table} c WHERE (c.scope IS NULL OR c.scope = r.type_id) ` + + `AND NOT ${holds('c', 'r')})`, ); } @@ -421,7 +440,7 @@ const recordConditions = (query: StackQuery, reach: Reach | undefined): Conditio } } - return { conditions, params, content }; + return s; }; /** @@ -453,26 +472,11 @@ const presentCursorCondition = ( ); }; -/** - * One statement reading `r` from `from`, whose own parameters bind ahead of - * the conditions'. When it tests content, the `cond` rows come first, - * materialized once so a walk reads columns rather than re-parsing JSON - * per record. - */ -const statement = ( - from: string, - { conditions, params, content }: Conditions, - tail: string, - fromParams: unknown[] = [], -): QueryStatement => { - const select = `SELECT r.* FROM ${from} WHERE ${conditions.join(' AND ')}${tail}`; - if (content.length === 0) return { sql: select, params: [...fromParams, ...params] }; - const head: unknown[] = []; - const cond = rows(CONDITION_COLUMNS, content, head); - return { - sql: `WITH cond AS MATERIALIZED ${cond} ${select}`, - params: [...head, ...fromParams, ...params], - }; +/** `s` as one statement reading `r` from `from`, its tables leading. */ +const statement = (s: Statement, from: string, tail: string): QueryStatement => { + const select = `SELECT r.* FROM ${from} WHERE ${s.conditions.join(' AND ')}${tail}`; + const sql = s.tables.length === 0 ? select : `WITH ${s.tables.join(', ')} ${select}`; + return { sql, params: [...s.tableParams, ...s.params] }; }; /** @@ -499,7 +503,7 @@ export const buildQueryPlan = (query: StackQuery, reach?: Reach): QueryStatement where.conditions.push(`(r.${col} ${op} ? OR (r.${col} = ? AND r.id ${op} ?))`); where.params.push(cursor.value, cursor.value, cursor.id); } - return [statement('records r', where, ` ORDER BY r.${col} ${dir}, r.id ${dir} LIMIT ?`)]; + return [statement(where, 'records r', ` ORDER BY r.${col} ${dir}, r.id ${dir} LIMIT ?`)]; } const cursor = decoded ? contentCursor(decoded, contentField) : null; @@ -519,7 +523,7 @@ export const buildQueryPlan = (query: StackQuery, reach?: Reach): QueryStatement `${SORT_ALIAS}.value_rank ${dir}, ${SORT_ALIAS}.num_value ${dir}, ` + `${SORT_ALIAS}.text_key ${dir}, ${SORT_ALIAS}.text_value ${dir}, ` + `${SORT_ALIAS}.record_id ${dir}`; - pages.push(statement(from, where, ` ORDER BY ${order} LIMIT ?`)); + pages.push(statement(where, from, ` ORDER BY ${order} LIMIT ?`)); } // Records holding nothing there, which trail the others whichever way @@ -535,7 +539,7 @@ export const buildQueryPlan = (query: StackQuery, reach?: Reach): QueryStatement where.conditions.push(`r.id ${op} ?`); where.params.push(cursor.id); } - pages.push(statement('records r', where, ` ORDER BY r.id ${dir} LIMIT ?`)); + pages.push(statement(where, 'records r', ` ORDER BY r.id ${dir} LIMIT ?`)); } return pages; @@ -569,10 +573,10 @@ const byTypePlan = ( const pages: QueryStatement[] = []; if (cursor?.kind !== 'absent' && fields.length + constants.length > 0) { + const where = recordConditions(query, reach); const sources: string[] = []; - const sourceParams: unknown[] = []; if (fields.length > 0) { - const byField = rows(['type_id', 'field'], fields, sourceParams); + const byField = rows(where, ['type_id', 'field'], fields); sources.push( 'SELECT ci.record_id AS record_id, ci.value_rank AS value_rank, ' + 'ci.num_value AS num_value, ci.text_key AS text_key, ci.text_value AS text_value ' + @@ -582,13 +586,13 @@ const byTypePlan = ( } if (constants.length > 0) { const byConstant = rows( + where, ['type_id', 'value_rank', 'num_value', 'text_key', 'text_value'], constants.map(([typeId, entry]) => entry.kind === 'num' ? [typeId, 0, entry.num, null, null] : [typeId, 1, null, entry.key, entry.text], ), - sourceParams, ); sources.push( 'SELECT t.id AS record_id, k.value_rank AS value_rank, k.num_value AS num_value, ' + @@ -596,7 +600,6 @@ const byTypePlan = ( `FROM ${byConstant} k JOIN records t ON t.type_id = k.type_id`, ); } - const where = recordConditions(query, reach); if (cursor) where.conditions.push(presentCursorCondition(cursor, op, where.params)); const from = `(${sources.join(' UNION ALL ')}) ${SORT_ALIAS} ` + @@ -605,7 +608,7 @@ const byTypePlan = ( `${SORT_ALIAS}.value_rank ${dir}, ${SORT_ALIAS}.num_value ${dir}, ` + `${SORT_ALIAS}.text_key ${dir}, ${SORT_ALIAS}.text_value ${dir}, ` + `${SORT_ALIAS}.record_id ${dir}`; - pages.push(statement(from, where, ` ORDER BY ${order} LIMIT ?`, sourceParams)); + pages.push(statement(where, from, ` ORDER BY ${order} LIMIT ?`)); } // Records holding nothing to order by: a Type's field with no value, a @@ -613,7 +616,7 @@ const byTypePlan = ( { const where = recordConditions(query, reach); if (fields.length > 0) { - const byField = rows(['type_id', 'field'], fields, where.params); + const byField = rows(where, ['type_id', 'field'], fields); where.conditions.push( `NOT EXISTS (SELECT 1 FROM ${byField} f JOIN content_index ${SORT_ALIAS} ` + `ON ${SORT_ALIAS}.record_id = r.id AND ${SORT_ALIAS}.field = f.field ` + @@ -622,13 +625,13 @@ const byTypePlan = ( } if (constants.length > 0) { const typeIds = constants.map(([typeId]) => typeId); - where.conditions.push(`NOT (${inList('r.type_id', typeIds, where.params)})`); + where.conditions.push(`NOT (${inList(where, 'r.type_id', typeIds)})`); } if (cursor?.kind === 'absent') { where.conditions.push(`r.id ${op} ?`); where.params.push(cursor.id); } - pages.push(statement('records r', where, ` ORDER BY r.id ${dir} LIMIT ?`)); + pages.push(statement(where, 'records r', ` ORDER BY r.id ${dir} LIMIT ?`)); } return pages; }; diff --git a/packages/wire-types/src/discovery.ts b/packages/wire-types/src/discovery.ts index d6907b8b..a6bd88f0 100644 --- a/packages/wire-types/src/discovery.ts +++ b/packages/wire-types/src/discovery.ts @@ -55,11 +55,11 @@ const SORT_FIELDS: ReadonlySet = new Set(NATIVE_SORT_FIELDS); /** * Resolves anything absent, malformed or unrecognized to the least capable * value it could stand for — one rule, applied once, rather than a default - * per key at each call site. `null` reads permissively for both: a limit - * the client cannot pre-check, absent or stated, since a limit gates - * nothing and its server still answers 413; and a cap with no limit, only - * where stated, since an absent cap gates a shape and reads as 0. See - * docs/spec/adapters.md § Adapter capabilities. + * per key at each call site. A cap or limit of `null` states no limit, + * and an absent one reads as the least it could stand for: `0` for a cap, + * which declines a shape, and `null` for a limit, which gates nothing. A + * cap is read as stated whatever the reach; perTypeCap() in core applies + * the reach. See docs/spec/adapters.md § Adapter capabilities. */ export function normalizeCapabilities( capabilities: DiscoveryCapabilities | undefined, @@ -70,20 +70,17 @@ export function normalizeCapabilities( const reach = CONTENT_FILTER_REACHES.has(filter?.content as string) ? (filter?.content as ContentFilterReach) : 'none'; - // The per-Type shapes travel in the POST /records/query body, which a - // server reaching no content does not serve. - const cap = (value: unknown): number | null => (reach === 'none' ? 0 : readCap(value)); return { filter: { content: reach, contentPresent: filter?.contentPresent === true, - contentByType: cap(filter?.contentByType), + contentByType: readCap(filter?.contentByType), search: filter?.search === true, }, sort: { fields: (sort?.fields ?? []).filter((f): f is NativeSortField => SORT_FIELDS.has(f)), contentField: sort?.contentField === true, - contentFieldByType: cap(sort?.contentFieldByType), + contentFieldByType: readCap(sort?.contentFieldByType), }, limits: { attachmentBytes: readLimit(limits?.attachmentBytes), @@ -93,15 +90,15 @@ export function normalizeCapabilities( } /** - * A cap on keys: a positive integer as stated, `null` — no limit — where - * the server says so, and 0 — declined — for anything else. + * A cap on keys: a positive integer or `null` — no limit — as stated, and + * 0 — declined — for anything else. */ const readCap = (value: unknown): number | null => { if (value === null) return null; return typeof value === 'number' && Number.isSafeInteger(value) && value > 0 ? value : 0; }; -/** A ceiling this client can compare against, or null for "cannot pre-check". */ +/** A ceiling as stated, and `null` — no limit stated — for anything else. */ const readLimit = (value: unknown): number | null => typeof value === 'number' && Number.isFinite(value) ? value : null; diff --git a/packages/wire-types/tests/discovery.test.ts b/packages/wire-types/tests/discovery.test.ts index c3c47742..5a7f60ed 100644 --- a/packages/wire-types/tests/discovery.test.ts +++ b/packages/wire-types/tests/discovery.test.ts @@ -100,15 +100,15 @@ describe('normalizeCapabilities', () => { expect([read.filter.contentByType, read.sort.contentFieldByType]).toEqual([null, null]); }); - // Both travel in the POST /records/query body, which a server reaching no - // content does not serve, so a cap it declares is one it cannot honor. - it('reads a per-Type cap as 0 beside a reach of none', () => { + // The reach is applied where a cap is used, by perTypeCap() in core, so + // reading stays one rule per entry. + it('reads a per-Type cap as stated beside a reach of none', () => { for (const cap of [16, null]) { const read = normalizeCapabilities({ filter: { content: 'none', contentByType: cap }, sort: { contentFieldByType: cap }, }); - expect([read.filter.contentByType, read.sort.contentFieldByType]).toEqual([0, 0]); + expect([read.filter.contentByType, read.sort.contentFieldByType]).toEqual([cap, cap]); } }); From a676ed8811642ee1f1c6bf53ef0f53af80811b5e Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 21 Oct 2026 21:24:07 +0000 Subject: [PATCH 7/9] refactor(core,wire-types): hold the reach invariant where capabilities enter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit What depends on filter.content — contentPresent and both per-Type caps — is false or 0 beside 'none'. That is now established where capabilities enter, as AGENTS.md asks: the conformance suite already holds a local adapter's declaration to it, and normalizeCapabilities() applies it to a discovery response, contentPresent included. Readers trust it, so perTypeCap() is gone and every cap is read directly through admitsKeys(). Also: collection narrowing skips an unused shape rather than calling it an unlimited cap; holds() names its isNull branch and derives depth from path instead of carrying both; valueKind() and kindOf() point at each other; holds() and normalizeCapabilities() docblocks fit the comment limit. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Mf3p8BVtrQJLZx1JFCHchC --- .changeset/fixed-statement-size.md | 2 +- docs/spec/adapters.md | 2 +- packages/adapter-conformance/src/record.ts | 6 +-- packages/core/src/adapter/index.ts | 2 - packages/core/src/collection.ts | 10 ++-- packages/core/src/query-validation.ts | 33 ++++--------- .../core/tests/query-capabilities.test.ts | 17 ------- packages/sqlite-shared/src/query.ts | 49 ++++++++++--------- packages/wire-types/src/discovery.ts | 18 +++---- packages/wire-types/tests/discovery.test.ts | 14 ++++-- 10 files changed, 62 insertions(+), 91 deletions(-) diff --git a/.changeset/fixed-statement-size.md b/.changeset/fixed-statement-size.md index 6fa3fc5e..a7661352 100644 --- a/.changeset/fixed-statement-size.md +++ b/.changeset/fixed-statement-size.md @@ -9,6 +9,6 @@ A SQLite statement's text and bound parameters no longer grow with a query's size. Every caller-sized part of a query binds as one table of rows: lists, permission sets, `sort.contentFieldByType`'s entries, and the content conditions — every `filter.content` and `filter.contentPresent` key and every `filter.contentByType` arm — whose paths one recursive walk per condition resolves. Durable Object SQLite no longer fails a query with `too many SQL variables` or `statement too long`, however many per-Type keys or content conditions it carries. -The per-Type caps `filter.contentByType` and `sort.contentFieldByType` are now `number | null`, `null` stating no limit. `MemoryAdapter` and the SQLite adapters declare `null`, since nothing bounds the count locally; a cap is for a server bounding its own cost. `null` means the same on a cap and a limit: no limit stated, with the server's own bound still answering. `normalizeCapabilities()` reads a cap as stated — `null` as no limit, absent or malformed as `0` — and no longer zeroes one beside `filter.content: 'none'`; `perTypeCap(capabilities, shape)` in `@haverstack/core/adapter` applies that reach wherever a cap is used, and `admitsKeys(cap, keys)` compares against it. `assertSortCapability()` now refuses `sort.contentFieldByType` against a reach of `'none'`, as the spec already required. A discovery fixture pins a stated `null` cap. +The per-Type caps `filter.contentByType` and `sort.contentFieldByType` are now `number | null`, `null` stating no limit. `MemoryAdapter` and the SQLite adapters declare `null`, since nothing bounds the count locally; a cap is for a server bounding its own cost. `null` means the same on a cap and a limit: no limit stated, with the server's own bound still answering. `normalizeCapabilities()` reads a stated `null` cap as no limit and an absent or malformed one as `0`, and now reads `filter.contentPresent` as `false` beside `filter.content: 'none'`, as it already read both caps as `0`. `admitsKeys(cap, keys)` in `@haverstack/core/adapter` is the one comparison against a cap. A discovery fixture pins a stated `null` cap. A content filter value now matches only a value of its own JSON type in the SQLite adapters, as `===` does: `true` no longer matches `1`. A content filter path may be 33 segments, the depth that reaches the deepest value a write accepts. The conformance suite holds every adapter to both, and to a multi-valued path satisfying both `contentPresent` and a `null` filter. diff --git a/docs/spec/adapters.md b/docs/spec/adapters.md index 553874fb..2b8bfdd4 100644 --- a/docs/spec/adapters.md +++ b/docs/spec/adapters.md @@ -219,7 +219,7 @@ One type, `StackCapabilities`, names both sides: an adapter declares it as `capa **`filter.contentPresent` sits beside the ladder rather than on it.** It gates `filter.contentPresent`, and is required `true` for local adapters on the same reasoning: asking whether a path holds a value is the same walk with a weaker predicate. It is not a rung because a server promising to match a content value has not thereby promised to answer whether one is there at all. It is meaningful only above `'none'` — presence travels in the `POST /records/query` body, which a server reaching no content does not expose — and a presence path is itself gated by the rung, so a multi-segment one needs `'path'`. -**`filter.contentByType` and `sort.contentFieldByType` are caps, not flags.** Each is the most keys the [per-Type shape](./data-model.md#content-by-type) may hold: `null` states no limit, and `0` declines the shape. **A cap states a real limit or none at all.** A local adapter declares `null` for both, on the reasoning that makes `'path'` required: an arm is the content filter it already runs, scoped to one `typeId`, and costs a SQLite adapter no statement text or parameter of its own (see [Statement size](#statement-size)), so nothing bounds the count. A cap is for a server, which bounds what a request may cost it — see [Wire format § Bounding query cost](./wire-format.md#bounding-query-cost). A server may declare any cap, `0` and `null` included, and a collection narrows the versions past it rather than failing the query — see [Data model § Content queries across versions](./data-model.md#content-queries-across-versions). Both are meaningful only above `filter.content: 'none'`, since they travel in the `POST /records/query` body: a cap is read as declared, and `perTypeCap()` in `@haverstack/core/adapter` applies the reach where it is used, so every reader holds a shape to the same cap. `admitsKeys()` beside it is the one comparison. +**`filter.contentByType` and `sort.contentFieldByType` are caps, not flags.** Each is the most keys the [per-Type shape](./data-model.md#content-by-type) may hold: `null` states no limit, and `0` declines the shape. **A cap states a real limit or none at all.** A local adapter declares `null` for both, on the reasoning that makes `'path'` required: an arm is the content filter it already runs, scoped to one `typeId`, and costs a SQLite adapter no statement text or parameter of its own (see [Statement size](#statement-size)), so nothing bounds the count. A cap is for a server, which bounds what a request may cost it — see [Wire format § Bounding query cost](./wire-format.md#bounding-query-cost). A server may declare any cap, `0` and `null` included, and a collection narrows the versions past it rather than failing the query — see [Data model § Content queries across versions](./data-model.md#content-queries-across-versions). Both are meaningful only above `filter.content: 'none'`, since they travel in the `POST /records/query` body, and are `0` beside it — as `filter.contentPresent` is `false`. That holds where capabilities enter, so nothing reading them re-derives it: the [conformance suite](#conformance) holds a local adapter's declaration to it, and `normalizeCapabilities()` applies it to a discovery response. `admitsKeys()` in `@haverstack/core/adapter` is the one comparison against a cap. **`filter.search` has no local-required rule.** A full-text index is a different mechanism from field matching, and neither implies the other, so a local adapter may legitimately decline it (see the JSON adapter note below). diff --git a/packages/adapter-conformance/src/record.ts b/packages/adapter-conformance/src/record.ts index bf0ffbd5..bc2828c8 100644 --- a/packages/adapter-conformance/src/record.ts +++ b/packages/adapter-conformance/src/record.ts @@ -29,7 +29,7 @@ import type { StackQuery, StackRecord, } from '@haverstack/core'; -import { ElementSetMovedError, admitsKeys, inReach, perTypeCap } from '@haverstack/core/adapter'; +import { ElementSetMovedError, admitsKeys, inReach } from '@haverstack/core/adapter'; import type { Reach, StackRecordAdapter } from '@haverstack/core/adapter'; type PermissionElement = Extract; @@ -1363,7 +1363,7 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt // ----------------------------------------------------------------- // Content by type — capability-gated on filter.contentByType // ----------------------------------------------------------------- - if (admitsKeys(perTypeCap(capabilities, 'filter.contentByType'), 2)) { + if (admitsKeys(capabilities.filter.contentByType, 2)) { describe('content by type', () => { const OTHER = 'org.haverstack.conformance/other@1'; const THIRD = 'org.haverstack.conformance/third@1'; @@ -1442,7 +1442,7 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt // ----------------------------------------------------------------- // Sort by content field per type — gated on sort.contentFieldByType // ----------------------------------------------------------------- - if (admitsKeys(perTypeCap(capabilities, 'sort.contentFieldByType'), 3)) { + if (admitsKeys(capabilities.sort.contentFieldByType, 3)) { describe('sort by content field per type', () => { const OTHER = 'org.haverstack.conformance/other@1'; const THIRD = 'org.haverstack.conformance/third@1'; diff --git a/packages/core/src/adapter/index.ts b/packages/core/src/adapter/index.ts index 139f0ab6..7dfab385 100644 --- a/packages/core/src/adapter/index.ts +++ b/packages/core/src/adapter/index.ts @@ -36,9 +36,7 @@ export { assertValidVersionsQuery, filtersContent, parseContentFilterKey, - perTypeCap, } from '../query-validation.js'; -export type { PerTypeShape } from '../query-validation.js'; export { contentSortEntry, contentSortKey, diff --git a/packages/core/src/collection.ts b/packages/core/src/collection.ts index f75afa64..b3911fb7 100644 --- a/packages/core/src/collection.ts +++ b/packages/core/src/collection.ts @@ -17,7 +17,7 @@ import { StackBadRequestError, StackMigrationError, StackPermissionError } from './errors.js'; import { baseIdOf, parseTypeId } from './schema.js'; -import { admitsKeys, perTypeCap } from './query-validation.js'; +import { admitsKeys } from './query-validation.js'; import { StoredVersionError, WRITE_EXPECTATION } from './stack/write-expectation.js'; import type { ExpectingClient, WriteExpectation } from './stack/write-expectation.js'; import type { MigrationRegistry } from './stack/migrations.js'; @@ -495,11 +495,9 @@ export const createCollection = ( const { capabilities } = client; // A version past the adapter's cap is narrowed, newest kept first: // the older a version, the fewer apps still write it. - const filterCap = - predicate === undefined ? null : perTypeCap(capabilities, 'filter.contentByType'); - const sortCap = - sortField === undefined ? null : perTypeCap(capabilities, 'sort.contentFieldByType'); - const admits = (keys: number) => admitsKeys(filterCap, keys) && admitsKeys(sortCap, keys); + const admits = (keys: number) => + (predicate === undefined || admitsKeys(capabilities.filter.contentByType, keys)) && + (sortField === undefined || admitsKeys(capabilities.sort.contentFieldByType, keys)); const sortKind = (() => { const def = sortField === undefined ? undefined : handle.schema[sortField]; return def && def.kind !== 'array' && def.kind !== 'object' ? def.kind : undefined; diff --git a/packages/core/src/query-validation.ts b/packages/core/src/query-validation.ts index b072ae87..2e354adb 100644 --- a/packages/core/src/query-validation.ts +++ b/packages/core/src/query-validation.ts @@ -43,7 +43,7 @@ import type { */ export function assertQueryCapabilities( filter: RecordFilter | undefined, - capabilities: StackCapabilities, + capabilities: Pick, ): void { const { content: reach, contentPresent, search } = capabilities.filter; if (filter?.search && !search) { @@ -72,7 +72,7 @@ export function assertQueryCapabilities( 'filter.content', ); } - if (arms.length > 0) assertAdmitted(capabilities, 'filter.contentByType', arms.length, 'arms'); + assertAdmitted(capabilities.filter.contentByType, 'filter.contentByType', arms.length, 'arms'); if (present.length > 0 && !contentPresent) { throw new StackBadRequestError( 'Query uses filter.contentPresent, but this adapter does not declare the ' + @@ -159,35 +159,22 @@ export function filtersContent(features: Pick): boo return features.filter.content !== 'none'; } -/** The query shapes a per-Type cap governs, each named for its capability. */ -export type PerTypeShape = 'filter.contentByType' | 'sort.contentFieldByType'; - /** - * The cap an adapter holds a per-Type shape to: as declared, except `0` - * below `filter.content: 'none'`, since both shapes travel in the - * POST /records/query body a server reaching no content does not serve. - * See docs/spec/adapters.md § Adapter capabilities. + * Whether a per-Type cap admits a shape of `keys` keys: `null` states no + * limit, and `0` admits none. See docs/spec/adapters.md § Adapter + * capabilities. */ -export function perTypeCap(capabilities: StackCapabilities, shape: PerTypeShape): number | null { - if (!filtersContent(capabilities)) return 0; - return shape === 'filter.contentByType' - ? capabilities.filter.contentByType - : capabilities.sort.contentFieldByType; -} - -/** Whether `cap`, a perTypeCap(), admits `keys` keys: `null` states no limit. */ export function admitsKeys(cap: number | null, keys: number): boolean { return cap === null || keys <= cap; } -/** Refuses a per-Type shape of `keys` keys its perTypeCap() does not admit. */ +/** Refuses a per-Type shape of `keys` keys that `cap` does not admit. */ function assertAdmitted( - capabilities: StackCapabilities, - shape: PerTypeShape, + cap: number | null, + shape: 'filter.contentByType' | 'sort.contentFieldByType', keys: number, unit: 'arms' | 'keys', ): void { - const cap = perTypeCap(capabilities, shape); if (admitsKeys(cap, keys)) return; throw new StackBadRequestError( cap === 0 @@ -352,12 +339,12 @@ export function normalizeSort( */ export function assertSortCapability( sort: QuerySort | undefined, - capabilities: StackCapabilities, + capabilities: Pick, ): void { if (!sort) return; if (sort.contentFieldByType !== undefined) { const keys = Object.keys(sort.contentFieldByType).length; - assertAdmitted(capabilities, 'sort.contentFieldByType', keys, 'keys'); + assertAdmitted(capabilities.sort.contentFieldByType, 'sort.contentFieldByType', keys, 'keys'); return; } if (sort.contentField !== undefined) { diff --git a/packages/core/tests/query-capabilities.test.ts b/packages/core/tests/query-capabilities.test.ts index 7527b3c3..c04f1cac 100644 --- a/packages/core/tests/query-capabilities.test.ts +++ b/packages/core/tests/query-capabilities.test.ts @@ -15,7 +15,6 @@ import { assertSortCapability, assertValidContentByType, assertValidSort, - perTypeCap, } from '../src/query-validation.js'; import { StackBadRequestError } from '../src/errors.js'; import type { @@ -162,22 +161,6 @@ describe('a refused query names the capability it needed', () => { ).not.toThrow(); }); - // Both shapes travel in the POST /records/query body, which a reach of - // 'none' does not serve, so a cap declared beside it admits nothing. - test('a per-Type cap admits nothing beside a reach of none', () => { - const none = without({ - filter: { content: 'none', contentByType: null }, - sort: { contentFieldByType: 16 }, - }); - expect(perTypeCap(none, 'filter.contentByType')).toBe(0); - expect(perTypeCap(none, 'sort.contentFieldByType')).toBe(0); - expect( - refusalFor(() => - assertSortCapability({ contentFieldByType: { 'com.example/x@1': { field: 'a' } } }, none), - ), - ).toBe('sort.contentFieldByType'); - }); - test('a null per-Type cap admits any number of keys', () => { const typeIds = Array.from({ length: 500 }, (_, i) => `com.example/x@${i + 1}`); const filter = { contentByType: Object.fromEntries(typeIds.map((id) => [id, {}])) }; diff --git a/packages/sqlite-shared/src/query.ts b/packages/sqlite-shared/src/query.ts index 973c07cc..07210969 100644 --- a/packages/sqlite-shared/src/query.ts +++ b/packages/sqlite-shared/src/query.ts @@ -119,34 +119,36 @@ const rows = (s: Statement, columns: readonly string[], tuples: unknown[][]): st }; /** The columns of a content condition; see contentRows(). */ -const CONDITION_COLUMNS = ['scope', 'path', 'depth', 'op', 'val', 'kind']; +const CONDITION_COLUMNS = ['scope', 'path', 'op', 'val', 'kind']; /** * A JSON value's type as json_each reports it, with integer and real as one * number: what an `equals` condition compares besides the value itself, as - * `===` does, so `true` never matches `1` and `'1'` never matches either. + * `===` does. valueKind() reads a filter value the same way. */ const kindOf = (type: string): string => `CASE ${type} WHEN 'integer' THEN 'number' WHEN 'real' THEN 'number' ELSE ${type} END`; /** - * Whether the condition row `c` holds for the record row `r`. The path is - * walked as data, a recursive CTE matching one segment per step against - * json_each's `key`, so no key is read as syntax and the text is the same - * for any path; one walk gathers what every op asks of the values it - * reaches. Each op is element-wise, so a path reaching a null and a value - * meets both `isNull` and `present`. See docs/spec/data-model.md § Filter. + * Whether condition row `c` holds for record row `r`: one walk of the path + * as data, a segment matched against json_each's `key` per step, gathering + * what each op asks of the values it reaches. See docs/spec/adapters.md + * § Statement size and docs/spec/data-model.md § Filter. */ -const holds = (c: string, r: string): string => - `(WITH RECURSIVE w(d, v, t) AS (SELECT 0, ${r}.content, 'object' UNION ALL ` + - `SELECT w.d + 1, e.value, e.type FROM w JOIN json_each(${members('w')}) m ` + - `JOIN json_each(${spread('m')}) e ` + - `WHERE w.d < ${c}.depth AND m.key = json_extract(${c}.path, '$[' || w.d || ']')) ` + - `SELECT CASE ${c}.op ` + - `WHEN 'equals' THEN IFNULL(MAX(${kindOf('w.t')} = ${c}.kind AND w.v = ${c}.val), 0) ` + - "WHEN 'present' THEN IFNULL(MAX(w.t != 'null'), 0) " + - "ELSE IFNULL(MAX(w.t = 'null'), 0) OR NOT IFNULL(MAX(w.t != 'null'), 0) END " + - `FROM w WHERE w.d = ${c}.depth)`; +const holds = (c: string, r: string): string => { + const depth = `json_array_length(${c}.path)`; + return ( + `(WITH RECURSIVE w(d, v, t) AS (SELECT 0, ${r}.content, 'object' UNION ALL ` + + `SELECT w.d + 1, e.value, e.type FROM w JOIN json_each(${members('w')}) m ` + + `JOIN json_each(${spread('m')}) e ` + + `WHERE w.d < ${depth} AND m.key = json_extract(${c}.path, '$[' || w.d || ']')) ` + + `SELECT CASE ${c}.op ` + + `WHEN 'equals' THEN IFNULL(MAX(${kindOf('w.t')} = ${c}.kind AND w.v = ${c}.val), 0) ` + + "WHEN 'present' THEN IFNULL(MAX(w.t != 'null'), 0) " + + "WHEN 'isNull' THEN IFNULL(MAX(w.t = 'null'), 0) OR NOT IFNULL(MAX(w.t != 'null'), 0) " + + `END FROM w WHERE w.d = ${depth})` + ); +}; /** The join alias the content index is read through. */ const SORT_ALIAS = 'cs'; @@ -173,7 +175,7 @@ const contentCursor = (cursor: DecodedCursor, field: string) => { return cursor; }; -/** A filter value's kindOf(), or null for one no stored scalar can equal. */ +/** A filter value's kindOf(), which it mirrors, or null for one nothing equals. */ const valueKind = (value: unknown): string | null => { if (typeof value === 'boolean') return `${value}`; if (typeof value === 'number') return 'number'; @@ -182,15 +184,14 @@ const valueKind = (value: unknown): string | null => { /** * A `content` and `contentPresent` predicate as condition rows scoped to - * `scope`: path segments, their count, the op, and for `equals` the value - * and its kindOf() (none for an object or array, which `===` never - * matches). A `null` value means "no value at the path, or a value that is - * null" (docs/spec/data-model.md § Filter). + * `scope`: path segments, the op, and for `equals` the value and its + * valueKind(). A `null` value means "no value at the path, or a value that + * is null" (docs/spec/data-model.md § Filter). */ const contentRows = (scope: string | null, predicate: ContentArm): unknown[][] => { const row = (key: string, op: string, value: unknown = null) => { const path = parseContentFilterKey(key); - return [scope, path, path.length, op, value, valueKind(value)]; + return [scope, path, op, value, valueKind(value)]; }; return [ ...Object.entries(predicate.content ?? {}).map(([key, value]) => diff --git a/packages/wire-types/src/discovery.ts b/packages/wire-types/src/discovery.ts index a6bd88f0..d6a0b55c 100644 --- a/packages/wire-types/src/discovery.ts +++ b/packages/wire-types/src/discovery.ts @@ -54,12 +54,9 @@ const SORT_FIELDS: ReadonlySet = new Set(NATIVE_SORT_FIELDS); /** * Resolves anything absent, malformed or unrecognized to the least capable - * value it could stand for — one rule, applied once, rather than a default - * per key at each call site. A cap or limit of `null` states no limit, - * and an absent one reads as the least it could stand for: `0` for a cap, - * which declines a shape, and `null` for a limit, which gates nothing. A - * cap is read as stated whatever the reach; perTypeCap() in core applies - * the reach. See docs/spec/adapters.md § Adapter capabilities. + * value it could stand for, and what depends on the content reach to + * nothing below `'none'` — applied once, where capabilities enter, so no + * reader re-derives either. See docs/spec/adapters.md § Adapter capabilities. */ export function normalizeCapabilities( capabilities: DiscoveryCapabilities | undefined, @@ -70,17 +67,20 @@ export function normalizeCapabilities( const reach = CONTENT_FILTER_REACHES.has(filter?.content as string) ? (filter?.content as ContentFilterReach) : 'none'; + // These travel in the POST /records/query body, which a server reaching + // no content does not serve. + const reaches = reach !== 'none'; return { filter: { content: reach, - contentPresent: filter?.contentPresent === true, - contentByType: readCap(filter?.contentByType), + contentPresent: reaches && filter?.contentPresent === true, + contentByType: reaches ? readCap(filter?.contentByType) : 0, search: filter?.search === true, }, sort: { fields: (sort?.fields ?? []).filter((f): f is NativeSortField => SORT_FIELDS.has(f)), contentField: sort?.contentField === true, - contentFieldByType: readCap(sort?.contentFieldByType), + contentFieldByType: reaches ? readCap(sort?.contentFieldByType) : 0, }, limits: { attachmentBytes: readLimit(limits?.attachmentBytes), diff --git a/packages/wire-types/tests/discovery.test.ts b/packages/wire-types/tests/discovery.test.ts index 5a7f60ed..30eeb033 100644 --- a/packages/wire-types/tests/discovery.test.ts +++ b/packages/wire-types/tests/discovery.test.ts @@ -100,15 +100,19 @@ describe('normalizeCapabilities', () => { expect([read.filter.contentByType, read.sort.contentFieldByType]).toEqual([null, null]); }); - // The reach is applied where a cap is used, by perTypeCap() in core, so - // reading stays one rule per entry. - it('reads a per-Type cap as stated beside a reach of none', () => { + // These travel in the POST /records/query body, which a server reaching + // no content does not serve, so what it declares for them is unhonorable. + it('reads what depends on the content reach as declined beside a reach of none', () => { for (const cap of [16, null]) { const read = normalizeCapabilities({ - filter: { content: 'none', contentByType: cap }, + filter: { content: 'none', contentPresent: true, contentByType: cap }, sort: { contentFieldByType: cap }, }); - expect([read.filter.contentByType, read.sort.contentFieldByType]).toEqual([cap, cap]); + expect([ + read.filter.contentPresent, + read.filter.contentByType, + read.sort.contentFieldByType, + ]).toEqual([false, 0, 0]); } }); From 9bdd74af22efbee98b30ffbe76e55c997a9193e2 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 21 Oct 2026 22:07:01 +0000 Subject: [PATCH 8/9] feat(core,wire-types): per-Type shapes are flags, not counts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The per-Type caps were the only capabilities holding a count, and the statement limits that once justified one are gone. Nothing else a query carries is counted — content keys, presence paths and tag lists are all unbounded, and a server bounds their cost by its query deadline — so filter.contentByType and sort.contentFieldByType are now booleans, like filter.contentPresent. A collection facing false queries its handle's version alone and reports skipped, as it did at a cap of 0; partial narrowing goes, and with it admitsKeys(), the null cap and the 0/null absence asymmetry. Also: a Stack test ties the 33-segment path cap to write validation, and the conformance suite checks everything that depends on the content reach in one test. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Mf3p8BVtrQJLZx1JFCHchC --- .changeset/fixed-statement-size.md | 2 +- docs/spec/adapters.md | 12 ++--- docs/spec/data-model.md | 6 +-- docs/spec/wire-format.md | 8 +-- packages/adapter-api/tests/api.test.ts | 6 +-- packages/adapter-api/tests/helpers.ts | 4 +- packages/adapter-conformance/src/record.ts | 34 +++++------- .../conformance-fixtures/src/discovery.ts | 48 +++++------------ packages/conformance-fixtures/src/query.ts | 4 +- packages/core/src/adapter/index.ts | 1 - packages/core/src/collection.ts | 17 +++--- packages/core/src/query-validation.ts | 42 +++++---------- packages/core/src/testing.ts | 8 +-- packages/core/src/types/capabilities.ts | 21 ++++---- packages/core/tests/collection.test.ts | 16 ++---- packages/core/tests/combine.test.ts | 4 +- .../core/tests/query-capabilities.test.ts | 36 +++++-------- packages/core/tests/stack.test.ts | 32 ++++++++--- .../tests/record.test.ts | 12 ++--- packages/sqlite-shared/src/record-adapter.ts | 6 +-- packages/wire-types/src/discovery.ts | 31 ++++------- packages/wire-types/tests/changes.test.ts | 4 +- packages/wire-types/tests/discovery.test.ts | 54 ++++++++----------- 23 files changed, 166 insertions(+), 242 deletions(-) diff --git a/.changeset/fixed-statement-size.md b/.changeset/fixed-statement-size.md index a7661352..a8b1d9de 100644 --- a/.changeset/fixed-statement-size.md +++ b/.changeset/fixed-statement-size.md @@ -9,6 +9,6 @@ A SQLite statement's text and bound parameters no longer grow with a query's size. Every caller-sized part of a query binds as one table of rows: lists, permission sets, `sort.contentFieldByType`'s entries, and the content conditions — every `filter.content` and `filter.contentPresent` key and every `filter.contentByType` arm — whose paths one recursive walk per condition resolves. Durable Object SQLite no longer fails a query with `too many SQL variables` or `statement too long`, however many per-Type keys or content conditions it carries. -The per-Type caps `filter.contentByType` and `sort.contentFieldByType` are now `number | null`, `null` stating no limit. `MemoryAdapter` and the SQLite adapters declare `null`, since nothing bounds the count locally; a cap is for a server bounding its own cost. `null` means the same on a cap and a limit: no limit stated, with the server's own bound still answering. `normalizeCapabilities()` reads a stated `null` cap as no limit and an absent or malformed one as `0`, and now reads `filter.contentPresent` as `false` beside `filter.content: 'none'`, as it already read both caps as `0`. `admitsKeys(cap, keys)` in `@haverstack/core/adapter` is the one comparison against a cap. A discovery fixture pins a stated `null` cap. +`filter.contentByType` and `sort.contentFieldByType` are now booleans, like `filter.contentPresent`: a per-Type shape holds any number of keys, as every other part of a query does, and a server bounds what one costs it by its query deadline. A collection facing `false` queries its handle's version alone and reports the rest as `skipped`, as it did at a cap of `0`; partial narrowing to the newest versions that fit is gone. `normalizeCapabilities()` reads both as `true` only when stated, and now reads `filter.contentPresent` as `false` beside `filter.content: 'none'`, as it reads both per-Type flags. A content filter value now matches only a value of its own JSON type in the SQLite adapters, as `===` does: `true` no longer matches `1`. A content filter path may be 33 segments, the depth that reaches the deepest value a write accepts. The conformance suite holds every adapter to both, and to a multi-valued path satisfying both `contentPresent` and a `null` filter. diff --git a/docs/spec/adapters.md b/docs/spec/adapters.md index 2b8bfdd4..3c712ced 100644 --- a/docs/spec/adapters.md +++ b/docs/spec/adapters.md @@ -174,7 +174,7 @@ A missing row is not only a stale read. `deleteUnreferencedAttachmentRecords()` ### Statement size -**A statement's text and its bound parameters never grow with how many entries a query's keys hold.** Durable Object SQLite caps a statement at 100 parameters and 100 KB of text, and a query's size is not the adapter's to choose: a reach is sized by the requester's grants and rosters, a filter by whoever wrote it. A statement that grew with either would fail on one engine for a query another answers, and a declared [cap](#adapter-capabilities) could not promise anything. **Every caller-sized part of a query is a table of rows**, bound as one JSON array of tuples and read through `json_each` as named columns: +**A statement's text and its bound parameters never grow with how many entries a query's keys hold.** Durable Object SQLite caps a statement at 100 parameters and 100 KB of text, and a query's size is not the adapter's to choose: a reach is sized by the requester's grants and rosters, a filter by whoever wrote it. A statement that grew with either would fail on one engine for a query another answers, and a declared [capability](#adapter-capabilities) could not promise anything. **Every caller-sized part of a query is a table of rows**, bound as one JSON array of tuples and read through `json_each` as named columns: - **A list is rows of one column** — a filter's type, app and author lists and its tags, and a reach's type lists. A list of one value binds as that value instead, so an index on its column keeps yielding rows in order; that is a smaller statement, never a larger one. - **A reach's permission sets** are rows of one element each, naming the set it belongs to and that set's size. @@ -192,13 +192,13 @@ type StackCapabilities = { filter: { content: 'none' | 'field' | 'path'; // how far a content filter key may reach contentPresent: boolean; // filter.contentPresent is honored - contentByType: number | null; // most arms filter.contentByType may hold; null = no limit, 0 = not honored + contentByType: boolean; // filter.contentByType is honored search: boolean; // filter.search is honored }; sort: { fields: ('createdAt' | 'updatedAt' | 'version')[]; // native columns this adapter can order by contentField: boolean; // sort.contentField is honored - contentFieldByType: number | null; // most keys sort.contentFieldByType may hold; null = no limit, 0 = not honored + contentFieldByType: boolean; // sort.contentFieldByType is honored }; limits: { attachmentBytes: number | null; // upload size ceiling, or null = client can't pre-check @@ -219,15 +219,15 @@ One type, `StackCapabilities`, names both sides: an adapter declares it as `capa **`filter.contentPresent` sits beside the ladder rather than on it.** It gates `filter.contentPresent`, and is required `true` for local adapters on the same reasoning: asking whether a path holds a value is the same walk with a weaker predicate. It is not a rung because a server promising to match a content value has not thereby promised to answer whether one is there at all. It is meaningful only above `'none'` — presence travels in the `POST /records/query` body, which a server reaching no content does not expose — and a presence path is itself gated by the rung, so a multi-segment one needs `'path'`. -**`filter.contentByType` and `sort.contentFieldByType` are caps, not flags.** Each is the most keys the [per-Type shape](./data-model.md#content-by-type) may hold: `null` states no limit, and `0` declines the shape. **A cap states a real limit or none at all.** A local adapter declares `null` for both, on the reasoning that makes `'path'` required: an arm is the content filter it already runs, scoped to one `typeId`, and costs a SQLite adapter no statement text or parameter of its own (see [Statement size](#statement-size)), so nothing bounds the count. A cap is for a server, which bounds what a request may cost it — see [Wire format § Bounding query cost](./wire-format.md#bounding-query-cost). A server may declare any cap, `0` and `null` included, and a collection narrows the versions past it rather than failing the query — see [Data model § Content queries across versions](./data-model.md#content-queries-across-versions). Both are meaningful only above `filter.content: 'none'`, since they travel in the `POST /records/query` body, and are `0` beside it — as `filter.contentPresent` is `false`. That holds where capabilities enter, so nothing reading them re-derives it: the [conformance suite](#conformance) holds a local adapter's declaration to it, and `normalizeCapabilities()` applies it to a discovery response. `admitsKeys()` in `@haverstack/core/adapter` is the one comparison against a cap. +**`filter.contentByType` and `sort.contentFieldByType` are flags, like `filter.contentPresent`.** Each says the [per-Type shape](./data-model.md#content-by-type) is honored, with any number of keys. A local adapter must honor both, on the reasoning that makes `'path'` required: an arm is the content filter it already runs, scoped to one `typeId`, and costs a SQLite adapter no statement text or parameter of its own (see [Statement size](#statement-size)). Neither carries a count, as nothing else a query holds does — `filter.content` keys, `contentPresent` paths and tag lists are unbounded too: a server bounds what a query costs it the one way it bounds all of them, under [Wire format § Bounding query cost](./wire-format.md#bounding-query-cost). A collection facing `false` queries its handle's version alone rather than failing — see [Data model § Content queries across versions](./data-model.md#content-queries-across-versions). Both are meaningful only above `filter.content: 'none'`, since they travel in the `POST /records/query` body, and are `false` beside it, as `filter.contentPresent` is. That holds where capabilities enter, so nothing reading them re-derives it: the [conformance suite](#conformance) holds a local adapter's declaration to it, and `normalizeCapabilities()` applies it to a discovery response. **`filter.search` has no local-required rule.** A full-text index is a different mechanism from field matching, and neither implies the other, so a local adapter may legitimately decline it (see the JSON adapter note below). **`sort.contentField` is a boolean, and `sort.fields` names native columns only.** Content fields are app-defined and unbounded, so an adapter cannot enumerate them; and an adapter that indexes content for sorting indexes [every top-level scalar](./data-model.md#sorting-by-a-content-field), so there is nothing per-field left to declare. `sort.contentField` is required `true` for local adapters, and independent of `filter.content`: the two describe ordering and filtering, and a server may offer either without the other — filtering is answerable at whatever scale a server already scans at, where ordering wants an index. Both are enforced: `Stack.query()` refuses a sort the adapter hasn't declared rather than letting it be answered in some other order, which a caller reading one bounded page cannot detect. -**Absent, malformed or unrecognized reads as the least capable value it could stand for.** One rule, for every entry: `'none'` for the reach, `false` for a flag, `0` for a cap, nothing for `sort.fields`, `null` for a limit. A discovery response that omits a group, omits `capabilities` entirely, or names a reach this client has never heard of describes a server this client will not send that query to — silence is never a claim, in either direction, and an unrecognized rung places nowhere on a ladder so it can only be read as its bottom. `normalizeCapabilities()` in `@haverstack/wire-types` applies the rule, so a client reads a foreign discovery response through one function rather than defaulting keys at each use. +**Absent, malformed or unrecognized reads as the least capable value it could stand for.** One rule, for every entry: `'none'` for the reach, `false` for a flag, nothing for `sort.fields`, `null` for a limit. A discovery response that omits a group, omits `capabilities` entirely, or names a reach this client has never heard of describes a server this client will not send that query to — silence is never a claim, in either direction, and an unrecognized rung places nowhere on a ladder so it can only be read as its bottom. `normalizeCapabilities()` in `@haverstack/wire-types` applies the rule, so a client reads a foreign discovery response through one function rather than defaulting keys at each use. -Two things this does not refuse. A query naming no sort asks for the server's own default order rather than a stated one, so it claims nothing that has to be declared and runs against an empty `sort.fields`. And `null` states no limit, on a cap and a limit alike: the client checks nothing before sending, and the server's own bound still answers — `413` for a request's size, `503` for its [cost](./wire-format.md#bounding-query-cost). Only absence reads differently between the two, and by the rule above rather than an exception to it: the least a cap could stand for is `0`, which declines a shape, and the least a limit could stand for is `null`, since a limit gates nothing. +Two things this does not refuse. A query naming no sort asks for the server's own default order rather than a stated one, so it claims nothing that has to be declared and runs against an empty `sort.fields`. And a `null` limit is permissive by design: it says this client cannot pre-check, not that nothing is enforced — the server's own ceiling still answers with a `413`. **Per-adapter notes:** diff --git a/docs/spec/data-model.md b/docs/spec/data-model.md index 4e3a74d7..8e0c393f 100644 --- a/docs/spec/data-model.md +++ b/docs/spec/data-model.md @@ -476,7 +476,7 @@ The derived types are a convenience over [runtime validation](#types), which sta A field the handle declares optional and no step produces is read under its own name, since a migration carries every field no step names — an older version may have gained it in place — and a name a step removed is the constant "no value". A `map` keeps a value it does not list, so more than one older value can arrive as the one a key asks for: `map('status', { done: 'finished' })` carries an unlisted `finished` through as well as remapping `done`. An arm matches one value per path, so it asks for a mapped value first, and each other source becomes a residual the first page probes for, as it probes a narrowed version: a match sets `skipped.older`. One such key per arm is split this way; a second narrows the version. A key no older value maps to drops the older arm, as a constant that does not match does. A key on a field the handle does not declare, which only an untyped caller can write, is narrowed. **A narrowed version is left out of the query**; a dropped arm is left out too, but only because no record there can match. Newer versions stay out of reach either way: the handle cannot represent them. -Records at an older version arrive migrated, as on an unfiltered page, and a sort orders them as their migrated values would order, since a carried field keeps its kind. The first page fixes which versions the query spans, and its cursor carries them: a later page queries the same arms and sort without listing types again, so a version defined mid-pagination cannot change what a cursor resumes. When no older version is left in, the query is sent as a plain `content` filter or `contentField` sort on the handle's `typeId`, which every adapter reaching content answers. An adapter declaring [`filter.contentByType`](./adapters.md#adapter-capabilities) `0` — or `sort.contentFieldByType` `0`, for a content sort — gets that query, with every older version narrowed. One declaring a lower cap than the family has versions to query gets the newest versions that fit, and the rest are narrowed. +Records at an older version arrive migrated, as on an unfiltered page, and a sort orders them as their migrated values would order, since a carried field keeps its kind. The first page fixes which versions the query spans, and its cursor carries them: a later page queries the same arms and sort without listing types again, so a version defined mid-pagination cannot change what a cursor resumes. When no older version is left in, the query is sent as a plain `content` filter or `contentField` sort on the handle's `typeId`, which every adapter reaching content answers. An adapter declaring [`filter.contentByType`](./adapters.md#adapter-capabilities) `false` — or `sort.contentFieldByType` `false`, for a content sort — gets that query, with every older version narrowed. **`skipped` says which side the query could not reach.** The first page sets `skipped.older` when records at a narrowed older version match the rest of the query, and `skipped.newer` when records at a newer version do: the results may then be missing records the filter was meant to find. A version is reported only when the filter or sort touches a field that does not translate across some step between it and the handle, or when a residual of its arm matches — a dropped arm is not a skip. A later page carries no `skipped`, since the flags depend on the query, not the page. Each is one `limit: 1` read, run beside the first page's own, and a scoped read that comes back empty with a cursor still set counts as a match rather than reading on, so a flag can be set when nothing was skipped but is never clear when something was. Mixed versions are a lasting state, not a backlog, since an app at an older version may keep writing its own, so the app decides what `skipped.older` means to it; `migrateAll()` clears it only until the next such write. `skipped.newer` says the app is behind, which an unfiltered page reports as misfits; a content query's page carries none. A query with no content keys reads the whole family. The untyped `query({ filter: { baseId, content } })` still matches across the family, returning records as stored. @@ -655,7 +655,7 @@ By default, `query()` (like `get()`) returns Records exactly as stored — see [ - **Every arm is pinned to a `typeId`**, and a filter carrying `contentByType` names at least one. An empty map is refused with `StackBadRequestError` (400), so the key cannot encode to a filter that matches everything. - **An arm is flat**: `content` and `contentPresent`, nothing else, and no nesting. An arm naming neither matches every Record at its `typeId`, which is what `typeId` alone matches. - **It never sits beside a top-level `content` or `contentPresent`**: a filter with both is refused with 400, since which one applies to a Record would be ambiguous. Every other key applies to every arm, and a `typeId` beside it intersects with its keys. -- **It is bounded** by the adapter's `filter.contentByType`, the most arms it takes, where it states a limit; a filter with more is refused with 400 naming that capability (see [Capability-gated filters](#capability-gated-filters)). +- **It is gated** by the adapter's `filter.contentByType`; an adapter declaring `false` refuses it with 400 naming that capability (see [Capability-gated filters](#capability-gated-filters)). It holds any number of arms, as `content` holds any number of keys. It discloses nothing new: each arm is a query the caller could already run, its `typeId` with its content filter, and the permission check applies per Record as for any query. Types partition Records, so no Record is matched by two arms, and the cost of _n_ arms stays within what _n_ queries would cost — what [Wire format § Bounding query cost](./wire-format.md#bounding-query-cost) already leaves to the server. **It is not a first step toward general OR.** A disjunction across Types is the only one a migration chain needs, and match-any forms stay refused for the reason [above](#filter): a filter that can encode to nothing can silently widen a query when it crosses the wire. @@ -754,7 +754,7 @@ Locale-correct ordering is out of reach for a stored key regardless of effort, b **`contentPresent` needs `filter.contentPresent` on top of a content reach**, a separate flag on the same reasoning the reach ladder rests on: a server promising to match a content value has not thereby promised to answer whether one is there, and reading it as such would hand a client the unfiltered superset that ignoring the filter produces. A multi-segment presence path needs the `'path'` rung too, exactly as a filter key does. -**`contentByType` and `contentFieldByType` need their own capabilities**, `filter.contentByType` and `sort.contentFieldByType`: each is the most keys the adapter takes, `null` for no limit, and `0` declines the shape. An arm's keys are gated as the same keys at the top of a filter are, so a multi-segment path in an arm needs `'path'`, and an arm's `contentPresent` needs `filter.contentPresent`. +**`contentByType` and `contentFieldByType` need their own capabilities**, `filter.contentByType` and `sort.contentFieldByType`: each says whether the adapter honors the shape, which holds any number of keys. An arm's keys are gated as the same keys at the top of a filter are, so a multi-segment path in an arm needs `'path'`, and an arm's `contentPresent` needs `filter.contentPresent`. **A sort is gated the same way.** `sort.contentField` needs the adapter's `sort.contentField`, and a native `sort.field` must appear in its `sort.fields`; a sort an adapter hasn't declared throws `StackBadRequestError` rather than being answered in some other order — which a caller reading one bounded page has no way to notice. `sort.contentField` and `filter.content` are independent: a server may order by a content field without offering to filter on one, or the reverse. diff --git a/docs/spec/wire-format.md b/docs/spec/wire-format.md index 2f2f9c03..7fb33d62 100644 --- a/docs/spec/wire-format.md +++ b/docs/spec/wire-format.md @@ -16,11 +16,11 @@ GET /.well-known/stack "entityId": "did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK", "timezone": "America/New_York", "capabilities": { - "filter": { "content": "path", "contentPresent": true, "contentByType": 16, "search": true }, + "filter": { "content": "path", "contentPresent": true, "contentByType": true, "search": true }, "sort": { "fields": ["createdAt", "updatedAt", "version"], "contentField": true, - "contentFieldByType": 16 + "contentFieldByType": true }, "limits": { "attachmentBytes": 52428800, "contentBytes": 1048576 } }, @@ -330,7 +330,7 @@ This is what lets a client report a mutation's outcome without a second read, an **`filter.contentPresent` travels in the `POST /records/query` body only**, like `filter.content` itself — there is no `GET /records` spelling, since a server exposing that endpoint alone declares `filter.content: "none"` and offers neither. A server declaring `filter.contentPresent: false` MUST refuse the filter with `400` rather than answer without it. -**`filter.contentByType` and `sort.contentFieldByType` travel in the `POST /records/query` body only**, for the reason `filter.contentPresent` does. A server declaring `0` for either MUST refuse it with `400`, and one receiving more keys than it declares MUST refuse the request with `400` rather than answer some of the arms. An empty map, an arm carrying anything but `content` and `contentPresent`, a `contentByType` beside a top-level `content` or `contentPresent`, and a sort key that is neither `{ "field": string }` nor `{ "value": string | number | null }` are `400` as well. See [Data model § Content by type](./data-model.md#content-by-type). +**`filter.contentByType` and `sort.contentFieldByType` travel in the `POST /records/query` body only**, for the reason `filter.contentPresent` does. A server declaring `false` for either MUST refuse it with `400`. An empty map, an arm carrying anything but `content` and `contentPresent`, a `contentByType` beside a top-level `content` or `contentPresent`, and a sort key that is neither `{ "field": string }` nor `{ "value": string | number | null }` are `400` as well. See [Data model § Content by type](./data-model.md#content-by-type). **A sort names a native column or a content field, and the parameter it arrives in says which** — the same shape the relationship filter uses above, and the only way to keep a content field named `version` distinct from the native column on a raw query string. A request carrying both `sort` and `sortContent` MUST be rejected with `400` rather than resolved in one direction; in a `POST /records/query` body the same two spellings are `sort.field` and `sort.contentField`, and `sort.contentFieldByType` is a third that no request may combine with either. A server declaring `sort.contentField: false` refuses `sortContent` with `400`. What the ordering itself must be — absent values last, numbers before text, and the [folded text key](./data-model.md#text-ordering) a server MUST reproduce — is in [Data model § Sorting by a content field](./data-model.md#sorting-by-a-content-field). @@ -413,7 +413,7 @@ The record adapters sanitize a `search` string's _complexity_ — wildcards stri The mitigation therefore lives where the engine is driven, not in the sanitizers: run record-adapter queries off the request thread (a worker per connection, or a worker pool), so a slow query can be interrupted — `sqlite3_interrupt`, or terminating the worker — and apply the per-request deadline at that boundary. A server that does none of this is conforming but will serve exactly as well as its slowest query. -A [`filter.contentByType`](./data-model.md#content-by-type) of _n_ arms costs what _n_ content queries would, so it adds nothing to bound but a count, which a server sets by declaring its cap. +A [`filter.contentByType`](./data-model.md#content-by-type) of _n_ arms costs what _n_ content queries would, and is bounded the same way: by this deadline, not by a count of arms. `timeout` is a distinct code because a client acts on it differently from every other failure in the table: nothing was applied, and the same request may well succeed if narrowed or retried. Reusing `bad_request` would say the opposite. See [Data model § Capability-gated filters](./data-model.md#capability-gated-filters). diff --git a/packages/adapter-api/tests/api.test.ts b/packages/adapter-api/tests/api.test.ts index f783e1ae..a5009c29 100644 --- a/packages/adapter-api/tests/api.test.ts +++ b/packages/adapter-api/tests/api.test.ts @@ -1226,8 +1226,8 @@ describe('queryRecords', () => { const adapter = await openAdapter(discovery); expect(adapter.capabilities).toEqual({ - filter: { content: 'none', contentPresent: false, contentByType: 0, search: false }, - sort: { fields: [], contentField: false, contentFieldByType: 0 }, + filter: { content: 'none', contentPresent: false, contentByType: false, search: false }, + sort: { fields: [], contentField: false, contentFieldByType: false }, limits: { attachmentBytes: null, contentBytes: null }, }); @@ -1295,7 +1295,7 @@ describe('queryRecords', () => { ], ] as const)('refuses %s locally where the server declines it', async (capability, query) => { const adapter = await openAdapter( - discoveryWith({ filter: { contentByType: 0 }, sort: { contentFieldByType: 0 } }), + discoveryWith({ filter: { contentByType: false }, sort: { contentFieldByType: false } }), ); await expect(adapter.queryRecords(query)).rejects.toMatchObject({ capability }); expect(mockFetch).toHaveBeenCalledTimes(1); // only the discovery call — no request sent diff --git a/packages/adapter-api/tests/helpers.ts b/packages/adapter-api/tests/helpers.ts index b201d0fa..17cf0d39 100644 --- a/packages/adapter-api/tests/helpers.ts +++ b/packages/adapter-api/tests/helpers.ts @@ -23,11 +23,11 @@ export const OWNER = 'entity-owner-123'; * limit and not the other is the case the null default exists for. */ export const FULL_CAPABILITIES = { - filter: { content: 'path', contentPresent: true, contentByType: 64, search: true }, + filter: { content: 'path', contentPresent: true, contentByType: true, search: true }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 64, + contentFieldByType: true, }, limits: { attachmentBytes: 52428800 }, } satisfies DiscoveryCapabilities; diff --git a/packages/adapter-conformance/src/record.ts b/packages/adapter-conformance/src/record.ts index bc2828c8..66a1ecf1 100644 --- a/packages/adapter-conformance/src/record.ts +++ b/packages/adapter-conformance/src/record.ts @@ -29,7 +29,7 @@ import type { StackQuery, StackRecord, } from '@haverstack/core'; -import { ElementSetMovedError, admitsKeys, inReach } from '@haverstack/core/adapter'; +import { ElementSetMovedError, inReach } from '@haverstack/core/adapter'; import type { Reach, StackRecordAdapter } from '@haverstack/core/adapter'; type PermissionElement = Extract; @@ -81,31 +81,25 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt describe('capabilities', () => { test('declares a well-formed StackCapabilities object', () => { expect(['none', 'field', 'path']).toContain(capabilities.filter.content); - expect(typeof capabilities.filter.contentPresent).toBe('boolean'); - expect(typeof capabilities.filter.search).toBe('boolean'); - expect(Array.isArray(capabilities.sort.fields)).toBe(true); - expect(typeof capabilities.sort.contentField).toBe('boolean'); - for (const cap of [ + for (const flag of [ + capabilities.filter.contentPresent, capabilities.filter.contentByType, + capabilities.filter.search, + capabilities.sort.contentField, capabilities.sort.contentFieldByType, ]) { - expect(cap === null || (Number.isInteger(cap) && cap >= 0)).toBe(true); - } - }); - - test('the per-Type caps are 0 whenever filter.content is "none"', () => { - // Both travel in the body of the endpoint a reach of "none" omits. - if (capabilities.filter.content === 'none') { - expect(capabilities.filter.contentByType).toBe(0); - expect(capabilities.sort.contentFieldByType).toBe(0); + expect(typeof flag).toBe('boolean'); } + expect(Array.isArray(capabilities.sort.fields)).toBe(true); }); - test('filter.contentPresent is false whenever filter.content is "none"', () => { - // contentPresent asks whether a value is there at all — meaningless - // to promise from an adapter that reaches no content to look at. + // Each asks something of content, and travels in the POST + // /records/query body a reach of "none" does not serve. + test('what depends on the content reach is false whenever filter.content is "none"', () => { if (capabilities.filter.content === 'none') { expect(capabilities.filter.contentPresent).toBe(false); + expect(capabilities.filter.contentByType).toBe(false); + expect(capabilities.sort.contentFieldByType).toBe(false); } }); }); @@ -1363,7 +1357,7 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt // ----------------------------------------------------------------- // Content by type — capability-gated on filter.contentByType // ----------------------------------------------------------------- - if (admitsKeys(capabilities.filter.contentByType, 2)) { + if (capabilities.filter.contentByType) { describe('content by type', () => { const OTHER = 'org.haverstack.conformance/other@1'; const THIRD = 'org.haverstack.conformance/third@1'; @@ -1442,7 +1436,7 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt // ----------------------------------------------------------------- // Sort by content field per type — gated on sort.contentFieldByType // ----------------------------------------------------------------- - if (admitsKeys(capabilities.sort.contentFieldByType, 3)) { + if (capabilities.sort.contentFieldByType) { describe('sort by content field per type', () => { const OTHER = 'org.haverstack.conformance/other@1'; const THIRD = 'org.haverstack.conformance/third@1'; diff --git a/packages/conformance-fixtures/src/discovery.ts b/packages/conformance-fixtures/src/discovery.ts index 9eafd153..986f9a83 100644 --- a/packages/conformance-fixtures/src/discovery.ts +++ b/packages/conformance-fixtures/src/discovery.ts @@ -21,13 +21,13 @@ export const discoveryFixtures: ConformanceFixture filter: { content: 'path', contentPresent: true, - contentByType: 16, + contentByType: true, search: true, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 16, + contentFieldByType: true, }, limits: { attachmentBytes: 52428800, @@ -36,30 +36,6 @@ export const discoveryFixtures: ConformanceFixture }, }, }, - { - name: 'discovery-declares-per-type-shapes-without-a-limit', - description: - 'A per-Type cap of null states no limit: the server takes filter.contentByType and ' + - 'sort.contentFieldByType with any number of keys, and a client sends them unrefused. ' + - 'Only a stated null reads this way — an absent cap declines the shape, as 0 does. See ' + - 'docs/spec/adapters.md § Adapter capabilities.', - method: 'GET', - path: '/.well-known/stack', - responseStatus: 200, - responseBody: { - version: WIRE_PROTOCOL_VERSION, - entityId: 'did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK', - capabilities: { - filter: { content: 'path', contentPresent: true, contentByType: null, search: true }, - sort: { - fields: ['createdAt', 'updatedAt', 'version'], - contentField: true, - contentFieldByType: null, - }, - limits: { attachmentBytes: null, contentBytes: null }, - }, - }, - }, { name: 'discovery-advertises-install-requests', description: @@ -73,11 +49,11 @@ export const discoveryFixtures: ConformanceFixture version: WIRE_PROTOCOL_VERSION, entityId: 'did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK', capabilities: { - filter: { content: 'none', contentPresent: false, contentByType: 0, search: false }, + filter: { content: 'none', contentPresent: false, contentByType: false, search: false }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: false, - contentFieldByType: 0, + contentFieldByType: false, }, limits: { attachmentBytes: null, contentBytes: null }, }, @@ -100,13 +76,13 @@ export const discoveryFixtures: ConformanceFixture filter: { content: 'none', contentPresent: false, - contentByType: 0, + contentByType: false, search: false, }, sort: { fields: ['createdAt'], contentField: false, - contentFieldByType: 0, + contentFieldByType: false, }, limits: { attachmentBytes: null, @@ -134,13 +110,13 @@ export const discoveryFixtures: ConformanceFixture filter: { content: 'path', contentPresent: true, - contentByType: 16, + contentByType: true, search: true, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 16, + contentFieldByType: true, }, limits: { attachmentBytes: 52428800, @@ -168,13 +144,13 @@ export const discoveryFixtures: ConformanceFixture filter: { content: 'path', contentPresent: true, - contentByType: 16, + contentByType: true, search: true, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 16, + contentFieldByType: true, }, limits: { attachmentBytes: 52428800, @@ -202,13 +178,13 @@ export const discoveryFixtures: ConformanceFixture filter: { content: 'none', contentPresent: false, - contentByType: 0, + contentByType: false, search: false, }, sort: { fields: ['createdAt'], contentField: false, - contentFieldByType: 0, + contentFieldByType: false, }, limits: { attachmentBytes: null, diff --git a/packages/conformance-fixtures/src/query.ts b/packages/conformance-fixtures/src/query.ts index abf534ab..eda27e44 100644 --- a/packages/conformance-fixtures/src/query.ts +++ b/packages/conformance-fixtures/src/query.ts @@ -176,7 +176,7 @@ export const queryRecordsFixtures: ConformanceFixture< 'filter across the versions of a family whose migrations renamed a field — here `by` at ' + '@1 and `owner` at @2. Types partition Records, so the arms split the query rather than ' + 'widening it, and every other filter key applies to each arm. A server declaring ' + - 'filter.contentByType: 0 MUST refuse it with 400. See docs/spec/data-model.md § Content ' + + 'filter.contentByType: false MUST refuse it with 400. See docs/spec/data-model.md § Content ' + 'by type.', method: 'POST', path: '/records/query', @@ -219,7 +219,7 @@ export const queryRecordsFixtures: ConformanceFixture< 'that Type declares it, or a constant already reduced to a number or text — here 5 for ' + 'every @1 Record, which predates the field. A Record whose typeId is not a key, or whose ' + 'value is null, holds no value and sorts last in either direction. A server declaring ' + - 'sort.contentFieldByType: 0 MUST refuse it with 400. See docs/spec/data-model.md ' + + 'sort.contentFieldByType: false MUST refuse it with 400. See docs/spec/data-model.md ' + '§ Sorting by a content field.', method: 'POST', path: '/records/query', diff --git a/packages/core/src/adapter/index.ts b/packages/core/src/adapter/index.ts index 7dfab385..dc54cdc2 100644 --- a/packages/core/src/adapter/index.ts +++ b/packages/core/src/adapter/index.ts @@ -27,7 +27,6 @@ export { combineAdapters } from './combine.js'; export { inReach } from './reach.js'; export type { ReachedRecord } from './reach.js'; export { - admitsKeys, assertQueryCapabilities, assertSortCapability, assertValidSort, diff --git a/packages/core/src/collection.ts b/packages/core/src/collection.ts index b3911fb7..3ec1133a 100644 --- a/packages/core/src/collection.ts +++ b/packages/core/src/collection.ts @@ -17,7 +17,6 @@ import { StackBadRequestError, StackMigrationError, StackPermissionError } from './errors.js'; import { baseIdOf, parseTypeId } from './schema.js'; -import { admitsKeys } from './query-validation.js'; import { StoredVersionError, WRITE_EXPECTATION } from './stack/write-expectation.js'; import type { ExpectingClient, WriteExpectation } from './stack/write-expectation.js'; import type { MigrationRegistry } from './stack/migrations.js'; @@ -493,11 +492,11 @@ export const createCollection = ( const predicate = contentPredicate(filter); const sortField = sort?.contentField; const { capabilities } = client; - // A version past the adapter's cap is narrowed, newest kept first: - // the older a version, the fewer apps still write it. - const admits = (keys: number) => - (predicate === undefined || admitsKeys(capabilities.filter.contentByType, keys)) && - (sortField === undefined || admitsKeys(capabilities.sort.contentFieldByType, keys)); + // Spanning versions takes the per-Type shape of each content key the + // query carries; an adapter without one gets the handle's version alone. + const spans = + (predicate === undefined || capabilities.filter.contentByType) && + (sortField === undefined || capabilities.sort.contentFieldByType); const sortKind = (() => { const def = sortField === undefined ? undefined : handle.schema[sortField]; return def && def.kind !== 'array' && def.kind !== 'object' ? def.kind : undefined; @@ -535,11 +534,7 @@ export const createCollection = ( : origins ? translateSortKey(origins, sortField, sortKind) : UNTRANSLATABLE; - if ( - translated === UNTRANSLATABLE || - key === UNTRANSLATABLE || - !admits(Object.keys(arms).length + 1) - ) { + if (translated === UNTRANSLATABLE || key === UNTRANSLATABLE || !spans) { narrowed.push(type.id); continue; } diff --git a/packages/core/src/query-validation.ts b/packages/core/src/query-validation.ts index 2e354adb..9a76e27e 100644 --- a/packages/core/src/query-validation.ts +++ b/packages/core/src/query-validation.ts @@ -72,7 +72,13 @@ export function assertQueryCapabilities( 'filter.content', ); } - assertAdmitted(capabilities.filter.contentByType, 'filter.contentByType', arms.length, 'arms'); + if (arms.length > 0 && !capabilities.filter.contentByType) { + throw new StackBadRequestError( + 'Query uses filter.contentByType, but this adapter does not declare the ' + + 'filter.contentByType capability.', + 'filter.contentByType', + ); + } if (present.length > 0 && !contentPresent) { throw new StackBadRequestError( 'Query uses filter.contentPresent, but this adapter does not declare the ' + @@ -159,31 +165,6 @@ export function filtersContent(features: Pick): boo return features.filter.content !== 'none'; } -/** - * Whether a per-Type cap admits a shape of `keys` keys: `null` states no - * limit, and `0` admits none. See docs/spec/adapters.md § Adapter - * capabilities. - */ -export function admitsKeys(cap: number | null, keys: number): boolean { - return cap === null || keys <= cap; -} - -/** Refuses a per-Type shape of `keys` keys that `cap` does not admit. */ -function assertAdmitted( - cap: number | null, - shape: 'filter.contentByType' | 'sort.contentFieldByType', - keys: number, - unit: 'arms' | 'keys', -): void { - if (admitsKeys(cap, keys)) return; - throw new StackBadRequestError( - cap === 0 - ? `Query uses ${shape}, but this adapter does not declare the ${shape} capability.` - : `Query uses ${keys} ${shape} ${unit}, but this adapter takes at most ${cap}.`, - shape, - ); -} - /** * The longest path that can reach a stored value: a top-level field sits at * depth 0 and each segment descends one level, so a path past this reaches @@ -343,8 +324,13 @@ export function assertSortCapability( ): void { if (!sort) return; if (sort.contentFieldByType !== undefined) { - const keys = Object.keys(sort.contentFieldByType).length; - assertAdmitted(capabilities.sort.contentFieldByType, 'sort.contentFieldByType', keys, 'keys'); + if (!capabilities.sort.contentFieldByType) { + throw new StackBadRequestError( + 'Query uses sort.contentFieldByType, but this adapter does not declare the ' + + 'sort.contentFieldByType capability.', + 'sort.contentFieldByType', + ); + } return; } if (sort.contentField !== undefined) { diff --git a/packages/core/src/testing.ts b/packages/core/src/testing.ts index 3658b723..f81f7ce6 100644 --- a/packages/core/src/testing.ts +++ b/packages/core/src/testing.ts @@ -83,13 +83,13 @@ export class MemoryAdapter implements StackAdapter { filter: { content: 'path', contentPresent: true, - contentByType: null, + contentByType: true, search: false, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: null, + contentFieldByType: true, }, limits: { attachmentBytes: null, @@ -725,13 +725,13 @@ export class IncapableMemoryAdapter extends MemoryAdapter { filter: { content: 'none', contentPresent: false, - contentByType: 0, + contentByType: false, search: false, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: false, - contentFieldByType: 0, + contentFieldByType: false, }, limits: { attachmentBytes: null, diff --git a/packages/core/src/types/capabilities.ts b/packages/core/src/types/capabilities.ts index 94ad7a1c..4ca216b2 100644 --- a/packages/core/src/types/capabilities.ts +++ b/packages/core/src/types/capabilities.ts @@ -40,12 +40,12 @@ export type StackCapabilities = { */ contentPresent: boolean; /** - * The most arms `filter.contentByType` may hold: `null` for no limit, - * `0` to decline it. A collection narrows the versions past a limit - * rather than failing. Local adapters declare `null`. See + * Whether `filter.contentByType` is honored. Local adapters must; a + * collection facing `false` queries its handle's version alone rather + * than failing. Meaningful only where `content` is not `'none'`. See * docs/spec/adapters.md § Adapter capabilities. */ - contentByType: number | null; + contentByType: boolean; /** * Whether `filter.search` is honored. Not a rung on the ladder: a * full-text index is a different mechanism from field matching, and @@ -62,17 +62,14 @@ export type StackCapabilities = { * scalar. Independent of `filter.content`. */ contentField: boolean; - /** - * The most keys `sort.contentFieldByType` may hold: `null` for no limit, - * `0` to decline it, as `filter.contentByType`. - */ - contentFieldByType: number | null; + /** Whether `sort.contentFieldByType` is honored, as `filter.contentByType`. */ + contentFieldByType: boolean; }; /** * Ceilings a client can check before spending a request; the server's - * own limit stays authoritative. `null` states no limit, as on a cap — - * nothing to pre-check, not nothing enforced — and local adapters - * declare it. See docs/spec/adapters.md § Adapter capabilities. + * own limit stays authoritative. `null` means this client cannot + * pre-check, not that nothing is enforced — local adapters declare it. + * See docs/spec/adapters.md § Adapter capabilities. */ limits: { /** Maximum attachment upload size in bytes. */ diff --git a/packages/core/tests/collection.test.ts b/packages/core/tests/collection.test.ts index df72a2fe..b682f0c9 100644 --- a/packages/core/tests/collection.test.ts +++ b/packages/core/tests/collection.test.ts @@ -692,12 +692,12 @@ describe.each([ let adapter: MemoryAdapter; let stack: Stack; - const open = async (filterArms = 64, sortKeys = 64) => { + const open = async (byType = true) => { adapter = await MemoryAdapter.open({ ownerEntityId: OWNER, timezone: 'UTC' }); (adapter as { capabilities: StackClient['capabilities'] }).capabilities = { ...adapter.capabilities, - filter: { ...adapter.capabilities.filter, contentByType: filterArms }, - sort: { ...adapter.capabilities.sort, contentFieldByType: sortKeys }, + filter: { ...adapter.capabilities.filter, contentByType: byType }, + sort: { ...adapter.capabilities.sort, contentFieldByType: byType }, }; stack = await Stack.open(adapter, { migrations: TASK_MIGRATIONS }); for (const type of [TaskV1, TaskV2, Task]) await stack.defineType(type); @@ -803,7 +803,7 @@ describe.each([ }); test('an adapter that declines the shape gets the handle-only query', async () => { - await open(0, 0); + await open(false); const [, , v3] = await seed('Ada'); const filtered = await tasks().query({ filter: { content: { owner: 'Ada' } } }); expect(filtered.records.map((r) => r.id)).toEqual([v3.id]); @@ -813,14 +813,6 @@ describe.each([ expect(sorted.skipped).toEqual({ older: true, newer: false }); }); - test('an adapter with a lower cap gets the newest versions that fit', async () => { - await open(2); - const [, v2, v3] = await seed('Ada'); - const page = await tasks().query({ filter: { content: { owner: 'Ada' } } }); - expect(page.records.map((r) => r.id).sort()).toEqual([v2.id, v3.id].sort()); - expect(page.skipped).toEqual({ older: true, newer: false }); - }); - test('a migration not built by migration() is narrowed', async () => { stack = await Stack.open(adapter, { migrations: [ diff --git a/packages/core/tests/combine.test.ts b/packages/core/tests/combine.test.ts index 61655d1e..c7037391 100644 --- a/packages/core/tests/combine.test.ts +++ b/packages/core/tests/combine.test.ts @@ -28,13 +28,13 @@ const capabilities: StackCapabilities = { filter: { content: 'none', contentPresent: false, - contentByType: 0, + contentByType: false, search: false, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: false, - contentFieldByType: 0, + contentFieldByType: false, }, limits: { attachmentBytes: null, diff --git a/packages/core/tests/query-capabilities.test.ts b/packages/core/tests/query-capabilities.test.ts index c04f1cac..495c38d5 100644 --- a/packages/core/tests/query-capabilities.test.ts +++ b/packages/core/tests/query-capabilities.test.ts @@ -25,11 +25,11 @@ import type { } from '../src/types/index.js'; const ALL: StackCapabilities = { - filter: { content: 'path', contentPresent: true, contentByType: 64, search: true }, + filter: { content: 'path', contentPresent: true, contentByType: true, search: true }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 64, + contentFieldByType: true, }, limits: { attachmentBytes: null, contentBytes: null }, }; @@ -110,19 +110,12 @@ describe('a refused query names the capability it needed', () => { test('filter.contentByType declined', () => { expect( - refusalFor(() => assertQueryCapabilities(ARMS, without({ filter: { contentByType: 0 } }))), + refusalFor(() => + assertQueryCapabilities(ARMS, without({ filter: { contentByType: false } })), + ), ).toBe('filter.contentByType'); }); - test('filter.contentByType past its cap', () => { - expect( - refusalFor(() => assertQueryCapabilities(ARMS, without({ filter: { contentByType: 1 } }))), - ).toBe('filter.contentByType'); - expect(() => - assertQueryCapabilities(ARMS, without({ filter: { contentByType: 2 } })), - ).not.toThrow(); - }); - test("filter.contentByType against reach 'none'", () => { expect( refusalFor(() => assertQueryCapabilities(ARMS, without({ filter: { content: 'none' } }))), @@ -151,28 +144,23 @@ describe('a refused query names the capability it needed', () => { ).toBe('filter.contentPresent'); }); - test('sort.contentFieldByType declined, or past its cap', () => { + test('sort.contentFieldByType declined', () => { const sort = { contentFieldByType: { 'com.example/x@1': { field: 'a' } } }; expect( - refusalFor(() => assertSortCapability(sort, without({ sort: { contentFieldByType: 0 } }))), + refusalFor(() => + assertSortCapability(sort, without({ sort: { contentFieldByType: false } })), + ), ).toBe('sort.contentFieldByType'); - expect(() => - assertSortCapability(sort, without({ sort: { contentFieldByType: 1 } })), - ).not.toThrow(); }); - test('a null per-Type cap admits any number of keys', () => { + test('a declared per-Type shape takes any number of keys', () => { const typeIds = Array.from({ length: 500 }, (_, i) => `com.example/x@${i + 1}`); const filter = { contentByType: Object.fromEntries(typeIds.map((id) => [id, {}])) }; const sort = { contentFieldByType: Object.fromEntries(typeIds.map((id) => [id, { field: 'a' }])), }; - const unbounded = without({ - filter: { contentByType: null }, - sort: { contentFieldByType: null }, - }); - expect(() => assertQueryCapabilities(filter, unbounded)).not.toThrow(); - expect(() => assertSortCapability(sort, unbounded)).not.toThrow(); + expect(() => assertQueryCapabilities(filter, ALL)).not.toThrow(); + expect(() => assertSortCapability(sort, ALL)).not.toThrow(); }); test('sort.fields', () => { diff --git a/packages/core/tests/stack.test.ts b/packages/core/tests/stack.test.ts index f917bf0b..73763859 100644 --- a/packages/core/tests/stack.test.ts +++ b/packages/core/tests/stack.test.ts @@ -1106,6 +1106,24 @@ describe('query — content filter null semantics', () => { ); }); + // The path cap and the content depth cap are one limit read from either + // side. See docs/spec/data-model.md § Nested content paths. + test('the longest legal path reaches the deepest value a write accepts', async () => { + const chain = (n: number): Record => + n === 1 ? { n: 'x' } : { n: chain(n - 1) }; + const path = (n: number) => Array(n).fill('n').join('.'); + + const deepest = await stack.create(NOTE_V1, { text: 'deep', ...chain(33) }); + const found = await stack.query({ filter: { content: { [path(33)]: 'x' } } }); + expect(found.records.map((r) => r.id)).toEqual([deepest.id]); + await expect(stack.create(NOTE_V1, { text: 'deeper', ...chain(34) })).rejects.toThrow( + /Content nesting exceeds maximum depth of 32/, + ); + await expect(stack.query({ filter: { content: { [path(34)]: 'x' } } })).rejects.toThrow( + StackBadRequestError, + ); + }); + // A dotted key is a path, and a field literally named `a.b` is refused // at write time, so the two readings can never both be available. test('a dotted key is a path, and the colliding field name is unwritable', async () => { @@ -4167,7 +4185,7 @@ describe('grantType', () => { const bare = await MemoryAdapter.open({ ownerEntityId: 'owner-123', timezone: 'UTC' }); (bare as { capabilities: StackAdapter['capabilities'] }).capabilities = { ...bare.capabilities, - sort: { fields: [], contentField: false, contentFieldByType: 0 }, + sort: { fields: [], contentField: false, contentFieldByType: false }, }; const bareStack = await Stack.open(bare); expect(await bareStack.listTypeGrants()).toEqual([]); @@ -6000,13 +6018,13 @@ describe('nested content paths', () => { filter: { content: 'field', contentPresent: false, - contentByType: 0, + contentByType: false, search: false, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: false, - contentFieldByType: 0, + contentFieldByType: false, }, limits: { attachmentBytes: null, @@ -6037,11 +6055,11 @@ describe('limits.contentBytes pre-check', () => { const withContentCeiling = async (contentBytes: number): Promise => Object.assign(await MemoryAdapter.open({ ownerEntityId: 'owner-123', timezone: 'UTC' }), { capabilities: { - filter: { content: 'path', contentPresent: true, contentByType: 64, search: false }, + filter: { content: 'path', contentPresent: true, contentByType: true, search: false }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 64, + contentFieldByType: true, }, limits: { attachmentBytes: null, contentBytes }, }, @@ -6095,11 +6113,11 @@ describe('putAttachment — limits.attachmentBytes pre-check', () => { const withCeiling = async (attachmentBytes: number): Promise => Object.assign(await MemoryAdapter.open({ ownerEntityId: 'owner-123', timezone: 'UTC' }), { capabilities: { - filter: { content: 'none', contentPresent: false, contentByType: 0, search: false }, + filter: { content: 'none', contentPresent: false, contentByType: false, search: false }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: false, - contentFieldByType: 0, + contentFieldByType: false, }, limits: { attachmentBytes, contentBytes: null }, }, diff --git a/packages/record-adapter-do-sqlite/tests/record.test.ts b/packages/record-adapter-do-sqlite/tests/record.test.ts index d527a918..156cc134 100644 --- a/packages/record-adapter-do-sqlite/tests/record.test.ts +++ b/packages/record-adapter-do-sqlite/tests/record.test.ts @@ -113,13 +113,13 @@ describe('construction', () => { filter: { content: 'path', contentPresent: true, - contentByType: null, + contentByType: true, search: true, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: null, + contentFieldByType: true, }, limits: { attachmentBytes: null, @@ -598,10 +598,10 @@ test('a tag filter far past the parameter cap', async () => { expect(result.records.map((r) => r.id)).toEqual([tagged.id]); }); -// The shared SQLite declaration sets no per-Type cap, so it has to hold on -// the engine with the tightest statement limits, far past any count a -// statement could have afforded. See docs/spec/adapters.md § Statement size. -describe('per-Type keys with no declared cap', () => { +// A per-Type shape carries no count, so it has to hold on the engine with +// the tightest statement limits, far past any count a statement could have +// afforded. See docs/spec/adapters.md § Statement size. +describe('per-Type keys far past the statement limits', () => { const KEYS = 200; const typeIds = (count: number) => Array.from({ length: count }, (_, i) => (i === 0 ? NOTE_TYPE_V1 : `com.example/t${i}@1`)); diff --git a/packages/sqlite-shared/src/record-adapter.ts b/packages/sqlite-shared/src/record-adapter.ts index 747a8207..7847d275 100644 --- a/packages/sqlite-shared/src/record-adapter.ts +++ b/packages/sqlite-shared/src/record-adapter.ts @@ -53,15 +53,13 @@ export const SQLITE_RECORD_CAPABILITIES: StackCapabilities = { filter: { content: 'path', contentPresent: true, - // A per-Type key costs no statement text or parameter of its own, so - // nothing bounds it; see docs/spec/adapters.md § Statement size. - contentByType: null, + contentByType: true, search: true, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: null, + contentFieldByType: true, }, limits: { attachmentBytes: null, diff --git a/packages/wire-types/src/discovery.ts b/packages/wire-types/src/discovery.ts index d6a0b55c..04805488 100644 --- a/packages/wire-types/src/discovery.ts +++ b/packages/wire-types/src/discovery.ts @@ -42,10 +42,10 @@ export type DiscoveryCapabilities = { filter?: { content?: string; contentPresent?: boolean; - contentByType?: number | null; + contentByType?: boolean; search?: boolean; }; - sort?: { fields?: string[]; contentField?: boolean; contentFieldByType?: number | null }; + sort?: { fields?: string[]; contentField?: boolean; contentFieldByType?: boolean }; limits?: { attachmentBytes?: number | null; contentBytes?: number | null }; }; @@ -54,9 +54,9 @@ const SORT_FIELDS: ReadonlySet = new Set(NATIVE_SORT_FIELDS); /** * Resolves anything absent, malformed or unrecognized to the least capable - * value it could stand for, and what depends on the content reach to - * nothing below `'none'` — applied once, where capabilities enter, so no - * reader re-derives either. See docs/spec/adapters.md § Adapter capabilities. + * value it could stand for, and what depends on the content reach to false + * below `'none'` — applied once, where capabilities enter. A `null` limit is + * the one permissive reading. See docs/spec/adapters.md § Adapter capabilities. */ export function normalizeCapabilities( capabilities: DiscoveryCapabilities | undefined, @@ -74,32 +74,23 @@ export function normalizeCapabilities( filter: { content: reach, contentPresent: reaches && filter?.contentPresent === true, - contentByType: reaches ? readCap(filter?.contentByType) : 0, + contentByType: reaches && filter?.contentByType === true, search: filter?.search === true, }, sort: { fields: (sort?.fields ?? []).filter((f): f is NativeSortField => SORT_FIELDS.has(f)), contentField: sort?.contentField === true, - contentFieldByType: reaches ? readCap(sort?.contentFieldByType) : 0, + contentFieldByType: reaches && sort?.contentFieldByType === true, }, limits: { - attachmentBytes: readLimit(limits?.attachmentBytes), - contentBytes: readLimit(limits?.contentBytes), + attachmentBytes: finiteOrNull(limits?.attachmentBytes), + contentBytes: finiteOrNull(limits?.contentBytes), }, }; } -/** - * A cap on keys: a positive integer or `null` — no limit — as stated, and - * 0 — declined — for anything else. - */ -const readCap = (value: unknown): number | null => { - if (value === null) return null; - return typeof value === 'number' && Number.isSafeInteger(value) && value > 0 ? value : 0; -}; - -/** A ceiling as stated, and `null` — no limit stated — for anything else. */ -const readLimit = (value: unknown): number | null => +/** A ceiling this client can compare against, or null for "cannot pre-check". */ +const finiteOrNull = (value: unknown): number | null => typeof value === 'number' && Number.isFinite(value) ? value : null; /** Splits a MAJOR.MINOR protocol version. Returns null if it isn't one. */ diff --git a/packages/wire-types/tests/changes.test.ts b/packages/wire-types/tests/changes.test.ts index 7c1e5c0c..83c59c78 100644 --- a/packages/wire-types/tests/changes.test.ts +++ b/packages/wire-types/tests/changes.test.ts @@ -42,13 +42,13 @@ const discovery = (overrides: Partial = {}): DiscoveryRespons filter: { content: 'path', contentPresent: true, - contentByType: 64, + contentByType: true, search: true, }, sort: { fields: ['createdAt'], contentField: true, - contentFieldByType: 64, + contentFieldByType: true, }, limits: { attachmentBytes: null, diff --git a/packages/wire-types/tests/discovery.test.ts b/packages/wire-types/tests/discovery.test.ts index 30eeb033..a9a30e7b 100644 --- a/packages/wire-types/tests/discovery.test.ts +++ b/packages/wire-types/tests/discovery.test.ts @@ -53,8 +53,8 @@ describe('isProtocolCompatible', () => { // ------------------------------------------------------- const NONE: StackCapabilities = { - filter: { content: 'none', contentPresent: false, contentByType: 0, search: false }, - sort: { fields: [], contentField: false, contentFieldByType: 0 }, + filter: { content: 'none', contentPresent: false, contentByType: false, search: false }, + sort: { fields: [], contentField: false, contentFieldByType: false }, limits: { attachmentBytes: null, contentBytes: null }, }; @@ -62,13 +62,13 @@ describe('normalizeCapabilities', () => { it('reads what a server declares', () => { expect( normalizeCapabilities({ - filter: { content: 'path', contentPresent: true, contentByType: 64, search: true }, - sort: { fields: ['createdAt', 'version'], contentField: true, contentFieldByType: 64 }, + filter: { content: 'path', contentPresent: true, contentByType: true, search: true }, + sort: { fields: ['createdAt', 'version'], contentField: true, contentFieldByType: true }, limits: { attachmentBytes: 52428800, contentBytes: 1048576 }, }), ).toEqual({ - filter: { content: 'path', contentPresent: true, contentByType: 64, search: true }, - sort: { fields: ['createdAt', 'version'], contentField: true, contentFieldByType: 64 }, + filter: { content: 'path', contentPresent: true, contentByType: true, search: true }, + sort: { fields: ['createdAt', 'version'], contentField: true, contentFieldByType: true }, limits: { attachmentBytes: 52428800, contentBytes: 1048576 }, }); }); @@ -82,38 +82,28 @@ describe('normalizeCapabilities', () => { ).toEqual([...NATIVE_SORT_FIELDS]); }); - it('reads a per-Type cap that is not a positive integer or null as 0', () => { - for (const cap of [-1, 2.5, '16', true, undefined, Number.POSITIVE_INFINITY]) { + it('reads a per-Type flag that is not true as false', () => { + for (const flag of [1, 'true', null, undefined]) { const read = normalizeCapabilities({ - filter: { content: 'path', contentByType: cap as number }, - sort: { contentFieldByType: cap as number }, + filter: { content: 'path', contentByType: flag as unknown as boolean }, + sort: { contentFieldByType: flag as unknown as boolean }, }); - expect([read.filter.contentByType, read.sort.contentFieldByType]).toEqual([0, 0]); + expect([read.filter.contentByType, read.sort.contentFieldByType]).toEqual([false, false]); } }); - it('reads a stated null per-Type cap as no limit', () => { - const read = normalizeCapabilities({ - filter: { content: 'path', contentByType: null }, - sort: { contentFieldByType: null }, - }); - expect([read.filter.contentByType, read.sort.contentFieldByType]).toEqual([null, null]); - }); - // These travel in the POST /records/query body, which a server reaching // no content does not serve, so what it declares for them is unhonorable. - it('reads what depends on the content reach as declined beside a reach of none', () => { - for (const cap of [16, null]) { - const read = normalizeCapabilities({ - filter: { content: 'none', contentPresent: true, contentByType: cap }, - sort: { contentFieldByType: cap }, - }); - expect([ - read.filter.contentPresent, - read.filter.contentByType, - read.sort.contentFieldByType, - ]).toEqual([false, 0, 0]); - } + it('reads what depends on the content reach as false beside a reach of none', () => { + const read = normalizeCapabilities({ + filter: { content: 'none', contentPresent: true, contentByType: true }, + sort: { contentFieldByType: true }, + }); + expect([ + read.filter.contentPresent, + read.filter.contentByType, + read.sort.contentFieldByType, + ]).toEqual([false, false, false]); }); it('reads an absent capabilities object as declaring nothing', () => { @@ -124,7 +114,7 @@ describe('normalizeCapabilities', () => { it('reads an absent group as declaring nothing, not as inheriting its siblings', () => { expect(normalizeCapabilities({ filter: { content: 'path' } })).toEqual({ ...NONE, - filter: { content: 'path', contentPresent: false, contentByType: 0, search: false }, + filter: { content: 'path', contentPresent: false, contentByType: false, search: false }, }); }); From 1c8b1780f11a2b9c461ff7dd856aaaa2f2958047 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 22 Oct 2026 02:01:56 +0000 Subject: [PATCH 9/9] refactor(sqlite,core): last spellings of one rule The invariant behind contentPresent and the per-Type flags is the POST /records/query body, not filtering: a per-Type sort needs that endpoint where sort.contentField, a query parameter, does not. The spec and the code that apply it now say so. elementsByRecord() reads its ids through rowsTable(), the shape rows() binds, and encodes them itself rather than taking a caller's JSON. The collection no longer orders a family by version, which only served the cap, and assertQueryCapabilities() reads contentByType with its siblings. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Mf3p8BVtrQJLZx1JFCHchC --- docs/spec/adapters.md | 2 +- packages/adapter-conformance/src/record.ts | 6 +++--- packages/core/src/collection.ts | 4 +--- packages/core/src/query-validation.ts | 4 ++-- packages/sqlite-shared/src/query.ts | 9 +++++++-- packages/sqlite-shared/src/record-logic.ts | 18 +++++++++--------- packages/wire-types/src/discovery.ts | 7 ++++--- packages/wire-types/tests/discovery.test.ts | 2 +- 8 files changed, 28 insertions(+), 24 deletions(-) diff --git a/docs/spec/adapters.md b/docs/spec/adapters.md index 3c712ced..af531c94 100644 --- a/docs/spec/adapters.md +++ b/docs/spec/adapters.md @@ -219,7 +219,7 @@ One type, `StackCapabilities`, names both sides: an adapter declares it as `capa **`filter.contentPresent` sits beside the ladder rather than on it.** It gates `filter.contentPresent`, and is required `true` for local adapters on the same reasoning: asking whether a path holds a value is the same walk with a weaker predicate. It is not a rung because a server promising to match a content value has not thereby promised to answer whether one is there at all. It is meaningful only above `'none'` — presence travels in the `POST /records/query` body, which a server reaching no content does not expose — and a presence path is itself gated by the rung, so a multi-segment one needs `'path'`. -**`filter.contentByType` and `sort.contentFieldByType` are flags, like `filter.contentPresent`.** Each says the [per-Type shape](./data-model.md#content-by-type) is honored, with any number of keys. A local adapter must honor both, on the reasoning that makes `'path'` required: an arm is the content filter it already runs, scoped to one `typeId`, and costs a SQLite adapter no statement text or parameter of its own (see [Statement size](#statement-size)). Neither carries a count, as nothing else a query holds does — `filter.content` keys, `contentPresent` paths and tag lists are unbounded too: a server bounds what a query costs it the one way it bounds all of them, under [Wire format § Bounding query cost](./wire-format.md#bounding-query-cost). A collection facing `false` queries its handle's version alone rather than failing — see [Data model § Content queries across versions](./data-model.md#content-queries-across-versions). Both are meaningful only above `filter.content: 'none'`, since they travel in the `POST /records/query` body, and are `false` beside it, as `filter.contentPresent` is. That holds where capabilities enter, so nothing reading them re-derives it: the [conformance suite](#conformance) holds a local adapter's declaration to it, and `normalizeCapabilities()` applies it to a discovery response. +**`filter.contentByType` and `sort.contentFieldByType` are flags, like `filter.contentPresent`.** Each says the [per-Type shape](./data-model.md#content-by-type) is honored, with any number of keys. A local adapter must honor both, on the reasoning that makes `'path'` required: an arm is the content filter it already runs, scoped to one `typeId`, and costs a SQLite adapter no statement text or parameter of its own (see [Statement size](#statement-size)). Neither carries a count, as nothing else a query holds does — `filter.content` keys, `contentPresent` paths and tag lists are unbounded too: a server bounds what a query costs it the one way it bounds all of them, under [Wire format § Bounding query cost](./wire-format.md#bounding-query-cost). A collection facing `false` queries its handle's version alone rather than failing — see [Data model § Content queries across versions](./data-model.md#content-queries-across-versions). Like `filter.contentPresent`, both travel only in the `POST /records/query` body, which a server reaching no content does not serve, so all three are `false` beside `filter.content: 'none'`. The dependency is on that endpoint, not on filtering: a per-Type sort needs it where `sort.contentField`, sent as a query parameter, does not. That holds where capabilities enter, so nothing reading them re-derives it: the [conformance suite](#conformance) holds a local adapter's declaration to it, and `normalizeCapabilities()` applies it to a discovery response. **`filter.search` has no local-required rule.** A full-text index is a different mechanism from field matching, and neither implies the other, so a local adapter may legitimately decline it (see the JSON adapter note below). diff --git a/packages/adapter-conformance/src/record.ts b/packages/adapter-conformance/src/record.ts index 66a1ecf1..651e599f 100644 --- a/packages/adapter-conformance/src/record.ts +++ b/packages/adapter-conformance/src/record.ts @@ -93,9 +93,9 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt expect(Array.isArray(capabilities.sort.fields)).toBe(true); }); - // Each asks something of content, and travels in the POST - // /records/query body a reach of "none" does not serve. - test('what depends on the content reach is false whenever filter.content is "none"', () => { + // Each travels only in the POST /records/query body, which a reach of + // "none" does not serve. + test('what needs the query body is false whenever filter.content is "none"', () => { if (capabilities.filter.content === 'none') { expect(capabilities.filter.contentPresent).toBe(false); expect(capabilities.filter.contentByType).toBe(false); diff --git a/packages/core/src/collection.ts b/packages/core/src/collection.ts index 3ec1133a..30ad21c9 100644 --- a/packages/core/src/collection.ts +++ b/packages/core/src/collection.ts @@ -506,9 +506,7 @@ export const createCollection = ( pinned ? pinned.map((id) => ({ id, baseId: handle.baseId, version: versionOf(id) })) : await listTypes() - ) - .filter((type) => type.baseId === handle.baseId && type.version !== handleVersion) - .sort((a, b) => b.version - a.version); + ).filter((type) => type.baseId === handle.baseId && type.version !== handleVersion); const arms: Record = { [handle.id]: predicate ?? {} }; const keys: Record = sortField ? { [handle.id]: { field: sortField } } diff --git a/packages/core/src/query-validation.ts b/packages/core/src/query-validation.ts index 9a76e27e..6a4288f6 100644 --- a/packages/core/src/query-validation.ts +++ b/packages/core/src/query-validation.ts @@ -45,7 +45,7 @@ export function assertQueryCapabilities( filter: RecordFilter | undefined, capabilities: Pick, ): void { - const { content: reach, contentPresent, search } = capabilities.filter; + const { content: reach, contentPresent, contentByType, search } = capabilities.filter; if (filter?.search && !search) { throw new StackBadRequestError( 'Query uses filter.search, but this adapter does not declare the filter.search capability.', @@ -72,7 +72,7 @@ export function assertQueryCapabilities( 'filter.content', ); } - if (arms.length > 0 && !capabilities.filter.contentByType) { + if (arms.length > 0 && !contentByType) { throw new StackBadRequestError( 'Query uses filter.contentByType, but this adapter does not declare the ' + 'filter.contentByType capability.', diff --git a/packages/sqlite-shared/src/query.ts b/packages/sqlite-shared/src/query.ts index 07210969..cc64a324 100644 --- a/packages/sqlite-shared/src/query.ts +++ b/packages/sqlite-shared/src/query.ts @@ -112,12 +112,17 @@ const emptyStatement = (): Statement => ({ */ const rows = (s: Statement, columns: readonly string[], tuples: unknown[][]): string => { const name = `t${s.tables.length}`; - const named = columns.map((column, i) => `json_extract(value, '$[${i}]') AS ${column}`); - s.tables.push(`${name} AS MATERIALIZED (SELECT ${named.join(', ')} FROM json_each(?))`); + s.tables.push(`${name} AS MATERIALIZED ${rowsTable(columns)}`); s.tableParams.push(JSON.stringify(tuples)); return name; }; +/** The table rows() binds, for a statement outside the query plan. */ +export const rowsTable = (columns: readonly string[]): string => { + const named = columns.map((column, i) => `json_extract(value, '$[${i}]') AS ${column}`); + return `(SELECT ${named.join(', ')} FROM json_each(?))`; +}; + /** The columns of a content condition; see contentRows(). */ const CONDITION_COLUMNS = ['scope', 'path', 'op', 'val', 'kind']; diff --git a/packages/sqlite-shared/src/record-logic.ts b/packages/sqlite-shared/src/record-logic.ts index 7c52cbbe..02d97ab7 100644 --- a/packages/sqlite-shared/src/record-logic.ts +++ b/packages/sqlite-shared/src/record-logic.ts @@ -48,7 +48,7 @@ import type { } from '@haverstack/core/adapter'; import type { SqlExecutor } from './executor.js'; import { isForeignKeyViolation, isUniqueConstraintViolation } from './executor.js'; -import { buildQueryPlan, atBudget } from './query.js'; +import { buildQueryPlan, atBudget, rowsTable } from './query.js'; import { ContentIndex, SearchIndex } from './indexes.js'; import { rowToRecord, @@ -306,7 +306,7 @@ export class SharedSqlRecordLogic { * reads however many records it holds. */ private hydrate(rows: Record[]): StackRecord[] { - const ids = JSON.stringify(rows.map((row) => row.id)); + const ids = rows.map((row) => row.id as string); const associations = this.elementsByRecord('associations', ids, rowToAssociation); const permissions = this.elementsByRecord('permissions', ids, rowToPermission); return rows.map((row) => @@ -804,9 +804,8 @@ export class SharedSqlRecordLogic { this.exec.transaction(() => { this.checkExpectedVersion(id, this.liveVersion(id), opts.ifVersion); if (opts.expectSet) { - const ids = JSON.stringify([id]); const map = table === 'associations' ? rowToAssociation : rowToPermission; - const stored = this.elementsByRecord(table, ids, map).get(id) ?? []; + const stored = this.elementsByRecord(table, [id], map).get(id) ?? []; if (!sameAssociationSet(stored, opts.expectSet)) throw new ElementSetMovedError(id); } write(); @@ -953,19 +952,20 @@ export class SharedSqlRecordLogic { } /** - * One element table's rows for every record in `ids` (a JSON array), in - * primary-key order — the order a single record's read has always had. + * One element table's rows for every record in `ids`, in primary-key + * order — the order a single record's read has always had. */ private elementsByRecord( table: 'associations' | 'permissions', - ids: string, + ids: readonly string[], map: (row: Record) => T, ): Map { const byRecord = new Map(); const rows = this.exec.all>( - `SELECT * FROM ${table} WHERE record_id IN (SELECT value FROM json_each(?)) + `WITH ids AS MATERIALIZED ${rowsTable(['id'])} + SELECT * FROM ${table} WHERE record_id IN (SELECT id FROM ids) ORDER BY ${table === 'associations' ? ASSOCIATION_KEY : PERMISSION_KEY}`, - [ids], + [JSON.stringify(ids.map((id) => [id]))], ); for (const row of rows) { const id = row.record_id as string; diff --git a/packages/wire-types/src/discovery.ts b/packages/wire-types/src/discovery.ts index 04805488..4148de02 100644 --- a/packages/wire-types/src/discovery.ts +++ b/packages/wire-types/src/discovery.ts @@ -54,9 +54,10 @@ const SORT_FIELDS: ReadonlySet = new Set(NATIVE_SORT_FIELDS); /** * Resolves anything absent, malformed or unrecognized to the least capable - * value it could stand for, and what depends on the content reach to false - * below `'none'` — applied once, where capabilities enter. A `null` limit is - * the one permissive reading. See docs/spec/adapters.md § Adapter capabilities. + * value it could stand for, and what needs the POST /records/query body to + * false below `'none'` — applied once, where capabilities enter. A `null` + * limit is the one permissive reading. See docs/spec/adapters.md § Adapter + * capabilities. */ export function normalizeCapabilities( capabilities: DiscoveryCapabilities | undefined, diff --git a/packages/wire-types/tests/discovery.test.ts b/packages/wire-types/tests/discovery.test.ts index a9a30e7b..1c20fa31 100644 --- a/packages/wire-types/tests/discovery.test.ts +++ b/packages/wire-types/tests/discovery.test.ts @@ -94,7 +94,7 @@ describe('normalizeCapabilities', () => { // These travel in the POST /records/query body, which a server reaching // no content does not serve, so what it declares for them is unhonorable. - it('reads what depends on the content reach as false beside a reach of none', () => { + it('reads what needs the query body as false beside a reach of none', () => { const read = normalizeCapabilities({ filter: { content: 'none', contentPresent: true, contentByType: true }, sort: { contentFieldByType: true },