diff --git a/.changeset/fixed-statement-size.md b/.changeset/fixed-statement-size.md new file mode 100644 index 00000000..a8b1d9de --- /dev/null +++ b/.changeset/fixed-statement-size.md @@ -0,0 +1,14 @@ +--- +'@haverstack/record-adapter-do-sqlite': minor +'@haverstack/record-adapter-sqlite': minor +'@haverstack/adapter-conformance': minor +'@haverstack/core': minor +'@haverstack/wire-types': minor +'@haverstack/conformance-fixtures': minor +--- + +A SQLite statement's text and bound parameters no longer grow with a query's size. Every caller-sized part of a query binds as one table of rows: lists, permission sets, `sort.contentFieldByType`'s entries, and the content conditions — every `filter.content` and `filter.contentPresent` key and every `filter.contentByType` arm — whose paths one recursive walk per condition resolves. Durable Object SQLite no longer fails a query with `too many SQL variables` or `statement too long`, however many per-Type keys or content conditions it carries. + +`filter.contentByType` and `sort.contentFieldByType` are now booleans, like `filter.contentPresent`: a per-Type shape holds any number of keys, as every other part of a query does, and a server bounds what one costs it by its query deadline. A collection facing `false` queries its handle's version alone and reports the rest as `skipped`, as it did at a cap of `0`; partial narrowing to the newest versions that fit is gone. `normalizeCapabilities()` reads both as `true` only when stated, and now reads `filter.contentPresent` as `false` beside `filter.content: 'none'`, as it reads both per-Type flags. + +A content filter value now matches only a value of its own JSON type in the SQLite adapters, as `===` does: `true` no longer matches `1`. A content filter path may be 33 segments, the depth that reaches the deepest value a write accepts. The conformance suite holds every adapter to both, and to a multi-valued path satisfying both `contentPresent` and a `null` filter. diff --git a/.changeset/scoped-query-reach.md b/.changeset/scoped-query-reach.md index 7452ca55..fc04028f 100644 --- a/.changeset/scoped-query-reach.md +++ b/.changeset/scoped-query-reach.md @@ -7,4 +7,4 @@ '@haverstack/conformance-fixtures': patch --- -State what a scoped requester may do as a `Reach` per verb — `create`, `read`, `update`, `delete` — and answer every grant- and permission-based gate from it: a type fence for a delegated principal, and within it the types reached whole, the types reached where the subject authored, and the permission-element sets that carry the verb, a `write` beside its grantee's `read` or `anyone`. `get()`, refusals and the change feed ask the read reach of one record, write gates ask the update or delete reach, `create()` asks the create reach of the record it would make, and `query()` hands the read reach to the adapter's new `queryReadable(query, reach)`, which matches it as one more condition — so pages, cursors and whether a cursor comes back are decided among readable records alone, and a cursor never names a record the requester cannot read. A requester's grants, rosters and reaches are each read at most once per operation, and a subscription holds them until a `_grant` or `_group` event clears them. `asEntity()`/`asActor()` refuse with `ScopeUnavailableError` — replacing `RelayScopeError` — over a stack whose adapter lacks `queryReadable()` or relays changes. `MemoryAdapter`, the SQLite adapters and `LocalAdapter` implement it; `inReach()` in `@haverstack/core/adapter` is the reference predicate, and the conformance suite holds adapters to it. The SQLite adapters bind every list a query carries — a filter's type, app and author lists, its tags, and every list in a reach — as one JSON parameter, so neither a reach nor a filter list of any length runs into Durable Object SQLite's 100-parameter cap; per-Type arms stay bounded by the declared `contentByType` and `contentFieldByType` caps. `MemoryAdapter`'s cursor names the last record a page held rather than an offset, so a page resumes after it however the matched set moved, as the SQL adapters' cursors do. +State what a scoped requester may do as a `Reach` per verb — `create`, `read`, `update`, `delete` — and answer every grant- and permission-based gate from it: a type fence for a delegated principal, and within it the types reached whole, the types reached where the subject authored, and the permission-element sets that carry the verb, a `write` beside its grantee's `read` or `anyone`. `get()`, refusals and the change feed ask the read reach of one record, write gates ask the update or delete reach, `create()` asks the create reach of the record it would make, and `query()` hands the read reach to the adapter's new `queryReadable(query, reach)`, which matches it as one more condition — so pages, cursors and whether a cursor comes back are decided among readable records alone, and a cursor never names a record the requester cannot read. A requester's grants, rosters and reaches are each read at most once per operation, and a subscription holds them until a `_grant` or `_group` event clears them. `asEntity()`/`asActor()` refuse with `ScopeUnavailableError` — replacing `RelayScopeError` — over a stack whose adapter lacks `queryReadable()` or relays changes. `MemoryAdapter`, the SQLite adapters and `LocalAdapter` implement it; `inReach()` in `@haverstack/core/adapter` is the reference predicate, and the conformance suite holds adapters to it. The SQLite adapters bind every list a query carries — a filter's type, app and author lists, its tags, and every list in a reach — as one JSON parameter, so neither a reach nor a filter list of any length runs into Durable Object SQLite's 100-parameter cap. `MemoryAdapter`'s cursor names the last record a page held rather than an offset, so a page resumes after it however the matched set moved, as the SQL adapters' cursors do. diff --git a/docs/spec/adapters.md b/docs/spec/adapters.md index 81f0b34e..af531c94 100644 --- a/docs/spec/adapters.md +++ b/docs/spec/adapters.md @@ -69,7 +69,7 @@ type Reach = { The shape and its reading are [Access control § Reach](./access-control.md#reach)'s, over TypeIds where that section's are over families: `Stack` resolves each family to its registered versions before the call, the resolution [`filter.baseId`](./data-model.md#filter) gets, so an adapter needs no family concept. The reach is one more condition of the query, beside its own `filter`, so it narrows and never widens. Every rule behind it is resolved before the adapter sees it, so the adapter applies none of access control: it matches a description, the way it matches a `filter`. `inReach()` in `@haverstack/core/adapter` is the reference reading, and the [conformance suite](#conformance) holds an adapter to it. -**Every local adapter implements it.** It is optional only so `APIAdapter` can leave it out: a stack whose adapter lacks it [cannot be scoped at all](./access-control.md#enforcement-stackasentity), which a server-backed stack never needs. A reach is sized by the requester's grants and rosters, not by anything the adapter controls, so **a list in it never costs a bound parameter per entry** — the rule a SQLite adapter holds every list a query carries to, a `filter`'s and a reach's alike: each binds as one JSON parameter read through `json_each`, the permission sets and a tag filter's all-of list included, since Durable Object SQLite caps a statement at 100 parameters. What is not a list, the per-Type arms of `contentByType` and `contentFieldByType`, is bounded instead by the cap the adapter [declares](#adapter-capabilities) for it. It matches each permission element through an index on the element's identity ending in `record_id`, the same semi-join shape as the association filters. +**Every local adapter implements it.** It is optional only so `APIAdapter` can leave it out: a stack whose adapter lacks it [cannot be scoped at all](./access-control.md#enforcement-stackasentity), which a server-backed stack never needs. A reach is sized by the requester's grants and rosters, not by anything the adapter controls, so a SQLite adapter binds each list in it as one parameter, the permission sets included — the rule it holds every part of a query to; see [Statement size](#statement-size). It matches each permission element through an index on the element's identity ending in `record_id`, the same semi-join shape as the association filters. ## Package naming convention @@ -172,6 +172,17 @@ This is why a SQLite-backed adapter's write is unindex, write, re-index rather t A missing row is not only a stale read. `deleteUnreferencedAttachmentRecords()` decides whether a file is still referenced by looking for one, so a `file-ref` field with no row makes a referenced file look unreferenced, and the attachment's metadata Records are purged rather than refused. A writer that touches `content` without rewriting the row is the one way to reach that. +### Statement size + +**A statement's text and its bound parameters never grow with how many entries a query's keys hold.** Durable Object SQLite caps a statement at 100 parameters and 100 KB of text, and a query's size is not the adapter's to choose: a reach is sized by the requester's grants and rosters, a filter by whoever wrote it. A statement that grew with either would fail on one engine for a query another answers, and a declared [capability](#adapter-capabilities) could not promise anything. **Every caller-sized part of a query is a table of rows**, bound as one JSON array of tuples and read through `json_each` as named columns: + +- **A list is rows of one column** — a filter's type, app and author lists and its tags, and a reach's type lists. A list of one value binds as that value instead, so an index on its column keeps yielding rows in order; that is a smaller statement, never a larger one. +- **A reach's permission sets** are rows of one element each, naming the set it belongs to and that set's size. +- **`sort.contentFieldByType`** is two tables, the Types ordered by a field and the Types ordered by a constant. +- **The content conditions** — every `filter.content` and `filter.contentPresent` key, and every key of every `filter.contentByType` arm — are one table, each row naming its path, what it asks, and the Type it is scoped to (none for `filter.content`). A record matches when every row scoped to nothing or to its own Type holds. One recursive walk per row matches a segment per step and gathers what the row asks of the values it reaches, so neither the number of conditions nor a path's length adds text. + +What remains binds a fixed count: a scalar filter's value, a date bound, a cursor's position, the page's row budget. Since a statement's size is set by its shape and not by its query, every engine running the shared query builder honors the same capability declaration. + ## Adapter capabilities Adapters expose a capabilities object so apps can check what's supported before relying on a feature: @@ -181,13 +192,13 @@ type StackCapabilities = { filter: { content: 'none' | 'field' | 'path'; // how far a content filter key may reach contentPresent: boolean; // filter.contentPresent is honored - contentByType: number; // most arms filter.contentByType may hold; 0 = not honored + contentByType: boolean; // filter.contentByType is honored search: boolean; // filter.search is honored }; sort: { fields: ('createdAt' | 'updatedAt' | 'version')[]; // native columns this adapter can order by contentField: boolean; // sort.contentField is honored - contentFieldByType: number; // most keys sort.contentFieldByType may hold; 0 = not honored + contentFieldByType: boolean; // sort.contentFieldByType is honored }; limits: { attachmentBytes: number | null; // upload size ceiling, or null = client can't pre-check @@ -208,13 +219,13 @@ One type, `StackCapabilities`, names both sides: an adapter declares it as `capa **`filter.contentPresent` sits beside the ladder rather than on it.** It gates `filter.contentPresent`, and is required `true` for local adapters on the same reasoning: asking whether a path holds a value is the same walk with a weaker predicate. It is not a rung because a server promising to match a content value has not thereby promised to answer whether one is there at all. It is meaningful only above `'none'` — presence travels in the `POST /records/query` body, which a server reaching no content does not expose — and a presence path is itself gated by the rung, so a multi-segment one needs `'path'`. -**`filter.contentByType` and `sort.contentFieldByType` are caps, not flags.** Each is the most keys the [per-Type shape](./data-model.md#content-by-type) may hold, and `0` declines it. A local adapter must honor both, on the reasoning that makes `'path'` required: an arm is the content filter it already runs, scoped to one `typeId`. The first-party ones declare `64`, far above any family's version count and inside SQLite's limits on expression depth and compound statements. A server may declare any cap, `0` included, and a collection then narrows the versions it cannot reach rather than failing the query — see [Data model § Content queries across versions](./data-model.md#content-queries-across-versions). Both are meaningful only above `filter.content: 'none'`, since they travel in the `POST /records/query` body. +**`filter.contentByType` and `sort.contentFieldByType` are flags, like `filter.contentPresent`.** Each says the [per-Type shape](./data-model.md#content-by-type) is honored, with any number of keys. A local adapter must honor both, on the reasoning that makes `'path'` required: an arm is the content filter it already runs, scoped to one `typeId`, and costs a SQLite adapter no statement text or parameter of its own (see [Statement size](#statement-size)). Neither carries a count, as nothing else a query holds does — `filter.content` keys, `contentPresent` paths and tag lists are unbounded too: a server bounds what a query costs it the one way it bounds all of them, under [Wire format § Bounding query cost](./wire-format.md#bounding-query-cost). A collection facing `false` queries its handle's version alone rather than failing — see [Data model § Content queries across versions](./data-model.md#content-queries-across-versions). Like `filter.contentPresent`, both travel only in the `POST /records/query` body, which a server reaching no content does not serve, so all three are `false` beside `filter.content: 'none'`. The dependency is on that endpoint, not on filtering: a per-Type sort needs it where `sort.contentField`, sent as a query parameter, does not. That holds where capabilities enter, so nothing reading them re-derives it: the [conformance suite](#conformance) holds a local adapter's declaration to it, and `normalizeCapabilities()` applies it to a discovery response. **`filter.search` has no local-required rule.** A full-text index is a different mechanism from field matching, and neither implies the other, so a local adapter may legitimately decline it (see the JSON adapter note below). **`sort.contentField` is a boolean, and `sort.fields` names native columns only.** Content fields are app-defined and unbounded, so an adapter cannot enumerate them; and an adapter that indexes content for sorting indexes [every top-level scalar](./data-model.md#sorting-by-a-content-field), so there is nothing per-field left to declare. `sort.contentField` is required `true` for local adapters, and independent of `filter.content`: the two describe ordering and filtering, and a server may offer either without the other — filtering is answerable at whatever scale a server already scans at, where ordering wants an index. Both are enforced: `Stack.query()` refuses a sort the adapter hasn't declared rather than letting it be answered in some other order, which a caller reading one bounded page cannot detect. -**Absent, malformed or unrecognized reads as the least capable value it could stand for.** One rule, for every entry: `'none'` for the reach, `false` for a flag, `0` for a cap, nothing for `sort.fields`, `null` for a limit. A discovery response that omits a group, omits `capabilities` entirely, or names a reach this client has never heard of describes a server this client will not send that query to — silence is never a claim, in either direction, and an unrecognized rung places nowhere on a ladder so it can only be read as its bottom. `normalizeCapabilities()` in `@haverstack/wire-types` applies the rule, so a client reads a foreign discovery response through one function rather than defaulting keys at each use. +**Absent, malformed or unrecognized reads as the least capable value it could stand for.** One rule, for every entry: `'none'` for the reach, `false` for a flag, nothing for `sort.fields`, `null` for a limit. A discovery response that omits a group, omits `capabilities` entirely, or names a reach this client has never heard of describes a server this client will not send that query to — silence is never a claim, in either direction, and an unrecognized rung places nowhere on a ladder so it can only be read as its bottom. `normalizeCapabilities()` in `@haverstack/wire-types` applies the rule, so a client reads a foreign discovery response through one function rather than defaulting keys at each use. Two things this does not refuse. A query naming no sort asks for the server's own default order rather than a stated one, so it claims nothing that has to be declared and runs against an empty `sort.fields`. And a `null` limit is permissive by design: it says this client cannot pre-check, not that nothing is enforced — the server's own ceiling still answers with a `413`. diff --git a/docs/spec/data-model.md b/docs/spec/data-model.md index cd99f229..8e0c393f 100644 --- a/docs/spec/data-model.md +++ b/docs/spec/data-model.md @@ -454,7 +454,7 @@ A record at an older version that no passed migration carries to the handle's is **A subscription covers the family.** A change's `record`, when present, is migrated to the handle's version; the change's own `typeId` stays the stored one, as `RecordChange.typeId` documents. A change whose record has `deletedAt` set carries neither `record` nor `misfit`, on either surface, and neither does a change that [migrates the record out](#type-migrations) of the family, which the filter matches by its previous `typeId`. The change filter takes every key but `typeId` and `baseId`. -**A content filter or sort spans the versions the family's migrations can carry it to**, and reaches no further: see [Content queries across versions](#content-queries-across-versions). The keys are typed against the handle's version: `filter.content` and `filter.contentPresent` take the content's paths (`ContentPathOf`), through declared objects and arrays, with any suffix below an `open` object or array, and a value of the field's type as a write takes it (an array's element type, since matching is element-wise) or `null`. A path reaching an object takes only `null`, since an adapter compares the value a path reaches and an object never equals one; a value below an `open` node is any scalar. Typing values as writes take them means an enum value the handle does not list is filtered for with a cast, the price of refusing a misspelled one; `sort.contentField` takes only a top-level scalar (`SortableFieldOf`). Path typing stops six segments deep and accepts any suffix below that, a limit of the compiler, not of the [32-segment path](#nested-content-paths). `query()` takes neither `typeId` nor `presentAt`, which would change which records a page holds or the shape they arrive in, and drops either if an untyped caller passes it; `subscribe()` drops a filter's `typeId` the same way. +**A content filter or sort spans the versions the family's migrations can carry it to**, and reaches no further: see [Content queries across versions](#content-queries-across-versions). The keys are typed against the handle's version: `filter.content` and `filter.contentPresent` take the content's paths (`ContentPathOf`), through declared objects and arrays, with any suffix below an `open` object or array, and a value of the field's type as a write takes it (an array's element type, since matching is element-wise) or `null`. A path reaching an object takes only `null`, since an adapter compares the value a path reaches and an object never equals one; a value below an `open` node is any scalar. Typing values as writes take them means an enum value the handle does not list is filtered for with a cast, the price of refusing a misspelled one; `sort.contentField` takes only a top-level scalar (`SortableFieldOf`). Path typing stops six segments deep and accepts any suffix below that, a limit of the compiler, not of the [33-segment path](#nested-content-paths). `query()` takes neither `typeId` nor `presentAt`, which would change which records a page holds or the shape they arrive in, and drops either if an untyped caller passes it; `subscribe()` drops a filter's `typeId` the same way. **Every verb that returns a record returns it as `get()` would.** The write already returned the record, so narrowing it costs no round trip, and a `restoreVersion()` to a pre-migration snapshot reads back migrated, like any older record. A write is checked against the read it already makes, before anything lands: a record outside the family is refused with `StackNotFoundError`, the answer `get()` gives, and so is a `restoreVersion()` to a snapshot taken before the record migrated into the family, which is not a version of a record in the collection. A content write to a record stored at a newer version is refused with `MisfitError` before anything is written, since its patch would be validated against a schema the handle does not know. Any other write that leaves a misfit lands, then throws `MisfitError` carrying the stored record, so the app knows the write landed. A deleted record, or a stale `ifVersion`, is refused first, as the untyped write refuses it. @@ -476,7 +476,7 @@ The derived types are a convenience over [runtime validation](#types), which sta A field the handle declares optional and no step produces is read under its own name, since a migration carries every field no step names — an older version may have gained it in place — and a name a step removed is the constant "no value". A `map` keeps a value it does not list, so more than one older value can arrive as the one a key asks for: `map('status', { done: 'finished' })` carries an unlisted `finished` through as well as remapping `done`. An arm matches one value per path, so it asks for a mapped value first, and each other source becomes a residual the first page probes for, as it probes a narrowed version: a match sets `skipped.older`. One such key per arm is split this way; a second narrows the version. A key no older value maps to drops the older arm, as a constant that does not match does. A key on a field the handle does not declare, which only an untyped caller can write, is narrowed. **A narrowed version is left out of the query**; a dropped arm is left out too, but only because no record there can match. Newer versions stay out of reach either way: the handle cannot represent them. -Records at an older version arrive migrated, as on an unfiltered page, and a sort orders them as their migrated values would order, since a carried field keeps its kind. The first page fixes which versions the query spans, and its cursor carries them: a later page queries the same arms and sort without listing types again, so a version defined mid-pagination cannot change what a cursor resumes. When no older version is left in, the query is sent as a plain `content` filter or `contentField` sort on the handle's `typeId`, which every adapter reaching content answers. An adapter declaring [`filter.contentByType`](./adapters.md#adapter-capabilities) `0` — or `sort.contentFieldByType` `0`, for a content sort — gets that query, with every older version narrowed. One declaring a lower cap than the family has versions to query gets the newest versions that fit, and the rest are narrowed. +Records at an older version arrive migrated, as on an unfiltered page, and a sort orders them as their migrated values would order, since a carried field keeps its kind. The first page fixes which versions the query spans, and its cursor carries them: a later page queries the same arms and sort without listing types again, so a version defined mid-pagination cannot change what a cursor resumes. When no older version is left in, the query is sent as a plain `content` filter or `contentField` sort on the handle's `typeId`, which every adapter reaching content answers. An adapter declaring [`filter.contentByType`](./adapters.md#adapter-capabilities) `false` — or `sort.contentFieldByType` `false`, for a content sort — gets that query, with every older version narrowed. **`skipped` says which side the query could not reach.** The first page sets `skipped.older` when records at a narrowed older version match the rest of the query, and `skipped.newer` when records at a newer version do: the results may then be missing records the filter was meant to find. A version is reported only when the filter or sort touches a field that does not translate across some step between it and the handle, or when a residual of its arm matches — a dropped arm is not a skip. A later page carries no `skipped`, since the flags depend on the query, not the page. Each is one `limit: 1` read, run beside the first page's own, and a scoped read that comes back empty with a cursor still set counts as a match rather than reading on, so a flag can be set when nothing was skipped but is never clear when something was. Mixed versions are a lasting state, not a backlog, since an app at an older version may keep writing its own, so the app decides what `skipped.older` means to it; `migrateAll()` clears it only until the next such write. `skipped.newer` says the app is behind, which an unfiltered page reports as misfits; a content query's page carries none. A query with no content keys reads the whole family. The untyped `query({ filter: { baseId, content } })` still matches across the family, returning records as stored. @@ -620,9 +620,11 @@ The motivating query is a listing: an article's `publishedAt` is optional, so dr **An array along the path is matched element-wise.** `contact@1` stores `emails` as an array of `{ value, label }`, and the question that motivates paths at all — which contact holds this address — is that shape. Traversal and containment are therefore one feature rather than two: a segment applies to each element of an array it meets, and the final comparison does the same, so `{ content: { tags: 'starred' } }` matches a record whose `tags` array contains `'starred'`. A record storing a scalar where another stores an array is matched by the same filter either way. -A path that descends through a scalar, or through a field that isn't there, reaches no value and matches nothing — it is not an error. A path may be at most **32 segments**; longer is `StackBadRequestError` (400), as is an empty segment (`a..b`) or a segment containing a reserved character. Those are structural faults in the request, refused rather than answered with an empty result, on the same reasoning as a malformed cursor. +**A value matches a value of its own JSON type.** `{ content: { flag: true } }` matches `true` and never `1`, and `1` never matches `'1'` — the comparison JavaScript's `===` makes, which is the reference reading every adapter is held to. A number is one type, however an engine stores it. A filter value that is an object or array never matches anything: a path is compared by the scalars it reaches, an array reached at its end standing for its elements. -**A segment is a value, never syntax.** An adapter resolving a path MUST carry each segment as a bound parameter matched against a key, rather than assembling it into a path expression — so no field name a write would accept can be reinterpreted as syntax, and no key can make the statement itself malformed. The 32-segment cap is what keeps that statement inside the engine's own limits: a SQLite adapter walks a segment with two `json_each` joins, and 32 segments is the longest path that fits SQLite's 64-table join limit. The cap and the generated shape move together — a longer path is refused because it could not be executed, not merely because it is unusual. +A path that descends through a scalar, or through a field that isn't there, reaches no value and matches nothing — it is not an error. A path may be at most **33 segments**; longer is `StackBadRequestError` (400), as is an empty segment (`a..b`) or a segment containing a reserved character. Those are structural faults in the request, refused rather than answered with an empty result, on the same reasoning as a malformed cursor. + +**A segment is a value, never syntax.** An adapter resolving a path MUST carry each segment as a bound value matched against a key, rather than assembling it into a path expression — so no field name a write would accept can be reinterpreted as syntax, and no key can make the statement itself malformed. The 33-segment cap is the [nesting cap](#types) read from the other side: a top-level field is reached by one segment and each further segment descends one level, so 33 reach the deepest value a write accepts and a longer path reaches none — it is refused as a fault in the request rather than answered with a match that cannot exist. **Multi-segment keys need the `'path'` rung of `filter.content`**, where a single-segment key needs only `'field'` — see [Capability-gated filters](#capability-gated-filters), and [Adapter capabilities](./adapters.md#adapter-capabilities) for why the reach is one ordered value rather than a flag per rung. @@ -653,7 +655,7 @@ By default, `query()` (like `get()`) returns Records exactly as stored — see [ - **Every arm is pinned to a `typeId`**, and a filter carrying `contentByType` names at least one. An empty map is refused with `StackBadRequestError` (400), so the key cannot encode to a filter that matches everything. - **An arm is flat**: `content` and `contentPresent`, nothing else, and no nesting. An arm naming neither matches every Record at its `typeId`, which is what `typeId` alone matches. - **It never sits beside a top-level `content` or `contentPresent`**: a filter with both is refused with 400, since which one applies to a Record would be ambiguous. Every other key applies to every arm, and a `typeId` beside it intersects with its keys. -- **It is bounded** by the adapter's `filter.contentByType`, the most arms it takes; a filter with more is refused with 400 naming that capability (see [Capability-gated filters](#capability-gated-filters)). +- **It is gated** by the adapter's `filter.contentByType`; an adapter declaring `false` refuses it with 400 naming that capability (see [Capability-gated filters](#capability-gated-filters)). It holds any number of arms, as `content` holds any number of keys. It discloses nothing new: each arm is a query the caller could already run, its `typeId` with its content filter, and the permission check applies per Record as for any query. Types partition Records, so no Record is matched by two arms, and the cost of _n_ arms stays within what _n_ queries would cost — what [Wire format § Bounding query cost](./wire-format.md#bounding-query-cost) already leaves to the server. **It is not a first step toward general OR.** A disjunction across Types is the only one a migration chain needs, and match-any forms stay refused for the reason [above](#filter): a filter that can encode to nothing can silently widen a query when it crosses the wire. @@ -752,7 +754,7 @@ Locale-correct ordering is out of reach for a stored key regardless of effort, b **`contentPresent` needs `filter.contentPresent` on top of a content reach**, a separate flag on the same reasoning the reach ladder rests on: a server promising to match a content value has not thereby promised to answer whether one is there, and reading it as such would hand a client the unfiltered superset that ignoring the filter produces. A multi-segment presence path needs the `'path'` rung too, exactly as a filter key does. -**`contentByType` and `contentFieldByType` need their own capabilities**, `filter.contentByType` and `sort.contentFieldByType`: each is the most keys the adapter takes, and `0` declines the shape. An arm's keys are gated as the same keys at the top of a filter are, so a multi-segment path in an arm needs `'path'`, and an arm's `contentPresent` needs `filter.contentPresent`. +**`contentByType` and `contentFieldByType` need their own capabilities**, `filter.contentByType` and `sort.contentFieldByType`: each says whether the adapter honors the shape, which holds any number of keys. An arm's keys are gated as the same keys at the top of a filter are, so a multi-segment path in an arm needs `'path'`, and an arm's `contentPresent` needs `filter.contentPresent`. **A sort is gated the same way.** `sort.contentField` needs the adapter's `sort.contentField`, and a native `sort.field` must appear in its `sort.fields`; a sort an adapter hasn't declared throws `StackBadRequestError` rather than being answered in some other order — which a caller reading one bounded page has no way to notice. `sort.contentField` and `filter.content` are independent: a server may order by a content field without offering to filter on one, or the reverse. diff --git a/docs/spec/wire-format.md b/docs/spec/wire-format.md index 2f2f9c03..7fb33d62 100644 --- a/docs/spec/wire-format.md +++ b/docs/spec/wire-format.md @@ -16,11 +16,11 @@ GET /.well-known/stack "entityId": "did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK", "timezone": "America/New_York", "capabilities": { - "filter": { "content": "path", "contentPresent": true, "contentByType": 16, "search": true }, + "filter": { "content": "path", "contentPresent": true, "contentByType": true, "search": true }, "sort": { "fields": ["createdAt", "updatedAt", "version"], "contentField": true, - "contentFieldByType": 16 + "contentFieldByType": true }, "limits": { "attachmentBytes": 52428800, "contentBytes": 1048576 } }, @@ -330,7 +330,7 @@ This is what lets a client report a mutation's outcome without a second read, an **`filter.contentPresent` travels in the `POST /records/query` body only**, like `filter.content` itself — there is no `GET /records` spelling, since a server exposing that endpoint alone declares `filter.content: "none"` and offers neither. A server declaring `filter.contentPresent: false` MUST refuse the filter with `400` rather than answer without it. -**`filter.contentByType` and `sort.contentFieldByType` travel in the `POST /records/query` body only**, for the reason `filter.contentPresent` does. A server declaring `0` for either MUST refuse it with `400`, and one receiving more keys than it declares MUST refuse the request with `400` rather than answer some of the arms. An empty map, an arm carrying anything but `content` and `contentPresent`, a `contentByType` beside a top-level `content` or `contentPresent`, and a sort key that is neither `{ "field": string }` nor `{ "value": string | number | null }` are `400` as well. See [Data model § Content by type](./data-model.md#content-by-type). +**`filter.contentByType` and `sort.contentFieldByType` travel in the `POST /records/query` body only**, for the reason `filter.contentPresent` does. A server declaring `false` for either MUST refuse it with `400`. An empty map, an arm carrying anything but `content` and `contentPresent`, a `contentByType` beside a top-level `content` or `contentPresent`, and a sort key that is neither `{ "field": string }` nor `{ "value": string | number | null }` are `400` as well. See [Data model § Content by type](./data-model.md#content-by-type). **A sort names a native column or a content field, and the parameter it arrives in says which** — the same shape the relationship filter uses above, and the only way to keep a content field named `version` distinct from the native column on a raw query string. A request carrying both `sort` and `sortContent` MUST be rejected with `400` rather than resolved in one direction; in a `POST /records/query` body the same two spellings are `sort.field` and `sort.contentField`, and `sort.contentFieldByType` is a third that no request may combine with either. A server declaring `sort.contentField: false` refuses `sortContent` with `400`. What the ordering itself must be — absent values last, numbers before text, and the [folded text key](./data-model.md#text-ordering) a server MUST reproduce — is in [Data model § Sorting by a content field](./data-model.md#sorting-by-a-content-field). @@ -413,7 +413,7 @@ The record adapters sanitize a `search` string's _complexity_ — wildcards stri The mitigation therefore lives where the engine is driven, not in the sanitizers: run record-adapter queries off the request thread (a worker per connection, or a worker pool), so a slow query can be interrupted — `sqlite3_interrupt`, or terminating the worker — and apply the per-request deadline at that boundary. A server that does none of this is conforming but will serve exactly as well as its slowest query. -A [`filter.contentByType`](./data-model.md#content-by-type) of _n_ arms costs what _n_ content queries would, so it adds nothing to bound but a count, which a server sets by declaring its cap. +A [`filter.contentByType`](./data-model.md#content-by-type) of _n_ arms costs what _n_ content queries would, and is bounded the same way: by this deadline, not by a count of arms. `timeout` is a distinct code because a client acts on it differently from every other failure in the table: nothing was applied, and the same request may well succeed if narrowed or retried. Reusing `bad_request` would say the opposite. See [Data model § Capability-gated filters](./data-model.md#capability-gated-filters). diff --git a/packages/adapter-api/tests/api.test.ts b/packages/adapter-api/tests/api.test.ts index f783e1ae..a5009c29 100644 --- a/packages/adapter-api/tests/api.test.ts +++ b/packages/adapter-api/tests/api.test.ts @@ -1226,8 +1226,8 @@ describe('queryRecords', () => { const adapter = await openAdapter(discovery); expect(adapter.capabilities).toEqual({ - filter: { content: 'none', contentPresent: false, contentByType: 0, search: false }, - sort: { fields: [], contentField: false, contentFieldByType: 0 }, + filter: { content: 'none', contentPresent: false, contentByType: false, search: false }, + sort: { fields: [], contentField: false, contentFieldByType: false }, limits: { attachmentBytes: null, contentBytes: null }, }); @@ -1295,7 +1295,7 @@ describe('queryRecords', () => { ], ] as const)('refuses %s locally where the server declines it', async (capability, query) => { const adapter = await openAdapter( - discoveryWith({ filter: { contentByType: 0 }, sort: { contentFieldByType: 0 } }), + discoveryWith({ filter: { contentByType: false }, sort: { contentFieldByType: false } }), ); await expect(adapter.queryRecords(query)).rejects.toMatchObject({ capability }); expect(mockFetch).toHaveBeenCalledTimes(1); // only the discovery call — no request sent diff --git a/packages/adapter-api/tests/helpers.ts b/packages/adapter-api/tests/helpers.ts index b201d0fa..17cf0d39 100644 --- a/packages/adapter-api/tests/helpers.ts +++ b/packages/adapter-api/tests/helpers.ts @@ -23,11 +23,11 @@ export const OWNER = 'entity-owner-123'; * limit and not the other is the case the null default exists for. */ export const FULL_CAPABILITIES = { - filter: { content: 'path', contentPresent: true, contentByType: 64, search: true }, + filter: { content: 'path', contentPresent: true, contentByType: true, search: true }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 64, + contentFieldByType: true, }, limits: { attachmentBytes: 52428800 }, } satisfies DiscoveryCapabilities; diff --git a/packages/adapter-conformance/src/record.ts b/packages/adapter-conformance/src/record.ts index e5c269f1..651e599f 100644 --- a/packages/adapter-conformance/src/record.ts +++ b/packages/adapter-conformance/src/record.ts @@ -81,31 +81,25 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt describe('capabilities', () => { test('declares a well-formed StackCapabilities object', () => { expect(['none', 'field', 'path']).toContain(capabilities.filter.content); - expect(typeof capabilities.filter.contentPresent).toBe('boolean'); - expect(typeof capabilities.filter.search).toBe('boolean'); - expect(Array.isArray(capabilities.sort.fields)).toBe(true); - expect(typeof capabilities.sort.contentField).toBe('boolean'); - for (const cap of [ + for (const flag of [ + capabilities.filter.contentPresent, capabilities.filter.contentByType, + capabilities.filter.search, + capabilities.sort.contentField, capabilities.sort.contentFieldByType, ]) { - expect(Number.isInteger(cap) && cap >= 0).toBe(true); - } - }); - - test('the per-Type caps are 0 whenever filter.content is "none"', () => { - // Both travel in the body of the endpoint a reach of "none" omits. - if (capabilities.filter.content === 'none') { - expect(capabilities.filter.contentByType).toBe(0); - expect(capabilities.sort.contentFieldByType).toBe(0); + expect(typeof flag).toBe('boolean'); } + expect(Array.isArray(capabilities.sort.fields)).toBe(true); }); - test('filter.contentPresent is false whenever filter.content is "none"', () => { - // contentPresent asks whether a value is there at all — meaningless - // to promise from an adapter that reaches no content to look at. + // Each travels only in the POST /records/query body, which a reach of + // "none" does not serve. + test('what needs the query body is false whenever filter.content is "none"', () => { if (capabilities.filter.content === 'none') { expect(capabilities.filter.contentPresent).toBe(false); + expect(capabilities.filter.contentByType).toBe(false); + expect(capabilities.sort.contentFieldByType).toBe(false); } }); }); @@ -1284,6 +1278,42 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt }); } + // A value matches as `===` compares it: its JSON type as well as + // its value. See docs/spec/data-model.md § Filter. + test('a content filter value matches its own JSON type only', async () => { + const bool = makeRecord({ content: { title: 'b', flag: true } }); + const num = makeRecord({ content: { title: 'n', flag: 1 } }); + const text = makeRecord({ content: { title: 't', flag: '1' } }); + for (const record of [bool, num, text]) await adapter.createRecord(record); + + for (const [value, record] of [ + [true, bool], + [1, num], + ['1', text], + ] as const) { + const result = await adapter.queryRecords({ filter: { content: { flag: value } } }); + expect(result.records.map((r) => r.id)).toEqual([record.id]); + } + }); + + if (capabilities.filter.contentPresent && capabilities.filter.content === 'path') { + // Both are element-wise, so a path reaching a null and a value + // satisfies each. See docs/spec/data-model.md § Filter. + test('a multi-valued path satisfies both contentPresent and a null filter', async () => { + const mixed = makeRecord({ + content: { title: 'x', tags: [null, 'a'], emails: [{ value: null }, { value: 1 }] }, + }); + await adapter.createRecord(mixed); + + for (const path of ['tags', 'emails.value']) { + const present = await adapter.queryRecords({ filter: { contentPresent: [path] } }); + expect(present.records.map((r) => r.id)).toEqual([mixed.id]); + const nulls = await adapter.queryRecords({ filter: { content: { [path]: null } } }); + expect(nulls.records.map((r) => r.id)).toEqual([mixed.id]); + } + }); + } + if (capabilities.filter.content === 'path') { test('an array is spread element-wise when filtering a nested path', async () => { const match = makeRecord({ @@ -1327,7 +1357,7 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt // ----------------------------------------------------------------- // Content by type — capability-gated on filter.contentByType // ----------------------------------------------------------------- - if (capabilities.filter.contentByType >= 2) { + if (capabilities.filter.contentByType) { describe('content by type', () => { const OTHER = 'org.haverstack.conformance/other@1'; const THIRD = 'org.haverstack.conformance/third@1'; @@ -1406,7 +1436,7 @@ export function runRecordAdapterConformance(options: RecordAdapterConformanceOpt // ----------------------------------------------------------------- // Sort by content field per type — gated on sort.contentFieldByType // ----------------------------------------------------------------- - if (capabilities.sort.contentFieldByType >= 3) { + if (capabilities.sort.contentFieldByType) { describe('sort by content field per type', () => { const OTHER = 'org.haverstack.conformance/other@1'; const THIRD = 'org.haverstack.conformance/third@1'; diff --git a/packages/conformance-fixtures/src/discovery.ts b/packages/conformance-fixtures/src/discovery.ts index 61af2eec..986f9a83 100644 --- a/packages/conformance-fixtures/src/discovery.ts +++ b/packages/conformance-fixtures/src/discovery.ts @@ -21,13 +21,13 @@ export const discoveryFixtures: ConformanceFixture filter: { content: 'path', contentPresent: true, - contentByType: 16, + contentByType: true, search: true, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 16, + contentFieldByType: true, }, limits: { attachmentBytes: 52428800, @@ -49,11 +49,11 @@ export const discoveryFixtures: ConformanceFixture version: WIRE_PROTOCOL_VERSION, entityId: 'did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK', capabilities: { - filter: { content: 'none', contentPresent: false, contentByType: 0, search: false }, + filter: { content: 'none', contentPresent: false, contentByType: false, search: false }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: false, - contentFieldByType: 0, + contentFieldByType: false, }, limits: { attachmentBytes: null, contentBytes: null }, }, @@ -76,13 +76,13 @@ export const discoveryFixtures: ConformanceFixture filter: { content: 'none', contentPresent: false, - contentByType: 0, + contentByType: false, search: false, }, sort: { fields: ['createdAt'], contentField: false, - contentFieldByType: 0, + contentFieldByType: false, }, limits: { attachmentBytes: null, @@ -110,13 +110,13 @@ export const discoveryFixtures: ConformanceFixture filter: { content: 'path', contentPresent: true, - contentByType: 16, + contentByType: true, search: true, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 16, + contentFieldByType: true, }, limits: { attachmentBytes: 52428800, @@ -144,13 +144,13 @@ export const discoveryFixtures: ConformanceFixture filter: { content: 'path', contentPresent: true, - contentByType: 16, + contentByType: true, search: true, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 16, + contentFieldByType: true, }, limits: { attachmentBytes: 52428800, @@ -178,13 +178,13 @@ export const discoveryFixtures: ConformanceFixture filter: { content: 'none', contentPresent: false, - contentByType: 0, + contentByType: false, search: false, }, sort: { fields: ['createdAt'], contentField: false, - contentFieldByType: 0, + contentFieldByType: false, }, limits: { attachmentBytes: null, diff --git a/packages/conformance-fixtures/src/query.ts b/packages/conformance-fixtures/src/query.ts index abf534ab..eda27e44 100644 --- a/packages/conformance-fixtures/src/query.ts +++ b/packages/conformance-fixtures/src/query.ts @@ -176,7 +176,7 @@ export const queryRecordsFixtures: ConformanceFixture< 'filter across the versions of a family whose migrations renamed a field — here `by` at ' + '@1 and `owner` at @2. Types partition Records, so the arms split the query rather than ' + 'widening it, and every other filter key applies to each arm. A server declaring ' + - 'filter.contentByType: 0 MUST refuse it with 400. See docs/spec/data-model.md § Content ' + + 'filter.contentByType: false MUST refuse it with 400. See docs/spec/data-model.md § Content ' + 'by type.', method: 'POST', path: '/records/query', @@ -219,7 +219,7 @@ export const queryRecordsFixtures: ConformanceFixture< 'that Type declares it, or a constant already reduced to a number or text — here 5 for ' + 'every @1 Record, which predates the field. A Record whose typeId is not a key, or whose ' + 'value is null, holds no value and sorts last in either direction. A server declaring ' + - 'sort.contentFieldByType: 0 MUST refuse it with 400. See docs/spec/data-model.md ' + + 'sort.contentFieldByType: false MUST refuse it with 400. See docs/spec/data-model.md ' + '§ Sorting by a content field.', method: 'POST', path: '/records/query', diff --git a/packages/core/src/collection.ts b/packages/core/src/collection.ts index 5b1489a7..30ad21c9 100644 --- a/packages/core/src/collection.ts +++ b/packages/core/src/collection.ts @@ -492,16 +492,11 @@ export const createCollection = ( const predicate = contentPredicate(filter); const sortField = sort?.contentField; const { capabilities } = client; - // A version past the adapter's cap is narrowed, newest kept first: - // the older a version, the fewer apps still write it. - const filterCap = - predicate === undefined - ? Infinity - : capabilities.filter.content === 'none' - ? 0 - : capabilities.filter.contentByType; - const sortCap = sortField === undefined ? Infinity : capabilities.sort.contentFieldByType; - const cap = Math.min(filterCap, sortCap); + // Spanning versions takes the per-Type shape of each content key the + // query carries; an adapter without one gets the handle's version alone. + const spans = + (predicate === undefined || capabilities.filter.contentByType) && + (sortField === undefined || capabilities.sort.contentFieldByType); const sortKind = (() => { const def = sortField === undefined ? undefined : handle.schema[sortField]; return def && def.kind !== 'array' && def.kind !== 'object' ? def.kind : undefined; @@ -511,9 +506,7 @@ export const createCollection = ( pinned ? pinned.map((id) => ({ id, baseId: handle.baseId, version: versionOf(id) })) : await listTypes() - ) - .filter((type) => type.baseId === handle.baseId && type.version !== handleVersion) - .sort((a, b) => b.version - a.version); + ).filter((type) => type.baseId === handle.baseId && type.version !== handleVersion); const arms: Record = { [handle.id]: predicate ?? {} }; const keys: Record = sortField ? { [handle.id]: { field: sortField } } @@ -539,11 +532,7 @@ export const createCollection = ( : origins ? translateSortKey(origins, sortField, sortKind) : UNTRANSLATABLE; - if ( - translated === UNTRANSLATABLE || - key === UNTRANSLATABLE || - Object.keys(arms).length >= cap - ) { + if (translated === UNTRANSLATABLE || key === UNTRANSLATABLE || !spans) { narrowed.push(type.id); continue; } diff --git a/packages/core/src/content-match.ts b/packages/core/src/content-match.ts index 5b2857b6..2ddd0b41 100644 --- a/packages/core/src/content-match.ts +++ b/packages/core/src/content-match.ts @@ -17,8 +17,8 @@ const spreadValue = (value: unknown): unknown[] => (Array.isArray(value) ? value /** * Every value reachable by walking `segments`, spreading arrays at each - * step — sqlite-shared's contentPathExists() walk, so a filter answers the - * same against either. Empty means the path reaches nothing, which a `null` + * step — the walk sqlite-shared runs in SQL, so a filter answers the same + * against either. Empty means the path reaches nothing, which a `null` * filter value matches. */ export const valuesAtContentPath = ( diff --git a/packages/core/src/query-validation.ts b/packages/core/src/query-validation.ts index 1fe0143f..6a4288f6 100644 --- a/packages/core/src/query-validation.ts +++ b/packages/core/src/query-validation.ts @@ -21,6 +21,7 @@ import { familyIdProblem, parseTypeId } from './schema.js'; import { targetErrors } from './associations/validation.js'; import { CONTENT_SEGMENT_METACHARACTERS, + MAX_VALIDATION_DEPTH, SEGMENT_METACHARACTER_RE, isPlainObject, } from './validate.js'; @@ -71,13 +72,10 @@ export function assertQueryCapabilities( 'filter.content', ); } - if (arms.length > contentByType) { + if (arms.length > 0 && !contentByType) { throw new StackBadRequestError( - contentByType === 0 - ? 'Query uses filter.contentByType, but this adapter does not declare the ' + - 'filter.contentByType capability.' - : `Query uses ${arms.length} filter.contentByType arms, but this adapter takes at ` + - `most ${contentByType}.`, + 'Query uses filter.contentByType, but this adapter does not declare the ' + + 'filter.contentByType capability.', 'filter.contentByType', ); } @@ -167,8 +165,13 @@ export function filtersContent(features: Pick): boo return features.filter.content !== 'none'; } -/** The longest path both SQLite engines can execute — see the spec link below. */ -const MAX_CONTENT_PATH_SEGMENTS = 32; +/** + * The longest path that can reach a stored value: a top-level field sits at + * depth 0 and each segment descends one level, so a path past this reaches + * nothing a write accepts. See docs/spec/data-model.md § Nested content + * paths. + */ +const MAX_CONTENT_PATH_SEGMENTS = MAX_VALIDATION_DEPTH + 1; /** * Split a content filter key into path segments. Write-time validation @@ -178,9 +181,6 @@ const MAX_CONTENT_PATH_SEGMENTS = 32; */ export function parseContentFilterKey(key: string): string[] { const segments = key.split('.'); - // A SQLite adapter spends two json_each joins per segment against a - // 64-table join limit, so a path past the cap is one the engine could - // not execute. See docs/spec/data-model.md § Nested content paths. if (segments.length > MAX_CONTENT_PATH_SEGMENTS) { throw new StackBadRequestError( `Invalid content filter path "${key}": at most ${MAX_CONTENT_PATH_SEGMENTS} segments.`, @@ -324,14 +324,10 @@ export function assertSortCapability( ): void { if (!sort) return; if (sort.contentFieldByType !== undefined) { - const keys = Object.keys(sort.contentFieldByType).length; - const cap = capabilities.sort.contentFieldByType; - if (keys > cap) { + if (!capabilities.sort.contentFieldByType) { throw new StackBadRequestError( - cap === 0 - ? 'Query uses sort.contentFieldByType, but this adapter does not declare the ' + - 'sort.contentFieldByType capability.' - : `Query uses ${keys} sort.contentFieldByType keys, but this adapter takes at most ${cap}.`, + 'Query uses sort.contentFieldByType, but this adapter does not declare the ' + + 'sort.contentFieldByType capability.', 'sort.contentFieldByType', ); } diff --git a/packages/core/src/testing.ts b/packages/core/src/testing.ts index b0daefa5..f81f7ce6 100644 --- a/packages/core/src/testing.ts +++ b/packages/core/src/testing.ts @@ -83,13 +83,13 @@ export class MemoryAdapter implements StackAdapter { filter: { content: 'path', contentPresent: true, - contentByType: 64, + contentByType: true, search: false, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 64, + contentFieldByType: true, }, limits: { attachmentBytes: null, @@ -725,13 +725,13 @@ export class IncapableMemoryAdapter extends MemoryAdapter { filter: { content: 'none', contentPresent: false, - contentByType: 0, + contentByType: false, search: false, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: false, - contentFieldByType: 0, + contentFieldByType: false, }, limits: { attachmentBytes: null, diff --git a/packages/core/src/types/capabilities.ts b/packages/core/src/types/capabilities.ts index 407c894d..4ca216b2 100644 --- a/packages/core/src/types/capabilities.ts +++ b/packages/core/src/types/capabilities.ts @@ -40,12 +40,12 @@ export type StackCapabilities = { */ contentPresent: boolean; /** - * The most arms `filter.contentByType` may hold; `0` declines it. A - * collection narrows the versions it cannot reach rather than failing. - * Local adapters must honor it. See docs/spec/adapters.md § Adapter - * capabilities. + * Whether `filter.contentByType` is honored. Local adapters must; a + * collection facing `false` queries its handle's version alone rather + * than failing. Meaningful only where `content` is not `'none'`. See + * docs/spec/adapters.md § Adapter capabilities. */ - contentByType: number; + contentByType: boolean; /** * Whether `filter.search` is honored. Not a rung on the ladder: a * full-text index is a different mechanism from field matching, and @@ -62,8 +62,8 @@ export type StackCapabilities = { * scalar. Independent of `filter.content`. */ contentField: boolean; - /** The most keys `sort.contentFieldByType` may hold; `0` declines it. */ - contentFieldByType: number; + /** Whether `sort.contentFieldByType` is honored, as `filter.contentByType`. */ + contentFieldByType: boolean; }; /** * Ceilings a client can check before spending a request; the server's diff --git a/packages/core/src/validate.ts b/packages/core/src/validate.ts index 3f260f04..9618e844 100644 --- a/packages/core/src/validate.ts +++ b/packages/core/src/validate.ts @@ -12,7 +12,8 @@ import type { TypeSchema, FieldDef, ScalarFieldKind } from './types/index.js'; -const MAX_VALIDATION_DEPTH = 32; +/** How deep content may nest below a top-level field, which is depth 0. */ +export const MAX_VALIDATION_DEPTH = 32; /** fileId format: SHA-256 hex, lowercase — matches blob-adapter-disk's assertFileId(). */ const FILE_ID_RE = /^[0-9a-f]{64}$/; diff --git a/packages/core/tests/collection.test.ts b/packages/core/tests/collection.test.ts index df72a2fe..b682f0c9 100644 --- a/packages/core/tests/collection.test.ts +++ b/packages/core/tests/collection.test.ts @@ -692,12 +692,12 @@ describe.each([ let adapter: MemoryAdapter; let stack: Stack; - const open = async (filterArms = 64, sortKeys = 64) => { + const open = async (byType = true) => { adapter = await MemoryAdapter.open({ ownerEntityId: OWNER, timezone: 'UTC' }); (adapter as { capabilities: StackClient['capabilities'] }).capabilities = { ...adapter.capabilities, - filter: { ...adapter.capabilities.filter, contentByType: filterArms }, - sort: { ...adapter.capabilities.sort, contentFieldByType: sortKeys }, + filter: { ...adapter.capabilities.filter, contentByType: byType }, + sort: { ...adapter.capabilities.sort, contentFieldByType: byType }, }; stack = await Stack.open(adapter, { migrations: TASK_MIGRATIONS }); for (const type of [TaskV1, TaskV2, Task]) await stack.defineType(type); @@ -803,7 +803,7 @@ describe.each([ }); test('an adapter that declines the shape gets the handle-only query', async () => { - await open(0, 0); + await open(false); const [, , v3] = await seed('Ada'); const filtered = await tasks().query({ filter: { content: { owner: 'Ada' } } }); expect(filtered.records.map((r) => r.id)).toEqual([v3.id]); @@ -813,14 +813,6 @@ describe.each([ expect(sorted.skipped).toEqual({ older: true, newer: false }); }); - test('an adapter with a lower cap gets the newest versions that fit', async () => { - await open(2); - const [, v2, v3] = await seed('Ada'); - const page = await tasks().query({ filter: { content: { owner: 'Ada' } } }); - expect(page.records.map((r) => r.id).sort()).toEqual([v2.id, v3.id].sort()); - expect(page.skipped).toEqual({ older: true, newer: false }); - }); - test('a migration not built by migration() is narrowed', async () => { stack = await Stack.open(adapter, { migrations: [ diff --git a/packages/core/tests/combine.test.ts b/packages/core/tests/combine.test.ts index 61655d1e..c7037391 100644 --- a/packages/core/tests/combine.test.ts +++ b/packages/core/tests/combine.test.ts @@ -28,13 +28,13 @@ const capabilities: StackCapabilities = { filter: { content: 'none', contentPresent: false, - contentByType: 0, + contentByType: false, search: false, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: false, - contentFieldByType: 0, + contentFieldByType: false, }, limits: { attachmentBytes: null, diff --git a/packages/core/tests/query-capabilities.test.ts b/packages/core/tests/query-capabilities.test.ts index 69cfea36..495c38d5 100644 --- a/packages/core/tests/query-capabilities.test.ts +++ b/packages/core/tests/query-capabilities.test.ts @@ -25,11 +25,11 @@ import type { } from '../src/types/index.js'; const ALL: StackCapabilities = { - filter: { content: 'path', contentPresent: true, contentByType: 64, search: true }, + filter: { content: 'path', contentPresent: true, contentByType: true, search: true }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 64, + contentFieldByType: true, }, limits: { attachmentBytes: null, contentBytes: null }, }; @@ -110,17 +110,10 @@ describe('a refused query names the capability it needed', () => { test('filter.contentByType declined', () => { expect( - refusalFor(() => assertQueryCapabilities(ARMS, without({ filter: { contentByType: 0 } }))), - ).toBe('filter.contentByType'); - }); - - test('filter.contentByType past its cap', () => { - expect( - refusalFor(() => assertQueryCapabilities(ARMS, without({ filter: { contentByType: 1 } }))), + refusalFor(() => + assertQueryCapabilities(ARMS, without({ filter: { contentByType: false } })), + ), ).toBe('filter.contentByType'); - expect(() => - assertQueryCapabilities(ARMS, without({ filter: { contentByType: 2 } })), - ).not.toThrow(); }); test("filter.contentByType against reach 'none'", () => { @@ -151,14 +144,23 @@ describe('a refused query names the capability it needed', () => { ).toBe('filter.contentPresent'); }); - test('sort.contentFieldByType declined, or past its cap', () => { + test('sort.contentFieldByType declined', () => { const sort = { contentFieldByType: { 'com.example/x@1': { field: 'a' } } }; expect( - refusalFor(() => assertSortCapability(sort, without({ sort: { contentFieldByType: 0 } }))), + refusalFor(() => + assertSortCapability(sort, without({ sort: { contentFieldByType: false } })), + ), ).toBe('sort.contentFieldByType'); - expect(() => - assertSortCapability(sort, without({ sort: { contentFieldByType: 1 } })), - ).not.toThrow(); + }); + + test('a declared per-Type shape takes any number of keys', () => { + const typeIds = Array.from({ length: 500 }, (_, i) => `com.example/x@${i + 1}`); + const filter = { contentByType: Object.fromEntries(typeIds.map((id) => [id, {}])) }; + const sort = { + contentFieldByType: Object.fromEntries(typeIds.map((id) => [id, { field: 'a' }])), + }; + expect(() => assertQueryCapabilities(filter, ALL)).not.toThrow(); + expect(() => assertSortCapability(sort, ALL)).not.toThrow(); }); test('sort.fields', () => { diff --git a/packages/core/tests/stack.test.ts b/packages/core/tests/stack.test.ts index f917bf0b..73763859 100644 --- a/packages/core/tests/stack.test.ts +++ b/packages/core/tests/stack.test.ts @@ -1106,6 +1106,24 @@ describe('query — content filter null semantics', () => { ); }); + // The path cap and the content depth cap are one limit read from either + // side. See docs/spec/data-model.md § Nested content paths. + test('the longest legal path reaches the deepest value a write accepts', async () => { + const chain = (n: number): Record => + n === 1 ? { n: 'x' } : { n: chain(n - 1) }; + const path = (n: number) => Array(n).fill('n').join('.'); + + const deepest = await stack.create(NOTE_V1, { text: 'deep', ...chain(33) }); + const found = await stack.query({ filter: { content: { [path(33)]: 'x' } } }); + expect(found.records.map((r) => r.id)).toEqual([deepest.id]); + await expect(stack.create(NOTE_V1, { text: 'deeper', ...chain(34) })).rejects.toThrow( + /Content nesting exceeds maximum depth of 32/, + ); + await expect(stack.query({ filter: { content: { [path(34)]: 'x' } } })).rejects.toThrow( + StackBadRequestError, + ); + }); + // A dotted key is a path, and a field literally named `a.b` is refused // at write time, so the two readings can never both be available. test('a dotted key is a path, and the colliding field name is unwritable', async () => { @@ -4167,7 +4185,7 @@ describe('grantType', () => { const bare = await MemoryAdapter.open({ ownerEntityId: 'owner-123', timezone: 'UTC' }); (bare as { capabilities: StackAdapter['capabilities'] }).capabilities = { ...bare.capabilities, - sort: { fields: [], contentField: false, contentFieldByType: 0 }, + sort: { fields: [], contentField: false, contentFieldByType: false }, }; const bareStack = await Stack.open(bare); expect(await bareStack.listTypeGrants()).toEqual([]); @@ -6000,13 +6018,13 @@ describe('nested content paths', () => { filter: { content: 'field', contentPresent: false, - contentByType: 0, + contentByType: false, search: false, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: false, - contentFieldByType: 0, + contentFieldByType: false, }, limits: { attachmentBytes: null, @@ -6037,11 +6055,11 @@ describe('limits.contentBytes pre-check', () => { const withContentCeiling = async (contentBytes: number): Promise => Object.assign(await MemoryAdapter.open({ ownerEntityId: 'owner-123', timezone: 'UTC' }), { capabilities: { - filter: { content: 'path', contentPresent: true, contentByType: 64, search: false }, + filter: { content: 'path', contentPresent: true, contentByType: true, search: false }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 64, + contentFieldByType: true, }, limits: { attachmentBytes: null, contentBytes }, }, @@ -6095,11 +6113,11 @@ describe('putAttachment — limits.attachmentBytes pre-check', () => { const withCeiling = async (attachmentBytes: number): Promise => Object.assign(await MemoryAdapter.open({ ownerEntityId: 'owner-123', timezone: 'UTC' }), { capabilities: { - filter: { content: 'none', contentPresent: false, contentByType: 0, search: false }, + filter: { content: 'none', contentPresent: false, contentByType: false, search: false }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: false, - contentFieldByType: 0, + contentFieldByType: false, }, limits: { attachmentBytes, contentBytes: null }, }, diff --git a/packages/record-adapter-do-sqlite/tests/record.test.ts b/packages/record-adapter-do-sqlite/tests/record.test.ts index 10945f61..156cc134 100644 --- a/packages/record-adapter-do-sqlite/tests/record.test.ts +++ b/packages/record-adapter-do-sqlite/tests/record.test.ts @@ -113,13 +113,13 @@ describe('construction', () => { filter: { content: 'path', contentPresent: true, - contentByType: 64, + contentByType: true, search: true, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 64, + contentFieldByType: true, }, limits: { attachmentBytes: null, @@ -333,20 +333,19 @@ describe('records — queries', () => { expect(result.records.map((r) => r.id)).toEqual(['r1']); }); - // The segment cap is sized against SQLite's join limit, and this build - // is where a difference in that limit would surface. The longest legal - // path has to execute, not merely be accepted. + // This build is where a difference in SQLite's limits would surface. The + // longest legal path has to execute, not merely be accepted. test('a path at the segment cap executes; one past it is refused', async () => { const stub = getStub(); await stub.createRecord(makeRecord({ id: 'r1', content: { text: 'shallow' } })); - const deepest = Array.from({ length: 32 }, (_, i) => `s${i}`).join('.'); + const deepest = Array.from({ length: 33 }, (_, i) => `s${i}`).join('.'); const miss = await stub.queryRecords({ filter: { content: { [deepest]: 'x' } } }); expect(miss.records).toEqual([]); const absent = await stub.queryRecords({ filter: { content: { [deepest]: null } } }); expect(absent.records.map((r) => r.id)).toEqual(['r1']); const err = await stub - .queryRecords({ filter: { content: { [`${deepest}.s32`]: 'x' } } }) + .queryRecords({ filter: { content: { [`${deepest}.s33`]: 'x' } } }) .catch((e: unknown) => e); expect((err as { code?: string }).code).toBe('bad_request'); }); @@ -552,7 +551,7 @@ describe('commitMigration', () => { // A reach is sized by the requester's grants and rosters, and this engine // caps a statement at 100 bound parameters, so a reach binds each of its -// lists as one. See docs/spec/adapters.md § Querying within a reach. +// lists as one. See docs/spec/adapters.md § Statement size. test('queryReadable answers a reach far past the parameter cap', async () => { const stub = getStub(); const shared = makeRecord({ @@ -598,3 +597,73 @@ test('a tag filter far past the parameter cap', async () => { const result = await stub.queryRecords({ filter: { tags: labels } }); expect(result.records.map((r) => r.id)).toEqual([tagged.id]); }); + +// A per-Type shape carries no count, so it has to hold on the engine with +// the tightest statement limits, far past any count a statement could have +// afforded. See docs/spec/adapters.md § Statement size. +describe('per-Type keys far past the statement limits', () => { + const KEYS = 200; + const typeIds = (count: number) => + Array.from({ length: count }, (_, i) => (i === 0 ? NOTE_TYPE_V1 : `com.example/t${i}@1`)); + + test('filter.contentByType', async () => { + const stub = getStub(); + await stub.createRecord(makeRecord({ id: 'r1', content: { text: 'a', meta: { n: 1 } } })); + await stub.createRecord(makeRecord({ id: 'r2', content: { text: 'b', meta: { n: 2 } } })); + const contentByType = Object.fromEntries( + typeIds(KEYS).map((typeId) => [ + typeId, + { content: { 'meta.n': 1, 'meta.missing': null }, contentPresent: ['text'] }, + ]), + ); + const result = await stub.queryRecords({ filter: { contentByType } }); + expect(result.records.map((r) => r.id)).toEqual(['r1']); + }); + + test('sort.contentFieldByType', async () => { + const stub = getStub(); + const other = 'com.example/t1@1'; + await stub.saveType({ + id: NOTE_TYPE_V1, + baseId: 'com.example.test/note', + version: 1, + name: 'Note', + schema: { text: { kind: 'string' }, n: { kind: 'number' } }, + schemaHash: 'note-hash', + createdAt: new Date(), + }); + await stub.createRecord(makeRecord({ id: 'r1', content: { text: 'a', n: 3 } })); + await stub.createRecord(makeRecord({ id: 'r2', content: { text: 'b', n: 1 } })); + await stub.createRecord(makeRecord({ id: 'r3', typeId: other, content: { text: 'c' } })); + await stub.createRecord(makeRecord({ id: 'r4', content: { text: 'd' } })); + const contentFieldByType = Object.fromEntries( + typeIds(KEYS).map((typeId, i) => [typeId, i % 2 === 0 ? { field: 'n' } : { value: 2 }]), + ); + const result = await stub.queryRecords({ + sort: { contentFieldByType, direction: 'asc' }, + limit: 2, + }); + expect(result.records.map((r) => r.id)).toEqual(['r2', 'r3']); + const rest = await stub.queryRecords({ + sort: { contentFieldByType, direction: 'asc' }, + cursor: result.cursor!, + }); + expect(rest.records.map((r) => r.id)).toEqual(['r1', 'r4']); + }); +}); + +// Nothing caps how many keys filter.content holds, so neither they nor their +// paths can cost statement text or a parameter each. See +// docs/spec/adapters.md § Statement size. +test('content conditions far past the statement limits execute', async () => { + const stub = getStub(); + await stub.createRecord(makeRecord({ id: 'r1', content: { text: 'a', n: 1 } })); + await stub.createRecord(makeRecord({ id: 'r2', content: { text: 'b', n: 2 } })); + const prefix = Array.from({ length: 32 }, (_, i) => `s${i}`).join('.'); + const content = Object.fromEntries( + Array.from({ length: 8 }, (_, i) => [`${prefix}.k${i}`, null] as [string, unknown]), + ); + content.n = 1; + const result = await stub.queryRecords({ filter: { content } }); + expect(result.records.map((r) => r.id)).toEqual(['r1']); +}); diff --git a/packages/record-adapter-sqlite/tests/record.test.ts b/packages/record-adapter-sqlite/tests/record.test.ts index 681274ac..a68bed18 100644 --- a/packages/record-adapter-sqlite/tests/record.test.ts +++ b/packages/record-adapter-sqlite/tests/record.test.ts @@ -1049,22 +1049,27 @@ describe('records — queries', () => { } }); - // The segment cap is what keeps the generated join list inside SQLite's - // 64-table limit, so the longest legal path has to be executable, not - // merely accepted. See docs/spec/data-model.md § Nested content paths. - test('a path at the segment cap executes; one past it is refused', async () => { - const adapter = await initAdapter(); - const deepest = Array.from({ length: 32 }, (_, i) => `s${i}`).join('.'); + // The longest legal path reaches the deepest value a write accepts, and + // has to execute, not merely be accepted. See docs/spec/data-model.md + // § Nested content paths. + test('a path at the segment cap reaches the deepest value; one past it is refused', async () => { + const adapter = await initAdapter(); + const segments = Array.from({ length: 33 }, (_, i) => `s${i}`); + const deepest = segments.join('.'); + const nested = segments.reduceRight((value, key) => ({ [key]: value }), 'x'); await adapter.createRecord(makeRecord({ id: 'r1', content: { text: 'shallow' } })); + await adapter.createRecord( + makeRecord({ id: 'r2', content: { text: 'deep', ...(nested as object) } }), + ); await expect( adapter.queryRecords({ filter: { content: { [deepest]: 'x' } } }), - ).resolves.toMatchObject({ records: [] }); + ).resolves.toMatchObject({ records: [{ id: 'r2' }] }); await expect( adapter.queryRecords({ filter: { content: { [deepest]: null } } }), ).resolves.toMatchObject({ records: [{ id: 'r1' }] }); await expect( - adapter.queryRecords({ filter: { content: { [`${deepest}.s32`]: 'x' } } }), + adapter.queryRecords({ filter: { content: { [`${deepest}.s33`]: 'x' } } }), ).rejects.toThrow(StackBadRequestError); }); diff --git a/packages/sqlite-shared/src/query.ts b/packages/sqlite-shared/src/query.ts index 5b51663d..cc64a324 100644 --- a/packages/sqlite-shared/src/query.ts +++ b/packages/sqlite-shared/src/query.ts @@ -66,65 +66,94 @@ const sqlDirection = (query: StackQuery): 'ASC' | 'DESC' => { /** * Normalize a json_each row's value so one array element and one bare * value are walked by the same clause: an array stands for its elements, - * anything else for itself. See docs/spec/data-model.md § Nested content - * paths. + * anything else for itself. A boolean is rewrapped from its type, since + * json_each's `value` reads it as 0 or 1. See docs/spec/data-model.md + * § Nested content paths. */ const spread = (alias: string): string => - `CASE WHEN ${alias}.type = 'array' THEN ${alias}.value ` + - `WHEN ${alias}.type = 'object' THEN json_array(json(${alias}.value)) ` + + `CASE ${alias}.type WHEN 'array' THEN ${alias}.value ` + + `WHEN 'object' THEN json_array(json(${alias}.value)) ` + + `WHEN 'true' THEN '[true]' WHEN 'false' THEN '[false]' ` + `ELSE json_array(${alias}.value) END`; /** - * The members of a json_each row that is an object, and none for a row - * that is not — a path descending through a scalar reaches no value rather - * than reaching an error. + * The members of a walk row that is an object, and none for a row that is + * not — a path descending through a scalar reaches no value rather than + * reaching an error. */ const members = (alias: string): string => - `CASE WHEN ${alias}.type = 'object' THEN ${alias}.value ELSE '{}' END`; + `CASE WHEN ${alias}.t = 'object' THEN ${alias}.v ELSE '{}' END`; /** - * True when some value reached by `segments` satisfies `leaf`, applied to - * the alias holding that value. Each segment is a bound parameter matched - * against json_each's `key`, never interpolated into a path expression, so - * no key can be read as syntax. See docs/spec/data-model.md § Filter. + * One statement as it is built: its tables, then its conditions on `r` and + * the parameters its text binds, in textual order. Every table's parameter + * binds ahead of the text's, since the tables lead the statement. */ -const contentPathExists = ( - segments: string[], - leaf: (alias: string) => string, - params: unknown[], - nextAlias: () => string, -): string => { - // The walk is a flat join list, not a subquery per segment: both SQLite - // builds cap expression-tree depth far below the segment cap, and - // nesting spends that budget while a join list spends the join budget - // the cap is actually sized against. - let key = nextAlias(); - let value = nextAlias(); - const from = [`json_each(r.content) AS ${key}`, `json_each(${spread(key)}) AS ${value}`]; - const where = [`${key}.key = ?`]; - params.push(segments[0]); - - for (const segment of segments.slice(1)) { - key = nextAlias(); - from.push(`json_each(${members(value)}) AS ${key}`); - value = nextAlias(); - from.push(`json_each(${spread(key)}) AS ${value}`); - where.push(`${key}.key = ?`); - params.push(segment); - } +type Statement = { + tables: string[]; + tableParams: unknown[]; + conditions: string[]; + params: unknown[]; +}; + +const emptyStatement = (): Statement => ({ + tables: [], + tableParams: [], + conditions: ["r.id != '_config'"], + params: [], +}); - where.push(leaf(value)); - return `EXISTS (SELECT 1 FROM ${from.join(', ')} WHERE ${where.join(' AND ')})`; +/** + * Tuples as a table of `s` with `columns`, named in tuple order: the one + * shape a caller-sized part of a query takes. It binds as one JSON + * parameter, so the statement never grows with it, and is materialized + * once, so nothing that reads it per record re-parses the JSON. Returns the + * table's name. See docs/spec/adapters.md § Statement size. + */ +const rows = (s: Statement, columns: readonly string[], tuples: unknown[][]): string => { + const name = `t${s.tables.length}`; + s.tables.push(`${name} AS MATERIALIZED ${rowsTable(columns)}`); + s.tableParams.push(JSON.stringify(tuples)); + return name; +}; + +/** The table rows() binds, for a statement outside the query plan. */ +export const rowsTable = (columns: readonly string[]): string => { + const named = columns.map((column, i) => `json_extract(value, '$[${i}]') AS ${column}`); + return `(SELECT ${named.join(', ')} FROM json_each(?))`; }; +/** The columns of a content condition; see contentRows(). */ +const CONDITION_COLUMNS = ['scope', 'path', 'op', 'val', 'kind']; + /** - * A scalar filter value never matches an object or array stored at the - * path: json_each exposes those as their JSON text, which would compare - * equal to a string spelling the same document while every non-SQL adapter - * compares the value itself. + * A JSON value's type as json_each reports it, with integer and real as one + * number: what an `equals` condition compares besides the value itself, as + * `===` does. valueKind() reads a filter value the same way. */ -const equalsLeaf = (alias: string): string => - `${alias}.type NOT IN ('object', 'array') AND ${alias}.value = ?`; +const kindOf = (type: string): string => + `CASE ${type} WHEN 'integer' THEN 'number' WHEN 'real' THEN 'number' ELSE ${type} END`; + +/** + * Whether condition row `c` holds for record row `r`: one walk of the path + * as data, a segment matched against json_each's `key` per step, gathering + * what each op asks of the values it reaches. See docs/spec/adapters.md + * § Statement size and docs/spec/data-model.md § Filter. + */ +const holds = (c: string, r: string): string => { + const depth = `json_array_length(${c}.path)`; + return ( + `(WITH RECURSIVE w(d, v, t) AS (SELECT 0, ${r}.content, 'object' UNION ALL ` + + `SELECT w.d + 1, e.value, e.type FROM w JOIN json_each(${members('w')}) m ` + + `JOIN json_each(${spread('m')}) e ` + + `WHERE w.d < ${depth} AND m.key = json_extract(${c}.path, '$[' || w.d || ']')) ` + + `SELECT CASE ${c}.op ` + + `WHEN 'equals' THEN IFNULL(MAX(${kindOf('w.t')} = ${c}.kind AND w.v = ${c}.val), 0) ` + + "WHEN 'present' THEN IFNULL(MAX(w.t != 'null'), 0) " + + "WHEN 'isNull' THEN IFNULL(MAX(w.t = 'null'), 0) OR NOT IFNULL(MAX(w.t != 'null'), 0) " + + `END FROM w WHERE w.d = ${depth})` + ); +}; /** The join alias the content index is read through. */ const SORT_ALIAS = 'cs'; @@ -151,87 +180,72 @@ const contentCursor = (cursor: DecodedCursor, field: string) => { return cursor; }; +/** A filter value's kindOf(), which it mirrors, or null for one nothing equals. */ +const valueKind = (value: unknown): string | null => { + if (typeof value === 'boolean') return `${value}`; + if (typeof value === 'number') return 'number'; + return typeof value === 'string' ? 'text' : null; +}; + /** - * A `content` and `contentPresent` predicate as conditions on `r`, binding - * in textual order. A dot-separated path is matched element-wise through - * arrays, and a `null` value means "no value at the path, or a value that - * is null" (docs/spec/data-model.md § Filter): the second arm below, "some - * value there is null" OR "nothing is there at all". + * A `content` and `contentPresent` predicate as condition rows scoped to + * `scope`: path segments, the op, and for `equals` the value and its + * valueKind(). A `null` value means "no value at the path, or a value that + * is null" (docs/spec/data-model.md § Filter). */ -const contentConditions = ( - predicate: ContentArm, - params: unknown[], - nextAlias: () => string, -): string[] => { - const conditions: string[] = []; - for (const [key, value] of Object.entries(predicate.content ?? {})) { - const segments = parseContentFilterKey(key); - // undefined is the same absence null names — JSON drops it on the - // way over the wire, so treating it as a value would make an - // in-process query and its wire equivalent disagree. - if (value === null || value === undefined) { - const isNull = contentPathExists(segments, (a) => `${a}.value IS NULL`, params, nextAlias); - const anyValue = contentPathExists( - segments, - (a) => `${a}.value IS NOT NULL`, - params, - nextAlias, - ); - conditions.push(`(${isNull} OR NOT ${anyValue})`); - } else { - const match = contentPathExists(segments, equalsLeaf, params, nextAlias); - params.push(value); - conditions.push(match); - } - } - // Presence — the question an exact-match value cannot ask: a path holds - // a value when at least one non-null value is reachable at it. - for (const key of predicate.contentPresent ?? []) { - conditions.push( - contentPathExists( - parseContentFilterKey(key), - (a) => `${a}.value IS NOT NULL`, - params, - nextAlias, - ), - ); - } - return conditions; +const contentRows = (scope: string | null, predicate: ContentArm): unknown[][] => { + const row = (key: string, op: string, value: unknown = null) => { + const path = parseContentFilterKey(key); + return [scope, path, op, value, valueKind(value)]; + }; + return [ + ...Object.entries(predicate.content ?? {}).map(([key, value]) => + // undefined is the same absence null names — JSON drops it on the way + // over the wire, so treating it as a value would make an in-process + // query and its wire equivalent disagree. + value === null || value === undefined ? row(key, 'isNull') : row(key, 'equals', value), + ), + ...(predicate.contentPresent ?? []).map((key) => row(key, 'present')), + ]; }; /** - * Values as the condition matching any of them. A list binds as one JSON - * parameter, since it is caller- or reach-sized and a statement's parameter - * count is capped (at 100 on Durable Object SQLite); one value binds as - * itself, so an index on the column still yields rows in its order. An - * empty list matches nothing. + * Values as the condition matching any of them: rows() of one column, or + * one value bound as itself, so an index on the column still yields rows in + * its order. An empty list matches nothing. */ -const inList = (column: string, values: readonly string[], params: unknown[]): string => { +const inList = (s: Statement, column: string, values: readonly string[]): string => { if (values.length === 1) { - params.push(values[0]); + s.params.push(values[0]); return `${column} = ?`; } - params.push(JSON.stringify(values)); - return `${column} IN (SELECT value FROM json_each(?))`; + const table = rows( + s, + ['value'], + values.map((value) => [value]), + ); + return `${column} IN (SELECT value FROM ${table})`; }; /** * Whether the record carries every element of one of `sets`, matched on the * permissions table's key — the identity editAccess() removes by — through - * idx_permissions_element. The sets travel as one JSON parameter, each row - * naming its set and that set's size; an empty set matches nothing. + * idx_permissions_element. Each element is a row naming its set and that + * set's size; an empty set matches nothing. */ -const carriesPermissions = (sets: AuthorityAssociation[][], params: unknown[]): string => { - const rows = sets.flatMap((set, i) => - set.map((p) => [i, set.length, p.kind, p.label, ...permissionKeyColumns(p)]), +const carriesPermissions = (s: Statement, sets: AuthorityAssociation[][]): string => { + const elements = rows( + s, + ['set_index', 'set_size', 'kind', 'label', 'grantee_kind', 'grantee_id', 'role'], + sets.flatMap((set, i) => + set.map((p) => [i, set.length, p.kind, p.label, ...permissionKeyColumns(p)]), + ), ); - params.push(JSON.stringify(rows)); - const at = (i: number) => `json_extract(e.value, '$[${i}]')`; return ( - `r.id IN (SELECT p.record_id FROM json_each(?) e JOIN permissions p ` + - `ON p.kind = ${at(2)} AND p.label = ${at(3)} AND p.grantee_kind = ${at(4)} ` + - `AND p.grantee_id = ${at(5)} AND p.role = ${at(6)} ` + - `GROUP BY p.record_id, ${at(0)} HAVING COUNT(*) = MAX(${at(1)}))` + `r.id IN (SELECT p.record_id FROM ${elements} e JOIN permissions p ` + + 'ON p.kind = e.kind AND p.label = e.label AND p.grantee_kind = e.grantee_kind ' + + 'AND p.grantee_id = e.grantee_id AND p.role = e.role ' + + 'GROUP BY p.record_id, e.set_index HAVING COUNT(*) = MAX(e.set_size))' ); }; @@ -239,20 +253,18 @@ const carriesPermissions = (sets: AuthorityAssociation[][], params: unknown[]): * The reach as conditions on `r`: inside the fence, and inside `within` by * one of its arms. See docs/spec/access-control.md § Reach. */ -const reachConditions = (reach: Reach, params: unknown[]): string[] => { - const conditions: string[] = []; - if (reach.fence) conditions.push(inList('r.type_id', reach.fence, params)); +const reachConditions = (s: Statement, reach: Reach): void => { + if (reach.fence) s.conditions.push(inList(s, 'r.type_id', reach.fence)); const within = reach.within; if (within) { - const arms = [inList('r.type_id', within.types, params)]; + const arms = [inList(s, 'r.type_id', within.types)]; if (within.authored) { - params.push(within.authored.subjectId); - arms.push(`(r.entity_id = ? AND ${inList('r.type_id', within.authored.types, params)})`); + s.params.push(within.authored.subjectId); + arms.push(`(r.entity_id = ? AND ${inList(s, 'r.type_id', within.authored.types)})`); } - arms.push(carriesPermissions(within.permissions, params)); - conditions.push(`(${arms.join(' OR ')})`); + arms.push(carriesPermissions(s, within.permissions)); + s.conditions.push(`(${arms.join(' OR ')})`); } - return conditions; }; /** @@ -260,15 +272,10 @@ const reachConditions = (reach: Reach, params: unknown[]): string[] => { * except where in the ordering to resume, which is per-partition and gets * appended by the caller. */ -const recordConditions = ( - query: StackQuery, - reach: Reach | undefined, -): { conditions: string[]; params: unknown[] } => { - const conditions: string[] = ["r.id != '_config'"]; - const params: unknown[] = []; +const recordConditions = (query: StackQuery, reach: Reach | undefined): Statement => { + const s = emptyStatement(); + const { conditions, params } = s; const f = query.filter ?? {}; - let aliasSeq = 0; - const nextContentAlias = () => `cp${aliasSeq++}`; /** * A filter that accepts one value or a list of them, as the one `IN` that @@ -276,9 +283,7 @@ const recordConditions = ( */ const inFilter = (column: string, value: string | readonly string[] | undefined): void => { if (value === undefined) return; - conditions.push( - inList(`r.${column}`, Array.isArray(value) ? value : [value as string], params), - ); + conditions.push(inList(s, `r.${column}`, Array.isArray(value) ? value : [value as string])); }; /** One side of a date range, as the open-ended comparison it is. */ @@ -304,7 +309,7 @@ const recordConditions = ( conditions.push('r.unlisted_at IS NULL'); } - if (reach) conditions.push(...reachConditions(reach, params)); + if (reach) reachConditions(s, reach); inFilter('type_id', f.typeId); @@ -345,7 +350,7 @@ const recordConditions = ( // counted against the distinct tags asked for, the list bound as one. if (f.tags?.length) { const tags = [...new Set(f.tags)]; - const label = inList('a.label', tags, params); + const label = inList(s, 'a.label', tags); conditions.push( `r.id IN (SELECT a.record_id FROM associations a WHERE a.kind = 'tag' AND ${label} ` + `GROUP BY a.record_id HAVING COUNT(DISTINCT a.label) = ?)`, @@ -407,18 +412,25 @@ const recordConditions = ( conditions.push(hasAssociation('relationship', clauses)); } - if (f.content || f.contentPresent?.length) { - conditions.push(...contentConditions(f, params, nextContentAlias)); + // An arm is the content filter scoped to one Type; types partition + // records, so a record meets at most the arm for its own type + // (docs/spec/data-model.md § Content by type). + const content = contentRows(null, f); + if (f.contentByType) { + const arms = Object.entries(f.contentByType); + // The Types themselves bound the result too: an empty arm adds no + // condition row, yet still admits only its own Type. + const typeIds = arms.map(([typeId]) => typeId); + conditions.push(inList(s, 'r.type_id', typeIds)); + for (const [typeId, arm] of arms) content.push(...contentRows(typeId, arm)); } - // One arm per Type, each pinned to its type_id; types partition records, - // so no record meets two arms (docs/spec/data-model.md § Content by type). - if (f.contentByType) { - const arms = Object.entries(f.contentByType).map(([typeId, arm]) => { - params.push(typeId); - return ['r.type_id = ?', ...contentConditions(arm, params, nextContentAlias)].join(' AND '); - }); - conditions.push(`(${arms.map((arm) => `(${arm})`).join(' OR ')})`); + if (content.length > 0) { + const table = rows(s, CONDITION_COLUMNS, content); + conditions.push( + `NOT EXISTS (SELECT 1 FROM ${table} c WHERE (c.scope IS NULL OR c.scope = r.type_id) ` + + `AND NOT ${holds('c', 'r')})`, + ); } if (f.search) { @@ -434,7 +446,7 @@ const recordConditions = ( } } - return { conditions, params }; + return s; }; /** @@ -466,16 +478,12 @@ const presentCursorCondition = ( ); }; -const statement = ( - select: string, - from: string, - conditions: string[], - params: unknown[], - tail = '', -): QueryStatement => ({ - sql: `SELECT ${select} FROM ${from} WHERE ${conditions.join(' AND ')}${tail}`, - params, -}); +/** `s` as one statement reading `r` from `from`, its tables leading. */ +const statement = (s: Statement, from: string, tail: string): QueryStatement => { + const select = `SELECT r.* FROM ${from} WHERE ${s.conditions.join(' AND ')}${tail}`; + const sql = s.tables.length === 0 ? select : `WITH ${s.tables.join(', ')} ${select}`; + return { sql, params: [...s.tableParams, ...s.params] }; +}; /** * Compile a query into the statements that answer it, read in order and @@ -493,23 +501,15 @@ export const buildQueryPlan = (query: StackQuery, reach?: Reach): QueryStatement const contentField = query.sort?.contentField; if (contentField === undefined) { - const { conditions, params } = recordConditions(query, reach); + const where = recordConditions(query, reach); const field = getSortField(query); const col = getSortColumn(field); if (decoded) { const cursor = nativeCursor(decoded, field); - conditions.push(`(r.${col} ${op} ? OR (r.${col} = ? AND r.id ${op} ?))`); - params.push(cursor.value, cursor.value, cursor.id); + where.conditions.push(`(r.${col} ${op} ? OR (r.${col} = ? AND r.id ${op} ?))`); + where.params.push(cursor.value, cursor.value, cursor.id); } - return [ - statement( - 'r.*', - 'records r', - conditions, - params, - ` ORDER BY r.${col} ${dir}, r.id ${dir} LIMIT ?`, - ), - ]; + return [statement(where, 'records r', ` ORDER BY r.${col} ${dir}, r.id ${dir} LIMIT ?`)]; } const cursor = decoded ? contentCursor(decoded, contentField) : null; @@ -518,34 +518,34 @@ export const buildQueryPlan = (query: StackQuery, reach?: Reach): QueryStatement // Records holding a value at the field, in that value's order. A cursor // sitting in the absent partition has already passed every one of them. if (cursor?.kind !== 'absent') { - const { conditions, params } = recordConditions(query, reach); + const where = recordConditions(query, reach); const from = `content_index ${SORT_ALIAS} JOIN records r ON r.id = ${SORT_ALIAS}.record_id`; - conditions.push(`${SORT_ALIAS}.field = ?`); - params.push(contentField); - if (cursor) conditions.push(presentCursorCondition(cursor, op, params)); + where.conditions.push(`${SORT_ALIAS}.field = ?`); + where.params.push(contentField); + if (cursor) where.conditions.push(presentCursorCondition(cursor, op, where.params)); // Ordered exactly as idx_content_index_sort stores it, so the page is // an ordered walk of the index rather than a sort of the matches. const order = `${SORT_ALIAS}.value_rank ${dir}, ${SORT_ALIAS}.num_value ${dir}, ` + `${SORT_ALIAS}.text_key ${dir}, ${SORT_ALIAS}.text_value ${dir}, ` + `${SORT_ALIAS}.record_id ${dir}`; - pages.push(statement('r.*', from, conditions, params, ` ORDER BY ${order} LIMIT ?`)); + pages.push(statement(where, from, ` ORDER BY ${order} LIMIT ?`)); } // Records holding nothing there, which trail the others whichever way // the sort runs (docs/spec/data-model.md § Sorting by a content field). { - const { conditions, params } = recordConditions(query, reach); - conditions.push( + const where = recordConditions(query, reach); + where.conditions.push( `NOT EXISTS (SELECT 1 FROM content_index ${SORT_ALIAS} ` + `WHERE ${SORT_ALIAS}.record_id = r.id AND ${SORT_ALIAS}.field = ?)`, ); - params.push(contentField); + where.params.push(contentField); if (cursor?.kind === 'absent') { - conditions.push(`r.id ${op} ?`); - params.push(cursor.id); + where.conditions.push(`r.id ${op} ?`); + where.params.push(cursor.id); } - pages.push(statement('r.*', 'records r', conditions, params, ` ORDER BY r.id ${dir} LIMIT ?`)); + pages.push(statement(where, 'records r', ` ORDER BY r.id ${dir} LIMIT ?`)); } return pages; @@ -554,8 +554,8 @@ export const buildQueryPlan = (query: StackQuery, reach?: Reach): QueryStatement /** * A sort naming its value per Type, read through a derived table in place * of the index: each Type's own index rows, and its constant in the index's - * columns, so one cursor condition and one ORDER BY serve both. See - * docs/spec/data-model.md § Sorting by a content field. + * columns, so one cursor condition and one ORDER BY serve both. Fields and + * constants are each rows(). See docs/spec/data-model.md § Sorting by a content field. */ const byTypePlan = ( query: StackQuery, @@ -577,32 +577,36 @@ const byTypePlan = ( if (entry) constants.push([typeId, entry]); } const pages: QueryStatement[] = []; - const fieldOf = (alias: string, record: string) => - fields.map(() => `(${record}.type_id = ? AND ${alias}.field = ?)`).join(' OR '); if (cursor?.kind !== 'absent' && fields.length + constants.length > 0) { + const where = recordConditions(query, reach); const sources: string[] = []; - const sourceParams: unknown[] = []; if (fields.length > 0) { + const byField = rows(where, ['type_id', 'field'], fields); sources.push( - 'SELECT c.record_id AS record_id, c.value_rank AS value_rank, c.num_value AS num_value, ' + - 'c.text_key AS text_key, c.text_value AS text_value ' + - `FROM content_index c JOIN records t ON t.id = c.record_id WHERE ${fieldOf('c', 't')}`, + 'SELECT ci.record_id AS record_id, ci.value_rank AS value_rank, ' + + 'ci.num_value AS num_value, ci.text_key AS text_key, ci.text_value AS text_value ' + + `FROM ${byField} f JOIN content_index ci ON ci.field = f.field ` + + 'JOIN records t ON t.id = ci.record_id AND t.type_id = f.type_id', ); - for (const [typeId, field] of fields) sourceParams.push(typeId, field); } - for (const [typeId, entry] of constants) { - sources.push( - 'SELECT t.id AS record_id, ? AS value_rank, ? AS num_value, ? AS text_key, ' + - '? AS text_value FROM records t WHERE t.type_id = ?', + if (constants.length > 0) { + const byConstant = rows( + where, + ['type_id', 'value_rank', 'num_value', 'text_key', 'text_value'], + constants.map(([typeId, entry]) => + entry.kind === 'num' + ? [typeId, 0, entry.num, null, null] + : [typeId, 1, null, entry.key, entry.text], + ), ); - sourceParams.push( - ...(entry.kind === 'num' ? [0, entry.num, null, null] : [1, null, entry.key, entry.text]), - typeId, + sources.push( + 'SELECT t.id AS record_id, k.value_rank AS value_rank, k.num_value AS num_value, ' + + 'k.text_key AS text_key, k.text_value AS text_value ' + + `FROM ${byConstant} k JOIN records t ON t.type_id = k.type_id`, ); } - const { conditions, params } = recordConditions(query, reach); - if (cursor) conditions.push(presentCursorCondition(cursor, op, params)); + if (cursor) where.conditions.push(presentCursorCondition(cursor, op, where.params)); const from = `(${sources.join(' UNION ALL ')}) ${SORT_ALIAS} ` + `JOIN records r ON r.id = ${SORT_ALIAS}.record_id`; @@ -610,37 +614,30 @@ const byTypePlan = ( `${SORT_ALIAS}.value_rank ${dir}, ${SORT_ALIAS}.num_value ${dir}, ` + `${SORT_ALIAS}.text_key ${dir}, ${SORT_ALIAS}.text_value ${dir}, ` + `${SORT_ALIAS}.record_id ${dir}`; - pages.push( - statement( - 'r.*', - from, - conditions, - [...sourceParams, ...params], - ` ORDER BY ${order} LIMIT ?`, - ), - ); + pages.push(statement(where, from, ` ORDER BY ${order} LIMIT ?`)); } // Records holding nothing to order by: a Type's field with no value, a // `null` constant, or a Type the map does not name. { - const { conditions, params } = recordConditions(query, reach); + const where = recordConditions(query, reach); if (fields.length > 0) { - conditions.push( - `NOT EXISTS (SELECT 1 FROM content_index ${SORT_ALIAS} ` + - `WHERE ${SORT_ALIAS}.record_id = r.id AND (${fieldOf(SORT_ALIAS, 'r')}))`, + const byField = rows(where, ['type_id', 'field'], fields); + where.conditions.push( + `NOT EXISTS (SELECT 1 FROM ${byField} f JOIN content_index ${SORT_ALIAS} ` + + `ON ${SORT_ALIAS}.record_id = r.id AND ${SORT_ALIAS}.field = f.field ` + + 'WHERE f.type_id = r.type_id)', ); - for (const [typeId, field] of fields) params.push(typeId, field); } if (constants.length > 0) { const typeIds = constants.map(([typeId]) => typeId); - conditions.push(`NOT (${inList('r.type_id', typeIds, params)})`); + where.conditions.push(`NOT (${inList(where, 'r.type_id', typeIds)})`); } if (cursor?.kind === 'absent') { - conditions.push(`r.id ${op} ?`); - params.push(cursor.id); + where.conditions.push(`r.id ${op} ?`); + where.params.push(cursor.id); } - pages.push(statement('r.*', 'records r', conditions, params, ` ORDER BY r.id ${dir} LIMIT ?`)); + pages.push(statement(where, 'records r', ` ORDER BY r.id ${dir} LIMIT ?`)); } return pages; }; diff --git a/packages/sqlite-shared/src/record-adapter.ts b/packages/sqlite-shared/src/record-adapter.ts index ba00f6fa..7847d275 100644 --- a/packages/sqlite-shared/src/record-adapter.ts +++ b/packages/sqlite-shared/src/record-adapter.ts @@ -53,15 +53,13 @@ export const SQLITE_RECORD_CAPABILITIES: StackCapabilities = { filter: { content: 'path', contentPresent: true, - // Inside SQLite's limits on expression depth and compound selects; see - // docs/spec/adapters.md § Adapter capabilities. - contentByType: 64, + contentByType: true, search: true, }, sort: { fields: ['createdAt', 'updatedAt', 'version'], contentField: true, - contentFieldByType: 64, + contentFieldByType: true, }, limits: { attachmentBytes: null, diff --git a/packages/sqlite-shared/src/record-logic.ts b/packages/sqlite-shared/src/record-logic.ts index 7c52cbbe..02d97ab7 100644 --- a/packages/sqlite-shared/src/record-logic.ts +++ b/packages/sqlite-shared/src/record-logic.ts @@ -48,7 +48,7 @@ import type { } from '@haverstack/core/adapter'; import type { SqlExecutor } from './executor.js'; import { isForeignKeyViolation, isUniqueConstraintViolation } from './executor.js'; -import { buildQueryPlan, atBudget } from './query.js'; +import { buildQueryPlan, atBudget, rowsTable } from './query.js'; import { ContentIndex, SearchIndex } from './indexes.js'; import { rowToRecord, @@ -306,7 +306,7 @@ export class SharedSqlRecordLogic { * reads however many records it holds. */ private hydrate(rows: Record[]): StackRecord[] { - const ids = JSON.stringify(rows.map((row) => row.id)); + const ids = rows.map((row) => row.id as string); const associations = this.elementsByRecord('associations', ids, rowToAssociation); const permissions = this.elementsByRecord('permissions', ids, rowToPermission); return rows.map((row) => @@ -804,9 +804,8 @@ export class SharedSqlRecordLogic { this.exec.transaction(() => { this.checkExpectedVersion(id, this.liveVersion(id), opts.ifVersion); if (opts.expectSet) { - const ids = JSON.stringify([id]); const map = table === 'associations' ? rowToAssociation : rowToPermission; - const stored = this.elementsByRecord(table, ids, map).get(id) ?? []; + const stored = this.elementsByRecord(table, [id], map).get(id) ?? []; if (!sameAssociationSet(stored, opts.expectSet)) throw new ElementSetMovedError(id); } write(); @@ -953,19 +952,20 @@ export class SharedSqlRecordLogic { } /** - * One element table's rows for every record in `ids` (a JSON array), in - * primary-key order — the order a single record's read has always had. + * One element table's rows for every record in `ids`, in primary-key + * order — the order a single record's read has always had. */ private elementsByRecord( table: 'associations' | 'permissions', - ids: string, + ids: readonly string[], map: (row: Record) => T, ): Map { const byRecord = new Map(); const rows = this.exec.all>( - `SELECT * FROM ${table} WHERE record_id IN (SELECT value FROM json_each(?)) + `WITH ids AS MATERIALIZED ${rowsTable(['id'])} + SELECT * FROM ${table} WHERE record_id IN (SELECT id FROM ids) ORDER BY ${table === 'associations' ? ASSOCIATION_KEY : PERMISSION_KEY}`, - [ids], + [JSON.stringify(ids.map((id) => [id]))], ); for (const row of rows) { const id = row.record_id as string; diff --git a/packages/wire-types/src/discovery.ts b/packages/wire-types/src/discovery.ts index d1fc4822..4148de02 100644 --- a/packages/wire-types/src/discovery.ts +++ b/packages/wire-types/src/discovery.ts @@ -39,8 +39,13 @@ export type DiscoveryResponse = { * checks rather than casts. See docs/spec/wire-format.md § Discovery. */ export type DiscoveryCapabilities = { - filter?: { content?: string; contentPresent?: boolean; contentByType?: number; search?: boolean }; - sort?: { fields?: string[]; contentField?: boolean; contentFieldByType?: number }; + filter?: { + content?: string; + contentPresent?: boolean; + contentByType?: boolean; + search?: boolean; + }; + sort?: { fields?: string[]; contentField?: boolean; contentFieldByType?: boolean }; limits?: { attachmentBytes?: number | null; contentBytes?: number | null }; }; @@ -49,10 +54,10 @@ const SORT_FIELDS: ReadonlySet = new Set(NATIVE_SORT_FIELDS); /** * Resolves anything absent, malformed or unrecognized to the least capable - * value it could stand for — one rule, applied once, rather than a default - * per key at each call site. A `null` limit is the one permissive reading, - * since the server's own ceiling still answers 413. See - * docs/spec/adapters.md § Adapter capabilities. + * value it could stand for, and what needs the POST /records/query body to + * false below `'none'` — applied once, where capabilities enter. A `null` + * limit is the one permissive reading. See docs/spec/adapters.md § Adapter + * capabilities. */ export function normalizeCapabilities( capabilities: DiscoveryCapabilities | undefined, @@ -63,20 +68,20 @@ export function normalizeCapabilities( const reach = CONTENT_FILTER_REACHES.has(filter?.content as string) ? (filter?.content as ContentFilterReach) : 'none'; - // The per-Type shapes travel in the POST /records/query body, which a - // server reaching no content does not serve. - const cap = (value: unknown): number => (reach === 'none' ? 0 : capOrZero(value)); + // These travel in the POST /records/query body, which a server reaching + // no content does not serve. + const reaches = reach !== 'none'; return { filter: { content: reach, - contentPresent: filter?.contentPresent === true, - contentByType: cap(filter?.contentByType), + contentPresent: reaches && filter?.contentPresent === true, + contentByType: reaches && filter?.contentByType === true, search: filter?.search === true, }, sort: { fields: (sort?.fields ?? []).filter((f): f is NativeSortField => SORT_FIELDS.has(f)), contentField: sort?.contentField === true, - contentFieldByType: cap(sort?.contentFieldByType), + contentFieldByType: reaches && sort?.contentFieldByType === true, }, limits: { attachmentBytes: finiteOrNull(limits?.attachmentBytes), @@ -85,10 +90,6 @@ export function normalizeCapabilities( }; } -/** A cap on keys, or 0 — declined — for anything but a non-negative integer. */ -const capOrZero = (value: unknown): number => - typeof value === 'number' && Number.isSafeInteger(value) && value > 0 ? value : 0; - /** A ceiling this client can compare against, or null for "cannot pre-check". */ const finiteOrNull = (value: unknown): number | null => typeof value === 'number' && Number.isFinite(value) ? value : null; diff --git a/packages/wire-types/tests/changes.test.ts b/packages/wire-types/tests/changes.test.ts index 7c1e5c0c..83c59c78 100644 --- a/packages/wire-types/tests/changes.test.ts +++ b/packages/wire-types/tests/changes.test.ts @@ -42,13 +42,13 @@ const discovery = (overrides: Partial = {}): DiscoveryRespons filter: { content: 'path', contentPresent: true, - contentByType: 64, + contentByType: true, search: true, }, sort: { fields: ['createdAt'], contentField: true, - contentFieldByType: 64, + contentFieldByType: true, }, limits: { attachmentBytes: null, diff --git a/packages/wire-types/tests/discovery.test.ts b/packages/wire-types/tests/discovery.test.ts index 54583d2d..1c20fa31 100644 --- a/packages/wire-types/tests/discovery.test.ts +++ b/packages/wire-types/tests/discovery.test.ts @@ -53,8 +53,8 @@ describe('isProtocolCompatible', () => { // ------------------------------------------------------- const NONE: StackCapabilities = { - filter: { content: 'none', contentPresent: false, contentByType: 0, search: false }, - sort: { fields: [], contentField: false, contentFieldByType: 0 }, + filter: { content: 'none', contentPresent: false, contentByType: false, search: false }, + sort: { fields: [], contentField: false, contentFieldByType: false }, limits: { attachmentBytes: null, contentBytes: null }, }; @@ -62,13 +62,13 @@ describe('normalizeCapabilities', () => { it('reads what a server declares', () => { expect( normalizeCapabilities({ - filter: { content: 'path', contentPresent: true, contentByType: 64, search: true }, - sort: { fields: ['createdAt', 'version'], contentField: true, contentFieldByType: 64 }, + filter: { content: 'path', contentPresent: true, contentByType: true, search: true }, + sort: { fields: ['createdAt', 'version'], contentField: true, contentFieldByType: true }, limits: { attachmentBytes: 52428800, contentBytes: 1048576 }, }), ).toEqual({ - filter: { content: 'path', contentPresent: true, contentByType: 64, search: true }, - sort: { fields: ['createdAt', 'version'], contentField: true, contentFieldByType: 64 }, + filter: { content: 'path', contentPresent: true, contentByType: true, search: true }, + sort: { fields: ['createdAt', 'version'], contentField: true, contentFieldByType: true }, limits: { attachmentBytes: 52428800, contentBytes: 1048576 }, }); }); @@ -82,24 +82,28 @@ describe('normalizeCapabilities', () => { ).toEqual([...NATIVE_SORT_FIELDS]); }); - it('reads a per-Type cap that is not a positive integer as 0', () => { - for (const cap of [-1, 2.5, '16', true, null, Number.POSITIVE_INFINITY]) { + it('reads a per-Type flag that is not true as false', () => { + for (const flag of [1, 'true', null, undefined]) { const read = normalizeCapabilities({ - filter: { content: 'path', contentByType: cap as number }, - sort: { contentFieldByType: cap as number }, + filter: { content: 'path', contentByType: flag as unknown as boolean }, + sort: { contentFieldByType: flag as unknown as boolean }, }); - expect([read.filter.contentByType, read.sort.contentFieldByType]).toEqual([0, 0]); + expect([read.filter.contentByType, read.sort.contentFieldByType]).toEqual([false, false]); } }); - // Both travel in the POST /records/query body, which a server reaching no - // content does not serve, so a cap it declares is one it cannot honor. - it('reads a per-Type cap as 0 beside a reach of none', () => { + // These travel in the POST /records/query body, which a server reaching + // no content does not serve, so what it declares for them is unhonorable. + it('reads what needs the query body as false beside a reach of none', () => { const read = normalizeCapabilities({ - filter: { content: 'none', contentByType: 16 }, - sort: { contentFieldByType: 16 }, + filter: { content: 'none', contentPresent: true, contentByType: true }, + sort: { contentFieldByType: true }, }); - expect([read.filter.contentByType, read.sort.contentFieldByType]).toEqual([0, 0]); + expect([ + read.filter.contentPresent, + read.filter.contentByType, + read.sort.contentFieldByType, + ]).toEqual([false, false, false]); }); it('reads an absent capabilities object as declaring nothing', () => { @@ -110,7 +114,7 @@ describe('normalizeCapabilities', () => { it('reads an absent group as declaring nothing, not as inheriting its siblings', () => { expect(normalizeCapabilities({ filter: { content: 'path' } })).toEqual({ ...NONE, - filter: { content: 'path', contentPresent: false, contentByType: 0, search: false }, + filter: { content: 'path', contentPresent: false, contentByType: false, search: false }, }); });