From df4292a84e0ebddd7d43734c6884f49cf8015a24 Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 15:06:23 -0600 Subject: [PATCH 01/46] test(db): transfer two index regressions into oracle --- docs/contributing/oracle-coverage.md | 2 +- .../index-unit-retirement-round-1.md | 94 ++++++++ .../btree-index-undefined-values.test.ts | 18 -- .../index-update-oracle.property.test.ts | 214 +++++++++++++++++- .../tests/index-update-short-circuit.test.ts | 13 -- 5 files changed, 303 insertions(+), 38 deletions(-) create mode 100644 docs/contributing/oracle-reviews/index-unit-retirement-round-1.md diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 2c53ec87d6..26003b0603 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -442,7 +442,7 @@ generic visitor behavior outside the named collectors. | Backend Collection E2E | [shared provider suites](https://github.com/TanStack/db/tree/main/packages/db-collection-e2e/src/suites); [Electric registration](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/e2e/electric.e2e.test.ts); [Query registration](https://github.com/TanStack/db/blob/main/packages/query-db-collection/e2e/query.e2e.test.ts); [TrailBase registration](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/e2e/trailbase.e2e.test.ts) | Electric, Query, and TrailBase register reusable predicate, pagination, join, and live-update checks at their provider boundaries; Electric also registers the move suite. These are receiving paths for core query and Collection laws, not framework render cuts. | | Window-controller overlap | [shared infinite-query conformance](https://github.com/TanStack/db/blob/main/packages/db/tests/conformance/infinite-suite-oracle.ts), [React driver](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-conformance.test.tsx), [Vue driver](https://github.com/TanStack/db/blob/main/packages/vue-db/tests/infinite-query-conformance.test.ts), [Svelte driver](https://github.com/TanStack/db/blob/main/packages/svelte-db/tests/infinite-query-conformance.svelte.test.ts), [review record](oracle-reviews/dec-03-window-overlap.md) | Four or five ordered source rows, two window-settlement orders, and public snapshots after each settlement, a subscribed observer update, a detached controller read, and explicit recovery. Insertion and removal of the fifth row distinguish both continuation directions while an earlier error remains visible. The detached read follows a source change with no controller subscriber; its preloaded Collection retains the five-row window. The exact overlap uses the exported DB controller in each package realm. Framework hooks expose no controller preload, so this cell does not establish a hook scheduling path; direct hook paging remains in the adjacent shared scenarios. An unsubscribed controller has no notification claim. | | Infinite-query publication | [shared publication laws](https://github.com/TanStack/db/blob/main/packages/db/tests/conformance/infinite-suite-oracle.ts), [React render cuts](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-render-cuts-oracle.test.tsx), [Svelte hook tests](https://github.com/TanStack/db/blob/main/packages/svelte-db/tests/useLiveInfiniteQuery.svelte.test.ts), [review record](oracle-reviews/2026-10-05-infinite-first-paint.md) | Each driver records every value its framework published: React layout commits, Vue pre-flushes, and the Svelte construction value plus pre-effect runs. The shared suite checks that a synchronous source shows its ready first page on the first value, for a query callback and for a supplied collection that has not started, that every ready value in a fixed-source, fixed-query scenario equals the source prefix for its own page count, including an equal dependency replacement that keeps page depth, and that a mount requests an on-demand first window once, for a query callback and for a supplied collection whose finite window is wider than or equal to the one the hook needs, judged by the rows the on-demand source holds after mount. Under deferred acquisition, an on-demand first value is either the ready first page or a value with no rows, by framework: React and Svelte publish before the hook subscribes, Vue after. `preloaded-collection-first-paint` checks the documented route-loader pattern: a limited live query preloaded before mount shows its ready first page on the first value in every framework and asks the source for nothing more; a preload that does not count as demand never settles there. Pre-commit work has no shared form, because Vue setup is its commit. The React driver owns renders that never commit, StrictMode double renders (one first-window request on React 19), retries after GC, a mounted replacement whose startup error must reach the error boundary, and a collection kept from an abandoned render that enters terminal error before its query is selected again, each as under `useLiveQuery`. The React source ID reuse oracle drives the infinite hook too. Svelte owns construction and superseded recomputes before the subscribing effect. In both, the hook acquires an on-demand source as often as `useLiveQuery` does under the same history, for either input form, direct or behind a live-query Collection. On-demand requests from renders that never commit are now owned by the live-query deferred acquisition oracle; the React render-cut oracle checks that an abandoned render makes no acquisition. Open cells, owned here: React 18 StrictMode, where neither hook keeps refs across the double render; a React commit held after the 50 ms unsubscribed GC floor, where a controlled-premise controller witness shows no non-ready value but no React scheduling witness exists; ready values in scenarios that write the source or replace the query, which need a per-step source model; a join of eager and on-demand sources, which needs a zero-acquisition check before commit; a supplied collection with no `.limit`, which sends one unbounded request on `main` too, because the controller cannot narrow it before the source request; and Svelte `state_unsafe_mutation` when a start inside `$derived.by` synchronously writes rows a peer is waiting for, shared with Svelte's `useLiveQuery`, which a two-component probe reproduces and the deferred-acquisition follow-up removes. | -| Structural values and ordered primitives | [hash values](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-oracle.property.test.ts), [hash identity](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-identity-oracle.property.test.ts), [MultiSet consolidation](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/multiset-consolidate-oracle.property.test.ts), [hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-graph-oracle.property.test.ts), [mixed hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-mixed-graph-oracle.property.test.ts), [hash retry](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-failure-retry-oracle.property.test.ts), [comparison](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts), [deep equality](https://github.com/TanStack/db/blob/main/packages/db/tests/utils-oracle.property.test.ts), [cursor](https://github.com/TanStack/db/blob/main/packages/db/tests/cursor-oracle.property.test.ts), [indexes](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update-oracle.property.test.ts), [BTree Map](https://github.com/TanStack/db/blob/main/packages/db/tests/btree-map-oracle.test.ts), [query identity](https://github.com/TanStack/db/blob/main/packages/db/tests/query/identity-output-shape-oracle.test.ts), [LIKE semantics](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/evaluators-oracle.test.ts) | Independent flat values, graph topology, algebraic laws, Map/group/sort recomputation, expression denotation, custom comparator dispatch/reference identity/index compatibility, LIKE wildcard refinement, compiled output bags, declared value-pair agreement between D2 equality keys and IR equality operands, and Collection collation with exact public scan/index order and one-index reuse across repeated reads. The auto-index cells cross BasicIndex and BTreeIndex; scan cells omit indexes. Both paths cover lexical and numeric `en-US` locale defaults plus an explicit ordinary locale override of a numeric default. An unindexed six-row work cell bounds Collection collation resolution to at most two reads per ordered scan: one index-path check and one scan clause. Other locales and ordered histories are outside this owner. The identity pair grammar covers primitives, signed zero, NaN/invalid Date, valid Date/timestamp, binary/Buffer, Temporal, nested references, symbols, and functions; fixed and sampled pairs do not prove every JavaScript value or hash collision freedom. Ref proxies are expressions rather than literal values in this lane. The deep-equality owner also pins that `deepEquals` ignores Map and Set insertion order, RegExp `lastIndex`, array holes, and typed-array class, the rules draft equality keeps. It also pins that objects of another class differ, that plain and null-prototype objects are one class, that a class instance without enumerable keys equals only itself, that URLs compare by `href`, and that typed-array elements compare like numbers. The hash-values owner checks equal-hash consequences for independently constructed flat values. Distinct type identities are checked by the hash-identity owner through `equalHashValues`, including forced digest collisions; distinct 32-bit digests are not promised. The hash-identity owner checks `hash` and `equalHashValues` against one spec-level identity model: primitives with signed zero, NaN, and bigint; reference leaves (functions, registered handles, `File`, binary over 128 bytes); Date, binary, Temporal, and RegExp headers; array length and holes; Map and Set insertion order; and object keys in any order, for acyclic values and for cycles through arrays, Map values, Sets, and plain objects. Map/Set order sensitivity is pinned current behavior, not a promise. Arrays from another realm, RegExps with a `NaN` `lastIndex`, getters, proxies, other cycle carriers, and pairs that differ only in a back-edge target are outside its grammar; pinned cases keep equality's cross-realm length check and strict RegExp field comparison. `hash-work.test.ts` pins how visited values and array/RegExp headers count toward the structural work cap, and that equality copies a shared Map's entries once per distinct pair ([code-weight hash dispatch review](oracle-reviews/code-weight-hash-dispatch.md)). Custom string indexes do not optimize ordinary ranges, and custom cursors remain unsupported. The LIKE owner covers boolean string matching and bounded work, not nullish three-valued logic or a general Unicode collation contract. Unsupported composite cursors reject. The MultiSet consolidation owner checks keyed, single-type, and structural identity, signed zero and NaN in keyed and single-number data, the retained first record, zero-sum removal, and input record contents. It does not assert output order. Its keyed grammar includes cross-type key and value collisions, non-finite numeric keys, the `\|` delimiter, symbol and function identity, and a direct-object/object-tuple delimiter control from #1948. Eight named collision witnesses and both generated campaigns are RED on `c879ba6d` and GREEN with the repair ([#1948 review record](oracle-reviews/issue-1948-keyed-consolidation.md)); the [code-weight consolidation review](oracle-reviews/code-weight-multiset-consolidation.md) records the earlier exclusion. The keyed reference uses direct value/reference equality, including SameValueZero for numeric keys. The unkeyed fallback keeps its original key and value domains. The direct `MultiSet` owner does not prove which `@tanstack/db` query shapes produce primitive keyed values or mixed-type source keys; that needs a live-query production witness. The index owner enforces one invalid-comparator law across both BasicIndex and BTreeIndex: for each comparator, accepted numeric prefix (including the empty prefix), and probe operation (add, update, eq/range lookup, take, build), the operation throws exactly when it receives a `NaN` or non-number result and never when every comparison is valid. A `signed infinity` comparator is the valid control, and a custom collation `compare` supplied through `compareOptions` is checked the same way. Two BTree Map split histories check inserted-key reads, exact whole-range payloads/order, size, and extrema, proving split placement follows the insertion index without a post-mutation comparison. A rejected index add or remove leaves the index refining its accepted rows; update and build are not atomic. At the Collection boundary, both index types and both write paths (optimistic insert, sync commit) crash the collection: status becomes `error` and the next mutation throws, so a usable collection never holds rows its subscribers were not told about. Open: a sync source can still commit writes on a collection in `error` state, which stores unpublished rows; this is existing `markError` behavior and needs a lifecycle-owner witness. Comparator transitivity is outside this evidence. | +| Structural values and ordered primitives | [hash values](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-oracle.property.test.ts), [hash identity](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-identity-oracle.property.test.ts), [MultiSet consolidation](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/multiset-consolidate-oracle.property.test.ts), [hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-graph-oracle.property.test.ts), [mixed hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-mixed-graph-oracle.property.test.ts), [hash retry](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-failure-retry-oracle.property.test.ts), [comparison](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts), [deep equality](https://github.com/TanStack/db/blob/main/packages/db/tests/utils-oracle.property.test.ts), [cursor](https://github.com/TanStack/db/blob/main/packages/db/tests/cursor-oracle.property.test.ts), [indexes](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update-oracle.property.test.ts), [BTree Map](https://github.com/TanStack/db/blob/main/packages/db/tests/btree-map-oracle.test.ts), [query identity](https://github.com/TanStack/db/blob/main/packages/db/tests/query/identity-output-shape-oracle.test.ts), [LIKE semantics](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/evaluators-oracle.test.ts) | Independent flat values, graph topology, algebraic laws, Map/group/sort recomputation, expression denotation, custom comparator dispatch/reference identity/index compatibility, LIKE wildcard refinement, compiled output bags, declared value-pair agreement between D2 equality keys and IR equality operands, and Collection collation with exact public scan/index order and one-index reuse across repeated reads. The auto-index cells cross BasicIndex and BTreeIndex; scan cells omit indexes. Both paths cover lexical and numeric `en-US` locale defaults plus an explicit ordinary locale override of a numeric default. An unindexed six-row work cell bounds Collection collation resolution to at most two reads per ordered scan: one index-path check and one scan clause. Other locales and ordered histories are outside this owner. The identity pair grammar covers primitives, signed zero, NaN/invalid Date, valid Date/timestamp, binary/Buffer, Temporal, nested references, symbols, and functions; fixed and sampled pairs do not prove every JavaScript value or hash collision freedom. Ref proxies are expressions rather than literal values in this lane. The deep-equality owner also pins that `deepEquals` ignores Map and Set insertion order, RegExp `lastIndex`, array holes, and typed-array class, the rules draft equality keeps. It also pins that objects of another class differ, that plain and null-prototype objects are one class, that a class instance without enumerable keys equals only itself, that URLs compare by `href`, and that typed-array elements compare like numbers. The hash-values owner checks equal-hash consequences for independently constructed flat values. Distinct type identities are checked by the hash-identity owner through `equalHashValues`, including forced digest collisions; distinct 32-bit digests are not promised. The hash-identity owner checks `hash` and `equalHashValues` against one spec-level identity model: primitives with signed zero, NaN, and bigint; reference leaves (functions, registered handles, `File`, binary over 128 bytes); Date, binary, Temporal, and RegExp headers; array length and holes; Map and Set insertion order; and object keys in any order, for acyclic values and for cycles through arrays, Map values, Sets, and plain objects. Map/Set order sensitivity is pinned current behavior, not a promise. Arrays from another realm, RegExps with a `NaN` `lastIndex`, getters, proxies, other cycle carriers, and pairs that differ only in a back-edge target are outside its grammar; pinned cases keep equality's cross-realm length check and strict RegExp field comparison. `hash-work.test.ts` pins how visited values and array/RegExp headers count toward the structural work cap, and that equality copies a shared Map's entries once per distinct pair ([code-weight hash dispatch review](oracle-reviews/code-weight-hash-dispatch.md)). Custom string indexes do not optimize ordinary ranges, and custom cursors remain unsupported. The LIKE owner covers boolean string matching and bounded work, not nullish three-valued logic or a general Unicode collation contract. Unsupported composite cursors reject. The MultiSet consolidation owner checks keyed, single-type, and structural identity, signed zero and NaN in keyed and single-number data, the retained first record, zero-sum removal, and input record contents. It does not assert output order. Its keyed grammar includes cross-type key and value collisions, non-finite numeric keys, the `\|` delimiter, symbol and function identity, and a direct-object/object-tuple delimiter control from #1948. Eight named collision witnesses and both generated campaigns are RED on `c879ba6d` and GREEN with the repair ([#1948 review record](oracle-reviews/issue-1948-keyed-consolidation.md)); the [code-weight consolidation review](oracle-reviews/code-weight-multiset-consolidation.md) records the earlier exclusion. The keyed reference uses direct value/reference equality, including SameValueZero for numeric keys. The unkeyed fallback keeps its original key and value domains. The direct `MultiSet` owner does not prove which `@tanstack/db` query shapes produce primitive keyed values or mixed-type source keys; that needs a live-query production witness. The index owner checks lower-bound presence on bounded numeric histories after each add, update, delete, and build: an upper-only range ignores an unused `fromInclusive: false`, while an explicit exclusive `from` excludes its boundary. A source Map supplies expected sets for BasicIndex and BTreeIndex; wrong-output controls reject both boundary mistakes. It also checks exact `equalityLookup` and `lookup('eq')` after singleton case-insensitive `A → a → A` replacements and populated comparator-group updates, plus fresh-object same-group replacements in fixed and generated histories. These direct-index checks do not cover every JavaScript value, arbitrary comparators, or long Collection histories. The index owner enforces one invalid-comparator law across both BasicIndex and BTreeIndex: for each comparator, accepted numeric prefix (including the empty prefix), and probe operation (add, update, eq/range lookup, take, build), the operation throws exactly when it receives a `NaN` or non-number result and never when every comparison is valid. A `signed infinity` comparator is the valid control, and a custom collation `compare` supplied through `compareOptions` is checked the same way. Two BTree Map split histories check inserted-key reads, exact whole-range payloads/order, size, and extrema, proving split placement follows the insertion index without a post-mutation comparison. A rejected index add or remove leaves the index refining its accepted rows; update and build are not atomic. At the Collection boundary, both index types and both write paths (optimistic insert, sync commit) crash the collection: status becomes `error` and the next mutation throws, so a usable collection never holds rows its subscribers were not told about. Open: a sync source can still commit writes on a collection in `error` state, which stores unpublished rows; this is existing `markError` behavior and needs a lifecycle-owner witness. Comparator transitivity is outside this evidence. | | WHERE predicate publication | [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts), [`in` evaluator](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts) ([review](oracle-reviews/perf-in-constant-set.md)) | An independent Kleene evaluator judges `eq`/`not`/`and`/`or` predicates over strings, a normalization-prefixed string, booleans, numbers, `NaN`, a valid Date, `null`, a missing field, and virtual fields. Snapshot histories with pending optimistic inserts compare a live query, direct subscribers with and without initial state, and `currentStateAsChanges`; change histories apply multi-key sync transactions, including reinsertion of a key deleted earlier, and compare every consumer's key set after each commit, on scan and `BasicIndex` paths. Peer subscribers share one field with different literals, plus one on a virtual field, so every published change must reach exactly the subscribers whose predicate it can satisfy; routing mutants that ignored the previous value, ignored the field, delivered a change twice, or routed while stale published rows awaited reconciliation fail here. Fixed witnesses cover a layout-only publication reaching a filtered subscriber as one empty batch, a filtered subscriber's empty Collection-readiness batch, retraction of a vanished row after eager cleanup and restart, and the new row plus exact subscriber update payload when a same-key row stays TRUE. A stale-live-value mutant survives the generated membership checks but fails the payload witness. A focused [property-visibility test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-prefilter-property-visibility.test.ts) compares the public unindexed snapshot with the enriched-row contract for inherited, non-enumerable, enumerable-own, and nested getter paths; the pre-fix stored-row shortcut failed three of its four controls. Other property-descriptor and stateful-getter histories remain outside those fixed controls. Hostile mutants for FALSE-for-UNKNOWN `eq`, `or` or number-literal subscription prefilters, a prefilter that ignores the previous value, a skipped readiness batch, a skip while stale published rows await reconciliation, and an unindexed snapshot scan that reads only synced rows while an optimistic insert or delete changes visibility passed the prior `@tanstack/db` suite and fail here. Change routing withholds a dropped row from an `eq` subscription before its sent key could be recorded, so a mutant that records dropped rows as sent is equivalent for routed predicates in this oracle. A focused `collection-subscription.test.ts` witness checks that a dropped insert or update, alone or beside a matching change, cannot advance a limited subscription's page offset under a routed `eq` and an unrouted `or` predicate; the unrouted cases beside a matching change kill that mutant, and the unrouted `alone` case does not. A loose-equality prefilter is an equivalent mutant: it only skips less. Skipping during truncate replay also survives; the replay's baseline diff re-derives the same retraction, so the guard keeps the prior dataflow without its own witness. Generated cleanup and restart histories for filtered subscribers remain open for the lifecycle publication owner. No-op updates, other comparison operators, Temporal and binary operands, joins, ordering, optimistic updates, and truncate are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. Change routing was later removed in favor of pooled live queries; the routing mutants above are historical, and the same peer laws now check plain subscription dispatch. The unindexed snapshot prefilter was restored for compiled queries as one stored-row `eq` test on a string or boolean literal. Its property-visibility test also covers the reverse direction: a stored row with an inherited or non-enumerable field must still match `isUndefined` on that field, which a scan that evaluated the full predicate on stored rows fails. The `in` evaluator owner checks `in` against an independent linear reference over Dates, `NaN`, `-0`, BigInt, bytes, Temporal values, objects and nullish items, for constant and row-supplied lists. | | Joined result keys | [Joined result key oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/join-result-key-oracle.property.test.ts) | A nested-loop model recomputes the pair set of an inner, left, or full join over source keys drawn from plain, comma-bearing, bracket-bearing, and quoted strings, numbers beside the strings that print the same, both infinities, and `NaN`. After preload and each synced group change, the published rows must equal the model's pairs and the result key count must equal the row count. The comma-joined key encoding fails its two pinned histories and both campaigns; plain `JSON.stringify`, which prints infinities as the missing-side `null`, fails the pinned infinity history and both campaigns. A pinned history in which a `NaN`-keyed pair leaves and re-forms fails when the join index compares source-key prefixes with `===`. Joins over subqueries, more than two sources, custom `getKey`, and optimistic mutations are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. | | D2 Index storage | [Index refinement oracle](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/index-refinement-oracle.property.test.ts) | A plain-`Map` model sums multiplicities under a type-and-value identity in which `NaN` equals `NaN` and `-0` equals `0`. Histories of up to twelve additions to two keys mix prefixed arrays with `NaN`, `0`, `-0`, `1`, `'1'`, `1n`, and `'a'` prefixes and unprefixed values, with cancellation and reappearance. After each addition, `get` and `has` must equal the model. Comparing prefixes with `===`, which disagrees with the `Map` that holds them, fails three pinned histories and both campaigns; the `-0` control passes under both. Compaction, presence tracking, and structural payloads are outside this owner; the join operator tests and incrementalization law own operator behavior. | diff --git a/docs/contributing/oracle-reviews/index-unit-retirement-round-1.md b/docs/contributing/oracle-reviews/index-unit-retirement-round-1.md new file mode 100644 index 0000000000..f570ee4ed7 --- /dev/null +++ b/docs/contributing/oracle-reviews/index-unit-retirement-round-1.md @@ -0,0 +1,94 @@ +# Index unit-test retirement: two-block ledger + +Base: `f6857fee9de59dc82cbe596ca7df4a18152004ed` (`origin/main` at worktree creation). +This ledger names the exact legacy assertions proposed for retirement. The +receiving owner is `packages/db/tests/index-update-oracle.property.test.ts`. +The other blocks in each legacy file remain outside this change. + +## Upper-only range ignores an absent lower bound + +- **Legacy source:** `packages/db/tests/btree-index-undefined-values.test.ts`, + `should not drop the minimum key when an upper-only range is exclusive on + the (absent) lower bound` (original line 252). It inserts values 1, 5, 10, + calls `rangeQuery({ to: 10, fromInclusive: false })`, and requires all three + keys. Its observation is the returned direct BTreeIndex set. +- **Law and authority:** An absent bound denotes the start of the index + (`packages/db/src/query/live/ARCHITECTURE.md`, Identity); the documented + `IndexInterface.rangeQuery` accepts separate bounds and inclusivity flags. + An inclusivity flag cannot exclude a value at a bound the caller did not + supply. This law says nothing about internal tree traversal or runtime cost. +- **Independent model and history:** The receiving oracle's source `Map` keeps + key/value ownership. For every upper bound -2, 0, 2, it filters the Map with + `value <= upper`; it makes no lower comparison for an absent lower bound. + Paired reads with an explicit exclusive `from` require `value > lower`, + preserving the distinction stated in the old test's comment. + The existing fixed and random campaigns apply add, same-key update, delete, + and full build to both BasicIndex and BTreeIndex, then check the upper-only + and explicit-bound reads after each action, including empty and + changed-minimum states. +- **Distinction and checkpoint:** A design that applies + `fromInclusive: false` to the implicit minimum drops a modeled key. The + comparison checks the exact returned set after each direct index action. + One wrong-output control removes the minimum from the upper-only read; + another adds the excluded boundary to an explicit-bound read. Both must + fail their intended comparison. +- **Retirement gate:** The new owner and both wrong-output controls passed. + `/root/review_absent_bound` attacked the original and revised claims, + executed the model and controls, and found no blocking survivor in the + bounded numeric scope. The named legacy block was removed; its sibling + blocks remain. + +## Comparator-equal update moves exact membership + +- **Legacy source:** `packages/db/tests/index-update-short-circuit.test.ts`, + `does not use comparator equality to skip an update` (original line 91). + With a case-insensitive comparator, it changes one key from `A` to `a` and + checks that exact `A` lookup becomes empty and exact `a` lookup owns the key. +- **Law and authority:** An ordered index keeps separate exact equality + buckets for values at one comparator position + (`packages/db/src/query/live/ARCHITECTURE.md`, Identity; the existing + comparator-group index oracle). Comparator equality alone does not authorize + skipping a source value replacement. This does not promise bucket object + identity or prohibit harmless maintenance work. +- **Independent model and history:** A source `Map` retains exact strings; + fixed input facts put `A/a` in one order group and `B/b` in another. The + receiving oracle checks both `equalityLookup` and `lookup('eq')`, plus + comparator-group range sets, after each direct action on BasicIndex and + BTreeIndex. The fixed trace updates a singleton `A → a → A` before adding + peers, then checks populated same-group and cross-group updates. Its + existing fixed/random + custom-comparator property now also replaces an object with a fresh object + in the same group before crossing to another group. The group IDs and exact + object handles are input facts, not production classifications. +- **Distinction and checkpoint:** A wrong design that skips a comparator-equal + `update` leaves the old exact lookup populated at the returned-action cut. + A singleton no-op update control must fail the same source-Map comparison. + A second control makes `lookup('eq')` return the comparator group while leaving + `equalityLookup` intact; it must fail as well. The fixed string trace + preserves the original value-domain and lookup-dispatch path; the generated + object lane probes the general distinction after representative retirement. +- **Retirement gate:** The string and generated witnesses and both controls + passed. `/root/review_equal_update` found two gaps in the first draft: it + checked only `equalityLookup`, and it updated only a populated index. The + revised witness exercises `lookup('eq')` and the original singleton path; + the reviewer re-ran the relevant cases and found no remaining blocker. + The named legacy block was removed; its sibling blocks remain. + +## Execution and workflow result + +The post-retirement run of the receiving oracle and both remaining legacy +files passed: 397 tests, zero type errors. Before retirement, the same three +files passed with both old blocks present. The new numeric and string controls +deliberately inject wrong returned results or skipped work and each reaches an +assertion failure at the intended public observation. No production defect was +demonstrated in these two blocks, so production code was not changed. + +This round exposed two useful review gates for the broader cleanup. A +same-group update on populated data did not cover the original singleton +history, and checking `equalityLookup` did not cover `lookup('eq')` dispatch. +Both gaps survived the author's first green run and were found by an +independent reviewer. The generated object fixture also initially reused a +retired key; distinct reuse keys repaired that model-fixture error before the +final run. Future blocks should record the exact public method, smallest +state, old comment's law, and a wrong-design control before deletion. A +passing test count by itself is not a retirement gate. diff --git a/packages/db/tests/btree-index-undefined-values.test.ts b/packages/db/tests/btree-index-undefined-values.test.ts index 252737e0ab..e2160f06da 100644 --- a/packages/db/tests/btree-index-undefined-values.test.ts +++ b/packages/db/tests/btree-index-undefined-values.test.ts @@ -249,24 +249,6 @@ describe(`BTreeIndex - undefined value handling`, () => { expect(withoutFrom.size).toBe(3) }) - it(`should not drop the minimum key when an upper-only range is exclusive on the (absent) lower bound`, () => { - // When no `from` bound is provided, `fromInclusive` must not cause the - // smallest key to be excluded: there is no lower bound to exclude - // against. Only an explicitly provided exclusive lower bound should - // drop its boundary value. - const index = createIndex(`value`) - index.add(`a`, { value: 1 }) - index.add(`b`, { value: 5 }) - index.add(`c`, { value: 10 }) - - const result = index.rangeQuery({ to: 10, fromInclusive: false }) - - expect(result.size).toBe(3) - expect(result).toContain(`a`) - expect(result).toContain(`b`) - expect(result).toContain(`c`) - }) - it(`should handle range query from undefined to undefined`, () => { const index = createIndex(`value`) index.add(`a`, { value: undefined }) diff --git a/packages/db/tests/index-update-oracle.property.test.ts b/packages/db/tests/index-update-oracle.property.test.ts index 0cb172df91..d3cfcd4024 100644 --- a/packages/db/tests/index-update-oracle.property.test.ts +++ b/packages/db/tests/index-update-oracle.property.test.ts @@ -30,6 +30,13 @@ import type { CompareOptions } from '../src/query/builder/types.js' * direct index observations are the documented IndexInterface boundary. * The Map retains each key's value because a later same-key put or delete can * distinguish states with the same value multiplicities but different owners. + * An absent lower range bound includes the minimum even when its unused + * `fromInclusive` flag is false. With an explicit exclusive lower bound, + * the value at that bound is excluded. + * Exact equality membership also stays distinct from comparator position: + * replacing a value with one in a different exact-equality bucket at the + * same comparator position must move its exact lookup membership. These laws + * concern returned index results, not bucket identity or maintenance work. */ type IndexValue = number @@ -91,7 +98,10 @@ const arbitraryAction: fc.Arbitrary = fc.oneof( * updates, delete supplies retirement, and build supplies arbitrary replacement * sets. Removing any one action loses that transition. Removing the narrow * key/value overlap loses collisions; removing -0 loses signed-zero equality. - * Range probes straddle -2, 0, and 2, including their exact boundaries. + * Range probes straddle -2, 0, and 2, including their exact boundaries and + * upper-only reads with an unused exclusive-lower flag and paired reads with + * an explicit exclusive lower bound. Both run after every action, including + * changes to the minimum and an empty build. * Duplicate keys passed directly to build are excluded: the source Map first * resolves them, since duplicate raw build entries have no contract here. */ @@ -146,6 +156,22 @@ function expectIndexMatchesModel( expect(index.rangeQuery({ from: boundary })).toEqual(keysAtOrAbove) expect(index.rangeQuery({ to: boundary })).toEqual(keysAtOrBelow) + // The model has no lower predicate here. An inclusivity flag cannot + // exclude the minimum when the caller supplied no lower bound. + expect(index.rangeQuery({ to: boundary, fromInclusive: false })).toEqual( + keysAtOrBelow, + ) + // Supplying the lower bound changes the model predicate. Keys at that + // bound are excluded, while later keys through the upper bound remain. + expect( + index.rangeQuery({ from: boundary, to: 2, fromInclusive: false }), + ).toEqual( + new Set( + [...rows] + .filter(([, value]) => value > boundary && value <= 2) + .map(([key]) => key), + ), + ) expect(index.rangeQueryReversed({ from: boundary })).toEqual(keysAtOrBelow) expect(index.rangeQueryReversed({ to: boundary })).toEqual(keysAtOrAbove) } @@ -327,6 +353,61 @@ describe.each(indexTypes)(`%s update properties`, (_indexName, IndexType) => { } }) + test(`upper-only range rejects a dropped minimum when the lower flag is exclusive`, () => { + const index = new IndexType(1, new PropRef([`value`])) + index.add(`minimum`, { value: -2 }) + const rangeQuery = index.rangeQuery.bind(index) + const wrongOutput = vi + .spyOn(index, `rangeQuery`) + .mockImplementation((options) => { + const result = rangeQuery(options) + if (!(`from` in options) && options.fromInclusive === false) { + result.delete(`minimum`) + } + return result + }) + try { + expect(() => + expectIndexMatchesModel(index, new Map([[`minimum`, -2]])), + ).toThrowError(/expected/) + } finally { + wrongOutput.mockRestore() + } + }) + + test(`explicit exclusive lower bound rejects an included boundary value`, () => { + const index = new IndexType(1, new PropRef([`value`])) + index.add(`boundary`, { value: -2 }) + index.add(`later`, { value: 0 }) + const rangeQuery = index.rangeQuery.bind(index) + const wrongOutput = vi + .spyOn(index, `rangeQuery`) + .mockImplementation((options) => { + const result = rangeQuery(options) + if ( + options.from === -2 && + options.to === 2 && + options.fromInclusive === false + ) { + result.add(`boundary`) + } + return result + }) + try { + expect(() => + expectIndexMatchesModel( + index, + new Map([ + [`boundary`, -2], + [`later`, 0], + ]), + ), + ).toThrowError(/expected/) + } finally { + wrongOutput.mockRestore() + } + }) + test(`accepts indexed values rather than row keys through the index interface`, () => { const index: IndexInterface = new IndexType( 1, @@ -819,14 +900,114 @@ describe.each(indexTypes)(`%s comparator groups`, (_indexName, IndexType) => { expectCustomGroups(index, rows, identities) }) + test(`moves exact string membership across a comparator-equal update`, () => { + // The input facts put A/a in one order group and B/b in another. The + // comparator implements that order, but the expected exact memberships + // come from the source Map's unchanged strings, not from the comparator. + const group = { A: 0, a: 0, B: 1, b: 1 } as const + type Value = keyof typeof group + const rows = new Map() + const index = new IndexType(1, new PropRef([`value`]), undefined, { + compareFn: (left, right) => + (left as string) + .toLowerCase() + .localeCompare((right as string).toLowerCase()), + }) + const expectMatchesSource = () => { + expect(index.keyCount).toBe(rows.size) + for (const value of Object.keys(group) as Array) { + const exactKeys = new Set( + [...rows] + .filter(([, current]) => current === value) + .map(([key]) => key), + ) + expect(index.equalityLookup(value)).toEqual(exactKeys) + expect(index.lookup(`eq`, value)).toEqual(exactKeys) + expect(index.rangeQuery({ from: value, to: value })).toEqual( + new Set( + [...rows] + .filter(([, current]) => group[current] === group[value]) + .map(([key]) => key), + ), + ) + } + } + + // Check after each direct index action. First update a singleton, as in + // the legacy witness; then add peers in both comparator groups. A shortcut + // limited to singleton indexes must fail before the peers can mask it. + index.add(`changed`, { value: `A` }) + rows.set(`changed`, `A`) + expectMatchesSource() + + const skipEqualUpdate = vi.spyOn(index, `update`).mockImplementation(() => { + if (index.keyCount === 1) return + }) + try { + index.update(`changed`, { value: `A` }, { value: `a` }) + rows.set(`changed`, `a`) + expect(() => expectMatchesSource()).toThrowError(/expected/) + } finally { + skipEqualUpdate.mockRestore() + rows.set(`changed`, `A`) + } + + for (const [key, oldValue, newValue] of [ + [`changed`, `A`, `a`], + [`changed`, `a`, `A`], + ] as Array<[string, Value, Value]>) { + index.update(key, { value: oldValue }, { value: newValue }) + rows.set(key, newValue) + expectMatchesSource() + } + + index.add(`anchor`, { value: `a` }) + rows.set(`anchor`, `a`) + expectMatchesSource() + index.add(`other`, { value: `B` }) + rows.set(`other`, `B`) + expectMatchesSource() + + // These histories additionally retain another exact value in the same + // order group and then move a key across groups. + for (const [key, oldValue, newValue] of [ + [`changed`, `A`, `a`], + [`other`, `B`, `b`], + [`other`, `b`, `a`], + ] as Array<[string, Value, Value]>) { + index.update(key, { value: oldValue }, { value: newValue }) + rows.set(key, newValue) + expectMatchesSource() + } + + // EqualityLookup and the public lookup dispatch are separate methods. + // A dispatch that substitutes the comparator group for exact equality + // would pass an equalityLookup-only check after the same-group update. + const lookup = index.lookup.bind(index) + const wrongLookup = vi + .spyOn(index, `lookup`) + .mockImplementation((operation, value) => + operation === `eq` + ? index.rangeQuery({ from: value, to: value }) + : lookup(operation, value), + ) + try { + expect(() => expectMatchesSource()).toThrowError(/expected/) + } finally { + wrongLookup.mockRestore() + } + }) + // Group grammar: 2..20 generated IDs in 0..4, with a forced duplicate first // group and a distinct group 5. The same bounded input is run in both index // types. The exact-identity lane gives every row a fresh array, then retires // and reuses handles. The custom-comparator lane additionally reverses // insertion. Without the duplicate, representative retirement is absent; // without the distinct group, group order is unobservable; without reversed - // insertion, insertion order can masquerade as lexical key order. The pinned - // [0, 1] case reconstructs all three distinctions. Invalid comparator + // insertion, insertion order can masquerade as lexical key order. Replacing + // one value with a fresh value in the same group checks exact membership + // independently of comparator position; a later update crosses groups. + // The pinned [0, 1] case reconstructs these distinctions. Invalid comparator // results and inconsistent comparator order are outside these two grammars. test.each([20260930, undefined])( `preserves exact equality while ordered traversal retains every row (seed %s)`, @@ -1023,6 +1204,23 @@ describe.each(indexTypes)(`%s comparator groups`, (_indexName, IndexType) => { expectCustomGroups(index, rows, identities) const previous = rows[0]! + const equalGroup = Object.freeze({ + key: previous.key, + groupId: previous.groupId, + value: { + groupId: previous.groupId, + position: previous.value.position, + }, + }) + identities.push(equalGroup.value) + index.update( + previous.key, + { value: previous.value }, + { value: equalGroup.value }, + ) + rows[0] = equalGroup + expectCustomGroups(index, rows, identities) + const changed = Object.freeze({ key: previous.key, groupId: 99, @@ -1031,16 +1229,20 @@ describe.each(indexTypes)(`%s comparator groups`, (_indexName, IndexType) => { identities.push(changed.value) index.update( previous.key, - { value: previous.value }, + { value: equalGroup.value }, { value: changed.value }, ) rows[0] = changed expectCustomGroups(index, rows, identities) - for (const retired of [removed, previous]) { + for (const [ordinal, retired] of [ + removed, + previous, + equalGroup, + ].entries()) { const reused = Object.freeze({ ...retired, - key: `reused-${retired.key}`, + key: `reused-${ordinal}-${retired.key}`, }) index.add(reused.key, { value: reused.value }) rows.push(reused) diff --git a/packages/db/tests/index-update-short-circuit.test.ts b/packages/db/tests/index-update-short-circuit.test.ts index 216bc5110d..0e22377ba6 100644 --- a/packages/db/tests/index-update-short-circuit.test.ts +++ b/packages/db/tests/index-update-short-circuit.test.ts @@ -88,19 +88,6 @@ describe.each(indexTypes)(`%s update`, (_indexName, IndexType) => { expect(index.lookup(`eq`, null)).toEqual(new Set([`a`])) }) - it(`does not use comparator equality to skip an update`, () => { - const index = createIndex({ - compareFn: (a: string, b: string) => - a.toLowerCase().localeCompare(b.toLowerCase()), - }) - index.add(`a`, { value: `A` }) - - index.update(`a`, { value: `A` }, { value: `a` }) - - expect(index.lookup(`eq`, `A`)).toEqual(new Set()) - expect(index.lookup(`eq`, `a`)).toEqual(new Set([`a`])) - }) - it(`preserves the previous error behavior when evaluation fails`, () => { const index = createIndex() index.add(`a`, { value: 1 }) From 0d21a22ec56bc06a47776690d7cdbfded0ae9acc Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 15:27:55 -0600 Subject: [PATCH 02/46] test(db): move nullish index membership cases into oracle --- docs/contributing/oracle-coverage.md | 12 ++ .../btree-index-undefined-values.test.ts | 49 ----- .../index-update-oracle.property.test.ts | 196 +++++++++++++++++- .../tests/index-update-short-circuit.test.ts | 10 - packages/db/tests/oracle-config.ts | 1 + packages/db/tests/oracle-replay-manifest.ts | 6 +- packages/db/tests/oracle-replay.test.ts | 2 +- 7 files changed, 211 insertions(+), 65 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index ea0075e005..47b23737aa 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -459,6 +459,18 @@ generic visitor behavior outside the named collectors. | Boundary refinements | [cleanup/restart](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-cleanup-restart-oracle.test.ts), [idle cleanup](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-idle-cleanup.test.ts), [issue #1891 async-cleanup review](oracle-reviews/issue-1891-async-cleanup.md), [issue #1891 cleanup-start follow-up](oracle-reviews/issue-1891-cleanup-start-followup.md), [metadata publication](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-metadata-publication-oracle.property.test.ts), [row metadata composition](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-row-metadata-composition-oracle.test.ts), [nested-begin metadata rebuild review](oracle-reviews/2026-10-06-nested-begin-metadata-rebuild.md), [state mutation round 4](oracle-reviews/2026-10-06-state-mutation-round-4.md), [state retention](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-state-retention-oracle.property.test.ts), [acquisition cells](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-subscription-lifecycle-oracle.test.ts), [D2 source reconciliation](https://github.com/TanStack/db/blob/main/packages/db/tests/d2-source-reconciliation-oracle.property.test.ts), [top-K support windows](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/operators/topk-support-window-oracle.test.ts), [nested Query work](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/includes-work-counter-oracle.test.ts) | Explicit lifecycle products, independent source maps and weighted relations, exact publication cuts, support/multiplicity, and value-plus-work observations. These refine the larger subsystem models; they do not replace them. | | Small structures and test mechanics | [SortedMap](https://github.com/TanStack/db/blob/main/packages/db/tests/SortedMap-oracle.test.ts), [cleanup queue](https://github.com/TanStack/db/blob/main/packages/db/tests/cleanup-queue-oracle.property.test.ts), [live-query GC clock](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-gc-clock.test.ts), [guarded replay](https://github.com/TanStack/db/blob/main/packages/db/tests/oracle-replay.test.ts) | Map/full-sort and elapsed appointment-list models with executed target/seed/path checks. SortedMap also checks deferred bulk writes, array-movement work, scan reuse, live iterators, runtime `undefined` keys, and public single-transaction and queued-transaction Collection sync paths. It does not settle the contract for comparator-observed values mutated after publication or every legal iterator/mutation interleaving. Wall-clock steps do not move due times, and a focused public witness checks source subscription release. Callback-reentrant scheduling, OS suspend/resume, environments without `performance.now()`, and fake/real Performance clock replacement while GC is pending remain outside this coverage. Tests must drain or reset pending GC before switching timer providers. | +The direct [index refinement oracle](../../packages/db/tests/index-update-oracle.property.test.ts) +also owns exact nullish membership for BasicIndex and BTreeIndex. Fresh +single-key updates cross `undefined→null`, `null→undefined`, +`1→undefined`, `undefined→1`, and `undefined→undefined`; a coexisting-add +history keeps null and undefined keys apart. A plain source Map supplies the +expected `equalityLookup` and `lookup('eq')` sets after each returned action. +Fixed and random three-key put/delete histories exercise nearby states. A +skipped update and a unioned lookup dispatch fail the check. This direct-index +evidence does not establish ordered, range, `IN`, live-query, bucket-identity, +or instrumentation behavior. The [transfer ledger](oracle-reviews/index-unit-retirement-round-2.md) +records the five retired legacy blocks and review boundary. + The alias-scope owner also has a compiler invariant witness for a malformed include plan: removing a captured ancestor's parent projection after the builder creates it must fail before child rows are read. Without that check, diff --git a/packages/db/tests/btree-index-undefined-values.test.ts b/packages/db/tests/btree-index-undefined-values.test.ts index e2160f06da..aea1a6fbd9 100644 --- a/packages/db/tests/btree-index-undefined-values.test.ts +++ b/packages/db/tests/btree-index-undefined-values.test.ts @@ -209,21 +209,6 @@ describe(`BTreeIndex - undefined value handling`, () => { expect(orderedEntriesArray(index)[0]![0]).toBe(undefined) }) - - it(`should handle mixed undefined and null values`, () => { - const index = createIndex(`value`) - index.add(`null1`, { value: null }) - index.add(`undef1`, { value: undefined }) - index.add(`num1`, { value: 1 }) - - const nullItems = index.equalityLookup(null) - const undefItems = index.equalityLookup(undefined) - - expect(nullItems.size).toBe(1) - expect(nullItems).toContain(`null1`) - expect(undefItems.size).toBe(1) - expect(undefItems).toContain(`undef1`) - }) }) describe(`rangeQuery with undefined bounds`, () => { @@ -351,40 +336,6 @@ describe(`BTreeIndex - undefined value handling`, () => { }) }) - describe(`update operation with undefined`, () => { - it(`should update item from defined to undefined value`, () => { - const index = createIndex(`value`) - index.add(`a`, { value: 1 }) - - index.update(`a`, { value: 1 }, { value: undefined }) - - expect(index.equalityLookup(1).size).toBe(0) - expect(index.equalityLookup(undefined).size).toBe(1) - expect(index.equalityLookup(undefined)).toContain(`a`) - }) - - it(`should update item from undefined to defined value`, () => { - const index = createIndex(`value`) - index.add(`a`, { value: undefined }) - - index.update(`a`, { value: undefined }, { value: 1 }) - - expect(index.equalityLookup(undefined).size).toBe(0) - expect(index.equalityLookup(1).size).toBe(1) - expect(index.equalityLookup(1)).toContain(`a`) - }) - - it(`should update item from undefined to undefined (same value)`, () => { - const index = createIndex(`value`) - index.add(`a`, { value: undefined }) - - index.update(`a`, { value: undefined }, { value: undefined }) - - expect(index.equalityLookup(undefined).size).toBe(1) - expect(index.equalityLookup(undefined)).toContain(`a`) - }) - }) - describe(`valueMapData getter`, () => { it(`should return undefined keys denormalized, not as sentinel`, () => { const index = createIndex(`value`) diff --git a/packages/db/tests/index-update-oracle.property.test.ts b/packages/db/tests/index-update-oracle.property.test.ts index d3cfcd4024..bb674210f8 100644 --- a/packages/db/tests/index-update-oracle.property.test.ts +++ b/packages/db/tests/index-update-oracle.property.test.ts @@ -109,10 +109,10 @@ const arbitraryAction: fc.Arbitrary = fc.oneof( const probeValues: Array = [-3, -2, -1, -0, 0, 1, 2, 3, 99] const rangeBoundaries: Array = [-2, 0, 2] -function groupKeysByValue( - rows: Map, -): Map> { - const groups = new Map>() +function groupKeysByValue( + rows: ReadonlyMap, +): Map> { + const groups = new Map>() for (const [key, value] of rows) { const keys = groups.get(value) if (keys) { @@ -470,6 +470,194 @@ describe.each(indexTypes)(`%s update properties`, (_indexName, IndexType) => { }) }) +/** + * Direct index equality keeps `undefined` and `null` in distinct exact-value + * buckets, even though the default ordering comparator places both at one + * position. The established direct-index tests below exercise this distinction + * through `equalityLookup` and `lookup('eq')`. This law does not describe the + * live-query WHERE evaluator, which uses three-valued nullish equality. + * + * A source Map predicts membership without index normalization or comparator + * classification. Fresh singleton histories preserve the old update cuts; + * coexisting values expose a wrong union of nullish buckets. The generated + * tail adds nearby put/delete histories over three keys and four values. An + * absent source key is added, while an owned key is updated. A direct update + * of an absent key is outside this grammar. Removing either nullish value + * loses the distinction; removing the defined values loses transitions across + * it; one key cannot expose surviving peers after a move or deletion. + * After each direct action returns, both exact-lookup methods and key count + * must agree with the Map. Ordered reads, range queries, `IN`, and bucket + * object identity belong to other observations. + */ +type NullishIndexValue = number | null | undefined +type NullishIndexAction = + | { type: `put`; key: string; value: NullishIndexValue } + | { type: `delete`; key: string } + +const nullishProbeValues: ReadonlyArray = [ + undefined, + null, + 1, + 2, +] +const arbitraryNullishAction: fc.Arbitrary = fc.oneof( + fc.record({ + type: fc.constant(`put` as const), + key: fc.constantFrom(`a`, `b`, `c`), + value: fc.constantFrom(...nullishProbeValues), + }), + fc.record({ + type: fc.constant(`delete` as const), + key: fc.constantFrom(`a`, `b`, `c`), + }), +) + +function expectNullishMembership( + index: BaseIndex, + rows: ReadonlyMap, +): void { + // Map key identity distinguishes null from undefined without consulting the + // index's normalized keys or its ordering comparator. + const groups = groupKeysByValue(rows) + expect(index.keyCount).toBe(rows.size) + expect(indexedKeysSet(index)).toEqual(new Set(rows.keys())) + for (const value of nullishProbeValues) { + const expected = groups.get(value) ?? new Set() + expect(index.equalityLookup(value)).toEqual(expected) + expect(index.lookup(`eq`, value)).toEqual(expected) + } +} + +function applyNullishAction( + index: BaseIndex, + rows: Map, + action: NullishIndexAction, +): void { + // The driver maps source ownership to the real add, update, and remove API. + if (action.type === `put`) { + if (rows.has(action.key)) { + index.update( + action.key, + { value: rows.get(action.key) }, + { value: action.value }, + ) + } else { + index.add(action.key, { value: action.value }) + } + rows.set(action.key, action.value) + } else if (rows.has(action.key)) { + index.remove(action.key, { value: rows.get(action.key) }) + rows.delete(action.key) + } + expectNullishMembership(index, rows) +} + +describe.each(indexTypes)( + `%s nullish exact membership`, + (_indexName, IndexType) => { + test.each([ + [`undefined to null`, undefined, null], + [`null to undefined`, null, undefined], + [`defined to undefined`, 1, undefined], + [`undefined to defined`, undefined, 1], + [`undefined to undefined`, undefined, undefined], + ] as const)(`keeps membership after %s`, (_label, before, after) => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + applyNullishAction(index, rows, { type: `put`, key: `a`, value: before }) + applyNullishAction(index, rows, { type: `put`, key: `a`, value: after }) + }) + + test(`keeps coexisting null and undefined buckets distinct`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + for (const [key, value] of [ + [`null1`, null], + [`undef1`, undefined], + [`num1`, 1], + ] as const) { + applyNullishAction(index, rows, { type: `put`, key, value }) + } + }) + + test.each([20261012, undefined])( + `refines nullish membership histories (seed %s)`, + async (seed) => { + await fc.assert( + fc.asyncProperty( + fc.array(arbitraryNullishAction, { minLength: 1, maxLength: 40 }), + async (actions) => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + expectNullishMembership(index, rows) + for (const action of actions) { + applyNullishAction(index, rows, action) + } + }, + ), + seed === undefined + ? oraclePropertyOptions(100, `index-update.nullish-membership`) + : { seed, numRuns: oracleRuns(100) }, + ) + }, + ) + + test(`rejects a skipped undefined-to-null update`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + applyNullishAction(index, rows, { + type: `put`, + key: `a`, + value: undefined, + }) + const skipped = vi.spyOn(index, `update`).mockImplementation(() => {}) + try { + expect(() => + applyNullishAction(index, rows, { + type: `put`, + key: `a`, + value: null, + }), + ).toThrowError(/expected/) + expect(skipped).toHaveBeenCalledOnce() + } finally { + skipped.mockRestore() + } + }) + + test(`rejects lookup dispatch that unites nullish buckets`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + applyNullishAction(index, rows, { + type: `put`, + key: `null1`, + value: null, + }) + applyNullishAction(index, rows, { + type: `put`, + key: `undef1`, + value: undefined, + }) + const lookup = index.lookup.bind(index) + const united = vi + .spyOn(index, `lookup`) + .mockImplementation((op, value) => + op === `eq` && value === undefined + ? new Set([`null1`, `undef1`]) + : lookup(op, value), + ) + try { + expect(() => expectNullishMembership(index, rows)).toThrowError( + /expected/, + ) + expect(united).toHaveBeenCalledWith(`eq`, undefined) + } finally { + united.mockRestore() + } + }) + }, +) + /** * A custom index comparator must return a number that is not NaN. * diff --git a/packages/db/tests/index-update-short-circuit.test.ts b/packages/db/tests/index-update-short-circuit.test.ts index 0e22377ba6..bf22354a6e 100644 --- a/packages/db/tests/index-update-short-circuit.test.ts +++ b/packages/db/tests/index-update-short-circuit.test.ts @@ -78,16 +78,6 @@ describe.each(indexTypes)(`%s update`, (_indexName, IndexType) => { expect(index.lookup(`eq`, 2)).toEqual(new Set([`a`])) }) - it(`does not conflate undefined and null`, () => { - const index = createIndex() - index.add(`a`, { value: undefined }) - - index.update(`a`, { value: undefined }, { value: null }) - - expect(index.lookup(`eq`, undefined)).toEqual(new Set()) - expect(index.lookup(`eq`, null)).toEqual(new Set([`a`])) - }) - it(`preserves the previous error behavior when evaluation fails`, () => { const index = createIndex() index.add(`a`, { value: 1 }) diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index 5aa174b416..0d1059f4dd 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -61,6 +61,7 @@ const staticOracleProperties = [ `cursor.partial-local-order`, `cursor.repeat-construction`, `index-update.reference-model`, + `index-update.nullish-membership`, `index-update.exact-identity`, `index-update.custom-comparator`, `query-identity.compiled-output`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index 37e85d04d9..bea5072439 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -32,7 +32,7 @@ const ownerGroups: ReadonlyArray = [ [ `db/tests/index-update-oracle.property.test.ts`, `index-update`, - `reference-model exact-identity custom-comparator`, + `reference-model nullish-membership exact-identity custom-comparator`, ], [ `db/tests/cursor-oracle.property.test.ts`, @@ -266,6 +266,10 @@ const indexReplayTestNames = new Map([ `index-update.reference-model`, `matches a reference model across valid operation sequences \\(seed undefined\\)`, ], + [ + `index-update.nullish-membership`, + `refines nullish membership histories \\(seed undefined\\)`, + ], [ `index-update.exact-identity`, `preserves exact equality while ordered traversal retains every row \\(seed undefined\\)`, diff --git a/packages/db/tests/oracle-replay.test.ts b/packages/db/tests/oracle-replay.test.ts index de172a716b..8a6c3c0ead 100644 --- a/packages/db/tests/oracle-replay.test.ts +++ b/packages/db/tests/oracle-replay.test.ts @@ -428,7 +428,7 @@ describe(`named oracle replay manifest`, () => { const direct = oracleReplayManifest.filter(({ property }) => /^(index-update|pagination\.matrix)\./.test(property), ) - expect(direct).toHaveLength(19) + expect(direct).toHaveLength(20) for (const entry of direct) { expect(entry.status).toBe(`assertion`) expect(entry.testNamePattern).toBeDefined() From b56d1a45c067f85ac8f4f079a7aa409d572a3785 Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 15:29:45 -0600 Subject: [PATCH 03/46] docs: record nullish index oracle retirement review --- .../index-unit-retirement-round-2.md | 92 +++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 docs/contributing/oracle-reviews/index-unit-retirement-round-2.md diff --git a/docs/contributing/oracle-reviews/index-unit-retirement-round-2.md b/docs/contributing/oracle-reviews/index-unit-retirement-round-2.md new file mode 100644 index 0000000000..e14bad4bfd --- /dev/null +++ b/docs/contributing/oracle-reviews/index-unit-retirement-round-2.md @@ -0,0 +1,92 @@ +# Index unit retirement: nullish exact membership + +The source-isolated loss audit used baseline `73a5d4ec4`. The receiving owner is +`packages/db/tests/index-update-oracle.property.test.ts`. No source block below +was an exact witness in that baseline: its generated update values were numeric, +and its fixed nullish case read ranges and cursors rather than equality buckets. + +## Transfer ledger before retirement + +| Legacy block | Smallest source state and direct action | Exact source observation and law | Plausible wrong result | +| --- | --- | --- | --- | +| `index-update-short-circuit.test.ts`, `does not conflate undefined and null` | BasicIndex and BTreeIndex: fresh `a=undefined`; update `a→null` | At update return, `lookup('eq', undefined)=∅` and `lookup('eq', null)={a}`. The title states distinct exact membership. | A loose-nullish shortcut leaves `a` in the old bucket; a dispatch that returns the comparator group includes it in both. | +| `btree-index-undefined-values.test.ts`, `should update item from defined to undefined value` | BTreeIndex: fresh `a=1`; update `a→undefined` | At update return, `equalityLookup(1)=∅` and `equalityLookup(undefined)={a}`. | The old key retires but the new undefined key is lost. | +| Same file, `should update item from undefined to defined value` | BTreeIndex: fresh `a=undefined`; update `a→1` | At update return, `equalityLookup(undefined)=∅` and `equalityLookup(1)={a}`. | The new key arrives but the undefined membership remains. | +| Same file, `should update item from undefined to undefined (same value)` | BTreeIndex: fresh `a=undefined`; update `a→undefined` | At update return, `equalityLookup(undefined)={a}`. This checks conservation, not bucket identity. | Remove without re-add drops `a`. | +| Same file, `should handle mixed undefined and null values` | BTreeIndex: fresh adds `null1=null`, `undef1=undefined`, `num1=1` | The two exact lookups return their respective singleton sets. The block checks no ordering despite its enclosing heading. | Returning the shared nullish comparator group for either value includes the other key. | + +The BTree file's opening infinite-loop and sentinel comments concern traversal; +they do not establish an equality law. `IndexInterface` exposes both lookup +methods, and the established direct-index tests distinguish nullish exact +values. The receiving oracle already states the separation between exact +membership and comparator position. This transfer is limited to direct index +methods. A live-query WHERE comparison with a nullish operand follows +three-valued logic, so it is outside this law. + +## Receiving law and evidence to check + +A plain source `Map` retains each key's exact value. The oracle recomputes +equality buckets without index normalization or comparator classification. +It drives each source action through BasicIndex and BTreeIndex and checks +`keyCount`, indexed keys, `equalityLookup`, and `lookup('eq')` after return. +Five fresh singleton or coexisting-add histories reconstruct the source cases; +an additional `null→undefined` singleton and fixed/random put/delete histories +challenge nearby legal transitions. The generated domain is three keys and +`undefined`, `null`, `1`, `2`. A direct update of an absent source key is +excluded; the driver uses `add` for that source action. + +A skipped `undefined→null` update and a wrong `lookup('eq')` union are intended +to fail the model comparison. Both controls must reach the relevant method and +fail at the returned-action lookup assertion. The nullish scope does not imply +ordered, range, `IN`, live-query, bucket-identity, or instrumentation coverage. + +## Execution and hostile review + +The reviewed code head is `0d21a22ec56bc06a47776690d7cdbfded0ae9acc`. +The five ledger blocks were removed at that head; no other test block in either +source file was retired. The oracle and both remaining source files passed +411/411 targeted tests with zero type errors. The named replay manifest's +selector tests passed. Fixed seed `20261012` and a seedless campaign each run +the same nullish property with 100 histories per index type. Guarded replay at +seed `20261012`, path `0` selected only the BasicIndex and BTreeIndex random +lanes and recorded one completed 100-run witness for each. + +The in-test skipped-update and unioned-lookup controls fail at the intended +Map-versus-lookup assertion. For replay failure fidelity, a temporary wrong +expected rule treated every `undefined` bucket as empty. It failed the +`equalityLookup(undefined)` comparison with seed `20261012`, shrink path +`0:1:1:1:1:1:0`, and counterexample `put(a, undefined)`. The guarded replay +reached one BasicIndex witness, marked it failed, and exited nonzero. After +restoring the oracle, that exact replay passed. This was a temporary model +fault for replay calibration, not a production mutant or a production bug. + +The independent hostile reviewer `/root/review_nullish_membership` first found +that the new property lacked a checked replay manifest entry. That setup +failure was not counted as an oracle kill. The manifest entry, selector, and +async property reporter were repaired. The reviewer rechecked the committed +head after the commit hook and found no remaining semantic blocker. No +production defect was demonstrated, so production code did not change. + +## Oracle guide audit at the reviewed head + +| Requirement | Outcome | +| --- | --- | +| ORC-001 | The established direct-index tests and `IndexInterface` exact-lookup methods supply the law. The claim excludes live-query WHERE nullish equality and the other observations listed above. | +| ORC-002 | A native source Map computes exact buckets without production normalization, comparator classification, or index storage. | +| ORC-003 | Opening prose states the law and limits; adjacent prose names the Map rule, legal actions, real index methods, observed sets, and returned-action checkpoint. | +| ORC-004 | Five fresh histories reconstruct the old witnesses; the adjacent `null→undefined` case and generated put/delete histories add nearby states. Removing either nullish value loses the distinguishing law; removing defined values loses defined/nullish moves; one key loses surviving-peer histories. The domain is three keys, four values, and at most 40 generated actions. An absent-key direct `update` is excluded because the source driver calls `add`. | +| ORC-005 | Both index types call real `add`, `update`, and `remove`, then compare exact returned sets and key count after each action. The fixed histories reach all five source states. | +| ORC-006 | A skipped `undefined→null` update and a unioned `lookup('eq')` result each cause an assertion failure at the exact-membership check. | +| ORC-007 | Fixed and seedless campaigns share one property, generator, recorder, comparison, and 100-run budget. The named guarded replay reaches the requested seed/path, and a captured controlled failure replays as a failed witness. | +| ORC-008 | The Map keeps key/value ownership because a later update or delete can distinguish sources with equal bucket sizes but different owners. | +| ORC-009 | `put` is a model source action mapped to index `add` or `update`; `delete` maps to `remove`. The shared terms `history`, `observation`, and `checkpoint` follow the glossary. | +| ORC-010 | The wrong-result spies restore in `finally`; the replay guard retains the failure witness separately from process status. No external resources are acquired. | +| ORC-011 | No plausible shared semantic fault requiring a second formulation was identified for the bounded nullish Map rule. Ordered comparator behavior is deliberately outside it. | +| ORC-012 | This record names the exact reviewed code head, audit result, controls, execution, and remaining scope. | +| ORC-013 | The `undefined→null` singleton and coexisting null/undefined adds distinguish no-op update and bucket-union alternatives at returned-action lookup checks. | +| ORC-014 | No controlled provider or host premise supports a claim about a real provider or host. | + +The next nullish candidates need their own observations: ordered and cursor +reads, `IN` union, custom comparator arguments, and the public live-query +path. The remaining source tests continue to own their exact assertions until +those transfers pass the same gates. From d880f3139bbbba7141dc658d84d3602217c2960f Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 15:44:44 -0600 Subject: [PATCH 04/46] test(db): move BTree nullish membership cases into oracle --- docs/contributing/oracle-coverage.md | 26 +- .../index-unit-retirement-round-3.md | 91 +++++++ .../btree-index-undefined-values.test.ts | 50 ---- .../index-update-oracle.property.test.ts | 232 ++++++++++++++++-- 4 files changed, 323 insertions(+), 76 deletions(-) create mode 100644 docs/contributing/oracle-reviews/index-unit-retirement-round-3.md diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 47b23737aa..d018ef1ab3 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -460,16 +460,22 @@ generic visitor behavior outside the named collectors. | Small structures and test mechanics | [SortedMap](https://github.com/TanStack/db/blob/main/packages/db/tests/SortedMap-oracle.test.ts), [cleanup queue](https://github.com/TanStack/db/blob/main/packages/db/tests/cleanup-queue-oracle.property.test.ts), [live-query GC clock](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-gc-clock.test.ts), [guarded replay](https://github.com/TanStack/db/blob/main/packages/db/tests/oracle-replay.test.ts) | Map/full-sort and elapsed appointment-list models with executed target/seed/path checks. SortedMap also checks deferred bulk writes, array-movement work, scan reuse, live iterators, runtime `undefined` keys, and public single-transaction and queued-transaction Collection sync paths. It does not settle the contract for comparator-observed values mutated after publication or every legal iterator/mutation interleaving. Wall-clock steps do not move due times, and a focused public witness checks source subscription release. Callback-reentrant scheduling, OS suspend/resume, environments without `performance.now()`, and fake/real Performance clock replacement while GC is pending remain outside this coverage. Tests must drain or reset pending GC before switching timer providers. | The direct [index refinement oracle](../../packages/db/tests/index-update-oracle.property.test.ts) -also owns exact nullish membership for BasicIndex and BTreeIndex. Fresh -single-key updates cross `undefined→null`, `null→undefined`, -`1→undefined`, `undefined→1`, and `undefined→undefined`; a coexisting-add -history keeps null and undefined keys apart. A plain source Map supplies the -expected `equalityLookup` and `lookup('eq')` sets after each returned action. -Fixed and random three-key put/delete histories exercise nearby states. A -skipped update and a unioned lookup dispatch fail the check. This direct-index -evidence does not establish ordered, range, `IN`, live-query, bucket-identity, -or instrumentation behavior. The [transfer ledger](oracle-reviews/index-unit-retirement-round-2.md) -records the five retired legacy blocks and review boundary. +also owns exact nullish membership and direct `IN` set selection for BasicIndex +and BTreeIndex. Fresh single-key updates cross `undefined→null`, +`null→undefined`, `1→undefined`, `undefined→1`, and +`undefined→undefined`; a coexisting-add history keeps null and undefined keys +apart. Four more fixed histories preserve two undefined owners beside string +values, an all-defined nonempty index, a mixed `[undefined, 1]` list with two +defined distractors, and a singleton `[undefined]` list with two owners. A +plain source Map supplies expected `equalityLookup`, `lookup('eq')`, +`inArrayLookup`, and `lookup('in')` sets after returned actions. Fixed and random +three-key put/delete histories exercise nearby states with empty, repeated, +both orders of a mixed list, and nullish-only `IN` lists. Skipped-update, wrong equality, lost-owner, +dropped-operand, and wrong-dispatch controls fail the check. This direct-index +evidence does not establish ordered, range, live-query, bucket-identity, or +instrumentation behavior. The [nullish transfer ledger](oracle-reviews/index-unit-retirement-round-2.md) +records the first five retired legacy blocks; the [IN transfer ledger](oracle-reviews/index-unit-retirement-round-3.md) +records the next four and their review boundary. The alias-scope owner also has a compiler invariant witness for a malformed include plan: removing a captured ancestor's parent projection after the diff --git a/docs/contributing/oracle-reviews/index-unit-retirement-round-3.md b/docs/contributing/oracle-reviews/index-unit-retirement-round-3.md new file mode 100644 index 0000000000..d4e7a3a1c4 --- /dev/null +++ b/docs/contributing/oracle-reviews/index-unit-retirement-round-3.md @@ -0,0 +1,91 @@ +# Index unit retirement: exact and IN membership + +The source-isolated Field Lab loss audits used `b56d1a45c` as the frozen +baseline. The receiving owner is +`packages/db/tests/index-update-oracle.property.test.ts`. Its prior nullish +Map law checked `equalityLookup` and `lookup('eq')`, but explicitly excluded +`IN`. The four legacy blocks below remain in the source file until the new +checks and hostile review pass. + +## Transfer ledger before retirement + +| Legacy block in `btree-index-undefined-values.test.ts` | Smallest source history and returned-add observation | Plausible wrong result | Receiving witness | +| --- | --- | --- | --- | +| `should find items with undefined indexed value` | Add `a=active`, `b=undefined`, `c=inactive`, `d=undefined` on `status`; `equalityLookup(undefined)={b,d}`. | A lookup drops one of the two owners or includes a string peer. | The fixed four-add string history computes every exact set from a source Map after each add. | +| `should return empty set when no undefined values exist` | Add `a=1`, `b=2`; `equalityLookup(undefined)=∅` in a nonempty index. | A missing-bucket fallback returns a defined key. | The fixed all-defined history reads the absent group after both adds. | +| `should handle undefined in inArrayLookup` | Add `undef=undefined`, `one=1`, `two=2`, `three=3`; `inArrayLookup([undefined,1])={undef,one}`. | The method ignores the undefined operand, returns only one group, or includes a distractor. | The fixed four-add history and mixed-list probe compare the full set from the Map. | +| `should handle inArrayLookup with only undefined` | Add `undef1=undefined`, `undef2=undefined`, `one=1`; `inArrayLookup([undefined])={undef1,undef2}`. | An undefined-only list returns empty or loses one owner. | The fixed three-add history and singleton-list probe compare the full set from the Map. | + +The `IndexInterface` exposes direct exact and `IN` methods. A source Map owns +keys and values; equality selects one exact-value group, while `IN` selects +keys whose source value occurs in its operand list. The expectation does not +call either production lookup, normalized-key helper, or comparator. The +driver checks both direct methods and `lookup('eq')`/`lookup('in')` for +BasicIndex and BTreeIndex after each source action returns. A generated +three-key, five-value put/delete grammar checks nearby transitions with empty, +repeated, both orders of a mixed list, and nullish-only operand lists. The fixed four-key and string +histories preserve source distinctions outside that generated domain. + +This is a direct-index set-selection law. It does not specify live-query WHERE +or `IN` nullish semantics, ordered traversal, range lookup, bucket identity, +or instrumentation. The `index-update-short-circuit.test.ts` block that also +calls `lookup('in')` stays: its `performance.now` non-call assertion cannot be +observed by this result oracle. The BTree file's traversal, range, custom +comparator, storage, and live-query blocks also stay. + +## Execution and hostile review + +The four ledger blocks were removed; the BTree file now has 12 tests instead +of 16. The receiving oracle, the remaining BTree file, and the short-circuit +file passed together: 423/423 tests and no type errors. The receiving oracle's +fixed-seed and seedless campaigns use the same 100-run property for each index +type. The guarded replay for `index-update.nullish-membership` at seed +`20261012`, path `0` reached two executions, one per index type, with 100 runs +each. The named replay manifest checks passed. + +The in-test wrong-result controls make a lookup lose one undefined owner, +drop the trailing undefined operand in `[1, undefined]`, lose one owner from +`[undefined]`, and return only the first group through `lookup('in')`. Each +reaches the relevant method and fails the Map comparison. The earlier +skipped-update and unioned-equality controls still pass their calibration. + +For failure replay, a temporary wrong expected rule cleared the Map result +for `[undefined, 1]`. The random lane failed at the direct `inArrayLookup` +comparison on `put(a, undefined)` with seed `20261012` and shrink path +`0:1:0:1:1:1:1:0`. Guarded replay reached both index lanes, recorded a failed +one-run witness for each, and exited nonzero. After removing the temporary +fault, that exact replay passed both lanes. This was a model fault for checker +and replay calibration, not a production mutant or a production defect. + +The fresh hostile reviewer `/root/review_in_membership` inspected the source +blocks, production implementations, ledger, oracle, and coverage map. It found +a position-sensitive survivor: all original mixed probes had undefined first. +Adding `[1, undefined]` and a control that drops its trailing undefined +operand killed that survivor. The reviewer independently ran the nullish +membership slice (36 passed, no type errors) and found no remaining semantic +blocker to retiring the four named blocks. It did not run a production mutant. +No production change was justified by this round. + +## Oracle guide audit + +| Requirement | Outcome | +| --- | --- | +| ORC-001 | `IndexInterface` and the established direct-index tests supply exact and `IN` set-selection obligations. The claim excludes WHERE semantics, ordered and range reads, bucket identity, timing, and arbitrary JavaScript values. | +| ORC-002 | A source Map supplies ownership. Expected exact sets come from Map grouping; expected `IN` sets come from operand membership over source rows. Neither uses production normalization, comparator classification, or another production lookup. | +| ORC-003 | Prose beside the model names the law, grammar, direct index driver, returned sets, and after-action checkpoint. | +| ORC-004 | The four fixed histories reconstruct every retired source state and operand. The generated grammar has three keys, five numeric/nullish values, up to 40 put/delete actions, and empty, repeated, mixed, reversed-mixed, and nullish-only lists. The fixed string and four-key histories cover values and width outside that grammar. Removing undefined loses the reported case; removing null loses nullish distinction; removing defined values loses mixed selection and moves; one key loses peer retention. Direct update of an absent source key is excluded. | +| ORC-005 | Both real index classes execute `add`, `update`, and `remove`. The checker compares `equalityLookup`, `lookup('eq')`, `inArrayLookup`, `lookup('in')`, key count, and indexed keys after each action returns. | +| ORC-006 | Four new wrong-result controls fail at the intended comparison; the temporary wrong expected rule also fails the generated `IN` comparison. No production mutant was run. | +| ORC-007 | Fixed and seedless campaigns share one generator and check with 100 runs each per type. Guarded seed/path `0` replay reached both lanes; the captured `0:1:0:1:1:1:1:0` failure replayed as failed, then passed after restoration. | +| ORC-008 | The Map retains each key's value because a later update or delete can distinguish states with the same bucket sizes but different owners. No model state was removed. | +| ORC-009 | `put` maps to index `add` or `update` according to source ownership; `delete` maps to `remove`. `IN` operands are lists of source values, and the after-action comparison is the checkpoint. | +| ORC-010 | Wrong-result spies restore in `finally`. Guarded replay records the property failure separately from child exit and cleans its temporary witness channel. The temporary model fault was removed after replay. | +| ORC-011 | No plausible shared semantic fault requiring a second independent formulation was identified for this bounded primitive-value Map rule. The review's order-sensitive survivor was an enforcement gap and is now checked. | +| ORC-012 | This versioned ledger records source assertions, law limits, execution, hostile review, and remaining scope. The committed code head is identified in the postcommit review addendum. | +| ORC-013 | `[undefined, 1]` versus `[1, undefined]` distinguishes operand-order dependence; `[undefined]` with two owners rejects first-owner-only selection; `[]` and `[1, 1]` check empty and duplicate boundaries. The wrong-result controls fail at returned-set comparisons. | +| ORC-014 | No controlled provider or host premise is used to claim real-provider or host behavior. | + +The remaining legacy blocks keep their distinct observations. The +short-circuit timing assertion, BTree traversal and range behavior, custom +comparator arguments, storage representation, and public live-query result +still need their own owners before retirement. diff --git a/packages/db/tests/btree-index-undefined-values.test.ts b/packages/db/tests/btree-index-undefined-values.test.ts index aea1a6fbd9..2942e0cec6 100644 --- a/packages/db/tests/btree-index-undefined-values.test.ts +++ b/packages/db/tests/btree-index-undefined-values.test.ts @@ -286,56 +286,6 @@ describe(`BTreeIndex - undefined value handling`, () => { }) }) - describe(`equalityLookup with undefined`, () => { - it(`should find items with undefined indexed value`, () => { - const index = createIndex(`status`) - index.add(`a`, { status: `active` }) - index.add(`b`, { status: undefined }) - index.add(`c`, { status: `inactive` }) - index.add(`d`, { status: undefined }) - - const undefinedItems = index.equalityLookup(undefined) - expect(undefinedItems.size).toBe(2) - expect(undefinedItems).toContain(`b`) - expect(undefinedItems).toContain(`d`) - }) - - it(`should return empty set when no undefined values exist`, () => { - const index = createIndex(`value`) - index.add(`a`, { value: 1 }) - index.add(`b`, { value: 2 }) - - expect(index.equalityLookup(undefined).size).toBe(0) - }) - }) - - describe(`inArrayLookup with undefined`, () => { - it(`should handle undefined in inArrayLookup`, () => { - const index = createIndex(`value`) - index.add(`undef`, { value: undefined }) - index.add(`one`, { value: 1 }) - index.add(`two`, { value: 2 }) - index.add(`three`, { value: 3 }) - - const result = index.inArrayLookup([undefined, 1]) - expect(result.size).toBe(2) - expect(result).toContain(`undef`) - expect(result).toContain(`one`) - }) - - it(`should handle inArrayLookup with only undefined`, () => { - const index = createIndex(`value`) - index.add(`undef1`, { value: undefined }) - index.add(`undef2`, { value: undefined }) - index.add(`one`, { value: 1 }) - - const result = index.inArrayLookup([undefined]) - expect(result.size).toBe(2) - expect(result).toContain(`undef1`) - expect(result).toContain(`undef2`) - }) - }) - describe(`valueMapData getter`, () => { it(`should return undefined keys denormalized, not as sentinel`, () => { const index = createIndex(`value`) diff --git a/packages/db/tests/index-update-oracle.property.test.ts b/packages/db/tests/index-update-oracle.property.test.ts index bb674210f8..20b5913a23 100644 --- a/packages/db/tests/index-update-oracle.property.test.ts +++ b/packages/db/tests/index-update-oracle.property.test.ts @@ -473,21 +473,27 @@ describe.each(indexTypes)(`%s update properties`, (_indexName, IndexType) => { /** * Direct index equality keeps `undefined` and `null` in distinct exact-value * buckets, even though the default ordering comparator places both at one - * position. The established direct-index tests below exercise this distinction - * through `equalityLookup` and `lookup('eq')`. This law does not describe the - * live-query WHERE evaluator, which uses three-valued nullish equality. + * position. An `IN` lookup returns the union of the requested exact-value + * memberships, including every owner of a repeated value and no unrequested + * value. The established direct-index tests exercise these methods. This law + * does not describe the live-query WHERE evaluator, which uses three-valued + * nullish equality. * * A source Map predicts membership without index normalization or comparator * classification. Fresh singleton histories preserve the old update cuts; - * coexisting values expose a wrong union of nullish buckets. The generated - * tail adds nearby put/delete histories over three keys and four values. An - * absent source key is added, while an owned key is updated. A direct update - * of an absent key is outside this grammar. Removing either nullish value - * loses the distinction; removing the defined values loses transitions across - * it; one key cannot expose surviving peers after a move or deletion. - * After each direct action returns, both exact-lookup methods and key count - * must agree with the Map. Ordered reads, range queries, `IN`, and bucket - * object identity belong to other observations. + * coexisting values expose a wrong union of nullish buckets. Fixed histories + * also retain two undefined owners beside strings, a nonempty state without + * undefined, and both mixed and undefined-only `IN` lists. The generated tail + * adds nearby put/delete histories over three keys and five values, with both + * orders of a mixed `IN` list, while a fixed four-key history keeps a third + * defined distractor. An absent source key is added, while an owned key is + * updated. A direct update of an absent key is outside this grammar. Each + * nullish value is needed to distinguish the buckets. Defined values exercise + * mixed-list selection and moves across the nullish boundary. At least two + * keys are needed to expose surviving peers. + * After each direct action returns, exact and `IN` methods, their `lookup` + * dispatches, and key count must agree with the Map. Ordered reads, range + * queries, bucket object identity, and instrumentation are outside this law. */ type NullishIndexValue = number | null | undefined type NullishIndexAction = @@ -499,6 +505,18 @@ const nullishProbeValues: ReadonlyArray = [ null, 1, 2, + 3, +] +const nullishInProbes: ReadonlyArray> = [ + [], + [undefined], + [null], + [undefined, 1], + [1, undefined], + [undefined, null], + [3], + [1, 2], + [1, 1], ] const arbitraryNullishAction: fc.Arbitrary = fc.oneof( fc.record({ @@ -512,20 +530,39 @@ const arbitraryNullishAction: fc.Arbitrary = fc.oneof( }), ) -function expectNullishMembership( +function expectExactMembership( index: BaseIndex, - rows: ReadonlyMap, + rows: ReadonlyMap, + exactProbes: ReadonlyArray, + inProbes: ReadonlyArray>, ): void { // Map key identity distinguishes null from undefined without consulting the - // index's normalized keys or its ordering comparator. + // index's normalized keys or its ordering comparator. IN membership follows + // from source-value membership in the operand list, including an empty list. const groups = groupKeysByValue(rows) expect(index.keyCount).toBe(rows.size) expect(indexedKeysSet(index)).toEqual(new Set(rows.keys())) - for (const value of nullishProbeValues) { + for (const value of exactProbes) { const expected = groups.get(value) ?? new Set() expect(index.equalityLookup(value)).toEqual(expected) expect(index.lookup(`eq`, value)).toEqual(expected) } + for (const values of inProbes) { + const expected = new Set( + [...rows] + .filter(([, value]) => values.includes(value)) + .map(([key]) => key), + ) + expect(index.inArrayLookup([...values])).toEqual(expected) + expect(index.lookup(`in`, [...values])).toEqual(expected) + } +} + +function expectNullishMembership( + index: BaseIndex, + rows: ReadonlyMap, +): void { + expectExactMembership(index, rows, nullishProbeValues, nullishInProbes) } function applyNullishAction( @@ -580,6 +617,55 @@ describe.each(indexTypes)( } }) + test(`finds every undefined owner beside distinct string values`, () => { + const index = new IndexType(1, new PropRef([`status`])) + const rows = new Map() + const exactProbes = [undefined, `active`, `inactive`] + const inProbes = [[undefined], [undefined, `active`]] + for (const [key, value] of [ + [`a`, `active`], + [`b`, undefined], + [`c`, `inactive`], + [`d`, undefined], + ] as const) { + index.add(key, { status: value }) + rows.set(key, value) + expectExactMembership(index, rows, exactProbes, inProbes) + } + }) + + test(`returns no undefined key from a nonempty all-defined index`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + applyNullishAction(index, rows, { type: `put`, key: `a`, value: 1 }) + applyNullishAction(index, rows, { type: `put`, key: `b`, value: 2 }) + }) + + test(`unites undefined and defined IN members without distractors`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + for (const [key, value] of [ + [`undef`, undefined], + [`one`, 1], + [`two`, 2], + [`three`, 3], + ] as const) { + applyNullishAction(index, rows, { type: `put`, key, value }) + } + }) + + test(`returns every undefined owner from a singleton IN list`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + for (const [key, value] of [ + [`undef1`, undefined], + [`undef2`, undefined], + [`one`, 1], + ] as const) { + applyNullishAction(index, rows, { type: `put`, key, value }) + } + }) + test.each([20261012, undefined])( `refines nullish membership histories (seed %s)`, async (seed) => { @@ -655,6 +741,120 @@ describe.each(indexTypes)( united.mockRestore() } }) + + test(`rejects an exact lookup that loses one undefined owner`, () => { + const index = new IndexType(1, new PropRef([`status`])) + const rows = new Map([ + [`a`, `active`], + [`b`, undefined], + [`c`, `inactive`], + [`d`, undefined], + ]) + for (const [key, status] of rows) index.add(key, { status }) + const lookup = index.equalityLookup.bind(index) + const truncated = vi + .spyOn(index, `equalityLookup`) + .mockImplementation((value) => + value === undefined ? new Set([`b`]) : lookup(value), + ) + try { + expect(() => + expectExactMembership(index, rows, [undefined], [[undefined]]), + ).toThrowError(/expected/) + expect(truncated).toHaveBeenCalledWith(undefined) + } finally { + truncated.mockRestore() + } + }) + + test(`rejects an IN lookup that drops an undefined operand`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + applyNullishAction(index, rows, { + type: `put`, + key: `undef`, + value: undefined, + }) + applyNullishAction(index, rows, { type: `put`, key: `one`, value: 1 }) + const lookup = index.inArrayLookup.bind(index) + const dropped = vi + .spyOn(index, `inArrayLookup`) + .mockImplementation((values) => + values.length === 2 && values[0] === 1 && values[1] === undefined + ? new Set([`one`]) + : lookup(values), + ) + try { + expect(() => expectNullishMembership(index, rows)).toThrowError( + /expected/, + ) + expect(dropped).toHaveBeenCalledWith([1, undefined]) + } finally { + dropped.mockRestore() + } + }) + + test(`rejects an IN lookup that loses a same-value owner`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + applyNullishAction(index, rows, { + type: `put`, + key: `undef1`, + value: undefined, + }) + applyNullishAction(index, rows, { + type: `put`, + key: `undef2`, + value: undefined, + }) + const lookup = index.inArrayLookup.bind(index) + const truncated = vi + .spyOn(index, `inArrayLookup`) + .mockImplementation((values) => + values.length === 1 && values[0] === undefined + ? new Set([`undef1`]) + : lookup(values), + ) + try { + expect(() => expectNullishMembership(index, rows)).toThrowError( + /expected/, + ) + expect(truncated).toHaveBeenCalledWith([undefined]) + } finally { + truncated.mockRestore() + } + }) + + test(`rejects IN dispatch that returns only its first value group`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + applyNullishAction(index, rows, { + type: `put`, + key: `undef`, + value: undefined, + }) + applyNullishAction(index, rows, { type: `put`, key: `one`, value: 1 }) + const lookup = index.lookup.bind(index) + const firstOnly = vi + .spyOn(index, `lookup`) + .mockImplementation((op, value) => + op === `in` && + Array.isArray(value) && + value.length === 2 && + value[0] === undefined && + value[1] === 1 + ? new Set([`undef`]) + : lookup(op, value), + ) + try { + expect(() => expectNullishMembership(index, rows)).toThrowError( + /expected/, + ) + expect(firstOnly).toHaveBeenCalledWith(`in`, [undefined, 1]) + } finally { + firstOnly.mockRestore() + } + }) }, ) From df18f41bba07589c4aa8a4b17ca676be7d39dc15 Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 15:45:58 -0600 Subject: [PATCH 05/46] docs: record exact-head IN oracle review --- .../index-unit-retirement-round-3.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/contributing/oracle-reviews/index-unit-retirement-round-3.md b/docs/contributing/oracle-reviews/index-unit-retirement-round-3.md index d4e7a3a1c4..93d4543428 100644 --- a/docs/contributing/oracle-reviews/index-unit-retirement-round-3.md +++ b/docs/contributing/oracle-reviews/index-unit-retirement-round-3.md @@ -4,8 +4,8 @@ The source-isolated Field Lab loss audits used `b56d1a45c` as the frozen baseline. The receiving owner is `packages/db/tests/index-update-oracle.property.test.ts`. Its prior nullish Map law checked `equalityLookup` and `lookup('eq')`, but explicitly excluded -`IN`. The four legacy blocks below remain in the source file until the new -checks and hostile review pass. +`IN`. The four legacy blocks below were retired at reviewed code head +`d880f3139bbbba7141dc658d84d3602217c2960f`. ## Transfer ledger before retirement @@ -23,8 +23,8 @@ call either production lookup, normalized-key helper, or comparator. The driver checks both direct methods and `lookup('eq')`/`lookup('in')` for BasicIndex and BTreeIndex after each source action returns. A generated three-key, five-value put/delete grammar checks nearby transitions with empty, -repeated, both orders of a mixed list, and nullish-only operand lists. The fixed four-key and string -histories preserve source distinctions outside that generated domain. +repeated, both orders of a mixed list, and nullish-only operand lists. The +fixed four-key and string histories preserve distinctions outside that domain. This is a direct-index set-selection law. It does not specify live-query WHERE or `IN` nullish semantics, ordered traversal, range lookup, bucket identity, @@ -66,6 +66,12 @@ membership slice (36 passed, no type errors) and found no remaining semantic blocker to retiring the four named blocks. It did not run a production mutant. No production change was justified by this round. +The reviewer rechecked committed head +`d880f3139bbbba7141dc658d84d3602217c2960f` after the commit hook. The +commit removed exactly the four ledger blocks and retained the reviewed oracle +and coverage claims. The reviewer reran all three targeted files: 423/423 +passed with no type errors. `git show --check` was clean. No new blocker arose. + ## Oracle guide audit | Requirement | Outcome | @@ -81,7 +87,7 @@ No production change was justified by this round. | ORC-009 | `put` maps to index `add` or `update` according to source ownership; `delete` maps to `remove`. `IN` operands are lists of source values, and the after-action comparison is the checkpoint. | | ORC-010 | Wrong-result spies restore in `finally`. Guarded replay records the property failure separately from child exit and cleans its temporary witness channel. The temporary model fault was removed after replay. | | ORC-011 | No plausible shared semantic fault requiring a second independent formulation was identified for this bounded primitive-value Map rule. The review's order-sensitive survivor was an enforcement gap and is now checked. | -| ORC-012 | This versioned ledger records source assertions, law limits, execution, hostile review, and remaining scope. The committed code head is identified in the postcommit review addendum. | +| ORC-012 | This versioned ledger records source assertions, law limits, execution, hostile review, and remaining scope against exact reviewed code head `d880f3139bbbba7141dc658d84d3602217c2960f`. | | ORC-013 | `[undefined, 1]` versus `[1, undefined]` distinguishes operand-order dependence; `[undefined]` with two owners rejects first-owner-only selection; `[]` and `[1, 1]` check empty and duplicate boundaries. The wrong-result controls fail at returned-set comparisons. | | ORC-014 | No controlled provider or host premise is used to claim real-provider or host behavior. | From a72a7043d5750b33100ab95b7d3c6a490b15626c Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 16:05:33 -0600 Subject: [PATCH 06/46] test(db): transfer undefined index traversal to oracle --- docs/contributing/oracle-coverage.md | 19 +- .../index-unit-retirement-round-4.md | 110 ++++++ .../btree-index-undefined-values.test.ts | 118 +----- .../index-update-oracle.property.test.ts | 364 +++++++++++++++++- 4 files changed, 488 insertions(+), 123 deletions(-) create mode 100644 docs/contributing/oracle-reviews/index-unit-retirement-round-4.md diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index d018ef1ab3..52e7915fbb 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -470,12 +470,21 @@ defined distractors, and a singleton `[undefined]` list with two owners. A plain source Map supplies expected `equalityLookup`, `lookup('eq')`, `inArrayLookup`, and `lookup('in')` sets after returned actions. Fixed and random three-key put/delete histories exercise nearby states with empty, repeated, -both orders of a mixed list, and nullish-only `IN` lists. Skipped-update, wrong equality, lost-owner, -dropped-operand, and wrong-dispatch controls fail the check. This direct-index -evidence does not establish ordered, range, live-query, bucket-identity, or -instrumentation behavior. The [nullish transfer ledger](oracle-reviews/index-unit-retirement-round-2.md) +both orders of a mixed list, and nullish-only `IN` lists. The same source Map +now predicts unfiltered ordered arrays and selected forward range sets over `undefined`, `null`, +and positive numbers. Fixed histories preserve endpoint versus explicit +undefined cursors, three undefined owners, removal of one of two owners, +explicit versus omitted lower bounds, and two owners between double undefined +bounds. The generated histories compare after each returned direct action. +Skipped-update, wrong equality, lost-owner, dropped-operand, wrong-dispatch, +wrong-cursor, and wrong-range controls fail the check. This direct-index +evidence does not establish mixed-type ordering, custom comparator behavior, +all forward bound combinations, reverse nullish ranges, filtered takes, +private ordered-entry representation, live-query +behavior, bucket identity, or instrumentation. The [nullish transfer ledger](oracle-reviews/index-unit-retirement-round-2.md) records the first five retired legacy blocks; the [IN transfer ledger](oracle-reviews/index-unit-retirement-round-3.md) -records the next four and their review boundary. +records the next four; the [ordered/range ledger](oracle-reviews/index-unit-retirement-round-4.md) +records the current transfer boundary and review. The alias-scope owner also has a compiler invariant witness for a malformed include plan: removing a captured ancestor's parent projection after the diff --git a/docs/contributing/oracle-reviews/index-unit-retirement-round-4.md b/docs/contributing/oracle-reviews/index-unit-retirement-round-4.md new file mode 100644 index 0000000000..46daca7c68 --- /dev/null +++ b/docs/contributing/oracle-reviews/index-unit-retirement-round-4.md @@ -0,0 +1,110 @@ +# Index unit retirement: undefined ordered reads and ranges + +The source-isolated Field Lab loss audits compared the remaining direct +traversal and range blocks in `btree-index-undefined-values.test.ts` against +`index-update-oracle.property.test.ts` at frozen head +`df18f41bba07589c4aa8a4b17ca676be7d39dc15`. The receiver checked one-owner +undefined cursors and numeric ranges, but its generated nullish lane checked +only exact and `IN` membership. The table freezes the source assertions before +changing the receiver. A proposed witness is not retirement evidence until it +executes, rejects a wrong result, and survives hostile review. + +| Legacy block | Source assertion at the returned-action checkpoint | Loss in frozen receiver | Proposed transfer | +| --- | --- | --- | --- | +| `take(n, undefined)` versus `takeFromStart(n)` | After `a=undefined,b=1,c=2`, `takeFromStart(3)` has three keys and `take(3, undefined)=[b,c]`. | The explicit cursor has a singleton analogue; the default-policy endpoint cut is absent. | Sort source rows with nullish values first; compare both exact arrays after each add, with a fixed three-row history. | +| `takeReversed(n, undefined)` versus `takeReversedFromEnd(n)` | In a fresh copy of the same state, the endpoint returns three keys and the explicit cursor returns none. | The explicit reverse cursor has a singleton analogue; the nullish endpoint cut is absent. | Compare both exact reverse arrays in the same fixed history. | +| Store and retrieve three undefined owners | After `a,b,c=undefined`, `takeFromStart(10)=[a,b,c]` and `equalityLookup(undefined).size=3`. | Two-owner exact membership is fixed; three-owner ordered output is absent. | Pin all three owners and compare exact arrays and source-derived equality sets. | +| Remove one of two undefined owners | After adding `a,b=undefined` and removing `a`, exact lookup has size one and contains `b`. | Reachable in generated put/delete histories, but the two-owner deletion cut is not pinned. | Pin the removal and compare the complete exact set after it returns. | +| Iterate past multiple undefined owners | After `undef1,undef2,undef3=undefined`, then `one=1,two=2`, `take(10,undefined)=[one,two]`. | One-owner cursor and multi-owner membership checks never meet. | Pin the five-row history and compare exact cursor arrays after each add. | +| Explicit undefined versus omitted lower range bound | After `undef=undefined,one=1,two=2,three=3`, `{from:undefined,to:2}` inclusive returns exactly `{undef,one,two}`; `{to:2}` returns three keys. | Numeric bounds and nullish upper-only reads are separate fixtures. | Source-derived range sets check explicit and omitted lower bounds, plus exclusive variants that distinguish them, after a fixed four-row history. | +| Double undefined range bound | After `a,b=undefined,c=1`, `{from:undefined,to:undefined}` inclusive returns exactly `{a,b}` and excludes `c`. | No double-bound range check with two undefined owners. | Pin this state and compare the full set against source-derived bounds. | + +The proposed law follows `DEFAULT_COMPARE_OPTIONS` and `defaultComparator` +in `packages/db/src/utils.ts` and `packages/db/src/utils/comparison.ts`, the +direct cursor comments in `packages/db/src/indexes/btree-index.ts`, the +key tie-order contract exercised by `deterministic-ordering.test.ts`, and the +frozen legacy assertions. `IndexInterface` establishes the direct methods but +its reference page does not specify these ordering details. The bounded law +uses default ascending order with nullish values first, positive numbers +ascending, and row keys ordered within one comparator position. Its domain is +`undefined`, `null`, and the numbers 1–3; negative numbers are outside it. +`null` and `undefined` occupy one comparator position for ordered reads while +remaining distinct exact lookup groups. A source Map retains key ownership. +The model sorts and filters that Map; it does not call the index comparator, +normalization, cursor, or range machinery to compute expected results. +`take` and `takeReversed` exclude the cursor's comparator position. Endpoint +methods have no cursor. A range bound exists only when its field is supplied; +an inclusivity flag without that bound has no effect. The production path is +the direct `IndexInterface` on both BasicIndex and BTreeIndex. Unfiltered +ordered arrays at n=1, 3, and 10 and full sets for ten selected forward range +shapes are compared after each add, update, or remove returns. The owner does +not claim all bound combinations or filtered takes. + +The default-order and string-order tests inspect `orderedEntriesArray`, a test +view of internal entries. Those two blocks remain until their private +observation has a receiving owner or the test's representation claim is +explicitly retired. The custom-comparator argument, `valueMapData` storage, +and public live-query blocks also remain. This direct-index law makes no claim +about those observations or live-query scheduling. + +## Execution and hostile review + +The seven ledger blocks were removed after the hostile review. The BTree file +now has five tests. The receiving oracle and that file passed together: +415/415 tests. The package TypeScript check and `git diff --check` passed. +The fixed-seed and seedless nullish campaigns run the same source actions, +checker, and 100-run budget for both index classes. A guarded replay at seed +`20261012`, path `0`, reached both classes with 100 successful runs each. The +named replay harness passed its 36 tests. + +Five wrong-result controls force an endpoint read to lose the undefined owner, +an explicit cursor to repeat one of three undefined owners, an inclusive +explicit lower bound to drop undefined, a double undefined range to lose a +peer, and a null bound to drop its undefined comparator peer. Each fails the +intended source-Map comparison for both index classes and restores its spy in +`finally`. The earlier exact and `IN` controls continue to run. + +For failure replay, a temporary wrong model rule cleared the expected set for +an explicit `undefined` lower bound and upper bound 2. The fixed campaign +failed at the forward range comparison on `put(a, undefined)`, seed `20261012`, +shrink path `0:0:0:0`. Guarded replay at that seed and path recorded one failed +run for each index class and exited nonzero. After removing the temporary rule, +the same guarded replay passed both classes. This was checker calibration, not +a production mutant or product defect. No production code changed. + +The fresh hostile reviewer `/root/review_ordered_range_round4` read the full +legacy file, ledger, changed oracle, coverage map, reference material, and both +index implementations. It identified a surviving wrong design in the broader +claim: a range implementation could treat a `null` bound as an exact-null +bucket because no probe used null as a bound. Inclusive and exclusive null +probes plus a wrong-result control closed that gap. It also required the +contract authority and domain limits to be precise. Its final block-by-block +verdict found no lost source assertion among the seven retired blocks. It did +not run a production mutant. The default-order internal-entry, string-order +internal-entry, custom-comparator argument, storage-view, and live-query tests +remain in the source file. + +## Oracle guide audit + +| Requirement | Evidence and limit | +| --- | --- | +| ORC-001 | The default comparator, BTree cursor comments, deterministic key tie ordering, and frozen direct tests support the bounded law. The interface supplies the methods, not the full semantics. The claim is limited to default options, values `undefined`/`null`/1–3, unfiltered takes at n=1/3/10, and ten selected forward range shapes. | +| ORC-002 | A source Map owns rows. A fresh value-position sort predicts arrays; separate model-bound objects filter source rows for ranges. Neither calls production normalization, comparator, or traversal to compute an expected result. | +| ORC-003 | Opening and adjacent prose distinguish the law, source Map, action grammar, direct index driver, returned arrays/sets, and after-action comparison. | +| ORC-004 | Fixed three-row endpoint, five-row repeated-owner, two-owner removal, four-row range, and double-bound histories reconstruct all retired assertions. The generated put/delete tail has three owners and five values over 1–40 actions. Removing an endpoint method loses the absent-cursor distinction; one owner loses peer traversal; removing null loses comparator-position overlap; removing numeric successors loses cursor continuation; omitting bound presence loses the exclusive comparison. Negative numbers, arbitrary types, filtered takes, reverse ranges, and direct update of an absent owner are excluded. | +| ORC-005 | BasicIndex and BTreeIndex direct methods run after returned add, update, or remove. Exact arrays and complete sets preserve omissions, duplicates in arrays, order, and wrong membership. | +| ORC-006 | Five injected wrong results fail at the intended comparisons. The temporary wrong model rule failed at the forward range comparison and was removed. | +| ORC-007 | Fixed and seedless lanes use the same generated property and 100-run budget. Guarded seed/path `0` replay reached both lanes; captured failure path `0:0:0:0` failed both lanes under the temporary rule and passed after restoration. | +| ORC-008 | No model state was added: the existing Map retains owner-to-value association, needed for later update and delete. Sorted rows and bounds are recomputed observations. | +| ORC-009 | A model `put` maps to add or update according to source ownership; `delete` maps to remove when owned. Model comparator position combines null and undefined only for order/range, while exact membership keeps them distinct. | +| ORC-010 | Spies restore in `finally`; the guarded runner recorded the primary mismatch separately from process exit and removed its private witness channel. | +| ORC-011 | The reviewer found a missing probe, not a fault shared by the model and production. The added null-bound probe distinguishes the model rule from an exact-bucket interpretation. | +| ORC-012 | This versioned ledger holds source assertions, transfer, limits, execution, and the hostile verdict. Exact committed-head recheck is recorded in a follow-up entry below. | +| ORC-013 | Explicit inclusive versus exclusive undefined and null bounds, paired with omitted bounds, distinguish presence and comparator-position grouping. The wrong-result controls fail at those comparisons. | +| ORC-014 | No host or external provider behavior is claimed. | + +The public live-query issue regression remains the only test in this file that +crosses the Collection/query path. The other four retained tests observe +private BTree ordering or storage and custom-comparator arguments; direct +source-Map arrays and sets cannot replace those observations without a +separate contract decision and receiving witness. diff --git a/packages/db/tests/btree-index-undefined-values.test.ts b/packages/db/tests/btree-index-undefined-values.test.ts index 2942e0cec6..df46e80fa3 100644 --- a/packages/db/tests/btree-index-undefined-values.test.ts +++ b/packages/db/tests/btree-index-undefined-values.test.ts @@ -110,84 +110,15 @@ describe(`BTreeIndex - Issue #1186: Infinite loop with undefined indexed values` }) /** - * Direct unit tests for BTreeIndex undefined value handling. - * Tests the sentinel mechanism that distinguishes "start from beginning" - * (no from parameter) from "the key is literally undefined". + * Direct BTreeIndex tests for observations outside the direct-index oracle: + * internal ordering and storage, custom-comparator arguments, and the public + * live-query path above. */ describe(`BTreeIndex - undefined value handling`, () => { function createIndex(propName: string): BTreeIndex { return new BTreeIndex(1, new PropRef([propName]), `test_index`) } - describe(`take vs takeFromStart`, () => { - it(`should distinguish take(n, undefined) from takeFromStart(n)`, () => { - const index = createIndex(`value`) - index.add(`a`, { value: undefined }) - index.add(`b`, { value: 1 }) - index.add(`c`, { value: 2 }) - - const fromStart = index.takeFromStart(3) - expect(fromStart).toHaveLength(3) - - // take(n, undefined) returns items AFTER undefined (smallest value) - const afterUndefined = index.take(3, undefined) - expect(afterUndefined).toEqual([`b`, `c`]) - }) - - it(`should distinguish takeReversed(n, undefined) from takeReversedFromEnd(n)`, () => { - const index = createIndex(`value`) - index.add(`a`, { value: undefined }) - index.add(`b`, { value: 1 }) - index.add(`c`, { value: 2 }) - - const fromEnd = index.takeReversedFromEnd(3) - expect(fromEnd).toHaveLength(3) - - // Since undefined is smallest, nothing comes before it - const beforeUndefined = index.takeReversed(3, undefined) - expect(beforeUndefined).toHaveLength(0) - }) - }) - - describe(`multiple undefined values`, () => { - it(`should store and retrieve multiple items with undefined indexed values`, () => { - const index = createIndex(`priority`) - index.add(`a`, { priority: undefined }) - index.add(`b`, { priority: undefined }) - index.add(`c`, { priority: undefined }) - - expect(index.takeFromStart(10)).toEqual([`a`, `b`, `c`]) - - const undefinedItems = index.equalityLookup(undefined) - expect(undefinedItems.size).toBe(3) - }) - - it(`should correctly remove items with undefined values`, () => { - const index = createIndex(`priority`) - index.add(`a`, { priority: undefined }) - index.add(`b`, { priority: undefined }) - - index.remove(`a`, { priority: undefined }) - - const remaining = index.equalityLookup(undefined) - expect(remaining.size).toBe(1) - expect(remaining).toContain(`b`) - }) - - it(`should iterate past multiple undefined values correctly`, () => { - const index = createIndex(`value`) - index.add(`undef1`, { value: undefined }) - index.add(`undef2`, { value: undefined }) - index.add(`undef3`, { value: undefined }) - index.add(`one`, { value: 1 }) - index.add(`two`, { value: 2 }) - - // Start from undefined, should get items after all undefined values - const afterUndefined = index.take(10, undefined) - expect(afterUndefined).toEqual([`one`, `two`]) - }) - }) - describe(`ordering of undefined relative to other values`, () => { it(`should sort undefined before numbers with default comparator`, () => { const index = createIndex(`value`) @@ -211,49 +142,6 @@ describe(`BTreeIndex - undefined value handling`, () => { }) }) - describe(`rangeQuery with undefined bounds`, () => { - it(`should distinguish explicit undefined from vs no from`, () => { - const index = createIndex(`value`) - index.add(`undef`, { value: undefined }) - index.add(`one`, { value: 1 }) - index.add(`two`, { value: 2 }) - index.add(`three`, { value: 3 }) - - const withExplicitFrom = index.rangeQuery({ - from: undefined, - to: 2, - fromInclusive: true, - toInclusive: true, - }) - expect(withExplicitFrom.size).toBe(3) - expect(withExplicitFrom).toContain(`undef`) - expect(withExplicitFrom).toContain(`one`) - expect(withExplicitFrom).toContain(`two`) - - const withoutFrom = index.rangeQuery({ to: 2, toInclusive: true }) - expect(withoutFrom.size).toBe(3) - }) - - it(`should handle range query from undefined to undefined`, () => { - const index = createIndex(`value`) - index.add(`a`, { value: undefined }) - index.add(`b`, { value: undefined }) - index.add(`c`, { value: 1 }) - - const justUndefined = index.rangeQuery({ - from: undefined, - to: undefined, - fromInclusive: true, - toInclusive: true, - }) - - expect(justUndefined.size).toBe(2) - expect(justUndefined).toContain(`a`) - expect(justUndefined).toContain(`b`) - expect(justUndefined).not.toContain(`c`) - }) - }) - describe(`custom comparator with undefined values`, () => { it(`should pass actual undefined to custom comparator, not sentinel`, () => { const comparisons: Array<[unknown, unknown]> = [] diff --git a/packages/db/tests/index-update-oracle.property.test.ts b/packages/db/tests/index-update-oracle.property.test.ts index 20b5913a23..29645e5e77 100644 --- a/packages/db/tests/index-update-oracle.property.test.ts +++ b/packages/db/tests/index-update-oracle.property.test.ts @@ -492,8 +492,22 @@ describe.each(indexTypes)(`%s update properties`, (_indexName, IndexType) => { * mixed-list selection and moves across the nullish boundary. At least two * keys are needed to expose surviving peers. * After each direct action returns, exact and `IN` methods, their `lookup` - * dispatches, and key count must agree with the Map. Ordered reads, range - * queries, bucket object identity, and instrumentation are outside this law. + * dispatches, unfiltered ordered reads, selected forward ranges, and key count must agree with + * the Map. The default comparator puts null and undefined at one ordered + * position before the positive numbers in this bounded domain, although + * exact lookup keeps them apart. Endpoint reads start at an end; an explicit + * cursor excludes its entire comparator position. Range bounds constrain + * positions only when supplied, even when the supplied value is undefined. + * The source Map is sorted anew for ordered reads and filtered for ranges; + * neither expectation uses the production comparator or traversal. Fixed + * histories keep three undefined owners, a remove-one-of-two cut, a mixed + * four-row range, and explicit versus omitted lower bounds. The generated + * tail varies owners and values at each returned-action cut. Removing the + * nullish values loses the boundary distinction, one owner loses repeated + * bucket traversal, and one defined value loses cursor continuation. Values + * outside the positive 1..3 domain, mixed-type numeric/string order, custom options, + * reverse ranges, all bound combinations, filtered takes, bucket object + * identity, and instrumentation are outside this ordered and range claim. */ type NullishIndexValue = number | null | undefined type NullishIndexAction = @@ -563,6 +577,131 @@ function expectNullishMembership( rows: ReadonlyMap, ): void { expectExactMembership(index, rows, nullishProbeValues, nullishInProbes) + expectNullishOrderedReads(index, rows) +} + +type ModelBound = Readonly<{ + value: NullishIndexValue + inclusive: boolean +}> +type ModelRange = Readonly<{ + lower?: ModelBound + upper?: ModelBound + unusedLowerInclusive?: false +}> + +const nullishRangeProbes: ReadonlyArray = [ + {}, + { upper: { value: 2, inclusive: true } }, + { upper: { value: 2, inclusive: true }, unusedLowerInclusive: false }, + { + lower: { value: undefined, inclusive: true }, + upper: { value: 2, inclusive: true }, + }, + { + lower: { value: undefined, inclusive: false }, + upper: { value: 2, inclusive: true }, + }, + { + lower: { value: undefined, inclusive: true }, + upper: { value: undefined, inclusive: true }, + }, + { + lower: { value: null, inclusive: true }, + upper: { value: null, inclusive: true }, + }, + { lower: { value: null, inclusive: false } }, + { upper: { value: undefined, inclusive: false } }, + { + lower: { value: 1, inclusive: true }, + upper: { value: 2, inclusive: false }, + }, +] + +// "Position" is a model label, not a stored index bucket. In the declared +// domain 1..3, zero names the shared nullish comparator position. Source key +// ownership remains in the Map even when two values share that position. +function nullishPosition(value: NullishIndexValue): number { + return value === null || value === undefined ? 0 : value +} + +function expectNullishOrderedReads( + index: BaseIndex, + rows: ReadonlyMap, +): void { + const ordered = [...rows] + .sort( + ([leftKey, leftValue], [rightKey, rightValue]) => + nullishPosition(leftValue) - nullishPosition(rightValue) || + (leftKey < rightKey ? -1 : leftKey > rightKey ? 1 : 0), + ) + .map(([key, value]) => ({ key, value })) + + // The public arrays preserve both ordering and multiplicity. A set-only + // check would miss a repeated undefined owner or a skipped peer. + for (const n of [1, 3, 10]) { + expect(index.takeFromStart(n)).toEqual( + ordered.slice(0, n).map(({ key }) => key), + ) + expect(index.takeReversedFromEnd(n)).toEqual( + [...ordered] + .reverse() + .slice(0, n) + .map(({ key }) => key), + ) + for (const cursor of nullishProbeValues) { + const cursorPosition = nullishPosition(cursor) + expect(index.take(n, cursor)).toEqual( + ordered + .filter(({ value }) => nullishPosition(value) > cursorPosition) + .slice(0, n) + .map(({ key }) => key), + ) + expect(index.takeReversed(n, cursor)).toEqual( + [...ordered] + .reverse() + .filter(({ value }) => nullishPosition(value) < cursorPosition) + .slice(0, n) + .map(({ key }) => key), + ) + } + } + + // Model bounds are separate objects, so an absent lower bound is not + // conflated with a present bound whose value is literally undefined. + for (const probe of nullishRangeProbes) { + const options = { + ...(probe.lower && { + from: probe.lower.value, + fromInclusive: probe.lower.inclusive, + }), + ...(probe.upper && { + to: probe.upper.value, + toInclusive: probe.upper.inclusive, + }), + ...(probe.unusedLowerInclusive === false && { fromInclusive: false }), + } + const expected = new Set( + ordered + .filter(({ value }) => { + const position = nullishPosition(value) + const lower = probe.lower + const upper = probe.upper + return ( + (lower === undefined || + (lower.inclusive + ? position >= nullishPosition(lower.value) + : position > nullishPosition(lower.value))) && + (upper === undefined || + (upper.inclusive + ? position <= nullishPosition(upper.value) + : position < nullishPosition(upper.value))) + ) + }) + .map(({ key }) => key), + ) + expect(index.rangeQuery(options)).toEqual(expected) + } } function applyNullishAction( @@ -590,7 +729,7 @@ function applyNullishAction( } describe.each(indexTypes)( - `%s nullish exact membership`, + `%s nullish direct index refinement`, (_indexName, IndexType) => { test.each([ [`undefined to null`, undefined, null], @@ -666,6 +805,73 @@ describe.each(indexTypes)( } }) + test(`distinguishes endpoint reads from explicit undefined cursors`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + for (const [key, value] of [ + [`a`, undefined], + [`b`, 1], + [`c`, 2], + ] as const) { + applyNullishAction(index, rows, { type: `put`, key, value }) + } + }) + + test(`traverses every undefined owner before continuing to numbers`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + for (const [key, value] of [ + [`undef1`, undefined], + [`undef2`, undefined], + [`undef3`, undefined], + [`one`, 1], + [`two`, 2], + ] as const) { + applyNullishAction(index, rows, { type: `put`, key, value }) + } + }) + + test(`retains the second undefined owner after removing the first`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + applyNullishAction(index, rows, { + type: `put`, + key: `a`, + value: undefined, + }) + applyNullishAction(index, rows, { + type: `put`, + key: `b`, + value: undefined, + }) + applyNullishAction(index, rows, { type: `delete`, key: `a` }) + }) + + test(`distinguishes explicit and omitted undefined lower bounds`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + for (const [key, value] of [ + [`undef`, undefined], + [`one`, 1], + [`two`, 2], + [`three`, 3], + ] as const) { + applyNullishAction(index, rows, { type: `put`, key, value }) + } + }) + + test(`includes both undefined owners within double undefined bounds`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + for (const [key, value] of [ + [`a`, undefined], + [`b`, undefined], + [`c`, 1], + ] as const) { + applyNullishAction(index, rows, { type: `put`, key, value }) + } + }) + test.each([20261012, undefined])( `refines nullish membership histories (seed %s)`, async (seed) => { @@ -855,6 +1061,158 @@ describe.each(indexTypes)( firstOnly.mockRestore() } }) + + test(`rejects an endpoint read that drops the undefined owner`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map([ + [`a`, undefined], + [`b`, 1], + [`c`, 2], + ]) + for (const [key, value] of rows) index.add(key, { value }) + const dropped = vi + .spyOn(index, `takeFromStart`) + .mockImplementation(() => [`b`, `c`]) + try { + expect(() => expectNullishOrderedReads(index, rows)).toThrowError( + /expected/, + ) + expect(dropped).toHaveBeenCalled() + } finally { + dropped.mockRestore() + } + }) + + test(`rejects a cursor that repeats the last undefined owner`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map([ + [`undef1`, undefined], + [`undef2`, undefined], + [`undef3`, undefined], + [`one`, 1], + [`two`, 2], + ]) + for (const [key, value] of rows) index.add(key, { value }) + const take = index.take.bind(index) + const repeated = vi + .spyOn(index, `take`) + .mockImplementation((n, from, filter) => + n === 10 && from === undefined + ? [`undef3`, `one`, `two`] + : take(n, from, filter), + ) + try { + expect(() => expectNullishOrderedReads(index, rows)).toThrowError( + /expected/, + ) + expect(repeated).toHaveBeenCalledWith(10, undefined) + } finally { + repeated.mockRestore() + } + }) + + test(`rejects a forward range that drops an explicit undefined lower bound`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map([ + [`undef`, undefined], + [`one`, 1], + [`two`, 2], + [`three`, 3], + ]) + for (const [key, value] of rows) index.add(key, { value }) + const rangeQuery = index.rangeQuery.bind(index) + const dropped = vi + .spyOn(index, `rangeQuery`) + .mockImplementation((options) => + `from` in options && + options.from === undefined && + options.to === 2 && + options.fromInclusive === true + ? new Set([`one`, `two`]) + : rangeQuery(options), + ) + try { + expect(() => expectNullishOrderedReads(index, rows)).toThrowError( + /expected/, + ) + expect(dropped).toHaveBeenCalledWith({ + from: undefined, + fromInclusive: true, + to: 2, + toInclusive: true, + }) + } finally { + dropped.mockRestore() + } + }) + + test(`rejects a double undefined range that loses one owner`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map([ + [`a`, undefined], + [`b`, undefined], + [`c`, 1], + ]) + for (const [key, value] of rows) index.add(key, { value }) + const rangeQuery = index.rangeQuery.bind(index) + const truncated = vi + .spyOn(index, `rangeQuery`) + .mockImplementation((options) => + `from` in options && + options.from === undefined && + `to` in options && + options.to === undefined + ? new Set([`a`]) + : rangeQuery(options), + ) + try { + expect(() => expectNullishOrderedReads(index, rows)).toThrowError( + /expected/, + ) + expect(truncated).toHaveBeenCalledWith({ + from: undefined, + fromInclusive: true, + to: undefined, + toInclusive: true, + }) + } finally { + truncated.mockRestore() + } + }) + + test(`rejects a null bound that drops its undefined peer`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map([ + [`a`, undefined], + [`b`, null], + [`c`, 1], + ]) + for (const [key, value] of rows) index.add(key, { value }) + const rangeQuery = index.rangeQuery.bind(index) + const exactOnly = vi + .spyOn(index, `rangeQuery`) + .mockImplementation((options) => + `from` in options && + options.from === null && + `to` in options && + options.to === null + ? new Set([`b`]) + : rangeQuery(options), + ) + try { + expect(() => expectNullishOrderedReads(index, rows)).toThrowError( + /expected/, + ) + expect(exactOnly).toHaveBeenCalledWith({ + from: null, + fromInclusive: true, + to: null, + toInclusive: true, + }) + } finally { + exactOnly.mockRestore() + } + }) }, ) From 175adce0cb32e29e5838853c821ee5c56fe0089a Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 16:06:32 -0600 Subject: [PATCH 07/46] docs: record exact-head index oracle review --- .../oracle-reviews/index-unit-retirement-round-4.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/contributing/oracle-reviews/index-unit-retirement-round-4.md b/docs/contributing/oracle-reviews/index-unit-retirement-round-4.md index 46daca7c68..114c47c58e 100644 --- a/docs/contributing/oracle-reviews/index-unit-retirement-round-4.md +++ b/docs/contributing/oracle-reviews/index-unit-retirement-round-4.md @@ -99,7 +99,7 @@ remain in the source file. | ORC-009 | A model `put` maps to add or update according to source ownership; `delete` maps to remove when owned. Model comparator position combines null and undefined only for order/range, while exact membership keeps them distinct. | | ORC-010 | Spies restore in `finally`; the guarded runner recorded the primary mismatch separately from process exit and removed its private witness channel. | | ORC-011 | The reviewer found a missing probe, not a fault shared by the model and production. The added null-bound probe distinguishes the model rule from an exact-bucket interpretation. | -| ORC-012 | This versioned ledger holds source assertions, transfer, limits, execution, and the hostile verdict. Exact committed-head recheck is recorded in a follow-up entry below. | +| ORC-012 | This versioned ledger holds source assertions, transfer, limits, execution, hostile verdict, and the exact committed-head recheck below. | | ORC-013 | Explicit inclusive versus exclusive undefined and null bounds, paired with omitted bounds, distinguish presence and comparator-position grouping. The wrong-result controls fail at those comparisons. | | ORC-014 | No host or external provider behavior is claimed. | @@ -108,3 +108,13 @@ crosses the Collection/query path. The other four retained tests observe private BTree ordering or storage and custom-comparator arguments; direct source-Map arrays and sets cannot replace those observations without a separate contract decision and receiving witness. + +## Exact committed-head recheck + +The same hostile reviewer rechecked committed code head +`a72a7043d5750b33100ab95b7d3c6a490b15626c` after the commit hook. The +commit deletes exactly the seven reviewed blocks and keeps all five intended +legacy tests. The committed oracle retains the Map comparisons, null-bound +probes, and five wrong-result controls; no temporary calibration fault remains. +The two targeted files passed 415/415 tests with no type errors, and +`git show --check --oneline HEAD` passed. The reviewer found no new blocker. From 9a145fba130f27c914a4a1fd4f44c5871a917483 Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 17:23:15 -0600 Subject: [PATCH 08/46] test(db): transfer comparison laws and fix oracle-found bugs --- .../fix-index-build-cross-realm-dates.md | 5 + docs/contributing/oracle-coverage.md | 31 +- .../basic-index-nullish-build.md | 59 +++ .../comparison-unit-retirement-round-5.md | 79 ++++ .../oracle-reviews/cross-realm-date-order.md | 95 +++++ .../cross-realm-date-where-publication.md | 88 ++++ packages/db/src/indexes/basic-index.ts | 7 +- packages/db/src/utils/comparison.ts | 45 +- .../tests/comparison-oracle.property.test.ts | 392 +++++++++++++++++- packages/db/tests/comparison.test.ts | 44 +- .../index-update-oracle.property.test.ts | 68 ++- packages/db/tests/query/subset-dedupe.test.ts | 13 +- ...dicate-publication-oracle.property.test.ts | 211 ++++++++++ 13 files changed, 1070 insertions(+), 67 deletions(-) create mode 100644 .changeset/fix-index-build-cross-realm-dates.md create mode 100644 docs/contributing/oracle-reviews/basic-index-nullish-build.md create mode 100644 docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md create mode 100644 docs/contributing/oracle-reviews/cross-realm-date-order.md create mode 100644 docs/contributing/oracle-reviews/cross-realm-date-where-publication.md diff --git a/.changeset/fix-index-build-cross-realm-dates.md b/.changeset/fix-index-build-cross-realm-dates.md new file mode 100644 index 0000000000..0fc2be832c --- /dev/null +++ b/.changeset/fix-index-build-cross-realm-dates.md @@ -0,0 +1,5 @@ +--- +'@tanstack/db': patch +--- + +Keep `undefined` values in nulls-first order when BasicIndex builds from existing rows. Compare native Dates from other JavaScript realms by timestamp in ordering and equality, including invalid Dates. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 52e7915fbb..6e5e460c1d 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -459,6 +459,31 @@ generic visitor behavior outside the named collectors. | Boundary refinements | [cleanup/restart](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-cleanup-restart-oracle.test.ts), [idle cleanup](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-idle-cleanup.test.ts), [issue #1891 async-cleanup review](oracle-reviews/issue-1891-async-cleanup.md), [issue #1891 cleanup-start follow-up](oracle-reviews/issue-1891-cleanup-start-followup.md), [metadata publication](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-metadata-publication-oracle.property.test.ts), [row metadata composition](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-row-metadata-composition-oracle.test.ts), [nested-begin metadata rebuild review](oracle-reviews/2026-10-06-nested-begin-metadata-rebuild.md), [state mutation round 4](oracle-reviews/2026-10-06-state-mutation-round-4.md), [state retention](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-state-retention-oracle.property.test.ts), [acquisition cells](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-subscription-lifecycle-oracle.test.ts), [D2 source reconciliation](https://github.com/TanStack/db/blob/main/packages/db/tests/d2-source-reconciliation-oracle.property.test.ts), [top-K support windows](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/operators/topk-support-window-oracle.test.ts), [nested Query work](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/includes-work-counter-oracle.test.ts) | Explicit lifecycle products, independent source maps and weighted relations, exact publication cuts, support/multiplicity, and value-plus-work observations. These refine the larger subsystem models; they do not replace them. | | Small structures and test mechanics | [SortedMap](https://github.com/TanStack/db/blob/main/packages/db/tests/SortedMap-oracle.test.ts), [cleanup queue](https://github.com/TanStack/db/blob/main/packages/db/tests/cleanup-queue-oracle.property.test.ts), [live-query GC clock](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-gc-clock.test.ts), [guarded replay](https://github.com/TanStack/db/blob/main/packages/db/tests/oracle-replay.test.ts) | Map/full-sort and elapsed appointment-list models with executed target/seed/path checks. SortedMap also checks deferred bulk writes, array-movement work, scan reuse, live iterators, runtime `undefined` keys, and public single-transaction and queued-transaction Collection sync paths. It does not settle the contract for comparator-observed values mutated after publication or every legal iterator/mutation interleaving. Wall-clock steps do not move due times, and a focused public witness checks source subscription release. Callback-reentrant scheduling, OS suspend/resume, environments without `performance.now()`, and fake/real Performance clock replacement while GC is pending remain outside this coverage. Tests must drain or reset pending GC before switching timer providers. | +The [comparison oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts) +now checks default finite/non-finite order for number-only and Date-only +histories with optional null, including exact comparator zeros and stable +sorted row IDs. Its [transfer ledger](oracle-reviews/comparison-unit-retirement-round-5.md) +records the five retired same-realm examples. A separate +[cross-realm Date witness](oracle-reviews/cross-realm-date-order.md) compares +native `node:vm` Dates by supplied timestamps and invalidity, including equal +times and numeric NaN with an invalid Date. It also checks one unindexed +public ordered scan. Exact `normalizeValue` results are direct utility +observations. Indexed order, public Date range predicates, cursor +continuation, proxies, and other Date-consuming paths remain outside that +witness. + +The [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts) +also has a [cross-realm Date equality matrix](oracle-reviews/cross-realm-date-where-publication.md). +For local and `node:vm` Dates with timestamps 1 and 2 or invalid timestamps, +plus number 1 and NaN, it compares public live-query keys, filtered subscriber +keys, and direct snapshot keys after preload and a three-row sync transaction. +Auto-indexing is off in both modes; one mode adds BasicIndex explicitly and +also checks its equality lookup and an optimized-only public snapshot. This +proves a public indexed snapshot result but does not prove that the live-query +planner chose the index. Other realm hosts, +proxies, subclasses, public range predicates, and arbitrary histories need +separate witnesses. + The direct [index refinement oracle](../../packages/db/tests/index-update-oracle.property.test.ts) also owns exact nullish membership and direct `IN` set selection for BasicIndex and BTreeIndex. Fresh single-key updates cross `undefined→null`, @@ -469,13 +494,17 @@ values, an all-defined nonempty index, a mixed `[undefined, 1]` list with two defined distractors, and a singleton `[undefined]` list with two owners. A plain source Map supplies expected `equalityLookup`, `lookup('eq')`, `inArrayLookup`, and `lookup('in')` sets after returned actions. Fixed and random -three-key put/delete histories exercise nearby states with empty, repeated, +three-key put/delete/build histories exercise nearby states with empty, repeated, both orders of a mixed list, and nullish-only `IN` lists. The same source Map now predicts unfiltered ordered arrays and selected forward range sets over `undefined`, `null`, and positive numbers. Fixed histories preserve endpoint versus explicit undefined cursors, three undefined owners, removal of one of two owners, explicit versus omitted lower bounds, and two owners between double undefined bounds. The generated histories compare after each returned direct action. +Fixed build histories use both `undefined`/`1` insertion orders and a larger +`undefined`/`null`/defined source; the generated build action replaces source +ownership. The [BasicIndex build bug record](oracle-reviews/basic-index-nullish-build.md) +contains the original `takeFromStart(1)` failure and repair evidence. Skipped-update, wrong equality, lost-owner, dropped-operand, wrong-dispatch, wrong-cursor, and wrong-range controls fail the check. This direct-index evidence does not establish mixed-type ordering, custom comparator behavior, diff --git a/docs/contributing/oracle-reviews/basic-index-nullish-build.md b/docs/contributing/oracle-reviews/basic-index-nullish-build.md new file mode 100644 index 0000000000..21efc48d10 --- /dev/null +++ b/docs/contributing/oracle-reviews/basic-index-nullish-build.md @@ -0,0 +1,59 @@ +# BasicIndex build with a nullish value + +At PR head `175adce0cb32e29e5838853c821ee5c56fe0089a`, a focused runtime +witness built a direct BasicIndex from `undefined` and `1`. The default +comparator places `undefined` first. `BasicIndex.build` returned +`takeFromStart(2) = [one, undefined]` and an empty inclusive upper range +bounded by `undefined`. Incremental `BasicIndex.add` and both BTreeIndex paths +returned `[undefined, one]` and the expected range. A recording comparator +received no calls during the BasicIndex build. The build path is also reached +when `Collection.createIndex` builds an index over existing rows. + +The accepted law is the default ascending, nulls-first order documented by +`DEFAULT_COMPARE_OPTIONS`, `defaultComparator`, and the direct `IndexInterface` +contract. The primary `index-update-oracle.property.test.ts` already models +nullish positions with a source Map, but its nullish history grammar previously +used add/update/remove only; its generated build grammar used numeric values. +The smallest missing cell is `undefined` plus `1` through `BasicIndex.build`, +observed with `takeFromStart(1)` and an explicit upper range. A two-order +fixed build history and generated nullish builds now use the same Map checker +after each returned action. The Map distinguishes exact null and undefined +membership while its ordered projection treats them as one comparator +position. Raw duplicate source keys are resolved before build; mixed value +classes and custom comparators remain outside this claim. + +## RED evidence before the production change + +The new fixed oracle witness failed on the unchanged implementation: +`BasicIndex nullish direct index refinement > build preserves nulls-first +ordered reads and ranges` compared `takeFromStart(1)`, receiving `[one]` +against source-model `[undefined]` at the intended observation checkpoint. +The BTreeIndex version passed. The command's focused run had one failed test, +one passed test, and no type errors. This is a product mismatch, not a setup +failure or a changed expectation. + +Native `Array.sort` moved a raw `undefined` element to the end without invoking +the supplied comparator. `BasicIndex.build` now sorts wrapper records, so every +stored value reaches the configured comparator, then unwraps the sorted values. +The fixed two-order build history and generated put/delete/build histories +pass through the same independent Map checker after each returned action. + +## GREEN evidence and hostile review + +The complete direct index oracle passed 412 tests with no type errors after the +repair. The fixed history includes both source insertion orders and a later +build with `undefined`, `null`, and a defined value. The generated grammar can +replace an existing source Map through build. The fresh hostile reviewer +`/root/review_nullish_build_oracle` inspected the original RED witness, the +model, both index implementations, and the repair. It found no blocker for the +bounded direct-index law. It did not claim that a Collection query using a +newly built index has been observed publicly. + +The oracle's model derives exact membership and sorted source keys from the +source Map, not from either index. A build that leaves `undefined` last fails +`takeFromStart(1)` at the returned-action checkpoint. Range and cursor probes +also compare against the Map after each action. The direct-index law excludes +raw duplicate-entry builds, mixed string/number order, custom comparators, +Collection queries, and work complexity. Its fixed and generated build cells +do not establish every possible source history. The exact committed-head +review is recorded in a later entry. diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md new file mode 100644 index 0000000000..f81d9f41b2 --- /dev/null +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md @@ -0,0 +1,79 @@ +# Comparison unit retirement: non-finite default order + +The source-isolated Field Lab loss audit compared `comparison.test.ts` with +`comparison-oracle.property.test.ts` at frozen head +`f6857fee9de59dc82cbe596ca7df4a18152004ed`. The receiving grammar used +finite numbers and valid Dates; it had no NaN input, invalid Date input, or +`defaultComparator` sort observation. This ledger freezes the five relevant +legacy blocks before retirement. The other five tests in that file have +separate Temporal, symbol, and `compareValues` obligations and remain. + +| Legacy block | Source assertion at the function-return or sort checkpoint | Loss in frozen receiver | Receiving check | +| --- | --- | --- | --- | +| `orders NaN greater than every number` | `ascComparator(NaN, 5)` is positive and the reverse negative under `DEFAULT_COMPARE_OPTIONS`. | NaN input excluded. | Tagged numeric finite/non-finite pairs compare signs in both directions. | +| `treats NaN as equal to NaN` | Direct comparator returns zero. | Reflexivity grammar excludes NaN. | Duplicate tagged NaNs compare as one order position. | +| `produces a stable total order with NaN sorting last` | Sorting `[3, NaN, 1, 5, NaN]` with `defaultComparator` yields finite prefix `[1,3,5]` and two NaNs at the end. | No default-sort production path. | The fixed tagged history reproduces these values; a stable source-rank sort predicts every output row ID. | +| `keeps null before non-null values regardless of NaN` | Sorting `[5, NaN, null, 1]` yields null, 1, 5, NaN. | Null and finite signs were separate from NaN/default-sort behavior. | The fixed tagged history combines null and NaN, and the generated grammar varies null presence. | +| `orders an invalid Date greater than valid Dates` | Direct comparator orders invalid Date after `2023-01-01` in both directions. | Date grammar excludes invalid Dates. | A fixed date history uses that day; generated valid epoch-day Dates and invalid Dates compare both ways and in sorted arrays. | + +The changelog entry for #1582 promises PostgreSQL-style NaN ordering and +applies it to invalid Dates, with null placement independent of it. The +`DEFAULT_COMPARE_OPTIONS` constant and `defaultComparator` document the +default ascending, nulls-first path. The model receives explicit tags for +null, finite, and non-finite input values and compares rank and finite epoch +day; it does not call the production unorderable classifier to predict a sign. +Direct `ascComparator` returns and `defaultComparator` sort row IDs are the +observations. The comparison occurs at each function return and after sorting. + +The bounded grammar uses 1–5 distinct finite numbers or valid Dates in +epoch-day range -20,000..20,000, one or two NaNs/invalid Dates, and optional +null. Fixed histories preserve the old examples. Mixed number/Date order, +Temporal values, symbols, other option sets, and public query/index paths are +outside this transfer. A separate direct `compareValues` behavior remains in +the old file because it has a different production entry point and fallback +law. + +## Review and execution + +The five listed blocks were removed. The receiving oracle and remaining legacy +file passed with 102 tests (including one Vitest-selected integration file), +and Vitest reported no type errors. Both the fixed seed and seedless campaigns +run the same 100-case tagged property. Its direct replay at property 23, +seed 20260911, path 0 passed. The direct comparator control ties a +non-finite value to a finite value and fails the sign comparison in both +number and Date families. A separate wrong-sort control ties NaN during +`defaultComparator` sort and fails the exact output-row-ID comparison. + +The fresh hostile reviewer `/root/review_nonfinite_round5` read the complete +source file, ledger, owner, production comparator, public authority, and +coverage map. It found one exact-assertion loss: Vitest `toBe(0)` on NaN/NaN +rejects negative zero, while the initial sign-only checker did not. The +revised oracle checks the raw result with `toBe(0)` for every non-finite pair; +the reviewer rechecked that change and the separate sort control and found no +remaining loss from the five retired blocks. Its challenge of the broader +Date law found a separate public cross-realm bug. That has its own RED witness +and review record in `cross-realm-date-order.md`; it does not make the retired +same-realm examples unique. + +## Oracle guide audit + +| Requirement | Evidence and limit | +| --- | --- | +| ORC-001 | Changelog #1582 and the live-query guide promise NaN/invalid-Date order; `DEFAULT_COMPARE_OPTIONS` and `defaultComparator` establish the tested default path. This transfer claims number-only and Date-only same-realm sorts with optional null. | +| ORC-002 | Input tags and supplied finite values determine expected signs and sort ranks. The model never imports production classification or comparator output. | +| ORC-003 | Opening and adjacent prose identify the law, tagged model, bounded inputs, comparator and sort drivers, result/sign and ordered-ID observations, and immediate checkpoints. | +| ORC-004 | Fixed inputs reconstruct all five source blocks. Generated families, finite values, duplicate non-finite values, and optional null each distinguish a missing behavior. The stated range is -20,000..20,000 finite epoch days, 1–5 distinct finite inputs, and 1–2 non-finite values. Mixed types and unrelated value classes are excluded. | +| ORC-005 | The production `ascComparator` receives `DEFAULT_COMPARE_OPTIONS`; `defaultComparator` drives an actual stable JavaScript sort. Pair signs, exact zero, and every sorted row ID are compared at return. | +| ORC-006 | Separate wrong-result controls reach and fail the direct-sign and sorted-ID comparisons. | +| ORC-007 | Fixed-seed and seedless lanes use the same property and 100-run budget through the existing direct seed/path replay harness. Property 23 replayed at seed 20260911, path 0. | +| ORC-008 | Not applicable: there is no stateful model. | +| ORC-009 | `category` and `order` are model-only tags, not production storage states. | +| ORC-010 | Existing comparison harness preserves primary checkpoint through shrinking and direct replay; this stateless check owns no cleanup. | +| ORC-011 | No shared production/model semantic fault was named within this bounded same-realm transfer. The cross-realm hostile finding has a separate public-query formulation. | +| ORC-012 | This versioned record contains the source ledger, execution, hostile verdict, limits, and the exact-head recheck below. | +| ORC-013 | Finite versus non-finite and null versus non-finite comparisons, plus the wrong-sort control, distinguish a tie-all or misplaced-nulls rule within the bounded default path. | +| ORC-014 | Not applicable: the direct comparator and JavaScript sort run in the package test host; no controlled external provider is extrapolated. | + +## Exact committed-head recheck + +Pending commit and fresh recheck. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-order.md b/docs/contributing/oracle-reviews/cross-realm-date-order.md new file mode 100644 index 0000000000..39ccf25e4e --- /dev/null +++ b/docs/contributing/oracle-reviews/cross-realm-date-order.md @@ -0,0 +1,95 @@ +# Date order across JavaScript realms + +The comparison loss audit led to a same-realm NaN/invalid-Date transfer. A +fresh hostile review then challenged the whole published Date ordering law, +including native Dates created in another JavaScript realm. The live-query +guide's comparison section and changelog #1582 promise that invalid Dates act +like NaN in ordering and equality, without a realm exception. A valid Date +orders by timestamp. This is a product-law question beyond the five retired +same-realm examples, so it has its own witness and review. + +At unchanged production head `175adce0cb32e29e5838853c821ee5c56fe0089a`, +`ascComparator(foreign Jan 2, local Jan 1)` returned a negative sign rather +than positive. The new tagged Date oracle constructed local and `node:vm` +Dates from model timestamps and compared both operand orders, then checked +`normalizeValue` against the supplied timestamp. The fixed valid-Date case +failed at the direct comparator sign. The fixed-seed generated campaign failed +normalization at seed `20260911`, path `0:0:1` and directly reproduced that +checkpoint; the seedless campaign likewise failed and replayed. These are +product mismatches, not setup or cleanup failures. + +A separate public ordered-scan witness inserted an invalid Date before a valid +Date in a Collection with `autoIndex: 'off'`. The local invalid-Date control +returned IDs `[2,1]`; the foreign invalid-Date case returned `[1,2]`, failing +the same exact public-row comparison. The source Collection and live query +cleaned up in `finally`. + +The model uses `null` only as an invalid-Date *tag*: production receives a +native Date with a NaN timestamp. Valid tags range from -1e12 to +1e12 epoch +milliseconds. Both local and foreign constructors, both argument positions, +equal times, and invalid values are legal. A fixed cross-family case also +checks that numeric NaN and either local or foreign invalid Date compare at +exactly one order position. A forged `[Symbol.toStringTag] = 'Date'` object is +a negative control: neither the classifier nor the normalizer accepts it as a +Date, and the comparator does not tie it to a Date. Proxies, subclasses, +other value classes, cursor pagination, and other realm mechanisms remain +outside the bounded grammar. + +## Repair, GREEN evidence, and hostile review + +`dateTimestamp` now recognizes a native Date created in another realm and +reads its timestamp with `Date.prototype.getTime.call`. A string tag alone +cannot establish the native Date brand: a forged tag must fail that intrinsic +call. Local Dates use the same intrinsic, so an own `getTime` method cannot +silently replace the timestamp. `ascComparator` classifies each operand once, +uses its timestamp for valid Date order, and treats an invalid Date like NaN. +`isUnorderable` and `normalizeValue` use the same timestamp extraction. + +After the repair, the comparison oracle, remaining comparison unit tests, +direct index oracle, and selected integration test passed 514 tests with no +type errors. The direct Date property ran with the fixed seed and the ordinary +seedless campaign. Its equal-timestamp branch forces ties independently of a +random collision. The new classifier assertions, both comparator directions, +and exact normalization diagnostics compare against supplied model tags. The +public scan checks complete ordered IDs after `preload()`; the local control +ensures this query path actually sorts before the foreign case is judged. + +The fresh hostile reviewer `/root/review_realm_date_oracle` followed the law +to the live-query guide and changelog #1582, inspected the unchanged-head RED +results, challenged model independence and a forged Date tag, and found no +blocker for this bounded repair. It identified the mixed NaN/invalid-Date +gap; the exact cross-family control now covers that comparison. It also found +that repeated Date classification could reread an unrelated object's tag; +the comparator now extracts each timestamp once per operand. A final +committed-head review follows below. + +## Oracle guide audit and limits + +The guide's law, authority, and scope requirements (ORC-001, -003, -011) +are stated in the oracle opening and beside this tagged Date model. Supplied +timestamps and invalidity tags predict order without production classification +(ORC-002, -009). Fixed foreign-valid, foreign-invalid, equal-time, mixed-NaN, +and forged-tag controls complement generated realm/time pairs (ORC-004, -013). +Direct classifier, comparator, and normalization checks compare at function +return; a public ordered scan compares after preload (ORC-005). The original +production fails both a direct and public-row observation, supplying an actual +wrong implementation rather than only a hypothetical mutant (ORC-006). +The shared property harness runs fixed-seed and seedless campaigns and supports +seed/path replay (ORC-007). The scan cleans up its source and live-query +Collections in `finally` (ORC-010). There is no stateful model or external +provider in this lane (ORC-008, -014). This record preserves the exact RED +head, reviewer verdict, and bounded GREEN result (ORC-012). + +`normalizeValue` is an internal index-key diagnostic, not public equality +evidence. The separate +[WHERE publication witness](cross-realm-date-where-publication.md) checks +public Date equality and explicit BasicIndex equality lookup. This record's +public scan uses `autoIndex: 'off'`; indexed order, range WHERE paths, +cursor/provider continuation, and other Date-consuming paths need separate +witnesses before they enter this closure claim. The subsystem has +realm-sensitive `instanceof Date` checks outside the three repaired utilities. +This record does not infer their correctness from the unindexed scan. + +## Exact committed-head recheck + +Pending commit and fresh reviewer recheck. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md new file mode 100644 index 0000000000..c3cf331b2a --- /dev/null +++ b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md @@ -0,0 +1,88 @@ +# Cross-realm Date equality at WHERE publication + +The [comparison transfer ledger](comparison-unit-retirement-round-5.md) +retired five same-realm NaN and invalid-Date examples. A hostile review of +their receiving law found a different defect: a native Date created in another +JavaScript realm was not normalized to its timestamp. The +[ordering record](cross-realm-date-order.md) preserves direct and public scan +RED witnesses. This record tests the separate equality and publication path. +It does not retire another unit test. + +The live-query guide's comparison semantics and changelog #1582 say that +valid Dates compare by timestamp and that invalid Dates share NaN's equality +class. The WHERE publication owner already models these classes for local +values. Its new focused matrix tags each source value and literal as a local +or `node:vm` Date with timestamp 1, timestamp 2, or invalid timestamp, or as +number 1 or NaN. The model reduces tags to equality classes before production +values exist. It predicts complete, sorted public key sets, independent of +`normalizeValue`, the evaluator, or either index implementation. + +The driver starts a source Collection with auto-indexing off, then exercises +both no-index and explicit BasicIndex modes. At each mode and literal it +compares public live-query Collection keys, filtered source subscriber keys, +and direct Collection snapshot keys after preload. The indexed mode also +compares `BasicIndex.lookup('eq')` and a public snapshot with +`optimizedOnly: true`; that snapshot cannot silently use the scan fallback. +One sync transaction moves three source rows across equality classes, and the +same checks run after commit. The test does not prove that the public live +query planner selected the index. The source and live-query Collections and +subscription are released through the owner's cleanup helper. + +## RED and GREEN evidence + +An isolated archive of unchanged head +`175adce0cb32e29e5838853c821ee5c56fe0089a` ran the new test without +swapping production files in the shared worktree. It failed at the intended +unindexed public live-query comparison: `eq(v, local Date(1))` expected keys +`[foreign-one, local-one, number-one]` and returned +`[local-one, number-one]`. The cross-realm premise itself passed: the foreign +value failed local `instanceof Date` but was accepted by the native Date +intrinsic. This is a product result mismatch at a public checkpoint. + +With the comparison utility repair, the complete WHERE publication oracle +passed 18 tests with no type errors. The DB oracle suite passed 5,006 tests +across 73 files with no type errors. The scope is native local and `node:vm` +Dates, finite timestamps 1 and 2, invalid Dates and NaN, one three-row sync +transaction, and the named observations. Other realm hosts, proxies, +subclasses, spoofed tags, arbitrary timestamp histories, and planner proof of +live-query index selection remain outside this witness. + +The full DB suite initially reported two failures in the existing +`subset-dedupe.test.ts` Date transport test. It expected local `Date(2)` and +foreign `Date(2)` to match different `eq` and `in` requests. That assertion +encoded the old realm-local implementation, contrary to timestamp equality. +The test now expects both Dates to match for either literal realm and directly +checks that the request object is passed through unchanged. Its separate +Uint8Array realm observations remain untouched. The focused subset-dedupe and +WHERE files then passed 53 tests with no type errors; the full +`@tanstack/db` test command also passed. + +## Oracle guide audit and hostile review + +The opening prose and adjacent comments state the equality law, model, +bounded grammar, production driver, public observations, and two checkpoints +(ORC-001 through -005, -009). The original implementation fails a public +comparison at the intended cut; `optimizedOnly: true` is a positive indexed +snapshot reach witness, although its frozen-head RED was not separately run +(ORC-006). This is a fixed matrix rather than a generated campaign; all +declared cells run on every execution, so seed/path replay is inapplicable +(ORC-004, -007). The model's source rows change only after a committed +transaction, and all consumers compare with that same authored state +(ORC-008). The cleanup helper preserves the primary failure while releasing +consumers (ORC-010). The comparison transfer and ordering records identify +the original test gap and rival identity-versus-timestamp rule (ORC-011 +through -013). The sync source is controlled; the witness makes no provider +claim (ORC-014). + +The fresh hostile reviewer `/root/review_realm_date_where` inspected the +model, the original unit ledger, current production, public authority, and +both lanes. It found one reach gap: direct index lookup did not prove a public +read used the index. The added optimized-only public snapshot closes that +gap. Its final verdict found no blocker for the bounded matrix. The reviewer +also traced the full-suite failure above to the old test's realm-specific +assumption, not to a request-identity contract. Final key sets do not prove +subscriber event multiplicity, payloads, or intermediate publication. + +## Exact committed-head recheck + +Pending commit and fresh reviewer recheck. diff --git a/packages/db/src/indexes/basic-index.ts b/packages/db/src/indexes/basic-index.ts index 0e5c91eec6..8caa057a9e 100644 --- a/packages/db/src/indexes/basic-index.ts +++ b/packages/db/src/indexes/basic-index.ts @@ -205,8 +205,11 @@ export class BasicIndex< } } - // Build sorted array from unique values - this.sortedValues = Array.from(this.valueMap.keys()).sort(this.compareFn) + // Native Array.sort moves raw undefined values last without comparing them. + // Wrap values so nulls-first comparison also runs during a full build. + const sortable = Array.from(this.valueMap.keys(), (value) => ({ value })) + sortable.sort((left, right) => this.compareFn(left.value, right.value)) + this.sortedValues = sortable.map(({ value }) => value) } /** diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index eb6ac39dc3..5b59e41da8 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -19,6 +19,29 @@ function getObjectId(obj: object): number { return id } +// A native Date can arrive from another JavaScript realm, where `instanceof` +// fails. Check its brand before borrowing the built-in method so a forged +// Symbol.toStringTag cannot make an ordinary object look like a Date. +function dateTimestamp(value: unknown): number | undefined { + if (value instanceof Date) return Date.prototype.getTime.call(value) + if (value === null || typeof value !== `object`) return undefined + try { + if (Object.prototype.toString.call(value) === `[object Date]`) { + return Date.prototype.getTime.call(value) + } + } catch { + // A proxy or forged Date tag has no native Date timestamp. + } + return undefined +} + +function hasUnorderableValue( + value: unknown, + timestamp: number | undefined, +): boolean { + return Number.isNaN(value) || Number.isNaN(timestamp) +} + /** * Whether a value has no IEEE-754 natural order: `NaN`, or an invalid Date * (whose timestamp is `NaN`). The query engine follows PostgreSQL float @@ -27,11 +50,8 @@ function getObjectId(obj: object): number { * evaluator treat them explicitly instead of letting `NaN` compare unequal to * everything (which has no consistent order and cannot be indexed or sorted). */ -export function isUnorderable(value: any): boolean { - return ( - (typeof value === `number` && Number.isNaN(value)) || - (value instanceof Date && Number.isNaN(value.getTime())) - ) +export function isUnorderable(value: unknown): boolean { + return hasUnorderableValue(value, dateTimestamp(value)) } /** @@ -51,8 +71,10 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { // all equal and sort greater than every other non-null value. This keeps the // order total (NaN would otherwise compare equal to everything), so such // values can be sorted and stored in tree-based indexes. - const aUnordered = isUnorderable(a) - const bUnordered = isUnorderable(b) + const aDateTime = dateTimestamp(a) + const bDateTime = dateTimestamp(b) + const aUnordered = hasUnorderableValue(a, aDateTime) + const bUnordered = hasUnorderableValue(b, bDateTime) if (aUnordered && bUnordered) return 0 if (aUnordered) return 1 if (bUnordered) return -1 @@ -81,8 +103,8 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { } // If both are dates, compare them - if (a instanceof Date && b instanceof Date) { - return a.getTime() - b.getTime() + if (aDateTime !== undefined && bDateTime !== undefined) { + return aDateTime - bDateTime } // If both are Temporal objects, use compareTemporalValues for correct semantic ordering @@ -239,9 +261,8 @@ export function normalizeValue(value: any): any { return value } - if (value instanceof Date) { - return value.getTime() - } + const timestamp = dateTimestamp(value) + if (timestamp !== undefined) return timestamp if (isTemporal(value)) { return normalizedKey( diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index 55ad28c4d2..2649600814 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -1,19 +1,29 @@ +import { runInNewContext } from 'node:vm' import { describe, expect, it } from 'vitest' import { fc, test as fcTest } from '@fast-check/vitest' +import { createCollection } from '../src/collection/index' +import { createLiveQueryCollection } from '../src/query/live-query-collection' import { areValuesEqual, ascComparator, + defaultComparator, descComparator, + isUnorderable, makeComparator, normalizeValue, } from '../src/utils/comparison' +import { DEFAULT_COMPARE_OPTIONS } from '../src/utils' import type { CompareOptions } from '../src/query/builder/types' /** * Comparison defines the order and equality domains used by queries and * indexes. The laws are reflexivity, deterministic sign, antisymmetry, and * transitivity under one resolved option set. Null placement, direction, and - * lexical/locale string modes are independent axes. + * lexical/locale string modes are independent axes. In the bounded default + * ascending domain below, NaN and invalid Dates share one non-finite order + * position and follow finite values; null remains first. The published NaN + * ordering change in `packages/db/CHANGELOG.md` (#1582) and the live-query + * guide establish that law for Dates regardless of their JavaScript realm. * * Authority: `OrderByOptions` and `CompareOptions` in the query builder's * option types declare these modes. The index comparator contract accepts @@ -21,7 +31,12 @@ import type { CompareOptions } from '../src/query/builder/types' * * The model uses direct primitive, Date, array, and byte comparisons from the * declared domain. It excludes unrelated object identities because production - * intentionally assigns those a creation-order ID. Equality laws separately + * intentionally assigns those a creation-order ID. The non-finite model uses + * input tags and finite epoch-day values, rather than the production + * unorderable classifier. It covers number-only and Date-only sorts with + * optional null. A separate realm-tagged Date model below checks native Date + * timestamps and invalidity across realms, plus one public ordered scan. + * Neither lane claims mixed-type order or every query/index path. Equality laws separately * cover the normalized value classes and changed-byte controls. Comparator * signs reject NaN and non-number results before reduction. Signed infinity * is a valid comparator result, including for custom string collation. @@ -200,8 +215,9 @@ function comparisonProperty< // locale/lexical, and custom-collation cases reconstruct those boundaries. // Array length 0/1 and binary length 0/1/128/129 are marginal cases. Plain // objects are outside this order model because their IDs depend on creation -// order. Mixed-type order and NaN inputs are outside this bounded grammar; an -// invalid comparator result is rejected by the control below. +// order. Mixed-type order is outside this bounded grammar. A separate tagged +// grammar below admits NaN and invalid Date inputs; an invalid comparator +// result is rejected by the control below. const arbitraryComparablePrimitive = fc.oneof( fc.string(), fc.integer(), @@ -211,6 +227,181 @@ const arbitraryComparablePrimitive = fc.oneof( const arbitraryDate = fc.date({ noInvalidDate: true }) +type NonFiniteFamily = `number` | `date` +type NonFiniteInput = { + family: NonFiniteFamily + finiteDays: Array + unorderedCount: number + includeNull: boolean +} +type TaggedOrderValue = { + id: string + value: number | Date | null + category: `null` | `finite` | `unordered` + order: number +} + +// The generator's tags are reference facts: finite epoch days have ordinary +// numeric order, null precedes them, and the family's non-finite values follow +// them. Constructing expected signs from these tags avoids asking production +// whether a Date or number is unorderable. Distinct IDs preserve duplicate +// NaNs/invalid Dates in the sorted observation. +function taggedOrderValues(input: NonFiniteInput): Array { + const unordered = () => + input.family === `number` ? Number.NaN : new Date(Number.NaN) + const finite = input.finiteDays.map((day, index) => ({ + id: `finite:${index}`, + value: input.family === `number` ? day : new Date(day * 86_400_000), + category: `finite` as const, + order: day, + })) + const rows: Array = [ + finite[0]!, + { id: `unordered:0`, value: unordered(), category: `unordered`, order: 0 }, + ...finite.slice(1), + ...Array.from({ length: input.unorderedCount - 1 }, (_, index) => ({ + id: `unordered:${index + 1}`, + value: unordered(), + category: `unordered` as const, + order: 0, + })), + ] + if (input.includeNull) { + rows.splice(2, 0, { id: `null`, value: null, category: `null`, order: 0 }) + } + return rows +} + +function taggedOrderSign(a: TaggedOrderValue, b: TaggedOrderValue): -1 | 0 | 1 { + const rank = { null: 0, finite: 1, unordered: 2 } + const difference = rank[a.category] - rank[b.category] + const valueDifference = + difference || (a.category === `finite` ? a.order - b.order : 0) + return valueDifference < 0 ? -1 : valueDifference > 0 ? 1 : 0 +} + +// Production observes direct comparator returns and the order of a stable +// JavaScript sort using the exported default comparator. The comparison runs +// immediately after each call, so it detects wrong signs and lost or reordered +// duplicate non-finite rows at the exact sort checkpoint. +function assertNonFiniteOrder( + input: NonFiniteInput, + compare: (a: number | Date | null, b: number | Date | null) => number = ( + a, + b, + ) => ascComparator(a, b, DEFAULT_COMPARE_OPTIONS), + sortCompare: ( + a: number | Date | null, + b: number | Date | null, + ) => number = defaultComparator, +): void { + const rows = taggedOrderValues(input) + for (const a of rows) { + for (const b of rows) { + const result = compare(a.value, b.value) + expect(sign(result)).toBe(taggedOrderSign(a, b)) + // Preserve the legacy exact-zero assertion for NaN/NaN, including its + // distinction between positive and negative zero in Vitest. + if (a.category === `unordered` && b.category === `unordered`) { + expect(result).toBe(0) + } + } + } + const actual = [...rows].sort((a, b) => sortCompare(a.value, b.value)) + const expected = [...rows].sort(taggedOrderSign) + expect(actual.map((row) => row.id)).toEqual(expected.map((row) => row.id)) +} + +// Legal inputs contain 1..5 distinct finite numbers or valid epoch-day Dates, +// one or two NaNs/invalid Dates, and optional null. A singleton finite value, +// duplicate non-finite values, negative days, and null presence are margins. +// Removing the family axis misses invalid Dates; removing the duplicate count +// misses equal non-finite rows; removing null misses its default placement. +// Mixed families, Temporal values, symbols, and invalid option objects are not +// generated. The fixed examples below reconstruct the legacy sort histories. +const arbitraryNonFiniteInput: fc.Arbitrary = fc.record({ + family: fc.constantFrom(`number` as const, `date` as const), + finiteDays: fc.uniqueArray(fc.integer({ min: -20_000, max: 20_000 }), { + minLength: 1, + maxLength: 5, + }), + unorderedCount: fc.integer({ min: 1, max: 2 }), + includeNull: fc.boolean(), +}) + +type RealmDateSample = { + leftTime: number | null + rightTime: number | null + leftForeign: boolean + rightForeign: boolean +} + +function dateInRealm(time: number | null, foreign: boolean): Date { + const timestamp = time ?? Number.NaN + return foreign + ? (runInNewContext(`new Date(timestamp)`, { timestamp }) as Date) + : new Date(timestamp) +} + +// `null` is a model-only invalid-Date tag; production receives a real Date +// whose timestamp is NaN. Two valid tags compare by their supplied timestamp, +// while an invalid tag follows every valid tag. The model never asks the +// production Date detector, normalizer, or comparator for an expected sign. +function modelDateSign(left: number | null, right: number | null): -1 | 0 | 1 { + if (left === null) return right === null ? 0 : 1 + if (right === null) return -1 + return left < right ? -1 : left > right ? 1 : 0 +} + +function assertRealmDateOrder(sample: RealmDateSample): void { + const left = dateInRealm(sample.leftTime, sample.leftForeign) + const right = dateInRealm(sample.rightTime, sample.rightForeign) + const expected = modelDateSign(sample.leftTime, sample.rightTime) + const reverseExpected = expected === 0 ? 0 : -expected + expect(isUnorderable(left)).toBe(sample.leftTime === null) + expect(isUnorderable(right)).toBe(sample.rightTime === null) + expect(sign(ascComparator(left, right, DEFAULT_COMPARE_OPTIONS))).toBe( + expected, + ) + expect(sign(ascComparator(right, left, DEFAULT_COMPARE_OPTIONS))).toBe( + reverseExpected, + ) + // Index equality keys use a Date's timestamp. This separate observation + // prevents a correct direct order from masking reference-key normalization. + expect(normalizeValue(left)).toBe(sample.leftTime ?? Number.NaN) + expect(normalizeValue(right)).toBe(sample.rightTime ?? Number.NaN) +} + +// Legal pairs cross local and foreign native Dates, valid epoch milliseconds +// in -1e12..1e12 and invalid Dates. The fixed witnesses below force the +// foreign-valid and foreign-invalid paths; generated pairs add equal times, +// both operand orders, and both invalid values. Spoofed Date tags, proxies, +// non-Date objects, and user-defined Date subclasses are outside this grammar. +const arbitraryDateTime = fc.oneof( + fc.integer({ min: -1_000_000_000_000, max: 1_000_000_000_000 }), + fc.constant(null), +) +const arbitraryRealmDateSample: fc.Arbitrary = fc.oneof( + fc.record({ + leftTime: arbitraryDateTime, + rightTime: arbitraryDateTime, + leftForeign: fc.boolean(), + rightForeign: fc.boolean(), + }), + fc + .record({ + time: arbitraryDateTime, + leftForeign: fc.boolean(), + rightForeign: fc.boolean(), + }) + .map(({ time, leftForeign, rightForeign }) => ({ + leftTime: time, + rightTime: time, + leftForeign, + rightForeign, + })), +) + const arbitraryComparableArray = fc.array(arbitraryComparablePrimitive, { maxLength: 5, }) @@ -677,6 +868,199 @@ describe(`ascComparator property-based tests`, () => { }) }) +describe(`NaN and invalid Date default-order refinement`, () => { + it(`reconstructs the finite, duplicate NaN, null, and invalid Date witnesses`, () => { + assertNonFiniteOrder({ + family: `number`, + finiteDays: [3, 1, 5], + unorderedCount: 2, + includeNull: false, + }) + assertNonFiniteOrder({ + family: `number`, + finiteDays: [5, 1], + unorderedCount: 1, + includeNull: true, + }) + assertNonFiniteOrder({ + family: `date`, + finiteDays: [19_358], // 2023-01-01, the legacy valid-Date operand. + unorderedCount: 1, + includeNull: false, + }) + }) + + comparisonProperty([arbitraryNonFiniteInput])( + `default comparator refines tagged finite and non-finite order`, + (input) => assertNonFiniteOrder(input), + ) + + it(`rejects a wrong comparison that ties non-finite and finite values`, () => { + for (const family of [`number`, `date`] as const) { + let reachedFault = false + const faultyComparator = ( + a: number | Date | null, + b: number | Date | null, + ) => { + const faultApplies = + family === `number` + ? typeof a === `number` && + Number.isNaN(a) && + typeof b === `number` && + Number.isFinite(b) + : a instanceof Date && + Number.isNaN(a.getTime()) && + b instanceof Date && + Number.isFinite(b.getTime()) + if (faultApplies) { + reachedFault = true + return 0 + } + return ascComparator(a, b, DEFAULT_COMPARE_OPTIONS) + } + expect(() => + assertNonFiniteOrder( + { + family, + finiteDays: [1], + unorderedCount: 1, + includeNull: false, + }, + faultyComparator, + ), + ).toThrowError() + expect(reachedFault).toBe(true) + } + }) + + it(`rejects a default sort that ties NaN to finite values`, () => { + let reachedFault = false + const faultySort = (a: number | Date | null, b: number | Date | null) => { + if (typeof a === `number` && Number.isNaN(a)) { + reachedFault = true + return 0 + } + if (typeof b === `number` && Number.isNaN(b)) { + reachedFault = true + return 0 + } + return defaultComparator(a, b) + } + expect(() => + assertNonFiniteOrder( + { + family: `number`, + finiteDays: [3, 1, 5], + unorderedCount: 2, + includeNull: false, + }, + (a, b) => ascComparator(a, b, DEFAULT_COMPARE_OPTIONS), + faultySort, + ), + ).toThrowError() + expect(reachedFault).toBe(true) + }) +}) + +describe(`native Date order across JavaScript realms`, () => { + it(`rejects forged or throwing Date tags during native-brand checks`, () => { + const forged = { [Symbol.toStringTag]: `Date` } + const throwing = Object.defineProperty({}, Symbol.toStringTag, { + get() { + throw new Error(`tag read`) + }, + }) + expect(isUnorderable(forged)).toBe(false) + expect(normalizeValue(forged)).toBe(forged) + expect( + sign(ascComparator(forged, new Date(0), DEFAULT_COMPARE_OPTIONS)), + ).not.toBe(0) + expect(isUnorderable(throwing)).toBe(false) + }) + + it(`treats numeric NaN and invalid Dates as one non-finite order position`, () => { + for (const foreign of [false, true]) { + const invalid = dateInRealm(null, foreign) + expect(ascComparator(Number.NaN, invalid, DEFAULT_COMPARE_OPTIONS)).toBe( + 0, + ) + expect(ascComparator(invalid, Number.NaN, DEFAULT_COMPARE_OPTIONS)).toBe( + 0, + ) + } + }) + + it(`orders foreign valid and invalid Dates by the promised timestamp law`, () => { + const jan1 = Date.UTC(2023, 0, 1) + const jan2 = Date.UTC(2023, 0, 2) + assertRealmDateOrder({ + leftTime: jan2, + rightTime: jan1, + leftForeign: true, + rightForeign: false, + }) + assertRealmDateOrder({ + leftTime: null, + rightTime: jan1, + leftForeign: true, + rightForeign: false, + }) + assertRealmDateOrder({ + leftTime: jan1, + rightTime: jan1, + leftForeign: true, + rightForeign: false, + }) + assertRealmDateOrder({ + leftTime: jan1, + rightTime: null, + leftForeign: false, + rightForeign: true, + }) + }) + + comparisonProperty([arbitraryRealmDateSample])( + `local and foreign Dates refine timestamp and invalidity order`, + (sample) => assertRealmDateOrder(sample), + ) + + it(`orders a foreign invalid Date after a valid Date in a public scan`, async () => { + for (const foreign of [false, true]) { + const invalid = dateInRealm(null, foreign) + const valid = new Date(Date.UTC(2023, 0, 1)) + const source = createCollection({ + id: `realm-date-order-${foreign}`, + getKey: (row: { id: number; when: Date }) => row.id, + autoIndex: `off`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, when: invalid } }) + write({ type: `insert`, value: { id: 2, when: valid } }) + commit() + markReady() + }, + }, + }) + const query = createLiveQueryCollection((q) => + q + .from({ rows: source }) + .orderBy(({ rows }) => rows.when, `asc`) + .select(({ rows }) => ({ id: rows.id, when: rows.when })), + ) + try { + await query.preload() + // The local control proves that this public query actually orders + // rows; the foreign case must have the same promised result. + expect([...query.values()].map(({ id }) => id)).toEqual([2, 1]) + } finally { + await query.cleanup() + await source.cleanup() + } + } + }) +}) + describe(`descComparator property-based tests`, () => { comparisonProperty([arbitrarySameTypePair])( `descComparator reverses ascComparator ordering (same types)`, diff --git a/packages/db/tests/comparison.test.ts b/packages/db/tests/comparison.test.ts index 870b23978b..066b46e380 100644 --- a/packages/db/tests/comparison.test.ts +++ b/packages/db/tests/comparison.test.ts @@ -1,50 +1,8 @@ import { describe, expect, it } from 'vitest' import { Temporal } from 'temporal-polyfill' -import { - ascComparator, - compareValues, - defaultComparator, -} from '../src/utils/comparison' +import { ascComparator, compareValues } from '../src/utils/comparison' import { DEFAULT_COMPARE_OPTIONS } from '../src/utils' -describe(`ascComparator - PostgreSQL float semantics for NaN`, () => { - const opts = DEFAULT_COMPARE_OPTIONS // nulls: `first` - - it(`orders NaN greater than every number`, () => { - expect(ascComparator(NaN, 5, opts)).toBeGreaterThan(0) - expect(ascComparator(5, NaN, opts)).toBeLessThan(0) - }) - - it(`treats NaN as equal to NaN`, () => { - expect(ascComparator(NaN, NaN, opts)).toBe(0) - }) - - it(`produces a stable total order with NaN sorting last`, () => { - const sorted = [3, NaN, 1, 5, NaN].sort((a, b) => defaultComparator(a, b)) - - expect(sorted.slice(0, 3)).toEqual([1, 3, 5]) - expect(sorted.slice(3).every((v) => Number.isNaN(v))).toBe(true) - }) - - it(`keeps null before non-null values regardless of NaN`, () => { - // nulls still sort first by default; NaN sorts last (greatest non-null) - const sorted = [5, NaN, null, 1].sort((a, b) => defaultComparator(a, b)) - - expect(sorted[0]).toBe(null) - expect(sorted[1]).toBe(1) - expect(sorted[2]).toBe(5) - expect(Number.isNaN(sorted[3])).toBe(true) - }) - - it(`orders an invalid Date greater than valid Dates`, () => { - const invalid = new Date(`not a date`) - const valid = new Date(`2023-01-01`) - - expect(ascComparator(invalid, valid, opts)).toBeGreaterThan(0) - expect(ascComparator(valid, invalid, opts)).toBeLessThan(0) - }) -}) - describe(`ascComparator - Temporal values`, () => { const opts = DEFAULT_COMPARE_OPTIONS diff --git a/packages/db/tests/index-update-oracle.property.test.ts b/packages/db/tests/index-update-oracle.property.test.ts index 29645e5e77..e1cef0d8af 100644 --- a/packages/db/tests/index-update-oracle.property.test.ts +++ b/packages/db/tests/index-update-oracle.property.test.ts @@ -37,6 +37,9 @@ import type { CompareOptions } from '../src/query/builder/types.js' * replacing a value with one in a different exact-equality bucket at the * same comparator position must move its exact lookup membership. These laws * concern returned index results, not bucket identity or maintenance work. + * A build is one of those direct actions even when a source value is + * `undefined`: default nulls-first order must hold after build just as it + * does after incremental add. The nullish lane below exercises both paths. */ type IndexValue = number @@ -483,8 +486,11 @@ describe.each(indexTypes)(`%s update properties`, (_indexName, IndexType) => { * classification. Fresh singleton histories preserve the old update cuts; * coexisting values expose a wrong union of nullish buckets. Fixed histories * also retain two undefined owners beside strings, a nonempty state without - * undefined, and both mixed and undefined-only `IN` lists. The generated tail - * adds nearby put/delete histories over three keys and five values, with both + * undefined, and both mixed and undefined-only `IN` lists. A fixed build + * history starts with `undefined` and 1 in both source orders, then continues + * with an add; it distinguishes full rebuild from incremental insertion. + * The generated tail + * adds nearby put/delete/build histories over three keys and five values, with both * orders of a mixed `IN` list, while a fixed four-key history keeps a third * defined distractor. An absent source key is added, while an owned key is * updated. A direct update of an absent key is outside this grammar. Each @@ -502,7 +508,10 @@ describe.each(indexTypes)(`%s update properties`, (_indexName, IndexType) => { * neither expectation uses the production comparator or traversal. Fixed * histories keep three undefined owners, a remove-one-of-two cut, a mixed * four-row range, and explicit versus omitted lower bounds. The generated - * tail varies owners and values at each returned-action cut. Removing the + * tail varies owners and values at each returned-action cut. Build replaces + * source ownership; duplicate generated keys are resolved by the Map before + * invoking production, so a raw duplicate-entry build is excluded. Removing + * build loses the native-sort path; removing the * nullish values loses the boundary distinction, one owner loses repeated * bucket traversal, and one defined value loses cursor continuation. Values * outside the positive 1..3 domain, mixed-type numeric/string order, custom options, @@ -513,6 +522,7 @@ type NullishIndexValue = number | null | undefined type NullishIndexAction = | { type: `put`; key: string; value: NullishIndexValue } | { type: `delete`; key: string } + | { type: `build`; entries: Array<[string, NullishIndexValue]> } const nullishProbeValues: ReadonlyArray = [ undefined, @@ -542,6 +552,16 @@ const arbitraryNullishAction: fc.Arbitrary = fc.oneof( type: fc.constant(`delete` as const), key: fc.constantFrom(`a`, `b`, `c`), }), + fc.record({ + type: fc.constant(`build` as const), + entries: fc.array( + fc.tuple( + fc.constantFrom(`a`, `b`, `c`), + fc.constantFrom(...nullishProbeValues), + ), + { maxLength: 3 }, + ), + }), ) function expectExactMembership( @@ -721,6 +741,13 @@ function applyNullishAction( index.add(action.key, { value: action.value }) } rows.set(action.key, action.value) + } else if (action.type === `build`) { + // Resolve duplicate generated source keys before calling build; their raw + // duplicate-entry policy is outside this law. Build replaces the Map. + const replacement = new Map(action.entries) + index.build([...replacement].map(([key, value]) => [key, { value }])) + rows.clear() + for (const [key, value] of replacement) rows.set(key, value) } else if (rows.has(action.key)) { index.remove(action.key, { value: rows.get(action.key) }) rows.delete(action.key) @@ -872,6 +899,41 @@ describe.each(indexTypes)( } }) + test(`build preserves nulls-first ordered reads and ranges`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + applyNullishAction(index, rows, { + type: `build`, + entries: [ + [`undefined`, undefined], + [`one`, 1], + ], + }) + // A second build changes insertion order; the model still predicts the + // same returned key order and complete upper-bound results. + applyNullishAction(index, rows, { + type: `build`, + entries: [ + [`one`, 1], + [`undefined`, undefined], + ], + }) + applyNullishAction(index, rows, { + type: `put`, + key: `two`, + value: 2, + }) + applyNullishAction(index, rows, { + type: `build`, + entries: [ + [`undef2`, undefined], + [`null`, null], + [`undef1`, undefined], + [`one`, 1], + ], + }) + }) + test.each([20261012, undefined])( `refines nullish membership histories (seed %s)`, async (seed) => { diff --git a/packages/db/tests/query/subset-dedupe.test.ts b/packages/db/tests/query/subset-dedupe.test.ts index a293c5a53b..911d7cc030 100644 --- a/packages/db/tests/query/subset-dedupe.test.ts +++ b/packages/db/tests/query/subset-dedupe.test.ts @@ -416,11 +416,20 @@ describe(`DeduplicatedLoadSubset`, () => { eq(ref(`value`), val(value)), new Func(`in`, [ref(`value`), val([value])]), ]) { - const request = captureRequest({ where }) + const options = { where } + const request = captureRequest(options) + expect(request).toBe(options) + expect(request.where).toBe(where) const matches = compileSingleRowExpression(request.where!) expect( [foreign, local].map((item) => matches({ value: item })), - ).toEqual(realm === `foreign` ? [true, false] : [false, true]) + ).toEqual( + type === `Date` + ? [true, true] + : realm === `foreign` + ? [true, false] + : [false, true], + ) } }) }, diff --git a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts index 67f05b11e7..0905ed406b 100644 --- a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts +++ b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts @@ -64,6 +64,10 @@ * source restarted after cleanup retracts a vanished row even when its first * batch cannot satisfy the predicate. A third witness checks that a same-key * update which stays TRUE publishes its new row and exact subscriber payload. + * A focused tagged-Date matrix checks native Dates from this and another + * JavaScript realm. It compares direct snapshot, filtered subscriber, and live + * query keys after preload and one sync transaction; an explicit BasicIndex + * lookup checks the indexed equality key when an index is present. * * Calibration: pinned histories separate Kleene from two-valued logic, and a * checker test requires the two-valued answer to fail. Hostile production @@ -82,7 +86,10 @@ * open; the lifecycle publication owner needs a predicate dimension to reach * them. The mock sync source is a controlled provider; this oracle claims only * how the Collection filters and routes the sync transactions it supplies. + * The tagged-Date matrix covers native `node:vm` Dates, not proxies, Date + * subclasses, spoofed tags, or all possible realm hosts. */ +import { runInNewContext } from 'node:vm' import { fc, test as fcTest } from '@fast-check/vitest' import { describe, expect, it } from 'vitest' import { createCollection } from '../../src/collection/index.js' @@ -253,6 +260,37 @@ function syncedModelRow(id: string, v: FieldValue): ModelRow { return { id, v, $synced: true, $origin: `remote`, $key: id } } +// This focused model starts from tags, not runtime Date classification. A +// native Date and a number with the same finite timestamp are equal. Invalid +// Dates and NaN occupy one equality class under the documented float rule. +type DateEqualityTag = + | { + kind: `date` + realm: `local` | `foreign` + timestamp: number | `invalid` + } + | { kind: `number`; value: number | `NaN` } + +type TaggedDateRow = { id: string; tag: DateEqualityTag } + +function dateEqualityClass(tag: DateEqualityTag): number | `unordered` { + if (tag.kind === `date`) { + return tag.timestamp === `invalid` ? `unordered` : tag.timestamp + } + return tag.value === `NaN` ? `unordered` : tag.value +} + +function expectedTaggedDateKeys( + rows: ReadonlyArray, + literal: DateEqualityTag, +): Array { + const expectedClass = dateEqualityClass(literal) + return rows + .filter((row) => dateEqualityClass(row.tag) === expectedClass) + .map((row) => row.id) + .sort() +} + // --------------------------------------------------------------------------- // History grammar // --------------------------------------------------------------------------- @@ -522,6 +560,36 @@ const pinnedChangeHistories: ReadonlyArray = [ }, ] +// The fixed realm grammar crosses the Date constructor's realm with equal, +// unequal, and invalid timestamps. Number literals test the documented +// Date/number and invalid-Date/NaN equality classes. A later sync transaction +// moves three rows between those classes without changing their public keys. +const taggedDateRows: ReadonlyArray = [ + { id: `local-one`, tag: { kind: `date`, realm: `local`, timestamp: 1 } }, + { id: `foreign-one`, tag: { kind: `date`, realm: `foreign`, timestamp: 1 } }, + { id: `foreign-two`, tag: { kind: `date`, realm: `foreign`, timestamp: 2 } }, + { + id: `local-invalid`, + tag: { kind: `date`, realm: `local`, timestamp: `invalid` }, + }, + { + id: `foreign-invalid`, + tag: { kind: `date`, realm: `foreign`, timestamp: `invalid` }, + }, + { id: `number-one`, tag: { kind: `number`, value: 1 } }, + { id: `number-nan`, tag: { kind: `number`, value: `NaN` } }, +] + +const taggedDateLiterals: ReadonlyArray = [ + { kind: `date`, realm: `local`, timestamp: 1 }, + { kind: `date`, realm: `foreign`, timestamp: 1 }, + { kind: `date`, realm: `foreign`, timestamp: 2 }, + { kind: `date`, realm: `local`, timestamp: `invalid` }, + { kind: `date`, realm: `foreign`, timestamp: `invalid` }, + { kind: `number`, value: 1 }, + { kind: `number`, value: `NaN` }, +] + // --------------------------------------------------------------------------- // Production driver // --------------------------------------------------------------------------- @@ -594,6 +662,132 @@ function sourceRow(id: string, v: FieldValue): Row { return v === MISSING ? { id } : { id, v } } +function runtimeTaggedDateValue(tag: DateEqualityTag): Date | number { + if (tag.kind === `number`) { + return tag.value === `NaN` ? Number.NaN : tag.value + } + const timestamp = tag.timestamp === `invalid` ? Number.NaN : tag.timestamp + return tag.realm === `foreign` + ? (runInNewContext(`new Date(timestamp)`, { timestamp }) as Date) + : new Date(timestamp) +} + +type TaggedValueRow = { id: string; v: Date | number } + +// The tag model predicts keys before the runtime values are constructed. The +// driver observes the public live query, filtered subscription, and direct +// Collection snapshot after preload and after a committed source transaction. +// The explicit index lookup checks the physical equality key at both cuts. +async function observeTaggedDateEquality( + literal: DateEqualityTag, + indexed: boolean, +): Promise { + const rows = taggedDateRows.map((row) => ({ ...row })) + const literalValue = runtimeTaggedDateValue(literal) + const collection = createCollection( + mockSyncCollectionOptions({ + id: `where-publication-date-${collectionSerial++}`, + getKey: (row) => row.id, + autoIndex: `off`, + initialData: rows.map((row) => ({ + id: row.id, + v: runtimeTaggedDateValue(row.tag), + })), + }), + ) + const live = createLiveQueryCollection((q) => + q.from({ row: collection }).where(({ row }) => eq(row.v, literalValue)), + ) + const predicate = new Func(`eq`, [ + new PropRef([`v`]), + new Value(literalValue), + ]) + const visible = new Set() + let subscription: { unsubscribe: () => void } | undefined + + await withOracleCleanup(async () => { + await collection.stateWhenReady() + const index = indexed + ? collection.createIndex((row) => row.v, { indexType: BasicIndex }) + : undefined + await live.preload() + subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else visible.add(change.key) + } + }, + { includeInitialState: true, whereExpression: predicate }, + ) + + const check = (checkpoint: string) => { + const expected = expectedTaggedDateKeys(rows, literal) + const label = `${checkpoint}, ${indexed ? `indexed` : `unindexed`}, ${JSON.stringify(literal)}` + expect( + [...live.keys()].map(String).sort(), + `${label}: live query`, + ).toEqual(expected) + expect( + [...visible].map(String).sort(), + `${label}: filtered subscriber`, + ).toEqual(expected) + const snapshot = collection.currentStateAsChanges({ where: predicate }) + expect( + snapshot?.map((change) => String(change.key)).sort(), + `${label}: direct snapshot`, + ).toEqual(expected) + if (index) { + expect( + [...index.lookup(`eq`, literalValue)].map(String).sort(), + `${label}: BasicIndex equality lookup`, + ).toEqual(expected) + const optimized = collection.currentStateAsChanges({ + where: predicate, + optimizedOnly: true, + }) + expect( + optimized?.map((change) => String(change.key)).sort(), + `${label}: optimized public snapshot`, + ).toEqual(expected) + } + } + + check(`after subscribe`) + const transitions: ReadonlyArray<{ index: number; tag: DateEqualityTag }> = + [ + { + index: 0, + tag: { kind: `date`, realm: `foreign`, timestamp: `invalid` }, + }, + { index: 2, tag: { kind: `date`, realm: `foreign`, timestamp: 1 } }, + { index: 3, tag: { kind: `date`, realm: `foreign`, timestamp: 2 } }, + ] + collection.utils.begin() + for (const transition of transitions) { + collection.utils.write({ + type: `update`, + value: { + id: rows[transition.index]!.id, + v: runtimeTaggedDateValue(transition.tag), + }, + }) + } + collection.utils.commit() + for (const transition of transitions) { + rows[transition.index] = { + id: rows[transition.index]!.id, + tag: transition.tag, + } + } + check(`after transaction`) + }, [ + () => subscription?.unsubscribe(), + () => live.cleanup(), + () => collection.cleanup(), + ]) +} + type Consumer = | `live query` | `subscriber with initial state` @@ -1103,6 +1297,23 @@ describe(`WHERE predicate publication oracle`, () => { }, [() => subscription.unsubscribe(), () => collection.cleanup()]) }) + it(`native Dates from another realm publish by timestamp and invalid-Date equality`, async () => { + const foreign = runtimeTaggedDateValue({ + kind: `date`, + realm: `foreign`, + timestamp: 1, + }) + // This checks the host premise: a native Date can fail local instanceof. + expect(foreign instanceof Date).toBe(false) + expect(Date.prototype.getTime.call(foreign)).toBe(1) + + for (const indexed of [false, true]) { + for (const literal of taggedDateLiterals) { + await observeTaggedDateEquality(literal, indexed) + } + } + }) + for (const history of [ ...pinnedHistories, pinnedPrefilteredSnapshot, From 636e71e3d6084da6badeb0055fef5aae19ca4f58 Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 17:40:06 -0600 Subject: [PATCH 09/46] fix(db): preserve native Date semantics across display tags --- docs/contributing/oracle-coverage.md | 22 +-- .../basic-index-nullish-build.md | 73 ++++++++++ .../comparison-unit-retirement-round-5.md | 13 +- .../oracle-reviews/cross-realm-date-order.md | 67 +++++++-- .../cross-realm-date-where-publication.md | 33 ++++- docs/guides/live-queries.md | 1 + packages/db/src/utils/comparison.ts | 27 ++-- .../tests/comparison-oracle.property.test.ts | 128 +++++++++++++++--- ...dicate-publication-oracle.property.test.ts | 41 ++++-- 9 files changed, 334 insertions(+), 71 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 6e5e460c1d..f914c94bfc 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -466,23 +466,25 @@ sorted row IDs. Its [transfer ledger](oracle-reviews/comparison-unit-retirement- records the five retired same-realm examples. A separate [cross-realm Date witness](oracle-reviews/cross-realm-date-order.md) compares native `node:vm` Dates by supplied timestamps and invalidity, including equal -times and numeric NaN with an invalid Date. It also checks one unindexed -public ordered scan. Exact `normalizeValue` results are direct utility -observations. Indexed order, public Date range predicates, cursor -continuation, proxies, and other Date-consuming paths remain outside that -witness. +times, numeric NaN with an invalid Date, and Dates with an own display tag. +It also checks an unindexed public ordered scan with a tagged foreign invalid +Date. Exact `normalizeValue` results are direct utility observations. Indexed +order, public Date range predicates, cursor continuation, proxies, subclasses, +replaced Date prototypes or `getTime` methods, and other Date-consuming paths +remain outside that witness. The [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts) also has a [cross-realm Date equality matrix](oracle-reviews/cross-realm-date-where-publication.md). For local and `node:vm` Dates with timestamps 1 and 2 or invalid timestamps, -plus number 1 and NaN, it compares public live-query keys, filtered subscriber -keys, and direct snapshot keys after preload and a three-row sync transaction. +including foreign Dates with an own display tag, plus number 1 and NaN, it +compares public live-query keys, filtered subscriber keys, and direct snapshot +keys after preload and a three-row sync transaction. Auto-indexing is off in both modes; one mode adds BasicIndex explicitly and also checks its equality lookup and an optimized-only public snapshot. This proves a public indexed snapshot result but does not prove that the live-query -planner chose the index. Other realm hosts, -proxies, subclasses, public range predicates, and arbitrary histories need -separate witnesses. +planner chose the index. Other realm hosts, proxies, subclasses, Date prototype +swaps or method replacement, public range predicates, and arbitrary histories +need separate witnesses. The direct [index refinement oracle](../../packages/db/tests/index-update-oracle.property.test.ts) also owns exact nullish membership and direct `IN` set selection for BasicIndex diff --git a/docs/contributing/oracle-reviews/basic-index-nullish-build.md b/docs/contributing/oracle-reviews/basic-index-nullish-build.md index 21efc48d10..23e400c030 100644 --- a/docs/contributing/oracle-reviews/basic-index-nullish-build.md +++ b/docs/contributing/oracle-reviews/basic-index-nullish-build.md @@ -57,3 +57,76 @@ raw duplicate-entry builds, mixed string/number order, custom comparators, Collection queries, and work complexity. Its fixed and generated build cells do not establish every possible source history. The exact committed-head review is recorded in a later entry. + +## Exact committed-head hostile recheck + +The reviewed commit is `9a145fba130f27c914a4a1fd4f44c5871a917483`. +`/root/exact_review_index_build` read the source-isolated index retirement +ledgers, the index oracle and coverage map, the direct index implementations, +and the default comparator and interface contracts. The commit preserves the +earlier RED witness: its parent still sorts raw values, so native `Array.sort` +moves `undefined` behind `1` without calling the comparator. The repaired +build sorts objects that hold the values; this invokes the checked comparator +for that pair and leaves equal comparator positions stable. This recheck +inspected the parent implementation and the recorded RED run; it did not +execute the parent commit again. + +The committed direct index oracle passed 412/412 tests with no type errors. +The guarded replay of `index-update.nullish-membership` at seed `20261012`, +path `0`, reached both index classes for 100 runs each and passed. `git show +--check` passed, and the worktree was clean after those runs. No new unit test +was retired by this build repair. The earlier retirement ledgers still delimit +the assertions removed from their source files. + +| Requirement | Exact-head result | +| --- | --- | +| ORC-001 | `DEFAULT_COMPARE_OPTIONS`, `defaultComparator`, and the direct `IndexInterface` methods support default ascending, nulls-first returned results. The interface lists methods but does not itself spell out this order. The claim is limited to direct default-option reads over `undefined`, `null`, and positive numbers. | +| ORC-002 | A source Map owns exact key/value pairs. Its `nullishPosition` projection freshly sorts and filters rows without production normalization, comparator, index storage, or traversal. | +| ORC-003 | Opening and adjacent oracle prose states the law and limits, Map rule, legal put/delete/build histories, direct BasicIndex/BTreeIndex path, returned arrays and sets, and the after-action comparison cut. | +| ORC-004 | The fixed trace reconstructs `undefined` plus `1` in both build insertion orders, a put after build, then a four-key replacement with `undefined`, `null`, and `1`. Generated histories use three keys, five values, 1–40 actions, and builds of at most three entries. Removing build loses the full-sort path; removing `undefined` loses the native-sort distinction; removing defined values loses the nullish boundary; removing replacement loses stale-owner detection. Duplicate generated keys are resolved by the Map before build. Raw duplicate-entry builds, other value domains, and direct updates of absent keys are excluded. | +| ORC-005 | The oracle invokes real direct `build`, `add`, `update`, and `remove` methods, then compares exact membership sets, complete forward range sets, and ordered key arrays after each returned action. `takeFromStart(1)` observes the original defect. No public Collection query is claimed. | +| ORC-006 | The recorded unchanged-production run failed at the `takeFromStart(1)` refinement comparison, with `[one]` instead of `[undefined]`; this is an assertion failure at the intended cut. The repaired version passes. No additional production mutant was run in this recheck. | +| ORC-007 | Fixed-seed and seedless lanes share the generated action grammar, recorder, comparison, and 100-run budget. The exact-head guarded seed/path replay reached two 100-run executions. The earlier round-4 ledger records a captured failure replay for this named property, before the build action changed its arbitrary. A captured failing seed/path for the revised arbitrary was not independently checked in this recheck; that replay-fidelity cell remains open. | +| ORC-008 | The existing Map retains key/value ownership because later put and delete actions distinguish owners. Build adds no independent model state; it replaces that Map. | +| ORC-009 | Model `put` maps to add or update according to Map ownership, `delete` to remove when owned, and `build` to a full replacement. The model-only comparator position combines null and undefined for order while exact membership keeps them separate. | +| ORC-010 | The action loop has no external resources. The guarded replay retains its named witness and removes its private temporary channel; the recorded original failure remains distinct from a setup or cleanup failure. | +| ORC-011 | No plausible fault shared by this bounded source-Map rule and production was identified. Nullish position follows the accepted default comparator contract; mixed domains and custom comparison need separate laws. | +| ORC-012 | This append-only entry ties the exact-head review, executed GREEN result, inspected RED path, guide audit, and limits to the reviewed commit. The revised-arbitrary failure replay named under ORC-007 is the remaining conformance evidence gap. | +| ORC-013 | The two-entry `undefined`/`1` build reaches the law's premise. The first fixed observation rejects the plausible wrong rule “native sort may place undefined last”; reversing source insertion order distinguishes an accidental insertion-order pass. | +| ORC-014 | No controlled provider or host premise supports a claim about real-provider behavior. Direct index results are the stated boundary. | + +There is no blocker for the bounded direct-index code or its source-Map +comparison. This result does not establish Collection query selection after +`createIndex`, custom comparator behavior, mixed string/number ordering, raw +duplicate-entry builds, all range shapes, or work complexity. The revised +arbitrary's captured-failure replay remains to be recorded before claiming +complete ORC-007 conformance for the expanded generated property. + +## Revised-grammar failure replay calibration + +This follow-up closes the ORC-007 evidence gap above for the oracle at reviewed +commit `9a145fba130f27c914a4a1fd4f44c5871a917483`. A temporary wrong +expected result was inserted after the nullish action checker: whenever a +generated `build` returned, it required `takeFromStart(1)` to equal the +impossible key `calibration-only-missing-key`. This deliberately faulty +expectation did not modify production or the action arbitrary. + +The seedless generated lane failed at the returned-build endpoint comparison +for BasicIndex after one test, seed `1770427226`, shrink path `0:0`. Its +counterexample was one empty build; the actual endpoint was `[]`, while the +temporary expectation was `[calibration-only-missing-key]`. BTreeIndex also +failed at the same temporary assertion in its seedless lane. Guarded direct +replay of `index-update.nullish-membership` with the BasicIndex test selector, +seed `1770427226`, and path `0:0` reached exactly one property execution. Its +witness recorded one run and `failed: true`; Vitest and the replay guard exited +nonzero with the same endpoint assertion. This is an assertion failure at the +intended comparison, not a setup error or timeout. It demonstrates replay +fidelity for the revised arbitrary; the empty-build case is a replay control, +not the nonempty nullish-order product witness. + +After removing the temporary expectation, the same guarded seed/path replay +reached one BasicIndex execution, recorded `failed: false`, and passed with no +type errors. `git diff --exit-code -- +packages/db/tests/index-update-oracle.property.test.ts` confirmed that the +oracle file again matched the reviewed commit. No calibration fault remains. +The preceding ORC-007 and ORC-012 gaps are closed by this executed evidence. diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md index f81d9f41b2..a05abbf730 100644 --- a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md @@ -76,4 +76,15 @@ same-realm examples unique. ## Exact committed-head recheck -Pending commit and fresh recheck. +At published commit `9a145fba130f27c914a4a1fd4f44c5871a917483`, the fresh +hostile reviewer `/root/exact_review_comparison_round5` found no loss from the +five retired same-realm blocks. It checked the complete source file against the +tagged model, exact comparator zeros, and stable sorted row IDs. The focused +comparison suite passed 102 tests with no type errors. + +The reviewer also found a separate bug in the broader cross-realm Date repair. +A native foreign Date with an own `Symbol.toStringTag` escaped the production +Date detector. The new direct and public oracle controls failed against that +published implementation. The repair and its fresh committed-head verdict are +recorded in [the Date order review](cross-realm-date-order.md). This finding +does not restore a unique assertion to any of the five retired blocks. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-order.md b/docs/contributing/oracle-reviews/cross-realm-date-order.md index 39ccf25e4e..9c88bbef11 100644 --- a/docs/contributing/oracle-reviews/cross-realm-date-order.md +++ b/docs/contributing/oracle-reviews/cross-realm-date-order.md @@ -27,21 +27,26 @@ cleaned up in `finally`. The model uses `null` only as an invalid-Date *tag*: production receives a native Date with a NaN timestamp. Valid tags range from -1e12 to +1e12 epoch milliseconds. Both local and foreign constructors, both argument positions, -equal times, and invalid values are legal. A fixed cross-family case also -checks that numeric NaN and either local or foreign invalid Date compare at -exactly one order position. A forged `[Symbol.toStringTag] = 'Date'` object is -a negative control: neither the classifier nor the normalizer accepts it as a -Date, and the comparator does not tie it to a Date. Proxies, subclasses, -other value classes, cursor pagination, and other realm mechanisms remain -outside the bounded grammar. +equal times, invalid values, and an own display tag are legal. One fixed +control changes the foreign Date prototype's display tag. Another checks that +numeric NaN and either local or foreign +invalid Date compare at exactly one order position. A forged +`[Symbol.toStringTag] = 'Date'` ordinary object is a negative control: neither +the classifier nor the normalizer accepts it as a Date, and the comparator +does not tie it to a Date. Proxies, subclasses, other value classes, cursor +pagination, and other realm mechanisms remain outside the bounded grammar. ## Repair, GREEN evidence, and hostile review `dateTimestamp` now recognizes a native Date created in another realm and -reads its timestamp with `Date.prototype.getTime.call`. A string tag alone -cannot establish the native Date brand: a forged tag must fail that intrinsic -call. Local Dates use the same intrinsic, so an own `getTime` method cannot -silently replace the timestamp. `ascComparator` classifies each operand once, +reads its timestamp with `Date.prototype.getTime.call`. A fast shape gate +avoids a caught exception for ordinary objects and Temporal values. For an +object with a display tag, it checks the immediate prototype's own `getTime` +descriptor without reading the tag. The intrinsic remains the final brand +check, so a forged tag cannot establish Date identity, while an own or +prototype display tag cannot hide a normal Date. Local Dates use the same +intrinsic, so an own `getTime` method cannot silently replace the timestamp. +`ascComparator` classifies each operand once, uses its timestamp for valid Date order, and treats an invalid Date like NaN. `isUnorderable` and `normalizeValue` use the same timestamp extraction. @@ -69,14 +74,15 @@ The guide's law, authority, and scope requirements (ORC-001, -003, -011) are stated in the oracle opening and beside this tagged Date model. Supplied timestamps and invalidity tags predict order without production classification (ORC-002, -009). Fixed foreign-valid, foreign-invalid, equal-time, mixed-NaN, -and forged-tag controls complement generated realm/time pairs (ORC-004, -013). +tagged-Date, and forged-tag controls complement generated realm/time/tag pairs +(ORC-004, -013). Direct classifier, comparator, and normalization checks compare at function return; a public ordered scan compares after preload (ORC-005). The original production fails both a direct and public-row observation, supplying an actual wrong implementation rather than only a hypothetical mutant (ORC-006). The shared property harness runs fixed-seed and seedless campaigns and supports -seed/path replay (ORC-007). The scan cleans up its source and live-query -Collections in `finally` (ORC-010). There is no stateful model or external +seed/path replay (ORC-007). The scan uses `withOracleCleanup` to attempt both +Collection cleanups and preserve the primary mismatch (ORC-010). There is no stateful model or external provider in this lane (ORC-008, -014). This record preserves the exact RED head, reviewer verdict, and bounded GREEN result (ORC-012). @@ -92,4 +98,35 @@ This record does not infer their correctness from the unindexed scan. ## Exact committed-head recheck -Pending commit and fresh reviewer recheck. +At published commit `9a145fba130f27c914a4a1fd4f44c5871a917483`, the fresh +hostile reviewer `/root/exact_review_comparison_round5` found a reachable +counterexample to the native-Date claim. A foreign native Date with an own +`Symbol.toStringTag = 'Other'` still passes `Date.prototype.getTime.call`, but +the production detector used `Object.prototype.toString` and treated it as an +ordinary object. The reviewer executed a public ordered query with a foreign +invalid Date carrying that tag. It returned IDs `[1,2]` instead of `[2,1]`. +The reviewer also observed that `isUnorderable` read an unrelated object's +display-tag getter. The existing oracle had plain foreign Dates and a forged +ordinary object, but no genuine Date whose tag differed. + +The expanded fixed controls and generated tag axis failed against that +published implementation. The focused RED run reported five failures: the +unrelated getter-count control, tagged-Date normalization, both generated +campaigns, and the public ordered scan. The fixed campaign captured seed +`20260911`, shrink path `0:0:0`, at the function-return checkpoint. The guarded +intrinsic detector passed the focused Date suite: seven tests and no type +errors. The public scan now includes the tagged foreign invalid Date. Fixed +controls also cover a genuine foreign Date with a misleading Temporal tag, a +tag getter that must not run, and an altered foreign Date prototype tag. + +The reviewer also found a harness risk by source inspection. A rejected +`query.cleanup()` in the original `finally` block could replace the ordered-ID +mismatch and skip `source.cleanup()`. The scan now uses the repository's +failure-preserving `withOracleCleanup` helper. A fresh review of the next +committed head remains pending. + +An intermediate intrinsic-only repair threw and caught for every ordinary +object and Temporal comparison. Hostile microbenchmarks found a substantial +Temporal cost. The final shape gate avoids that hot path without trusting tag +text as a Date brand. Dates with a replaced or stripped Date prototype, Date +subclasses, and proxies remain outside this claim. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md index c3cf331b2a..ee6f07f50a 100644 --- a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md +++ b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md @@ -8,12 +8,13 @@ JavaScript realm was not normalized to its timestamp. The RED witnesses. This record tests the separate equality and publication path. It does not retire another unit test. -The live-query guide's comparison semantics and changelog #1582 say that -valid Dates compare by timestamp and that invalid Dates share NaN's equality -class. The WHERE publication owner already models these classes for local -values. Its new focused matrix tags each source value and literal as a local -or `node:vm` Date with timestamp 1, timestamp 2, or invalid timestamp, or as -number 1 or NaN. The model reduces tags to equality classes before production +The live-query guide now states explicitly that native valid Dates compare by +timestamp across realms and display tags. Changelog #1582 covers the separate +invalid-Date/NaN equality class. The WHERE publication owner already models +these classes for local values. Its focused matrix tags each source value and +literal as a local or `node:vm` Date with timestamp 1, timestamp 2, or an +invalid timestamp, or as number 1 or NaN. Some foreign Dates have an own +display tag. The model reduces tags to equality classes before production values exist. It predicts complete, sorted public key sets, independent of `normalizeValue`, the evaluator, or either index implementation. @@ -85,4 +86,22 @@ subscriber event multiplicity, payloads, or intermediate publication. ## Exact committed-head recheck -Pending commit and fresh reviewer recheck. +At published commit `9a145fba130f27c914a4a1fd4f44c5871a917483`, the fresh +hostile reviewer `/root/exact_review_where_realm` found no blocker in the +bounded matrix for unmodified native Dates. The focused WHERE and subset suites +passed 53 tests with no type errors. Source inspection confirmed that the +`optimizedOnly: true` read cannot fall back to a scan. + +The same reviewer independently found the foreign native Date with an own +`Symbol.toStringTag`. The published comparison utility classified it by object +identity, so the broader native-Date claim was open. The matrix now includes +tagged foreign source rows, tagged literals, and a tagged post-commit update. +The reviewer ran this expanded matrix in an isolated archive of `9a145fba1`. +At its first unindexed public live-query checkpoint, a local `Date(1)` literal +expected `foreign-tagged-one` in the key set, but production omitted it. The +complete expected set was `[foreign-one, foreign-tagged-one, local-one, +number-one]`; production returned `[foreign-one, local-one, number-one]`. +Vitest reported an assertion failure with no type errors. The unmodified +matrix passed against the same archived code. After the guarded intrinsic +repair, the expanded matrix passed in both indexed and unindexed modes. A +fresh committed-head review of the repaired matrix remains pending. diff --git a/docs/guides/live-queries.md b/docs/guides/live-queries.md index 8b1ea145c8..b28a6aeb9c 100644 --- a/docs/guides/live-queries.md +++ b/docs/guides/live-queries.md @@ -778,6 +778,7 @@ For a complete reference of all available functions, see the [Expression Functio Comparisons follow SQL/PostgreSQL conventions rather than raw JavaScript: - **`null` / `undefined` use three-valued logic.** Any comparison involving `null` or `undefined` evaluates to `UNKNOWN`, so the row is not matched. For example `eq(user.score, null)` matches nothing — use a dedicated null check (e.g. `isUndefined`) to match missing values. +- **Native `Date` values with a Date prototype that retains `getTime` compare by timestamp.** Dates with the same millisecond timestamp match equality and occupy the same order position. This includes Dates from another JavaScript realm and Dates with their own `Symbol.toStringTag`. Replacing a Date's prototype is outside this rule. - **`NaN` follows PostgreSQL float semantics.** `NaN` is treated as equal to itself and greater than every other (non-null) value. So `eq(row.value, NaN)` matches `NaN` rows, `gt(row.value, x)` includes `NaN`, and ordering by such a field places `NaN` last. (Invalid `Date` values, whose timestamp is `NaN`, behave the same way.) This differs from JavaScript, where `NaN === NaN` is `false`, and matches how PostgreSQL orders and indexes floating-point values. ## Select diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index 5b59e41da8..08b299035c 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -19,20 +19,31 @@ function getObjectId(obj: object): number { return id } -// A native Date can arrive from another JavaScript realm, where `instanceof` -// fails. Check its brand before borrowing the built-in method so a forged -// Symbol.toStringTag cannot make an ordinary object look like a Date. +// The Date intrinsic checks the native brand across realms. Cheap shape gates +// avoid that throwing probe for ordinary objects and Temporal values. A Date +// display tag cannot hide its timestamp, and a forged tag cannot supply one. function dateTimestamp(value: unknown): number | undefined { - if (value instanceof Date) return Date.prototype.getTime.call(value) if (value === null || typeof value !== `object`) return undefined + if (value instanceof Date) return Date.prototype.getTime.call(value) try { - if (Object.prototype.toString.call(value) === `[object Date]`) { - return Date.prototype.getTime.call(value) + if (Symbol.toStringTag in value) { + // A normal Date prototype owns getTime. Inspect its descriptor without + // reading a user display-tag getter or trusting a claimed tag value. + const prototype = Object.getPrototypeOf(value) + if ( + typeof Object.getOwnPropertyDescriptor(prototype, `getTime`)?.value !== + `function` + ) { + return undefined + } + } else if (Object.prototype.toString.call(value) !== `[object Date]`) { + return undefined } + return Date.prototype.getTime.call(value) } catch { - // A proxy or forged Date tag has no native Date timestamp. + // An ordinary object or proxy has no native Date timestamp. + return undefined } - return undefined } function hasUnorderableValue( diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index 2649600814..aa3fdc63bd 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -13,6 +13,7 @@ import { normalizeValue, } from '../src/utils/comparison' import { DEFAULT_COMPARE_OPTIONS } from '../src/utils' +import { withOracleCleanup } from './utils' import type { CompareOptions } from '../src/query/builder/types' /** @@ -23,7 +24,8 @@ import type { CompareOptions } from '../src/query/builder/types' * ascending domain below, NaN and invalid Dates share one non-finite order * position and follow finite values; null remains first. The published NaN * ordering change in `packages/db/CHANGELOG.md` (#1582) and the live-query - * guide establish that law for Dates regardless of their JavaScript realm. + * guide establish that law for Dates with native prototypes regardless of + * their JavaScript realm. * * Authority: `OrderByOptions` and `CompareOptions` in the query builder's * option types declare these modes. The index comparator contract accepts @@ -334,13 +336,23 @@ type RealmDateSample = { rightTime: number | null leftForeign: boolean rightForeign: boolean + leftOwnTag?: boolean + rightOwnTag?: boolean } -function dateInRealm(time: number | null, foreign: boolean): Date { +function dateInRealm( + time: number | null, + foreign: boolean, + ownTag = false, +): Date { const timestamp = time ?? Number.NaN - return foreign + const date = foreign ? (runInNewContext(`new Date(timestamp)`, { timestamp }) as Date) : new Date(timestamp) + if (ownTag) { + Object.defineProperty(date, Symbol.toStringTag, { value: `Other` }) + } + return date } // `null` is a model-only invalid-Date tag; production receives a real Date @@ -354,8 +366,16 @@ function modelDateSign(left: number | null, right: number | null): -1 | 0 | 1 { } function assertRealmDateOrder(sample: RealmDateSample): void { - const left = dateInRealm(sample.leftTime, sample.leftForeign) - const right = dateInRealm(sample.rightTime, sample.rightForeign) + const left = dateInRealm( + sample.leftTime, + sample.leftForeign, + sample.leftOwnTag, + ) + const right = dateInRealm( + sample.rightTime, + sample.rightForeign, + sample.rightOwnTag, + ) const expected = modelDateSign(sample.leftTime, sample.rightTime) const reverseExpected = expected === 0 ? 0 : -expected expect(isUnorderable(left)).toBe(sample.leftTime === null) @@ -375,8 +395,9 @@ function assertRealmDateOrder(sample: RealmDateSample): void { // Legal pairs cross local and foreign native Dates, valid epoch milliseconds // in -1e12..1e12 and invalid Dates. The fixed witnesses below force the // foreign-valid and foreign-invalid paths; generated pairs add equal times, -// both operand orders, and both invalid values. Spoofed Date tags, proxies, -// non-Date objects, and user-defined Date subclasses are outside this grammar. +// both operand orders, both invalid values, and Dates with an own display tag. +// A display tag cannot change native Date identity. Proxies, non-Date objects, +// and user-defined Date subclasses are outside this generated grammar. const arbitraryDateTime = fc.oneof( fc.integer({ min: -1_000_000_000_000, max: 1_000_000_000_000 }), fc.constant(null), @@ -387,18 +408,24 @@ const arbitraryRealmDateSample: fc.Arbitrary = fc.oneof( rightTime: arbitraryDateTime, leftForeign: fc.boolean(), rightForeign: fc.boolean(), + leftOwnTag: fc.boolean(), + rightOwnTag: fc.boolean(), }), fc .record({ time: arbitraryDateTime, leftForeign: fc.boolean(), rightForeign: fc.boolean(), + leftOwnTag: fc.boolean(), + rightOwnTag: fc.boolean(), }) - .map(({ time, leftForeign, rightForeign }) => ({ + .map(({ time, leftForeign, rightForeign, leftOwnTag, rightOwnTag }) => ({ leftTime: time, rightTime: time, leftForeign, rightForeign, + leftOwnTag, + rightOwnTag, })), ) @@ -963,11 +990,13 @@ describe(`NaN and invalid Date default-order refinement`, () => { }) describe(`native Date order across JavaScript realms`, () => { - it(`rejects forged or throwing Date tags during native-brand checks`, () => { + it(`uses the native Date brand without reading an unrelated display tag`, () => { const forged = { [Symbol.toStringTag]: `Date` } - const throwing = Object.defineProperty({}, Symbol.toStringTag, { + let tagReads = 0 + const unrelated = Object.defineProperty({}, Symbol.toStringTag, { get() { - throw new Error(`tag read`) + tagReads++ + return `Other` }, }) expect(isUnorderable(forged)).toBe(false) @@ -975,7 +1004,63 @@ describe(`native Date order across JavaScript realms`, () => { expect( sign(ascComparator(forged, new Date(0), DEFAULT_COMPARE_OPTIONS)), ).not.toBe(0) - expect(isUnorderable(throwing)).toBe(false) + expect(isUnorderable(unrelated)).toBe(false) + expect(tagReads).toBe(0) + }) + + it(`keeps native Date semantics when a foreign Date has an own display tag`, () => { + const timestamp = 123 + const foreignValid = dateInRealm(timestamp, true, true) + const foreignInvalid = dateInRealm(null, true, true) + const foreignTemporalTagged = dateInRealm(timestamp, true) + Object.defineProperty(foreignTemporalTagged, Symbol.toStringTag, { + value: `Temporal.PlainDate`, + }) + const foreignPrototypeTagged = dateInRealm(timestamp, true) + Object.defineProperty( + Object.getPrototypeOf(foreignPrototypeTagged), + Symbol.toStringTag, + { value: `Other` }, + ) + const foreignGetter = dateInRealm(timestamp, true) + let tagReads = 0 + Object.defineProperty(foreignGetter, Symbol.toStringTag, { + get() { + tagReads++ + return `Other` + }, + }) + const localValid = new Date(timestamp) + // The intrinsic establishes the native Date premise without consulting + // the production detector or the object's display tag. + expect(Date.prototype.getTime.call(foreignValid)).toBe(timestamp) + expect(Date.prototype.getTime.call(foreignInvalid)).toBeNaN() + expect(normalizeValue(foreignValid)).toBe(timestamp) + expect(normalizeValue(foreignTemporalTagged)).toBe(timestamp) + expect(normalizeValue(foreignPrototypeTagged)).toBe(timestamp) + expect(normalizeValue(foreignGetter)).toBe(timestamp) + expect(tagReads).toBe(0) + expect(normalizeValue(foreignInvalid)).toBeNaN() + expect(isUnorderable(foreignInvalid)).toBe(true) + expect( + ascComparator(foreignValid, localValid, DEFAULT_COMPARE_OPTIONS), + ).toBe(0) + expect( + ascComparator(localValid, foreignValid, DEFAULT_COMPARE_OPTIONS), + ).toBe(0) + expect( + ascComparator(foreignTemporalTagged, localValid, DEFAULT_COMPARE_OPTIONS), + ).toBe(0) + expect( + ascComparator( + foreignPrototypeTagged, + localValid, + DEFAULT_COMPARE_OPTIONS, + ), + ).toBe(0) + expect( + sign(ascComparator(foreignInvalid, localValid, DEFAULT_COMPARE_OPTIONS)), + ).toBe(1) }) it(`treats numeric NaN and invalid Dates as one non-finite order position`, () => { @@ -1025,11 +1110,15 @@ describe(`native Date order across JavaScript realms`, () => { ) it(`orders a foreign invalid Date after a valid Date in a public scan`, async () => { - for (const foreign of [false, true]) { - const invalid = dateInRealm(null, foreign) + for (const [foreign, ownTag] of [ + [false, false], + [true, false], + [true, true], + ] as const) { + const invalid = dateInRealm(null, foreign, ownTag) const valid = new Date(Date.UTC(2023, 0, 1)) const source = createCollection({ - id: `realm-date-order-${foreign}`, + id: `realm-date-order-${foreign}-${ownTag}`, getKey: (row: { id: number; when: Date }) => row.id, autoIndex: `off`, sync: { @@ -1048,15 +1137,14 @@ describe(`native Date order across JavaScript realms`, () => { .orderBy(({ rows }) => rows.when, `asc`) .select(({ rows }) => ({ id: rows.id, when: rows.when })), ) - try { + // Preserve an ordered-row mismatch if either resource also fails to + // clean up, and attempt both cleanups in all cases. + await withOracleCleanup(async () => { await query.preload() // The local control proves that this public query actually orders // rows; the foreign case must have the same promised result. expect([...query.values()].map(({ id }) => id)).toEqual([2, 1]) - } finally { - await query.cleanup() - await source.cleanup() - } + }, [() => query.cleanup(), () => source.cleanup()]) } }) }) diff --git a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts index 0905ed406b..6c423cda7f 100644 --- a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts +++ b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts @@ -86,8 +86,9 @@ * open; the lifecycle publication owner needs a predicate dimension to reach * them. The mock sync source is a controlled provider; this oracle claims only * how the Collection filters and routes the sync transactions it supplies. - * The tagged-Date matrix covers native `node:vm` Dates, not proxies, Date - * subclasses, spoofed tags, or all possible realm hosts. + * The tagged-Date matrix covers native `node:vm` Dates with ordinary Date + * prototypes, including own display tags. It excludes proxies, subclasses, + * prototype swaps, ordinary objects with spoofed Date tags, and other hosts. */ import { runInNewContext } from 'node:vm' import { fc, test as fcTest } from '@fast-check/vitest' @@ -263,11 +264,13 @@ function syncedModelRow(id: string, v: FieldValue): ModelRow { // This focused model starts from tags, not runtime Date classification. A // native Date and a number with the same finite timestamp are equal. Invalid // Dates and NaN occupy one equality class under the documented float rule. +// An own display tag does not change a native Date's equality class. type DateEqualityTag = | { kind: `date` realm: `local` | `foreign` timestamp: number | `invalid` + ownTag?: boolean } | { kind: `number`; value: number | `NaN` } @@ -561,12 +564,16 @@ const pinnedChangeHistories: ReadonlyArray = [ ] // The fixed realm grammar crosses the Date constructor's realm with equal, -// unequal, and invalid timestamps. Number literals test the documented -// Date/number and invalid-Date/NaN equality classes. A later sync transaction -// moves three rows between those classes without changing their public keys. +// unequal, and invalid timestamps, plus foreign Dates with an own display tag. +// Number literals test the Date/number and invalid-Date/NaN equality classes. +// A later sync transaction moves three rows between those classes. const taggedDateRows: ReadonlyArray = [ { id: `local-one`, tag: { kind: `date`, realm: `local`, timestamp: 1 } }, { id: `foreign-one`, tag: { kind: `date`, realm: `foreign`, timestamp: 1 } }, + { + id: `foreign-tagged-one`, + tag: { kind: `date`, realm: `foreign`, timestamp: 1, ownTag: true }, + }, { id: `foreign-two`, tag: { kind: `date`, realm: `foreign`, timestamp: 2 } }, { id: `local-invalid`, @@ -576,6 +583,10 @@ const taggedDateRows: ReadonlyArray = [ id: `foreign-invalid`, tag: { kind: `date`, realm: `foreign`, timestamp: `invalid` }, }, + { + id: `foreign-tagged-invalid`, + tag: { kind: `date`, realm: `foreign`, timestamp: `invalid`, ownTag: true }, + }, { id: `number-one`, tag: { kind: `number`, value: 1 } }, { id: `number-nan`, tag: { kind: `number`, value: `NaN` } }, ] @@ -583,9 +594,11 @@ const taggedDateRows: ReadonlyArray = [ const taggedDateLiterals: ReadonlyArray = [ { kind: `date`, realm: `local`, timestamp: 1 }, { kind: `date`, realm: `foreign`, timestamp: 1 }, + { kind: `date`, realm: `foreign`, timestamp: 1, ownTag: true }, { kind: `date`, realm: `foreign`, timestamp: 2 }, { kind: `date`, realm: `local`, timestamp: `invalid` }, { kind: `date`, realm: `foreign`, timestamp: `invalid` }, + { kind: `date`, realm: `foreign`, timestamp: `invalid`, ownTag: true }, { kind: `number`, value: 1 }, { kind: `number`, value: `NaN` }, ] @@ -667,9 +680,14 @@ function runtimeTaggedDateValue(tag: DateEqualityTag): Date | number { return tag.value === `NaN` ? Number.NaN : tag.value } const timestamp = tag.timestamp === `invalid` ? Number.NaN : tag.timestamp - return tag.realm === `foreign` - ? (runInNewContext(`new Date(timestamp)`, { timestamp }) as Date) - : new Date(timestamp) + const date = + tag.realm === `foreign` + ? (runInNewContext(`new Date(timestamp)`, { timestamp }) as Date) + : new Date(timestamp) + if (tag.ownTag) { + Object.defineProperty(date, Symbol.toStringTag, { value: `Other` }) + } + return date } type TaggedValueRow = { id: string; v: Date | number } @@ -760,8 +778,11 @@ async function observeTaggedDateEquality( index: 0, tag: { kind: `date`, realm: `foreign`, timestamp: `invalid` }, }, - { index: 2, tag: { kind: `date`, realm: `foreign`, timestamp: 1 } }, - { index: 3, tag: { kind: `date`, realm: `foreign`, timestamp: 2 } }, + { + index: 3, + tag: { kind: `date`, realm: `foreign`, timestamp: 1, ownTag: true }, + }, + { index: 4, tag: { kind: `date`, realm: `foreign`, timestamp: 2 } }, ] collection.utils.begin() for (const transition of transitions) { From c038b2b673a601d3bfbc70261f2cbb61f6f934ea Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 17:42:12 -0600 Subject: [PATCH 10/46] docs: record exact-head Date oracle reviews --- .../oracle-reviews/cross-realm-date-order.md | 11 +++++++++++ .../cross-realm-date-where-publication.md | 15 ++++++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/docs/contributing/oracle-reviews/cross-realm-date-order.md b/docs/contributing/oracle-reviews/cross-realm-date-order.md index 9c88bbef11..036ca76cad 100644 --- a/docs/contributing/oracle-reviews/cross-realm-date-order.md +++ b/docs/contributing/oracle-reviews/cross-realm-date-order.md @@ -130,3 +130,14 @@ object and Temporal comparison. Hostile microbenchmarks found a substantial Temporal cost. The final shape gate avoids that hot path without trusting tag text as a Date brand. Dates with a replaced or stripped Date prototype, Date subclasses, and proxies remain outside this claim. + +At clean commit `636e71e3d6084da6badeb0055fef5aae19ca4f58`, the fresh +hostile reviewer `/root/exact_review_comparison_round5` found no blocker for +the declared Date and retired-test scope. The focused suite passed 103 tests +with no type errors. The reviewer checked that genuine tagged foreign Dates +retain timestamp semantics, forged ordinary objects fail the native brand +check, unrelated display-tag getters remain unread, and the comparator avoids +the rejected exception-heavy path for Temporal values. The public scan cleanup +preserves a primary mismatch if cleanup also fails. This verdict does not +extend to replaced Date prototypes or `getTime` methods, proxies, subclasses, +indexed order, range queries, or cursor continuation. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md index ee6f07f50a..027b306636 100644 --- a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md +++ b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md @@ -104,4 +104,17 @@ number-one]`; production returned `[foreign-one, local-one, number-one]`. Vitest reported an assertion failure with no type errors. The unmodified matrix passed against the same archived code. After the guarded intrinsic repair, the expanded matrix passed in both indexed and unindexed modes. A -fresh committed-head review of the repaired matrix remains pending. +fresh committed-head review of the repaired matrix follows. + +At clean commit `636e71e3d6084da6badeb0055fef5aae19ca4f58`, the fresh +hostile reviewer `/root/exact_review_where_realm` found no blocker for this +bounded WHERE Date equality matrix. The focused WHERE and subset suites passed +53 tests with no type errors. The reviewer reconstructed the model classes, +checked tagged source rows and literals at initial and post-commit cuts, and +traced the live keys, subscriber reconstruction, and public snapshots in both +modes. The explicit BasicIndex lookup and `optimizedOnly: true` snapshot prove +an indexed public read, without claiming the live-query planner selected it. +The archived `9a145fba1` run failed at the intended unindexed public equality +comparison by omitting `foreign-tagged-one`; its unmodified controls passed. +The verdict does not cover other hosts, public ranges, indexed order, +arbitrary histories, or Date objects with replaced prototypes or methods. From 8b04592724fb943b04a56c6682757448d26a8ff6 Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 18:04:35 -0600 Subject: [PATCH 11/46] fix(db): preserve Temporal order through comparator and indexes --- .changeset/fix-temporal-index-order.md | 5 + docs/contributing/oracle-coverage.md | 12 + .../comparison-unit-retirement-round-6.md | 88 +++++ .../oracle-reviews/temporal-index-order.md | 83 +++++ .../functions/compareTemporalValues.md | 15 +- packages/db/package.json | 2 +- packages/db/src/indexes/basic-index.ts | 98 +++-- packages/db/src/indexes/btree-index.ts | 50 ++- packages/db/src/utils/comparison.ts | 44 +-- .../tests/comparison-oracle.property.test.ts | 336 +++++++++++++++++- packages/db/tests/comparison.test.ts | 23 -- .../tests/temporal-index-order-oracle.test.ts | 328 +++++++++++++++++ 12 files changed, 979 insertions(+), 105 deletions(-) create mode 100644 .changeset/fix-temporal-index-order.md create mode 100644 docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md create mode 100644 docs/contributing/oracle-reviews/temporal-index-order.md create mode 100644 packages/db/tests/temporal-index-order-oracle.test.ts diff --git a/.changeset/fix-temporal-index-order.md b/.changeset/fix-temporal-index-order.md new file mode 100644 index 0000000000..d1817e5c77 --- /dev/null +++ b/.changeset/fix-temporal-index-order.md @@ -0,0 +1,5 @@ +--- +'@tanstack/db': patch +--- + +Order Temporal values by their calendar or instant semantics in indexed reads. Keep exact equality distinct for ZonedDateTimes in different zones at the same instant. Prevent one Temporal value from changing the comparator used for later values. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index f914c94bfc..f2a7b82bf6 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -473,6 +473,18 @@ order, public Date range predicates, cursor continuation, proxies, subclasses, replaced Date prototypes or `getTime` methods, and other Date-consuming paths remain outside that witness. +The comparison oracle also owns a bounded [Temporal order transfer](oracle-reviews/comparison-unit-retirement-round-6.md). Supplied ISO calendar fields and epoch-nanosecond offsets predict direct `ascComparator` signs and exact zeros for ordinary `PlainDate` and `ZonedDateTime` values from one polyfill copy. Fixed cases preserve the two retired unit blocks; generated cases vary calendar fields, offset zones, equal instants, and adjacent nanoseconds. Fresh-module histories reject a first-input constructor cache poisoned by a genuine instance or forged tag, including the direct `compareValues` path. This lane does not establish public query or index behavior. Custom Temporal subclasses, replaced instance prototypes, mixed native/polyfill implementations, other Temporal kinds, and public ordering/filter publication need separate witnesses. + +The [Temporal index order oracle](../../packages/db/tests/temporal-index-order-oracle.test.ts) +and its [review record](oracle-reviews/temporal-index-order.md) own direct +BasicIndex and BTreeIndex ordered takes after add/build and one mutation +continuation, plus an optimized-only public Collection snapshot with an +explicit index. Supplied epoch nanoseconds establish order without promising +a key tie break for equal instants; distinct zones retain exact lookup +identities. Custom comparators, mixed value domains, other Temporal kinds, +public WHERE ranges, provider cursors, longer histories, and live-query +publication require separate witnesses. + The [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts) also has a [cross-realm Date equality matrix](oracle-reviews/cross-realm-date-where-publication.md). For local and `node:vm` Dates with timestamps 1 and 2 or invalid timestamps, diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md new file mode 100644 index 0000000000..5ede87cc42 --- /dev/null +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md @@ -0,0 +1,88 @@ +# Comparison unit retirement: Temporal order + +A source-isolated Field Lab loss audit compared the two Temporal blocks in +`packages/db/tests/comparison.test.ts` with the comparison oracle at frozen +head `c038b2b673a601d3bfbc70261f2cbb61f6f934ea`. The receiver expressly +excluded Temporal. The audit recovered these four assertions before the two +blocks were removed. The symbol block and both `compareValues` NaN blocks stay +in the unit file because they have different obligations. + +| Retired block | Original observation | Receiving oracle | +| --- | --- | --- | +| `orders PlainDate values by calendar date` | January 2024 precedes June 2024; the reverse comparison is positive; a fresh January value compares at exact zero. | Supplied ISO year/month/day tuples predict both signs and exact equal-date zero. The fixed case reconstructs January and June; generated tuples vary all three fields. | +| `treats ZonedDateTime values at the same instant as equal regardless of zone` | The 06:30 UTC and 12:00 +05:30 displays of one instant compare at exact zero. | Supplied epoch-nanosecond offsets predict both comparison directions. The fixed case reconstructs the two displays; generated cases vary offset zones, equal instants, and adjacent instants. | + +Changelog #1519 promises Temporal static comparison for `orderBy` and range +operators. It says `ZonedDateTime` comparison uses the instant while `eq` +keeps the zone as part of identity. The receiving model uses calendar tuples +and supplied nanosecond offsets, not production's Temporal dispatch or +normalization. The driver constructs ordinary values from one polyfill copy. +Each `ascComparator` return under default options is the comparison cut. +This direct lane also probes the shared `compareValues` helper in hostile +dispatch histories. It does not claim public query, index, or provider behavior. + +The grammar uses ISO PlainDates in years 2000–2030 with days 1–28. It uses +ZonedDateTimes within two seconds of a fixed 2024 instant in UTC, +05:30, +-05:00, and +14:00 zones. Equal-value branches construct distinct instances; +an adjacent-instant branch differs by one nanosecond. Fixed text-order and +millisecond-truncation controls fail the model comparison. Mixed Temporal +kinds, unsupported `PlainMonthDay` ordering, custom subclasses or prototype +replacement, and mixed native/polyfill implementations are outside this +transfer. The separate index owner below tests another production path. + +## RED witness and hostile review + +At unchanged production head `c038b2b673a601d3bfbc70261f2cbb61f6f934ea`, +the tag-wide comparator cache could learn a bogus static `compare` from the +first operand. A genuine PlainDate with an own `constructor.compare` returning +zero made January and June compare equal. A plain object with a forged +`Temporal.PlainDate` display tag could poison later genuine comparisons too. +Fresh-module histories prevent test order from seeding the cache first. The +focused run had eight passing Temporal checks and three intended failures at +the comparator-result assertions, with no type errors. A right-operand shadow +case also rejects a plausible repair that reads the second operand's own +constructor. A `compareValues`-first history reaches the other direct entry +point. The original four assertions pass in the fixed reconstruction. + +The fresh hostile reviewer `/root/hostile_temporal_direct` compared each +retired assertion with the new model and original source. It found the forged +tag, right-operand, and nanosecond precision gaps; the revised oracle added +those controls before production changed. The reviewer then rechecked the RED +run and found no remaining loss in the two retired blocks. It found a separate +published API-reference description of the old cache; this change updates +that page. Replaced prototypes, overridden subclass static comparators, and +mixed native/polyfill values remain excluded. These are recorded limits, not +claims that the comparator handles every altered Temporal object. + +The production repair removes the tag-wide cache and reads the static compare +function from the value's prototype on each call. The repaired direct and +evaluator suite passed 248 tests with no type errors. The exact committed-head +verdict follows after commit. + +## Oracle guide audit + +| Requirement | Evidence and limit | +| --- | --- | +| ORC-001 | Changelog #1519 supplies calendar/instant order and zone-sensitive equality. The opening and this record bound value kinds, implementation, paths, and checkpoints. | +| ORC-002 | Supplied date fields and nanosecond offsets predict signs without calling production comparison, normalization, or Temporal static compare. | +| ORC-003 | Prose beside the models explains the law, grammar, direct production paths, return observations, and refinement checks. | +| ORC-004 | Fixed cases reconstruct all four source assertions. Additional year-, month-, and day-separated pairs distinguish calendar-field omissions; distinct instances, zones, equal instants, and one-nanosecond neighbors distinguish the other axes. The bounded ranges and excluded value classes are stated above. | +| ORC-005 | Both direct exported helpers run on real Temporal values. Their exact returns, signs, and equal-value zeros are compared immediately. This is not public query evidence. | +| ORC-006 | A text-order control and a millisecond-truncation control fail at the sign check. The original cache fails three hostile histories at the same comparison cut. | +| ORC-007 | The shared comparison harness runs fixed-seed and seedless 100-case campaigns with the same property and supports direct seed/path replay. Replay evidence is recorded below. | +| ORC-008 | Not applicable: tuple and instant models have no mutable state. Fresh modules control production cache history, not reference-model state. | +| ORC-009 | Date tuples and nanosecond offsets are model input labels. Comparator zero describes order position, while `eq` identity retains the zone. | +| ORC-010 | The shared harness keeps the original failure checkpoint through shrinking. The direct tests own no cleanup resource. | +| ORC-011 | The hostile text-order and millisecond formulations distinguish shared semantic mistakes. The index/public path has its own oracle. | +| ORC-012 | This record contains the source ledger, RED/GREEN evidence, hostile verdict, limits, and committed-head recheck. | +| ORC-013 | Equal instants in different zones and adjacent nanoseconds distinguish text order and millisecond truncation from the promised instant law. | +| ORC-014 | Not applicable: this lane has no controlled provider or claim across a host boundary. | + +## Direct replay and exact committed-head review + +The shared harness directly replayed the PlainDate and ZonedDateTime generated +properties as selectors 39 and 40 with seed `20260911` and path `0`. Each +requested replay passed with no type errors; 39 other properties were skipped +in each run. After the final index oracle edit, the full DB suite passed +12,154 tests in 278 files with no type errors. The DB package build passed. +The exact committed-head review remains pending. diff --git a/docs/contributing/oracle-reviews/temporal-index-order.md b/docs/contributing/oracle-reviews/temporal-index-order.md new file mode 100644 index 0000000000..0f9358c9f0 --- /dev/null +++ b/docs/contributing/oracle-reviews/temporal-index-order.md @@ -0,0 +1,83 @@ +# Temporal indexed order + +The direct comparison transfer in [round 6](comparison-unit-retirement-round-6.md) +does not exercise an index. A separate oracle checks the public index contract: +an ordered take must follow `ZonedDateTime` instant order, while exact equality +keeps zones distinct. The law comes from Temporal's `ZonedDateTime.compare` and +`equals` semantics, changelog #1519, and the Collection's ordered-read and +IndexInterface contracts. No public contract fixes the key order of two rows +at the same instant, so the oracle accepts either tie order. + +## Original failure and repair + +At frozen head `c038b2b673a601d3bfbc70261f2cbb61f6f934ea`, both BasicIndex +and BTreeIndex normalized Temporal values to display strings before ordering. +For the supplied rows, `d-minus-five` has a later instant than the UTC and ++14:00 rows but an earlier display string. Both direct index types returned it +first after `add` and `build`. An optimized-only public Collection snapshot +with `limit: 1` exposed the same wrong first row; the unindexed control returned +an earliest-instant row. These failures occurred at the intended ordered-key +and public-snapshot comparisons. The exact same-instant zones retained separate +equality lookups. + +The repair retains normalized exact keys for equality and stores raw Temporal +values for ordered positions. The BTree's comparator group tracks each exact +key's ordered value so removal can replace a departing representative. A +separate removal witness failed against the first repair because the BTree +retained the removed raw value as its representative. Public ordered output +was still correct under a stable comparator; the witness establishes a storage +invariant, not a second public-order defect. The repaired focused oracle passed +13 tests, including a wrong-result control, and the existing index refinement +oracle passed 412 tests. The final +DB suite passed 12,154 tests in 278 files with no type errors. The DB package +build passed. The exact-commit review is recorded below after commit. + +The independent fixture supplies epoch nanoseconds and zone-identity classes. +It never calls the index's normalization or comparator to compute expected +results. A prefix checker requires all earlier instants, no later instants, +unique keys, and instant order. It permits either choice at an equal-instant +limit boundary. Exact lookup compares zone-sensitive key sets. The grammar has +two opposing offset zones at one instant, a successor one nanosecond later, +a later instant with earlier display text, and three pre-epoch values that +distinguish signed numeric order from decimal-string order. It exercises direct +`add`, `build`, a value cursor past both equal-instant peers, one +remove/insert/update continuation, and an explicit index in an optimized-only +Collection read after sync readiness. The collection driver spies on the +chosen index's `takeFromStart` to prove the intended path ran. + +## Hostile review and limits + +A separate hostile reviewer traced the exact-key and ordered-value paths in +both indexes, examined the original failure, and checked the repaired public +index selection. It found no blocker within the bounded default-comparator +domain. It identified a callback compatibility risk: a custom `compareFn` +now receives raw Temporal values where it previously received normalized +strings. Published types accept `any` and do not promise that representation. +Custom comparators require a separate product decision and witness before a +claim about them. Mixed domains, other Temporal kinds, public WHERE ranges, +provider cursors, longer mutation histories, and live-query publication also +remain outside this oracle. Existing range optimization does not claim +Temporal WHERE correctness. + +## Oracle guide audit + +| Requirement | Evidence and limit | +| --- | --- | +| ORC-001 | The opening names instant order, zone-sensitive equality, authority, and the bounded domain. Equal-instant tie order is deliberately unspecified. | +| ORC-002 | Supplied BigInt instants and zone classes determine expected order and exact matches independently of normalization and comparison code. | +| ORC-003 | The oracle describes its law, fixture model, legal actions, direct and public drivers, observations, and checkpoints beside executable code. | +| ORC-004 | Not triggered: the bounded fixture matrix and one continuation are fixed histories, not a generated-history claim. The nearby invalid mixed-kind domain is excluded. | +| ORC-005 | Direct takes and equality lookups observe both index implementations. An optimized-only Collection snapshot invokes the explicit index's `takeFromStart` after readiness. | +| ORC-006 | The original normalized-string design failed at direct ordered takes and the public `limit: 1` snapshot. A stable wrong-result control rejects that later-instant first row while allowing both legitimate equal-instant ties. The first repair's stale representative failed the internal invariant assertion. | +| ORC-007 | Not triggered: this is a bounded fixed oracle, not an important generated property. The existing index refinement property keeps its own campaigns. | +| ORC-008 | Not triggered: the fixture's independent model has no mutable reference state. | +| ORC-009 | The model's instant label represents order only; its zone class represents exact equality. One comparator position can contain two exact values. | +| ORC-010 | `withOracleCleanup` preserves a primary public mismatch and releases the Collection. Direct indexes own no external resources. | +| ORC-011 | The unindexed public scan is a second production formulation for the ordered result. The numeric BigInt model also distinguishes a shared display-string mistake. | +| ORC-012 | This record contains RED/GREEN evidence, a hostile review, and scoped unresolved cells. An exact-commit verdict follows below. | +| ORC-013 | The equal-instant zones and one-nanosecond neighbor distinguish collapse and truncation; the later instant with earlier display text distinguishes lexical order. | +| ORC-014 | Not triggered: the Collection has controlled sync and the claim ends at its in-memory ordered snapshot, with no real-provider claim. | + +## Exact committed-head review + +Pending. diff --git a/docs/reference/functions/compareTemporalValues.md b/docs/reference/functions/compareTemporalValues.md index 2f412b437a..c3c7ccf1e1 100644 --- a/docs/reference/functions/compareTemporalValues.md +++ b/docs/reference/functions/compareTemporalValues.md @@ -7,17 +7,18 @@ title: compareTemporalValues function compareTemporalValues(a, b): number; ``` -Defined in: [packages/db/src/utils/comparison.ts:313](https://github.com/TanStack/db/blob/main/packages/db/src/utils/comparison.ts#L313) +Defined in: [packages/db/src/utils/comparison.ts:345](https://github.com/TanStack/db/blob/main/packages/db/src/utils/comparison.ts#L345) Compare two Temporal values of the same type, returning -1, 0, or 1. -Dispatch is keyed on `Symbol.toStringTag` (the brand already checked by -`isTemporal`) rather than `a.constructor`, making it robust across realms -and resistant to a shadowed `constructor` property. Types without a static -`.compare` (e.g. `PlainMonthDay`) throw rather than fall back to string -comparison, matching Temporal's design intent. +The tag checks type agreement. The first value's prototype supplies the static +compare function, so an own `constructor` property cannot change dispatch or +poison later comparisons. Types without a static `.compare` (e.g. +`PlainMonthDay`) throw instead of using text order. -Callers must ensure both arguments are Temporal objects; mixed types throw. +Callers must ensure both arguments are ordinary Temporal objects from one +compatible implementation. Mixed types throw. Replaced prototypes and custom +subclass comparators are outside this contract. ## Parameters diff --git a/packages/db/package.json b/packages/db/package.json index 81dc7f5217..734692d965 100644 --- a/packages/db/package.json +++ b/packages/db/package.json @@ -23,7 +23,7 @@ "test": "vitest --run", "test:dist": "vitest run --config vitest.dist.config.ts", "test:facade-retention": "node --expose-gc --import tsx tests/facade-retention.probe.ts", - "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", + "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/temporal-index-order-oracle.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", "bench:nested-includes": "vitest bench tests/query/includes-performance.bench.ts --run", "test:indexed-db:package": "pnpm --filter @tanstack/db... build && vitest run --config vitest.indexed-db.package.config.ts", "test:indexed-db:browser": "playwright test --config playwright.indexed-db.config.ts", diff --git a/packages/db/src/indexes/basic-index.ts b/packages/db/src/indexes/basic-index.ts index 8caa057a9e..fd995b832f 100644 --- a/packages/db/src/indexes/basic-index.ts +++ b/packages/db/src/indexes/basic-index.ts @@ -1,5 +1,10 @@ import { compareKeys } from '@tanstack/db-ivm' -import { areSameValueZeroEqual, normalizeValue } from '../utils/comparison.js' +import { + areSameValueZeroEqual, + indexExactKey, + indexOrderValue, + normalizeValue, +} from '../utils/comparison.js' import { compareKeysReversed, findInsertPositionInArray, @@ -55,7 +60,8 @@ export class BasicIndex< // Map for O(1) equality lookups: indexedValue -> Set of PKs private valueMap = new Map>() - // Sorted array of unique indexed values for range queries + // One ordered representative per exact value; Temporal stays unnormalized + // here so equal instants in different zones share a comparator position. private sortedValues: Array = [] // Set of all indexed PKs private indexedKeys = new Set() @@ -69,13 +75,21 @@ export class BasicIndex< const indexedValue = this.evaluateAddedValue(key, item) const normalizedValue = normalizeValue(indexedValue) - this.addToBucket(key, normalizedValue) + this.addToBucket( + key, + normalizedValue, + indexOrderValue(indexedValue, normalizedValue), + ) this.addRangeValue(indexedValue) this.indexedKeys.add(key) } - private addToBucket(key: TKey, normalizedValue: unknown): void { + private addToBucket( + key: TKey, + normalizedValue: unknown, + orderValue: unknown, + ): void { const keySet = this.valueMap.get(normalizedValue) if (keySet) { // Value already exists, just add the key to the set @@ -85,11 +99,11 @@ export class BasicIndex< // throw before the index changes. const insertIdx = findInsertPositionInArray( this.sortedValues, - normalizedValue, + orderValue, this.compareFn, ) this.valueMap.set(normalizedValue, new Set([key])) - this.sortedValues.splice(insertIdx, 0, normalizedValue) + this.sortedValues.splice(insertIdx, 0, orderValue) } } @@ -108,13 +122,21 @@ export class BasicIndex< const normalizedValue = normalizeValue(indexedValue) - this.removeFromBucket(key, normalizedValue) + this.removeFromBucket( + key, + normalizedValue, + indexOrderValue(indexedValue, normalizedValue), + ) this.removeRangeValue(indexedValue) this.indexedKeys.delete(key) } - private removeFromBucket(key: TKey, normalizedValue: unknown): void { + private removeFromBucket( + key: TKey, + normalizedValue: unknown, + orderValue: unknown, + ): void { const keySet = this.valueMap.get(normalizedValue) if (keySet?.has(key)) { // The last key for this value leaves the sorted array too. Locate it @@ -122,17 +144,17 @@ export class BasicIndex< if (keySet.size === 1) { let sortedIndex = findInsertPositionInArray( this.sortedValues, - normalizedValue, + orderValue, this.compareFn, ) // Distinct equality keys may share one comparator position. while ( sortedIndex < this.sortedValues.length && - this.compareFn(this.sortedValues[sortedIndex], normalizedValue) === 0 + this.compareFn(this.sortedValues[sortedIndex], orderValue) === 0 ) { if ( areSameValueZeroEqual( - this.sortedValues[sortedIndex], + indexExactKey(this.sortedValues[sortedIndex]), normalizedValue, ) ) { @@ -174,9 +196,13 @@ export class BasicIndex< return } - this.removeFromBucket(key, oldValue) + this.removeFromBucket( + key, + oldValue, + indexOrderValue(oldIndexedValue, oldValue), + ) this.removeRangeValue(oldIndexedValue) - this.addToBucket(key, newValue) + this.addToBucket(key, newValue, indexOrderValue(newIndexedValue, newValue)) this.addRangeValue(newIndexedValue) this.indexedKeys.add(key) } @@ -188,26 +214,33 @@ export class BasicIndex< this.clear() // Collect all entries first - const entriesArray: Array<{ key: TKey; value: any }> = [] + const entriesArray: Array<{ key: TKey; value: any; orderValue: any }> = [] for (const [key, item] of entries) { const indexedValue = this.evaluateAddedValue(key, item) - entriesArray.push({ key, value: normalizeValue(indexedValue) }) + const value = normalizeValue(indexedValue) + entriesArray.push({ + key, + value, + orderValue: indexOrderValue(indexedValue, value), + }) this.addRangeValue(indexedValue) this.indexedKeys.add(key) } // Group by value - for (const { key, value } of entriesArray) { + const orderValues: Array = [] + for (const { key, value, orderValue } of entriesArray) { if (this.valueMap.has(value)) { this.valueMap.get(value)!.add(key) } else { this.valueMap.set(value, new Set([key])) + orderValues.push(orderValue) } } // Native Array.sort moves raw undefined values last without comparing them. // Wrap values so nulls-first comparison also runs during a full build. - const sortable = Array.from(this.valueMap.keys(), (value) => ({ value })) + const sortable = orderValues.map((value) => ({ value })) sortable.sort((left, right) => this.compareFn(left.value, right.value)) this.sortedValues = sortable.map(({ value }) => value) } @@ -285,6 +318,8 @@ export class BasicIndex< const normalizedFrom = normalizeValue(from) const normalizedTo = normalizeValue(to) + const orderFrom = indexOrderValue(from, normalizedFrom) + const orderTo = indexOrderValue(to, normalizedTo) const hasFrom = `from` in options const hasTo = `to` in options @@ -293,7 +328,7 @@ export class BasicIndex< if (hasFrom) { startIdx = findInsertPositionInArray( this.sortedValues, - normalizedFrom, + orderFrom, this.compareFn, ) // Comparator-equal values form one range boundary even when they are @@ -301,7 +336,7 @@ export class BasicIndex< while ( !fromInclusive && startIdx < this.sortedValues.length && - this.compareFn(this.sortedValues[startIdx], normalizedFrom) === 0 + this.compareFn(this.sortedValues[startIdx], orderFrom) === 0 ) { startIdx++ } @@ -312,14 +347,14 @@ export class BasicIndex< if (hasTo) { endIdx = findInsertPositionInArray( this.sortedValues, - normalizedTo, + orderTo, this.compareFn, ) // Include the whole comparator group at an inclusive upper boundary. while ( toInclusive && endIdx < this.sortedValues.length && - this.compareFn(this.sortedValues[endIdx], normalizedTo) === 0 + this.compareFn(this.sortedValues[endIdx], orderTo) === 0 ) { endIdx++ } @@ -327,7 +362,7 @@ export class BasicIndex< // Collect all keys in range for (let i = startIdx; i < endIdx; i++) { - const keys = this.valueMap.get(this.sortedValues[i]) + const keys = this.valueMap.get(indexExactKey(this.sortedValues[i])) if (keys) { keys.forEach((key) => result.add(key)) } @@ -341,15 +376,16 @@ export class BasicIndex< */ take(n: number, from: any, filterFn?: (key: TKey) => boolean): Array { const normalizedFrom = normalizeValue(from) + const orderFrom = indexOrderValue(from, normalizedFrom) let startIdx = findInsertPositionInArray( this.sortedValues, - normalizedFrom, + orderFrom, this.compareFn, ) // Skip past the 'from' value (exclusive) while ( startIdx < this.sortedValues.length && - this.compareFn(this.sortedValues[startIdx], normalizedFrom) <= 0 + this.compareFn(this.sortedValues[startIdx], orderFrom) <= 0 ) { startIdx++ } @@ -366,16 +402,14 @@ export class BasicIndex< filterFn?: (key: TKey) => boolean, ): Array { const normalizedFrom = normalizeValue(from) + const orderFrom = indexOrderValue(from, normalizedFrom) let startIdx = - findInsertPositionInArray( - this.sortedValues, - normalizedFrom, - this.compareFn, - ) - 1 + findInsertPositionInArray(this.sortedValues, orderFrom, this.compareFn) - + 1 // Skip past the 'from' value (exclusive) while ( startIdx >= 0 && - this.compareFn(this.sortedValues[startIdx], normalizedFrom) >= 0 + this.compareFn(this.sortedValues[startIdx], orderFrom) >= 0 ) { startIdx-- } @@ -416,7 +450,9 @@ export class BasicIndex< const groupValue = this.sortedValues[index] const groupKeys: Array = [] do { - for (const key of this.valueMap.get(this.sortedValues[index]) ?? []) { + for (const key of this.valueMap.get( + indexExactKey(this.sortedValues[index]), + ) ?? []) { groupKeys.push(key) } index += step diff --git a/packages/db/src/indexes/btree-index.ts b/packages/db/src/indexes/btree-index.ts index 2266c18a1f..23573d44a8 100644 --- a/packages/db/src/indexes/btree-index.ts +++ b/packages/db/src/indexes/btree-index.ts @@ -4,6 +4,7 @@ import { BTree } from '../utils/btree.js' import { areSameValueZeroEqual, denormalizeUndefined, + indexOrderValue, normalizeForBTree, } from '../utils/comparison.js' import { codedMessage, devBuild } from '../error-message.js' @@ -35,7 +36,9 @@ export interface RangeQueryOptions { type OrderedBucket = { representative: unknown - exactValues: Set + // Exact equality keys can share one comparator position. Their ordered + // values let removal replace a departing representative without a scan. + exactValues: Map keys: Set } @@ -84,13 +87,21 @@ export class BTreeIndex< // Normalize the value for Map key usage const normalizedValue = normalizeForBTree(indexedValue) - this.addToBucket(key, normalizedValue) + this.addToBucket( + key, + normalizedValue, + indexOrderValue(indexedValue, normalizedValue), + ) this.addRangeValue(indexedValue) this.indexedKeys.add(key) } - private addToBucket(key: TKey, normalizedValue: unknown): void { + private addToBucket( + key: TKey, + normalizedValue: unknown, + orderValue: unknown, + ): void { const exact = this.valueMap.get(normalizedValue) if (exact) { exact.keys.add(key) @@ -98,17 +109,17 @@ export class BTreeIndex< return } - let orderedBucket = this.orderedEntries.get(normalizedValue) + let orderedBucket = this.orderedEntries.get(orderValue) if (orderedBucket) { orderedBucket.keys.add(key) - orderedBucket.exactValues.add(normalizedValue) + orderedBucket.exactValues.set(normalizedValue, orderValue) } else { orderedBucket = { - representative: normalizedValue, - exactValues: new Set([normalizedValue]), + representative: orderValue, + exactValues: new Map([[normalizedValue, orderValue]]), keys: new Set([key]), } - this.orderedEntries.set(normalizedValue, orderedBucket) + this.orderedEntries.set(orderValue, orderedBucket) } this.valueMap.set(normalizedValue, { keys: new Set([key]), @@ -145,16 +156,19 @@ export class BTreeIndex< const replacesRepresentative = removedExactValue && orderedBucket.keys.size > 1 && - areSameValueZeroEqual(orderedBucket.representative, normalizedValue) + areSameValueZeroEqual( + orderedBucket.representative, + orderedBucket.exactValues.get(normalizedValue), + ) // Tree writes compare first, so a comparator failure throws before the // index changes. A comparator-equal `set` overwrites the entry's key. if (orderedBucket.keys.size === 1) { - this.orderedEntries.delete(normalizedValue) + this.orderedEntries.delete(orderedBucket.representative) } else if (replacesRepresentative) { const representative = [...orderedBucket.exactValues].find( - (value) => !areSameValueZeroEqual(value, normalizedValue), - ) + ([value]) => !areSameValueZeroEqual(value, normalizedValue), + )?.[1] this.orderedEntries.set(representative, orderedBucket) orderedBucket.representative = representative } @@ -194,7 +208,7 @@ export class BTreeIndex< this.removeFromBucket(key, oldValue) this.removeRangeValue(oldIndexedValue) - this.addToBucket(key, newValue) + this.addToBucket(key, newValue, indexOrderValue(newIndexedValue, newValue)) this.addRangeValue(newIndexedValue) this.indexedKeys.add(key) } @@ -286,9 +300,11 @@ export class BTreeIndex< const hasTo = `to` in options const fromKey = hasFrom - ? normalizeForBTree(from) + ? indexOrderValue(from, normalizeForBTree(from)) : this.orderedEntries.minKey() - const toKey = hasTo ? normalizeForBTree(to) : this.orderedEntries.maxKey() + const toKey = hasTo + ? indexOrderValue(to, normalizeForBTree(to)) + : this.orderedEntries.maxKey() this.orderedEntries.forRange( fromKey, @@ -361,7 +377,7 @@ export class BTreeIndex< take(n: number, from: any, filterFn?: (key: TKey) => boolean): Array { const nextPair = (k?: any) => this.orderedEntries.nextHigherPair(k) // Normalize the from value - const normalizedFrom = normalizeForBTree(from) + const normalizedFrom = indexOrderValue(from, normalizeForBTree(from)) return this.takeInternal(n, nextPair, normalizedFrom, filterFn) } @@ -390,7 +406,7 @@ export class BTreeIndex< ): Array { const nextPair = (k?: any) => this.orderedEntries.nextLowerPair(k) // Normalize the from value - const normalizedFrom = normalizeForBTree(from) + const normalizedFrom = indexOrderValue(from, normalizeForBTree(from)) return this.takeInternal(n, nextPair, normalizedFrom, filterFn, true) } diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index 08b299035c..f6f9381bf6 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -304,6 +304,16 @@ export function normalizeForBTree(value: any): any { return normalizeValue(value) } +/** Keep Temporal's native order separate from its exact index lookup key. */ +export function indexOrderValue(value: any, exactKey: any): any { + return isTemporal(value) ? value : exactKey +} + +/** Recover the exact lookup key from a stored ordered value. */ +export function indexExactKey(value: any): any { + return isTemporal(value) ? normalizeValue(value) : value +} + /** * Compare values using the equality semantics used by Map keys. */ @@ -322,23 +332,13 @@ export function denormalizeUndefined(value: any): any { return value } -// Cached map from Symbol.toStringTag → static compare function (null = none defined). -// Populated lazily on first encounter of each Temporal type so we never access -// `.constructor` more than once per type, and dispatch is keyed on the already- -// computed brand tag rather than on the constructor itself. -const temporalCompareByTag = new Map< - string, - ((a: unknown, b: unknown) => number) | null ->() - /** * Compare two Temporal values of the same type, returning -1, 0, or 1. * - * Dispatch is keyed on `Symbol.toStringTag` (the brand already checked by - * `isTemporal`) rather than `a.constructor`, making it robust across realms - * and resistant to a shadowed `constructor` property. Types without a static - * `.compare` (e.g. `PlainMonthDay`) throw rather than fall back to string - * comparison, matching Temporal's design intent. + * The tag checks type agreement; the prototype supplies the static compare + * function. An instance's own `constructor` can be shadowed, and a claimed tag + * must not install a comparator that later genuine values use. Types without + * a static `.compare` (e.g. `PlainMonthDay`) throw instead of using text order. * * Callers must ensure both arguments are Temporal objects; mixed types throw. */ @@ -352,17 +352,11 @@ export function compareTemporalValues(a: unknown, b: unknown): number { : codedMessage(167, { aTag, bTag }), ) } - let compare = temporalCompareByTag.get(aTag) - if (compare === undefined) { - const fn = ( - (a as { constructor: unknown }).constructor as { - compare?: (x: unknown, y: unknown) => number - } - ).compare - compare = typeof fn === `function` ? fn : null - temporalCompareByTag.set(aTag, compare) - } - if (compare === null) { + const prototype = Object.getPrototypeOf(a) as { + constructor?: { compare?: (x: unknown, y: unknown) => number } + } | null + const compare = prototype?.constructor?.compare + if (typeof compare !== `function`) { throw new TypeError( devBuild() && process.env.NODE_ENV !== `production` ? `${aTag} has no defined ordering` diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index aa3fdc63bd..b2a05ee7f2 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -1,6 +1,7 @@ import { runInNewContext } from 'node:vm' -import { describe, expect, it } from 'vitest' +import { describe, expect, it, vi } from 'vitest' import { fc, test as fcTest } from '@fast-check/vitest' +import { Temporal } from 'temporal-polyfill' import { createCollection } from '../src/collection/index' import { createLiveQueryCollection } from '../src/query/live-query-collection' import { @@ -1354,6 +1355,339 @@ describe(`areValuesEqual property-based tests`, () => { ) }) +/** + * Temporal ordering uses the value's calendar position or instant, not its + * display text. The published #1519 contract gives PlainDate calendar order + * and ZonedDateTime instant order even when zones differ. These models use + * supplied ISO date fields and epoch-nanosecond offsets; neither asks the + * production comparator or Temporal's static compare for an expected sign. + * + * Legal inputs are ISO PlainDates from years 2000..2030 with days 1..28 and + * ZonedDateTimes within two seconds of a fixed instant in four offset zones. + * Equal-value branches construct separate objects, and the zoned branch forces + * different zones at the same instant. The production paths are the exported + * `ascComparator` under the default options and `compareValues` in hostile + * dispatch histories. Each return is the observation checkpoint; + * this lane makes no indexed or public-query claim. Comparator + * zero for same-instant ZonedDateTimes is distinct from `eq`/`in` identity: + * the zone remains part of identity under the #1519 contract. + */ +type IsoDateFields = { year: number; month: number; day: number } +type PlainDateOrderInput = { left: IsoDateFields; right: IsoDateFields } + +function modelPlainDateSign( + left: IsoDateFields, + right: IsoDateFields, +): -1 | 0 | 1 { + for (const field of [`year`, `month`, `day`] as const) { + if (left[field] < right[field]) return -1 + if (left[field] > right[field]) return 1 + } + return 0 +} + +function assertPlainDateOrder(input: PlainDateOrderInput): void { + const left = new Temporal.PlainDate( + input.left.year, + input.left.month, + input.left.day, + ) + const right = new Temporal.PlainDate( + input.right.year, + input.right.month, + input.right.day, + ) + const expected = modelPlainDateSign(input.left, input.right) + const actual = ascComparator(left, right, DEFAULT_COMPARE_OPTIONS) + const reverse = ascComparator(right, left, DEFAULT_COMPARE_OPTIONS) + expect(sign(actual)).toBe(expected) + expect(sign(reverse)).toBe(expected === 0 ? 0 : -expected) + if (expected === 0) { + // The old unit block asserted an exact zero for two distinct instances. + expect(left).not.toBe(right) + expect(actual).toBe(0) + expect(reverse).toBe(0) + } +} + +const arbitraryIsoDateFields: fc.Arbitrary = fc.record({ + year: fc.integer({ min: 2000, max: 2030 }), + month: fc.integer({ min: 1, max: 12 }), + day: fc.integer({ min: 1, max: 28 }), +}) +const arbitraryPlainDateOrderInput: fc.Arbitrary = + fc.oneof( + fc.record({ + left: arbitraryIsoDateFields, + right: arbitraryIsoDateFields, + }), + arbitraryIsoDateFields.map((fields) => ({ left: fields, right: fields })), + ) + +type OffsetZone = `+00:00` | `+05:30` | `-05:00` | `+14:00` +type ZonedDateTimeOrderInput = { + leftNanoOffset: number + rightNanoOffset: number + leftZone: OffsetZone + rightZone: OffsetZone +} + +// The base instant reconstructs the old 06:30 UTC / 12:00 +05:30 witness. +// Offsets are safe integers, so their ordinary numeric order is the reference +// for the generated epoch nanoseconds. +const zonedBaseEpochNanoseconds = + BigInt(Date.UTC(2024, 0, 15, 6, 30)) * 1_000_000n + +function modelNanosecondSign(left: number, right: number): -1 | 0 | 1 { + return left < right ? -1 : left > right ? 1 : 0 +} + +function zonedValue(nanoOffset: number, zone: OffsetZone) { + return Temporal.Instant.fromEpochNanoseconds( + zonedBaseEpochNanoseconds + BigInt(nanoOffset), + ).toZonedDateTimeISO(zone) +} + +function assertZonedDateTimeOrder( + input: ZonedDateTimeOrderInput, + compare: (left: unknown, right: unknown) => number = (left, right) => + ascComparator(left, right, DEFAULT_COMPARE_OPTIONS), +): void { + const left = zonedValue(input.leftNanoOffset, input.leftZone) + const right = zonedValue(input.rightNanoOffset, input.rightZone) + const expected = modelNanosecondSign( + input.leftNanoOffset, + input.rightNanoOffset, + ) + const actual = compare(left, right) + const reverse = compare(right, left) + expect(sign(actual)).toBe(expected) + expect(sign(reverse)).toBe(expected === 0 ? 0 : -expected) + if (expected === 0) { + expect(actual).toBe(0) + expect(reverse).toBe(0) + if (input.leftZone !== input.rightZone) { + // A text-order mutant sees different strings for one model instant. + expect(left.toString()).not.toBe(right.toString()) + } + } +} + +const arbitraryNanoOffset = fc.integer({ + min: -1_999_999_999, + max: 1_999_999_999, +}) +const arbitraryOffsetZone: fc.Arbitrary = fc.constantFrom( + `+00:00`, + `+05:30`, + `-05:00`, + `+14:00`, +) +const arbitraryZonedDateTimeOrderInput: fc.Arbitrary = + fc.oneof( + fc.record({ + leftNanoOffset: arbitraryNanoOffset, + rightNanoOffset: arbitraryNanoOffset, + leftZone: arbitraryOffsetZone, + rightZone: arbitraryOffsetZone, + }), + fc + .tuple(arbitraryNanoOffset, arbitraryOffsetZone, arbitraryOffsetZone) + .filter(([, leftZone, rightZone]) => leftZone !== rightZone) + .map(([nanoOffset, leftZone, rightZone]) => ({ + leftNanoOffset: nanoOffset, + rightNanoOffset: nanoOffset, + leftZone, + rightZone, + })), + fc + .tuple( + arbitraryNanoOffset, + fc.boolean(), + arbitraryOffsetZone, + arbitraryOffsetZone, + ) + .map(([nanoOffset, forward, leftZone, rightZone]) => ({ + leftNanoOffset: nanoOffset, + rightNanoOffset: nanoOffset + (forward ? 1 : -1), + leftZone, + rightZone, + })), + ) + +// Dispatch previously cached the first comparison function seen for a tag. +// Reload before each hostile history so its first dispatch, rather than test +// execution order, determines whether a bad function can poison later values. +async function freshTemporalComparison() { + vi.resetModules() + return import('../src/utils/comparison') +} + +describe(`Temporal default-order refinement`, () => { + it(`reconstructs the three PlainDate and one ZonedDateTime assertions`, () => { + const jan = { year: 2024, month: 1, day: 1 } + const june = { year: 2024, month: 6, day: 1 } + assertPlainDateOrder({ + left: { year: 2023, month: 12, day: 28 }, + right: jan, + }) + assertPlainDateOrder({ left: jan, right: june }) + assertPlainDateOrder({ + left: jan, + right: { year: 2024, month: 1, day: 2 }, + }) + assertPlainDateOrder({ left: jan, right: jan }) + assertZonedDateTimeOrder({ + leftNanoOffset: 0, + rightNanoOffset: 0, + leftZone: `+00:00`, + rightZone: `+05:30`, + }) + assertZonedDateTimeOrder({ + leftNanoOffset: 0, + rightNanoOffset: 1, + leftZone: `+05:30`, + rightZone: `+00:00`, + }) + }) + + comparisonProperty([arbitraryPlainDateOrderInput])( + `ISO PlainDate order refines supplied calendar fields`, + assertPlainDateOrder, + ) + + comparisonProperty([arbitraryZonedDateTimeOrderInput])( + `ZonedDateTime order refines supplied instants across offset zones`, + assertZonedDateTimeOrder, + ) + + it(`rejects text order for one instant shown in two zones`, () => { + const sample: ZonedDateTimeOrderInput = { + leftNanoOffset: 0, + rightNanoOffset: 0, + leftZone: `+00:00`, + rightZone: `+05:30`, + } + const lexical = (left: unknown, right: unknown) => + String(left).localeCompare(String(right)) + expect(() => assertZonedDateTimeOrder(sample, lexical)).toThrow() + }) + + it(`rejects millisecond truncation across adjacent instants`, () => { + const sample: ZonedDateTimeOrderInput = { + leftNanoOffset: 0, + rightNanoOffset: 1, + leftZone: `+05:30`, + rightZone: `+00:00`, + } + const truncated = (left: unknown, right: unknown) => { + const leftEpoch = (left as { epochNanoseconds: bigint }).epochNanoseconds + const rightEpoch = (right as { epochNanoseconds: bigint }) + .epochNanoseconds + return Number(leftEpoch / 1_000_000n - rightEpoch / 1_000_000n) + } + expect(() => assertZonedDateTimeOrder(sample, truncated)).toThrow() + }) + + it(`keeps a PlainDate constructor override from poisoning later comparisons`, async () => { + const { ascComparator: isolatedComparator, compareValues: isolatedValues } = + await freshTemporalComparison() + // A genuine Temporal instance may have an own data property; its calendar + // value is unchanged by that property. It is the first dispatch here. + const shadowed = new Temporal.PlainDate(2024, 1, 1) + Object.defineProperty(shadowed, `constructor`, { + value: { compare: () => 0 }, + }) + const later = new Temporal.PlainDate(2024, 6, 1) + const ordinary = new Temporal.PlainDate(2024, 1, 1) + const expected = modelPlainDateSign( + { year: 2024, month: 1, day: 1 }, + { year: 2024, month: 6, day: 1 }, + ) + const observed = [ + isolatedComparator(shadowed, later, DEFAULT_COMPARE_OPTIONS), + isolatedComparator(ordinary, later, DEFAULT_COMPARE_OPTIONS), + isolatedValues(ordinary, later), + ] + expect(observed.map(sign)).toEqual([expected, expected, expected]) + }) + + it(`keeps a right-hand constructor override from changing PlainDate order`, async () => { + const { ascComparator: isolatedComparator, compareValues: isolatedValues } = + await freshTemporalComparison() + const january = new Temporal.PlainDate(2024, 1, 1) + const shadowedJune = new Temporal.PlainDate(2024, 6, 1) + Object.defineProperty(shadowedJune, `constructor`, { + value: { compare: () => 0 }, + }) + const ordinaryJanuary = new Temporal.PlainDate(2024, 1, 1) + const ordinaryJune = new Temporal.PlainDate(2024, 6, 1) + const expected = modelPlainDateSign( + { year: 2024, month: 1, day: 1 }, + { year: 2024, month: 6, day: 1 }, + ) + const observed = [ + isolatedComparator(january, shadowedJune, DEFAULT_COMPARE_OPTIONS), + isolatedComparator( + ordinaryJanuary, + ordinaryJune, + DEFAULT_COMPARE_OPTIONS, + ), + isolatedValues(ordinaryJanuary, ordinaryJune), + ] + expect(observed.map(sign)).toEqual([expected, expected, expected]) + }) + + it(`keeps a compareValues-first constructor override from poisoning later comparisons`, async () => { + const { ascComparator: isolatedComparator, compareValues: isolatedValues } = + await freshTemporalComparison() + const shadowedJanuary = new Temporal.PlainDate(2024, 1, 1) + Object.defineProperty(shadowedJanuary, `constructor`, { + value: { compare: () => 0 }, + }) + const june = new Temporal.PlainDate(2024, 6, 1) + const ordinaryJanuary = new Temporal.PlainDate(2024, 1, 1) + const expected = modelPlainDateSign( + { year: 2024, month: 1, day: 1 }, + { year: 2024, month: 6, day: 1 }, + ) + const observed = [ + isolatedValues(shadowedJanuary, june), + isolatedValues(ordinaryJanuary, june), + isolatedComparator(ordinaryJanuary, june, DEFAULT_COMPARE_OPTIONS), + ] + expect(observed.map(sign)).toEqual([expected, expected, expected]) + }) + + it(`keeps a forged Temporal tag from poisoning genuine PlainDate order`, async () => { + // A claimed display tag may enter the dispatch path, but must not change + // the result of a later comparison between genuine Temporal values. + const { ascComparator: isolatedComparator, compareValues: isolatedValues } = + await freshTemporalComparison() + const forged = { + [Symbol.toStringTag]: `Temporal.PlainDate`, + constructor: { compare: () => 0 }, + } + const june = new Temporal.PlainDate(2024, 6, 1) + try { + isolatedComparator(forged, june, DEFAULT_COMPARE_OPTIONS) + } catch { + // Rejecting the forged value is a valid outcome for this history. + } + const january = new Temporal.PlainDate(2024, 1, 1) + const expected = modelPlainDateSign( + { year: 2024, month: 1, day: 1 }, + { year: 2024, month: 6, day: 1 }, + ) + expect( + [ + isolatedComparator(january, june, DEFAULT_COMPARE_OPTIONS), + isolatedValues(january, june), + ].map(sign), + ).toEqual([expected, expected]) + }) +}) + if (requestedProperty !== undefined) { it(`selects a registered comparison oracle property`, () => { expect(requestedProperty).toMatch(/^\d+$/) diff --git a/packages/db/tests/comparison.test.ts b/packages/db/tests/comparison.test.ts index 066b46e380..7c60bc5d0c 100644 --- a/packages/db/tests/comparison.test.ts +++ b/packages/db/tests/comparison.test.ts @@ -1,30 +1,7 @@ import { describe, expect, it } from 'vitest' -import { Temporal } from 'temporal-polyfill' import { ascComparator, compareValues } from '../src/utils/comparison' import { DEFAULT_COMPARE_OPTIONS } from '../src/utils' -describe(`ascComparator - Temporal values`, () => { - const opts = DEFAULT_COMPARE_OPTIONS - - it(`orders PlainDate values by calendar date`, () => { - const earlier = new Temporal.PlainDate(2024, 1, 1) - const later = new Temporal.PlainDate(2024, 6, 1) - expect(ascComparator(earlier, later, opts)).toBeLessThan(0) - expect(ascComparator(later, earlier, opts)).toBeGreaterThan(0) - expect( - ascComparator(earlier, new Temporal.PlainDate(2024, 1, 1), opts), - ).toBe(0) - }) - - it(`treats ZonedDateTime values at the same instant as equal regardless of zone`, () => { - // 2024-01-15T06:30Z and 2024-01-15T12:00+05:30 are the same instant; - // lexicographic toString comparison would order them differently. - const utc = Temporal.ZonedDateTime.from(`2024-01-15T06:30:00+00:00[+00:00]`) - const ist = Temporal.ZonedDateTime.from(`2024-01-15T12:00:00+05:30[+05:30]`) - expect(ascComparator(utc, ist, opts)).toBe(0) - }) -}) - describe(`ascComparator - symbols`, () => { const opts = DEFAULT_COMPARE_OPTIONS diff --git a/packages/db/tests/temporal-index-order-oracle.test.ts b/packages/db/tests/temporal-index-order-oracle.test.ts new file mode 100644 index 0000000000..84c73f9d05 --- /dev/null +++ b/packages/db/tests/temporal-index-order-oracle.test.ts @@ -0,0 +1,328 @@ +import { describe, expect, it, vi } from 'vitest' +import { Temporal } from 'temporal-polyfill' +import { createCollection } from '../src/collection/index.js' +import { BasicIndex } from '../src/indexes/basic-index.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { PropRef } from '../src/query/ir.js' +import { orderedEntriesArray, withOracleCleanup } from './utils.js' + +/** + * A ZonedDateTime's ordered position is its instant. Two zones at the same + * instant share an ordered position but remain distinct equality values. + * Authority: Temporal.ZonedDateTime.compare/equals, the established WHERE + * evaluator contract in query/compiler/evaluators-oracle.test.ts, and the + * ordered Collection and IndexInterface contracts in docs/reference. + * + * This bounded oracle covers one polyfill copy, two opposing offset zones, + * one equal-instant pair, a one-nanosecond successor, and negative instants + * around the epoch. Equal-instant peers may appear in either key order. It + * checks direct BasicIndex/BTreeIndex adds, builds, and value cursors, one + * removal/insertion/update continuation, then the public ordered Collection + * snapshot with limit 1. It does not claim arbitrary Temporal + * kinds, time-zone transitions, longer mutation histories, public/provider + * cursor continuation, or live-query publication. + */ + +type Row = { id: string; when: Temporal.ZonedDateTime } +type Fixture = Readonly<{ + id: string + text: string + epochNanoseconds: bigint + equalityClass: string +}> + +// The supplied epoch nanoseconds are the model's order labels. Neither the +// index's normalization nor its comparator computes an expected position. +// The first and second rows have the same instant in different zones; the +// third is only one nanosecond later. The fourth has later instant but earlier +// local clock text than the first, so textual index order is observably wrong. +const fixtures: ReadonlyArray = [ + { + id: `b-plus-fourteen`, + text: `2024-01-15T23:00:00+14:00[+14:00]`, + epochNanoseconds: 1_705_309_200_000_000_000n, + equalityClass: `plus-fourteen`, + }, + { + id: `c-utc`, + text: `2024-01-15T09:00:00+00:00[+00:00]`, + epochNanoseconds: 1_705_309_200_000_000_000n, + equalityClass: `utc`, + }, + { + id: `a-one-nanosecond-later`, + text: `2024-01-15T09:00:00.000000001+00:00[+00:00]`, + epochNanoseconds: 1_705_309_200_000_000_001n, + equalityClass: `one-nanosecond-later`, + }, + { + id: `d-minus-five`, + text: `2024-01-15T05:00:00-05:00[-05:00]`, + epochNanoseconds: 1_705_312_800_000_000_000n, + equalityClass: `minus-five`, + }, +] + +// Textual sorting of signed decimal epoch values puts -10 before -100. These +// pre-epoch neighbors reject that tempting numeric-string repair, even though +// their ZonedDateTime display strings happen to be in chronological order. +const negativeFixtures: ReadonlyArray = [ + { + id: `a-minus-ten`, + text: `1969-12-31T23:59:59.999999990+00:00[+00:00]`, + epochNanoseconds: -10n, + equalityClass: `minus-ten`, + }, + { + id: `b-minus-hundred`, + text: `1969-12-31T23:59:59.999999900+00:00[+00:00]`, + epochNanoseconds: -100n, + equalityClass: `minus-hundred`, + }, + { + id: `c-zero`, + text: `1970-01-01T00:00:00+00:00[+00:00]`, + epochNanoseconds: 0n, + equalityClass: `zero`, + }, +] + +function rowsFromFixtures( + samples: ReadonlyArray = fixtures, +): Array { + return samples.map(({ id, text, epochNanoseconds }) => { + const when = Temporal.ZonedDateTime.from(text) + // This checks fixture construction, not the production comparison rule. + expect(when.epochNanoseconds).toBe(epochNanoseconds) + return { id, when } + }) +} + +// The order contract fixes instant order, but gives no public tie break for +// different zones at the same instant. A prefix may choose any members at its +// boundary, provided it includes every row at an earlier instant. +function expectOrderedPrefix( + actual: Array | undefined, + count: number, + samples: ReadonlyArray = fixtures, +): void { + expect(actual).toHaveLength(count) + const byId = new Map(samples.map((sample) => [sample.id, sample])) + const instants = actual!.map((id) => { + if (typeof id !== `string`) + throw new Error(`index returned a non-string key`) + const sample = byId.get(id) + expect(sample).toBeDefined() + return sample!.epochNanoseconds + }) + expect(new Set(actual).size).toBe(count) + const sortedInstants = samples + .map(({ epochNanoseconds }) => epochNanoseconds) + .sort((left, right) => (left < right ? -1 : left > right ? 1 : 0)) + const boundary = sortedInstants[count - 1]! + for (const sample of samples) { + if (sample.epochNanoseconds < boundary) { + expect(actual).toContain(sample.id) + } + } + for (let index = 0; index < instants.length; index++) { + expect(instants[index]! <= boundary).toBe(true) + if (index > 0) expect(instants[index - 1]! <= instants[index]!).toBe(true) + } +} + +const indexTypes = [ + [`BasicIndex`, BasicIndex], + [`BTreeIndex`, BTreeIndex], +] as const + +it(`accepts either equal-instant boundary row but rejects a later instant`, () => { + // A lexical index chose d-minus-five at the first-row checkpoint. The + // checker must reject that result while allowing either earliest zone. + expect(() => expectOrderedPrefix([`b-plus-fourteen`], 1)).not.toThrow() + expect(() => expectOrderedPrefix([`c-utc`], 1)).not.toThrow() + expect(() => expectOrderedPrefix([`d-minus-five`], 1)).toThrow() +}) + +describe.each(indexTypes)( + `%s ZonedDateTime indexed order`, + (_name, IndexType) => { + it.each([`add`, `build`] as const)( + `refines instant order and zone-sensitive equality after %s`, + (construction) => { + const rows = rowsFromFixtures() + const index = new IndexType(1, new PropRef([`when`])) + + // The legal direct history either adds each unique source key or builds + // once from that same source snapshot. No source key changes afterward. + if (construction === `add`) { + for (const row of rows) index.add(row.id, row) + } else { + index.build(rows.map((row): [string, Row] => [row.id, row])) + } + + // Exact lookup and ordered position are distinct observations. An + // epoch-only key would incorrectly merge the two equal-instant zones. + for (const row of rows) { + const fixture = fixtures.find(({ id }) => id === row.id)! + const expected = new Set( + fixtures + .filter( + ({ equalityClass }) => equalityClass === fixture.equalityClass, + ) + .map(({ id }) => id), + ) + const equalButDistinct = Temporal.ZonedDateTime.from( + row.when.toString(), + ) + expect(equalButDistinct).not.toBe(row.when) + expect(index.equalityLookup(equalButDistinct)).toEqual(expected) + } + expect(rows[0]!.when.epochNanoseconds).toBe( + rows[1]!.when.epochNanoseconds, + ) + expect(index.equalityLookup(rows[1]!.when).has(rows[0]!.id)).toBe(false) + + // Returned key arrays preserve instant order and multiplicity at the + // direct IndexInterface checkpoint. Limit 1 detects a later instant + // before the equal-instant first group, without imposing a tie break. + expectOrderedPrefix(index.takeFromStart(rows.length), rows.length) + expectOrderedPrefix(index.takeFromStart(1), 1) + // A cursor at the shared instant must skip both zones, while retaining + // the one-nanosecond successor. An ordered key with a zone suffix would + // incorrectly leave one equal-instant row beyond this cursor. + expect(index.take(2, rows[0]!.when)).toEqual([ + `a-one-nanosecond-later`, + `d-minus-five`, + ]) + }, + ) + + it(`retains an equal-instant peer through removal, insertion, and zone update`, () => { + const [plusFourteen, utc, nanosecondLater] = rowsFromFixtures() + const index = new IndexType(1, new PropRef([`when`])) + for (const row of [plusFourteen!, utc!, nanosecondLater!]) { + index.add(row.id, row) + } + + // A source removal changes membership but leaves the peer at the same + // instant. The peer becomes the ordered representative of that position. + index.remove(plusFourteen!.id, plusFourteen) + expect(index.takeFromStart(3)).toEqual([ + `c-utc`, + `a-one-nanosecond-later`, + ]) + expect(index.equalityLookup(plusFourteen!.when)).toEqual(new Set()) + expect(index.equalityLookup(utc!.when)).toEqual(new Set([`c-utc`])) + + // The ordered tree or array must now hold a live value at this position. + // This storage invariant catches a stale BTree representative even when + // its instant still compares equal to the surviving public row. + expect(orderedEntriesArray(index)[0]?.[0]).toBe(utc!.when) + + const inserted: Fixture = { + id: `one-nanosecond-earlier`, + text: `2024-01-15T08:59:59.999999999+00:00[+00:00]`, + epochNanoseconds: fixtures[1]!.epochNanoseconds - 1n, + equalityClass: `one-nanosecond-earlier`, + } + const [earlier] = rowsFromFixtures([inserted]) + index.add(earlier!.id, earlier) + expectOrderedPrefix(index.takeFromStart(3), 3, [ + inserted, + fixtures[1]!, + fixtures[2]!, + ]) + expect(index.take(2, earlier!.when)).toEqual([ + `c-utc`, + `a-one-nanosecond-later`, + ]) + + // Updating that peer to another zone at the same supplied instant moves + // its exact lookup bucket without changing its ordered position. + const moved = { + id: utc!.id, + when: Temporal.ZonedDateTime.from(`2024-01-15T04:00:00-05:00[-05:00]`), + } + expect(moved.when.epochNanoseconds).toBe(fixtures[1]!.epochNanoseconds) + index.update(utc!.id, utc, moved) + expect(index.equalityLookup(utc!.when)).toEqual(new Set()) + expect(index.equalityLookup(moved.when)).toEqual(new Set([`c-utc`])) + expectOrderedPrefix(index.takeFromStart(3), 3, [ + inserted, + fixtures[1]!, + fixtures[2]!, + ]) + expect(index.take(2, utc!.when)).toEqual([`a-one-nanosecond-later`]) + }) + + it.each([`add`, `build`] as const)( + `orders negative nanosecond instants after %s`, + (construction) => { + const rows = rowsFromFixtures(negativeFixtures) + const index = new IndexType(1, new PropRef([`when`])) + if (construction === `add`) { + for (const row of rows) index.add(row.id, row) + } else { + index.build(rows.map((row): [string, Row] => [row.id, row])) + } + // The model sorts supplied signed BigInts, not their decimal strings. + expectOrderedPrefix( + index.takeFromStart(rows.length), + rows.length, + negativeFixtures, + ) + }, + ) + + it(`publishes the earliest instant from an indexed ordered Collection snapshot`, async () => { + const rows = rowsFromFixtures() + const collection = createCollection({ + id: `temporal-index-order-${_name}`, + getKey: (row) => row.id, + autoIndex: `off`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of rows) write({ type: `insert`, value: row }) + commit() + markReady() + }, + }, + }) + + const orderBy = [ + { + expression: new PropRef([`when`]), + compareOptions: { + direction: `asc` as const, + nulls: `first` as const, + }, + }, + ] + const readFirst = (optimizedOnly: boolean) => + collection + .currentStateAsChanges({ orderBy, limit: 1, optimizedOnly }) + ?.map(({ value }) => value.id) + + await withOracleCleanup(async () => { + await collection.stateWhenReady() + expect(collection.indexes.size).toBe(0) + // The unindexed public scan checks the same ordering law through a + // separate production path before index creation. + expectOrderedPrefix(readFirst(false), 1) + + const index = collection.createIndex((row) => row.when, { + indexType: IndexType, + }) + const take = vi.spyOn(index, `takeFromStart`) + // optimizedOnly forbids a scan fallback; the spy shows that the intended + // index path supplied this public snapshot after the build. + const indexed = readFirst(true) + expect(take).toHaveBeenCalled() + expectOrderedPrefix(indexed, 1) + }, [() => collection.cleanup()]) + }) + }, +) From 1265abeb5743f873d78e7be742064146087e005b Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 18:29:22 -0600 Subject: [PATCH 12/46] fix(db): match Temporal equality across zone aliases --- .changeset/fix-temporal-index-order.md | 2 +- docs/contributing/oracle-coverage.md | 21 +- .../comparison-unit-retirement-round-6.md | 11 +- .../oracle-reviews/temporal-equality-alias.md | 69 ++++++ .../oracle-reviews/temporal-index-order.md | 53 ++-- packages/db/package.json | 2 +- packages/db/src/indexes/base-index.ts | 21 +- packages/db/src/indexes/basic-index.ts | 19 +- packages/db/src/indexes/btree-index.ts | 13 +- packages/db/src/utils/comparison.ts | 78 +++++- .../in-evaluator-oracle.property.test.ts | 105 ++++++-- .../temporal-equality-alias-oracle.test.ts | 233 ++++++++++++++++++ .../tests/temporal-index-order-oracle.test.ts | 30 ++- 13 files changed, 584 insertions(+), 73 deletions(-) create mode 100644 docs/contributing/oracle-reviews/temporal-equality-alias.md create mode 100644 packages/db/tests/temporal-equality-alias-oracle.test.ts diff --git a/.changeset/fix-temporal-index-order.md b/.changeset/fix-temporal-index-order.md index d1817e5c77..5dc6b9970c 100644 --- a/.changeset/fix-temporal-index-order.md +++ b/.changeset/fix-temporal-index-order.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Order Temporal values by their calendar or instant semantics in indexed reads. Keep exact equality distinct for ZonedDateTimes in different zones at the same instant. Prevent one Temporal value from changing the comparator used for later values. +Order Temporal values by their calendar or instant semantics in indexed reads. Match ZonedDateTime equality across equivalent time-zone aliases while keeping distinct zones separate, and preserve custom index comparator inputs. Prevent one Temporal value from changing the comparator used for later values. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index f2a7b82bf6..49c0a78f67 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -481,9 +481,24 @@ BasicIndex and BTreeIndex ordered takes after add/build and one mutation continuation, plus an optimized-only public Collection snapshot with an explicit index. Supplied epoch nanoseconds establish order without promising a key tie break for equal instants; distinct zones retain exact lookup -identities. Custom comparators, mixed value domains, other Temporal kinds, -public WHERE ranges, provider cursors, longer histories, and live-query -publication require separate witnesses. +identities. A focused callback witness preserves the former string operands +for a custom comparator over two distinct pre-epoch values. Custom comparators +that distinguish values in one query-equality class, mixed value domains, +other Temporal kinds, public WHERE ranges, provider cursors, longer histories, +and live-query publication require separate witnesses. + +The [Temporal alias equality oracle](../../packages/db/tests/temporal-equality-alias-oracle.test.ts) +and its [review record](oracle-reviews/temporal-equality-alias.md) own the +bounded `ZonedDateTime.equals()` partition for UTC/Etc/UTC aliases, a distinct +fixed-offset zone, adjacent nanoseconds, and a different calendar. Direct +BasicIndex/BTreeIndex add, build, and removal checks compare independent +fixture-assigned classes. Unindexed and optimized-only indexed Collection +snapshots check public `eq`; the `in` evaluator oracle checks constant and +row-supplied alias lists. A plain object with a forged Temporal tag and a +genuine value with a shadowed zone accessor must stay outside the UTC class. +Other zone aliases, mixed Temporal implementations, joins, grouping, query +identity, live-query publication, and provider paths remain unproved. Direct +D2 `groupBy` has its separately approved kind-and-string identity policy. The [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts) also has a [cross-realm Date equality matrix](oracle-reviews/cross-realm-date-where-publication.md). diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md index 5ede87cc42..49d6af5723 100644 --- a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md @@ -73,7 +73,7 @@ verdict follows after commit. | ORC-008 | Not applicable: tuple and instant models have no mutable state. Fresh modules control production cache history, not reference-model state. | | ORC-009 | Date tuples and nanosecond offsets are model input labels. Comparator zero describes order position, while `eq` identity retains the zone. | | ORC-010 | The shared harness keeps the original failure checkpoint through shrinking. The direct tests own no cleanup resource. | -| ORC-011 | The hostile text-order and millisecond formulations distinguish shared semantic mistakes. The index/public path has its own oracle. | +| ORC-011 | Not triggered: the reviewer named no plausible fault shared by the calendar/instant model and production within this bounded lane. Text-order and millisecond injections are ORC-006 wrong-result controls. The index/public path has its own oracle. | | ORC-012 | This record contains the source ledger, RED/GREEN evidence, hostile verdict, limits, and committed-head recheck. | | ORC-013 | Equal instants in different zones and adjacent nanoseconds distinguish text order and millisecond truncation from the promised instant law. | | ORC-014 | Not applicable: this lane has no controlled provider or claim across a host boundary. | @@ -85,4 +85,11 @@ properties as selectors 39 and 40 with seed `20260911` and path `0`. Each requested replay passed with no type errors; 39 other properties were skipped in each run. After the final index oracle edit, the full DB suite passed 12,154 tests in 278 files with no type errors. The DB package build passed. -The exact committed-head review remains pending. +At committed head `8b04592724fb943b04a56c6682757448d26a8ff6`, the fresh +hostile reviewer `/root/exact_temporal_comparison_review` found no lost +assertion in the two retired blocks. It reconstructed the original PlainDate +and ZonedDateTime operands and checked all four assertions at the direct +`ascComparator` return. Its focused run passed 112 comparison tests and 149 +evaluator/index tests with no type errors. It replayed properties 39 and 40 +with seed `20260911` and path `0`. The reviewer corrected the ORC-011 +classification above. Cross-realm Temporal values remain outside this verdict. diff --git a/docs/contributing/oracle-reviews/temporal-equality-alias.md b/docs/contributing/oracle-reviews/temporal-equality-alias.md new file mode 100644 index 0000000000..277981ca6d --- /dev/null +++ b/docs/contributing/oracle-reviews/temporal-equality-alias.md @@ -0,0 +1,69 @@ +# Temporal alias equality + +Changelog #1519 promises `ZonedDateTime` query equality consistent with +Temporal's `.equals()`. At one instant, `[UTC]` and `[Etc/UTC]` are equal, +while `[+00:00]` is a distinct zone. Exact nanoseconds and calendar identity +also matter. The new oracle assigns those classes in its fixture, then checks +direct BasicIndex and BTreeIndex lookups after add, build, and removal, plus +unindexed and optimized-only indexed public `eq` snapshots. The existing `in` +evaluator oracle now uses `.equals()` as its independent Temporal reference +and checks both constant and row-supplied lists, including the forged and +shadowed controls. + +## Failure and repair + +At frozen commit `8b04592724fb943b04a56c6682757448d26a8ff6`, exact +keys used display strings. Eight new direct and public alias checks failed at +their equality comparisons; the distinct-zone fixture control passed. +Four `in` checks also failed against the corrected reference, with no type +errors. The shared production cause was `normalizeValue`, so a repair limited +to an index would leave unindexed `eq` and `in` wrong. + +The repair keys ordinary ZonedDateTimes by kind, epoch nanoseconds, +canonical zone ID, and calendar ID. It reads those three fields through +prototype getters so an own property cannot change a Temporal instance's +equality class. A plain object with a forged Temporal tag remains an ordinary +object. Before that guard, the forged object and a genuine fixed-offset value +whose `timeZoneId` property shadowed its internal zone both matched a UTC +probe; the new direct-index and public-scan control rejected them. + +An independent read-only sweep compared the proposed zone key with native +`.equals()` on 177,906 same-instant pairs across 597 host zone IDs, finding +no partition mismatch. This bounds evidence for that host, not a proof for +every Temporal implementation or time-zone database. The focused index, +evaluator, and Temporal runs passed 453 tests with no type errors after the +repair. The full DB suite passed 12,171 tests in 279 files with no type +errors. The exact-commit review follows after commit. + +## Oracle guide audit + +| Requirement | Evidence and limit | +| --- | --- | +| ORC-001 | Changelog #1519 and Temporal `.equals()` authorize the equality class; the opening oracle comment limits implementations, paths, and observations. | +| ORC-002 | Fixture-assigned classes predict key sets without calling production normalization. A separate native `.equals()` check calibrates the fixture. | +| ORC-003 | Prose beside the model, add/build history, Collection driver, and assertions explains their roles. | +| ORC-004 | Fixed classes distinguish alias equality, distinct zone, adjacent nanosecond, calendar, forged tag, and shadowed accessor. No generated-history claim is made. | +| ORC-005 | Direct lookups observe both index types; an indexed public snapshot spies on `lookup`, and the unindexed snapshot checks the evaluator path after readiness. | +| ORC-006 | The frozen display-string design failed alias checks. The first tuple design failed forged-tag and shadowed-zone controls at direct lookups. A lexical `in` mutant fails its check. | +| ORC-007 | Not triggered by this fixed oracle. The adjacent `in` owner retains fixed and random generated campaigns. | +| ORC-008 | The fixture model is immutable and has no reference-model state. | +| ORC-009 | The model's class labels denote query equality, distinct from the order oracle's instant labels. | +| ORC-010 | Collection cleanup preserves the primary mismatch. Direct index checks own no external resource. | +| ORC-011 | The direct lookup and public scan cross different production paths, but share normalization. Native `.equals()` is the independent semantic calibration; this record does not claim a second full reference model. | +| ORC-012 | This record names the owner, RED controls, limits, and required exact-commit review. | +| ORC-013 | Alias equality versus fixed-offset inequality separates the candidate key from display-only and instant-only keys; forged and shadowed values reject structural impostors. | +| ORC-014 | The Collection uses controlled in-memory sync and ends its claim at the snapshot; no provider handoff is claimed. | + +## Limits and exact-commit review + +The oracle covers ordinary values from one polyfill copy and one instant. +Other IANA aliases, mixed Temporal implementations, joins, grouping, query +identity, live-query publication, provider paths, and longer histories need +their own witnesses. The `in` oracle adds generated list membership but not +those other boundaries. `Intl` canonicalization has a raw-ID fallback when +the host rejects a valid Temporal zone; alias equivalence in that case is +unproved. A custom comparator that distinguishes two query-equal aliases +also needs a separate law and index witness. Direct D2 `groupBy` has a +separately approved kind-and-string policy. + +Pending fresh hostile review of the exact final commit. diff --git a/docs/contributing/oracle-reviews/temporal-index-order.md b/docs/contributing/oracle-reviews/temporal-index-order.md index 0f9358c9f0..94d9600c40 100644 --- a/docs/contributing/oracle-reviews/temporal-index-order.md +++ b/docs/contributing/oracle-reviews/temporal-index-order.md @@ -5,8 +5,10 @@ does not exercise an index. A separate oracle checks the public index contract: an ordered take must follow `ZonedDateTime` instant order, while exact equality keeps zones distinct. The law comes from Temporal's `ZonedDateTime.compare` and `equals` semantics, changelog #1519, and the Collection's ordered-read and -IndexInterface contracts. No public contract fixes the key order of two rows -at the same instant, so the oracle accepts either tie order. +IndexInterface contracts. The live-query architecture also requires an +ordered comparator group to keep a representative from a live exact-value +bucket. No public contract fixes the key order of two rows at the same +instant, so the oracle accepts either tie order. ## Original failure and repair @@ -21,16 +23,17 @@ and public-snapshot comparisons. The exact same-instant zones retained separate equality lookups. The repair retains normalized exact keys for equality and stores raw Temporal -values for ordered positions. The BTree's comparator group tracks each exact -key's ordered value so removal can replace a departing representative. A -separate removal witness failed against the first repair because the BTree -retained the removed raw value as its representative. Public ordered output -was still correct under a stable comparator; the witness establishes a storage -invariant, not a second public-order defect. The repaired focused oracle passed -13 tests, including a wrong-result control, and the existing index refinement -oracle passed 412 tests. The final -DB suite passed 12,154 tests in 278 files with no type errors. The DB package -build passed. The exact-commit review is recorded below after commit. +values for ordered positions. A removal control caught the first repair +retaining a BTree representative whose exact-value bucket had departed. +The corrected BTree replaces that representative with one from a remaining +bucket. An initial hostile objection treated this as an unwarranted private +assertion; rechecking the normative architecture established its authority. +The existing index refinement oracle also rejects the version that removed +the replacement. The original repaired commit passed 13 focused tests and +the index refinement oracle's 412 tests, then the full DB suite passed +12,154 tests in 278 files. After the alias and comparator controls, the full +DB suite passed 12,171 tests in 279 files with no type errors. The final +exact-commit review remains pending. The independent fixture supplies epoch nanoseconds and zone-identity classes. It never calls the index's normalization or comparator to compute expected @@ -50,28 +53,32 @@ chosen index's `takeFromStart` to prove the intended path ran. A separate hostile reviewer traced the exact-key and ordered-value paths in both indexes, examined the original failure, and checked the repaired public index selection. It found no blocker within the bounded default-comparator -domain. It identified a callback compatibility risk: a custom `compareFn` -now receives raw Temporal values where it previously received normalized -strings. Published types accept `any` and do not promise that representation. -Custom comparators require a separate product decision and witness before a -claim about them. Mixed domains, other Temporal kinds, public WHERE ranges, -provider cursors, longer mutation histories, and live-query publication also -remain outside this oracle. Existing range optimization does not claim -Temporal WHERE correctness. +domain. It identified a callback compatibility regression: a custom +`compareFn` received raw Temporal objects where it previously received +normalized strings. A first follow-up kept the string type but supplied a +new epoch tuple; a pre-epoch callback witness rejected its wrong direct +ordered result in both index types. The repair now passes the former display +key to custom callbacks while storing raw values for index order. This is a +bounded compatibility decision; published types do not promise the operand +representation. A custom comparator that distinguishes two values in one +query-equality class remains open. Mixed domains, other Temporal kinds, +public WHERE ranges, provider cursors, longer histories, and live-query +publication also remain outside this oracle. Existing range optimization +does not claim Temporal WHERE correctness. ## Oracle guide audit | Requirement | Evidence and limit | | --- | --- | -| ORC-001 | The opening names instant order, zone-sensitive equality, authority, and the bounded domain. Equal-instant tie order is deliberately unspecified. | +| ORC-001 | The opening names instant order, zone-sensitive equality, live representatives, authority, and the bounded domain. Equal-instant tie order is deliberately unspecified. | | ORC-002 | Supplied BigInt instants and zone classes determine expected order and exact matches independently of normalization and comparison code. | | ORC-003 | The oracle describes its law, fixture model, legal actions, direct and public drivers, observations, and checkpoints beside executable code. | | ORC-004 | Not triggered: the bounded fixture matrix and one continuation are fixed histories, not a generated-history claim. The nearby invalid mixed-kind domain is excluded. | | ORC-005 | Direct takes and equality lookups observe both index implementations. An optimized-only Collection snapshot invokes the explicit index's `takeFromStart` after readiness. | -| ORC-006 | The original normalized-string design failed at direct ordered takes and the public `limit: 1` snapshot. A stable wrong-result control rejects that later-instant first row while allowing both legitimate equal-instant ties. The first repair's stale representative failed the internal invariant assertion. | +| ORC-006 | The original normalized-string design failed at direct ordered takes and the public `limit: 1` snapshot. A stable wrong-result control rejects that later-instant first row while allowing both legitimate equal-instant ties. The first BTree repair failed the live-bucket representative check; a tuple-fed custom comparator failed the pre-epoch take check. | | ORC-007 | Not triggered: this is a bounded fixed oracle, not an important generated property. The existing index refinement property keeps its own campaigns. | | ORC-008 | Not triggered: the fixture's independent model has no mutable reference state. | -| ORC-009 | The model's instant label represents order only; its zone class represents exact equality. One comparator position can contain two exact values. | +| ORC-009 | The model's instant label represents order only; its zone class represents exact equality. One comparator position can contain two exact values. The custom callback witness checks backward compatibility for its input text, not Temporal's default instant law. | | ORC-010 | `withOracleCleanup` preserves a primary public mismatch and releases the Collection. Direct indexes own no external resources. | | ORC-011 | The unindexed public scan is a second production formulation for the ordered result. The numeric BigInt model also distinguishes a shared display-string mistake. | | ORC-012 | This record contains RED/GREEN evidence, a hostile review, and scoped unresolved cells. An exact-commit verdict follows below. | diff --git a/packages/db/package.json b/packages/db/package.json index 734692d965..3b71220d32 100644 --- a/packages/db/package.json +++ b/packages/db/package.json @@ -23,7 +23,7 @@ "test": "vitest --run", "test:dist": "vitest run --config vitest.dist.config.ts", "test:facade-retention": "node --expose-gc --import tsx tests/facade-retention.probe.ts", - "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/temporal-index-order-oracle.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", + "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/temporal-index-order-oracle.test.ts tests/temporal-equality-alias-oracle.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", "bench:nested-includes": "vitest bench tests/query/includes-performance.bench.ts --run", "test:indexed-db:package": "pnpm --filter @tanstack/db... build && vitest run --config vitest.indexed-db.package.config.ts", "test:indexed-db:browser": "playwright test --config playwright.indexed-db.config.ts", diff --git a/packages/db/src/indexes/base-index.ts b/packages/db/src/indexes/base-index.ts index b8f3a59bb4..6862c14052 100644 --- a/packages/db/src/indexes/base-index.ts +++ b/packages/db/src/indexes/base-index.ts @@ -1,7 +1,12 @@ import { compileSingleRowExpression } from '../query/compiler/evaluators.js' import { comparisonFunctions } from '../query/builder/functions.js' import { DEFAULT_COMPARE_OPTIONS, deepEquals } from '../utils.js' -import { makeCheckedComparator, makeComparator } from '../utils/comparison.js' +import { + indexCustomComparatorValue, + indexOrderValue, + makeCheckedComparator, + makeComparator, +} from '../utils/comparison.js' import { codedMessage, codedWarning, devBuild } from '../error-message.js' import type { CompiledSingleRowExpression } from '../query/compiler/evaluators.js' import type { RangeQueryOptions } from './btree-index.js' @@ -147,8 +152,15 @@ export abstract class BaseIndex< this.expression = expression this.compareOptions = options?.compareOptions ?? DEFAULT_COMPARE_OPTIONS this.hasCustomComparator = options?.compareFn != null + const compareFn = options?.compareFn this.compareFn = makeCheckedComparator( - options?.compareFn ?? makeComparator(this.compareOptions), + compareFn == null + ? makeComparator(this.compareOptions) + : (a, b) => + compareFn( + indexCustomComparatorValue(a), + indexCustomComparatorValue(b), + ), ) this.name = name this.initialize(options) @@ -207,6 +219,11 @@ export abstract class BaseIndex< return !this.hasCustomComparator } + /** Temporal uses raw values for order while retaining normalized exact keys. */ + protected orderValue(value: unknown, exactKey: unknown): unknown { + return indexOrderValue(value, exactKey) + } + protected addRangeValue(value: unknown): void { const domain = rangeValueDomain(value) if (domain === undefined) return diff --git a/packages/db/src/indexes/basic-index.ts b/packages/db/src/indexes/basic-index.ts index fd995b832f..41a2649f71 100644 --- a/packages/db/src/indexes/basic-index.ts +++ b/packages/db/src/indexes/basic-index.ts @@ -2,7 +2,6 @@ import { compareKeys } from '@tanstack/db-ivm' import { areSameValueZeroEqual, indexExactKey, - indexOrderValue, normalizeValue, } from '../utils/comparison.js' import { @@ -78,7 +77,7 @@ export class BasicIndex< this.addToBucket( key, normalizedValue, - indexOrderValue(indexedValue, normalizedValue), + this.orderValue(indexedValue, normalizedValue), ) this.addRangeValue(indexedValue) @@ -125,7 +124,7 @@ export class BasicIndex< this.removeFromBucket( key, normalizedValue, - indexOrderValue(indexedValue, normalizedValue), + this.orderValue(indexedValue, normalizedValue), ) this.removeRangeValue(indexedValue) @@ -199,10 +198,10 @@ export class BasicIndex< this.removeFromBucket( key, oldValue, - indexOrderValue(oldIndexedValue, oldValue), + this.orderValue(oldIndexedValue, oldValue), ) this.removeRangeValue(oldIndexedValue) - this.addToBucket(key, newValue, indexOrderValue(newIndexedValue, newValue)) + this.addToBucket(key, newValue, this.orderValue(newIndexedValue, newValue)) this.addRangeValue(newIndexedValue) this.indexedKeys.add(key) } @@ -221,7 +220,7 @@ export class BasicIndex< entriesArray.push({ key, value, - orderValue: indexOrderValue(indexedValue, value), + orderValue: this.orderValue(indexedValue, value), }) this.addRangeValue(indexedValue) this.indexedKeys.add(key) @@ -318,8 +317,8 @@ export class BasicIndex< const normalizedFrom = normalizeValue(from) const normalizedTo = normalizeValue(to) - const orderFrom = indexOrderValue(from, normalizedFrom) - const orderTo = indexOrderValue(to, normalizedTo) + const orderFrom = this.orderValue(from, normalizedFrom) + const orderTo = this.orderValue(to, normalizedTo) const hasFrom = `from` in options const hasTo = `to` in options @@ -376,7 +375,7 @@ export class BasicIndex< */ take(n: number, from: any, filterFn?: (key: TKey) => boolean): Array { const normalizedFrom = normalizeValue(from) - const orderFrom = indexOrderValue(from, normalizedFrom) + const orderFrom = this.orderValue(from, normalizedFrom) let startIdx = findInsertPositionInArray( this.sortedValues, orderFrom, @@ -402,7 +401,7 @@ export class BasicIndex< filterFn?: (key: TKey) => boolean, ): Array { const normalizedFrom = normalizeValue(from) - const orderFrom = indexOrderValue(from, normalizedFrom) + const orderFrom = this.orderValue(from, normalizedFrom) let startIdx = findInsertPositionInArray(this.sortedValues, orderFrom, this.compareFn) - 1 diff --git a/packages/db/src/indexes/btree-index.ts b/packages/db/src/indexes/btree-index.ts index 23573d44a8..c83f1bc1c8 100644 --- a/packages/db/src/indexes/btree-index.ts +++ b/packages/db/src/indexes/btree-index.ts @@ -4,7 +4,6 @@ import { BTree } from '../utils/btree.js' import { areSameValueZeroEqual, denormalizeUndefined, - indexOrderValue, normalizeForBTree, } from '../utils/comparison.js' import { codedMessage, devBuild } from '../error-message.js' @@ -90,7 +89,7 @@ export class BTreeIndex< this.addToBucket( key, normalizedValue, - indexOrderValue(indexedValue, normalizedValue), + this.orderValue(indexedValue, normalizedValue), ) this.addRangeValue(indexedValue) @@ -208,7 +207,7 @@ export class BTreeIndex< this.removeFromBucket(key, oldValue) this.removeRangeValue(oldIndexedValue) - this.addToBucket(key, newValue, indexOrderValue(newIndexedValue, newValue)) + this.addToBucket(key, newValue, this.orderValue(newIndexedValue, newValue)) this.addRangeValue(newIndexedValue) this.indexedKeys.add(key) } @@ -300,10 +299,10 @@ export class BTreeIndex< const hasTo = `to` in options const fromKey = hasFrom - ? indexOrderValue(from, normalizeForBTree(from)) + ? this.orderValue(from, normalizeForBTree(from)) : this.orderedEntries.minKey() const toKey = hasTo - ? indexOrderValue(to, normalizeForBTree(to)) + ? this.orderValue(to, normalizeForBTree(to)) : this.orderedEntries.maxKey() this.orderedEntries.forRange( @@ -377,7 +376,7 @@ export class BTreeIndex< take(n: number, from: any, filterFn?: (key: TKey) => boolean): Array { const nextPair = (k?: any) => this.orderedEntries.nextHigherPair(k) // Normalize the from value - const normalizedFrom = indexOrderValue(from, normalizeForBTree(from)) + const normalizedFrom = this.orderValue(from, normalizeForBTree(from)) return this.takeInternal(n, nextPair, normalizedFrom, filterFn) } @@ -406,7 +405,7 @@ export class BTreeIndex< ): Array { const nextPair = (k?: any) => this.orderedEntries.nextLowerPair(k) // Normalize the from value - const normalizedFrom = indexOrderValue(from, normalizeForBTree(from)) + const normalizedFrom = this.orderValue(from, normalizeForBTree(from)) return this.takeInternal(n, nextPair, normalizedFrom, filterFn, true) } diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index f6f9381bf6..bb4d1220e0 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -238,6 +238,24 @@ function normalizedKey(kind: string, value: string): string { return `${NORMALIZED_KEY_PREFIX}${kind}:${value}` } +const canonicalTemporalZones = new Map() + +function canonicalTemporalZone(zoneId: string): string { + const cached = canonicalTemporalZones.get(zoneId) + if (cached !== undefined) return cached + + let canonical = zoneId + try { + canonical = new Intl.DateTimeFormat(`en-US`, { + timeZone: zoneId, + }).resolvedOptions().timeZone + } catch { + // Keep valid Temporal zones that this host's Intl does not recognize. + } + canonicalTemporalZones.set(zoneId, canonical) + return canonical +} + function normalizeBinary(value: Uint8Array): string { let bytes = `` for (let index = 0; index < value.byteLength; index++) { @@ -276,10 +294,52 @@ export function normalizeValue(value: any): any { if (timestamp !== undefined) return timestamp if (isTemporal(value)) { - return normalizedKey( - `temporal`, - `${value[Symbol.toStringTag]}:${value.toString()}`, - ) + const kind = value[Symbol.toStringTag] + if (kind === `Temporal.ZonedDateTime`) { + // Read Temporal's internal slots through its prototype getters. Own + // properties can shadow the public accessors without changing equals(). + const prototype = Object.getPrototypeOf(value) + if (prototype?.[Symbol.toStringTag] !== kind) return value + let epochNanoseconds: unknown + let timeZoneId: unknown + let calendarId: unknown + try { + epochNanoseconds = Object.getOwnPropertyDescriptor( + prototype, + `epochNanoseconds`, + )?.get?.call(value) + timeZoneId = Object.getOwnPropertyDescriptor( + prototype, + `timeZoneId`, + )?.get?.call(value) + calendarId = Object.getOwnPropertyDescriptor( + prototype, + `calendarId`, + )?.get?.call(value) + } catch { + // A borrowed prototype is not a branded Temporal instance. + return value + } + if ( + typeof epochNanoseconds === `bigint` && + typeof timeZoneId === `string` && + typeof calendarId === `string` + ) { + // Equality retains the instant, calendar, and canonical zone. A fixed + // +00:00 zone stays distinct from named UTC, while UTC aliases merge. + return normalizedKey( + `temporal`, + JSON.stringify([ + kind, + String(epochNanoseconds), + canonicalTemporalZone(timeZoneId), + calendarId, + ]), + ) + } + return value + } + return normalizedKey(`temporal`, `${kind}:${value.toString()}`) } // Normalize Uint8Arrays/Buffers to a string representation for Map key usage @@ -309,6 +369,16 @@ export function indexOrderValue(value: any, exactKey: any): any { return isTemporal(value) ? value : exactKey } +/** Supply custom comparators the Temporal display key used before raw ordering. */ +export function indexCustomComparatorValue(value: unknown): unknown { + return isTemporal(value) + ? normalizedKey( + `temporal`, + `${value[Symbol.toStringTag]}:${value.toString()}`, + ) + : value +} + /** Recover the exact lookup key from a stored ordered value. */ export function indexExactKey(value: any): any { return isTemporal(value) ? normalizeValue(value) : value diff --git a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts index f5ea6b05cb..19f0744427 100644 --- a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts +++ b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts @@ -19,7 +19,9 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * - A valid Date equals its millisecond timestamp, and another Date with the * same timestamp. * - Two Uint8Arrays (Buffers included) are equal when their bytes are equal. - * - A Temporal value equals a Temporal value of the same kind and string. + * - A Temporal value equals another of the same kind when its native + * `.equals()` relation holds. This includes equivalent time-zone aliases + * for ZonedDateTime, but excludes a distinct fixed-offset zone. * - Every other pair compares with `===`, so `-0` equals `0`, `1n` does not * equal `1`, and objects compare by reference. * @@ -31,10 +33,11 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * of each rule: equal and unequal strings, a string that begins with the * internal normalization prefix, booleans, `0`, `-0`, `1`, `NaN`, `1n`, a * valid Date and its timestamp, an invalid Date, equal and unequal byte - * arrays, a Buffer, Temporal values, one shared object and a structurally - * equal copy, `null` and `undefined`. Lists have zero to six items and may - * repeat items. The list is a constant `Value`, the shape that join demand - * builds, or a row field, so a list that changes between rows is also checked. + * arrays, a Buffer, Temporal values including ZonedDateTime zone aliases, + * a forged tag and shadowed zone accessor, one shared object and a + * structurally equal copy, `null` and `undefined`. Lists have zero to six + * items and may repeat. The list is a constant `Value`, the shape that join + * demand builds, or a row field, so changing lists are also checked. * * Production driver: compile `in` with the public IR and evaluate it for * each row, as a WHERE clause does. @@ -51,6 +54,24 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' const PROPERTY = `evaluators.in` const PREFIX = `\u0000tanstack-db:binary:` const shared = { id: 1 } +const utcZoned = Temporal.ZonedDateTime.from(`2024-01-15T09:00:00+00:00[UTC]`) +const aliasZoned = Temporal.ZonedDateTime.from( + `2024-01-15T09:00:00+00:00[Etc/UTC]`, +) +const offsetZoned = Temporal.ZonedDateTime.from( + `2024-01-15T09:00:00+00:00[+00:00]`, +) +const shadowedOffsetZoned = Temporal.ZonedDateTime.from( + `2024-01-15T09:00:00+00:00[+00:00]`, +) +Object.defineProperty(shadowedOffsetZoned, `timeZoneId`, { value: `UTC` }) +const forgedZoned = { + [Symbol.toStringTag]: `Temporal.ZonedDateTime`, + epochNanoseconds: utcZoned.epochNanoseconds, + timeZoneId: `UTC`, + calendarId: `iso8601`, + toString: () => `unrelated plain object`, +} const pool: ReadonlyArray = [ `a`, `b`, @@ -73,6 +94,11 @@ const pool: ReadonlyArray = [ Temporal.PlainDate.from(`2024-01-01`), Temporal.PlainDate.from(`2024-01-01`), Temporal.PlainTime.from(`10:00`), + utcZoned, + aliasZoned, + offsetZoned, + shadowedOffsetZoned, + forgedZoned, shared, { id: 1 }, null, @@ -88,15 +114,21 @@ function isBytes(value: unknown): value is Uint8Array { ) } -function isTemporalValue(value: unknown): value is { toString: () => string } { +function isTemporalValue(value: unknown): value is { + [Symbol.toStringTag]: string + equals: (other: unknown) => boolean +} { + const tag = + typeof value === `object` && value !== null + ? (value as { [Symbol.toStringTag]?: unknown })[Symbol.toStringTag] + : undefined return ( typeof value === `object` && value !== null && - typeof (value as { [Symbol.toStringTag]?: unknown })[Symbol.toStringTag] === - `string` && - (value as { [Symbol.toStringTag]: string })[Symbol.toStringTag].startsWith( - `Temporal.`, - ) + typeof (value as { equals?: unknown }).equals === `function` && + typeof tag === `string` && + tag.startsWith(`Temporal.`) && + Object.getPrototypeOf(value)?.[Symbol.toStringTag] === tag ) } @@ -127,9 +159,8 @@ function referenceEqual(left: unknown, right: unknown): boolean { return ( isTemporalValue(left) && isTemporalValue(right) && - (left as Record)[Symbol.toStringTag] === - (right as Record)[Symbol.toStringTag] && - left.toString() === right.toString() + left[Symbol.toStringTag] === right[Symbol.toStringTag] && + left.equals(right) ) } return asComparable(left) === asComparable(right) @@ -226,6 +257,52 @@ describe(`in evaluator`, () => { check(evaluateWithProduction, testCase) }) + it.each([false, true])( + `matches equivalent zone aliases in a %s list and rejects a distinct zone`, + (listFromRow) => { + // Temporal's own equality is the external semantic base, while the + // model below keeps the two expected outcomes explicit. + expect(utcZoned.equals(aliasZoned)).toBe(true) + expect(offsetZoned.equals(aliasZoned)).toBe(false) + check(evaluateWithProduction, { + value: aliasZoned, + list: [utcZoned], + listFromRow, + }) + check(evaluateWithProduction, { + value: offsetZoned, + list: [utcZoned], + listFromRow, + }) + }, + ) + + it.each([false, true])( + `rejects a forged tag and a shadowed zone in a %s list`, + (listFromRow) => { + expect(utcZoned.equals(shadowedOffsetZoned)).toBe(false) + for (const value of [forgedZoned, shadowedOffsetZoned]) { + check(evaluateWithProduction, { + value, + list: [utcZoned], + listFromRow, + }) + } + }, + ) + + it(`rejects display-string equality for equivalent zone aliases`, () => { + const lexical: Evaluate = ({ value, list }) => + list.some((candidate) => String(candidate) === String(value)) + expect(() => + check(lexical, { + value: aliasZoned, + list: [utcZoned], + listFromRow: false, + }), + ).toThrow(/in mismatch/) + }) + it(`rejects an evaluator that compares raw values`, () => { // Calibration: without normalization a Date never equals its timestamp // and equal byte arrays are different objects. diff --git a/packages/db/tests/temporal-equality-alias-oracle.test.ts b/packages/db/tests/temporal-equality-alias-oracle.test.ts new file mode 100644 index 0000000000..3e8e1dad6a --- /dev/null +++ b/packages/db/tests/temporal-equality-alias-oracle.test.ts @@ -0,0 +1,233 @@ +import { describe, expect, it, vi } from 'vitest' +import { Temporal } from 'temporal-polyfill' +import { createCollection } from '../src/collection/index.js' +import { BasicIndex } from '../src/indexes/basic-index.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { Func, PropRef, Value } from '../src/query/ir.js' +import { withOracleCleanup } from './utils.js' + +/** + * A ZonedDateTime's exact query equality follows Temporal's `.equals()`. + * Changelog #1519 promises that relation for `eq`, distinct from instant + * order. At one instant, UTC and Etc/UTC are equivalent zone aliases, while + * the fixed-offset +00:00 zone remains a different identity. + * + * This bounded oracle covers ordinary values from one polyfill copy and one + * instant. The fixture assigns equality classes independently of production + * normalization. Direct BasicIndex/BTreeIndex add and build histories compare + * exact key sets after each returned action. Public Collection reads compare + * an unindexed scan and an optimized-only indexed snapshot after readiness. + * A forged display tag and an own property shadowing a genuine Temporal zone + * must not join UTC's class. This oracle does not claim all IANA aliases, + * mixed implementations, joins, live-query publication, or provider behavior. + */ +type Row = { id: string; when: Temporal.ZonedDateTime } +type Sample = { + id: string + text: string + identity: `utc` | `offset` | `later` | `calendar` +} + +const samples: ReadonlyArray = [ + { + id: `utc`, + text: `2024-01-15T09:00:00+00:00[UTC]`, + identity: `utc`, + }, + { + id: `utc-alias`, + text: `2024-01-15T09:00:00+00:00[Etc/UTC]`, + identity: `utc`, + }, + { + id: `fixed-offset`, + text: `2024-01-15T09:00:00+00:00[+00:00]`, + identity: `offset`, + }, + { + id: `one-nanosecond-later`, + text: `2024-01-15T09:00:00.000000001+00:00[UTC]`, + identity: `later`, + }, + { + id: `gregory-calendar`, + text: `2024-01-15T09:00:00+00:00[UTC][u-ca=gregory]`, + identity: `calendar`, + }, +] + +function rows(): Array { + return samples.map(({ id, text }) => ({ + id, + when: Temporal.ZonedDateTime.from(text), + })) +} + +function expectedKeys(identity: Sample[`identity`]): Set { + return new Set( + samples + .filter((sample) => sample.identity === identity) + .map(({ id }) => id), + ) +} + +const aliasProbe = Temporal.ZonedDateTime.from( + `2024-01-15T09:00:00+00:00[Etc/UTC]`, +) +const offsetProbe = Temporal.ZonedDateTime.from( + `2024-01-15T09:00:00+00:00[+00:00]`, +) + +const indexTypes = [ + [`BasicIndex`, BasicIndex], + [`BTreeIndex`, BTreeIndex], +] as const + +it(`distinguishes equivalent aliases from a fixed-offset zone`, () => { + const [utc, alias, fixed, later, gregory] = rows() + // This checks the fixture against the external Temporal relation. Expected + // index and query sets still come from the assigned identity classes. + expect(utc!.when.equals(alias!.when)).toBe(true) + expect(utc!.when.equals(fixed!.when)).toBe(false) + expect(utc!.when.equals(later!.when)).toBe(false) + expect(utc!.when.equals(gregory!.when)).toBe(false) + expect(utc!.when.toString()).not.toBe(alias!.when.toString()) + expect(expectedKeys(`utc`)).toEqual(new Set([`utc`, `utc-alias`])) + expect(expectedKeys(`offset`)).toEqual(new Set([`fixed-offset`])) +}) + +describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { + it.each([`add`, `build`] as const)( + `finds the exact equality class after %s and one removal`, + (construction) => { + const source = rows() + const index = new IndexType(1, new PropRef([`when`])) + if (construction === `add`) { + const seenAliasKeys = new Set() + for (const [position, row] of source.entries()) { + index.add(row.id, row) + if (samples[position]!.identity === `utc`) seenAliasKeys.add(row.id) + expect(index.equalityLookup(aliasProbe)).toEqual(seenAliasKeys) + } + } else { + index.build(source.map((row): [string, Row] => [row.id, row])) + } + + // The query value is a fresh instance. A display-string key would omit + // UTC; an instant-only key would also include fixed-offset. + expect(index.equalityLookup(aliasProbe)).toEqual(expectedKeys(`utc`)) + expect(index.lookup(`eq`, aliasProbe)).toEqual(expectedKeys(`utc`)) + expect(index.inArrayLookup([aliasProbe])).toEqual(expectedKeys(`utc`)) + expect(index.equalityLookup(offsetProbe)).toEqual(expectedKeys(`offset`)) + + index.remove(source[0]!.id, source[0]!) + expect(index.equalityLookup(aliasProbe)).toEqual(new Set([`utc-alias`])) + expect(index.equalityLookup(offsetProbe)).toEqual(expectedKeys(`offset`)) + }, + ) + + it.each([false, true])( + `returns alias-equal rows in a public snapshot with optimizedOnly=%s`, + async (optimizedOnly) => { + const source = rows() + const collection = createCollection({ + id: `temporal-alias-equality-${_name}`, + getKey: (row) => row.id, + autoIndex: `off`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of source) write({ type: `insert`, value: row }) + commit() + markReady() + }, + }, + }) + const predicate = new Func(`eq`, [ + new PropRef([`when`]), + new Value(aliasProbe), + ]) + const readKeys = () => + new Set( + collection + .currentStateAsChanges({ where: predicate, optimizedOnly }) + ?.map(({ value }) => value.id), + ) + + await withOracleCleanup(async () => { + await collection.stateWhenReady() + expect(collection.indexes.size).toBe(0) + const lookup = optimizedOnly + ? vi.spyOn( + collection.createIndex((row) => row.when, { + indexType: IndexType, + }), + `lookup`, + ) + : undefined + const result = readKeys() + if (lookup) expect(lookup).toHaveBeenCalled() + expect(result).toEqual(expectedKeys(`utc`)) + }, [() => collection.cleanup()]) + }, + ) +}) + +it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () => { + const fake = { + [Symbol.toStringTag]: `Temporal.ZonedDateTime`, + epochNanoseconds: aliasProbe.epochNanoseconds, + timeZoneId: `UTC`, + calendarId: `iso8601`, + toString: () => `unrelated plain object`, + } + const shadowed = Temporal.ZonedDateTime.from( + `2024-01-15T09:00:00+00:00[+00:00]`, + ) + Object.defineProperty(shadowed, `timeZoneId`, { value: `UTC` }) + expect(aliasProbe.equals(shadowed)).toBe(false) + expect(() => Reflect.apply(aliasProbe.equals, aliasProbe, [fake])).toThrow() + + const source = [ + { id: `forged`, when: fake }, + { id: `shadowed`, when: shadowed }, + ] + for (const [, IndexType] of indexTypes) { + for (const row of source) { + // One row avoids asking the default ordering comparator to order a + // forged object and a genuine Temporal value against each other. + const index = new IndexType(1, new PropRef([`when`])) + index.add(row.id, row) + expect(index.equalityLookup(aliasProbe)).toEqual(new Set()) + } + } + + const collection = createCollection<{ id: string; when: unknown }, string>({ + id: `temporal-alias-spoof-control`, + getKey: (row) => row.id, + autoIndex: `off`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of source) write({ type: `insert`, value: row }) + commit() + markReady() + }, + }, + }) + await withOracleCleanup(async () => { + await collection.stateWhenReady() + const predicate = new Func(`eq`, [ + new PropRef([`when`]), + new Value(aliasProbe), + ]) + const keys = new Set( + collection + .currentStateAsChanges({ where: predicate }) + ?.map(({ value }) => value.id), + ) + expect(keys).toEqual(new Set()) + }, [() => collection.cleanup()]) +}) diff --git a/packages/db/tests/temporal-index-order-oracle.test.ts b/packages/db/tests/temporal-index-order-oracle.test.ts index 84c73f9d05..f062b01c5a 100644 --- a/packages/db/tests/temporal-index-order-oracle.test.ts +++ b/packages/db/tests/temporal-index-order-oracle.test.ts @@ -18,9 +18,11 @@ import { orderedEntriesArray, withOracleCleanup } from './utils.js' * around the epoch. Equal-instant peers may appear in either key order. It * checks direct BasicIndex/BTreeIndex adds, builds, and value cursors, one * removal/insertion/update continuation, then the public ordered Collection - * snapshot with limit 1. It does not claim arbitrary Temporal - * kinds, time-zone transitions, longer mutation histories, public/provider - * cursor continuation, or live-query publication. + * snapshot with limit 1. A separate direct-index witness preserves a custom + * comparator that accepts the string keys supplied before raw Temporal + * ordering was introduced. It does not claim arbitrary Temporal kinds, + * time-zone transitions, longer mutation histories, public/provider cursor + * continuation, or live-query publication. */ type Row = { id: string; when: Temporal.ZonedDateTime } @@ -215,9 +217,8 @@ describe.each(indexTypes)( expect(index.equalityLookup(plusFourteen!.when)).toEqual(new Set()) expect(index.equalityLookup(utc!.when)).toEqual(new Set([`c-utc`])) - // The ordered tree or array must now hold a live value at this position. - // This storage invariant catches a stale BTree representative even when - // its instant still compares equal to the surviving public row. + // The ordered group must keep a live representative after removal. + // The live-query architecture requires this for stable traversal. expect(orderedEntriesArray(index)[0]?.[0]).toBe(utc!.when) const inserted: Fixture = { @@ -275,6 +276,23 @@ describe.each(indexTypes)( }, ) + it(`retains a string-compatible custom comparator across negative instants`, () => { + const [minusTen, minusHundred] = rowsFromFixtures(negativeFixtures) + const index = new IndexType(1, new PropRef([`when`]), undefined, { + compareFn: (left: string, right: string) => left.localeCompare(right), + }) + + // This legal callback ordered normalized strings before the Temporal + // index repair. Negative decimal strings distinguish the former display + // operands from the new exact-equality tuple. + index.add(minusTen!.id, minusTen) + index.add(minusHundred!.id, minusHundred) + expect(index.takeFromStart(2)).toEqual([`b-minus-hundred`, `a-minus-ten`]) + expect(index.equalityLookup(minusHundred!.when)).toEqual( + new Set([`b-minus-hundred`]), + ) + }) + it(`publishes the earliest instant from an indexed ordered Collection snapshot`, async () => { const rows = rowsFromFixtures() const collection = createCollection({ From c4c07da42c00414b31689a1f57e9c862dd5ae8a2 Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 18:59:02 -0600 Subject: [PATCH 13/46] fix(db): preserve exact and ordered index semantics --- .../fix-custom-index-order-selection.md | 5 + .changeset/fix-temporal-index-order.md | 2 +- docs/contributing/oracle-coverage.md | 41 ++- .../comparison-unit-retirement-round-6.md | 8 + .../custom-comparator-signed-zero.md | 24 ++ .../custom-index-order-selection.md | 51 ++++ .../oracle-reviews/temporal-equality-alias.md | 54 ++-- .../oracle-reviews/temporal-index-order.md | 45 +++- .../utils-unit-retirement-round-7.md | 58 +++++ packages/db/package.json | 2 +- packages/db/src/collection/change-events.ts | 5 +- packages/db/src/indexes/base-index.ts | 6 +- packages/db/src/indexes/basic-index.ts | 187 ++++++++------ packages/db/src/indexes/btree-index.ts | 103 +++++--- packages/db/src/query/live/ARCHITECTURE.md | 8 +- packages/db/src/utils/comparison.ts | 77 ++++-- .../tests/comparison-oracle.property.test.ts | 19 +- ...ustom-index-order-selection-oracle.test.ts | 169 +++++++++++++ .../index-update-oracle.property.test.ts | 40 +++ .../in-evaluator-oracle.property.test.ts | 236 +++++++++++++++--- .../temporal-equality-alias-oracle.test.ts | 125 +++++++++- .../tests/temporal-index-order-oracle.test.ts | 128 +++++++++- .../db/tests/utils-oracle.property.test.ts | 73 +++++- packages/db/tests/utils.test.ts | 36 --- 24 files changed, 1246 insertions(+), 256 deletions(-) create mode 100644 .changeset/fix-custom-index-order-selection.md create mode 100644 docs/contributing/oracle-reviews/custom-comparator-signed-zero.md create mode 100644 docs/contributing/oracle-reviews/custom-index-order-selection.md create mode 100644 docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md create mode 100644 packages/db/tests/custom-index-order-selection-oracle.test.ts diff --git a/.changeset/fix-custom-index-order-selection.md b/.changeset/fix-custom-index-order-selection.md new file mode 100644 index 0000000000..29d70c28c8 --- /dev/null +++ b/.changeset/fix-custom-index-order-selection.md @@ -0,0 +1,5 @@ +--- +'@tanstack/db': patch +--- + +Use a custom-comparator index for exact lookup without selecting its order for an ordinary Collection `orderBy` read. diff --git a/.changeset/fix-temporal-index-order.md b/.changeset/fix-temporal-index-order.md index 5dc6b9970c..a951ea4a2c 100644 --- a/.changeset/fix-temporal-index-order.md +++ b/.changeset/fix-temporal-index-order.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Order Temporal values by their calendar or instant semantics in indexed reads. Match ZonedDateTime equality across equivalent time-zone aliases while keeping distinct zones separate, and preserve custom index comparator inputs. Prevent one Temporal value from changing the comparator used for later values. +Order Temporal values by their calendar or instant semantics in indexed reads. Match ZonedDateTime equality across equivalent time-zone aliases while keeping distinct zones separate. Keep Temporal equality when genuine values shadow display members, and reject ordinary objects with forged tags. Preserve custom index comparator operands and per-row order even when values share one query-equality class, including signed zero. Prevent one Temporal value from changing the comparator used for later values. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 49c0a78f67..b63b936fc6 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -442,8 +442,8 @@ generic visitor behavior outside the named collectors. | Backend Collection E2E | [shared provider suites](https://github.com/TanStack/db/tree/main/packages/db-collection-e2e/src/suites); [Electric registration](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/e2e/electric.e2e.test.ts); [Query registration](https://github.com/TanStack/db/blob/main/packages/query-db-collection/e2e/query.e2e.test.ts); [TrailBase registration](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/e2e/trailbase.e2e.test.ts) | Electric, Query, and TrailBase register reusable predicate, pagination, join, and live-update checks at their provider boundaries; Electric also registers the move suite. These are receiving paths for core query and Collection laws, not framework render cuts. | | Window-controller overlap | [shared infinite-query conformance](https://github.com/TanStack/db/blob/main/packages/db/tests/conformance/infinite-suite-oracle.ts), [React driver](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-conformance.test.tsx), [Vue driver](https://github.com/TanStack/db/blob/main/packages/vue-db/tests/infinite-query-conformance.test.ts), [Svelte driver](https://github.com/TanStack/db/blob/main/packages/svelte-db/tests/infinite-query-conformance.svelte.test.ts), [review record](oracle-reviews/dec-03-window-overlap.md) | Four or five ordered source rows, two window-settlement orders, and public snapshots after each settlement, a subscribed observer update, a detached controller read, and explicit recovery. Insertion and removal of the fifth row distinguish both continuation directions while an earlier error remains visible. The detached read follows a source change with no controller subscriber; its preloaded Collection retains the five-row window. The exact overlap uses the exported DB controller in each package realm. Framework hooks expose no controller preload, so this cell does not establish a hook scheduling path; direct hook paging remains in the adjacent shared scenarios. An unsubscribed controller has no notification claim. | | Infinite-query publication | [shared publication laws](https://github.com/TanStack/db/blob/main/packages/db/tests/conformance/infinite-suite-oracle.ts), [React render cuts](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-render-cuts-oracle.test.tsx), [Svelte hook tests](https://github.com/TanStack/db/blob/main/packages/svelte-db/tests/useLiveInfiniteQuery.svelte.test.ts), [review record](oracle-reviews/2026-10-05-infinite-first-paint.md) | Each driver records every value its framework published: React layout commits, Vue pre-flushes, and the Svelte construction value plus pre-effect runs. The shared suite checks that a synchronous source shows its ready first page on the first value, for a query callback and for a supplied collection that has not started, that every ready value in a fixed-source, fixed-query scenario equals the source prefix for its own page count, including an equal dependency replacement that keeps page depth, and that a mount requests an on-demand first window once, for a query callback and for a supplied collection whose finite window is wider than or equal to the one the hook needs, judged by the rows the on-demand source holds after mount. Under deferred acquisition, an on-demand first value is either the ready first page or a value with no rows, by framework: React and Svelte publish before the hook subscribes, Vue after. `preloaded-collection-first-paint` checks the documented route-loader pattern: a limited live query preloaded before mount shows its ready first page on the first value in every framework and asks the source for nothing more; a preload that does not count as demand never settles there. Pre-commit work has no shared form, because Vue setup is its commit. The React driver owns renders that never commit, StrictMode double renders (one first-window request on React 19), retries after GC, a mounted replacement whose startup error must reach the error boundary, and a collection kept from an abandoned render that enters terminal error before its query is selected again, each as under `useLiveQuery`. The React source ID reuse oracle drives the infinite hook too. Svelte owns construction and superseded recomputes before the subscribing effect. In both, the hook acquires an on-demand source as often as `useLiveQuery` does under the same history, for either input form, direct or behind a live-query Collection. On-demand requests from renders that never commit are now owned by the live-query deferred acquisition oracle; the React render-cut oracle checks that an abandoned render makes no acquisition. Open cells, owned here: React 18 StrictMode, where neither hook keeps refs across the double render; a React commit held after the 50 ms unsubscribed GC floor, where a controlled-premise controller witness shows no non-ready value but no React scheduling witness exists; ready values in scenarios that write the source or replace the query, which need a per-step source model; a join of eager and on-demand sources, which needs a zero-acquisition check before commit; a supplied collection with no `.limit`, which sends one unbounded request on `main` too, because the controller cannot narrow it before the source request; and Svelte `state_unsafe_mutation` when a start inside `$derived.by` synchronously writes rows a peer is waiting for, shared with Svelte's `useLiveQuery`, which a two-component probe reproduces and the deferred-acquisition follow-up removes. | -| Structural values and ordered primitives | [hash values](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-oracle.property.test.ts), [hash identity](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-identity-oracle.property.test.ts), [MultiSet consolidation](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/multiset-consolidate-oracle.property.test.ts), [hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-graph-oracle.property.test.ts), [mixed hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-mixed-graph-oracle.property.test.ts), [hash retry](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-failure-retry-oracle.property.test.ts), [comparison](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts), [deep equality](https://github.com/TanStack/db/blob/main/packages/db/tests/utils-oracle.property.test.ts), [cursor](https://github.com/TanStack/db/blob/main/packages/db/tests/cursor-oracle.property.test.ts), [indexes](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update-oracle.property.test.ts), [BTree Map](https://github.com/TanStack/db/blob/main/packages/db/tests/btree-map-oracle.test.ts), [query identity](https://github.com/TanStack/db/blob/main/packages/db/tests/query/identity-output-shape-oracle.test.ts), [LIKE semantics](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/evaluators-oracle.test.ts) | Independent flat values, graph topology, algebraic laws, Map/group/sort recomputation, expression denotation, custom comparator dispatch/reference identity/index compatibility, LIKE wildcard refinement, compiled output bags, declared value-pair agreement between D2 equality keys and IR equality operands, and Collection collation with exact public scan/index order and one-index reuse across repeated reads. The auto-index cells cross BasicIndex and BTreeIndex; scan cells omit indexes. Both paths cover lexical and numeric `en-US` locale defaults plus an explicit ordinary locale override of a numeric default. An unindexed six-row work cell bounds Collection collation resolution to at most two reads per ordered scan: one index-path check and one scan clause. Other locales and ordered histories are outside this owner. The identity pair grammar covers primitives, signed zero, NaN/invalid Date, valid Date/timestamp, binary/Buffer, Temporal, nested references, symbols, and functions; fixed and sampled pairs do not prove every JavaScript value or hash collision freedom. Ref proxies are expressions rather than literal values in this lane. The deep-equality owner also pins that `deepEquals` ignores Map and Set insertion order, RegExp `lastIndex`, array holes, and typed-array class, the rules draft equality keeps. It also pins that objects of another class differ, that plain and null-prototype objects are one class, that a class instance without enumerable keys equals only itself, that URLs compare by `href`, and that typed-array elements compare like numbers. The hash-values owner checks equal-hash consequences for independently constructed flat values. Distinct type identities are checked by the hash-identity owner through `equalHashValues`, including forced digest collisions; distinct 32-bit digests are not promised. The hash-identity owner checks `hash` and `equalHashValues` against one spec-level identity model: primitives with signed zero, NaN, and bigint; reference leaves (functions, registered handles, `File`, binary over 128 bytes); Date, binary, Temporal, and RegExp headers; array length and holes; Map and Set insertion order; and object keys in any order, for acyclic values and for cycles through arrays, Map values, Sets, and plain objects. Map/Set order sensitivity is pinned current behavior, not a promise. Arrays from another realm, RegExps with a `NaN` `lastIndex`, getters, proxies, other cycle carriers, and pairs that differ only in a back-edge target are outside its grammar; pinned cases keep equality's cross-realm length check and strict RegExp field comparison. `hash-work.test.ts` pins how visited values and array/RegExp headers count toward the structural work cap, and that equality copies a shared Map's entries once per distinct pair ([code-weight hash dispatch review](oracle-reviews/code-weight-hash-dispatch.md)). Custom string indexes do not optimize ordinary ranges, and custom cursors remain unsupported. The LIKE owner covers boolean string matching and bounded work, not nullish three-valued logic or a general Unicode collation contract. Unsupported composite cursors reject. The MultiSet consolidation owner checks keyed, single-type, and structural identity, signed zero and NaN in keyed and single-number data, the retained first record, zero-sum removal, and input record contents. It does not assert output order. Its keyed grammar includes cross-type key and value collisions, non-finite numeric keys, the `\|` delimiter, symbol and function identity, and a direct-object/object-tuple delimiter control from #1948. Eight named collision witnesses and both generated campaigns are RED on `c879ba6d` and GREEN with the repair ([#1948 review record](oracle-reviews/issue-1948-keyed-consolidation.md)); the [code-weight consolidation review](oracle-reviews/code-weight-multiset-consolidation.md) records the earlier exclusion. The keyed reference uses direct value/reference equality, including SameValueZero for numeric keys. The unkeyed fallback keeps its original key and value domains. The direct `MultiSet` owner does not prove which `@tanstack/db` query shapes produce primitive keyed values or mixed-type source keys; that needs a live-query production witness. The index owner checks lower-bound presence on bounded numeric histories after each add, update, delete, and build: an upper-only range ignores an unused `fromInclusive: false`, while an explicit exclusive `from` excludes its boundary. A source Map supplies expected sets for BasicIndex and BTreeIndex; wrong-output controls reject both boundary mistakes. It also checks exact `equalityLookup` and `lookup('eq')` after singleton case-insensitive `A → a → A` replacements and populated comparator-group updates, plus fresh-object same-group replacements in fixed and generated histories. These direct-index checks do not cover every JavaScript value, arbitrary comparators, or long Collection histories. The index owner enforces one invalid-comparator law across both BasicIndex and BTreeIndex: for each comparator, accepted numeric prefix (including the empty prefix), and probe operation (add, update, eq/range lookup, take, build), the operation throws exactly when it receives a `NaN` or non-number result and never when every comparison is valid. A `signed infinity` comparator is the valid control, and a custom collation `compare` supplied through `compareOptions` is checked the same way. Two BTree Map split histories check inserted-key reads, exact whole-range payloads/order, size, and extrema, proving split placement follows the insertion index without a post-mutation comparison. A rejected index add or remove leaves the index refining its accepted rows; update and build are not atomic. At the Collection boundary, both index types and both write paths (optimistic insert, sync commit) crash the collection: status becomes `error` and the next mutation throws, so a usable collection never holds rows its subscribers were not told about. Open: a sync source can still commit writes on a collection in `error` state, which stores unpublished rows; this is existing `markError` behavior and needs a lifecycle-owner witness. Comparator transitivity is outside this evidence. | -| WHERE predicate publication | [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts), [`in` evaluator](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts) ([review](oracle-reviews/perf-in-constant-set.md)) | An independent Kleene evaluator judges `eq`/`not`/`and`/`or` predicates over strings, a normalization-prefixed string, booleans, numbers, `NaN`, a valid Date, `null`, a missing field, and virtual fields. Snapshot histories with pending optimistic inserts compare a live query, direct subscribers with and without initial state, and `currentStateAsChanges`; change histories apply multi-key sync transactions, including reinsertion of a key deleted earlier, and compare every consumer's key set after each commit, on scan and `BasicIndex` paths. Peer subscribers share one field with different literals, plus one on a virtual field, so every published change must reach exactly the subscribers whose predicate it can satisfy; routing mutants that ignored the previous value, ignored the field, delivered a change twice, or routed while stale published rows awaited reconciliation fail here. Fixed witnesses cover a layout-only publication reaching a filtered subscriber as one empty batch, a filtered subscriber's empty Collection-readiness batch, retraction of a vanished row after eager cleanup and restart, and the new row plus exact subscriber update payload when a same-key row stays TRUE. A stale-live-value mutant survives the generated membership checks but fails the payload witness. A focused [property-visibility test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-prefilter-property-visibility.test.ts) compares the public unindexed snapshot with the enriched-row contract for inherited, non-enumerable, enumerable-own, and nested getter paths; the pre-fix stored-row shortcut failed three of its four controls. Other property-descriptor and stateful-getter histories remain outside those fixed controls. Hostile mutants for FALSE-for-UNKNOWN `eq`, `or` or number-literal subscription prefilters, a prefilter that ignores the previous value, a skipped readiness batch, a skip while stale published rows await reconciliation, and an unindexed snapshot scan that reads only synced rows while an optimistic insert or delete changes visibility passed the prior `@tanstack/db` suite and fail here. Change routing withholds a dropped row from an `eq` subscription before its sent key could be recorded, so a mutant that records dropped rows as sent is equivalent for routed predicates in this oracle. A focused `collection-subscription.test.ts` witness checks that a dropped insert or update, alone or beside a matching change, cannot advance a limited subscription's page offset under a routed `eq` and an unrouted `or` predicate; the unrouted cases beside a matching change kill that mutant, and the unrouted `alone` case does not. A loose-equality prefilter is an equivalent mutant: it only skips less. Skipping during truncate replay also survives; the replay's baseline diff re-derives the same retraction, so the guard keeps the prior dataflow without its own witness. Generated cleanup and restart histories for filtered subscribers remain open for the lifecycle publication owner. No-op updates, other comparison operators, Temporal and binary operands, joins, ordering, optimistic updates, and truncate are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. Change routing was later removed in favor of pooled live queries; the routing mutants above are historical, and the same peer laws now check plain subscription dispatch. The unindexed snapshot prefilter was restored for compiled queries as one stored-row `eq` test on a string or boolean literal. Its property-visibility test also covers the reverse direction: a stored row with an inherited or non-enumerable field must still match `isUndefined` on that field, which a scan that evaluated the full predicate on stored rows fails. The `in` evaluator owner checks `in` against an independent linear reference over Dates, `NaN`, `-0`, BigInt, bytes, Temporal values, objects and nullish items, for constant and row-supplied lists. | +| Structural values and ordered primitives | [hash values](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-oracle.property.test.ts), [hash identity](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-identity-oracle.property.test.ts), [MultiSet consolidation](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/multiset-consolidate-oracle.property.test.ts), [hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-graph-oracle.property.test.ts), [mixed hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-mixed-graph-oracle.property.test.ts), [hash retry](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-failure-retry-oracle.property.test.ts), [comparison](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts), [deep equality](https://github.com/TanStack/db/blob/main/packages/db/tests/utils-oracle.property.test.ts), [cursor](https://github.com/TanStack/db/blob/main/packages/db/tests/cursor-oracle.property.test.ts), [indexes](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update-oracle.property.test.ts), [BTree Map](https://github.com/TanStack/db/blob/main/packages/db/tests/btree-map-oracle.test.ts), [query identity](https://github.com/TanStack/db/blob/main/packages/db/tests/query/identity-output-shape-oracle.test.ts), [LIKE semantics](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/evaluators-oracle.test.ts) | Independent flat values, graph topology, algebraic laws, Map/group/sort recomputation, expression denotation, custom comparator dispatch/reference identity/index compatibility, LIKE wildcard refinement, compiled output bags, declared value-pair agreement between D2 equality keys and IR equality operands, and Collection collation with exact public scan/index order and one-index reuse across repeated reads. The auto-index cells cross BasicIndex and BTreeIndex; scan cells omit indexes. Both paths cover lexical and numeric `en-US` locale defaults plus an explicit ordinary locale override of a numeric default. An unindexed six-row work cell bounds Collection collation resolution to at most two reads per ordered scan: one index-path check and one scan clause. Other locales and ordered histories are outside this owner. The identity pair grammar covers primitives, signed zero, NaN/invalid Date, valid Date/timestamp, binary/Buffer, Temporal, nested references, symbols, and functions; fixed and sampled pairs do not prove every JavaScript value or hash collision freedom. Ref proxies are expressions rather than literal values in this lane. The deep-equality owner also pins that `deepEquals` ignores Map and Set insertion order, RegExp `lastIndex`, array holes, and typed-array class, the rules draft equality keeps. It also pins that objects of another class differ, that plain and null-prototype objects are one class, that a class instance without enumerable keys equals only itself, that URLs compare by `href`, and that typed-array elements compare like numbers. Its [Date/RegExp unit transfer](oracle-reviews/utils-unit-retirement-round-7.md) adds fixed and generated direct-return witnesses that Date differs from its ISO text and epoch number, and RegExp differs from its source text, while preserving the original copy, changed-time, changed-flag, and changed-source operands. Other cross-type pairs and public change-event traces remain outside that transfer. The hash-values owner checks equal-hash consequences for independently constructed flat values. Distinct type identities are checked by the hash-identity owner through `equalHashValues`, including forced digest collisions; distinct 32-bit digests are not promised. The hash-identity owner checks `hash` and `equalHashValues` against one spec-level identity model: primitives with signed zero, NaN, and bigint; reference leaves (functions, registered handles, `File`, binary over 128 bytes); Date, binary, Temporal, and RegExp headers; array length and holes; Map and Set insertion order; and object keys in any order, for acyclic values and for cycles through arrays, Map values, Sets, and plain objects. Map/Set order sensitivity is pinned current behavior, not a promise. Arrays from another realm, RegExps with a `NaN` `lastIndex`, getters, proxies, other cycle carriers, and pairs that differ only in a back-edge target are outside its grammar; pinned cases keep equality's cross-realm length check and strict RegExp field comparison. `hash-work.test.ts` pins how visited values and array/RegExp headers count toward the structural work cap, and that equality copies a shared Map's entries once per distinct pair ([code-weight hash dispatch review](oracle-reviews/code-weight-hash-dispatch.md)). Custom string indexes do not optimize ordinary ranges, and custom cursors remain unsupported. The LIKE owner covers boolean string matching and bounded work, not nullish three-valued logic or a general Unicode collation contract. Unsupported composite cursors reject. The MultiSet consolidation owner checks keyed, single-type, and structural identity, signed zero and NaN in keyed and single-number data, the retained first record, zero-sum removal, and input record contents. It does not assert output order. Its keyed grammar includes cross-type key and value collisions, non-finite numeric keys, the `\|` delimiter, symbol and function identity, and a direct-object/object-tuple delimiter control from #1948. Eight named collision witnesses and both generated campaigns are RED on `c879ba6d` and GREEN with the repair ([#1948 review record](oracle-reviews/issue-1948-keyed-consolidation.md)); the [code-weight consolidation review](oracle-reviews/code-weight-multiset-consolidation.md) records the earlier exclusion. The keyed reference uses direct value/reference equality, including SameValueZero for numeric keys. The unkeyed fallback keeps its original key and value domains. The direct `MultiSet` owner does not prove which `@tanstack/db` query shapes produce primitive keyed values or mixed-type source keys; that needs a live-query production witness. The index owner checks lower-bound presence on bounded numeric histories after each add, update, delete, and build: an upper-only range ignores an unused `fromInclusive: false`, while an explicit exclusive `from` excludes its boundary. A source Map supplies expected sets for BasicIndex and BTreeIndex; wrong-output controls reject both boundary mistakes. It also checks exact `equalityLookup` and `lookup('eq')` after singleton case-insensitive `A → a → A` replacements and populated comparator-group updates, plus fresh-object same-group replacements in fixed and generated histories. These direct-index checks do not cover every JavaScript value, arbitrary comparators, or long Collection histories. The index owner enforces one invalid-comparator law across both BasicIndex and BTreeIndex: for each comparator, accepted numeric prefix (including the empty prefix), and probe operation (add, update, eq/range lookup, take, build), the operation throws exactly when it receives a `NaN` or non-number result and never when every comparison is valid. A `signed infinity` comparator is the valid control, and a custom collation `compare` supplied through `compareOptions` is checked the same way. Two BTree Map split histories check inserted-key reads, exact whole-range payloads/order, size, and extrema, proving split placement follows the insertion index without a post-mutation comparison. A rejected index add or remove leaves the index refining its accepted rows; update and build are not atomic. At the Collection boundary, both index types and both write paths (optimistic insert, sync commit) crash the collection: status becomes `error` and the next mutation throws, so a usable collection never holds rows its subscribers were not told about. Open: a sync source can still commit writes on a collection in `error` state, which stores unpublished rows; this is existing `markError` behavior and needs a lifecycle-owner witness. Comparator transitivity is outside this evidence. | +| WHERE predicate publication | [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts), [`in` evaluator](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts) ([review](oracle-reviews/perf-in-constant-set.md)) | An independent Kleene evaluator judges `eq`/`not`/`and`/`or` predicates over strings, a normalization-prefixed string, booleans, numbers, `NaN`, a valid Date, `null`, a missing field, and virtual fields. Snapshot histories with pending optimistic inserts compare a live query, direct subscribers with and without initial state, and `currentStateAsChanges`; change histories apply multi-key sync transactions, including reinsertion of a key deleted earlier, and compare every consumer's key set after each commit, on scan and `BasicIndex` paths. Peer subscribers share one field with different literals, plus one on a virtual field, so every published change must reach exactly the subscribers whose predicate it can satisfy; routing mutants that ignored the previous value, ignored the field, delivered a change twice, or routed while stale published rows awaited reconciliation fail here. Fixed witnesses cover a layout-only publication reaching a filtered subscriber as one empty batch, a filtered subscriber's empty Collection-readiness batch, retraction of a vanished row after eager cleanup and restart, and the new row plus exact subscriber update payload when a same-key row stays TRUE. A stale-live-value mutant survives the generated membership checks but fails the payload witness. A focused [property-visibility test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-prefilter-property-visibility.test.ts) compares the public unindexed snapshot with the enriched-row contract for inherited, non-enumerable, enumerable-own, and nested getter paths; the pre-fix stored-row shortcut failed three of its four controls. Other property-descriptor and stateful-getter histories remain outside those fixed controls. Hostile mutants for FALSE-for-UNKNOWN `eq`, `or` or number-literal subscription prefilters, a prefilter that ignores the previous value, a skipped readiness batch, a skip while stale published rows await reconciliation, and an unindexed snapshot scan that reads only synced rows while an optimistic insert or delete changes visibility passed the prior `@tanstack/db` suite and fail here. Change routing withholds a dropped row from an `eq` subscription before its sent key could be recorded, so a mutant that records dropped rows as sent is equivalent for routed predicates in this oracle. A focused `collection-subscription.test.ts` witness checks that a dropped insert or update, alone or beside a matching change, cannot advance a limited subscription's page offset under a routed `eq` and an unrouted `or` predicate; the unrouted cases beside a matching change kill that mutant, and the unrouted `alone` case does not. A loose-equality prefilter is an equivalent mutant: it only skips less. Skipping during truncate replay also survives; the replay's baseline diff re-derives the same retraction, so the guard keeps the prior dataflow without its own witness. Generated cleanup and restart histories for filtered subscribers remain open for the lifecycle publication owner. No-op updates, other comparison operators, Temporal and binary operands, joins, ordering, optimistic updates, and truncate are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. Change routing was later removed in favor of pooled live queries; the routing mutants above are historical, and the same peer laws now check plain subscription dispatch. The unindexed snapshot prefilter was restored for compiled queries as one stored-row `eq` test on a string or boolean literal. Its property-visibility test also covers the reverse direction: a stored row with an inherited or non-enumerable field must still match `isUndefined` on that field, which a scan that evaluated the full predicate on stored rows fails. The `in` evaluator owner checks TanStack DB `inArray` against an independent linear reference over Dates, `NaN`, `-0`, BigInt, bytes, Temporal values, objects and nullish items, for constant and row-supplied lists. Its one-polyfill Temporal model uses captured intrinsic methods to reject forged and borrowed brands, accepts genuine own display shadows, and models Duration by ten fields, including PT1H versus PT60M. Arbitrary custom prototype forgery, mixed Temporal implementations, WHERE publication, and indexed mixed ordinary-object/Temporal range order remain outside that owner. | | Joined result keys | [Joined result key oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/join-result-key-oracle.property.test.ts) | A nested-loop model recomputes the pair set of an inner, left, or full join over source keys drawn from plain, comma-bearing, bracket-bearing, and quoted strings, numbers beside the strings that print the same, both infinities, and `NaN`. After preload and each synced group change, the published rows must equal the model's pairs and the result key count must equal the row count. The comma-joined key encoding fails its two pinned histories and both campaigns; plain `JSON.stringify`, which prints infinities as the missing-side `null`, fails the pinned infinity history and both campaigns. A pinned history in which a `NaN`-keyed pair leaves and re-forms fails when the join index compares source-key prefixes with `===`. Joins over subqueries, more than two sources, custom `getKey`, and optimistic mutations are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. | | D2 Index storage | [Index refinement oracle](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/index-refinement-oracle.property.test.ts) | A plain-`Map` model sums multiplicities under a type-and-value identity in which `NaN` equals `NaN` and `-0` equals `0`. Histories of up to twelve additions to two keys mix prefixed arrays with `NaN`, `0`, `-0`, `1`, `'1'`, `1n`, and `'a'` prefixes and unprefixed values, with cancellation and reappearance. After each addition, `get` and `has` must equal the model. Comparing prefixes with `===`, which disagrees with the `Map` that holds them, fails three pinned histories and both campaigns; the `-0` control passes under both. Compaction, presence tracking, and structural payloads are outside this owner; the join operator tests and incrementalization law own operator behavior. | | Pooled live queries | [pooled live query oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/pooled-live-query-oracle.property.test.ts) | An independent `eq` model (nullish UNKNOWN, Date as timestamp, `NaN` equal to `NaN`, `-0` equal to `0`) gives membership; a live-query Collection compiled for the same query gives order, values, and status. Histories mix sync transactions, optimistic writes confirmed or rolled back, peer mounts and unmounts, and source cleanup and restart, including cleanup while a write is pending and a mount between cleanup and restart. Wholesale observers must match after every step, granular changes must match a reference observer by type, key, value, and previous value and replay to the model's rows, the layout revision must advance when keys reorder, and observing must never build the live-query Collection. Peers may add a residual `not(eq(...))` conjunct, which the model evaluates itself; a view that ignores it, reports an unfiltered `entries()`, or delivers a row entering or leaving it as an update fails it. Other residual operators use the same compiler evaluator but are not generated. Peers may also order their rows by `id` or by a `g` that can be null, with explicit `nulls`; order comes from the reference. A comparator that ignores direction or `nulls`, an order left out of the partition key, and key order instead of the query's order each fail the pinned reorder history and both campaigns. A partition that ignores a row's previous group, orders by arrival, skips normalization, skips the initial snapshot, follows the source's status after cleanup, or never terminates fails it. Two shared conformance scenarios (`eq-filter-rows`, `eq-filter-peers`) run the pooled path under React, Vue, Solid, Svelte, and Angular; a partition that ignores a row's previous group fails both under every adapter, and `eq-filter-peers` checks through `subscriberCount` that each adapter shares one source subscription per partition. A focused [release-timing test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/pooled-live-query-gc-oracle.test.ts) compares when a pooled query and a live-query Collection release their sources across `gcTime` 1, 100, 0, and `Infinity`, with and without a subscriber, and with no `gcTime`, where both use the live-query Collection default, checks that a partition waits for its views' longest `gcTime`, checks that a view resubscribed after its partition released follows later source writes as its live-query Collection does, including a view with a residual conjunct, and checks that a partition that subscribes again still serves new mounts from one source subscription. Further focused witnesses check that a partition keeps groups only for values that have rows or watchers, that a detached reader never sees a dropped group's revision reused, keep a pooled query's public Collection live while the view is observed, accept it as a query source, and check that a pooled query turns terminal when source cleanup starts while the adapter's cleanup is still pending, as a live-query Collection does. A pinned oracle witness checks that an `eq` path whose getter throws excludes the row on both paths; a pooled path that rethrows fails it. The pooled oracle's grammar never releases a partition, so release and resubscribe histories rely on these fixed witnesses; a fixed delay, a missing unsubscribed floor, a last-`gcTime`-wins rule, and releasing at `gcTime` 0 each fail it. The [2026-10-01 review](oracle-reviews/2026-10-01-pooled-live-queries.md) records each ORC outcome. | @@ -473,7 +473,7 @@ order, public Date range predicates, cursor continuation, proxies, subclasses, replaced Date prototypes or `getTime` methods, and other Date-consuming paths remain outside that witness. -The comparison oracle also owns a bounded [Temporal order transfer](oracle-reviews/comparison-unit-retirement-round-6.md). Supplied ISO calendar fields and epoch-nanosecond offsets predict direct `ascComparator` signs and exact zeros for ordinary `PlainDate` and `ZonedDateTime` values from one polyfill copy. Fixed cases preserve the two retired unit blocks; generated cases vary calendar fields, offset zones, equal instants, and adjacent nanoseconds. Fresh-module histories reject a first-input constructor cache poisoned by a genuine instance or forged tag, including the direct `compareValues` path. This lane does not establish public query or index behavior. Custom Temporal subclasses, replaced instance prototypes, mixed native/polyfill implementations, other Temporal kinds, and public ordering/filter publication need separate witnesses. +The comparison oracle also owns a bounded [Temporal order transfer](oracle-reviews/comparison-unit-retirement-round-6.md). Supplied ISO calendar fields and epoch-nanosecond offsets predict direct `ascComparator` signs and exact zeros for ordinary `PlainDate` and `ZonedDateTime` values from one polyfill copy. Fixed cases preserve the two retired unit blocks; generated cases vary calendar fields, offset zones, equal instants, and adjacent nanoseconds. Fresh-module histories reject a first-input constructor cache poisoned by a genuine instance or forged tag, including the direct `compareValues` path. A genuine ZonedDateTime with an own shadowed display tag retains its instant position through both direct helpers. This lane does not establish public query or index behavior. Custom Temporal subclasses, replaced instance prototypes, mixed native/polyfill implementations, other Temporal kinds, and public ordering/filter publication need separate witnesses. The [Temporal index order oracle](../../packages/db/tests/temporal-index-order-oracle.test.ts) and its [review record](oracle-reviews/temporal-index-order.md) own direct @@ -482,10 +482,14 @@ continuation, plus an optimized-only public Collection snapshot with an explicit index. Supplied epoch nanoseconds establish order without promising a key tie break for equal instants; distinct zones retain exact lookup identities. A focused callback witness preserves the former string operands -for a custom comparator over two distinct pre-epoch values. Custom comparators -that distinguish values in one query-equality class, mixed value domains, -other Temporal kinds, public WHERE ranges, provider cursors, longer histories, -and live-query publication require separate witnesses. +for a custom comparator over two distinct pre-epoch values. A second callback +witness splits query-equal UTC aliases into per-row comparator positions after +add/build and checks direct range, cursors, removal, and same-class update. +An add/build witness checks that a genuine value's own shadowed display tag +cannot move its default ordered position or the value cursor. +Direct removal supplies the exact prior row. Mixed value domains, other +Temporal kinds, public WHERE ranges, provider cursors, longer histories, and +live-query publication require separate witnesses. The [Temporal alias equality oracle](../../packages/db/tests/temporal-equality-alias-oracle.test.ts) and its [review record](oracle-reviews/temporal-equality-alias.md) own the @@ -496,10 +500,23 @@ fixture-assigned classes. Unindexed and optimized-only indexed Collection snapshots check public `eq`; the `in` evaluator oracle checks constant and row-supplied alias lists. A plain object with a forged Temporal tag and a genuine value with a shadowed zone accessor must stay outside the UTC class. -Other zone aliases, mixed Temporal implementations, joins, grouping, query +The neighboring PlainDate lane checks forged tags, borrowed prototypes, and +genuine own-tag or display shadows at direct lookup and unindexed public `eq`. +Other zone aliases, custom spoofed prototypes, mixed Temporal implementations, +mixed ordinary-object/Temporal ordered indexes, joins, grouping, query identity, live-query publication, and provider paths remain unproved. Direct D2 `groupBy` has its separately approved kind-and-string identity policy. +The [custom index order selection oracle](../../packages/db/tests/custom-index-order-selection-oracle.test.ts) +and its [review record](oracle-reviews/custom-index-order-selection.md) check +that a public ordered Collection snapshot does not select a reverse-number +custom comparator by default ascending or descending `compareOptions`. +Unindexed and ordinary-index snapshots supply controls; optimized-only reads +reject the custom order while optimized exact equality still uses that index. +The fixed ready-state fixture covers BasicIndex and BTreeIndex, one numeric +field, and limit one. Mutation histories, arbitrary callbacks, and provider or +live-query publication remain outside this owner. + The [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts) also has a [cross-realm Date equality matrix](oracle-reviews/cross-realm-date-where-publication.md). For local and `node:vm` Dates with timestamps 1 and 2 or invalid timestamps, @@ -544,6 +561,14 @@ records the first five retired legacy blocks; the [IN transfer ledger](oracle-re records the next four; the [ordered/range ledger](oracle-reviews/index-unit-retirement-round-4.md) records the current transfer boundary and review. +The [signed-zero custom comparator witness](oracle-reviews/custom-comparator-signed-zero.md) +in the index refinement oracle separates `-0` and `+0` into two ordered +positions while exact lookup retains one SameValueZero class. Fixed add/build +and removal histories check BasicIndex and BTreeIndex takes, a cursor, a +bounded range, and exact membership. It does not establish arbitrary custom +callbacks or public planner selection; the custom index order selection owner +above checks that separate boundary. + The alias-scope owner also has a compiler invariant witness for a malformed include plan: removing a captured ancestor's parent projection after the builder creates it must fail before child rows are read. Without that check, diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md index 49d6af5723..e072598bd9 100644 --- a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md @@ -59,6 +59,14 @@ function from the value's prototype on each call. The repaired direct and evaluator suite passed 248 tests with no type errors. The exact committed-head verdict follows after commit. +The later equality review exposed a separate order defect: a genuine +ZonedDateTime with an own `Symbol.toStringTag` shadow was treated as an +ordinary object. A direct witness failed at the equal-instant +`ascComparator` return on the unrepaired code. The shared branded dispatch +now reads the Temporal prototype and checks native slots; direct +`ascComparator` and `compareValues` preserve the supplied instant order. +The index-order owner has a corresponding value-cursor witness. + ## Oracle guide audit | Requirement | Evidence and limit | diff --git a/docs/contributing/oracle-reviews/custom-comparator-signed-zero.md b/docs/contributing/oracle-reviews/custom-comparator-signed-zero.md new file mode 100644 index 0000000000..10dfaa67fe --- /dev/null +++ b/docs/contributing/oracle-reviews/custom-comparator-signed-zero.md @@ -0,0 +1,24 @@ +# Signed zero under a custom index comparator + +The direct IndexInterface contract permits a numeric custom comparator. +JavaScript `Map` equality treats `-0` and `+0` as one exact lookup class, while +a callback can distinguish their order. For this PR, the maintainer chose +per-row callback order inside a query-equality class. The index refinement +oracle supplies independent rank labels for `-0`, `+0`, and `1`, then checks +exact lookup, full ordered take, a bounded range, and a cursor after `add` or +`build`. It removes the negative row and checks the surviving class and order. + +The unmodified BasicIndex ordered-bucket `Map` used SameValueZero keys for the +callback positions. Both new add/build histories returned `positive` twice +and omitted `negative` at `takeFromStart`; BTreeIndex passed both controls. +BasicIndex now encodes negative zero separately for its private ordered-bucket +key, while exact membership keeps the ordinary `Map` key. All four new +histories and the full 416-test index refinement owner pass with no type errors. +The independent rank model is supplied by the fixture, not copied from either +index. This fixed witness covers only the stated signed-zero callback and +direct operations; arbitrary callbacks, longer histories, public order +selection, and provider paths remain separate. + +## Exact-commit hostile review + +Pending fresh review of the committed oracle and production paths. diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md new file mode 100644 index 0000000000..2749e8b619 --- /dev/null +++ b/docs/contributing/oracle-reviews/custom-index-order-selection.md @@ -0,0 +1,51 @@ +# Custom comparator index selection + +An index with a user-supplied `compareFn` cannot advertise the ordinary +Collection `orderBy` comparator merely through `compareOptions`. The +live-query architecture's Identity contract requires advertised comparison +options to describe the comparator the index executes. Exact equality lookup +remains a separate index operation. + +The new oracle supplies three numeric rows in source order 20, 30, 10. Its +independent model sorts their supplied ranks numerically, so an ascending +`limit: 1` read must return rank 10. It compares unindexed and indexed public +Collection snapshots after readiness for both BasicIndex and BTreeIndex. A +custom callback orders ranks descending, and an ordinary index is a control. +The driver also checks descending reads, `optimizedOnly` refusal of a custom +index for ordinary order, and optimized exact `eq` on that same custom index. + +At frozen head `1265abeb5`, both custom index types claimed default ascending +eligibility and returned rank 30. The new oracle failed twice at the public +first-key comparison, with no type errors. The repair makes public ordered +selection respect the existing `supportsRangeOptimization` capability, which +is false for custom comparators. This leaves exact lookup available and does +not change the custom callback's direct index behavior. The focused 460-test +index/query campaign passed with no type errors after repair. + +The oracle covers a ready in-memory Collection, one numeric field, one order +term, limit one, and two built-in index types. It does not establish arbitrary +custom comparator laws, mutation histories, live-query publication, provider +ordering, or a performance guarantee that another eligible index is created. + +## Oracle guide audit + +| Requirement | Evidence and limit | +| --- | --- | +| ORC-001 | The architecture's index metadata law and public ordered-read contract authorize the numeric first-key observation. | +| ORC-002 | Supplied ranks determine expected order without reading the index comparator or its metadata. | +| ORC-003 | The oracle explains law, fixture, production paths, snapshot, and comparison beside its code. | +| ORC-004 | Fixed three-row source order distinguishes insertion order from ascending and descending rank; no generated-history claim is made. | +| ORC-005 | Both public Collection snapshot paths are observed after readiness; optimized-only reads check planner admission. | +| ORC-006 | The original wrong planner fails at both public first-key assertions; an ordinary index and exact lookup are adjacent controls. | +| ORC-007 | Not triggered: this owner is a bounded fixed witness. | +| ORC-008 | Not triggered: the model has no mutable state. | +| ORC-009 | Ranks are model facts; `supportsRangeOptimization` is production capability observed separately from expected row order. | +| ORC-010 | `withOracleCleanup` releases each Collection while preserving the primary assertion. | +| ORC-011 | The unindexed scan and numeric model check the indexed result through different formulations; both production paths still share the Collection API. | +| ORC-012 | This record names RED/GREEN evidence, owner, limits, and pending exact-commit review. | +| ORC-013 | A reverse-number callback makes the wrong index choice visibly return rank 30 instead of rank 10. | +| ORC-014 | Controlled in-memory sync ends at the ready snapshot; no provider handoff is claimed. | + +## Exact-commit hostile review + +Pending fresh review of the final committed oracle and production path. diff --git a/docs/contributing/oracle-reviews/temporal-equality-alias.md b/docs/contributing/oracle-reviews/temporal-equality-alias.md index 277981ca6d..93e2cc9ae1 100644 --- a/docs/contributing/oracle-reviews/temporal-equality-alias.md +++ b/docs/contributing/oracle-reviews/temporal-equality-alias.md @@ -6,8 +6,9 @@ while `[+00:00]` is a distinct zone. Exact nanoseconds and calendar identity also matter. The new oracle assigns those classes in its fixture, then checks direct BasicIndex and BTreeIndex lookups after add, build, and removal, plus unindexed and optimized-only indexed public `eq` snapshots. The existing `in` -evaluator oracle now uses `.equals()` as its independent Temporal reference -and checks both constant and row-supplied lists, including the forged and +evaluator oracle now uses captured native Temporal methods as its independent +reference for kinds with `.equals()` and ten intrinsic fields for Duration. It +checks constant and row-supplied lists, including forged, borrowed, and shadowed controls. ## Failure and repair @@ -33,7 +34,31 @@ no partition mismatch. This bounds evidence for that host, not a proof for every Temporal implementation or time-zone database. The focused index, evaluator, and Temporal runs passed 453 tests with no type errors after the repair. The full DB suite passed 12,171 tests in 279 files with no type -errors. The exact-commit review follows after commit. +errors. The exact-commit review then found the gaps below, so that count +belongs to an intermediate commit. + +## Exact-commit counterexamples and follow-up + +At commit `1265abeb5`, hostile reviewers found that a genuine PlainDate with +an own `Symbol.toStringTag` or `toString` shadow could miss a native-equal +value through compiled `in`. A plain object with a forged PlainDate tag could +match a genuine value. A genuine ZonedDateTime with a shadowed tag could miss +through direct indexes and public `eq`. The `in` reference had trusted an +instance's display tag and therefore agreed with one wrong production answer. +Its Duration wording also claimed `.equals()` even though Duration has no such +method. Eleven new brand, shadow, and Duration checks failed at the intended +comparisons on the original code, with no type errors. + +The follow-up reference calls captured polyfill intrinsics on the candidate to +establish its brand and models Duration with ten field getters. Production +normalization reads slot-checking methods from the value's prototype rather +than its own display members. The alias owner also checks the later-instant +and alternate-calendar classes and a PlainDate forged/borrowed/shadowed lane +through direct singleton lookups and an unindexed public snapshot. The focused +39-test alias and `in` oracle run passed with no type errors after repair. +The mixed ordinary-object/Temporal ordered index is excluded because its +range order is unspecified. Arbitrary custom prototypes that imitate genuine +Temporal methods remain outside the one-polyfill brand guarantee. ## Oracle guide audit @@ -42,28 +67,29 @@ errors. The exact-commit review follows after commit. | ORC-001 | Changelog #1519 and Temporal `.equals()` authorize the equality class; the opening oracle comment limits implementations, paths, and observations. | | ORC-002 | Fixture-assigned classes predict key sets without calling production normalization. A separate native `.equals()` check calibrates the fixture. | | ORC-003 | Prose beside the model, add/build history, Collection driver, and assertions explains their roles. | -| ORC-004 | Fixed classes distinguish alias equality, distinct zone, adjacent nanosecond, calendar, forged tag, and shadowed accessor. No generated-history claim is made. | +| ORC-004 | Fixed classes distinguish alias equality, distinct zone, adjacent nanosecond, calendar, forged and borrowed brands, and genuine display shadows. The `in` owner generates list combinations over its bounded pool; no arbitrary object grammar is claimed. | | ORC-005 | Direct lookups observe both index types; an indexed public snapshot spies on `lookup`, and the unindexed snapshot checks the evaluator path after readiness. | -| ORC-006 | The frozen display-string design failed alias checks. The first tuple design failed forged-tag and shadowed-zone controls at direct lookups. A lexical `in` mutant fails its check. | +| ORC-006 | The frozen display-string design failed alias checks. The first tuple design failed forged-tag and shadowed-zone controls. Commit `1265abeb5` failed eleven brand/shadow/Duration checks; a lexical `in` mutant fails its comparison. | | ORC-007 | Not triggered by this fixed oracle. The adjacent `in` owner retains fixed and random generated campaigns. | | ORC-008 | The fixture model is immutable and has no reference-model state. | | ORC-009 | The model's class labels denote query equality, distinct from the order oracle's instant labels. | | ORC-010 | Collection cleanup preserves the primary mismatch. Direct index checks own no external resource. | | ORC-011 | The direct lookup and public scan cross different production paths, but share normalization. Native `.equals()` is the independent semantic calibration; this record does not claim a second full reference model. | | ORC-012 | This record names the owner, RED controls, limits, and required exact-commit review. | -| ORC-013 | Alias equality versus fixed-offset inequality separates the candidate key from display-only and instant-only keys; forged and shadowed values reject structural impostors. | +| ORC-013 | Alias equality versus fixed-offset inequality separates display-only and instant-only keys; native-brand controls reject forged/borrowed impostors while genuine display shadows preserve equality. | | ORC-014 | The Collection uses controlled in-memory sync and ends its claim at the snapshot; no provider handoff is claimed. | ## Limits and exact-commit review -The oracle covers ordinary values from one polyfill copy and one instant. -Other IANA aliases, mixed Temporal implementations, joins, grouping, query -identity, live-query publication, provider paths, and longer histories need -their own witnesses. The `in` oracle adds generated list membership but not -those other boundaries. `Intl` canonicalization has a raw-ID fallback when +The ZonedDateTime lane covers one polyfill copy and one instant; the +PlainDate lane checks brand shadows at one calendar date. Other IANA aliases, +arbitrary spoofed prototypes, mixed Temporal implementations, joins, grouping, +query identity, live-query publication, provider paths, and longer histories +need their own witnesses. The `in` owner adds generated list membership but +not those other boundaries. `Intl` canonicalization has a raw-ID fallback when the host rejects a valid Temporal zone; alias equivalence in that case is -unproved. A custom comparator that distinguishes two query-equal aliases -also needs a separate law and index witness. Direct D2 `groupBy` has a -separately approved kind-and-string policy. +unproved. The index-order owner now checks a custom comparator that +separates query-equal aliases. Direct D2 `groupBy` has a separately approved +kind-and-string policy. Pending fresh hostile review of the exact final commit. diff --git a/docs/contributing/oracle-reviews/temporal-index-order.md b/docs/contributing/oracle-reviews/temporal-index-order.md index 94d9600c40..40859fe519 100644 --- a/docs/contributing/oracle-reviews/temporal-index-order.md +++ b/docs/contributing/oracle-reviews/temporal-index-order.md @@ -32,8 +32,9 @@ The existing index refinement oracle also rejects the version that removed the replacement. The original repaired commit passed 13 focused tests and the index refinement oracle's 412 tests, then the full DB suite passed 12,154 tests in 278 files. After the alias and comparator controls, the full -DB suite passed 12,171 tests in 279 files with no type errors. The final -exact-commit review remains pending. +DB suite passed 12,171 tests in 279 files with no type errors. A later exact +commit review found the custom alias gap below, so those counts describe an +intermediate commit rather than final verification. The independent fixture supplies epoch nanoseconds and zone-identity classes. It never calls the index's normalization or comparator to compute expected @@ -60,11 +61,33 @@ new epoch tuple; a pre-epoch callback witness rejected its wrong direct ordered result in both index types. The repair now passes the former display key to custom callbacks while storing raw values for index order. This is a bounded compatibility decision; published types do not promise the operand -representation. A custom comparator that distinguishes two values in one -query-equality class remains open. Mixed domains, other Temporal kinds, -public WHERE ranges, provider cursors, longer histories, and live-query -publication also remain outside this oracle. Existing range optimization -does not claim Temporal WHERE correctness. +representation. The exact-commit reviewer then found that a callback could +distinguish UTC and Etc/UTC values in one query-equality class. Both indexes +merged the exact key before comparing those rows, so ordered takes, ranges, +and cursors disagreed with the callback. Six new add/build/update witnesses +failed at `takeFromStart` on `1265abeb5`, with no type errors. The repair keeps +exact lookup and comparator positions as separate partitions. The witnesses +now check the callback's exact former display-string operands, equality lookup, +range boundaries, both cursor directions, representative removal, and a +same-class update. The focused 470-test index campaign passed with no type +errors after repair. Direct removal supplies the actual prior source value; +no equivalence-substitution guarantee is claimed for a different old item. +For this PR, the maintainer selected per-row callback order when a comparator +distinguishes values in one query-equality class. This is the contract the +separate partitions implement. +Mixed domains, other Temporal kinds, public WHERE ranges, provider cursors, +longer histories, and live-query publication remain outside this oracle. +Existing range optimization does not claim Temporal WHERE correctness. + +A subsequent direct comparison witness found that a genuine ZonedDateTime +with an own shadowed `Symbol.toStringTag` lost its native instant order. +The matching index witness failed in all four BasicIndex/BTreeIndex add/build +histories at the value cursor: it returned the equal-instant peer instead of +the one-nanosecond successor. Classification now reads a slot-checking +prototype method, so the own display tag cannot choose an ordinary-object +position. The direct and index witnesses pass after that repair. The fixed +index witness does not establish custom comparator behavior with a shadowed +tag or mixed Temporal implementations. ## Oracle guide audit @@ -74,11 +97,11 @@ does not claim Temporal WHERE correctness. | ORC-002 | Supplied BigInt instants and zone classes determine expected order and exact matches independently of normalization and comparison code. | | ORC-003 | The oracle describes its law, fixture model, legal actions, direct and public drivers, observations, and checkpoints beside executable code. | | ORC-004 | Not triggered: the bounded fixture matrix and one continuation are fixed histories, not a generated-history claim. The nearby invalid mixed-kind domain is excluded. | -| ORC-005 | Direct takes and equality lookups observe both index implementations. An optimized-only Collection snapshot invokes the explicit index's `takeFromStart` after readiness. | -| ORC-006 | The original normalized-string design failed at direct ordered takes and the public `limit: 1` snapshot. A stable wrong-result control rejects that later-instant first row while allowing both legitimate equal-instant ties. The first BTree repair failed the live-bucket representative check; a tuple-fed custom comparator failed the pre-epoch take check. | +| ORC-005 | Direct takes, ranges, cursors, and equality lookups observe both index implementations. An optimized-only Collection snapshot invokes the explicit index's `takeFromStart` after readiness. | +| ORC-006 | The original normalized-string design failed at direct ordered takes and the public `limit: 1` snapshot. A stable wrong-result control rejects that later-instant first row while allowing both legitimate equal-instant ties. The first BTree repair failed the live-bucket representative check; a tuple-fed custom comparator failed the pre-epoch take check; the merged-exact-key design failed six alias/callback takes. Own-tag shadows failed all four value-cursor checks before the branded dispatch repair. | | ORC-007 | Not triggered: this is a bounded fixed oracle, not an important generated property. The existing index refinement property keeps its own campaigns. | | ORC-008 | Not triggered: the fixture's independent model has no mutable reference state. | -| ORC-009 | The model's instant label represents order only; its zone class represents exact equality. One comparator position can contain two exact values. The custom callback witness checks backward compatibility for its input text, not Temporal's default instant law. | +| ORC-009 | The model's instant label represents default order only; its zone class represents exact equality. One comparator position can contain two exact values, and a custom callback can split one exact value into two positions. The callback-argument witness checks exact former input text. | | ORC-010 | `withOracleCleanup` preserves a primary public mismatch and releases the Collection. Direct indexes own no external resources. | | ORC-011 | The unindexed public scan is a second production formulation for the ordered result. The numeric BigInt model also distinguishes a shared display-string mistake. | | ORC-012 | This record contains RED/GREEN evidence, a hostile review, and scoped unresolved cells. An exact-commit verdict follows below. | @@ -87,4 +110,4 @@ does not claim Temporal WHERE correctness. ## Exact committed-head review -Pending. +Pending fresh hostile review of the final committed repair. diff --git a/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md b/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md new file mode 100644 index 0000000000..3aa5c83671 --- /dev/null +++ b/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md @@ -0,0 +1,58 @@ +# Deep equality unit retirement: Date and RegExp + +Two source-isolated Field Lab loss audits compared the Date and RegExp blocks +of `packages/db/tests/utils.test.ts` at frozen head `1265abeb5` with the +existing `utils-oracle.property.test.ts`. The proposed deletion said that the +oracle's copy/change and cross-type laws already covered both blocks. The +audits found three missing primitive boundaries before either block was +removed. No production behavior was changed for this transfer. + +| Retired block and source observation | Frozen receiver | Receiving law and checkpoint | +| --- | --- | --- | +| Date: separately allocated 2023-01-01 Dates are equal; 2023-01-02 differs. | Date copy/change properties and a fixed Date-time triple covered the law, but not those exact operands. | The fixed 2023 triple reconstructs both results; the existing generated Date copy/change property checks neighboring timestamps at `deepEquals` return. | +| Date: a Date differs from its ISO string. | No Date/string pair; the broad cross-type description had compressed away this boundary. | A fixed 2023 probe and generated finite Dates compare with their own ISO text in both directions. | +| Date: a Date differs from its numeric timestamp. | No Date/number pair, even when the number encoded the same instant. | The same fixed and generated law compares with `getTime()` in both directions. | +| RegExp: separate `/abc/g` and `/test/i` copies are equal; changed flags or source differ. | Source/flags fixed triples and a generated copy/changed-flags property covered the law, but not those exact operands. | Fixed `/abc/g`, `/abc/i`, `/def/g`, and `/test/i` checks reconstruct each return; generated RegExps vary source and flags. | +| RegExp: `/abc/g` differs from its source string `abc`. | The receiver had RegExp/object and number/string pairs, but no RegExp/string pair. | A fixed source-text probe and generated RegExps compare with their own source text in both directions. | + +The `deepEquals` documentation names Date and RegExp as supported value +classes. The old examples supply the more precise rule that their primitive +encodings are not coerced into those classes. The receiving model assigns +expected inequality from those distinct value classes; it does not call the +production classifier. The fixed operands preserve every old assertion, and +the generated properties extend each missing boundary across finite Dates and +valid RegExps. A deliberately coercing Date/number, Date/string, or +RegExp/string comparator fails the receiving checker at its forward inequality +assertion. Both directions are checked for the real production function. + +The two legacy blocks contained six tests. After transfer, the focused oracle +and remaining unit file passed 208 tests with no type errors. The generator +uses finite Date timestamps from -1,000,000,000,000 through 2,000,000,000,000 +milliseconds and the existing valid RegExp source/flag grammar; fixed and +seedless campaigns use the same properties. This establishes the direct +`deepEquals` return for those pairs. It does not establish cross-realm values, +RegExp subclasses, arbitrary prototype changes, every cross-type pair, or a +public change-event trace. Those need their own owner or receiving witness. + +## Oracle guide audit + +| Requirement | Evidence and limit | +| --- | --- | +| ORC-001 | Existing `deepEquals` documentation and retained examples supply the class-preserving law; the direct value classes and return checkpoint are explicit. | +| ORC-002 | Separate constructors and primitive encodings determine expected copy, change, and cross-type outcomes without production classification. | +| ORC-003 | The receiver explains its law, bounded grammar, production call, and return comparison beside the checks. | +| ORC-004 | Every removed operand and outcome has a fixed reconstruction; generated finite Dates and RegExps vary the new cross-type boundaries. | +| ORC-005 | Each check calls exported `deepEquals` and observes the immediate boolean return. No public publication claim is made. | +| ORC-006 | Injected primitive-coercing comparators fail the exact inequality assertion for all three newly recovered boundaries. | +| ORC-007 | Existing `utilsProperty` runs each property with fixed seed and random seed and supports direct seed/path replay. | +| ORC-008 | Not triggered: the value model has no mutable state. | +| ORC-009 | Date, RegExp, ISO text, epoch number, and source text are distinct value classes in this model; no production storage state is modeled. | +| ORC-010 | The direct check owns no cleanup resource; its assertion is the failure checkpoint. | +| ORC-011 | The source-isolated audits found a shared broad-category omission. The old exact examples and generated checks now enforce each primitive boundary; no independent second production formulation is claimed. | +| ORC-012 | This ledger records the source assertions, recovered losses, receiving checks, execution, limits, and pending exact-commit review. | +| ORC-013 | A primitive with matching displayed or numeric content distinguishes class-preserving equality from a plausible coercing design. | +| ORC-014 | Not triggered: the direct call has no provider or host boundary. | + +## Exact-commit hostile review + +Pending fresh review of the final committed oracle and the removed blocks. diff --git a/packages/db/package.json b/packages/db/package.json index 3b71220d32..df69b9cdc0 100644 --- a/packages/db/package.json +++ b/packages/db/package.json @@ -23,7 +23,7 @@ "test": "vitest --run", "test:dist": "vitest run --config vitest.dist.config.ts", "test:facade-retention": "node --expose-gc --import tsx tests/facade-retention.probe.ts", - "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/temporal-index-order-oracle.test.ts tests/temporal-equality-alias-oracle.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", + "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/temporal-index-order-oracle.test.ts tests/temporal-equality-alias-oracle.test.ts tests/custom-index-order-selection-oracle.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", "bench:nested-includes": "vitest bench tests/query/includes-performance.bench.ts --run", "test:indexed-db:package": "pnpm --filter @tanstack/db... build && vitest run --config vitest.indexed-db.package.config.ts", "test:indexed-db:browser": "playwright test --config playwright.indexed-db.config.ts", diff --git a/packages/db/src/collection/change-events.ts b/packages/db/src/collection/change-events.ts index f69ab13d43..a0df8fb6cb 100644 --- a/packages/db/src/collection/change-events.ts +++ b/packages/db/src/collection/change-events.ts @@ -356,7 +356,10 @@ function getOrderedKeys( // Find the index const index = findIndexForField(collection, fieldPath, compareOpts) - if (index && index.supports(`gt`)) { + // A user-supplied index comparator is not described by compareOptions. + // Exact equality lookup may still use that index, but its ordered walk + // cannot stand in for the requested order. + if (index && index.supports(`gt`) && index.supportsRangeOptimization) { // Use index optimization const filterFn = (key: TKey): boolean => { const value = collection.get(key) diff --git a/packages/db/src/indexes/base-index.ts b/packages/db/src/indexes/base-index.ts index 6862c14052..d791ca759c 100644 --- a/packages/db/src/indexes/base-index.ts +++ b/packages/db/src/indexes/base-index.ts @@ -101,8 +101,10 @@ export interface IndexInterface< * Whether range lookups (gt/gte/lt/lte) on this index can be trusted to * return every matching key. Range traversal relies on the index ordering, so * it is unsafe when the index uses a custom comparator, whose order may not - * match the WHERE evaluator's relational operators. Callers must fall back to - * a full scan when this is `false`. + * match the WHERE evaluator's relational operators. A custom comparator is + * also not represented by compareOptions, so an ordered Collection read must + * not select this index by compareOptions alone. Callers must fall back to a + * full scan when this is `false`. */ get supportsRangeOptimization(): boolean diff --git a/packages/db/src/indexes/basic-index.ts b/packages/db/src/indexes/basic-index.ts index 41a2649f71..21bf4a4fb9 100644 --- a/packages/db/src/indexes/basic-index.ts +++ b/packages/db/src/indexes/basic-index.ts @@ -1,9 +1,5 @@ import { compareKeys } from '@tanstack/db-ivm' -import { - areSameValueZeroEqual, - indexExactKey, - normalizeValue, -} from '../utils/comparison.js' +import { areSameValueZeroEqual, normalizeValue } from '../utils/comparison.js' import { compareKeysReversed, findInsertPositionInArray, @@ -17,6 +13,13 @@ import { import type { CompareOptions } from '../query/builder/types.js' import type { IndexOperation } from './base-index.js' +// Map collapses signed zero, while a user comparator may order them apart. +const NEGATIVE_ZERO_ORDER_KEY = Symbol(`negative zero order key`) + +function orderedBucketKey(value: unknown): unknown { + return Object.is(value, -0) ? NEGATIVE_ZERO_ORDER_KEY : value +} + /** * Options for range queries */ @@ -59,9 +62,10 @@ export class BasicIndex< // Map for O(1) equality lookups: indexedValue -> Set of PKs private valueMap = new Map>() - // One ordered representative per exact value; Temporal stays unnormalized - // here so equal instants in different zones share a comparator position. + // Equality and comparator positions are independent: a custom comparator + // may distinguish values that have one exact query-equality key. private sortedValues: Array = [] + private orderedBuckets = new Map>() // Set of all indexed PKs private indexedKeys = new Set() @@ -89,20 +93,34 @@ export class BasicIndex< normalizedValue: unknown, orderValue: unknown, ): void { - const keySet = this.valueMap.get(normalizedValue) - if (keySet) { - // Value already exists, just add the key to the set - keySet.add(key) + this.addToOrder(key, orderValue) + const exactKeys = this.valueMap.get(normalizedValue) + if (exactKeys) exactKeys.add(key) + else this.valueMap.set(normalizedValue, new Set([key])) + } + + private addToOrder(key: TKey, orderValue: unknown): void { + // Compare before changing either partition. One comparator position can + // contain several exact values, and one exact value can occupy several + // custom comparator positions. + const insertIdx = findInsertPositionInArray( + this.sortedValues, + orderValue, + this.compareFn, + ) + if ( + insertIdx < this.sortedValues.length && + this.compareFn(this.sortedValues[insertIdx], orderValue) === 0 + ) { + this.orderedBuckets + .get(orderedBucketKey(this.sortedValues[insertIdx]))! + .set(key, orderValue) } else { - // New value. Find its sorted position first: a comparator failure must - // throw before the index changes. - const insertIdx = findInsertPositionInArray( - this.sortedValues, - orderValue, - this.compareFn, - ) - this.valueMap.set(normalizedValue, new Set([key])) this.sortedValues.splice(insertIdx, 0, orderValue) + this.orderedBuckets.set( + orderedBucketKey(orderValue), + new Map([[key, orderValue]]), + ) } } @@ -136,36 +154,44 @@ export class BasicIndex< normalizedValue: unknown, orderValue: unknown, ): void { - const keySet = this.valueMap.get(normalizedValue) - if (keySet?.has(key)) { - // The last key for this value leaves the sorted array too. Locate it - // before any change, so a comparator failure leaves the index unchanged. - if (keySet.size === 1) { - let sortedIndex = findInsertPositionInArray( - this.sortedValues, - orderValue, - this.compareFn, - ) - // Distinct equality keys may share one comparator position. - while ( - sortedIndex < this.sortedValues.length && - this.compareFn(this.sortedValues[sortedIndex], orderValue) === 0 - ) { - if ( - areSameValueZeroEqual( - indexExactKey(this.sortedValues[sortedIndex]), - normalizedValue, - ) - ) { - this.sortedValues.splice(sortedIndex, 1) - break - } - sortedIndex++ - } - this.valueMap.delete(normalizedValue) - } - keySet.delete(key) + const exactKeys = this.valueMap.get(normalizedValue) + if (!exactKeys?.has(key)) return + this.removeFromOrder(key, orderValue) + exactKeys.delete(key) + if (exactKeys.size === 0) this.valueMap.delete(normalizedValue) + } + + private removeFromOrder(key: TKey, orderValue: unknown): void { + const index = findInsertPositionInArray( + this.sortedValues, + orderValue, + this.compareFn, + ) + if (index >= this.sortedValues.length) return + const representative = this.sortedValues[index] + if (this.compareFn(representative, orderValue) !== 0) return + const members = this.orderedBuckets.get(orderedBucketKey(representative)) + if (!members?.has(key)) return + if (members.size === 1) { + this.sortedValues.splice(index, 1) + this.orderedBuckets.delete(orderedBucketKey(representative)) + return + } + // Keep a live representative when its row leaves the comparator group. + if (areSameValueZeroEqual(members.get(key), representative)) { + const values = members.entries() + const first = values.next().value! + const successor = first[0] === key ? values.next().value![1] : first[1] + // A custom comparator may reject the promoted value against a neighbor. + // Check before changing either the sorted or exact partition. + if (index > 0) this.compareFn(this.sortedValues[index - 1], successor) + if (index + 1 < this.sortedValues.length) + this.compareFn(successor, this.sortedValues[index + 1]) + this.sortedValues[index] = successor + this.orderedBuckets.delete(orderedBucketKey(representative)) + this.orderedBuckets.set(orderedBucketKey(successor), members) } + members.delete(key) } /** @@ -185,23 +211,25 @@ export class BasicIndex< const oldValue = normalizeValue(oldIndexedValue) const newValue = normalizeValue(newIndexedValue) + const oldOrder = this.orderValue(oldIndexedValue, oldValue) + const newOrder = this.orderValue(newIndexedValue, newValue) if ( areSameValueZeroEqual(oldValue, newValue) && this.valueMap.get(newValue)?.has(key) && this.indexedKeys.has(key) ) { + if (!areSameValueZeroEqual(oldOrder, newOrder)) { + this.removeFromOrder(key, oldOrder) + this.addToOrder(key, newOrder) + } this.removeRangeValue(oldIndexedValue) this.addRangeValue(newIndexedValue) return } - this.removeFromBucket( - key, - oldValue, - this.orderValue(oldIndexedValue, oldValue), - ) + this.removeFromBucket(key, oldValue, oldOrder) this.removeRangeValue(oldIndexedValue) - this.addToBucket(key, newValue, this.orderValue(newIndexedValue, newValue)) + this.addToBucket(key, newValue, newOrder) this.addRangeValue(newIndexedValue) this.indexedKeys.add(key) } @@ -226,22 +254,33 @@ export class BasicIndex< this.indexedKeys.add(key) } - // Group by value - const orderValues: Array = [] - for (const { key, value, orderValue } of entriesArray) { - if (this.valueMap.has(value)) { - this.valueMap.get(value)!.add(key) + // Native Array.sort moves raw undefined values last without comparing them. + // Wrap values so nulls-first comparison also runs during a full build. + const sortable = entriesArray.map((entry) => ({ entry })) + sortable.sort((left, right) => + this.compareFn(left.entry.orderValue, right.entry.orderValue), + ) + for (const { entry } of sortable) { + const exactKeys = this.valueMap.get(entry.value) + if (exactKeys) exactKeys.add(entry.key) + else this.valueMap.set(entry.value, new Set([entry.key])) + + const representative = this.sortedValues.at(-1) + if ( + this.sortedValues.length > 0 && + this.compareFn(representative, entry.orderValue) === 0 + ) { + this.orderedBuckets + .get(orderedBucketKey(representative))! + .set(entry.key, entry.orderValue) } else { - this.valueMap.set(value, new Set([key])) - orderValues.push(orderValue) + this.sortedValues.push(entry.orderValue) + this.orderedBuckets.set( + orderedBucketKey(entry.orderValue), + new Map([[entry.key, entry.orderValue]]), + ) } } - - // Native Array.sort moves raw undefined values last without comparing them. - // Wrap values so nulls-first comparison also runs during a full build. - const sortable = orderValues.map((value) => ({ value })) - sortable.sort((left, right) => this.compareFn(left.value, right.value)) - this.sortedValues = sortable.map(({ value }) => value) } /** @@ -250,6 +289,7 @@ export class BasicIndex< clear(): void { this.valueMap.clear() this.sortedValues = [] + this.orderedBuckets.clear() this.indexedKeys.clear() this.clearRangeValues() } @@ -361,9 +401,10 @@ export class BasicIndex< // Collect all keys in range for (let i = startIdx; i < endIdx; i++) { - const keys = this.valueMap.get(indexExactKey(this.sortedValues[i])) - if (keys) { - keys.forEach((key) => result.add(key)) + for (const key of this.orderedBuckets + .get(orderedBucketKey(this.sortedValues[i]))! + .keys()) { + result.add(key) } } @@ -449,9 +490,9 @@ export class BasicIndex< const groupValue = this.sortedValues[index] const groupKeys: Array = [] do { - for (const key of this.valueMap.get( - indexExactKey(this.sortedValues[index]), - ) ?? []) { + for (const key of this.orderedBuckets + .get(orderedBucketKey(this.sortedValues[index])) + ?.keys() ?? []) { groupKeys.push(key) } index += step diff --git a/packages/db/src/indexes/btree-index.ts b/packages/db/src/indexes/btree-index.ts index c83f1bc1c8..af20944eb4 100644 --- a/packages/db/src/indexes/btree-index.ts +++ b/packages/db/src/indexes/btree-index.ts @@ -4,6 +4,7 @@ import { BTree } from '../utils/btree.js' import { areSameValueZeroEqual, denormalizeUndefined, + indexCustomComparatorValue, normalizeForBTree, } from '../utils/comparison.js' import { codedMessage, devBuild } from '../error-message.js' @@ -35,12 +36,15 @@ export interface RangeQueryOptions { type OrderedBucket = { representative: unknown - // Exact equality keys can share one comparator position. Their ordered - // values let removal replace a departing representative without a scan. - exactValues: Map + // One comparator position may contain several exact values, while one + // exact value may occupy several custom comparator positions. + values: Map keys: Set } +// Map uses SameValueZero, but a custom callback may order -0 apart from +0. +const NEGATIVE_ZERO_ORDER_OPERAND = Symbol(`negative zero order operand`) + /** * B+Tree index for sorted data with range queries * This maintains items in sorted order and provides efficient range operations @@ -63,9 +67,12 @@ export class BTreeIndex< private orderedEntries = new BTree>((a, b) => this.compareStored(a, b), ) - private valueMap = new Map< + private valueMap = new Map }>() + // Repeated identical callback operands belong to the same comparator + // position. This avoids another tree search for every row in a hot bucket. + private orderOperands = new Map< unknown, - { keys: Set; ordered: OrderedBucket } + { bucket: OrderedBucket; owners: number } >() private indexedKeys = new Set() @@ -75,6 +82,13 @@ export class BTreeIndex< return this.compareFn(denormalizeUndefined(a), denormalizeUndefined(b)) } + private orderOperand(value: unknown): unknown { + const operand = this.hasCustomComparator + ? indexCustomComparatorValue(value) + : value + return Object.is(operand, -0) ? NEGATIVE_ZERO_ORDER_OPERAND : operand + } + protected initialize(_options?: BTreeIndexOptions): void {} /** @@ -101,29 +115,35 @@ export class BTreeIndex< normalizedValue: unknown, orderValue: unknown, ): void { + this.addToOrder(key, orderValue) const exact = this.valueMap.get(normalizedValue) - if (exact) { - exact.keys.add(key) - exact.ordered.keys.add(key) + if (exact) exact.keys.add(key) + else this.valueMap.set(normalizedValue, { keys: new Set([key]) }) + } + + private addToOrder(key: TKey, orderValue: unknown): void { + const operand = this.orderOperand(orderValue) + const cached = this.orderOperands.get(operand) + if (cached) { + cached.bucket.keys.add(key) + cached.bucket.values.set(key, orderValue) + cached.owners++ return } let orderedBucket = this.orderedEntries.get(orderValue) if (orderedBucket) { orderedBucket.keys.add(key) - orderedBucket.exactValues.set(normalizedValue, orderValue) + orderedBucket.values.set(key, orderValue) } else { orderedBucket = { representative: orderValue, - exactValues: new Map([[normalizedValue, orderValue]]), + values: new Map([[key, orderValue]]), keys: new Set([key]), } this.orderedEntries.set(orderValue, orderedBucket) } - this.valueMap.set(normalizedValue, { - keys: new Set([key]), - ordered: orderedBucket, - }) + this.orderOperands.set(operand, { bucket: orderedBucket, owners: 1 }) } /** @@ -141,23 +161,38 @@ export class BTreeIndex< // Normalize the value for Map key usage const normalizedValue = normalizeForBTree(indexedValue) - this.removeFromBucket(key, normalizedValue) + this.removeFromBucket( + key, + normalizedValue, + this.orderValue(indexedValue, normalizedValue), + ) this.removeRangeValue(indexedValue) this.indexedKeys.delete(key) } - private removeFromBucket(key: TKey, normalizedValue: unknown): void { + private removeFromBucket( + key: TKey, + normalizedValue: unknown, + orderValue: unknown, + ): void { const exact = this.valueMap.get(normalizedValue) if (!exact?.keys.has(key)) return - const orderedBucket = exact.ordered - const removedExactValue = exact.keys.size === 1 + this.removeFromOrder(key, orderValue) + exact.keys.delete(key) + if (exact.keys.size === 0) this.valueMap.delete(normalizedValue) + } + + private removeFromOrder(key: TKey, orderValue: unknown): void { + const operand = this.orderOperand(orderValue) + const cached = this.orderOperands.get(operand) + const orderedBucket = cached?.bucket ?? this.orderedEntries.get(orderValue) + if (!orderedBucket?.keys.has(key)) return const replacesRepresentative = - removedExactValue && orderedBucket.keys.size > 1 && areSameValueZeroEqual( orderedBucket.representative, - orderedBucket.exactValues.get(normalizedValue), + orderedBucket.values.get(key), ) // Tree writes compare first, so a comparator failure throws before the @@ -165,18 +200,17 @@ export class BTreeIndex< if (orderedBucket.keys.size === 1) { this.orderedEntries.delete(orderedBucket.representative) } else if (replacesRepresentative) { - const representative = [...orderedBucket.exactValues].find( - ([value]) => !areSameValueZeroEqual(value, normalizedValue), - )?.[1] - this.orderedEntries.set(representative, orderedBucket) + const values = orderedBucket.values.entries() + const first = values.next().value! + const representative = + first[0] === key ? values.next().value![1] : first[1] + if (!areSameValueZeroEqual(representative, orderedBucket.representative)) + this.orderedEntries.set(representative, orderedBucket) orderedBucket.representative = representative } - exact.keys.delete(key) orderedBucket.keys.delete(key) - if (removedExactValue) { - this.valueMap.delete(normalizedValue) - orderedBucket.exactValues.delete(normalizedValue) - } + orderedBucket.values.delete(key) + if (cached && --cached.owners === 0) this.orderOperands.delete(operand) } /** @@ -196,18 +230,24 @@ export class BTreeIndex< const oldValue = normalizeForBTree(oldIndexedValue) const newValue = normalizeForBTree(newIndexedValue) + const oldOrder = this.orderValue(oldIndexedValue, oldValue) + const newOrder = this.orderValue(newIndexedValue, newValue) if ( areSameValueZeroEqual(oldValue, newValue) && this.valueMap.get(newValue)?.keys.has(key) ) { + if (!areSameValueZeroEqual(oldOrder, newOrder)) { + this.removeFromOrder(key, oldOrder) + this.addToOrder(key, newOrder) + } this.removeRangeValue(oldIndexedValue) this.addRangeValue(newIndexedValue) return } - this.removeFromBucket(key, oldValue) + this.removeFromBucket(key, oldValue, oldOrder) this.removeRangeValue(oldIndexedValue) - this.addToBucket(key, newValue, this.orderValue(newIndexedValue, newValue)) + this.addToBucket(key, newValue, newOrder) this.addRangeValue(newIndexedValue) this.indexedKeys.add(key) } @@ -229,6 +269,7 @@ export class BTreeIndex< clear(): void { this.orderedEntries.clear() this.valueMap.clear() + this.orderOperands.clear() this.indexedKeys.clear() this.clearRangeValues() } diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 7561e33172..a2fbd9dc99 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -232,9 +232,11 @@ domain. An index's advertised comparison options also define its executable comparator; metadata cannot claim an order that the index does not use. Explicit `undefined` range and cursor bounds denote the indexed nullish comparator group, while an absent bound denotes the start or end of the index. An ordered index -groups exact value buckets that compare at the same position and keeps a live -representative for each group, so range traversal and ordered limits cannot drop -rows whose distinct values are comparator-equal. Compiler tokens belong to one +groups row values by comparator position independently of exact equality +buckets and keeps a live representative for each group. One exact bucket may +span several positions under a custom comparator, while distinct exact values +may share a position. Range traversal and ordered limits must retain every row +in either case. Compiler tokens belong to one compiled graph. This keeps every operator in the graph on the same identity relation. Objects, functions, and local symbols are weakly keyed where the runtime supports weak symbol keys. Older runtimes retain local symbols strongly diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index bb4d1220e0..72a1f82685 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -119,8 +119,10 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { } // If both are Temporal objects, use compareTemporalValues for correct semantic ordering - if (isTemporal(a) && isTemporal(b)) { - return compareTemporalValues(a, b) + const aTemporal = brandedTemporal(a) + const bTemporal = brandedTemporal(b) + if (aTemporal && bTemporal) { + return compareBrandedTemporalValues(a, b, aTemporal, bTemporal) } // Symbols have identity but no built-in order: relational comparison throws. @@ -264,6 +266,27 @@ function normalizeBinary(value: Uint8Array): string { return normalizedKey(`binary`, bytes) } +/** Read a Temporal brand and display through its prototype, not own shadows. */ +type BrandedTemporal = { prototype: object; kind: string; text: string } + +function brandedTemporal(value: unknown): BrandedTemporal | undefined { + if (value === null || typeof value !== `object`) return undefined + const prototype = Object.getPrototypeOf(value) + if (!prototype || !isTemporal(prototype)) return undefined + const toString = Object.getOwnPropertyDescriptor(prototype, `toString`)?.value + if (typeof toString !== `function`) return undefined + try { + // The native/polyfill method checks internal slots. A forged tag or a + // borrowed Temporal prototype cannot claim a Temporal ordered position. + const text = Reflect.apply(toString, value, []) + return typeof text === `string` + ? { prototype, kind: prototype[Symbol.toStringTag], text } + : undefined + } catch { + return undefined + } +} + /** * Sentinel value representing undefined in normalized form. * This allows distinguishing between "start from beginning" (undefined parameter) @@ -293,13 +316,13 @@ export function normalizeValue(value: any): any { const timestamp = dateTimestamp(value) if (timestamp !== undefined) return timestamp - if (isTemporal(value)) { - const kind = value[Symbol.toStringTag] + // Query equality comes from the branded value, never an own display shadow. + const temporal = brandedTemporal(value) + if (temporal) { + const { prototype, kind, text } = temporal if (kind === `Temporal.ZonedDateTime`) { // Read Temporal's internal slots through its prototype getters. Own // properties can shadow the public accessors without changing equals(). - const prototype = Object.getPrototypeOf(value) - if (prototype?.[Symbol.toStringTag] !== kind) return value let epochNanoseconds: unknown let timeZoneId: unknown let calendarId: unknown @@ -339,7 +362,7 @@ export function normalizeValue(value: any): any { } return value } - return normalizedKey(`temporal`, `${kind}:${value.toString()}`) + return normalizedKey(`temporal`, `${kind}:${text}`) } // Normalize Uint8Arrays/Buffers to a string representation for Map key usage @@ -366,22 +389,20 @@ export function normalizeForBTree(value: any): any { /** Keep Temporal's native order separate from its exact index lookup key. */ export function indexOrderValue(value: any, exactKey: any): any { - return isTemporal(value) ? value : exactKey + return brandedTemporal(value) ? value : exactKey } /** Supply custom comparators the Temporal display key used before raw ordering. */ export function indexCustomComparatorValue(value: unknown): unknown { - return isTemporal(value) - ? normalizedKey( - `temporal`, - `${value[Symbol.toStringTag]}:${value.toString()}`, - ) + const temporal = brandedTemporal(value) + return temporal + ? normalizedKey(`temporal`, `${temporal.kind}:${temporal.text}`) : value } /** Recover the exact lookup key from a stored ordered value. */ export function indexExactKey(value: any): any { - return isTemporal(value) ? normalizeValue(value) : value + return brandedTemporal(value) ? normalizeValue(value) : value } /** @@ -413,16 +434,30 @@ export function denormalizeUndefined(value: any): any { * Callers must ensure both arguments are Temporal objects; mixed types throw. */ export function compareTemporalValues(a: unknown, b: unknown): number { - const aTag = (a as Record)[Symbol.toStringTag] as string - const bTag = (b as Record)[Symbol.toStringTag] as string - if (aTag !== bTag) { + return compareBrandedTemporalValues( + a, + b, + brandedTemporal(a), + brandedTemporal(b), + ) +} + +function compareBrandedTemporalValues( + a: unknown, + b: unknown, + aTemporal: BrandedTemporal | undefined, + bTemporal: BrandedTemporal | undefined, +): number { + const aTag = aTemporal?.kind ?? `unbranded` + const bTag = bTemporal?.kind ?? `unbranded` + if (!aTemporal || !bTemporal || aTag !== bTag) { throw new TypeError( devBuild() && process.env.NODE_ENV !== `production` ? `Cannot order Temporal values of different types: ${aTag} vs ${bTag}` : codedMessage(167, { aTag, bTag }), ) } - const prototype = Object.getPrototypeOf(a) as { + const prototype = aTemporal.prototype as { constructor?: { compare?: (x: unknown, y: unknown) => number } } | null const compare = prototype?.constructor?.compare @@ -448,8 +483,10 @@ export function compareTemporalValues(a: unknown, b: unknown): number { * arguments are non-null. */ export function compareValues(a: unknown, b: unknown): number { - if (isTemporal(a) && isTemporal(b)) { - return compareTemporalValues(a, b) + const aTemporal = brandedTemporal(a) + const bTemporal = brandedTemporal(b) + if (aTemporal && bTemporal) { + return compareBrandedTemporalValues(a, b, aTemporal, bTemporal) } return (a as any) < (b as any) ? -1 : (a as any) > (b as any) ? 1 : 0 } diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index b2a05ee7f2..1d127bbff5 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -7,6 +7,7 @@ import { createLiveQueryCollection } from '../src/query/live-query-collection' import { areValuesEqual, ascComparator, + compareValues, defaultComparator, descComparator, isUnorderable, @@ -1365,7 +1366,9 @@ describe(`areValuesEqual property-based tests`, () => { * Legal inputs are ISO PlainDates from years 2000..2030 with days 1..28 and * ZonedDateTimes within two seconds of a fixed instant in four offset zones. * Equal-value branches construct separate objects, and the zoned branch forces - * different zones at the same instant. The production paths are the exported + * different zones at the same instant. A fixed genuine ZonedDateTime also + * shadows its own display tag; the supplied instant still determines order. + * The production paths are the exported * `ascComparator` under the default options and `compareValues` in hostile * dispatch histories. Each return is the observation checkpoint; * this lane makes no indexed or public-query claim. Comparator @@ -1589,6 +1592,20 @@ describe(`Temporal default-order refinement`, () => { expect(() => assertZonedDateTimeOrder(sample, truncated)).toThrow() }) + it(`keeps a genuine ZonedDateTime's instant order when its own tag is shadowed`, () => { + const utc = zonedValue(0, `+00:00`) + const peer = zonedValue(0, `+05:30`) + const later = zonedValue(1, `+05:30`) + Object.defineProperty(utc, Symbol.toStringTag, { value: `ordinary` }) + + // The supplied offsets, not the instance's display property, define the + // model positions. The direct comparator and helper are separate paths. + expect(ascComparator(utc, peer, DEFAULT_COMPARE_OPTIONS)).toBe(0) + expect(ascComparator(peer, utc, DEFAULT_COMPARE_OPTIONS)).toBe(0) + expect(compareValues(utc, peer)).toBe(0) + expect(sign(ascComparator(utc, later, DEFAULT_COMPARE_OPTIONS))).toBe(-1) + }) + it(`keeps a PlainDate constructor override from poisoning later comparisons`, async () => { const { ascComparator: isolatedComparator, compareValues: isolatedValues } = await freshTemporalComparison() diff --git a/packages/db/tests/custom-index-order-selection-oracle.test.ts b/packages/db/tests/custom-index-order-selection-oracle.test.ts new file mode 100644 index 0000000000..cf4c22b425 --- /dev/null +++ b/packages/db/tests/custom-index-order-selection-oracle.test.ts @@ -0,0 +1,169 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { BasicIndex } from '../src/indexes/basic-index.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { Func, PropRef, Value } from '../src/query/ir.js' +import { DEFAULT_COMPARE_OPTIONS } from '../src/utils.js' +import { withOracleCleanup } from './utils.js' + +/** + * An ordered Collection snapshot may use an index only when the index executes + * the requested comparison. The live-query ARCHITECTURE.md Identity contract + * says that advertised comparison options define the executable comparator. + * A custom index comparator has no identity in the ordinary ascending query + * options, so it cannot supply that query's order. Exact equality lookup on + * the index remains a separate, valid operation. + * + * This bounded witness compares public snapshots of three unique numeric rows + * in an unindexed Collection and a Collection with a descending custom index. + * It covers BasicIndex and BTreeIndex at the ready checkpoint, with one order + * term and limit one. It does not claim arbitrary custom comparator laws, + * mutation histories, provider ordering, or live-query publication. + */ +type Row = { id: string; rank: number } + +const rows: ReadonlyArray = [ + { id: `middle`, rank: 20 }, + { id: `last`, rank: 30 }, + { id: `first`, rank: 10 }, +] + +// The reference sorts source facts numerically. It knows neither index order +// nor the index-selection code; the first public key must be `first`. +const expectedFirstKey = [...rows].sort((a, b) => a.rank - b.rank)[0]!.id +const ascendingFirst = { + orderBy: [ + { + expression: new PropRef([`rank`]), + compareOptions: DEFAULT_COMPARE_OPTIONS, + }, + ], + limit: 1, +} +const descendingFirst = { + ...ascendingFirst, + orderBy: [ + { + expression: new PropRef([`rank`]), + compareOptions: { + ...DEFAULT_COMPARE_OPTIONS, + direction: `desc` as const, + }, + }, + ], +} + +function collectionWithRows(id: string) { + return createCollection({ + id, + getKey: (row) => row.id, + autoIndex: `off`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of rows) write({ type: `insert`, value: row }) + commit() + markReady() + }, + }, + }) +} + +const indexTypes = [ + [`BasicIndex`, BasicIndex], + [`BTreeIndex`, BTreeIndex], +] as const + +describe.each(indexTypes)( + `%s custom comparator order selection`, + (name, IndexType) => { + it(`keeps a descending custom index out of an ascending public snapshot`, async () => { + const scanned = collectionWithRows(`custom-index-order-scan-${name}`) + const indexed = collectionWithRows(`custom-index-order-indexed-${name}`) + const ordinary = collectionWithRows(`custom-index-order-ordinary-${name}`) + + await withOracleCleanup(async () => { + await Promise.all([ + scanned.stateWhenReady(), + indexed.stateWhenReady(), + ordinary.stateWhenReady(), + ]) + const customIndex = indexed.createIndex((row) => row.rank, { + indexType: IndexType, + options: { compareFn: (a: number, b: number) => b - a }, + }) + const ordinaryIndex = ordinary.createIndex((row) => row.rank, { + indexType: IndexType, + }) + + // Legal history: install the same three rows, then add the custom index. + // The exact lookup control checks that excluding the index from order + // selection does not change its exact-value membership. + expect(customIndex.equalityLookup(20)).toEqual(new Set([`middle`])) + expect(customIndex.lookup(`eq`, 20)).toEqual(new Set([`middle`])) + expect(customIndex.supportsRangeOptimization).toBe(false) + expect(ordinaryIndex.supportsRangeOptimization).toBe(true) + expect( + indexed + .currentStateAsChanges({ + where: new Func(`eq`, [new PropRef([`rank`]), new Value(20)]), + optimizedOnly: true, + }) + ?.map(({ key }) => key), + ).toEqual([`middle`]) + + // Both reads are public Collection snapshots after readiness. The + // unindexed read is a second formulation of the same first-row law. + const scannedKeys = scanned + .currentStateAsChanges(ascendingFirst) + ?.map(({ key }) => key) + const indexedKeys = indexed + .currentStateAsChanges(ascendingFirst) + ?.map(({ key }) => key) + expect(scannedKeys).toEqual([expectedFirstKey]) + expect(indexedKeys).toEqual(scannedKeys) + expect( + ordinary + .currentStateAsChanges({ ...ascendingFirst, optimizedOnly: true }) + ?.map(({ key }) => key), + ).toEqual(scannedKeys) + + // A read that forbids a scan must decline the custom index. Returning + // `last` here is the hostile design this witness rejects. + expect( + indexed.currentStateAsChanges({ + ...ascendingFirst, + optimizedOnly: true, + }), + ).toBeUndefined() + + // Reversing the advertised direction would also reverse the unknown + // custom comparator. The fallback and ordinary reversed index must + // still agree with an unindexed descending read. + const scannedDescending = scanned + .currentStateAsChanges(descendingFirst) + ?.map(({ key }) => key) + expect(scannedDescending).toEqual([`last`]) + expect( + indexed.currentStateAsChanges(descendingFirst)?.map(({ key }) => key), + ).toEqual(scannedDescending) + expect( + ordinary + .currentStateAsChanges({ ...descendingFirst, optimizedOnly: true }) + ?.map(({ key }) => key), + ).toEqual(scannedDescending) + expect( + indexed.currentStateAsChanges({ + ...descendingFirst, + optimizedOnly: true, + }), + ).toBeUndefined() + }, [ + () => indexed.cleanup(), + () => ordinary.cleanup(), + () => scanned.cleanup(), + ]) + }) + }, +) diff --git a/packages/db/tests/index-update-oracle.property.test.ts b/packages/db/tests/index-update-oracle.property.test.ts index e1cef0d8af..5cb79a9500 100644 --- a/packages/db/tests/index-update-oracle.property.test.ts +++ b/packages/db/tests/index-update-oracle.property.test.ts @@ -36,6 +36,8 @@ import type { CompareOptions } from '../src/query/builder/types.js' * Exact equality membership also stays distinct from comparator position: * replacing a value with one in a different exact-equality bucket at the * same comparator position must move its exact lookup membership. These laws + * also allow a custom callback to put signed zero values in different order + * positions while Map equality keeps them in one exact bucket. These laws * concern returned index results, not bucket identity or maintenance work. * A build is one of those direct actions even when a source value is * `undefined`: default nulls-first order must hold after build just as it @@ -1618,6 +1620,44 @@ describe.each([ ) describe.each(indexTypes)(`%s comparator groups`, (_indexName, IndexType) => { + test.each([`add`, `build`] as const)( + `follows a custom callback that separates signed zero after %s`, + (construction) => { + const rows = [ + { key: `negative`, value: -0 }, + { key: `positive`, value: 0 }, + { key: `later`, value: 1 }, + ] + const rank = (value: unknown): number => + Object.is(value, -0) ? 0 : value === 0 ? 1 : 2 + const index = new IndexType(1, new PropRef([`value`]), undefined, { + compareFn: (left, right) => rank(left) - rank(right), + }) + if (construction === `add`) { + for (const row of rows) index.add(row.key, row) + } else { + index.build( + rows.map((row): [string, { value: number }] => [row.key, row]), + ) + } + + // Map's SameValueZero defines the expected exact membership; the + // independent rank labels define the callback's three order positions. + expect(index.equalityLookup(-0)).toEqual( + new Set([`negative`, `positive`]), + ) + expect(index.takeFromStart(3)).toEqual([`negative`, `positive`, `later`]) + expect(index.rangeQuery({ from: -0, to: -0 })).toEqual( + new Set([`negative`]), + ) + expect(index.take(2, -0)).toEqual([`positive`, `later`]) + + index.remove(`negative`, rows[0]) + expect(index.equalityLookup(0)).toEqual(new Set([`positive`])) + expect(index.takeFromStart(2)).toEqual([`positive`, `later`]) + }, + ) + test(`rejects stale retired identity outputs without rejecting live reuse`, () => { const symbol = Symbol(`same group`) const retired = { key: `old`, groupId: 0, value: [symbol] } diff --git a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts index 19f0744427..9479a91d26 100644 --- a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts +++ b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts @@ -8,8 +8,8 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' /** * # Which values does `in` find in a list? * - * Contract: `in(value, list)` is SQL `IN` under three-valued logic. A null or - * undefined value is UNKNOWN (`null`). A list that is not an array is FALSE. + * Contract: TanStack DB's `inArray(value, list)` returns UNKNOWN (`null`) for a + * null or undefined value. A list that is not an array is FALSE. * Otherwise the result is TRUE when some list item equals the value under the * evaluator's equality, and FALSE when none does. A null or undefined item * never equals a value. The equality is the one `eq` documents in @@ -19,9 +19,10 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * - A valid Date equals its millisecond timestamp, and another Date with the * same timestamp. * - Two Uint8Arrays (Buffers included) are equal when their bytes are equal. - * - A Temporal value equals another of the same kind when its native - * `.equals()` relation holds. This includes equivalent time-zone aliases - * for ZonedDateTime, but excludes a distinct fixed-offset zone. + * - Temporal kinds with an intrinsic `.equals()` use that relation. This + * includes equivalent time-zone aliases for ZonedDateTime, but excludes a + * distinct fixed-offset zone. Duration has no `.equals()`; its ten fields + * must match, so PT1H and PT60M are distinct. * - Every other pair compares with `===`, so `-0` equals `0`, `1n` does not * equal `1`, and objects compare by reference. * @@ -34,7 +35,8 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * internal normalization prefix, booleans, `0`, `-0`, `1`, `NaN`, `1n`, a * valid Date and its timestamp, an invalid Date, equal and unequal byte * arrays, a Buffer, Temporal values including ZonedDateTime zone aliases, - * a forged tag and shadowed zone accessor, one shared object and a + * a forged tag, a borrowed prototype, shadowed display members, and Duration + * forms, one shared object and a * structurally equal copy, `null` and `undefined`. Lists have zero to six * items and may repeat. The list is a constant `Value`, the shape that join * demand builds, or a row field, so changing lists are also checked. @@ -47,13 +49,44 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * A calibration test feeds the checker an evaluator that compares raw values * without normalization and requires the checker to reject it. * - * Limits: this owner checks the boolean result of `in` only. WHERE + * Limits: this owner checks the boolean result of `in` for ordinary values + * from one Temporal polyfill copy and the listed display shadows. It does not + * claim mixed Temporal implementations, replaced genuine prototypes, or + * arbitrary custom prototypes that imitate Temporal intrinsics. WHERE * publication, indexes, and subset loading have other owners. */ const PROPERTY = `evaluators.in` const PREFIX = `\u0000tanstack-db:binary:` const shared = { id: 1 } +const plainDate = Temporal.PlainDate.from(`2024-01-01`) +const forgedPlainDate = { + [Symbol.toStringTag]: `Temporal.PlainDate`, + toString: () => `2024-01-01`, +} +const borrowedPlainDate = Object.assign( + Object.create(Temporal.PlainDate.prototype) as object, + { toString: () => `2024-01-01` }, +) +const shadowedTagPlainDate = Temporal.PlainDate.from(`2024-01-01`) +Object.defineProperty(shadowedTagPlainDate, Symbol.toStringTag, { + value: `ordinary`, +}) +const shadowedTextPlainDate = Temporal.PlainDate.from(`2024-01-01`) +Object.defineProperty(shadowedTextPlainDate, `toString`, { + value: () => `unrelated text`, +}) +const hourDuration = Temporal.Duration.from(`PT1H`) +const sameHourDuration = Temporal.Duration.from(`PT1H`) +const sixtyMinuteDuration = Temporal.Duration.from(`PT60M`) +const shadowedHourDuration = Temporal.Duration.from(`PT1H`) +Object.defineProperty(shadowedHourDuration, `toString`, { + value: () => `PT2H`, +}) +const forgedHourDuration = { + [Symbol.toStringTag]: `Temporal.Duration`, + toString: () => `PT1H`, +} const utcZoned = Temporal.ZonedDateTime.from(`2024-01-15T09:00:00+00:00[UTC]`) const aliasZoned = Temporal.ZonedDateTime.from( `2024-01-15T09:00:00+00:00[Etc/UTC]`, @@ -91,9 +124,18 @@ const pool: ReadonlyArray = [ new Uint8Array([1, 2]), new Uint8Array([1, 3]), Buffer.from([1, 2]), - Temporal.PlainDate.from(`2024-01-01`), + plainDate, Temporal.PlainDate.from(`2024-01-01`), Temporal.PlainTime.from(`10:00`), + forgedPlainDate, + borrowedPlainDate, + shadowedTagPlainDate, + shadowedTextPlainDate, + hourDuration, + sameHourDuration, + sixtyMinuteDuration, + shadowedHourDuration, + forgedHourDuration, utcZoned, aliasZoned, offsetZoned, @@ -114,22 +156,63 @@ function isBytes(value: unknown): value is Uint8Array { ) } -function isTemporalValue(value: unknown): value is { - [Symbol.toStringTag]: string - equals: (other: unknown) => boolean -} { - const tag = - typeof value === `object` && value !== null - ? (value as { [Symbol.toStringTag]?: unknown })[Symbol.toStringTag] - : undefined - return ( - typeof value === `object` && - value !== null && - typeof (value as { equals?: unknown }).equals === `function` && - typeof tag === `string` && - tag.startsWith(`Temporal.`) && - Object.getPrototypeOf(value)?.[Symbol.toStringTag] === tag - ) +// The trusted polyfill methods inspect Temporal's internal slots. A user can +// shadow an instance's tag, equals, or toString, and can borrow its prototype; +// none of those visible properties establishes the brand. The bounded model +// tests the trusted methods directly and never imports production's classifier. +const temporalEqualities = [ + [`Instant`, Temporal.Instant.prototype.equals], + [`PlainDate`, Temporal.PlainDate.prototype.equals], + [`PlainDateTime`, Temporal.PlainDateTime.prototype.equals], + [`PlainMonthDay`, Temporal.PlainMonthDay.prototype.equals], + [`PlainTime`, Temporal.PlainTime.prototype.equals], + [`PlainYearMonth`, Temporal.PlainYearMonth.prototype.equals], + [`ZonedDateTime`, Temporal.ZonedDateTime.prototype.equals], +] as const + +const durationFields = [ + `years`, + `months`, + `weeks`, + `days`, + `hours`, + `minutes`, + `seconds`, + `milliseconds`, + `microseconds`, + `nanoseconds`, +] as const +const durationGetters = durationFields.map( + (field) => + Object.getOwnPropertyDescriptor(Temporal.Duration.prototype, field)!.get!, +) + +type TemporalBrand = + | { kind: string; equals: (other: unknown) => boolean } + | { kind: `Duration`; fields: ReadonlyArray } + +function temporalBrand(value: unknown): TemporalBrand | undefined { + if (typeof value !== `object` || value === null) return undefined + for (const [kind, equals] of temporalEqualities) { + try { + if (Reflect.apply(equals, value, [value]) === true) { + return { + kind, + equals: (other) => Reflect.apply(equals, value, [other]), + } + } + } catch { + // A borrowed or forged prototype has no Temporal internal slots. + } + } + try { + return { + kind: `Duration`, + fields: durationGetters.map((get) => Reflect.apply(get, value, [])), + } + } catch { + return undefined + } } function isNotANumber(value: unknown): boolean { @@ -155,13 +238,22 @@ function referenceEqual(left: unknown, right: unknown): boolean { Array.from(left).every((byte, index) => byte === right[index]) ) } - if (isTemporalValue(left) || isTemporalValue(right)) { - return ( - isTemporalValue(left) && - isTemporalValue(right) && - left[Symbol.toStringTag] === right[Symbol.toStringTag] && - left.equals(right) - ) + const leftTemporal = temporalBrand(left) + const rightTemporal = temporalBrand(right) + if (leftTemporal || rightTemporal) { + if ( + !leftTemporal || + !rightTemporal || + leftTemporal.kind !== rightTemporal.kind + ) { + return false + } + if (`fields` in leftTemporal && `fields` in rightTemporal) { + return leftTemporal.fields.every( + (field, index) => field === rightTemporal.fields[index], + ) + } + return `equals` in leftTemporal && leftTemporal.equals(right) } return asComparable(left) === asComparable(right) } @@ -175,7 +267,7 @@ function referenceIn(value: unknown, list: unknown): boolean | null { ) } -type InCase = { value: unknown; list: Array; listFromRow: boolean } +type InCase = { value: unknown; list: unknown; listFromRow: boolean } const item = fc.constantFrom(...pool) const inCase: fc.Arbitrary = fc.record({ @@ -210,8 +302,14 @@ function check(evaluate: Evaluate, testCase: InCase): void { } function describeCase(testCase: InCase): string { - const show = (value: unknown): string => - typeof value === `bigint` + const show = (value: unknown): string => { + if (value === forgedPlainDate) return `forged PlainDate tag` + if (value === borrowedPlainDate) return `borrowed PlainDate prototype` + if (value === shadowedTagPlainDate) return `PlainDate with shadowed tag` + if (value === shadowedTextPlainDate) return `PlainDate with shadowed text` + if (value === shadowedHourDuration) return `Duration with shadowed text` + if (value === forgedHourDuration) return `forged Duration tag` + return typeof value === `bigint` ? `${value}n` : Object.is(value, -0) ? `-0` @@ -222,11 +320,21 @@ function describeCase(testCase: InCase): string { : value === shared ? `shared` : typeof value === `object` && value !== null - ? String(value) + ? (() => { + try { + return String(value) + } catch { + return `object with borrowed Temporal prototype` + } + })() : typeof value === `string` ? JSON.stringify(value) : String(value) - return `value=${show(testCase.value)} list=[${testCase.list.map(show).join(`, `)}] listFromRow=${testCase.listFromRow}` + } + const list = Array.isArray(testCase.list) + ? `[${testCase.list.map(show).join(`, `)}]` + : show(testCase.list) + return `value=${show(testCase.value)} list=${list} listFromRow=${testCase.listFromRow}` } describe(`in evaluator`, () => { @@ -257,6 +365,13 @@ describe(`in evaluator`, () => { check(evaluateWithProduction, testCase) }) + it.each([false, true])( + `returns false for a non-array %s list`, + (listFromRow) => { + check(evaluateWithProduction, { value: `a`, list: `a`, listFromRow }) + }, + ) + it.each([false, true])( `matches equivalent zone aliases in a %s list and rejects a distinct zone`, (listFromRow) => { @@ -291,8 +406,55 @@ describe(`in evaluator`, () => { }, ) + it.each( + [false, true].flatMap((listFromRow) => + [ + [`forged own tag`, forgedPlainDate], + [`borrowed prototype`, borrowedPlainDate], + [`shadowed own tag`, shadowedTagPlainDate], + [`shadowed toString`, shadowedTextPlainDate], + ].map(([name, value]) => ({ name, value, listFromRow })), + ), + )( + `uses the native brand for $name with listFromRow=$listFromRow`, + ({ value, listFromRow }) => { + check(evaluateWithProduction, { + value, + list: [plainDate], + listFromRow, + }) + }, + ) + + it.each([false, true])( + `matches Duration fields, while keeping equivalent representations distinct in a %s list`, + (listFromRow) => { + check(evaluateWithProduction, { + value: sameHourDuration, + list: [hourDuration], + listFromRow, + }) + check(evaluateWithProduction, { + value: sixtyMinuteDuration, + list: [hourDuration], + listFromRow, + }) + check(evaluateWithProduction, { + value: shadowedHourDuration, + list: [hourDuration], + listFromRow, + }) + check(evaluateWithProduction, { + value: forgedHourDuration, + list: [hourDuration], + listFromRow, + }) + }, + ) + it(`rejects display-string equality for equivalent zone aliases`, () => { const lexical: Evaluate = ({ value, list }) => + Array.isArray(list) && list.some((candidate) => String(candidate) === String(value)) expect(() => check(lexical, { @@ -309,7 +471,7 @@ describe(`in evaluator`, () => { const raw: Evaluate = ({ value, list }) => value === null || value === undefined ? null - : list.some((entry) => entry === value) + : Array.isArray(list) && list.some((entry) => entry === value) expect(() => fc.assert( fc.property(inCase, (testCase) => check(raw, testCase)), diff --git a/packages/db/tests/temporal-equality-alias-oracle.test.ts b/packages/db/tests/temporal-equality-alias-oracle.test.ts index 3e8e1dad6a..10609a2ab2 100644 --- a/packages/db/tests/temporal-equality-alias-oracle.test.ts +++ b/packages/db/tests/temporal-equality-alias-oracle.test.ts @@ -12,14 +12,18 @@ import { withOracleCleanup } from './utils.js' * order. At one instant, UTC and Etc/UTC are equivalent zone aliases, while * the fixed-offset +00:00 zone remains a different identity. * - * This bounded oracle covers ordinary values from one polyfill copy and one - * instant. The fixture assigns equality classes independently of production - * normalization. Direct BasicIndex/BTreeIndex add and build histories compare - * exact key sets after each returned action. Public Collection reads compare - * an unindexed scan and an optimized-only indexed snapshot after readiness. - * A forged display tag and an own property shadowing a genuine Temporal zone - * must not join UTC's class. This oracle does not claim all IANA aliases, - * mixed implementations, joins, live-query publication, or provider behavior. + * The ZonedDateTime lane covers ordinary values from one polyfill copy and one + * instant. A neighboring PlainDate lane checks that native brand and equality + * survive own display shadows while forged tags retain ordinary object + * identity. Fixture-assigned classes, not production normalization, predict + * exact key sets. Direct BasicIndex/BTreeIndex add, build, and removal + * histories compare those sets after each returned action. Public Collection + * reads compare an unindexed scan and an optimized-only indexed snapshot after + * readiness. The mixed forged/genuine PlainDate Collection has no index: + * default range order across an ordinary object and Temporal is unspecified. + * This oracle does not claim all IANA aliases, arbitrary spoofed prototypes, + * mixed Temporal implementations, joins, live-query publication, or provider + * behavior. */ type Row = { id: string; when: Temporal.ZonedDateTime } type Sample = { @@ -119,6 +123,14 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { expect(index.lookup(`eq`, aliasProbe)).toEqual(expectedKeys(`utc`)) expect(index.inArrayLookup([aliasProbe])).toEqual(expectedKeys(`utc`)) expect(index.equalityLookup(offsetProbe)).toEqual(expectedKeys(`offset`)) + // The neighboring instant and calendar classes ensure that a repair + // does not collapse every Temporal value sharing a named zone. + expect(index.equalityLookup(source[3]!.when)).toEqual( + expectedKeys(`later`), + ) + expect(index.equalityLookup(source[4]!.when)).toEqual( + expectedKeys(`calendar`), + ) index.remove(source[0]!.id, source[0]!) expect(index.equalityLookup(aliasProbe)).toEqual(new Set([`utc-alias`])) @@ -231,3 +243,100 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () expect(keys).toEqual(new Set()) }, [() => collection.cleanup()]) }) + +it(`uses the PlainDate brand for direct lookups and a public unindexed snapshot`, async () => { + const genuine = Temporal.PlainDate.from(`2024-01-01`) + const fake = { + [Symbol.toStringTag]: `Temporal.PlainDate`, + toString: () => `2024-01-01`, + } + const borrowed = Object.assign( + Object.create(Temporal.PlainDate.prototype) as object, + { toString: () => `2024-01-01` }, + ) + const shadowedTag = Temporal.PlainDate.from(`2024-01-01`) + Object.defineProperty(shadowedTag, Symbol.toStringTag, { + value: `ordinary`, + }) + const shadowedText = Temporal.PlainDate.from(`2024-01-01`) + Object.defineProperty(shadowedText, `toString`, { + value: () => `unrelated text`, + }) + expect(Temporal.PlainDate.prototype.equals.call(shadowedTag, genuine)).toBe( + true, + ) + expect(Temporal.PlainDate.prototype.equals.call(shadowedText, genuine)).toBe( + true, + ) + + const brandSamples: ReadonlyArray<{ + id: string + when: unknown + matches: boolean + }> = [ + { id: `genuine`, when: genuine, matches: true }, + { id: `forged-tag`, when: fake, matches: false }, + { id: `borrowed-prototype`, when: borrowed, matches: false }, + { id: `shadowed-tag`, when: shadowedTag, matches: true }, + { id: `shadowed-text`, when: shadowedText, matches: true }, + ] + const probe = Temporal.PlainDate.from(`2024-01-01`) + expect(() => + Reflect.apply(Temporal.PlainDate.prototype.equals, fake, [probe]), + ).toThrow() + expect(() => + Reflect.apply(Temporal.PlainDate.prototype.equals, borrowed, [probe]), + ).toThrow() + + // A one-row index tests each exact equality class without ordering mixed + // ordinary objects and Temporal values, whose range order is unspecified. + for (const [, IndexType] of indexTypes) { + for (const sample of brandSamples) { + const index = new IndexType(1, new PropRef([`when`])) + index.add(sample.id, sample) + expect(index.equalityLookup(probe)).toEqual( + sample.matches ? new Set([sample.id]) : new Set(), + ) + } + // Both insertion orders must retain the same equality class when a live + // representative is removed. The forged objects are intentionally kept + // out of this ordered index: the default comparator has no promised order + // for a mixed ordinary-object and Temporal domain. + for (const first of [genuine, shadowedTag]) { + const second = first === genuine ? shadowedTag : genuine + const index = new IndexType(1, new PropRef([`when`])) + index.add(`first`, { when: first }) + index.add(`second`, { when: second }) + expect(index.equalityLookup(probe)).toEqual(new Set([`first`, `second`])) + index.remove(`first`, { when: first }) + expect(index.equalityLookup(probe)).toEqual(new Set([`second`])) + } + } + + const collection = createCollection<{ id: string; when: unknown }, string>({ + id: `temporal-plain-date-brand`, + getKey: (row) => row.id, + autoIndex: `off`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const { id, when } of brandSamples) { + write({ type: `insert`, value: { id, when } }) + } + commit() + markReady() + }, + }, + }) + await withOracleCleanup(async () => { + await collection.stateWhenReady() + const predicate = new Func(`eq`, [new PropRef([`when`]), new Value(probe)]) + const keys = new Set( + collection + .currentStateAsChanges({ where: predicate }) + ?.map(({ value }) => value.id), + ) + expect(keys).toEqual(new Set([`genuine`, `shadowed-tag`, `shadowed-text`])) + }, [() => collection.cleanup()]) +}) diff --git a/packages/db/tests/temporal-index-order-oracle.test.ts b/packages/db/tests/temporal-index-order-oracle.test.ts index f062b01c5a..ac247931da 100644 --- a/packages/db/tests/temporal-index-order-oracle.test.ts +++ b/packages/db/tests/temporal-index-order-oracle.test.ts @@ -18,11 +18,15 @@ import { orderedEntriesArray, withOracleCleanup } from './utils.js' * around the epoch. Equal-instant peers may appear in either key order. It * checks direct BasicIndex/BTreeIndex adds, builds, and value cursors, one * removal/insertion/update continuation, then the public ordered Collection - * snapshot with limit 1. A separate direct-index witness preserves a custom - * comparator that accepts the string keys supplied before raw Temporal - * ordering was introduced. It does not claim arbitrary Temporal kinds, - * time-zone transitions, longer mutation histories, public/provider cursor - * continuation, or live-query publication. + * snapshot with limit 1. Direct custom-comparator witnesses preserve the + * former display-string operands and let the callback order two rows whose + * values share one query-equality class. They check add/build, exact lookup, + * range and cursor boundaries, representative removal, and an update that + * changes callback order without changing query equality. Fixed add/build + * histories also shadow a genuine value's display tag; the value cursor must + * still cross its native instant position. These checks do not claim arbitrary + * Temporal kinds, time-zone transitions, longer histories, + * public/provider cursor continuation, or live-query publication. */ type Row = { id: string; when: Temporal.ZonedDateTime } @@ -276,6 +280,28 @@ describe.each(indexTypes)( }, ) + it.each([`add`, `build`] as const)( + `keeps instant order when a genuine value shadows its display tag after %s`, + (construction) => { + const rows = rowsFromFixtures().slice(0, 3) + Object.defineProperty(rows[0]!.when, Symbol.toStringTag, { + value: `ordinary`, + }) + const index = new IndexType(1, new PropRef([`when`])) + if (construction === `add`) { + for (const row of rows) index.add(row.id, row) + } else { + index.build(rows.map((row): [string, Row] => [row.id, row])) + } + + // The source fixture supplies the instant labels. An instance display + // property cannot move its row out of the shared earliest position. + expectOrderedPrefix(index.takeFromStart(3), 3, fixtures.slice(0, 3)) + expectOrderedPrefix(index.takeFromStart(1), 1, fixtures.slice(0, 3)) + expect(index.take(1, rows[0]!.when)).toEqual([`a-one-nanosecond-later`]) + }, + ) + it(`retains a string-compatible custom comparator across negative instants`, () => { const [minusTen, minusHundred] = rowsFromFixtures(negativeFixtures) const index = new IndexType(1, new PropRef([`when`]), undefined, { @@ -293,6 +319,98 @@ describe.each(indexTypes)( ) }) + it.each([`add`, `build`] as const)( + `orders equal UTC aliases by a custom callback after %s`, + (construction) => { + const source = [ + { + id: `a-utc`, + when: Temporal.ZonedDateTime.from(`2024-01-15T09:00:00+00:00[UTC]`), + }, + { + id: `z-alias`, + when: Temporal.ZonedDateTime.from( + `2024-01-15T09:00:00+00:00[Etc/UTC]`, + ), + }, + ] + const [utc, alias] = source + expect(utc!.when.equals(alias!.when)).toBe(true) + const callbackArguments: Array<[string, string]> = [] + const index = new IndexType(1, new PropRef([`when`]), undefined, { + compareFn: (left: string, right: string) => { + callbackArguments.push([left, right]) + return left.localeCompare(right) + }, + }) + + // The exact-equality class has both keys. The callback's legacy + // display keys place Etc/UTC before UTC, despite the opposite key order. + if (construction === `add`) { + for (const row of source) index.add(row.id, row) + } else { + index.build(source.map((row): [string, Row] => [row.id, row])) + } + expect(index.equalityLookup(utc!.when)).toEqual( + new Set([`a-utc`, `z-alias`]), + ) + expect(index.takeFromStart(2)).toEqual([`z-alias`, `a-utc`]) + expect(index.rangeQuery({ to: alias!.when })).toEqual( + new Set([`z-alias`]), + ) + expect( + index.rangeQuery({ from: alias!.when, fromInclusive: false }), + ).toEqual(new Set([`a-utc`])) + expect(index.take(2, alias!.when)).toEqual([`a-utc`]) + expect(index.takeReversed(2, utc!.when)).toEqual([`z-alias`]) + expect(callbackArguments.length).toBeGreaterThan(0) + const expectedOperands = new Set( + source.map( + ({ when }) => + `\u0000tanstack-db:temporal:Temporal.ZonedDateTime:${when.toString()}`, + ), + ) + for (const [left, right] of callbackArguments) { + expect(expectedOperands.has(left)).toBe(true) + expect(expectedOperands.has(right)).toBe(true) + } + + // Removal supplies the exact prior source row. The first stored + // representative must be replaced by a surviving row's value. + index.remove(utc!.id, utc) + expect(index.takeFromStart(2)).toEqual([`z-alias`]) + expect(orderedEntriesArray(index)[0]?.[0]).toBe(alias!.when) + expect(index.equalityLookup(alias!.when)).toEqual(new Set([`z-alias`])) + }, + ) + + it(`moves one row within its exact class when its custom order changes`, () => { + const utc = { + id: `a-utc`, + when: Temporal.ZonedDateTime.from(`2024-01-15T09:00:00+00:00[UTC]`), + } + const alias = { + id: `z-alias`, + when: Temporal.ZonedDateTime.from(`2024-01-15T09:00:00+00:00[Etc/UTC]`), + } + const index = new IndexType(1, new PropRef([`when`]), undefined, { + compareFn: (left: string, right: string) => left.localeCompare(right), + }) + index.add(utc.id, utc) + index.add(alias.id, alias) + expect(index.takeFromStart(2)).toEqual([`z-alias`, `a-utc`]) + + // The query equality class stays fixed, while the callback position + // changes from the later UTC display string to the earlier alias text. + const moved = { id: utc.id, when: alias.when } + index.update(utc.id, utc, moved) + expect(index.equalityLookup(alias.when)).toEqual( + new Set([`a-utc`, `z-alias`]), + ) + expect(index.takeFromStart(2)).toEqual([`a-utc`, `z-alias`]) + expect(index.take(2, alias.when)).toEqual([]) + }) + it(`publishes the earliest instant from an indexed ordered Collection snapshot`, async () => { const rows = rowsFromFixtures() const collection = createCollection({ diff --git a/packages/db/tests/utils-oracle.property.test.ts b/packages/db/tests/utils-oracle.property.test.ts index c4cda036c1..53d9210f2e 100644 --- a/packages/db/tests/utils-oracle.property.test.ts +++ b/packages/db/tests/utils-oracle.property.test.ts @@ -11,8 +11,9 @@ import { deepEquals, equalPersistedSnapshotValues } from '../src/utils' * * Independent value constructors exercise reflexivity, symmetry, copied * structure, and changed-leaf inequality for primitives, arrays, records, - * dates, regexes, typed bytes, and selected Temporal types. Cross-type values - * remain unequal. Separate bounded graph laws construct corresponding rings + * dates, regexes, typed bytes, and selected Temporal types. Named cross-type + * pairs remain unequal, even when a primitive displays or encodes a Date or + * RegExp value. Separate bounded graph laws construct corresponding rings * and acyclic copies without reusing the production walk. * * Arbitrary different cycle topologies are outside this model. Naming that @@ -88,6 +89,9 @@ const arbitraryPrimitive = fc.oneof( ) const arbitraryDate = fc.date().map((d) => new Date(d.getTime())) +const arbitraryFiniteDate = fc + .integer({ min: -1_000_000_000_000, max: 2_000_000_000_000 }) + .map((time) => new Date(time)) const arbitraryRegExp = fc .tuple(fc.string(), fc.constantFrom(``, `g`, `i`, `gi`, `m`, `gim`)) @@ -190,8 +194,9 @@ const arbitraryDefinedValue = arbitrarySingleValue.filter( // independently generated values, so both equal and unequal pairs are legal. // The fixed witnesses exercise those two outcomes; deleting either outcome // would leave an always-true or always-false comparator undetected. The -// distinct-type grammar is deliberately restricted to Date/Temporal pairs; -// it does not claim every cross-class combination. String lengths, integer +// generated distinct-type grammar names Date/Temporal pairs, Date versus its +// ISO text and epoch number, and RegExp versus its source text; it does not +// claim every cross-class combination. String lengths, integer // magnitude, and object keys are fast-check defaults; explicit array/record // bounds are 0–5 for equivalence and 0–10 for structural copies, bytes 0–50, // Float32Array 0–10, Temporal years 1–9999/months 1–12/days 1–28, and @@ -691,6 +696,66 @@ describe(`deepEquals property-based tests`, () => { }) describe(`cross-type comparisons`, () => { + // A Date's display and numeric timestamp encode the same instant, but + // change-event equality keeps the Date value class distinct. The finite + // generator makes both encodings legal; the 2023 case preserves the old + // unit test's exact operands. + utilsProperty([arbitraryFiniteDate])( + `Date differs from its ISO text and epoch number in both directions`, + (date) => { + assertDistinctTypes(date, date.toISOString(), deepEquals) + assertDistinctTypes(date, date.getTime(), deepEquals) + }, + ) + + it(`keeps a 2023 Date distinct from its old primitive probes`, () => { + const date = new Date(`2023-01-01T00:00:00Z`) + expectCopyAndChanged( + date, + new Date(`2023-01-01T00:00:00Z`), + new Date(`2023-01-02T00:00:00Z`), + ) + assertDistinctTypes(date, `2023-01-01T00:00:00Z`, deepEquals) + assertDistinctTypes(date, date.getTime(), deepEquals) + }) + + // RegExp source text does not acquire the regex's value class, even when + // it spells the same pattern. The source and flags still govern equality + // between two genuine RegExps in the laws below. + utilsProperty([arbitraryRegExp])( + `RegExp differs from its source text in both directions`, + (regex) => assertDistinctTypes(regex, regex.source, deepEquals), + ) + + it(`keeps /abc/g distinct from its old source-text probe`, () => { + expectCopyAndChanged(/abc/g, /abc/g, /abc/i) + expectCopyAndChanged(/abc/g, /abc/g, /def/g) + expectEqualityPair(/test/i, /test/i, true) + assertDistinctTypes(/abc/g, `abc`, deepEquals) + }) + + it(`rejects primitive-coercing equality for Date and RegExp`, () => { + const date = new Date(`2023-01-01T00:00:00Z`) + const dateAsNumber = (left: unknown, right: unknown) => + (left instanceof Date ? left.getTime() : left) === + (right instanceof Date ? right.getTime() : right) + const dateAsString = (left: unknown, right: unknown) => + (left instanceof Date ? left.toISOString() : left) === + (right instanceof Date ? right.toISOString() : right) + const regexpAsSource = (left: unknown, right: unknown) => + (left instanceof RegExp ? left.source : left) === + (right instanceof RegExp ? right.source : right) + expect(() => + assertDistinctTypes(date, date.getTime(), dateAsNumber), + ).toThrow(`distinct types must be unequal (forward)`) + expect(() => + assertDistinctTypes(date, date.toISOString(), dateAsString), + ).toThrow(`distinct types must be unequal (forward)`) + expect(() => assertDistinctTypes(/abc/g, `abc`, regexpAsSource)).toThrow( + `distinct types must be unequal (forward)`, + ) + }) + utilsProperty([arbitraryDate, arbitraryTemporalDuration])( `generated Date and Temporal.Duration are unequal in both directions`, (date, duration) => assertDistinctTypes(date, duration, deepEquals), diff --git a/packages/db/tests/utils.test.ts b/packages/db/tests/utils.test.ts index acffea3b69..1bbf4e4730 100644 --- a/packages/db/tests/utils.test.ts +++ b/packages/db/tests/utils.test.ts @@ -288,42 +288,6 @@ describe(`deepEquals`, () => { }) }) - describe(`Date objects`, () => { - it(`should handle identical dates`, () => { - const date1 = new Date(`2023-01-01T00:00:00Z`) - const date2 = new Date(`2023-01-01T00:00:00Z`) - expect(deepEquals(date1, date2)).toBe(true) - }) - - it(`should handle different dates`, () => { - const date1 = new Date(`2023-01-01T00:00:00Z`) - const date2 = new Date(`2023-01-02T00:00:00Z`) - expect(deepEquals(date1, date2)).toBe(false) - }) - - it(`should handle date vs non-date`, () => { - const date = new Date(`2023-01-01T00:00:00Z`) - expect(deepEquals(date, `2023-01-01T00:00:00Z`)).toBe(false) - expect(deepEquals(date, date.getTime())).toBe(false) - }) - }) - - describe(`RegExp objects`, () => { - it(`should handle identical regexes`, () => { - expect(deepEquals(/abc/g, /abc/g)).toBe(true) - expect(deepEquals(/test/i, /test/i)).toBe(true) - }) - - it(`should handle different regexes`, () => { - expect(deepEquals(/abc/g, /abc/i)).toBe(false) - expect(deepEquals(/abc/g, /def/g)).toBe(false) - }) - - it(`should handle regex vs non-regex`, () => { - expect(deepEquals(/abc/g, `abc`)).toBe(false) - }) - }) - describe(`Map objects`, () => { it(`should handle identical maps`, () => { const map1 = new Map([ From b7e01b5e2e24426409744f343156d4672f27bdc0 Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 19:26:42 -0600 Subject: [PATCH 14/46] fix(db): close hostile oracle counterexamples --- .../fix-custom-index-order-selection.md | 2 +- .changeset/fix-temporal-index-order.md | 2 +- docs/contributing/oracle-coverage.md | 38 +- .../comparison-unit-retirement-round-6.md | 16 + .../custom-comparator-signed-zero.md | 46 ++- .../custom-index-order-selection.md | 33 +- .../oracle-reviews/temporal-equality-alias.md | 52 ++- .../utils-unit-retirement-round-7.md | 25 +- packages/db/src/collection/change-events.ts | 4 +- packages/db/src/indexes/auto-index.ts | 43 ++- packages/db/src/indexes/basic-index.ts | 2 +- packages/db/src/indexes/btree-index.ts | 2 +- packages/db/src/utils/btree.ts | 10 +- packages/db/src/utils/comparison.ts | 54 ++- packages/db/src/utils/index-optimization.ts | 24 +- .../tests/comparison-oracle.property.test.ts | 3 +- ...ustom-index-order-selection-oracle.test.ts | 146 ++++++- .../index-update-oracle.property.test.ts | 148 +++++++ .../in-evaluator-oracle.property.test.ts | 160 +++++++- .../temporal-equality-alias-oracle.test.ts | 364 +++++++++++++++++- .../db/tests/utils-oracle.property.test.ts | 30 +- 21 files changed, 1111 insertions(+), 93 deletions(-) diff --git a/.changeset/fix-custom-index-order-selection.md b/.changeset/fix-custom-index-order-selection.md index 29d70c28c8..9dd7278679 100644 --- a/.changeset/fix-custom-index-order-selection.md +++ b/.changeset/fix-custom-index-order-selection.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Use a custom-comparator index for exact lookup without selecting its order for an ordinary Collection `orderBy` read. +Use a custom-comparator index for exact lookup without selecting its order for an ordinary Collection `orderBy` read. When a compatible ordinary index also exists, select it for ordered reads; eager indexing now creates one if needed. diff --git a/.changeset/fix-temporal-index-order.md b/.changeset/fix-temporal-index-order.md index a951ea4a2c..e10846d874 100644 --- a/.changeset/fix-temporal-index-order.md +++ b/.changeset/fix-temporal-index-order.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Order Temporal values by their calendar or instant semantics in indexed reads. Match ZonedDateTime equality across equivalent time-zone aliases while keeping distinct zones separate. Keep Temporal equality when genuine values shadow display members, and reject ordinary objects with forged tags. Preserve custom index comparator operands and per-row order even when values share one query-equality class, including signed zero. Prevent one Temporal value from changing the comparator used for later values. +Order Temporal values by their calendar or instant semantics in indexed reads. Match ZonedDateTime equality across equivalent time-zone aliases while keeping distinct zones separate. Preserve PlainMonthDay reference years in unindexed equality, and PlainYearMonth reference days and Duration fields in indexed equality, when default display strings collide. Keep Temporal equality when genuine values shadow display members, and reject ordinary objects with forged tags. Preserve custom index comparator operands and per-row order even when values share one query-equality class, including signed zero through updates and BTree ranges. Prevent one Temporal value from changing the comparator used for later values. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index b63b936fc6..68be474a80 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -473,7 +473,7 @@ order, public Date range predicates, cursor continuation, proxies, subclasses, replaced Date prototypes or `getTime` methods, and other Date-consuming paths remain outside that witness. -The comparison oracle also owns a bounded [Temporal order transfer](oracle-reviews/comparison-unit-retirement-round-6.md). Supplied ISO calendar fields and epoch-nanosecond offsets predict direct `ascComparator` signs and exact zeros for ordinary `PlainDate` and `ZonedDateTime` values from one polyfill copy. Fixed cases preserve the two retired unit blocks; generated cases vary calendar fields, offset zones, equal instants, and adjacent nanoseconds. Fresh-module histories reject a first-input constructor cache poisoned by a genuine instance or forged tag, including the direct `compareValues` path. A genuine ZonedDateTime with an own shadowed display tag retains its instant position through both direct helpers. This lane does not establish public query or index behavior. Custom Temporal subclasses, replaced instance prototypes, mixed native/polyfill implementations, other Temporal kinds, and public ordering/filter publication need separate witnesses. +The comparison oracle also owns a bounded [Temporal order transfer](oracle-reviews/comparison-unit-retirement-round-6.md). Supplied ISO calendar fields and epoch-nanosecond offsets predict direct `ascComparator` signs and exact zeros for ordinary `PlainDate` and `ZonedDateTime` values from one polyfill copy. Fixed cases preserve the two retired unit blocks; generated cases vary calendar fields, offset zones, equal instants, and adjacent nanoseconds. Fresh-module histories reject a first-input constructor cache poisoned by a genuine instance or forged tag, including the direct `compareValues` path. Both direct helpers also receive a right-hand PlainDate with a constructor shadow. A genuine ZonedDateTime with an own shadowed display tag retains its instant position through both direct helpers. This lane does not establish public query or index behavior. Custom Temporal subclasses, spoofed or replaced instance prototypes, mixed native/polyfill implementations, other Temporal kinds, and public ordering/filter publication need separate witnesses. Spoofed Temporal prototypes can still cause asymmetric comparisons; the comparison owner needs a dedicated brand and mixed-value witness before that class can be called closed. The [Temporal index order oracle](../../packages/db/tests/temporal-index-order-oracle.test.ts) and its [review record](oracle-reviews/temporal-index-order.md) own direct @@ -502,6 +502,21 @@ row-supplied alias lists. A plain object with a forged Temporal tag and a genuine value with a shadowed zone accessor must stay outside the UTC class. The neighboring PlainDate lane checks forged tags, borrowed prototypes, and genuine own-tag or display shadows at direct lookup and unindexed public `eq`. +The ZonedDateTime fixture also contains genuine UTC rows with own display-tag +and `toString` shadows; direct indexes and both public `eq` snapshot paths +retain their alias equality. The `in` evaluator owner distinguishes +PlainMonthDay reference years, PlainYearMonth reference days, and Duration +field vectors that share default display text. Fixed controls reach compiled +`eq`, constant-list `in`, and row-supplied `in` in both operand directions. +The alias owner also checks a public unindexed `eq` scan with two native-equal +PlainMonthDay rows and a distinct reference year. Direct BasicIndex/BTreeIndex +add, build, removal, re-add, and public indexed `eq` cover the orderable +PlainYearMonth and Duration cases. Indexed PlainMonthDay equality with multiple +classes remains open: the ordered indexes cannot compare those classes. A +future exact-only path must preserve equality while making public `orderBy` +throw for PlainMonthDay, as changelog #1519 requires. A one-class index bypass +failed that order law and was not retained. +Its byte model excludes a genuine Uint8Array with an own display-tag shadow. Other zone aliases, custom spoofed prototypes, mixed Temporal implementations, mixed ordinary-object/Temporal ordered indexes, joins, grouping, query identity, live-query publication, and provider paths remain unproved. Direct @@ -513,9 +528,14 @@ that a public ordered Collection snapshot does not select a reverse-number custom comparator by default ascending or descending `compareOptions`. Unindexed and ordinary-index snapshots supply controls; optimized-only reads reject the custom order while optimized exact equality still uses that index. -The fixed ready-state fixture covers BasicIndex and BTreeIndex, one numeric -field, and limit one. Mutation histories, arbitrary callbacks, and provider or -live-query publication remain outside this owner. +With a custom index created before a compatible ordinary index, ordered reads +select the latter in both directions. Eager indexing adds one compatible +index when the custom index alone is present and does not add a duplicate on +repeated reads. Single and compound numeric range predicates also select +the compatible index; an eager range subscriber creates one when needed. +The fixed ready-state fixture covers BasicIndex and +BTreeIndex, one numeric field, and limit one. Mutation histories, arbitrary +callbacks, and provider or live-query publication remain outside this owner. The [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts) also has a [cross-realm Date equality matrix](oracle-reviews/cross-realm-date-where-publication.md). @@ -564,10 +584,12 @@ records the current transfer boundary and review. The [signed-zero custom comparator witness](oracle-reviews/custom-comparator-signed-zero.md) in the index refinement oracle separates `-0` and `+0` into two ordered positions while exact lookup retains one SameValueZero class. Fixed add/build -and removal histories check BasicIndex and BTreeIndex takes, a cursor, a -bounded range, and exact membership. It does not establish arbitrary custom -callbacks or public planner selection; the custom index order selection owner -above checks that separate boundary. +and removal histories check BasicIndex and BTreeIndex takes, cursors, ranges, +and exact membership. Sign-changing updates and removal/re-addition must move +or release the ordered position; mixed-sign inclusive/exclusive and reversed +ranges are checked at a BTree leaf and across a 33-row split. It does not +establish arbitrary custom callbacks or public planner selection; the custom +index order selection owner above checks that separate boundary. The alias-scope owner also has a compiler invariant witness for a malformed include plan: removing a captured ancestor's parent projection after the diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md index e072598bd9..2e243feb1e 100644 --- a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md @@ -101,3 +101,19 @@ and ZonedDateTime operands and checked all four assertions at the direct evaluator/index tests with no type errors. It replayed properties 39 and 40 with seed `20260911` and path `0`. The reviewer corrected the ORC-011 classification above. Cross-realm Temporal values remain outside this verdict. + +## Review of the next committed head + +At `c4c07da42`, a fresh reviewer again found all four retired assertions +preserved. It found a helper-specific false green: the right-hand +constructor-shadow history exercised `ascComparator` with the shadowed +PlainDate, but passed ordinary values to `compareValues`. The direct helper +now receives that same shadowed right operand and must return the supplied +negative calendar sign. The focused comparison, alias, and utils run passed +258 tests with no type errors after this addition. + +The reviewer also reproduced asymmetric comparisons for ordinary objects +given a spoofed Temporal prototype. This is outside the declared +custom-prototype domain and is not evidence of closure for that domain. The +comparison owner needs a separate brand and mixed-value witness before +claiming it. Fresh review of the final committed head remains pending. diff --git a/docs/contributing/oracle-reviews/custom-comparator-signed-zero.md b/docs/contributing/oracle-reviews/custom-comparator-signed-zero.md index 10dfaa67fe..81c8d59e64 100644 --- a/docs/contributing/oracle-reviews/custom-comparator-signed-zero.md +++ b/docs/contributing/oracle-reviews/custom-comparator-signed-zero.md @@ -16,9 +16,47 @@ key, while exact membership keeps the ordinary `Map` key. All four new histories and the full 416-test index refinement owner pass with no type errors. The independent rank model is supplied by the fixture, not copied from either index. This fixed witness covers only the stated signed-zero callback and -direct operations; arbitrary callbacks, longer histories, public order -selection, and provider paths remain separate. +direct operations; arbitrary callbacks, public order selection, and provider +paths remain separate. -## Exact-commit hostile review +## Exact-commit hostile review and follow-up -Pending fresh review of the committed oracle and production paths. +At `c4c07da42`, a fresh reviewer found three legal histories the first +witness missed. Both index types skipped ordered maintenance for `-0 → +0` +because SameValueZero treats the positions as equal, even though the callback +does not. BTreeIndex could then keep a stale ordered bucket after update, +remove, and re-add. BTree range traversal also used JavaScript `===` to +collapse distinct signed-zero bounds at leaf and internal nodes; a 33-row +fixture reached the split-tree case. + +The extended direct oracle uses an independent source Map and a supplied +numeric rank to predict membership and order after sign-changing updates, +removal, and re-addition. It checks exact SameValueZero lookup, ordered takes, +cursors, and inclusive, exclusive, reversed, and point ranges on both index +types; the split case crosses BTree children. Four focused cases failed on +the old production at the intended ordered or range comparison. Changing +only the BTree point-range shortcuts left the two update cases red. Ordered +maintenance now uses `Object.is` in both indexes; the BTree shortcuts use it +to recognize only identical bounds. The index/BTree owner passed 436 tests +with no type errors after repair. The five lint warnings were preexisting. + +## Oracle guide audit + +| Requirement | Evidence and limit | +| --- | --- | +| ORC-001 | The direct index contract accepts a numeric callback. The maintainer selected per-row callback order within a SameValueZero exact class; the architecture's comparator-group law requires every row to survive ordered traversal. | +| ORC-002 | A source Map records owners and raw values. Fixture ranks predict order independently of index buckets and traversal code. The callback itself is the input contract, not an imported production comparator. | +| ORC-003 | Prose beside the source model, actions, driver, and assertions states the law, legal history, direct path, observations, and after-action comparison. | +| ORC-004 | Not triggered: the sign-changing and split-tree histories are fixed witnesses, not a generated grammar claim. | +| ORC-005 | Both real index types return exact sets, ordered arrays, cursor arrays, and range sets after each action. Those recordings expose missing, duplicate, and misplaced rows. | +| ORC-006 | Four new cases failed on the pre-repair code at intended take or range comparisons; the BTree-only fix left the update cases red. | +| ORC-007 | Not triggered by these fixed witnesses. Other generated index properties keep their own campaigns. | +| ORC-008 | The Map retains each owner's raw sign because a later update or removal can distinguish `-0` from `+0`; merging those states would lose the next ordered observation. | +| ORC-009 | Model ownership maps to indexed row keys. Model rank is a fixture abstraction of callback position, separate from exact SameValueZero membership. | +| ORC-010 | Direct indexes own no external resources or cleanup action; an assertion is the failure checkpoint. | +| ORC-011 | The source Map and fresh rank sort differ from the incremental indexes. No plausible shared semantic fault within this bounded callback remains identified; arbitrary callback laws remain open. | +| ORC-012 | This record names the counterexamples, RED/GREEN sequence, scoped claim, and pending final review. | +| ORC-013 | `-0` and `+0` share exact lookup but have distinct callback positions; inclusive versus exclusive and reversed mixed-sign bounds reject a SameValueZero range rule. | +| ORC-014 | No provider or host handoff is claimed. | + +Fresh review of the final committed head remains pending. diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md index 2749e8b619..9aed82266d 100644 --- a/docs/contributing/oracle-reviews/custom-index-order-selection.md +++ b/docs/contributing/oracle-reviews/custom-index-order-selection.md @@ -24,8 +24,8 @@ index/query campaign passed with no type errors after repair. The oracle covers a ready in-memory Collection, one numeric field, one order term, limit one, and two built-in index types. It does not establish arbitrary -custom comparator laws, mutation histories, live-query publication, provider -ordering, or a performance guarantee that another eligible index is created. +custom comparator laws, mutation histories, live-query publication, or +provider ordering. ## Oracle guide audit @@ -46,6 +46,31 @@ ordering, or a performance guarantee that another eligible index is created. | ORC-013 | A reverse-number callback makes the wrong index choice visibly return rank 30 instead of rank 10. | | ORC-014 | Controlled in-memory sync ends at the ready snapshot; no provider handoff is claimed. | -## Exact-commit hostile review +## Exact-commit hostile review and follow-up -Pending fresh review of the final committed oracle and production path. +At `c4c07da42`, a fresh reviewer reproduced a planner failure with two +indexes on the same field: when the custom index was created first and a +compatible ordinary index second, `optimizedOnly` ordered reads still +returned no result. The original oracle put the indexes in separate +Collections, so first-match selection was never challenged. An eager +Collection also treated the custom index as sufficient and created no +compatible ordinary index. Numeric range predicates had the same first-match +problem. The reviewer reproduced both index types and both read directions. + +The extended fixture now puts both indexes in one Collection and checks +ascending and descending public first rows. It also starts with a custom +index under eager mode, requires one compatible index to be created, and +checks a repeated read does not create another. Optimized exact `eq` still +uses the first custom index, verified through its lookup spy. The fixture +also checks optimized single and compound numeric ranges and an eager range +subscriber. Four ordered-selection cases failed before that repair at the +missing first row. Four more cases failed before the range repair: both index +types returned no optimized `gt` result, and both eager subscriptions failed +to create a compatible index. The compound assertion followed the single +range assertion in one test, so no separate pre-fix compound failure was +observed. Selection and eager admission now skip a custom index only when +ordered traversal is required. The new oracle passed 8 cases, and the +six-file related campaign passed 145 tests with no type errors. Lint and +diff checks passed. + +Fresh review of the final committed head remains pending. diff --git a/docs/contributing/oracle-reviews/temporal-equality-alias.md b/docs/contributing/oracle-reviews/temporal-equality-alias.md index 93e2cc9ae1..534d0e3960 100644 --- a/docs/contributing/oracle-reviews/temporal-equality-alias.md +++ b/docs/contributing/oracle-reviews/temporal-equality-alias.md @@ -64,19 +64,19 @@ Temporal methods remain outside the one-polyfill brand guarantee. | Requirement | Evidence and limit | | --- | --- | -| ORC-001 | Changelog #1519 and Temporal `.equals()` authorize the equality class; the opening oracle comment limits implementations, paths, and observations. | -| ORC-002 | Fixture-assigned classes predict key sets without calling production normalization. A separate native `.equals()` check calibrates the fixture. | +| ORC-001 | Changelog #1519 and Temporal `.equals()` authorize the equality classes. The Duration field-vector rule follows #1519's explicit `eq` distinction. The opening oracle comment limits implementations, paths, and observations. | +| ORC-002 | Fixture-assigned classes predict key sets without calling production normalization. Native `.equals()` calibrates supported kinds; captured intrinsic Duration fields calibrate Duration. | | ORC-003 | Prose beside the model, add/build history, Collection driver, and assertions explains their roles. | -| ORC-004 | Fixed classes distinguish alias equality, distinct zone, adjacent nanosecond, calendar, forged and borrowed brands, and genuine display shadows. The `in` owner generates list combinations over its bounded pool; no arbitrary object grammar is claimed. | -| ORC-005 | Direct lookups observe both index types; an indexed public snapshot spies on `lookup`, and the unindexed snapshot checks the evaluator path after readiness. | -| ORC-006 | The frozen display-string design failed alias checks. The first tuple design failed forged-tag and shadowed-zone controls. Commit `1265abeb5` failed eleven brand/shadow/Duration checks; a lexical `in` mutant fails its comparison. | +| ORC-004 | Not triggered by this fixed oracle. The separate generated `in` owner controls its bounded list grammar. | +| ORC-005 | Direct lookups observe both index types for orderable kinds; an indexed public snapshot spies on `lookup`, and unindexed snapshots check the evaluator path after readiness. PlainMonthDay has no indexed claim. | +| ORC-006 | The frozen display-string design failed alias checks. The first tuple design failed forged-tag and shadowed-zone controls. Commit `1265abeb5` failed eleven brand/shadow/Duration checks; `c4c07da42` failed the intrinsic-field equality controls. A lexical `in` mutant fails its comparison. | | ORC-007 | Not triggered by this fixed oracle. The adjacent `in` owner retains fixed and random generated campaigns. | | ORC-008 | The fixture model is immutable and has no reference-model state. | | ORC-009 | The model's class labels denote query equality, distinct from the order oracle's instant labels. | | ORC-010 | Collection cleanup preserves the primary mismatch. Direct index checks own no external resource. | -| ORC-011 | The direct lookup and public scan cross different production paths, but share normalization. Native `.equals()` is the independent semantic calibration; this record does not claim a second full reference model. | -| ORC-012 | This record names the owner, RED controls, limits, and required exact-commit review. | -| ORC-013 | Alias equality versus fixed-offset inequality separates display-only and instant-only keys; native-brand controls reject forged/borrowed impostors while genuine display shadows preserve equality. | +| ORC-011 | Direct lookup and public scan cross different production paths, but share normalization. Native `.equals()` and the Duration field-vector rule provide independent semantic calibration; this record does not claim a second full reference model. | +| ORC-012 | This record names the owner, RED controls, limits, the rejected PlainMonthDay index design, and required exact-commit review. | +| ORC-013 | Alias equality versus fixed-offset inequality separates display-only and instant-only keys; same-display reference-field and Duration pairs reject string-only keys. Native-brand controls reject forged/borrowed impostors while genuine display shadows preserve equality. | | ORC-014 | The Collection uses controlled in-memory sync and ends its claim at the snapshot; no provider handoff is claimed. | ## Limits and exact-commit review @@ -92,4 +92,38 @@ unproved. The index-order owner now checks a custom comparator that separates query-equal aliases. Direct D2 `groupBy` has a separately approved kind-and-string policy. -Pending fresh hostile review of the exact final commit. +At committed head `c4c07da42`, a fresh reviewer found no mismatch in the +exercised alias cases but found that the previously reported genuine +ZonedDateTime own-tag failure lacked a direct witness here. The fixture now +adds genuine UTC rows with an own shadowed display tag and an own shadowed +`toString`. Native `ZonedDateTime.prototype.equals` confirms their equality +with the alias probe. Add, build, removal, unindexed public `eq`, and +optimized-only indexed public `eq` must all retain those rows. The focused +run passed 11 alias tests with no type errors. Final committed-head review +remains pending. + +## Intrinsic-key counterexamples and index boundary + +The same `c4c07da42` review found three Temporal equality classes whose +default display omits identity fields. Native `.equals()` distinguishes +PlainMonthDay reference years and PlainYearMonth reference days. Changelog +#1519 keeps Duration field representations distinct under `eq`, even when +both display as `PT1S`. Compiled `eq`, constant `in`, and row-supplied `in` +returned false matches before the production repair. New fixed pairs check +equal copies and unequal same-display values in both operand directions. +Normalization now reads the intrinsic reference fields and ten Duration +fields. The receiving `in` model reads captured polyfill methods and getters. + +This alias oracle carries the same equality classes to a public unindexed +PlainMonthDay scan. Its PlainYearMonth and Duration lanes check direct index +add, build, removal, and re-add, plus public scan and indexed `eq`. Each +fixture assigns classes from constructed fields without using production +normalization. The six focused suites passed 619 tests after the repair. + +An attempted one-class PlainMonthDay index bypass was rejected during law +review. With two native-equal rows, a BasicIndex made public `orderBy` return +keys, while the unindexed query threw as changelog #1519 specifies. Adding a +later distinct reference-year row then threw during a valid sync commit. The +bypass was removed. Direct and public indexed PlainMonthDay equality across +multiple classes remains open. An exact-only index path needs a separate +oracle for add, build, remove, update, and the orderBy rejection. diff --git a/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md b/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md index 3aa5c83671..573209be39 100644 --- a/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md +++ b/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md @@ -44,7 +44,7 @@ public change-event trace. Those need their own owner or receiving witness. | ORC-004 | Every removed operand and outcome has a fixed reconstruction; generated finite Dates and RegExps vary the new cross-type boundaries. | | ORC-005 | Each check calls exported `deepEquals` and observes the immediate boolean return. No public publication claim is made. | | ORC-006 | Injected primitive-coercing comparators fail the exact inequality assertion for all three newly recovered boundaries. | -| ORC-007 | Existing `utilsProperty` runs each property with fixed seed and random seed and supports direct seed/path replay. | +| ORC-007 | `utilsProperty` runs the same property with fixed and random seeds. The source/flags property replayed a captured failing seed/path directly, as recorded below. | | ORC-008 | Not triggered: the value model has no mutable state. | | ORC-009 | Date, RegExp, ISO text, epoch number, and source text are distinct value classes in this model; no production storage state is modeled. | | ORC-010 | The direct check owns no cleanup resource; its assertion is the failure checkpoint. | @@ -53,6 +53,25 @@ public change-event trace. Those need their own owner or receiving witness. | ORC-013 | A primitive with matching displayed or numeric content distinguishes class-preserving equality from a plausible coercing design. | | ORC-014 | Not triggered: the direct call has no provider or host boundary. | -## Exact-commit hostile review +## Exact-commit hostile review and follow-up -Pending fresh review of the final committed oracle and the removed blocks. +At committed head `c4c07da42`, the fresh reviewer confirmed that all six +retired assertions had fixed receiving witnesses and found no production bug +in this transfer. It found two broader oracle weaknesses: the generated +RegExp law varied flags but not source within a pair, and the two newly +registered cross-type properties had shifted existing ordinal replay IDs. +The generated source law now compares a same-flags, changed-source neighbor. +The two added properties are registered after the original properties, so +their old selectors retain their meanings. A focused run passed 139 oracle +tests with no type errors. + +For ORC-007 calibration, a temporary wrong model rule expected an arbitrary +RegExp to equal its changed-source neighbor. The fixed campaign failed at the +`deepEquals` return comparison after one test with seed `20260911` and shrink +path `0:0`. Direct replay through `UTILS_ORACLE_PROPERTY=36`, +`UTILS_ORACLE_SEED=20260911`, and `UTILS_ORACLE_PATH=0:0` reproduced that +same assertion failure. Restoring the model rule made the direct replay pass. +The temporary rule is absent from the final oracle. This demonstrates replay +fidelity for the changed-source property, not every property in the file. + +Fresh review of the final committed head remains pending. diff --git a/packages/db/src/collection/change-events.ts b/packages/db/src/collection/change-events.ts index a0df8fb6cb..621eb179b4 100644 --- a/packages/db/src/collection/change-events.ts +++ b/packages/db/src/collection/change-events.ts @@ -351,10 +351,12 @@ function getOrderedKeys( fieldPath, collection as CollectionImpl, compareOpts, + undefined, + true, ) // Find the index - const index = findIndexForField(collection, fieldPath, compareOpts) + const index = findIndexForField(collection, fieldPath, compareOpts, true) // A user-supplied index comparator is not described by compareOptions. // Exact equality lookup may still use that index, but its ordered walk diff --git a/packages/db/src/indexes/auto-index.ts b/packages/db/src/indexes/auto-index.ts index c72c82807a..2d7217ed43 100644 --- a/packages/db/src/indexes/auto-index.ts +++ b/packages/db/src/indexes/auto-index.ts @@ -22,6 +22,7 @@ export function ensureIndexForField< collection: CollectionImpl, compareOptions?: CompareOptions, compareFn?: (a: any, b: any) => number, + requireOrderedTraversal = false, ) { if (hasVirtualPropPath(fieldPath)) { return @@ -36,10 +37,13 @@ export function ensureIndexForField< ...collection.compareOptions, } - // Check if we already have an index for this field + // An ordered read needs an index whose comparator matches its options; + // an exact predicate can still use a custom index's equality buckets. const existingIndex = Array.from(collection.indexes.values()).find( (index) => - index.matchesField(fieldPath) && index.matchesCompareOptions(compareOpts), + index.matchesField(fieldPath) && + index.matchesCompareOptions(compareOpts) && + (!requireOrderedTraversal || index.supportsRangeOptimization), ) if (existingIndex) { @@ -94,18 +98,33 @@ export function ensureIndexForExpression< const indexableExpressions = extractIndexableExpressions(expression) - for (const { fieldName, fieldPath } of indexableExpressions) { - ensureIndexForField(fieldName, fieldPath, collection) + for (const { + fieldName, + fieldPath, + requireOrderedTraversal, + } of indexableExpressions) { + ensureIndexForField( + fieldName, + fieldPath, + collection, + undefined, + undefined, + requireOrderedTraversal, + ) } } -/** - * Extracts all indexable expressions from a where expression - */ +type IndexableExpression = { + fieldName: string + fieldPath: Array + requireOrderedTraversal: boolean +} + +/** Extracts indexable predicates and whether they need relational order. */ function extractIndexableExpressions( expression: BasicExpression, -): Array<{ fieldName: string; fieldPath: Array }> { - const results: Array<{ fieldName: string; fieldPath: Array }> = [] +): Array { + const results: Array = [] function extractFromExpression(expr: BasicExpression): void { if (expr.type !== `func`) { @@ -144,7 +163,11 @@ function extractIndexableExpressions( // For nested paths, use the full path joined with underscores as the field name // For simple paths, use the first (and only) element const fieldName = fieldPath.join(`_`) - results.push({ fieldName, fieldPath }) + results.push({ + fieldName, + fieldPath, + requireOrderedTraversal: func.name !== `eq` && func.name !== `in`, + }) } extractFromExpression(expression) diff --git a/packages/db/src/indexes/basic-index.ts b/packages/db/src/indexes/basic-index.ts index 21bf4a4fb9..2dc74330e6 100644 --- a/packages/db/src/indexes/basic-index.ts +++ b/packages/db/src/indexes/basic-index.ts @@ -218,7 +218,7 @@ export class BasicIndex< this.valueMap.get(newValue)?.has(key) && this.indexedKeys.has(key) ) { - if (!areSameValueZeroEqual(oldOrder, newOrder)) { + if (!Object.is(oldOrder, newOrder)) { this.removeFromOrder(key, oldOrder) this.addToOrder(key, newOrder) } diff --git a/packages/db/src/indexes/btree-index.ts b/packages/db/src/indexes/btree-index.ts index af20944eb4..69cddcc202 100644 --- a/packages/db/src/indexes/btree-index.ts +++ b/packages/db/src/indexes/btree-index.ts @@ -236,7 +236,7 @@ export class BTreeIndex< areSameValueZeroEqual(oldValue, newValue) && this.valueMap.get(newValue)?.keys.has(key) ) { - if (!areSameValueZeroEqual(oldOrder, newOrder)) { + if (!Object.is(oldOrder, newOrder)) { this.removeFromOrder(key, oldOrder) this.addToOrder(key, newOrder) } diff --git a/packages/db/src/utils/btree.ts b/packages/db/src/utils/btree.ts index 82366f6deb..42d841c90f 100644 --- a/packages/db/src/utils/btree.ts +++ b/packages/db/src/utils/btree.ts @@ -303,8 +303,9 @@ class BNode { ): void { const cmp = tree._compare let iLow, iHigh - if (high === low) { - // A point range (as used by delete) needs only one search. + if (Object.is(high, low)) { + // A point range (as used by delete) needs only one search. `===` also + // equates signed zero values that a supplied comparator can separate. if (!includeHigh) return iHigh = (iLow = this.indexOf(low, -1, cmp)) + 1 if (iLow < 0) return @@ -474,9 +475,10 @@ class BNodeInternal extends BNode { children = this.children let iLow = this.indexOf(low, 0, cmp), i = iLow - // A point range (as used by delete) needs only one search. + // Only identical bounds can use the one-child point-range shortcut. + // A supplied comparator may place -0 and +0 in different children. const iHigh = Math.min( - high === low ? iLow : this.indexOf(high, 0, cmp), + Object.is(high, low) ? iLow : this.indexOf(high, 0, cmp), keys.length - 1, ) for (; i <= iHigh; i++) { diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index 72a1f82685..e48d5edeb4 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -266,6 +266,19 @@ function normalizeBinary(value: Uint8Array): string { return normalizedKey(`binary`, bytes) } +const durationFields = [ + `years`, + `months`, + `weeks`, + `days`, + `hours`, + `minutes`, + `seconds`, + `milliseconds`, + `microseconds`, + `nanoseconds`, +] as const + /** Read a Temporal brand and display through its prototype, not own shadows. */ type BrandedTemporal = { prototype: object; kind: string; text: string } @@ -276,8 +289,9 @@ function brandedTemporal(value: unknown): BrandedTemporal | undefined { const toString = Object.getOwnPropertyDescriptor(prototype, `toString`)?.value if (typeof toString !== `function`) return undefined try { - // The native/polyfill method checks internal slots. A forged tag or a - // borrowed Temporal prototype cannot claim a Temporal ordered position. + // The native/polyfill method checks internal slots. An own forged tag or + // borrowed Temporal prototype cannot claim an ordered position. Arbitrary + // custom prototypes that spoof these methods remain outside this guard. const text = Reflect.apply(toString, value, []) return typeof text === `string` ? { prototype, kind: prototype[Symbol.toStringTag], text } @@ -362,6 +376,42 @@ export function normalizeValue(value: any): any { } return value } + if ( + kind === `Temporal.PlainMonthDay` || + kind === `Temporal.PlainYearMonth` + ) { + // Default display omits the reference ISO year/day that native equals() + // compares. The intrinsic string form with the calendar shown includes + // those fields and ignores an instance's own toString shadow. + const toString = Object.getOwnPropertyDescriptor( + prototype, + `toString`, + )?.value + try { + const fullText = Reflect.apply(toString, value, [ + { calendarName: `always` }, + ]) + return normalizedKey(`temporal`, `${kind}:${fullText}`) + } catch { + return value + } + } + if (kind === `Temporal.Duration`) { + // Duration's default display balances subsecond units. Equality under + // the #1519 contract instead preserves all ten intrinsic fields. + try { + const fields = durationFields.map((field) => { + const get = Object.getOwnPropertyDescriptor(prototype, field)?.get + return Reflect.apply(get!, value, []) as number + }) + if (fields.every((field) => typeof field === `number`)) { + return normalizedKey(`temporal`, JSON.stringify([kind, ...fields])) + } + } catch { + // A borrowed prototype has no Temporal internal slots. + } + return value + } return normalizedKey(`temporal`, `${kind}:${text}`) } diff --git a/packages/db/src/utils/index-optimization.ts b/packages/db/src/utils/index-optimization.ts index 4ee63233b1..d26195154b 100644 --- a/packages/db/src/utils/index-optimization.ts +++ b/packages/db/src/utils/index-optimization.ts @@ -40,12 +40,15 @@ export interface OptimizationResult { } /** - * Finds an index that matches a given field path + * Finds an index that matches a field path. Exact predicates can use a custom + * comparator's equality buckets; ordered traversal needs an index whose order + * is represented by its comparison options. */ export function findIndexForField( collection: CollectionLike, fieldPath: Array, compareOptions?: CompareOptions, + requireOrderedTraversal = false, ): IndexReader | undefined { if (hasVirtualPropPath(fieldPath)) { return undefined @@ -58,7 +61,8 @@ export function findIndexForField( for (const index of collection.indexes.values()) { if ( index.matchesField(fieldPath) && - index.matchesCompareOptions(compareOpts) + index.matchesCompareOptions(compareOpts) && + (!requireOrderedTraversal || index.supportsRangeOptimization) ) { if (!index.matchesDirection(compareOpts.direction)) { // Reversing the index also reverses where it keeps nulls, so the @@ -363,7 +367,7 @@ function optimizeCompoundRangeQuery< for (const [fieldKey, operations] of fieldOperations) { if (operations.length >= 2) { const fieldPath = JSON.parse(fieldKey) as Array - const index = findIndexForField(collection, fieldPath) + const index = findIndexForField(collection, fieldPath, undefined, true) // Only collapse this field into a range query when every bound's domain // orders the same way the index does and the index supports trustworthy @@ -526,7 +530,12 @@ function optimizeSimpleComparison< if (fieldArg && valueArg) { const fieldPath = (fieldArg as any).path - const index = findIndexForField(collection, fieldPath) + const index = findIndexForField( + collection, + fieldPath, + undefined, + operation !== `eq`, + ) if (index) { const queryValue = (valueArg as any).value @@ -602,7 +611,12 @@ function canOptimizeSimpleComparison< } if (fieldPath) { - const index = findIndexForField(collection, fieldPath) + const index = findIndexForField( + collection, + fieldPath, + undefined, + expression.name !== `eq`, + ) return index !== undefined } diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index 1d127bbff5..6e07038879 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -1645,6 +1645,7 @@ describe(`Temporal default-order refinement`, () => { ) const observed = [ isolatedComparator(january, shadowedJune, DEFAULT_COMPARE_OPTIONS), + isolatedValues(january, shadowedJune), isolatedComparator( ordinaryJanuary, ordinaryJune, @@ -1652,7 +1653,7 @@ describe(`Temporal default-order refinement`, () => { ), isolatedValues(ordinaryJanuary, ordinaryJune), ] - expect(observed.map(sign)).toEqual([expected, expected, expected]) + expect(observed.map(sign)).toEqual([expected, expected, expected, expected]) }) it(`keeps a compareValues-first constructor override from poisoning later comparisons`, async () => { diff --git a/packages/db/tests/custom-index-order-selection-oracle.test.ts b/packages/db/tests/custom-index-order-selection-oracle.test.ts index cf4c22b425..b63c6ae7fd 100644 --- a/packages/db/tests/custom-index-order-selection-oracle.test.ts +++ b/packages/db/tests/custom-index-order-selection-oracle.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it } from 'vitest' +import { describe, expect, it, vi } from 'vitest' import { createCollection } from '../src/collection/index.js' import { BasicIndex } from '../src/indexes/basic-index.js' import { BTreeIndex } from '../src/indexes/btree-index.js' @@ -16,9 +16,13 @@ import { withOracleCleanup } from './utils.js' * * This bounded witness compares public snapshots of three unique numeric rows * in an unindexed Collection and a Collection with a descending custom index. - * It covers BasicIndex and BTreeIndex at the ready checkpoint, with one order - * term and limit one. It does not claim arbitrary custom comparator laws, - * mutation histories, provider ordering, or live-query publication. + * It also checks a custom index followed by a compatible ordinary index, and + * eager index creation when the custom index is present first. Numeric range + * predicates must likewise find a later compatible index; a range subscriber + * must trigger eager creation when only a custom index exists. It covers + * BasicIndex and BTreeIndex at the ready checkpoint, with one order term and + * limit one. It does not claim arbitrary custom comparator laws, mutation + * histories, provider ordering, or live-query publication. */ type Row = { id: string; rank: number } @@ -31,6 +35,11 @@ const rows: ReadonlyArray = [ // The reference sorts source facts numerically. It knows neither index order // nor the index-selection code; the first public key must be `first`. const expectedFirstKey = [...rows].sort((a, b) => a.rank - b.rank)[0]!.id +const greaterThanTen = new Func(`gt`, [new PropRef([`rank`]), new Value(10)]) +const betweenTenAndThirty = new Func(`and`, [ + greaterThanTen, + new Func(`lt`, [new PropRef([`rank`]), new Value(30)]), +]) const ascendingFirst = { orderBy: [ { @@ -53,11 +62,15 @@ const descendingFirst = { ], } -function collectionWithRows(id: string) { +function collectionWithRows( + id: string, + defaultIndexType?: typeof BasicIndex | typeof BTreeIndex, +) { return createCollection({ id, getKey: (row) => row.id, - autoIndex: `off`, + autoIndex: defaultIndexType ? `eager` : `off`, + defaultIndexType, startSync: true, sync: { sync: ({ begin, write, commit, markReady }) => { @@ -78,6 +91,127 @@ const indexTypes = [ describe.each(indexTypes)( `%s custom comparator order selection`, (name, IndexType) => { + it(`selects a later compatible index while exact equality keeps the first custom index`, async () => { + const collection = collectionWithRows(`custom-index-coexistence-${name}`) + await withOracleCleanup(async () => { + await collection.stateWhenReady() + const custom = collection.createIndex((row) => row.rank, { + indexType: IndexType, + options: { compareFn: (a: number, b: number) => b - a }, + }) + collection.createIndex((row) => row.rank, { indexType: IndexType }) + + // Legal declaration order matters to first-match index search. The + // custom index is first and still serves exact equality; the ordinary + // index is second and must serve the ordered read. + const lookup = vi.spyOn(custom, `lookup`) + expect( + collection + .currentStateAsChanges({ + where: new Func(`eq`, [new PropRef([`rank`]), new Value(20)]), + optimizedOnly: true, + }) + ?.map(({ key }) => key), + ).toEqual([`middle`]) + expect(lookup).toHaveBeenCalledWith(`eq`, 20) + expect( + collection + .currentStateAsChanges({ ...ascendingFirst, optimizedOnly: true }) + ?.map(({ key }) => key), + ).toEqual([expectedFirstKey]) + expect( + collection + .currentStateAsChanges({ ...descendingFirst, optimizedOnly: true }) + ?.map(({ key }) => key), + ).toEqual([`last`]) + + // Ordinary numeric predicates use relational order rather than the + // custom index's descending walk. Both single and paired bounds must + // search past the custom index to use the ordinary one. + expect( + collection + .currentStateAsChanges({ + where: greaterThanTen, + optimizedOnly: true, + }) + ?.map(({ key }) => key) + .sort(), + ).toEqual([`last`, `middle`]) + expect( + collection + .currentStateAsChanges({ + where: betweenTenAndThirty, + optimizedOnly: true, + }) + ?.map(({ key }) => key), + ).toEqual([`middle`]) + expect(lookup).not.toHaveBeenCalledWith(`gt`, 10) + }, [() => collection.cleanup()]) + }) + + it(`creates a compatible eager index after an existing custom index`, async () => { + const collection = collectionWithRows( + `custom-index-eager-${name}`, + IndexType, + ) + await withOracleCleanup(async () => { + await collection.stateWhenReady() + collection.createIndex((row) => row.rank, { + indexType: IndexType, + options: { compareFn: (a: number, b: number) => b - a }, + }) + expect(collection.indexes.size).toBe(1) + + // The custom index cannot satisfy the ordinary order request. Eager + // indexing must add a compatible one before optimizedOnly reads it. + expect( + collection + .currentStateAsChanges({ ...ascendingFirst, optimizedOnly: true }) + ?.map(({ key }) => key), + ).toEqual([expectedFirstKey]) + expect(collection.indexes.size).toBe(2) + collection.currentStateAsChanges({ + ...ascendingFirst, + optimizedOnly: true, + }) + expect(collection.indexes.size).toBe(2) + }, [() => collection.cleanup()]) + }) + + it(`creates a compatible eager index for a range subscriber`, async () => { + const collection = collectionWithRows( + `custom-index-eager-range-${name}`, + IndexType, + ) + await withOracleCleanup(async () => { + await collection.stateWhenReady() + collection.createIndex((row) => row.rank, { + indexType: IndexType, + options: { compareFn: (a: number, b: number) => b - a }, + }) + const initialKeys: Array = [] + const subscription = collection.subscribeChanges( + (changes) => initialKeys.push(...changes.map(({ key }) => key)), + { whereExpression: greaterThanTen, includeInitialState: true }, + ) + try { + expect(initialKeys.sort()).toEqual([`last`, `middle`]) + expect(collection.indexes.size).toBe(2) + expect( + collection + .currentStateAsChanges({ + where: greaterThanTen, + optimizedOnly: true, + }) + ?.map(({ key }) => key) + .sort(), + ).toEqual([`last`, `middle`]) + } finally { + subscription.unsubscribe() + } + }, [() => collection.cleanup()]) + }) + it(`keeps a descending custom index out of an ascending public snapshot`, async () => { const scanned = collectionWithRows(`custom-index-order-scan-${name}`) const indexed = collectionWithRows(`custom-index-order-indexed-${name}`) diff --git a/packages/db/tests/index-update-oracle.property.test.ts b/packages/db/tests/index-update-oracle.property.test.ts index 5cb79a9500..2f004948f5 100644 --- a/packages/db/tests/index-update-oracle.property.test.ts +++ b/packages/db/tests/index-update-oracle.property.test.ts @@ -1619,6 +1619,86 @@ describe.each([ }, ) +// The caller's callback gives each signed zero its own order position, while +// JavaScript Map gives them one exact membership class. This small source Map +// is the authority for ownership; a fresh sort by the caller's stated numeric +// rank predicts ordered reads. No index bucket or traversal helper is reused. +function signedZeroPosition(value: number): number { + return Object.is(value, -0) ? -0.5 : value +} + +const signedZeroCompare = (left: unknown, right: unknown): number => + signedZeroPosition(left as number) - signedZeroPosition(right as number) + +function expectSignedZeroRefinement( + index: BaseIndex, + rows: ReadonlyMap, +): void { + const ordered = [...rows].sort( + ([leftKey, leftValue], [rightKey, rightValue]) => + signedZeroPosition(leftValue) - signedZeroPosition(rightValue) || + (leftKey < rightKey ? -1 : leftKey > rightKey ? 1 : 0), + ) + const keys = ordered.map(([key]) => key) + const zeroOwners = new Set( + [...rows].filter(([, value]) => value === 0).map(([key]) => key), + ) + + // Observe the documented direct IndexInterface after every returned action. + // A set-only comparison would miss a skipped or duplicated ordered row. + expect(index.keyCount).toBe(rows.size) + expect(index.equalityLookup(-0)).toEqual(zeroOwners) + expect(index.equalityLookup(0)).toEqual(zeroOwners) + expect(index.takeFromStart(rows.size + 1)).toEqual(keys) + expect(index.takeReversedFromEnd(rows.size + 1)).toEqual([...keys].reverse()) + for (const cursor of [-0, 0]) { + const position = signedZeroPosition(cursor) + expect(index.take(rows.size + 1, cursor)).toEqual( + ordered + .filter(([, value]) => signedZeroPosition(value) > position) + .map(([key]) => key), + ) + expect(index.takeReversed(rows.size + 1, cursor)).toEqual( + ordered + .filter(([, value]) => signedZeroPosition(value) < position) + .reverse() + .map(([key]) => key), + ) + } + + // Distinct bounds with `===` equality are legal when a callback separates + // their positions. The reverse bound is empty; exclusive edges select one + // sign only. These probes reach both leaf and split BTree range traversal. + for (const options of [ + { from: -0, to: -0 }, + { from: 0, to: 0 }, + { from: -0, to: 0 }, + { from: -0, to: 0, fromInclusive: false }, + { from: -0, to: 0, toInclusive: false }, + { from: -0, to: 0, fromInclusive: false, toInclusive: false }, + { from: 0, to: -0 }, + ]) { + const lower = signedZeroPosition(options.from) + const upper = signedZeroPosition(options.to) + const expected = new Set( + [...rows] + .filter(([, value]) => { + const position = signedZeroPosition(value) + return ( + (options.fromInclusive === false + ? position > lower + : position >= lower) && + (options.toInclusive === false + ? position < upper + : position <= upper) + ) + }) + .map(([key]) => key), + ) + expect(index.rangeQuery(options)).toEqual(expected) + } +} + describe.each(indexTypes)(`%s comparator groups`, (_indexName, IndexType) => { test.each([`add`, `build`] as const)( `follows a custom callback that separates signed zero after %s`, @@ -1658,6 +1738,74 @@ describe.each(indexTypes)(`%s comparator groups`, (_indexName, IndexType) => { }, ) + test(`tracks signed-zero order through updates, removal, and reuse`, () => { + const index = new IndexType(1, new PropRef([`value`]), undefined, { + compareFn: signedZeroCompare, + }) + const rows = new Map() + const put = (key: string, value: number): void => { + if (rows.has(key)) { + index.update(key, { value: rows.get(key) }, { value }) + } else { + index.add(key, { value }) + } + rows.set(key, value) + expectSignedZeroRefinement(index, rows) + } + const remove = (key: string): void => { + index.remove(key, { value: rows.get(key) }) + rows.delete(key) + expectSignedZeroRefinement(index, rows) + } + + // A sign-changing update keeps exact membership but must move the row's + // comparator position. Peers expose both stale and missing order buckets. + put(`negative-peer`, -0) + put(`moving`, -0) + put(`positive-peer`, 0) + put(`later`, 1) + put(`moving`, 0) + put(`moving`, -0) + remove(`moving`) + put(`moving`, 0) + remove(`negative-peer`) + put(`new-negative`, -0) + }) + + test(`preserves mixed-sign ranges across a BTree-sized split`, () => { + const index = new IndexType(1, new PropRef([`value`]), undefined, { + compareFn: signedZeroCompare, + }) + const rows = new Map() + // More than the BTree's default 32 keys places the two zero positions + // across an internal traversal. BasicIndex receives the same history. + const values = [ + ...Array.from({ length: 15 }, (_, offset) => offset - 15), + -0, + 0, + ...Array.from({ length: 16 }, (_, offset) => offset + 1), + ] + for (const [ordinal, value] of values.entries()) { + const key = `row-${String(ordinal).padStart(2, `0`)}` + index.add(key, { value }) + rows.set(key, value) + } + expectSignedZeroRefinement(index, rows) + + // Continue after the split so a stale old-position row cannot hide behind + // correct initial build order or leave an empty bucket after retirement. + const negative = `row-15` + index.update(negative, { value: -0 }, { value: 0 }) + rows.set(negative, 0) + expectSignedZeroRefinement(index, rows) + index.remove(negative, { value: 0 }) + rows.delete(negative) + expectSignedZeroRefinement(index, rows) + index.add(negative, { value: -0 }) + rows.set(negative, -0) + expectSignedZeroRefinement(index, rows) + }) + test(`rejects stale retired identity outputs without rejecting live reuse`, () => { const symbol = Symbol(`same group`) const retired = { key: `old`, groupId: 0, value: [symbol] } diff --git a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts index 9479a91d26..042d88c27b 100644 --- a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts +++ b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts @@ -21,8 +21,11 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * - Two Uint8Arrays (Buffers included) are equal when their bytes are equal. * - Temporal kinds with an intrinsic `.equals()` use that relation. This * includes equivalent time-zone aliases for ZonedDateTime, but excludes a - * distinct fixed-offset zone. Duration has no `.equals()`; its ten fields - * must match, so PT1H and PT60M are distinct. + * distinct fixed-offset zone. PlainMonthDay and PlainYearMonth also retain + * their reference ISO year and day, even when default display omits them. + * Duration has no `.equals()`; its ten fields must match, so PT1H and PT60M + * are distinct. Changelog #1519 explicitly preserves that distinction for + * equivalent Duration forms under `eq`. * - Every other pair compares with `===`, so `-0` equals `0`, `1n` does not * equal `1`, and objects compare by reference. * @@ -35,8 +38,9 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * internal normalization prefix, booleans, `0`, `-0`, `1`, `NaN`, `1n`, a * valid Date and its timestamp, an invalid Date, equal and unequal byte * arrays, a Buffer, Temporal values including ZonedDateTime zone aliases, - * a forged tag, a borrowed prototype, shadowed display members, and Duration - * forms, one shared object and a + * a forged tag, a borrowed prototype, shadowed display and field members, + * reference ISO year/day differences hidden by default display, and Duration carry + * forms with equal display but distinct fields, one shared object and a * structurally equal copy, `null` and `undefined`. Lists have zero to six * items and may repeat. The list is a constant `Value`, the shape that join * demand builds, or a row field, so changing lists are also checked. @@ -52,7 +56,9 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * Limits: this owner checks the boolean result of `in` for ordinary values * from one Temporal polyfill copy and the listed display shadows. It does not * claim mixed Temporal implementations, replaced genuine prototypes, or - * arbitrary custom prototypes that imitate Temporal intrinsics. WHERE + * arbitrary custom prototypes that imitate Temporal intrinsics. The byte + * classifier also excludes genuine Uint8Arrays with an own display-tag + * shadow. WHERE * publication, indexes, and subset loading have other owners. */ @@ -76,9 +82,38 @@ const shadowedTextPlainDate = Temporal.PlainDate.from(`2024-01-01`) Object.defineProperty(shadowedTextPlainDate, `toString`, { value: () => `unrelated text`, }) +const leapMonthDay = new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972) +const sameLeapMonthDay = new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972) +const otherReferenceMonthDay = new Temporal.PlainMonthDay( + 2, + 29, + `iso8601`, + 2004, +) +const firstYearMonth = new Temporal.PlainYearMonth(2024, 2, `iso8601`, 1) +const sameFirstYearMonth = new Temporal.PlainYearMonth(2024, 2, `iso8601`, 1) +const otherReferenceYearMonth = new Temporal.PlainYearMonth( + 2024, + 2, + `iso8601`, + 20, +) const hourDuration = Temporal.Duration.from(`PT1H`) const sameHourDuration = Temporal.Duration.from(`PT1H`) const sixtyMinuteDuration = Temporal.Duration.from(`PT60M`) +const secondDuration = Temporal.Duration.from({ seconds: 1 }) +const sameSecondDuration = Temporal.Duration.from({ seconds: 1 }) +const shadowedSecondDuration = Temporal.Duration.from({ seconds: 1 }) +Object.defineProperty(shadowedSecondDuration, `seconds`, { value: 2 }) +const thousandMillisecondDuration = Temporal.Duration.from({ + milliseconds: 1000, +}) +const sameThousandMillisecondDuration = Temporal.Duration.from({ + milliseconds: 1000, +}) +const millionMicrosecondDuration = Temporal.Duration.from({ + microseconds: 1_000_000, +}) const shadowedHourDuration = Temporal.Duration.from(`PT1H`) Object.defineProperty(shadowedHourDuration, `toString`, { value: () => `PT2H`, @@ -131,9 +166,21 @@ const pool: ReadonlyArray = [ borrowedPlainDate, shadowedTagPlainDate, shadowedTextPlainDate, + leapMonthDay, + sameLeapMonthDay, + otherReferenceMonthDay, + firstYearMonth, + sameFirstYearMonth, + otherReferenceYearMonth, hourDuration, sameHourDuration, sixtyMinuteDuration, + secondDuration, + sameSecondDuration, + shadowedSecondDuration, + thousandMillisecondDuration, + sameThousandMillisecondDuration, + millionMicrosecondDuration, shadowedHourDuration, forgedHourDuration, utcZoned, @@ -291,6 +338,45 @@ function evaluateWithProduction(testCase: InCase): unknown { return evaluate({ row: { value: testCase.value, list: testCase.list } }) } +// The same normalized equality also feeds compiled `eq`. Check its public +// result alongside both `in` paths so a repaired list key cannot mask a +// remaining direct-predicate collision. +function evaluateEqWithProduction(left: unknown, right: unknown): unknown { + const evaluate = compileExpression( + new Func(`eq`, [ + new PropRef([`row`, `value`]), + new PropRef([`row`, `other`]), + ]), + ) + return evaluate({ row: { value: left, other: right } }) +} + +function checkEqualityPaths( + left: unknown, + right: unknown, + expected: boolean, +): void { + expect(referenceEqual(left, right)).toBe(expected) + for (const [direction, value, other] of [ + [`forward`, left, right], + [`reverse`, right, left], + ] as const) { + expect + .soft(evaluateEqWithProduction(value, other), `eq ${direction}`) + .toBe(expected) + for (const listFromRow of [false, true]) { + const testCase = { value, list: [other], listFromRow } + expect(referenceIn(value, testCase.list)).toBe(expected) + expect + .soft( + evaluateWithProduction(testCase), + `in ${listFromRow ? `row-supplied` : `constant`} ${direction}`, + ) + .toBe(expected) + } + } +} + function check(evaluate: Evaluate, testCase: InCase): void { const expected = referenceIn(testCase.value, testCase.list) const actual = evaluate(testCase) @@ -452,6 +538,70 @@ describe(`in evaluator`, () => { }, ) + it(`keeps reference ISO fields in PlainMonthDay and PlainYearMonth equality`, () => { + // Native equals, not default display, is the independent semantic base. + // Both pairs have equal display text but different reference ISO fields. + expect(leapMonthDay.toString()).toBe(otherReferenceMonthDay.toString()) + expect(leapMonthDay.equals(sameLeapMonthDay)).toBe(true) + expect(leapMonthDay.equals(otherReferenceMonthDay)).toBe(false) + expect(firstYearMonth.toString()).toBe(otherReferenceYearMonth.toString()) + expect(firstYearMonth.equals(sameFirstYearMonth)).toBe(true) + expect(firstYearMonth.equals(otherReferenceYearMonth)).toBe(false) + for (const value of [ + leapMonthDay, + sameLeapMonthDay, + otherReferenceMonthDay, + firstYearMonth, + sameFirstYearMonth, + otherReferenceYearMonth, + ]) { + expect(pool).toContain(value) + } + + checkEqualityPaths(leapMonthDay, sameLeapMonthDay, true) + checkEqualityPaths(leapMonthDay, otherReferenceMonthDay, false) + checkEqualityPaths(firstYearMonth, sameFirstYearMonth, true) + checkEqualityPaths(firstYearMonth, otherReferenceYearMonth, false) + }) + + it(`keeps Duration fields distinct across second and subsecond carries`, () => { + // Changelog #1519 promises that equivalent Duration representations stay + // unequal under `eq`. The captured intrinsic field getters in the model + // preserve that difference while their default display text collides. + expect(secondDuration.toString()).toBe( + thousandMillisecondDuration.toString(), + ) + expect(thousandMillisecondDuration.toString()).toBe( + millionMicrosecondDuration.toString(), + ) + for (const value of [ + secondDuration, + sameSecondDuration, + shadowedSecondDuration, + thousandMillisecondDuration, + sameThousandMillisecondDuration, + millionMicrosecondDuration, + ]) { + expect(pool).toContain(value) + } + + checkEqualityPaths(secondDuration, sameSecondDuration, true) + // An own getter shadow cannot change the intrinsic Duration fields. + expect(shadowedSecondDuration.seconds).toBe(2) + checkEqualityPaths(secondDuration, shadowedSecondDuration, true) + checkEqualityPaths( + thousandMillisecondDuration, + sameThousandMillisecondDuration, + true, + ) + checkEqualityPaths(secondDuration, thousandMillisecondDuration, false) + checkEqualityPaths( + thousandMillisecondDuration, + millionMicrosecondDuration, + false, + ) + }) + it(`rejects display-string equality for equivalent zone aliases`, () => { const lexical: Evaluate = ({ value, list }) => Array.isArray(list) && diff --git a/packages/db/tests/temporal-equality-alias-oracle.test.ts b/packages/db/tests/temporal-equality-alias-oracle.test.ts index 10609a2ab2..6881708a2b 100644 --- a/packages/db/tests/temporal-equality-alias-oracle.test.ts +++ b/packages/db/tests/temporal-equality-alias-oracle.test.ts @@ -12,15 +12,27 @@ import { withOracleCleanup } from './utils.js' * order. At one instant, UTC and Etc/UTC are equivalent zone aliases, while * the fixed-offset +00:00 zone remains a different identity. * - * The ZonedDateTime lane covers ordinary values from one polyfill copy and one - * instant. A neighboring PlainDate lane checks that native brand and equality - * survive own display shadows while forged tags retain ordinary object - * identity. Fixture-assigned classes, not production normalization, predict - * exact key sets. Direct BasicIndex/BTreeIndex add, build, and removal - * histories compare those sets after each returned action. Public Collection + * The ZonedDateTime lane covers values from one polyfill copy and one instant, + * including genuine instances with own display shadows. A neighboring + * PlainDate lane checks that native brand and equality survive those shadows + * while forged tags retain ordinary object identity. Fixture-assigned classes, + * independent of production normalization, predict exact key sets. Direct + * BasicIndex/BTreeIndex add, build, and removal histories compare those sets + * after each returned action. Public Collection * reads compare an unindexed scan and an optimized-only indexed snapshot after * readiness. The mixed forged/genuine PlainDate Collection has no index: * default range order across an ordinary object and Temporal is unspecified. + * A second lane uses one polyfill copy to check `PlainMonthDay` and + * `PlainYearMonth` reference ISO fields, and Duration's ten fields, under + * the same exact query equality. Changelog #1519 explicitly keeps equivalent + * Duration representations unequal for `eq`. Its fixture assigns equality + * classes from native constructors and intrinsic relations, never from + * production key encoding. Compiled `eq` and `in` for these classes are + * checked by the in-evaluator oracle. PlainMonthDay has no natural order, so + * this file checks its public Collection equality through an unindexed scan. + * Indexed PlainMonthDay equality remains open. YearMonth and Duration store + * several classes with identical default display and check indexed add, + * build, removal, re-add, and public reads. * This oracle does not claim all IANA aliases, arbitrary spoofed prototypes, * mixed Temporal implementations, joins, live-query publication, or provider * behavior. @@ -30,6 +42,7 @@ type Sample = { id: string text: string identity: `utc` | `offset` | `later` | `calendar` + ownDisplay?: `tag` | `text` } const samples: ReadonlyArray = [ @@ -58,13 +71,32 @@ const samples: ReadonlyArray = [ text: `2024-01-15T09:00:00+00:00[UTC][u-ca=gregory]`, identity: `calendar`, }, + { + id: `utc-shadowed-tag`, + text: `2024-01-15T09:00:00+00:00[UTC]`, + identity: `utc`, + ownDisplay: `tag`, + }, + { + id: `utc-shadowed-text`, + text: `2024-01-15T09:00:00+00:00[UTC]`, + identity: `utc`, + ownDisplay: `text`, + }, ] function rows(): Array { - return samples.map(({ id, text }) => ({ - id, - when: Temporal.ZonedDateTime.from(text), - })) + return samples.map(({ id, text, ownDisplay }) => { + const when = Temporal.ZonedDateTime.from(text) + if (ownDisplay === `tag`) { + Object.defineProperty(when, Symbol.toStringTag, { value: `ordinary` }) + } else if (ownDisplay === `text`) { + Object.defineProperty(when, `toString`, { + value: () => `unrelated text`, + }) + } + return { id, when } + }) } function expectedKeys(identity: Sample[`identity`]): Set { @@ -87,16 +119,149 @@ const indexTypes = [ [`BTreeIndex`, BTreeIndex], ] as const -it(`distinguishes equivalent aliases from a fixed-offset zone`, () => { - const [utc, alias, fixed, later, gregory] = rows() +type IntrinsicKind = `PlainMonthDay` | `PlainYearMonth` | `Duration` +type IntrinsicRow = { id: string; when: unknown; identity: string } +type IntrinsicProbe = { when: unknown; identity: string } +type IntrinsicFixture = { + rows: Array + probes: Array + checkNativeRelation: () => void +} + +const indexedIntrinsicKinds: ReadonlyArray< + Exclude +> = [`PlainYearMonth`, `Duration`] + +// The fixture identity is a model-only equality class assigned from the +// constructed Temporal fields. It combines no production state. Fresh probes +// distinguish equal copies from different fields with the same display text. +function intrinsicFixture(kind: IntrinsicKind): IntrinsicFixture { + if (kind === `PlainMonthDay`) { + const stored = new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972) + const storedCopy = new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972) + const equal = new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972) + const other = new Temporal.PlainMonthDay(2, 29, `iso8601`, 2004) + return { + rows: [ + { id: `reference-1972`, when: stored, identity: `1972` }, + { id: `reference-1972-copy`, when: storedCopy, identity: `1972` }, + { id: `reference-2004`, when: other, identity: `2004` }, + ], + probes: [ + { when: equal, identity: `1972` }, + { when: other, identity: `2004` }, + ], + checkNativeRelation: () => { + expect(stored.toString()).toBe(other.toString()) + expect(stored.equals(storedCopy)).toBe(true) + expect(stored.equals(equal)).toBe(true) + expect(stored.equals(other)).toBe(false) + }, + } + } + + if (kind === `PlainYearMonth`) { + const stored = new Temporal.PlainYearMonth(2024, 2, `iso8601`, 1) + const equal = new Temporal.PlainYearMonth(2024, 2, `iso8601`, 1) + const other = new Temporal.PlainYearMonth(2024, 2, `iso8601`, 20) + return { + rows: [ + { id: `reference-day-1`, when: stored, identity: `day-1` }, + { id: `reference-day-1-copy`, when: equal, identity: `day-1` }, + { id: `reference-day-20`, when: other, identity: `day-20` }, + ], + probes: [ + { + when: new Temporal.PlainYearMonth(2024, 2, `iso8601`, 1), + identity: `day-1`, + }, + { + when: new Temporal.PlainYearMonth(2024, 2, `iso8601`, 20), + identity: `day-20`, + }, + ], + checkNativeRelation: () => { + expect(stored.toString()).toBe(other.toString()) + expect(stored.equals(equal)).toBe(true) + expect(stored.equals(other)).toBe(false) + }, + } + } + + const seconds = Temporal.Duration.from({ seconds: 1 }) + const secondsCopy = Temporal.Duration.from({ seconds: 1 }) + const shadowedSeconds = Temporal.Duration.from({ seconds: 1 }) + Object.defineProperty(shadowedSeconds, `seconds`, { value: 2 }) + const milliseconds = Temporal.Duration.from({ milliseconds: 1000 }) + const microseconds = Temporal.Duration.from({ microseconds: 1_000_000 }) + const intrinsicSeconds = Object.getOwnPropertyDescriptor( + Temporal.Duration.prototype, + `seconds`, + )!.get! + return { + rows: [ + { id: `seconds`, when: seconds, identity: `seconds` }, + { id: `seconds-copy`, when: secondsCopy, identity: `seconds` }, + { id: `seconds-shadow`, when: shadowedSeconds, identity: `seconds` }, + { id: `milliseconds`, when: milliseconds, identity: `milliseconds` }, + { id: `microseconds`, when: microseconds, identity: `microseconds` }, + ], + probes: [ + { when: Temporal.Duration.from({ seconds: 1 }), identity: `seconds` }, + { + when: Temporal.Duration.from({ milliseconds: 1000 }), + identity: `milliseconds`, + }, + { + when: Temporal.Duration.from({ microseconds: 1_000_000 }), + identity: `microseconds`, + }, + ], + checkNativeRelation: () => { + expect(seconds.toString()).toBe(milliseconds.toString()) + expect(milliseconds.toString()).toBe(microseconds.toString()) + expect(intrinsicSeconds.call(shadowedSeconds)).toBe(1) + expect(shadowedSeconds.seconds).toBe(2) + expect(seconds.seconds).toBe(1) + expect(milliseconds.milliseconds).toBe(1000) + expect(microseconds.microseconds).toBe(1_000_000) + }, + } +} + +function intrinsicExpectedKeys( + source: ReadonlyArray, + identity: string, +): Set { + return new Set( + source.filter((row) => row.identity === identity).map((row) => row.id), + ) +} + +it(`distinguishes equivalent aliases and own display shadows from a fixed-offset zone`, () => { + const [utc, alias, fixed, later, gregory, shadowedTag, shadowedText] = rows() // This checks the fixture against the external Temporal relation. Expected // index and query sets still come from the assigned identity classes. expect(utc!.when.equals(alias!.when)).toBe(true) expect(utc!.when.equals(fixed!.when)).toBe(false) expect(utc!.when.equals(later!.when)).toBe(false) expect(utc!.when.equals(gregory!.when)).toBe(false) + expect( + Temporal.ZonedDateTime.prototype.equals.call( + shadowedTag!.when, + alias!.when, + ), + ).toBe(true) + expect( + Temporal.ZonedDateTime.prototype.equals.call( + shadowedText!.when, + alias!.when, + ), + ).toBe(true) expect(utc!.when.toString()).not.toBe(alias!.when.toString()) - expect(expectedKeys(`utc`)).toEqual(new Set([`utc`, `utc-alias`])) + expect(expectedKeys(`utc`)).toEqual( + new Set([`utc`, `utc-alias`, `utc-shadowed-tag`, `utc-shadowed-text`]), + ) expect(expectedKeys(`offset`)).toEqual(new Set([`fixed-offset`])) }) @@ -133,7 +298,9 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { ) index.remove(source[0]!.id, source[0]!) - expect(index.equalityLookup(aliasProbe)).toEqual(new Set([`utc-alias`])) + expect(index.equalityLookup(aliasProbe)).toEqual( + new Set([`utc-alias`, `utc-shadowed-tag`, `utc-shadowed-text`]), + ) expect(index.equalityLookup(offsetProbe)).toEqual(expectedKeys(`offset`)) }, ) @@ -186,6 +353,175 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { ) }) +it(`keeps PlainMonthDay reference years distinct in an unindexed public equality scan`, async () => { + const fixture = intrinsicFixture(`PlainMonthDay`) + fixture.checkNativeRelation() + const collection = createCollection({ + id: `temporal-monthday-unindexed-equality`, + getKey: (row) => row.id, + autoIndex: `off`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of fixture.rows) write({ type: `insert`, value: row }) + commit() + markReady() + }, + }, + }) + + // The source contains two native-equal copies and a distinct reference + // year with the same default display. The public scan has no index order + // obligation; each query must return exactly its modeled equality class. + await withOracleCleanup(async () => { + await collection.stateWhenReady() + expect(collection.indexes.size).toBe(0) + for (const probe of fixture.probes) { + const predicate = new Func(`eq`, [ + new PropRef([`when`]), + new Value(probe.when), + ]) + const keys = new Set( + collection + .currentStateAsChanges({ where: predicate, optimizedOnly: false }) + ?.map(({ value }) => value.id), + ) + expect(collection.indexes.size).toBe(0) + expect(keys).toEqual(intrinsicExpectedKeys(fixture.rows, probe.identity)) + } + }, [() => collection.cleanup()]) +}) + +describe.each(indexTypes)( + `%s Temporal intrinsic field equality`, + (indexName, IndexType) => { + it.each(indexedIntrinsicKinds)( + `keeps %s exact classes after each add, removal, and re-add`, + (kind) => { + const fixture = intrinsicFixture(kind) + fixture.checkNativeRelation() + const index = new IndexType(1, new PropRef([`when`])) + const added: Array = [] + + // Each returned add is a checkpoint. The model recomputes membership + // from fixture classes, independent of the index's equality key. + for (const row of fixture.rows) { + index.add(row.id, row) + added.push(row) + for (const probe of fixture.probes) { + expect(index.equalityLookup(probe.when)).toEqual( + intrinsicExpectedKeys(added, probe.identity), + ) + } + } + + for (const probe of fixture.probes) { + const expected = intrinsicExpectedKeys(added, probe.identity) + expect(index.lookup(`eq`, probe.when)).toEqual(expected) + expect(index.inArrayLookup([probe.when])).toEqual(expected) + } + + const removed = added.shift()! + index.remove(removed.id, removed) + for (const probe of fixture.probes) { + expect(index.equalityLookup(probe.when)).toEqual( + intrinsicExpectedKeys(added, probe.identity), + ) + } + + const freshEqual = fixture.probes.find( + (probe) => probe.identity === removed.identity, + )! + const readded = { ...removed, when: freshEqual.when } + index.add(readded.id, readded) + added.push(readded) + for (const probe of fixture.probes) { + expect(index.equalityLookup(probe.when)).toEqual( + intrinsicExpectedKeys(added, probe.identity), + ) + } + }, + ) + + it.each(indexedIntrinsicKinds)( + `keeps %s exact classes after build`, + (kind) => { + const fixture = intrinsicFixture(kind) + fixture.checkNativeRelation() + const index = new IndexType(1, new PropRef([`when`])) + index.build( + fixture.rows.map((row): [string, IntrinsicRow] => [row.id, row]), + ) + for (const probe of fixture.probes) { + expect(index.equalityLookup(probe.when)).toEqual( + intrinsicExpectedKeys(fixture.rows, probe.identity), + ) + } + }, + ) + + it.each( + indexedIntrinsicKinds.flatMap((kind) => [ + { kind, optimizedOnly: false }, + { kind, optimizedOnly: true }, + ]), + )( + `returns $kind exact classes in a public snapshot with optimizedOnly=$optimizedOnly`, + async ({ kind, optimizedOnly }) => { + const fixture = intrinsicFixture(kind) + fixture.checkNativeRelation() + const collection = createCollection({ + id: `temporal-intrinsic-${kind}-${indexName}-${optimizedOnly}`, + getKey: (row) => row.id, + autoIndex: `off`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of fixture.rows) + write({ type: `insert`, value: row }) + commit() + markReady() + }, + }, + }) + + // A scan supplies the public control. With optimizedOnly, a spy proves + // that the same `eq` predicate reaches the explicit index lookup. + await withOracleCleanup(async () => { + await collection.stateWhenReady() + expect(collection.indexes.size).toBe(0) + const lookup = optimizedOnly + ? vi.spyOn( + collection.createIndex((row) => row.when, { + indexType: IndexType, + }), + `lookup`, + ) + : undefined + for (const probe of fixture.probes) { + lookup?.mockClear() + const predicate = new Func(`eq`, [ + new PropRef([`when`]), + new Value(probe.when), + ]) + const keys = new Set( + collection + .currentStateAsChanges({ where: predicate, optimizedOnly }) + ?.map(({ value }) => value.id), + ) + if (lookup) expect(lookup).toHaveBeenCalled() + expect(keys).toEqual( + intrinsicExpectedKeys(fixture.rows, probe.identity), + ) + } + }, [() => collection.cleanup()]) + }, + ) + }, +) + it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () => { const fake = { [Symbol.toStringTag]: `Temporal.ZonedDateTime`, diff --git a/packages/db/tests/utils-oracle.property.test.ts b/packages/db/tests/utils-oracle.property.test.ts index 53d9210f2e..1713888f54 100644 --- a/packages/db/tests/utils-oracle.property.test.ts +++ b/packages/db/tests/utils-oracle.property.test.ts @@ -700,14 +700,6 @@ describe(`deepEquals property-based tests`, () => { // change-event equality keeps the Date value class distinct. The finite // generator makes both encodings legal; the 2023 case preserves the old // unit test's exact operands. - utilsProperty([arbitraryFiniteDate])( - `Date differs from its ISO text and epoch number in both directions`, - (date) => { - assertDistinctTypes(date, date.toISOString(), deepEquals) - assertDistinctTypes(date, date.getTime(), deepEquals) - }, - ) - it(`keeps a 2023 Date distinct from its old primitive probes`, () => { const date = new Date(`2023-01-01T00:00:00Z`) expectCopyAndChanged( @@ -722,11 +714,6 @@ describe(`deepEquals property-based tests`, () => { // RegExp source text does not acquire the regex's value class, even when // it spells the same pattern. The source and flags still govern equality // between two genuine RegExps in the laws below. - utilsProperty([arbitraryRegExp])( - `RegExp differs from its source text in both directions`, - (regex) => assertDistinctTypes(regex, regex.source, deepEquals), - ) - it(`keeps /abc/g distinct from its old source-text probe`, () => { expectCopyAndChanged(/abc/g, /abc/g, /abc/i) expectCopyAndChanged(/abc/g, /abc/g, /def/g) @@ -1187,6 +1174,19 @@ describe(`deepEquals property-based tests`, () => { // Keep newly added generated laws after the original registration order: // UTILS_ORACLE_PROPERTY uses ordinal selectors for saved seed/path replays. describe(`additional non-ring value boundaries`, () => { + utilsProperty([arbitraryFiniteDate])( + `Date differs from its ISO text and epoch number in both directions`, + (date) => { + assertDistinctTypes(date, date.toISOString(), deepEquals) + assertDistinctTypes(date, date.getTime(), deepEquals) + }, + ) + + utilsProperty([arbitraryRegExp])( + `RegExp differs from its source text in both directions`, + (regex) => assertDistinctTypes(regex, regex.source, deepEquals), + ) + utilsProperty([arbitraryRegExp])( `RegExp source and flags determine equality`, (regex) => { @@ -1194,11 +1194,15 @@ describe(`deepEquals property-based tests`, () => { const changedFlags = regex.flags.includes(`i`) ? regex.flags.replace(`i`, ``) : `${regex.flags}i` + // The source neighbor keeps flags fixed; a source-blind comparator + // must fail even when generated flags happen to match. + const changedSource = new RegExp(`${regex.source}(?:x)`, regex.flags) expectCopyAndChanged( regex, copy, new RegExp(regex.source, changedFlags), ) + expectEqualityPair(regex, changedSource, false) }, ) From ae0b67fd286b12a235867663cfacb951b7a91ead Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 19:41:50 -0600 Subject: [PATCH 15/46] test(db): close oracle review gaps and guarded replay --- .../fix-custom-index-order-selection.md | 2 +- docs/contributing/oracle-coverage.md | 30 +++- .../custom-index-order-selection.md | 15 ++ .../oracle-reviews/temporal-equality-alias.md | 43 +++++ packages/db/src/indexes/auto-index.ts | 28 +-- ...ustom-index-order-selection-oracle.test.ts | 131 ++++++++++---- packages/db/tests/oracle-replay-manifest.ts | 7 + packages/db/tests/oracle-replay.test.ts | 5 + .../in-evaluator-oracle.property.test.ts | 161 +++++++++++++++--- .../temporal-equality-alias-oracle.test.ts | 73 ++++++-- 10 files changed, 413 insertions(+), 82 deletions(-) diff --git a/.changeset/fix-custom-index-order-selection.md b/.changeset/fix-custom-index-order-selection.md index 9dd7278679..74c14e600c 100644 --- a/.changeset/fix-custom-index-order-selection.md +++ b/.changeset/fix-custom-index-order-selection.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Use a custom-comparator index for exact lookup without selecting its order for an ordinary Collection `orderBy` read. When a compatible ordinary index also exists, select it for ordered reads; eager indexing now creates one if needed. +Use a custom-comparator index for exact lookup without selecting its order for an ordinary Collection `orderBy` read. When a compatible ordinary index also exists, select it for ordered reads; eager indexing now creates one if needed, including for equivalent value-left range predicates. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 80df33c2cc..56e9c45d97 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -502,12 +502,18 @@ row-supplied alias lists. A plain object with a forged Temporal tag and a genuine value with a shadowed zone accessor must stay outside the UTC class. The neighboring PlainDate lane checks forged tags, borrowed prototypes, and genuine own-tag or display shadows at direct lookup and unindexed public `eq`. -The ZonedDateTime fixture also contains genuine UTC rows with own display-tag -and `toString` shadows; direct indexes and both public `eq` snapshot paths -retain their alias equality. The `in` evaluator owner distinguishes +The ZonedDateTime fixture also contains genuine UTC rows with own display-tag, +`toString`, `epochNanoseconds`, and `calendarId` shadows; direct indexes and +both public `eq` snapshot paths retain their alias equality. The `in` +evaluator owner distinguishes PlainMonthDay reference years, PlainYearMonth reference days, and Duration field vectors that share default display text. Fixed controls reach compiled `eq`, constant-list `in`, and row-supplied `in` in both operand directions. +Equal and unequal values within Instant, PlainDate, PlainDateTime, and +PlainTime reject a kind-only Temporal equality key at those same paths. A +genuine Uint8Array with a shadowed display tag and a spoof-tagged Uint16Array +check the intrinsic byte-class boundary. The `in` property has a guarded +direct replay selector that runs only its random/replay campaign. The alias owner also checks a public unindexed `eq` scan with two native-equal PlainMonthDay rows and a distinct reference year. Direct BasicIndex/BTreeIndex add, build, removal, re-add, and public indexed `eq` cover the orderable @@ -515,8 +521,11 @@ PlainYearMonth and Duration cases. Indexed PlainMonthDay equality with multiple classes remains open: the ordered indexes cannot compare those classes. A future exact-only path must preserve equality while making public `orderBy` throw for PlainMonthDay, as changelog #1519 requires. A one-class index bypass -failed that order law and was not retained. -Its byte model excludes a genuine Uint8Array with an own display-tag shadow. +failed that order law and was not retained. The `in` model treats a non-null +value tested against a null-only list as FALSE, while the live-query guide's +broad SQL/PostgreSQL comparison statement suggests UNKNOWN. The compiled `in` +owner tracks this contract-authority question; a product decision is needed +before changing its expected result or claiming SQL-style null-list behavior. Other zone aliases, custom spoofed prototypes, mixed Temporal implementations, mixed ordinary-object/Temporal ordered indexes, joins, grouping, query identity, live-query publication, and provider paths remain unproved. Direct @@ -532,10 +541,13 @@ With a custom index created before a compatible ordinary index, ordered reads select the latter in both directions. Eager indexing adds one compatible index when the custom index alone is present and does not add a duplicate on repeated reads. Single and compound numeric range predicates also select -the compatible index; an eager range subscriber creates one when needed. -The fixed ready-state fixture covers BasicIndex and -BTreeIndex, one numeric field, and limit one. Mutation histories, arbitrary -callbacks, and provider or live-query publication remain outside this owner. +the compatible index; an eager range subscriber creates one when needed for +field-left and value-left single or paired relational predicates. A +custom-only index cannot serve an optimized range read. The fixed ready-state +fixture covers BasicIndex and BTreeIndex, one numeric field, and limit one. +Mutation histories, arbitrary callbacks, and provider publication remain +outside this owner. Live-query compiler admission remains a separate +receiving-path risk. The [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts) also has a [cross-realm Date equality matrix](oracle-reviews/cross-realm-date-where-publication.md). diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md index 9aed82266d..b2609ce8df 100644 --- a/docs/contributing/oracle-reviews/custom-index-order-selection.md +++ b/docs/contributing/oracle-reviews/custom-index-order-selection.md @@ -74,3 +74,18 @@ six-file related campaign passed 145 tests with no type errors. Lint and diff checks passed. Fresh review of the final committed head remains pending. + +At `5df06de2e`, another reviewer found that eager admission recognized +`gt(rank, 10)` but not its equivalent value-left form `lt(10, rank)` after a +custom index. Initial subscription rows were correct, but the value-left +history failed its optimized-only snapshot because no compatible index was +created. The author added single and paired value-left RED witnesses for +BasicIndex and BTreeIndex before extending eager expression extraction to +identify the field on either side. The repaired tests compare initial rows, +optimized rows, and index count against independent numeric row filters. A +custom-only optimized range refusal is an adjacent control. The focused +oracle passed eight tests and the neighboring index campaign passed 547. +This establishes neither arbitrary value domains nor mutation histories. +The live-query compiler has a separate index-admission path; simple public +probes were correct, but the comparable coexisting-index schedule remains +unproved there. diff --git a/docs/contributing/oracle-reviews/temporal-equality-alias.md b/docs/contributing/oracle-reviews/temporal-equality-alias.md index 534d0e3960..65282cbf96 100644 --- a/docs/contributing/oracle-reviews/temporal-equality-alias.md +++ b/docs/contributing/oracle-reviews/temporal-equality-alias.md @@ -127,3 +127,46 @@ later distinct reference-year row then threw during a valid sync commit. The bypass was removed. Direct and public indexed PlainMonthDay equality across multiple classes remains open. An exact-only index path needs a separate oracle for add, build, remove, update, and the orderBy rejection. + +## Review of committed head `5df06de2e` + +Fresh review found no mismatch in the exercised alias cases. It identified +one remaining sensitivity gap in the own-property controls: genuine +ZonedDateTimes with own `epochNanoseconds` or `calendarId` shadows were not +present. The fixture now gives both values an independently assigned UTC +identity class. Native `ZonedDateTime.prototype.equals` confirms each equals +the alias probe, and direct add/build/removal plus unindexed and optimized-only +public `eq` must return them. The focused alias owner passed 28 tests. + +The separate compiled `in` review found a false-green kind-only possibility: +its pool contained only one intrinsic value of several Temporal kinds. +Equal and unequal Instant, PlainDate, PlainDateTime, and PlainTime values now +reach compiled `eq`, constant `in`, and row-supplied `in` in both directions. +A kind-only membership mutant reaches the shared comparison and is rejected +as an expected-false/actual-true mismatch. The review also found that the +byte reference classifier trusted an instance's display tag. Captured +typed-array intrinsic classification now admits a genuine Uint8Array with an +own tag shadow and rejects spoof-tagged Uint16Arrays; fixed controls reach all +three compiled paths. The focused `in` owner passed 33 tests. + +`evaluators.in` previously had no guarded replay owner. Its replay manifest +now names this oracle and selects only the random/replay campaign. The +property uses an async fast-check runner so the guarded witness reporter +records the completed run; a targeted replay test passed with one selected +property and the fixed campaign skipped. +For failure calibration, a temporary kind-only driver substitution failed the +random campaign after 11 tests at seed `42`, path `10:2:3`. Guarded replay of +that coordinate ran only this property, reproduced the same unequal PlainDate +membership mismatch in one test, recorded `failed: true`, and exited nonzero. +The substitution was removed; the production driver again evaluates compiled +`in`. + +One contract-authority question stays open. For a non-null value and a +null-only list, the current model and evaluator return FALSE, so a compiled +`not(inArray(...))` returns TRUE. PostgreSQL would return UNKNOWN. The +[live-query guide](../../guides/live-queries.md#comparison-semantics) broadly +says comparisons follow SQL/PostgreSQL null conventions, while its `inArray` +description states array membership without defining null-list behavior. The +compiled `in` owner records the discrepancy; changing the law requires a +product decision and separate public WHERE/negation witnesses. No SQL-style +null-list claim is made for this transfer. diff --git a/packages/db/src/indexes/auto-index.ts b/packages/db/src/indexes/auto-index.ts index 2d7217ed43..b96b067b2f 100644 --- a/packages/db/src/indexes/auto-index.ts +++ b/packages/db/src/indexes/auto-index.ts @@ -3,7 +3,7 @@ import { hasVirtualPropPath } from '../virtual-props' import { codedWarning, devBuild } from '../error-message.js' import { checkCollectionSizeForIndex, isDevModeEnabled } from './index-registry' import type { CompareOptions } from '../query/builder/types' -import type { BasicExpression } from '../query/ir' +import type { BasicExpression, PropRef } from '../query/ir' import type { CollectionImpl } from '../collection/index.js' function shouldAutoIndex(collection: CollectionImpl) { @@ -131,11 +131,9 @@ function extractIndexableExpressions( return } - const func = expr as any - // Handle 'and' expressions by recursively processing all arguments - if (func.name === `and`) { - for (const arg of func.args) { + if (expr.name === `and`) { + for (const arg of expr.args) { extractFromExpression(arg) } return @@ -143,16 +141,26 @@ function extractIndexableExpressions( // Check if this is a supported operation const supportedOperations = [`eq`, `gt`, `gte`, `lt`, `lte`, `in`] - if (!supportedOperations.includes(func.name)) { + if (!supportedOperations.includes(expr.name)) { return } - // Check if the first argument is a property reference - if (func.args.length < 1 || func.args[0].type !== `ref`) { + // Preserve field-first admission for existing expressions. A scalar on + // the left of a comparison can also put the indexable field on the right. + // IN keeps its field-first shape. + let fieldRef: PropRef | undefined + if (expr.args[0]?.type === `ref`) { + fieldRef = expr.args[0] + } else if (expr.name !== `in` && expr.args.length === 2) { + const [left, right] = expr.args + if (left?.type === `val` && right?.type === `ref`) { + fieldRef = right + } + } + if (!fieldRef) { return } - const fieldRef = func.args[0] const fieldPath = fieldRef.path // Skip if the path is empty @@ -166,7 +174,7 @@ function extractIndexableExpressions( results.push({ fieldName, fieldPath, - requireOrderedTraversal: func.name !== `eq` && func.name !== `in`, + requireOrderedTraversal: expr.name !== `eq` && expr.name !== `in`, }) } diff --git a/packages/db/tests/custom-index-order-selection-oracle.test.ts b/packages/db/tests/custom-index-order-selection-oracle.test.ts index b63c6ae7fd..16f39789e6 100644 --- a/packages/db/tests/custom-index-order-selection-oracle.test.ts +++ b/packages/db/tests/custom-index-order-selection-oracle.test.ts @@ -19,7 +19,8 @@ import { withOracleCleanup } from './utils.js' * It also checks a custom index followed by a compatible ordinary index, and * eager index creation when the custom index is present first. Numeric range * predicates must likewise find a later compatible index; a range subscriber - * must trigger eager creation when only a custom index exists. It covers + * must trigger eager creation when only a custom index exists, whichever side + * of a single or paired relational predicate holds the field. It covers * BasicIndex and BTreeIndex at the ready checkpoint, with one order term and * limit one. It does not claim arbitrary custom comparator laws, mutation * histories, provider ordering, or live-query publication. @@ -36,6 +37,19 @@ const rows: ReadonlyArray = [ // nor the index-selection code; the first public key must be `first`. const expectedFirstKey = [...rows].sort((a, b) => a.rank - b.rank)[0]!.id const greaterThanTen = new Func(`gt`, [new PropRef([`rank`]), new Value(10)]) +const tenLessThanRank = new Func(`lt`, [new Value(10), new PropRef([`rank`])]) +const expectedGreaterThanTen = rows + .filter(({ rank }) => rank > 10) + .map(({ id }) => id) + .sort() +const reversedBetweenTenAndThirty = new Func(`and`, [ + tenLessThanRank, + new Func(`gt`, [new Value(30), new PropRef([`rank`])]), +]) +const expectedBetweenTenAndThirty = rows + .filter(({ rank }) => 10 < rank && rank < 30) + .map(({ id }) => id) + .sort() const betweenTenAndThirty = new Func(`and`, [ greaterThanTen, new Func(`lt`, [new PropRef([`rank`]), new Value(30)]), @@ -178,38 +192,79 @@ describe.each(indexTypes)( }, [() => collection.cleanup()]) }) - it(`creates a compatible eager index for a range subscriber`, async () => { - const collection = collectionWithRows( - `custom-index-eager-range-${name}`, - IndexType, + it(`creates a compatible eager index for reversed single and paired ranges`, async () => { + const forms = [ + { + name: `field-left`, + where: greaterThanTen, + expectedKeys: expectedGreaterThanTen, + }, + { + name: `field-right`, + where: tenLessThanRank, + expectedKeys: expectedGreaterThanTen, + }, + { + name: `paired-field-right`, + where: reversedBetweenTenAndThirty, + expectedKeys: expectedBetweenTenAndThirty, + }, + ] as const + const collections = forms.map(({ name: formName }) => + collectionWithRows( + `custom-index-eager-range-${name}-${formName}`, + IndexType, + ), ) - await withOracleCleanup(async () => { - await collection.stateWhenReady() - collection.createIndex((row) => row.rank, { - indexType: IndexType, - options: { compareFn: (a: number, b: number) => b - a }, - }) - const initialKeys: Array = [] - const subscription = collection.subscribeChanges( - (changes) => initialKeys.push(...changes.map(({ key }) => key)), - { whereExpression: greaterThanTen, includeInitialState: true }, - ) - try { - expect(initialKeys.sort()).toEqual([`last`, `middle`]) - expect(collection.indexes.size).toBe(2) - expect( - collection - .currentStateAsChanges({ - where: greaterThanTen, - optimizedOnly: true, + await withOracleCleanup( + async () => { + await Promise.all( + collections.map((collection) => collection.stateWhenReady()), + ) + const observations = [] + for (const [index, form] of forms.entries()) { + const collection = collections[index]! + collection.createIndex((row) => row.rank, { + indexType: IndexType, + options: { compareFn: (a: number, b: number) => b - a }, + }) + const initialKeys: Array = [] + const subscription = collection.subscribeChanges( + (changes) => initialKeys.push(...changes.map(({ key }) => key)), + { whereExpression: form.where, includeInitialState: true }, + ) + try { + observations.push({ + form: form.name, + initialKeys: initialKeys.sort(), + optimizedKeys: collection + .currentStateAsChanges({ + where: form.where, + optimizedOnly: true, + }) + ?.map(({ key }) => key) + .sort(), + indexCount: collection.indexes.size, }) - ?.map(({ key }) => key) - .sort(), - ).toEqual([`last`, `middle`]) - } finally { - subscription.unsubscribe() - } - }, [() => collection.cleanup()]) + } finally { + subscription.unsubscribe() + } + } + // The first two expressions denote the same numeric relation; the + // paired form narrows it. Initial publication and optimized-only + // snapshots must follow those independent row filters after the + // subscription's eager index construction. + expect(observations).toEqual( + forms.map(({ name: formName, expectedKeys }) => ({ + form: formName, + initialKeys: expectedKeys, + optimizedKeys: expectedKeys, + indexCount: 2, + })), + ) + }, + collections.map((collection) => () => collection.cleanup()), + ) }) it(`keeps a descending custom index out of an ascending public snapshot`, async () => { @@ -271,6 +326,20 @@ describe.each(indexTypes)( optimizedOnly: true, }), ).toBeUndefined() + // A range read also cannot use the custom comparator when it is the + // only index, even if no later ordinary index is available. + expect( + indexed.currentStateAsChanges({ + where: greaterThanTen, + optimizedOnly: true, + }), + ).toBeUndefined() + expect( + indexed.currentStateAsChanges({ + where: betweenTenAndThirty, + optimizedOnly: true, + }), + ).toBeUndefined() // Reversing the advertised direction would also reverse the unknown // custom comparator. The fallback and ordinary reversed index must diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index bea5072439..31bac27019 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -34,6 +34,11 @@ const ownerGroups: ReadonlyArray = [ `index-update`, `reference-model nullish-membership exact-identity custom-comparator`, ], + [ + `db/tests/query/compiler/in-evaluator-oracle.property.test.ts`, + `evaluators`, + `in`, + ], [ `db/tests/cursor-oracle.property.test.ts`, `cursor`, @@ -281,6 +286,8 @@ const indexReplayTestNames = new Map([ ]) function directReplayTestName(property: string): string | undefined { + if (property === `evaluators.in`) + return `agrees with the reference over a random or replayed campaign` if (property === `cold-join.compound`) return `recomputes generated compound histories, seed=undefined` if (property === `oracle-replay.calibration`) diff --git a/packages/db/tests/oracle-replay.test.ts b/packages/db/tests/oracle-replay.test.ts index 8a6c3c0ead..144dacacd1 100644 --- a/packages/db/tests/oracle-replay.test.ts +++ b/packages/db/tests/oracle-replay.test.ts @@ -95,6 +95,11 @@ describe(`guarded oracle replay`, () => { }, 40_000) it.each([ + [ + `evaluators.in`, + `tests/query/compiler/in-evaluator-oracle.property.test.ts`, + 1, + ], [ `index-update.reference-model`, `tests/index-update-oracle.property.test.ts`, diff --git a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts index 042d88c27b..2c424ad1b8 100644 --- a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts +++ b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts @@ -8,11 +8,14 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' /** * # Which values does `in` find in a list? * - * Contract: TanStack DB's `inArray(value, list)` returns UNKNOWN (`null`) for a - * null or undefined value. A list that is not an array is FALSE. + * Transfer law: The public `inArray` API describes array membership. The + * current evaluator returns UNKNOWN (`null`) for a null or undefined value + * and FALSE for a list that is not an array. * Otherwise the result is TRUE when some list item equals the value under the * evaluator's equality, and FALSE when none does. A null or undefined item - * never equals a value. The equality is the one `eq` documents in + * never equals a value under that membership interpretation. The public + * guide's broad SQL null statement leaves this last rule open for a separate + * product decision. The equality is the one `eq` documents in * `src/query/compiler/evaluators.ts`: * * - `NaN` and an invalid Date equal each other and nothing else. @@ -37,11 +40,14 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * of each rule: equal and unequal strings, a string that begins with the * internal normalization prefix, booleans, `0`, `-0`, `1`, `NaN`, `1n`, a * valid Date and its timestamp, an invalid Date, equal and unequal byte - * arrays, a Buffer, Temporal values including ZonedDateTime zone aliases, - * a forged tag, a borrowed prototype, shadowed display and field members, - * reference ISO year/day differences hidden by default display, and Duration carry - * forms with equal display but distinct fields, one shared object and a - * structurally equal copy, `null` and `undefined`. Lists have zero to six + * arrays, a Buffer, genuine bytes with an own display-tag shadow, a wide + * typed array with a spoofed byte tag, Temporal values including ZonedDateTime + * zone aliases, a forged tag, a borrowed prototype, shadowed display and + * field members, + * unequal values within each modeled Temporal kind, reference ISO year/day + * differences hidden by default display, and Duration carry forms with equal + * display but distinct fields, one shared object and a structurally equal + * copy, `null` and `undefined`. Lists have zero to six * items and may repeat. The list is a constant `Value`, the shape that join * demand builds, or a row field, so changing lists are also checked. * @@ -56,9 +62,7 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * Limits: this owner checks the boolean result of `in` for ordinary values * from one Temporal polyfill copy and the listed display shadows. It does not * claim mixed Temporal implementations, replaced genuine prototypes, or - * arbitrary custom prototypes that imitate Temporal intrinsics. The byte - * classifier also excludes genuine Uint8Arrays with an own display-tag - * shadow. WHERE + * arbitrary custom prototypes that imitate Temporal intrinsics. WHERE * publication, indexes, and subset loading have other owners. */ @@ -66,6 +70,28 @@ const PROPERTY = `evaluators.in` const PREFIX = `\u0000tanstack-db:binary:` const shared = { id: 1 } const plainDate = Temporal.PlainDate.from(`2024-01-01`) +const otherPlainDate = Temporal.PlainDate.from(`2024-01-02`) +const instant = Temporal.Instant.from(`2024-01-01T00:00:00Z`) +const sameInstant = Temporal.Instant.from(`2024-01-01T00:00:00Z`) +const otherInstant = Temporal.Instant.from(`2024-01-02T00:00:00Z`) +const plainDateTime = Temporal.PlainDateTime.from(`2024-01-01T10:00`) +const samePlainDateTime = Temporal.PlainDateTime.from(`2024-01-01T10:00`) +const otherPlainDateTime = Temporal.PlainDateTime.from(`2024-01-01T11:00`) +const plainTime = Temporal.PlainTime.from(`10:00`) +const samePlainTime = Temporal.PlainTime.from(`10:00`) +const otherPlainTime = Temporal.PlainTime.from(`11:00`) +const shadowedTagBytes = new Uint8Array([1, 2]) +Object.defineProperty(shadowedTagBytes, Symbol.toStringTag, { + value: `ordinary`, +}) +const spoofedWideBytes = new Uint16Array([1, 2]) +Object.defineProperty(spoofedWideBytes, Symbol.toStringTag, { + value: `Uint8Array`, +}) +const otherSpoofedWideBytes = new Uint16Array([1, 2]) +Object.defineProperty(otherSpoofedWideBytes, Symbol.toStringTag, { + value: `Uint8Array`, +}) const forgedPlainDate = { [Symbol.toStringTag]: `Temporal.PlainDate`, toString: () => `2024-01-01`, @@ -159,9 +185,21 @@ const pool: ReadonlyArray = [ new Uint8Array([1, 2]), new Uint8Array([1, 3]), Buffer.from([1, 2]), + shadowedTagBytes, + spoofedWideBytes, + otherSpoofedWideBytes, plainDate, Temporal.PlainDate.from(`2024-01-01`), - Temporal.PlainTime.from(`10:00`), + otherPlainDate, + instant, + sameInstant, + otherInstant, + plainDateTime, + samePlainDateTime, + otherPlainDateTime, + plainTime, + samePlainTime, + otherPlainTime, forgedPlainDate, borrowedPlainDate, shadowedTagPlainDate, @@ -194,12 +232,17 @@ const pool: ReadonlyArray = [ undefined, ] -// A Buffer from another realm, such as Node's under jsdom, is not an -// `instanceof Uint8Array` here, so check the view's tag instead. +// The intrinsic typed-array tag reads the internal class across realms. An +// instance's own display tag cannot hide genuine bytes or forge wide bytes. +const typedArrayTag = Object.getOwnPropertyDescriptor( + Object.getPrototypeOf(Uint8Array.prototype), + Symbol.toStringTag, +)!.get! + function isBytes(value: unknown): value is Uint8Array { return ( ArrayBuffer.isView(value) && - Object.prototype.toString.call(value) === `[object Uint8Array]` + Reflect.apply(typedArrayTag, value, []) === `Uint8Array` ) } @@ -387,6 +430,21 @@ function check(evaluate: Evaluate, testCase: InCase): void { } } +// A kind-only Temporal list key is a plausible wrong normalization: it finds +// an equal copy, but also finds a different value of the same kind. Feed that +// answer through the same comparison used by the generated campaigns. +function evaluateWithKindOnlyTemporalKey(testCase: InCase): boolean | null { + const kind = temporalBrand(testCase.value)?.kind + if ( + kind !== undefined && + Array.isArray(testCase.list) && + testCase.list.some((item) => temporalBrand(item)?.kind === kind) + ) { + return true + } + return referenceIn(testCase.value, testCase.list) +} + function describeCase(testCase: InCase): string { const show = (value: unknown): string => { if (value === forgedPlainDate) return `forged PlainDate tag` @@ -424,16 +482,16 @@ function describeCase(testCase: InCase): string { } describe(`in evaluator`, () => { - const property = fc.property(inCase, (testCase) => + const property = fc.asyncProperty(inCase, async (testCase) => check(evaluateWithProduction, testCase), ) - it(`agrees with the reference over a fixed campaign`, () => { - fc.assert(property, { numRuns: oracleRuns(2000), seed: 0x2029 }) + it(`agrees with the reference over a fixed campaign`, async () => { + await fc.assert(property, { numRuns: oracleRuns(2000), seed: 0x2029 }) }) - it(`agrees with the reference over a random or replayed campaign`, () => { - fc.assert(property, oraclePropertyOptions(2000, PROPERTY)) + it(`agrees with the reference over a random or replayed campaign`, async () => { + await fc.assert(property, oraclePropertyOptions(2000, PROPERTY)) }) it.each([ @@ -451,6 +509,69 @@ describe(`in evaluator`, () => { check(evaluateWithProduction, testCase) }) + it(`uses intrinsic byte class when a typed array shadows its display tag`, () => { + expect(Object.prototype.toString.call(shadowedTagBytes)).toBe( + `[object ordinary]`, + ) + expect(Object.prototype.toString.call(spoofedWideBytes)).toBe( + `[object Uint8Array]`, + ) + expect(isBytes(shadowedTagBytes)).toBe(true) + expect(isBytes(spoofedWideBytes)).toBe(false) + checkEqualityPaths(shadowedTagBytes, new Uint8Array([1, 2]), true) + checkEqualityPaths(spoofedWideBytes, otherSpoofedWideBytes, false) + }) + + it(`distinguishes unequal intrinsic values within each Temporal kind`, () => { + // Native equals supplies each kind's expected identity relation. The + // public compiler must reject a kind-only equality key in all three paths. + expect(plainDate.equals(otherPlainDate)).toBe(false) + expect(instant.equals(sameInstant)).toBe(true) + expect(instant.equals(otherInstant)).toBe(false) + expect(plainDateTime.equals(samePlainDateTime)).toBe(true) + expect(plainDateTime.equals(otherPlainDateTime)).toBe(false) + expect(plainTime.equals(samePlainTime)).toBe(true) + expect(plainTime.equals(otherPlainTime)).toBe(false) + for (const [left, equal, other] of [ + [plainDate, shadowedTagPlainDate, otherPlainDate], + [instant, sameInstant, otherInstant], + [plainDateTime, samePlainDateTime, otherPlainDateTime], + [plainTime, samePlainTime, otherPlainTime], + ] as const) { + expect(pool).toContain(left) + expect(pool).toContain(equal) + expect(pool).toContain(other) + checkEqualityPaths(left, equal, true) + checkEqualityPaths(left, other, false) + } + }) + + it(`rejects a kind-only Temporal list key at the membership comparison`, () => { + for (const [value, equal, unequal] of [ + [plainDate, shadowedTagPlainDate, otherPlainDate], + [instant, sameInstant, otherInstant], + [plainDateTime, samePlainDateTime, otherPlainDateTime], + [plainTime, samePlainTime, otherPlainTime], + ] as const) { + for (const listFromRow of [false, true]) { + expect( + evaluateWithKindOnlyTemporalKey({ + value, + list: [equal], + listFromRow, + }), + ).toBe(true) + expect(() => + check(evaluateWithKindOnlyTemporalKey, { + value, + list: [unequal], + listFromRow, + }), + ).toThrow(/in mismatch: expected false, got true/) + } + } + }) + it.each([false, true])( `returns false for a non-array %s list`, (listFromRow) => { diff --git a/packages/db/tests/temporal-equality-alias-oracle.test.ts b/packages/db/tests/temporal-equality-alias-oracle.test.ts index 6881708a2b..5712b32590 100644 --- a/packages/db/tests/temporal-equality-alias-oracle.test.ts +++ b/packages/db/tests/temporal-equality-alias-oracle.test.ts @@ -13,7 +13,7 @@ import { withOracleCleanup } from './utils.js' * the fixed-offset +00:00 zone remains a different identity. * * The ZonedDateTime lane covers values from one polyfill copy and one instant, - * including genuine instances with own display shadows. A neighboring + * including genuine instances with own display and intrinsic-getter shadows. A neighboring * PlainDate lane checks that native brand and equality survive those shadows * while forged tags retain ordinary object identity. Fixture-assigned classes, * independent of production normalization, predict exact key sets. Direct @@ -42,7 +42,7 @@ type Sample = { id: string text: string identity: `utc` | `offset` | `later` | `calendar` - ownDisplay?: `tag` | `text` + ownShadow?: `tag` | `text` | `epoch` | `calendar` } const samples: ReadonlyArray = [ @@ -75,25 +75,41 @@ const samples: ReadonlyArray = [ id: `utc-shadowed-tag`, text: `2024-01-15T09:00:00+00:00[UTC]`, identity: `utc`, - ownDisplay: `tag`, + ownShadow: `tag`, }, { id: `utc-shadowed-text`, text: `2024-01-15T09:00:00+00:00[UTC]`, identity: `utc`, - ownDisplay: `text`, + ownShadow: `text`, + }, + { + id: `utc-shadowed-epoch`, + text: `2024-01-15T09:00:00+00:00[UTC]`, + identity: `utc`, + ownShadow: `epoch`, + }, + { + id: `utc-shadowed-calendar`, + text: `2024-01-15T09:00:00+00:00[UTC]`, + identity: `utc`, + ownShadow: `calendar`, }, ] function rows(): Array { - return samples.map(({ id, text, ownDisplay }) => { + return samples.map(({ id, text, ownShadow }) => { const when = Temporal.ZonedDateTime.from(text) - if (ownDisplay === `tag`) { + if (ownShadow === `tag`) { Object.defineProperty(when, Symbol.toStringTag, { value: `ordinary` }) - } else if (ownDisplay === `text`) { + } else if (ownShadow === `text`) { Object.defineProperty(when, `toString`, { value: () => `unrelated text`, }) + } else if (ownShadow === `epoch`) { + Object.defineProperty(when, `epochNanoseconds`, { value: 0n }) + } else if (ownShadow === `calendar`) { + Object.defineProperty(when, `calendarId`, { value: `gregory` }) } return { id, when } }) @@ -238,8 +254,18 @@ function intrinsicExpectedKeys( ) } -it(`distinguishes equivalent aliases and own display shadows from a fixed-offset zone`, () => { - const [utc, alias, fixed, later, gregory, shadowedTag, shadowedText] = rows() +it(`distinguishes equivalent aliases and own property shadows from a fixed-offset zone`, () => { + const [ + utc, + alias, + fixed, + later, + gregory, + shadowedTag, + shadowedText, + shadowedEpoch, + shadowedCalendar, + ] = rows() // This checks the fixture against the external Temporal relation. Expected // index and query sets still come from the assigned identity classes. expect(utc!.when.equals(alias!.when)).toBe(true) @@ -258,9 +284,28 @@ it(`distinguishes equivalent aliases and own display shadows from a fixed-offset alias!.when, ), ).toBe(true) + expect( + Temporal.ZonedDateTime.prototype.equals.call( + shadowedEpoch!.when, + alias!.when, + ), + ).toBe(true) + expect( + Temporal.ZonedDateTime.prototype.equals.call( + shadowedCalendar!.when, + alias!.when, + ), + ).toBe(true) expect(utc!.when.toString()).not.toBe(alias!.when.toString()) expect(expectedKeys(`utc`)).toEqual( - new Set([`utc`, `utc-alias`, `utc-shadowed-tag`, `utc-shadowed-text`]), + new Set([ + `utc`, + `utc-alias`, + `utc-shadowed-tag`, + `utc-shadowed-text`, + `utc-shadowed-epoch`, + `utc-shadowed-calendar`, + ]), ) expect(expectedKeys(`offset`)).toEqual(new Set([`fixed-offset`])) }) @@ -299,7 +344,13 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { index.remove(source[0]!.id, source[0]!) expect(index.equalityLookup(aliasProbe)).toEqual( - new Set([`utc-alias`, `utc-shadowed-tag`, `utc-shadowed-text`]), + new Set([ + `utc-alias`, + `utc-shadowed-tag`, + `utc-shadowed-text`, + `utc-shadowed-epoch`, + `utc-shadowed-calendar`, + ]), ) expect(index.equalityLookup(offsetProbe)).toEqual(expectedKeys(`offset`)) }, From be5b7a3c45693c8b1871df56e6846b739d859bc3 Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 19:56:57 -0600 Subject: [PATCH 16/46] fix(db): cover cached ordered windows and Date subclasses --- docs/contributing/oracle-coverage.md | 41 ++-- .../oracle-reviews/cross-realm-date-order.md | 19 ++ .../cross-realm-date-where-publication.md | 17 ++ .../custom-comparator-signed-zero.md | 12 +- .../custom-index-order-selection.md | 46 +++++ .../oracle-reviews/perf-in-constant-set.md | 13 +- .../oracle-reviews/temporal-equality-alias.md | 21 ++ .../oracle-reviews/temporal-index-order.md | 7 +- .../utils-unit-retirement-round-7.md | 14 +- packages/db/package.json | 2 +- packages/db/src/query/compiler/order-by.ts | 14 +- packages/db/src/utils/comparison.ts | 20 +- .../tests/comparison-oracle.property.test.ts | 64 +++++- .../in-evaluator-oracle.property.test.ts | 6 +- ...stom-index-window-selection-oracle.test.ts | 185 ++++++++++++++++++ ...dicate-publication-oracle.property.test.ts | 138 +++++++++++-- .../temporal-equality-alias-oracle.test.ts | 64 ++++-- .../db/tests/utils-oracle.property.test.ts | 26 +-- 18 files changed, 627 insertions(+), 82 deletions(-) create mode 100644 packages/db/tests/query/custom-index-window-selection-oracle.test.ts diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 56e9c45d97..4ae656dee2 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -467,11 +467,13 @@ records the five retired same-realm examples. A separate [cross-realm Date witness](oracle-reviews/cross-realm-date-order.md) compares native `node:vm` Dates by supplied timestamps and invalidity, including equal times, numeric NaN with an invalid Date, and Dates with an own display tag. -It also checks an unindexed public ordered scan with a tagged foreign invalid +A fixed local/foreign subclass witness checks native timestamps inherited +through a Date prototype and rejects a borrowed-prototype impostor. The owner +also checks an unindexed public ordered scan with a tagged foreign invalid Date. Exact `normalizeValue` results are direct utility observations. Indexed -order, public Date range predicates, cursor continuation, proxies, subclasses, -replaced Date prototypes or `getTime` methods, and other Date-consuming paths -remain outside that witness. +order, public Date range predicates, cursor continuation, proxies, deeper +subclass chains, replaced Date prototypes or `getTime` methods, and other +Date-consuming paths remain outside that witness. The comparison oracle also owns a bounded [Temporal order transfer](oracle-reviews/comparison-unit-retirement-round-6.md). Supplied ISO calendar fields and epoch-nanosecond offsets predict direct `ascComparator` signs and exact zeros for ordinary `PlainDate` and `ZonedDateTime` values from one polyfill copy. Fixed cases preserve the two retired unit blocks; generated cases vary calendar fields, offset zones, equal instants, and adjacent nanoseconds. Fresh-module histories reject a first-input constructor cache poisoned by a genuine instance or forged tag, including the direct `compareValues` path. Both direct helpers also receive a right-hand PlainDate with a constructor shadow. A genuine ZonedDateTime with an own shadowed display tag retains its instant position through both direct helpers. This lane does not establish public query or index behavior. Custom Temporal subclasses, spoofed or replaced instance prototypes, mixed native/polyfill implementations, other Temporal kinds, and public ordering/filter publication need separate witnesses. Spoofed Temporal prototypes can still cause asymmetric comparisons; the comparison owner needs a dedicated brand and mixed-value witness before that class can be called closed. @@ -500,6 +502,9 @@ fixture-assigned classes. Unindexed and optimized-only indexed Collection snapshots check public `eq`; the `in` evaluator oracle checks constant and row-supplied alias lists. A plain object with a forged Temporal tag and a genuine value with a shadowed zone accessor must stay outside the UTC class. +Forged and borrowed ordinary objects also retain reference self-identity in +direct lookup and unindexed public `eq`; distinct structural copies do not +match them. The neighboring PlainDate lane checks forged tags, borrowed prototypes, and genuine own-tag or display shadows at direct lookup and unindexed public `eq`. The ZonedDateTime fixture also contains genuine UTC rows with own display-tag, @@ -546,21 +551,32 @@ field-left and value-left single or paired relational predicates. A custom-only index cannot serve an optimized range read. The fixed ready-state fixture covers BasicIndex and BTreeIndex, one numeric field, and limit one. Mutation histories, arbitrary callbacks, and provider publication remain -outside this owner. Live-query compiler admission remains a separate -receiving-path risk. +outside this direct snapshot owner. The separate +[custom-index window selection oracle](../../packages/db/tests/query/custom-index-window-selection-oracle.test.ts) +checks the live-query compiler's receiving path. A controlled on-demand source +starts with ranks 20 and 40 cached, then serves exact ordered requests from +ranks 10, 20, 30, and 40. Moving a public window from offset 0/limit 1 to +offset 1/limit 2 must publish ranks 20 and 30 once, even though the provider +does not rewrite cached rank 20. Custom-first, ordinary-first, no-index, and +eager-custom-only layouts cross BasicIndex and BTreeIndex. Provider reply and +write observations prove that the cached row reaches the local ordered path. +The owner does not establish ties, nullish or nonnumeric orders, mutations, +replays, multiple order terms, arbitrary callbacks, real providers, or other +window schedules. The [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts) also has a [cross-realm Date equality matrix](oracle-reviews/cross-realm-date-where-publication.md). For local and `node:vm` Dates with timestamps 1 and 2 or invalid timestamps, -including foreign Dates with an own display tag, plus number 1 and NaN, it +including tagged foreign Dates and fixed local/foreign subclasses with an own +display tag, plus number 1, NaN, and a borrowed-prototype impostor, it compares public live-query keys, filtered subscriber keys, and direct snapshot keys after preload and a three-row sync transaction. Auto-indexing is off in both modes; one mode adds BasicIndex explicitly and also checks its equality lookup and an optimized-only public snapshot. This proves a public indexed snapshot result but does not prove that the live-query -planner chose the index. Other realm hosts, proxies, subclasses, Date prototype -swaps or method replacement, public range predicates, and arbitrary histories -need separate witnesses. +planner chose the index. Other realm hosts, proxies, deeper subclass chains, +Date prototype swaps or method replacement, public range predicates, and +generated subclass histories need separate witnesses. The direct [index refinement oracle](../../packages/db/tests/index-update-oracle.property.test.ts) also owns exact nullish membership and direct `IN` set selection for BasicIndex @@ -598,8 +614,9 @@ in the index refinement oracle separates `-0` and `+0` into two ordered positions while exact lookup retains one SameValueZero class. Fixed add/build and removal histories check BasicIndex and BTreeIndex takes, cursors, ranges, and exact membership. Sign-changing updates and removal/re-addition must move -or release the ordered position; mixed-sign inclusive/exclusive and reversed -ranges are checked at a BTree leaf and across a 33-row split. It does not +or release the ordered position; mixed-sign inclusive/exclusive ranges with +reversed bounds are checked at a BTree leaf and across a 33-row split. The +signed-zero checker does not directly call `rangeQueryReversed`. It does not establish arbitrary custom callbacks or public planner selection; the custom index order selection owner above checks that separate boundary. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-order.md b/docs/contributing/oracle-reviews/cross-realm-date-order.md index 036ca76cad..56411eed17 100644 --- a/docs/contributing/oracle-reviews/cross-realm-date-order.md +++ b/docs/contributing/oracle-reviews/cross-realm-date-order.md @@ -141,3 +141,22 @@ the rejected exception-heavy path for Temporal values. The public scan cleanup preserves a primary mismatch if cleanup also fails. This verdict does not extend to replaced Date prototypes or `getTime` methods, proxies, subclasses, indexed order, range queries, or cursor continuation. + +## Tagged Date subclass follow-up + +At `ae0b67fd2`, a new hostile review found a native `node:vm` Date subclass +with an own display tag that the immediate-prototype `getTime` descriptor gate +rejected. The native intrinsic returned timestamp 123, yet normalization left +the object unchanged and a public `eq` query missed its row. This is a separate +product bug under the guide's native-Date timestamp law, not a lost assertion +from the five retired same-realm blocks. The comparison owner now fixes local +and foreign subclasses with an inherited Date method, valid and invalid +timestamps, direct normalization and order, and an ordinary object borrowing +the foreign subclass prototype. On the original code the direct witness failed +at `normalizeValue(foreignSubclass)`, returning the object instead of 123. The +repair walks prototype descriptors without reading user getters, then calls +the native `Date.prototype.getTime` as the final brand check. Both receiving +oracles passed 127 focused tests after the repair. The separate WHERE owner +records the public RED and GREEN observations. Proxies, deeper subclass chains, +replaced methods or prototypes, indexed order, ranges, and cursor continuation +remain unproved. Fresh exact-commit review of this follow-up remains pending. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md index 027b306636..2d5c5a88a0 100644 --- a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md +++ b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md @@ -118,3 +118,20 @@ The archived `9a145fba1` run failed at the intended unindexed public equality comparison by omitting `foreign-tagged-one`; its unmodified controls passed. The verdict does not cover other hosts, public ranges, indexed order, arbitrary histories, or Date objects with replaced prototypes or methods. + +## Tagged Date subclass follow-up + +The hostile review at `ae0b67fd2` found that a native foreign Date subclass +with an own display tag missed `eq(Date(1))` in a public live query. The matrix +now assigns fixed local and foreign subclass rows and literals to timestamp +classes independently of production classification. It also includes an +invalid foreign subclass and an ordinary object borrowing a foreign subclass +prototype as a negative control. Initial and post-commit observations compare +the complete public live-query rows, filtered subscriber keys, direct snapshot +keys, explicit BasicIndex lookup, and an optimized-only snapshot. The original +implementation failed first at the unindexed live-query key set by omitting +`foreign-subclass-one`. The descriptor-chain repair passed both receiving +oracles, 127 tests, with no type errors. Live-query planner index use, generated +subclass histories, deeper subclass chains, other hosts, public ranges, and +prototype/method replacement remain outside this witness. Fresh exact-commit +review is pending. diff --git a/docs/contributing/oracle-reviews/custom-comparator-signed-zero.md b/docs/contributing/oracle-reviews/custom-comparator-signed-zero.md index 81c8d59e64..752666a915 100644 --- a/docs/contributing/oracle-reviews/custom-comparator-signed-zero.md +++ b/docs/contributing/oracle-reviews/custom-comparator-signed-zero.md @@ -32,7 +32,7 @@ fixture reached the split-tree case. The extended direct oracle uses an independent source Map and a supplied numeric rank to predict membership and order after sign-changing updates, removal, and re-addition. It checks exact SameValueZero lookup, ordered takes, -cursors, and inclusive, exclusive, reversed, and point ranges on both index +cursors, and inclusive, exclusive, reversed-bound, and point ranges on both index types; the split case crosses BTree children. Four focused cases failed on the old production at the intended ordered or range comparison. Changing only the BTree point-range shortcuts left the two update cases red. Ordered @@ -56,7 +56,13 @@ with no type errors after repair. The five lint warnings were preexisting. | ORC-010 | Direct indexes own no external resources or cleanup action; an assertion is the failure checkpoint. | | ORC-011 | The source Map and fresh rank sort differ from the incremental indexes. No plausible shared semantic fault within this bounded callback remains identified; arbitrary callback laws remain open. | | ORC-012 | This record names the counterexamples, RED/GREEN sequence, scoped claim, and pending final review. | -| ORC-013 | `-0` and `+0` share exact lookup but have distinct callback positions; inclusive versus exclusive and reversed mixed-sign bounds reject a SameValueZero range rule. | +| ORC-013 | `-0` and `+0` share exact lookup but have distinct callback positions; inclusive versus exclusive and reversed mixed-sign bounds reject a SameValueZero range rule. The signed-zero checker does not directly call `rangeQueryReversed`. | | ORC-014 | No provider or host handoff is claimed. | -Fresh review of the final committed head remains pending. +At `ae0b67fd2`, a fresh hostile reviewer found no blocker in the declared +direct-index scope. The 420-test focused oracle and targeted generated replays +passed. Independent bounded histories matched BasicIndex and BTreeIndex for +signed-zero exact lookup, order, cursors, and forward range queries. The +reviewer also noted that part of the broader numeric property inspects private +layout, while separate public method checks protect the transferred behavior. +Arbitrary callbacks and public planner selection remain outside this verdict. diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md index b2609ce8df..78003724f1 100644 --- a/docs/contributing/oracle-reviews/custom-index-order-selection.md +++ b/docs/contributing/oracle-reviews/custom-index-order-selection.md @@ -89,3 +89,49 @@ This establishes neither arbitrary value domains nor mutation histories. The live-query compiler has a separate index-admission path; simple public probes were correct, but the comparable coexisting-index schedule remains unproved there. + +## Live-query receiving path after `ae0b67fd2` + +A fresh hostile review found a public counterexample to the compiler's ordered +index admission. An on-demand source had rank-20 and rank-40 rows cached. +It held four authoritative rows at ranks 10, 20, 30, and 40. With a reverse +custom index declared before an ordinary index, an ascending live query +preloaded its first row correctly. Moving to offset one and limit two then +published ranks 30 and 40, instead of 20 and 30. The provider fulfilled its +exact ordered request for ranks 10, 20, and 30. It did not rewrite cached +rank 20, which the wrong local index traversal had failed to send into the +graph. Ordinary-first, ordinary-only, and no-index controls returned the +correct rows. The reviewer reproduced both index types and found 16 public +mismatches in a separate 64-scenario cache/window sweep. + +The new [window oracle](../../../packages/db/tests/query/custom-index-window-selection-oracle.test.ts) +assigns numeric ranks in its source model and sorts and slices those four +facts independently of index selection. Its controlled adapter evaluates +each request's predicate and cursor against the source facts, records the +selected provider rows, and writes only uncached rows. It checks the live +query's rows after preload and after `setWindow()` settles, plus the sole +publication batch. A reply `[1,2,3]` with writes `[1,3]` proves that row 2 +was already cached and still belonged to the requested result. The fixed +history crosses custom-first, ordinary-first, no-index, and eager-custom-only +layouts with BasicIndex and BTreeIndex. On the original compiler, four cases +failed at the moved public window: `[3,4]` instead of `[2,3]`. Controls +passed. All eight cases passed after the repair; 677 related tests and the +explicit test TypeScript check passed. + +The compiler now requires an ordinary ordered-traversal index when it creates +or selects the first order term's source index. Eager creation lets BaseIndex +derive the comparator from `compareOptions`; passing an explicit `compareFn` +would mark that index custom and make it ineligible. This changes only source +index admission. Direct custom callbacks still order each row according to +the maintainer's decision. The production edit removes two net lines. + +The oracle states the ordered window law and its source limits beside its +model, request grammar, driver, and two public cuts (ORC-001–005). The +original compiler fails at the promised post-window comparison, while +ordinary-first and no-index controls reach the same public boundary +(ORC-006, -011, -013). This is a fixed finite matrix, so generated replay +does not apply (ORC-004, -007). The provider is controlled and awaits applied +receipts; the claim does not extend to remote hosts (ORC-014). Ties, nullish +or nonnumeric orders, mutations, replay, multiple order terms, arbitrary +callbacks, and other window schedules remain open. Fresh hostile review of +the final committed oracle is pending. diff --git a/docs/contributing/oracle-reviews/perf-in-constant-set.md b/docs/contributing/oracle-reviews/perf-in-constant-set.md index f8d3fcc6a8..fe09c2883d 100644 --- a/docs/contributing/oracle-reviews/perf-in-constant-set.md +++ b/docs/contributing/oracle-reviews/perf-in-constant-set.md @@ -5,8 +5,17 @@ production change is `18ea072b5`. This record adds the audit only. ## Contract and evidence -`inArray(value, list)` is SQL `IN` under three-valued logic. A null or -undefined value is UNKNOWN (`null`). A list that is not an array is FALSE. The +The original review called `inArray(value, list)` SQL `IN` under three-valued +logic. That wording is superseded for null-containing lists by the +[compiled `in` contract review](temporal-equality-alias.md#review-of-committed-head-5df06de2e) +and its coverage-map entry. The current evaluator treats a non-null value +against a null-only list as FALSE; SQL would yield UNKNOWN. This is an open +product-law decision, not a claim that the current rule is SQL-equivalent. +The normalization details below describe the reviewed `perf-in-set` head; +the linked review records later Temporal equality changes. + +Under the reviewed evaluator rule, a null or undefined value is UNKNOWN +(`null`). A list that is not an array is FALSE. The result is TRUE when some non-null list item equals the value under the evaluator's equality, and FALSE otherwise. diff --git a/docs/contributing/oracle-reviews/temporal-equality-alias.md b/docs/contributing/oracle-reviews/temporal-equality-alias.md index 65282cbf96..7ee1ee4a4d 100644 --- a/docs/contributing/oracle-reviews/temporal-equality-alias.md +++ b/docs/contributing/oracle-reviews/temporal-equality-alias.md @@ -170,3 +170,24 @@ description states array membership without defining null-list behavior. The compiled `in` owner records the discrepancy; changing the law requires a product decision and separate public WHERE/negation witnesses. No SQL-style null-list claim is made for this transfer. + +## Review of committed head `ae0b67fd2` + +Fresh hostile review found no lost retired comparison assertion in the alias +owner. It found one sensitivity gap: forged and borrowed ordinary objects were +checked only for inequality with genuine Temporal values, so an index could +drop their own entries and still pass. Direct BasicIndex/BTreeIndex lookups now +require each impostor to find itself, while a distinct copy must not match. +Unindexed public `eq` makes the same self-identity and distinct-copy checks. +The focused alias owner passed 28 tests after the extension. This guards the +ordinary-object side of brand rejection without treating a spoofed display +tag as Temporal identity. Fresh exact-commit review of these new controls is +pending. Indexed PlainMonthDay equality across multiple classes remains open. + +The same committed-head review found no blocker in the compiled `in` owner's +stated grammar: 33 tests and guarded replay passed. It confirmed the +null-only-list authority question above and identified the older +`perf-in-constant-set.md` review's SQL-equivalence sentence as too broad. That +record now explicitly supersedes the sentence for null-containing lists. +Two lint-only edits to the compiled `in` owner follow this review and require +a fresh exact-commit check. diff --git a/docs/contributing/oracle-reviews/temporal-index-order.md b/docs/contributing/oracle-reviews/temporal-index-order.md index 40859fe519..5cf9e0a0db 100644 --- a/docs/contributing/oracle-reviews/temporal-index-order.md +++ b/docs/contributing/oracle-reviews/temporal-index-order.md @@ -110,4 +110,9 @@ tag or mixed Temporal implementations. ## Exact committed-head review -Pending fresh hostile review of the final committed repair. +At `ae0b67fd2`, a fresh hostile reviewer found no blocker in the stated +default and custom ordered-index domain. All 25 focused tests passed. Its +independent bounded histories exercised both index types under both order +rules, and a wrong-result probe confirmed that the optimized public snapshot +uses the index result. Provider cursors, arbitrary callbacks, and live-query +publication remain outside this verdict. diff --git a/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md b/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md index 573209be39..355871d0fa 100644 --- a/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md +++ b/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md @@ -68,10 +68,20 @@ tests with no type errors. For ORC-007 calibration, a temporary wrong model rule expected an arbitrary RegExp to equal its changed-source neighbor. The fixed campaign failed at the `deepEquals` return comparison after one test with seed `20260911` and shrink -path `0:0`. Direct replay through `UTILS_ORACLE_PROPERTY=36`, +path `0:0`. Direct replay through `UTILS_ORACLE_PROPERTY=34`, `UTILS_ORACLE_SEED=20260911`, and `UTILS_ORACLE_PATH=0:0` reproduced that same assertion failure. Restoring the model rule made the direct replay pass. The temporary rule is absent from the final oracle. This demonstrates replay fidelity for the changed-source property, not every property in the file. -Fresh review of the final committed head remains pending. +At `ae0b67fd2`, a fresh hostile reviewer confirmed all six retired assertions +and found no production mismatch in this transfer. It caught a replay defect: +the two new cross-type registrations preceded the existing RegExp, Float32Array, +and Duration registrations, shifting saved selectors 34–36. Moving the new +registrations after them restored 34 to RegExp source and flags, 35 to +Float32Array, and 36 to Duration; the new properties are 37 and 38. A temporary +wrong expected result for the RegExp changed-source neighbor failed on direct +replay of selector 34, seed `20260911`, path `0:0`, after one test at the +forward `deepEquals` return. Removing the temporary rule made that replay pass +with only the selected generated property active. Fresh review of the next +committed head remains pending. diff --git a/packages/db/package.json b/packages/db/package.json index df69b9cdc0..31a5f6eb61 100644 --- a/packages/db/package.json +++ b/packages/db/package.json @@ -23,7 +23,7 @@ "test": "vitest --run", "test:dist": "vitest run --config vitest.dist.config.ts", "test:facade-retention": "node --expose-gc --import tsx tests/facade-retention.probe.ts", - "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/temporal-index-order-oracle.test.ts tests/temporal-equality-alias-oracle.test.ts tests/custom-index-order-selection-oracle.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", + "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/temporal-index-order-oracle.test.ts tests/temporal-equality-alias-oracle.test.ts tests/custom-index-order-selection-oracle.test.ts tests/query/custom-index-window-selection-oracle.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", "bench:nested-includes": "vitest bench tests/query/includes-performance.bench.ts --run", "test:indexed-db:package": "pnpm --filter @tanstack/db... build && vitest run --config vitest.indexed-db.package.config.ts", "test:indexed-db:browser": "playwright test --config playwright.indexed-db.config.ts", diff --git a/packages/db/src/query/compiler/order-by.ts b/packages/db/src/query/compiler/order-by.ts index 9782dd7662..3f3b304f06 100644 --- a/packages/db/src/query/compiler/order-by.ts +++ b/packages/db/src/query/compiler/order-by.ts @@ -181,19 +181,16 @@ export function processOrderBy( const compareOpts = buildCompareOptions(firstClause, collection) if (fieldName) { - // Use a single-column comparator for the index, not the - // multi-column `compare` function. The multi-column comparator - // expects array values [col1, col2, ...] but the index stores - // individual field values. Passing `compare` here causes the - // BTree to treat all single values as equal (since number[0] - // === undefined for both sides of the comparison). - const firstColumnCompareFn = makeComparator(compareOpts) + // The index derives this term's comparator from compareOptions. + // Supplying a compareFn would mark it as a custom comparator whose + // order cannot be used for ordinary ordered traversal. ensureIndexForField( fieldName, followRefResult.path, followRefCollection, compareOpts, - firstColumnCompareFn, + undefined, + true, ) } @@ -201,6 +198,7 @@ export function processOrderBy( followRefCollection, followRefResult.path, compareOpts, + true, ) // Only use the index if it supports range queries diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index e48d5edeb4..adef608d35 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -27,15 +27,19 @@ function dateTimestamp(value: unknown): number | undefined { if (value instanceof Date) return Date.prototype.getTime.call(value) try { if (Symbol.toStringTag in value) { - // A normal Date prototype owns getTime. Inspect its descriptor without - // reading a user display-tag getter or trusting a claimed tag value. - const prototype = Object.getPrototypeOf(value) - if ( - typeof Object.getOwnPropertyDescriptor(prototype, `getTime`)?.value !== - `function` - ) { - return undefined + // A Date subclass can inherit getTime from its Date prototype. Walk + // descriptors without reading a user display-tag or getTime getter; + // the intrinsic below still checks for the native Date slot. + let prototype = Object.getPrototypeOf(value) + while (prototype !== null) { + const descriptor = Object.getOwnPropertyDescriptor(prototype, `getTime`) + if (descriptor) { + if (typeof descriptor.value !== `function`) return undefined + break + } + prototype = Object.getPrototypeOf(prototype) } + if (prototype === null) return undefined } else if (Object.prototype.toString.call(value) !== `[object Date]`) { return undefined } diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index 6e07038879..5d29b70394 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -39,7 +39,9 @@ import type { CompareOptions } from '../src/query/builder/types' * input tags and finite epoch-day values, rather than the production * unorderable classifier. It covers number-only and Date-only sorts with * optional null. A separate realm-tagged Date model below checks native Date - * timestamps and invalidity across realms, plus one public ordered scan. + * timestamps and invalidity across realms, plus one public ordered scan. A + * fixed Date-subclass witness covers an inherited native `getTime` beyond the + * generated plain-Date grammar. * Neither lane claims mixed-type order or every query/index path. Equality laws separately * cover the normalized value classes and changed-byte controls. Comparator * signs reject NaN and non-number results before reduction. Signed infinity @@ -399,7 +401,8 @@ function assertRealmDateOrder(sample: RealmDateSample): void { // foreign-valid and foreign-invalid paths; generated pairs add equal times, // both operand orders, both invalid values, and Dates with an own display tag. // A display tag cannot change native Date identity. Proxies, non-Date objects, -// and user-defined Date subclasses are outside this generated grammar. +// and user-defined Date subclasses are outside this generated grammar. The +// fixed witness below covers local and foreign subclasses with an own tag. const arbitraryDateTime = fc.oneof( fc.integer({ min: -1_000_000_000_000, max: 1_000_000_000_000 }), fc.constant(null), @@ -1065,6 +1068,63 @@ describe(`native Date order across JavaScript realms`, () => { ).toBe(1) }) + it(`uses the native timestamp of a tagged foreign Date subclass`, () => { + class LocalDate extends Date {} + const timestamp = 123 + const localSubclass = new LocalDate(timestamp) + const foreignSubclass = runInNewContext( + `class D extends Date {}; new D(timestamp)`, + { timestamp }, + ) as Date + const foreignInvalid = runInNewContext( + `class D extends Date {}; new D(Number.NaN)`, + ) as Date + for (const value of [localSubclass, foreignSubclass, foreignInvalid]) { + Object.defineProperty(value, Symbol.toStringTag, { value: `Other` }) + } + // The supplied time and native slot establish the Date premise. The + // foreign subclass's immediate prototype inherits native getTime. + expect(Date.prototype.getTime.call(foreignSubclass)).toBe(timestamp) + expect(foreignSubclass instanceof Date).toBe(false) + expect( + Object.getOwnPropertyDescriptor( + Object.getPrototypeOf(foreignSubclass), + `getTime`, + ), + ).toBeUndefined() + + for (const value of [localSubclass, foreignSubclass]) { + expect(normalizeValue(value)).toBe(timestamp) + expect( + ascComparator(value, new Date(timestamp), DEFAULT_COMPARE_OPTIONS), + ).toBe(0) + expect( + sign( + ascComparator( + value, + new Date(timestamp + 1), + DEFAULT_COMPARE_OPTIONS, + ), + ), + ).toBe(-1) + } + expect(normalizeValue(foreignInvalid)).toBeNaN() + expect(isUnorderable(foreignInvalid)).toBe(true) + expect( + ascComparator(foreignInvalid, Number.NaN, DEFAULT_COMPARE_OPTIONS), + ).toBe(0) + + // Borrowing that prototype and display tag cannot forge native Date + // slots. The intrinsic rejects this adjacent non-Date control. + const forged = Object.create(Object.getPrototypeOf(foreignSubclass)) as Date + Object.defineProperty(forged, Symbol.toStringTag, { value: `Other` }) + expect(() => Date.prototype.getTime.call(forged)).toThrow(TypeError) + expect(normalizeValue(forged)).toBe(forged) + expect( + ascComparator(forged, foreignSubclass, DEFAULT_COMPARE_OPTIONS), + ).not.toBe(0) + }) + it(`treats numeric NaN and invalid Dates as one non-finite order position`, () => { for (const foreign of [false, true]) { const invalid = dateInRealm(null, foreign) diff --git a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts index 2c424ad1b8..867e39f696 100644 --- a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts +++ b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts @@ -438,7 +438,7 @@ function evaluateWithKindOnlyTemporalKey(testCase: InCase): boolean | null { if ( kind !== undefined && Array.isArray(testCase.list) && - testCase.list.some((item) => temporalBrand(item)?.kind === kind) + testCase.list.some((candidate) => temporalBrand(candidate)?.kind === kind) ) { return true } @@ -482,8 +482,8 @@ function describeCase(testCase: InCase): string { } describe(`in evaluator`, () => { - const property = fc.asyncProperty(inCase, async (testCase) => - check(evaluateWithProduction, testCase), + const property = fc.asyncProperty(inCase, (testCase) => + Promise.resolve().then(() => check(evaluateWithProduction, testCase)), ) it(`agrees with the reference over a fixed campaign`, async () => { diff --git a/packages/db/tests/query/custom-index-window-selection-oracle.test.ts b/packages/db/tests/query/custom-index-window-selection-oracle.test.ts new file mode 100644 index 0000000000..1b7025e098 --- /dev/null +++ b/packages/db/tests/query/custom-index-window-selection-oracle.test.ts @@ -0,0 +1,185 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { BasicIndex } from '../../src/indexes/basic-index.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' +import { createLiveQueryCollection } from '../../src/query/live-query-collection.js' +import { evaluateReferenceExpression } from '../reference-expression-oracle.js' +import { withOracleCleanup } from '../utils.js' +import type { LoadSubsetOptions } from '../../src/types.js' + +/** + * A live query ordered by a numeric field publishes the requested slice of + * the provider's rows in ascending numeric order. The live-query architecture's + * ordered-request and atomic-window laws require the completed setWindow() + * result to include already cached rows as well as newly loaded rows. An index + * with a custom reverse comparator cannot stand in for that ordinary order. + * + * The bounded history caches ranks 20 and 40, installs indexes in one of four + * legal layouts, preloads a one-row window, then moves to offset one with two + * rows. A controlled provider fulfills each exact request from four immutable + * source rows; it never rewrites cached rows. A plain numeric sort and slice + * predicts both public checkpoints independently of index selection. Public + * snapshots and publication batches are compared after preload and after the + * window promise settles. This checks BasicIndex and BTreeIndex; it does not + * claim arbitrary comparators, ties, mutations, or remote provider behavior. + */ +type Row = { id: number; rank: number } +type IndexType = typeof BasicIndex | typeof BTreeIndex +type Layout = + `custom-first` | `ordinary-first` | `no-index` | `eager-custom-only` + +const rows: ReadonlyArray = [ + { id: 1, rank: 10 }, + { id: 2, rank: 20 }, + { id: 3, rank: 30 }, + { id: 4, rank: 40 }, +] +const expected = [...rows].sort((a, b) => a.rank - b.rank || a.id - b.id) +const expectedWindow = (offset: number, limit: number) => + expected.slice(offset, offset + limit).map(({ id }) => id) + +// This adapter models only the request grammar exercised here: a whole-source +// request, a finite prefix, or a cursor's current tie plus following rows. +// Its answers come from source facts and the request predicate, never indexes. +function requestedRows(options: LoadSubsetOptions): Array { + if (options.orderBy) { + expect(options.orderBy).toHaveLength(1) + expect(options.orderBy[0]!.expression).toMatchObject({ + type: `ref`, + path: [`rank`], + }) + expect(options.orderBy[0]!.compareOptions.direction).toBe(`asc`) + } + const matching = options.where + ? expected.filter( + (row) => evaluateReferenceExpression(options.where!, row) === true, + ) + : expected + if (!options.cursor) { + const start = options.offset ?? 0 + return matching.slice( + start, + options.limit === undefined ? undefined : start + options.limit, + ) + } + const current = matching.filter((row) => + Boolean(evaluateReferenceExpression(options.cursor!.whereCurrent, row)), + ) + const following = matching + .filter((row) => + Boolean(evaluateReferenceExpression(options.cursor!.whereFrom, row)), + ) + .slice(0, options.limit) + return [ + ...new Map([...current, ...following].map((row) => [row.id, row])).values(), + ] +} + +async function observeWindowMove( + indexType: IndexType, + layout: Layout, + suffix: string, +) { + const replies: Array> = [] + const writes: Array = [] + const delivered = new Set([2, 4]) + const source = createCollection({ + id: `custom-window-source-${suffix}-${layout}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: layout === `eager-custom-only` ? `eager` : `off`, + defaultIndexType: indexType, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of rows.filter(({ id }) => delivered.has(id))) { + write({ type: `insert`, value: row }) + } + commit() + markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + const reply = requestedRows(options) + replies.push(reply.map(({ id }) => id)) + begin() + for (const row of reply) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + writes.push(row.id) + write({ type: `insert`, value: row }) + } + const receipt = commit(options.signal) + return receipt === true ? Promise.resolve() : receipt + }, + } + }, + }, + }) + await source.stateWhenReady() + const custom = () => + source.createIndex((row) => row.rank, { + indexType, + options: { compareFn: (a: number, b: number) => b - a }, + }) + const ordinary = () => source.createIndex((row) => row.rank, { indexType }) + if (layout === `custom-first`) { + custom() + ordinary() + } else if (layout === `ordinary-first`) { + ordinary() + custom() + } else if (layout === `eager-custom-only`) { + custom() + } + + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .offset(0) + .limit(1) + .select(({ row }) => ({ id: row.id, rank: row.rank })), + ) + const publications: Array> = [] + let subscription: ReturnType | undefined + return withOracleCleanup(async () => { + await live.preload() + const initial = Array.from(live.values(), ({ id }) => id) + subscription = live.subscribeChanges(() => { + publications.push(Array.from(live.values(), ({ id }) => id)) + }) + publications.length = 0 + await live.utils.setWindow({ offset: 1, limit: 2 }) + const moved = Array.from(live.values(), ({ id }) => id) + + // The finite provider answer contains the cached rank-20 row, while + // its write stream does not. This reaches the cached-row merge path. + expect(replies.some((reply) => reply.join(`,`) === `1,2,3`)).toBe(true) + expect(writes).toEqual([1, 3]) + expect(initial).toEqual(expectedWindow(0, 1)) + expect(moved).toEqual(expectedWindow(1, 2)) + expect(publications).toEqual([expectedWindow(1, 2)]) + if (layout === `eager-custom-only`) { + expect(source.indexes.size).toBe(2) + } + }, [ + () => subscription?.unsubscribe(), + () => live.cleanup(), + () => source.cleanup(), + ]) +} + +describe.each([ + [`BasicIndex`, BasicIndex], + [`BTreeIndex`, BTreeIndex], +] as const)(`%s ordered live window index selection`, (name, indexType) => { + it.each([ + `custom-first`, + `ordinary-first`, + `no-index`, + `eager-custom-only`, + ] as const)(`publishes the complete moved window with %s`, async (layout) => { + await observeWindowMove(indexType, layout, name) + }) +}) diff --git a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts index 6c423cda7f..e927b16e49 100644 --- a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts +++ b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts @@ -67,7 +67,10 @@ * A focused tagged-Date matrix checks native Dates from this and another * JavaScript realm. It compares direct snapshot, filtered subscriber, and live * query keys after preload and one sync transaction; an explicit BasicIndex - * lookup checks the indexed equality key when an index is present. + * lookup checks the indexed equality key when an index is present. Fixed local + * and foreign Date subclasses with inherited `getTime` join the same + * timestamp classes. A forged object with a borrowed subclass prototype does + * not join a Date class. * * Calibration: pinned histories separate Kleene from two-valued logic, and a * checker test requires the two-valued answer to fail. Hostile production @@ -87,8 +90,9 @@ * them. The mock sync source is a controlled provider; this oracle claims only * how the Collection filters and routes the sync transactions it supplies. * The tagged-Date matrix covers native `node:vm` Dates with ordinary Date - * prototypes, including own display tags. It excludes proxies, subclasses, - * prototype swaps, ordinary objects with spoofed Date tags, and other hosts. + * prototypes, including own display tags, and fixed local and foreign Date + * subclasses with inherited `getTime`. It excludes proxies, prototype swaps, + * replaced `getTime` methods, other hosts, and generated subclass histories. */ import { runInNewContext } from 'node:vm' import { fc, test as fcTest } from '@fast-check/vitest' @@ -264,17 +268,22 @@ function syncedModelRow(id: string, v: FieldValue): ModelRow { // This focused model starts from tags, not runtime Date classification. A // native Date and a number with the same finite timestamp are equal. Invalid // Dates and NaN occupy one equality class under the documented float rule. -// An own display tag does not change a native Date's equality class. +// An own display tag or a subclass does not change a native Date's equality +// class. A borrowed prototype without the native Date slot matches no Date. type DateEqualityTag = | { kind: `date` realm: `local` | `foreign` timestamp: number | `invalid` ownTag?: boolean + subclass?: boolean } | { kind: `number`; value: number | `NaN` } -type TaggedDateRow = { id: string; tag: DateEqualityTag } +type TaggedDateRow = { + id: string + tag: DateEqualityTag | { kind: `forged` } +} function dateEqualityClass(tag: DateEqualityTag): number | `unordered` { if (tag.kind === `date`) { @@ -289,7 +298,11 @@ function expectedTaggedDateKeys( ): Array { const expectedClass = dateEqualityClass(literal) return rows - .filter((row) => dateEqualityClass(row.tag) === expectedClass) + .filter( + (row) => + row.tag.kind !== `forged` && + dateEqualityClass(row.tag) === expectedClass, + ) .map((row) => row.id) .sort() } @@ -565,6 +578,8 @@ const pinnedChangeHistories: ReadonlyArray = [ // The fixed realm grammar crosses the Date constructor's realm with equal, // unequal, and invalid timestamps, plus foreign Dates with an own display tag. +// Fixed subclass rows and literals exercise inherited getTime on both sides of +// eq; the forged row controls against trusting only the borrowed prototype. // Number literals test the Date/number and invalid-Date/NaN equality classes. // A later sync transaction moves three rows between those classes. const taggedDateRows: ReadonlyArray = [ @@ -589,6 +604,47 @@ const taggedDateRows: ReadonlyArray = [ }, { id: `number-one`, tag: { kind: `number`, value: 1 } }, { id: `number-nan`, tag: { kind: `number`, value: `NaN` } }, + { + id: `local-subclass-one`, + tag: { + kind: `date`, + realm: `local`, + timestamp: 1, + subclass: true, + ownTag: true, + }, + }, + { + id: `foreign-subclass-one`, + tag: { + kind: `date`, + realm: `foreign`, + timestamp: 1, + subclass: true, + ownTag: true, + }, + }, + { + id: `foreign-subclass-two`, + tag: { + kind: `date`, + realm: `foreign`, + timestamp: 2, + subclass: true, + ownTag: true, + }, + }, + { + id: `foreign-subclass-invalid`, + tag: { + kind: `date`, + realm: `foreign`, + timestamp: `invalid`, + subclass: true, + ownTag: true, + }, + }, + { id: `borrowed-prototype-object`, tag: { kind: `forged` } }, ] const taggedDateLiterals: ReadonlyArray = [ @@ -601,6 +657,27 @@ const taggedDateLiterals: ReadonlyArray = [ { kind: `date`, realm: `foreign`, timestamp: `invalid`, ownTag: true }, { kind: `number`, value: 1 }, { kind: `number`, value: `NaN` }, + { + kind: `date`, + realm: `local`, + timestamp: 1, + subclass: true, + ownTag: true, + }, + { + kind: `date`, + realm: `foreign`, + timestamp: 1, + subclass: true, + ownTag: true, + }, + { + kind: `date`, + realm: `foreign`, + timestamp: `invalid`, + subclass: true, + ownTag: true, + }, ] // --------------------------------------------------------------------------- @@ -675,22 +752,42 @@ function sourceRow(id: string, v: FieldValue): Row { return v === MISSING ? { id } : { id, v } } -function runtimeTaggedDateValue(tag: DateEqualityTag): Date | number { +function runtimeTaggedDateValue( + tag: TaggedDateRow[`tag`], +): Date | number | object { + if (tag.kind === `forged`) { + const foreignSubclass = runInNewContext( + `class D extends Date {}; new D(1)`, + ) as Date + const forged = Object.create( + Object.getPrototypeOf(foreignSubclass), + ) as object + Object.defineProperty(forged, Symbol.toStringTag, { value: `Other` }) + return forged + } if (tag.kind === `number`) { return tag.value === `NaN` ? Number.NaN : tag.value } const timestamp = tag.timestamp === `invalid` ? Number.NaN : tag.timestamp + class LocalDate extends Date {} const date = tag.realm === `foreign` - ? (runInNewContext(`new Date(timestamp)`, { timestamp }) as Date) - : new Date(timestamp) + ? (runInNewContext( + tag.subclass + ? `class D extends Date {}; new D(timestamp)` + : `new Date(timestamp)`, + { timestamp }, + ) as Date) + : tag.subclass + ? new LocalDate(timestamp) + : new Date(timestamp) if (tag.ownTag) { Object.defineProperty(date, Symbol.toStringTag, { value: `Other` }) } return date } -type TaggedValueRow = { id: string; v: Date | number } +type TaggedValueRow = { id: string; v: Date | number | object } // The tag model predicts keys before the runtime values are constructed. The // driver observes the public live query, filtered subscription, and direct @@ -783,6 +880,16 @@ async function observeTaggedDateEquality( tag: { kind: `date`, realm: `foreign`, timestamp: 1, ownTag: true }, }, { index: 4, tag: { kind: `date`, realm: `foreign`, timestamp: 2 } }, + { + index: 10, + tag: { + kind: `date`, + realm: `foreign`, + timestamp: 2, + subclass: true, + ownTag: true, + }, + }, ] collection.utils.begin() for (const transition of transitions) { @@ -1318,7 +1425,7 @@ describe(`WHERE predicate publication oracle`, () => { }, [() => subscription.unsubscribe(), () => collection.cleanup()]) }) - it(`native Dates from another realm publish by timestamp and invalid-Date equality`, async () => { + it(`native Dates and subclasses publish by timestamp and invalid-Date equality`, async () => { const foreign = runtimeTaggedDateValue({ kind: `date`, realm: `foreign`, @@ -1327,6 +1434,15 @@ describe(`WHERE predicate publication oracle`, () => { // This checks the host premise: a native Date can fail local instanceof. expect(foreign instanceof Date).toBe(false) expect(Date.prototype.getTime.call(foreign)).toBe(1) + const foreignSubclass = runtimeTaggedDateValue({ + kind: `date`, + realm: `foreign`, + timestamp: 1, + subclass: true, + ownTag: true, + }) + expect(Date.prototype.getTime.call(foreignSubclass)).toBe(1) + expect(foreignSubclass instanceof Date).toBe(false) for (const indexed of [false, true]) { for (const literal of taggedDateLiterals) { diff --git a/packages/db/tests/temporal-equality-alias-oracle.test.ts b/packages/db/tests/temporal-equality-alias-oracle.test.ts index 5712b32590..dee3ce0c5e 100644 --- a/packages/db/tests/temporal-equality-alias-oracle.test.ts +++ b/packages/db/tests/temporal-equality-alias-oracle.test.ts @@ -592,6 +592,7 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () { id: `forged`, when: fake }, { id: `shadowed`, when: shadowed }, ] + const fakeCopy = { ...fake } for (const [, IndexType] of indexTypes) { for (const row of source) { // One row avoids asking the default ordering comparator to order a @@ -599,6 +600,12 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () const index = new IndexType(1, new PropRef([`when`])) index.add(row.id, row) expect(index.equalityLookup(aliasProbe)).toEqual(new Set()) + // Rejecting the forged brand must not discard the ordinary object's + // exact self-identity. A distinct structural copy stays outside it. + expect(index.equalityLookup(row.when)).toEqual(new Set([row.id])) + if (row.id === `forged`) { + expect(index.equalityLookup(fakeCopy)).toEqual(new Set()) + } } } @@ -618,16 +625,20 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () }) await withOracleCleanup(async () => { await collection.stateWhenReady() - const predicate = new Func(`eq`, [ - new PropRef([`when`]), - new Value(aliasProbe), - ]) - const keys = new Set( - collection - .currentStateAsChanges({ where: predicate }) - ?.map(({ value }) => value.id), - ) - expect(keys).toEqual(new Set()) + const keysFor = (probe: unknown) => { + const predicate = new Func(`eq`, [ + new PropRef([`when`]), + new Value(probe), + ]) + return new Set( + collection + .currentStateAsChanges({ where: predicate }) + ?.map(({ value }) => value.id), + ) + } + expect(keysFor(aliasProbe)).toEqual(new Set()) + expect(keysFor(fake)).toEqual(new Set([`forged`])) + expect(keysFor(fakeCopy)).toEqual(new Set()) }, [() => collection.cleanup()]) }) @@ -684,6 +695,7 @@ it(`uses the PlainDate brand for direct lookups and a public unindexed snapshot` expect(index.equalityLookup(probe)).toEqual( sample.matches ? new Set([sample.id]) : new Set(), ) + expect(index.equalityLookup(sample.when)).toEqual(new Set([sample.id])) } // Both insertion orders must retain the same equality class when a live // representative is removed. The forged objects are intentionally kept @@ -718,12 +730,32 @@ it(`uses the PlainDate brand for direct lookups and a public unindexed snapshot` }) await withOracleCleanup(async () => { await collection.stateWhenReady() - const predicate = new Func(`eq`, [new PropRef([`when`]), new Value(probe)]) - const keys = new Set( - collection - .currentStateAsChanges({ where: predicate }) - ?.map(({ value }) => value.id), + const keysFor = (probeValue: unknown) => { + const predicate = new Func(`eq`, [ + new PropRef([`when`]), + new Value(probeValue), + ]) + return new Set( + collection + .currentStateAsChanges({ where: predicate }) + ?.map(({ value }) => value.id), + ) + } + expect(keysFor(probe)).toEqual( + new Set([`genuine`, `shadowed-tag`, `shadowed-text`]), ) - expect(keys).toEqual(new Set([`genuine`, `shadowed-tag`, `shadowed-text`])) + // An impostor remains an ordinary object: querying by its own reference + // finds it, while a distinct object with the same display cannot. + const borrowedCopy = Object.assign( + Object.create(Temporal.PlainDate.prototype) as object, + { toString: () => `2024-01-01` }, + ) + for (const [impostor, id, copy] of [ + [fake, `forged-tag`, { ...fake }], + [borrowed, `borrowed-prototype`, borrowedCopy], + ] as const) { + expect(keysFor(impostor)).toEqual(new Set([id])) + expect(keysFor(copy)).toEqual(new Set()) + } }, [() => collection.cleanup()]) }) diff --git a/packages/db/tests/utils-oracle.property.test.ts b/packages/db/tests/utils-oracle.property.test.ts index 1713888f54..23c943460d 100644 --- a/packages/db/tests/utils-oracle.property.test.ts +++ b/packages/db/tests/utils-oracle.property.test.ts @@ -1174,19 +1174,6 @@ describe(`deepEquals property-based tests`, () => { // Keep newly added generated laws after the original registration order: // UTILS_ORACLE_PROPERTY uses ordinal selectors for saved seed/path replays. describe(`additional non-ring value boundaries`, () => { - utilsProperty([arbitraryFiniteDate])( - `Date differs from its ISO text and epoch number in both directions`, - (date) => { - assertDistinctTypes(date, date.toISOString(), deepEquals) - assertDistinctTypes(date, date.getTime(), deepEquals) - }, - ) - - utilsProperty([arbitraryRegExp])( - `RegExp differs from its source text in both directions`, - (regex) => assertDistinctTypes(regex, regex.source, deepEquals), - ) - utilsProperty([arbitraryRegExp])( `RegExp source and flags determine equality`, (regex) => { @@ -1233,6 +1220,19 @@ describe(`deepEquals property-based tests`, () => { expectCopyAndChanged(duration, copy, changed) }, ) + + utilsProperty([arbitraryFiniteDate])( + `Date differs from its ISO text and epoch number in both directions`, + (date) => { + assertDistinctTypes(date, date.toISOString(), deepEquals) + assertDistinctTypes(date, date.getTime(), deepEquals) + }, + ) + + utilsProperty([arbitraryRegExp])( + `RegExp differs from its source text in both directions`, + (regex) => assertDistinctTypes(regex, regex.source, deepEquals), + ) }) }) From 3668d1b88b93dd52fd7085bc6b6a8ade9b711115 Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 20:21:26 -0600 Subject: [PATCH 17/46] fix(db): preserve mixed index values and cross-realm equality --- .../fix-custom-index-order-selection.md | 2 +- .../fix-index-build-cross-realm-dates.md | 2 +- .../fix-mixed-index-and-foreign-bytes.md | 5 + docs/contributing/oracle-coverage.md | 47 +++- .../oracle-reviews/cross-realm-date-order.md | 19 ++ .../cross-realm-date-where-publication.md | 14 ++ .../custom-index-order-selection.md | 14 +- .../oracle-reviews/mixed-value-index-order.md | 62 +++++ .../oracle-reviews/temporal-equality-alias.md | 36 +++ .../utils-unit-retirement-round-7.md | 10 + packages/db/src/utils/comparison.ts | 49 +++- .../tests/comparison-oracle.property.test.ts | 46 ++++ .../index-update-oracle.property.test.ts | 232 ++++++++++++++++++ .../in-evaluator-oracle.property.test.ts | 33 ++- ...stom-index-window-selection-oracle.test.ts | 129 +++++----- ...dicate-publication-oracle.property.test.ts | 63 ++++- .../temporal-equality-alias-oracle.test.ts | 12 + 17 files changed, 689 insertions(+), 86 deletions(-) create mode 100644 .changeset/fix-mixed-index-and-foreign-bytes.md create mode 100644 docs/contributing/oracle-reviews/mixed-value-index-order.md diff --git a/.changeset/fix-custom-index-order-selection.md b/.changeset/fix-custom-index-order-selection.md index 74c14e600c..a7a2251cf0 100644 --- a/.changeset/fix-custom-index-order-selection.md +++ b/.changeset/fix-custom-index-order-selection.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Use a custom-comparator index for exact lookup without selecting its order for an ordinary Collection `orderBy` read. When a compatible ordinary index also exists, select it for ordered reads; eager indexing now creates one if needed, including for equivalent value-left range predicates. +Use a custom-comparator index for exact lookup without selecting its order for an ordinary Collection `orderBy` read. Select a compatible ordinary index for ordered reads, including live-query window moves with cached rows. Eager indexing creates a compatible index when needed, including for equivalent value-left range predicates. diff --git a/.changeset/fix-index-build-cross-realm-dates.md b/.changeset/fix-index-build-cross-realm-dates.md index 0fc2be832c..678f3676d3 100644 --- a/.changeset/fix-index-build-cross-realm-dates.md +++ b/.changeset/fix-index-build-cross-realm-dates.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Keep `undefined` values in nulls-first order when BasicIndex builds from existing rows. Compare native Dates from other JavaScript realms by timestamp in ordering and equality, including invalid Dates. +Keep `undefined` values in nulls-first order when BasicIndex builds from existing rows. Compare native Dates from other JavaScript realms by timestamp in ordering and equality, including invalid Dates and subclasses. Treat an ordinary object with a borrowed Date prototype as an ordinary object. diff --git a/.changeset/fix-mixed-index-and-foreign-bytes.md b/.changeset/fix-mixed-index-and-foreign-bytes.md new file mode 100644 index 0000000000..ff73a9e85c --- /dev/null +++ b/.changeset/fix-mixed-index-and-foreign-bytes.md @@ -0,0 +1,5 @@ +--- +'@tanstack/db': patch +--- + +Keep mixed values in default ordered indexes through deletes and updates. Compare native Uint8Arrays from different JavaScript realms by byte content in compiled equality and membership predicates. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 4ae656dee2..1f2641a6d4 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -468,7 +468,9 @@ records the five retired same-realm examples. A separate native `node:vm` Dates by supplied timestamps and invalidity, including equal times, numeric NaN with an invalid Date, and Dates with an own display tag. A fixed local/foreign subclass witness checks native timestamps inherited -through a Date prototype and rejects a borrowed-prototype impostor. The owner +through a Date prototype, checks valid and invalid subclass order in both +directions against finite Dates, and rejects local and foreign +borrowed-prototype impostors. The owner also checks an unindexed public ordered scan with a tagged foreign invalid Date. Exact `normalizeValue` results are direct utility observations. Indexed order, public Date range predicates, cursor continuation, proxies, deeper @@ -477,7 +479,7 @@ Date-consuming paths remain outside that witness. The comparison oracle also owns a bounded [Temporal order transfer](oracle-reviews/comparison-unit-retirement-round-6.md). Supplied ISO calendar fields and epoch-nanosecond offsets predict direct `ascComparator` signs and exact zeros for ordinary `PlainDate` and `ZonedDateTime` values from one polyfill copy. Fixed cases preserve the two retired unit blocks; generated cases vary calendar fields, offset zones, equal instants, and adjacent nanoseconds. Fresh-module histories reject a first-input constructor cache poisoned by a genuine instance or forged tag, including the direct `compareValues` path. Both direct helpers also receive a right-hand PlainDate with a constructor shadow. A genuine ZonedDateTime with an own shadowed display tag retains its instant position through both direct helpers. This lane does not establish public query or index behavior. Custom Temporal subclasses, spoofed or replaced instance prototypes, mixed native/polyfill implementations, other Temporal kinds, and public ordering/filter publication need separate witnesses. Spoofed Temporal prototypes can still cause asymmetric comparisons; the comparison owner needs a dedicated brand and mixed-value witness before that class can be called closed. -The [Temporal index order oracle](../../packages/db/tests/temporal-index-order-oracle.test.ts) +The [Temporal index order oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/temporal-index-order-oracle.test.ts) and its [review record](oracle-reviews/temporal-index-order.md) own direct BasicIndex and BTreeIndex ordered takes after add/build and one mutation continuation, plus an optimized-only public Collection snapshot with an @@ -493,7 +495,7 @@ Direct removal supplies the exact prior row. Mixed value domains, other Temporal kinds, public WHERE ranges, provider cursors, longer histories, and live-query publication require separate witnesses. -The [Temporal alias equality oracle](../../packages/db/tests/temporal-equality-alias-oracle.test.ts) +The [Temporal alias equality oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/temporal-equality-alias-oracle.test.ts) and its [review record](oracle-reviews/temporal-equality-alias.md) own the bounded `ZonedDateTime.equals()` partition for UTC/Etc/UTC aliases, a distinct fixed-offset zone, adjacent nanoseconds, and a different calendar. Direct @@ -517,7 +519,10 @@ field vectors that share default display text. Fixed controls reach compiled Equal and unequal values within Instant, PlainDate, PlainDateTime, and PlainTime reject a kind-only Temporal equality key at those same paths. A genuine Uint8Array with a shadowed display tag and a spoof-tagged Uint16Array -check the intrinsic byte-class boundary. The `in` property has a guarded +check the intrinsic byte-class boundary. Native `node:vm` Uint8Arrays with +equal and unequal content, including an own tag shadow, also reach compiled +`eq`, constant `in`, and row-supplied `in` in both operand directions; the +original classifier failed equal foreign/local byte pairs. The `in` property has a guarded direct replay selector that runs only its random/replay campaign. The alias owner also checks a public unindexed `eq` scan with two native-equal PlainMonthDay rows and a distinct reference year. Direct BasicIndex/BTreeIndex @@ -535,8 +540,13 @@ Other zone aliases, custom spoofed prototypes, mixed Temporal implementations, mixed ordinary-object/Temporal ordered indexes, joins, grouping, query identity, live-query publication, and provider paths remain unproved. Direct D2 `groupBy` has its separately approved kind-and-string identity policy. +The alias owner records a concrete custom-prototype fake that currently +matches a genuine PlainDate and a distinct fake copy in direct indexes and +public `eq`. Its stated grammar excludes that boundary. A future contract +decision about non-branded object identity and a trusted Temporal intrinsic +witness are needed before it can be claimed closed. -The [custom index order selection oracle](../../packages/db/tests/custom-index-order-selection-oracle.test.ts) +The [custom index order selection oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/custom-index-order-selection-oracle.test.ts) and its [review record](oracle-reviews/custom-index-order-selection.md) check that a public ordered Collection snapshot does not select a reverse-number custom comparator by default ascending or descending `compareOptions`. @@ -552,7 +562,7 @@ custom-only index cannot serve an optimized range read. The fixed ready-state fixture covers BasicIndex and BTreeIndex, one numeric field, and limit one. Mutation histories, arbitrary callbacks, and provider publication remain outside this direct snapshot owner. The separate -[custom-index window selection oracle](../../packages/db/tests/query/custom-index-window-selection-oracle.test.ts) +[custom-index window selection oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/custom-index-window-selection-oracle.test.ts) checks the live-query compiler's receiving path. A controlled on-demand source starts with ranks 20 and 40 cached, then serves exact ordered requests from ranks 10, 20, 30, and 40. Moving a public window from offset 0/limit 1 to @@ -562,13 +572,15 @@ eager-custom-only layouts cross BasicIndex and BTreeIndex. Provider reply and write observations prove that the cached row reaches the local ordered path. The owner does not establish ties, nullish or nonnumeric orders, mutations, replays, multiple order terms, arbitrary callbacks, real providers, or other -window schedules. +window schedules. The eager-custom-only layout is eager auto-indexing on an +on-demand source. The public result rejects wrong custom-index admission; +ordinary-index use and an eager source path need distinct witnesses. The [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts) also has a [cross-realm Date equality matrix](oracle-reviews/cross-realm-date-where-publication.md). For local and `node:vm` Dates with timestamps 1 and 2 or invalid timestamps, including tagged foreign Dates and fixed local/foreign subclasses with an own -display tag, plus number 1, NaN, and a borrowed-prototype impostor, it +display tag, plus number 1, NaN, and a foreign borrowed-prototype impostor, it compares public live-query keys, filtered subscriber keys, and direct snapshot keys after preload and a three-row sync transaction. Auto-indexing is off in both modes; one mode adds BasicIndex explicitly and @@ -577,8 +589,25 @@ proves a public indexed snapshot result but does not prove that the live-query planner chose the index. Other realm hosts, proxies, deeper subclass chains, Date prototype swaps or method replacement, public range predicates, and generated subclass histories need separate witnesses. +A focused unindexed public query also keeps one local +`Object.create(DateSubclass.prototype)` impostor outside Date equality while +retaining its same-object match. Indexed publication for that local impostor +remains open. -The direct [index refinement oracle](../../packages/db/tests/index-update-oracle.property.test.ts) +The [mixed-value index order review](oracle-reviews/mixed-value-index-order.md) +links the direct index and public WHERE owners. The direct owner checks +BasicIndex and BTreeIndex key conservation, exact membership, and within-kind +order across build, incremental admission, delete, and update for fixed +string/number, array/object, Date/object, PlainDate/object, and +callable/number histories. The WHERE owner pins a generated string/number +delete/update counterexample and compares public key sets after each commit. +The original default comparator failed these histories because cross-family +ties and runtime-ID order conflicted with within-family order. Cross-family +relative positions remain unspecified. Longer mixed histories, proxies, +mixed Temporal kinds, and non-transitive user comparators need separate +witnesses before a universal sorted-index claim. + +The direct [index refinement oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update-oracle.property.test.ts) also owns exact nullish membership and direct `IN` set selection for BasicIndex and BTreeIndex. Fresh single-key updates cross `undefined→null`, `null→undefined`, `1→undefined`, `undefined→1`, and diff --git a/docs/contributing/oracle-reviews/cross-realm-date-order.md b/docs/contributing/oracle-reviews/cross-realm-date-order.md index 56411eed17..da8445116b 100644 --- a/docs/contributing/oracle-reviews/cross-realm-date-order.md +++ b/docs/contributing/oracle-reviews/cross-realm-date-order.md @@ -160,3 +160,22 @@ oracles passed 127 focused tests after the repair. The separate WHERE owner records the public RED and GREEN observations. Proxies, deeper subclass chains, replaced methods or prototypes, indexed order, ranges, and cursor continuation remain unproved. Fresh exact-commit review of this follow-up remains pending. + +At committed head `be5b7a3c4`, a fresh hostile reviewer found no loss from the +five retired comparison blocks and confirmed the tagged-subclass repair. It +found a narrower order-check gap: valid subclasses were compared with the +subclass only on the left, and an invalid subclass was compared with NaN but +not a finite Date. The receiving oracle now checks both operand directions +against equal and later finite Dates and places an invalid subclass after a +finite Date in both directions. These are direct order observations; they do +not extend the indexed or public range claim. + +The same review wave found a local ordinary object that borrowed a Date +subclass prototype. `instanceof Date` accepted it, but the native Date +intrinsic rejected its missing internal slot. At `be5b7a3c4`, direct +`normalizeValue` threw instead of retaining the object's reference identity. +The repair places the fast local branch inside the existing guarded intrinsic +call, so this object falls through as an ordinary object. The direct witness +failed before the repair and passed after it. The separate WHERE record owns +the public key-set witness. Proxies, arbitrary prototype changes, and indexed +behavior for the impostor remain outside this addition. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md index 2d5c5a88a0..364e069ed3 100644 --- a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md +++ b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md @@ -135,3 +135,17 @@ oracles, 127 tests, with no type errors. Live-query planner index use, generated subclass histories, deeper subclass chains, other hosts, public ranges, and prototype/method replacement remain outside this witness. Fresh exact-commit review is pending. + +## Local borrowed-prototype counterexample at `be5b7a3c4` + +A fresh hostile reviewer found that the matrix's foreign borrowed-prototype +object never reached the local `instanceof Date` fast path. An ordinary object +made with `Object.create(LocalDateSubclass.prototype)` has no native Date +slot, although local `instanceof Date` returns true. In an unindexed public +query, `not(eq(v, Date(1)))` omitted its only row instead of publishing that +row. The new focused witness establishes the missing slot with the native +intrinsic, then checks both Date inequality and same-object identity after +preload. It failed at the public key set on the reviewed commit and passed +after the guarded Date-classifier repair. The matrix still preserves the +foreign borrowed-prototype control. This addition covers one unindexed row; +indexed publication for the local impostor and other host realms remain open. diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md index 78003724f1..27e1846184 100644 --- a/docs/contributing/oracle-reviews/custom-index-order-selection.md +++ b/docs/contributing/oracle-reviews/custom-index-order-selection.md @@ -104,7 +104,7 @@ graph. Ordinary-first, ordinary-only, and no-index controls returned the correct rows. The reviewer reproduced both index types and found 16 public mismatches in a separate 64-scenario cache/window sweep. -The new [window oracle](../../../packages/db/tests/query/custom-index-window-selection-oracle.test.ts) +The new [window oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/custom-index-window-selection-oracle.test.ts) assigns numeric ranks in its source model and sorts and slices those four facts independently of index selection. Its controlled adapter evaluates each request's predicate and cursor against the source facts, records the @@ -135,3 +135,15 @@ receipts; the claim does not extend to remote hosts (ORC-014). Ties, nullish or nonnumeric orders, mutations, replay, multiple order terms, arbitrary callbacks, and other window schedules remain open. Fresh hostile review of the final committed oracle is pending. + +At committed head `be5b7a3c4`, a fresh hostile reviewer found no additional +numeric window mismatch in 128 controlled cache/layout scenarios. A read-only +wrong-index admission fault changed the settled public rows, confirming the +comparison rejects that error. The reviewer found that readiness and live +query construction happened before cleanup protection; the driver now enters +`withOracleCleanup` before those steps and releases any handles created before +a setup failure. The `eager-custom-only` layout means eager *auto-indexing on +an on-demand source*. The eight-case oracle proves the public result and the +provider's cached-row interaction. It does not measure ordinary-index use or +an eager source path. The architecture's index work law needs a separate +index-use witness before this owner claims that property. diff --git a/docs/contributing/oracle-reviews/mixed-value-index-order.md b/docs/contributing/oracle-reviews/mixed-value-index-order.md new file mode 100644 index 0000000000..21c49ea541 --- /dev/null +++ b/docs/contributing/oracle-reviews/mixed-value-index-order.md @@ -0,0 +1,62 @@ +# Mixed values in ordered indexes + +An indexed field can contain values from different JavaScript families. The +index API accepts these values, while mixed-domain range optimization is +disabled. Exact lookup, public WHERE membership, and unfiltered traversal +must still preserve each live key through legal deletes and updates. The +contract does not assign a public order between unlike families. Within one +orderable family, the configured comparator still applies. + +## Discovery and RED evidence + +An existing generated WHERE publication history failed with seed `907500611` +and path `78:8:6:6:8:7:6:7:8:10:10:9`. Its legal source held `b`, `1`, and +`a`, then deleted and updated rows. `BasicIndex.addToOrder` threw while it +looked up an ordered bucket. The default comparator had tied both strings +with the number through native relational coercion, although it ordered `a` +before `b`. Those ties do not form a partition that an ordered index can +maintain. + +The WHERE oracle now pins that history and compares public key sets after +each commit. The direct index owner also checks build and incremental +admission, reversed source order, exact lookup, and key conservation after +deletion and replacement for both BasicIndex and BTreeIndex. A source Map +predicts live keys and exact members. The test checks string collation only +within the string family; it does not prescribe where the number appears. + +The same comparison fault reached other legal values. A fixed +PlainDate/ordinary-object history updated one row from an ordinary object to +a later PlainDate, then an earlier PlainDate. A later update lost or duplicated +keys, and the original BasicIndex implementation could throw at the same +bucket lookup. A callable/number history also lost keys because two distinct +functions each tied the number after native coercion. Direct witnesses now +check arrays, Dates, PlainDates, and callables beside ordinary objects or +numbers. They compare a full ordered take with the source Map as a key +permutation, check exact membership after each action, and retain order only +within comparable values of the same family. The original comparator failed +the PlainDate and callable witnesses before repair. + +## Repair and scope + +The default comparator now separates strings from other primitives. It +separates arrays, native Dates, branded Temporal values, and ordinary +references before the runtime object-ID fallback. Functions use that same +reference path. These family positions prevent a cross-family comparison +from tying or reversing an order established within another family. Same-kind +string, array, Date, and Temporal comparisons remain in their existing paths. +Mixed Temporal kinds that lack a common defined order still reject ordering. +The relational WHERE helper keeps its existing comparison semantics. + +After the repair, six affected test files passed 600 tests with no Vitest type +errors. The explicit test TypeScript check, formatting, and diff checks +passed. ESLint reported five existing `require-await` warnings in the older +part of the direct index oracle and no warning in the new code. + +This repair covers the named finite direct histories and the public WHERE +history. It does not claim a specific cross-family sort order, arbitrary +mixed-value histories, proxy behavior, mixed Temporal-kind ordering, or +validity of an arbitrary user comparator. A user comparator that violates +transitivity can still corrupt an ordered index. The direct index and +comparison owners need further generated family-mix histories before a +universal sorted-index claim. A fresh exact-commit hostile review and final +package verification follow after commit. diff --git a/docs/contributing/oracle-reviews/temporal-equality-alias.md b/docs/contributing/oracle-reviews/temporal-equality-alias.md index 7ee1ee4a4d..b2a0d7ed7d 100644 --- a/docs/contributing/oracle-reviews/temporal-equality-alias.md +++ b/docs/contributing/oracle-reviews/temporal-equality-alias.md @@ -191,3 +191,39 @@ null-only-list authority question above and identified the older record now explicitly supersedes the sentence for null-containing lists. Two lint-only edits to the compiled `in` owner follow this review and require a fresh exact-commit check. + +## Hostile review of `be5b7a3c4` + +The alias reviewer found no loss from the retired comparison assertions. It +identified a positive native-equality gap: a genuine `[+00:00]` +ZonedDateTime with an own `timeZoneId = 'UTC'` shadow had been checked against +a UTC value and itself, but not against a fresh native-equal `[+00:00]` +value. The oracle now calibrates native `.equals()` on that fresh probe and +requires direct BasicIndex/BTreeIndex lookup and unindexed public `eq` to find +the shadowed row. Current production passed this probe. It rejects a wrong +classifier that treats every shadowed value as reference-only. + +The reviewer also demonstrated a custom-prototype ordinary object that +supplies a `Temporal.PlainDate` tag and matching display text. Current direct +indexes and public `eq` match it with a genuine PlainDate and a distinct fake +copy, while the native PlainDate intrinsic rejects it as a receiver. This is +outside this owner's declared custom-prototype grammar, so it does not undo +the bounded retirement verdict. It remains an open brand/ordinary-object +contract question under this owner: decide whether all non-branded objects +retain reference identity, then add genuine/fake/copy direct-index and public +`eq` witnesses. A shape or candidate-method guard can be spoofed; a robust +repair needs a trusted Temporal intrinsic boundary that supports the imported +polyfill without relying on `globalThis.Temporal`. + +The separate compiled `in` reviewer found native `Uint8Array` values from a +second JavaScript realm missing from its otherwise independent byte model. +Equal foreign/local bytes failed compiled `eq`, constant `in`, and row-supplied +`in` in both operand directions at `be5b7a3c4`. The receiving owner now +generates equal and unequal foreign bytes, including an own display-tag +shadow, and keeps a spoof-tagged wide-array negative control. The production +classifier uses a captured native typed-array getter after `ArrayBuffer.isView` +so realm and user display tags cannot change the byte class. The fixed +witness failed in twelve intended assertions before that repair and passed +after it; 359 related tests and static checks passed. This proves compiled +boolean equality in the `node:vm` realm, not browser iframe behavior or public +WHERE publication. The null-only-list law question above remains open. diff --git a/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md b/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md index 355871d0fa..833bea4954 100644 --- a/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md +++ b/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md @@ -85,3 +85,13 @@ replay of selector 34, seed `20260911`, path `0:0`, after one test at the forward `deepEquals` return. Removing the temporary rule made that replay pass with only the selected generated property active. Fresh review of the next committed head remains pending. + +At committed head `be5b7a3c4`, a fresh hostile reviewer found no blocker in +the six-test Date and RegExp transfer. All six retired assertions still have +fixed witnesses and broader generated laws. The owner passed 139 tests with +no type errors. Direct replays confirmed that selectors 34–38 name the +recorded properties, including selector 34 with seed `20260911` and path +`0:0`. The result covers direct `deepEquals` calls. It does not establish +public change events, cross-realm values, RegExp subclasses, or arbitrary +cross-type pairs. Ordinal selectors can shift if future registrations are +inserted ahead of them. diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index adef608d35..09650b1b7b 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -24,8 +24,8 @@ function getObjectId(obj: object): number { // display tag cannot hide its timestamp, and a forged tag cannot supply one. function dateTimestamp(value: unknown): number | undefined { if (value === null || typeof value !== `object`) return undefined - if (value instanceof Date) return Date.prototype.getTime.call(value) try { + if (value instanceof Date) return Date.prototype.getTime.call(value) if (Symbol.toStringTag in value) { // A Date subclass can inherit getTime from its Date prototype. Walk // descriptors without reading a user display-tag or getTime getter; @@ -105,6 +105,12 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { // For lexical sort we rely on direct comparison for primitive values } + // Keep the string collation class separate from other value types. Native + // relational comparison ties every nonnumeric string with every number, + // although those same strings have a distinct order among themselves. + if (typeof a === `string` && typeof b !== `string`) return -1 + if (typeof b === `string` && typeof a !== `string`) return 1 + // if a and b are both arrays, compare them element by element if (Array.isArray(a) && Array.isArray(b)) { for (let i = 0; i < Math.min(a.length, b.length); i++) { @@ -141,13 +147,32 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { if (aIsSymbol) return 1 if (bIsSymbol) return -1 - // If at least one of the values is an object, use stable IDs for comparison - const aIsObject = typeof a === `object` - const bIsObject = typeof b === `object` + // Function values are references too; coercing them against numbers can + // create the same non-transitive ties as unlike ordered object kinds. + const aIsObject = typeof a === `object` || typeof a === `function` + const bIsObject = typeof b === `object` || typeof b === `function` if (aIsObject || bIsObject) { - // If both are objects, compare their stable IDs + // Compare unlike ordered kinds before assigning stable IDs to ordinary objects. if (aIsObject && bIsObject) { + // Array, Date, and Temporal comparisons order values within their own + // kinds above. Keep those kinds apart from ordinary objects: mixing a + // calendar comparison with runtime object IDs creates order cycles. + const aKind = Array.isArray(a) + ? 0 + : aDateTime !== undefined + ? 1 + : aTemporal + ? 2 + : 3 + const bKind = Array.isArray(b) + ? 0 + : bDateTime !== undefined + ? 1 + : bTemporal + ? 2 + : 3 + if (aKind !== bKind) return aKind - bKind const aId = getObjectId(a) const bId = getObjectId(b) return aId - bId @@ -217,11 +242,19 @@ export const defaultComparator = makeComparator({ stringSort: `locale`, }) -/** Include host Buffers when the current realm has a different Uint8Array. */ +// The native getter reads a view's typed-array class across realms even when +// its own display tag shadows Symbol.toStringTag. Buffer has the same native +// byte class as Uint8Array. +const typedArrayTag = Object.getOwnPropertyDescriptor( + Object.getPrototypeOf(Uint8Array.prototype), + Symbol.toStringTag, +)!.get! + +/** Recognize native Uint8Array values, including Buffers and foreign realms. */ export function isUint8Array(value: unknown): value is Uint8Array { return ( - value instanceof Uint8Array || - (typeof Buffer !== `undefined` && value instanceof Buffer) + ArrayBuffer.isView(value) && + Reflect.apply(typedArrayTag, value, []) === `Uint8Array` ) } diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index 5d29b70394..58842b58be 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -1098,6 +1098,9 @@ describe(`native Date order across JavaScript realms`, () => { expect( ascComparator(value, new Date(timestamp), DEFAULT_COMPARE_OPTIONS), ).toBe(0) + expect( + ascComparator(new Date(timestamp), value, DEFAULT_COMPARE_OPTIONS), + ).toBe(0) expect( sign( ascComparator( @@ -1107,12 +1110,39 @@ describe(`native Date order across JavaScript realms`, () => { ), ), ).toBe(-1) + expect( + sign( + ascComparator( + new Date(timestamp + 1), + value, + DEFAULT_COMPARE_OPTIONS, + ), + ), + ).toBe(1) } expect(normalizeValue(foreignInvalid)).toBeNaN() expect(isUnorderable(foreignInvalid)).toBe(true) expect( ascComparator(foreignInvalid, Number.NaN, DEFAULT_COMPARE_OPTIONS), ).toBe(0) + expect( + sign( + ascComparator( + foreignInvalid, + new Date(timestamp), + DEFAULT_COMPARE_OPTIONS, + ), + ), + ).toBe(1) + expect( + sign( + ascComparator( + new Date(timestamp), + foreignInvalid, + DEFAULT_COMPARE_OPTIONS, + ), + ), + ).toBe(-1) // Borrowing that prototype and display tag cannot forge native Date // slots. The intrinsic rejects this adjacent non-Date control. @@ -1125,6 +1155,22 @@ describe(`native Date order across JavaScript realms`, () => { ).not.toBe(0) }) + it(`rejects a local Date subclass prototype without a native Date slot`, () => { + class LocalDate extends Date {} + const forged = Object.create(LocalDate.prototype) as object + const real = new Date(1) + // instanceof sees the borrowed local prototype, but the Date intrinsic + // rejects the missing native slot. The object keeps reference identity. + expect(forged instanceof Date).toBe(true) + expect(() => Date.prototype.getTime.call(forged)).toThrow(TypeError) + expect(normalizeValue(forged)).toBe(forged) + expect(isUnorderable(forged)).toBe(false) + const forward = sign(ascComparator(forged, real, DEFAULT_COMPARE_OPTIONS)) + const reverse = sign(ascComparator(real, forged, DEFAULT_COMPARE_OPTIONS)) + expect(forward).not.toBe(0) + expect(reverse).toBe(-forward) + }) + it(`treats numeric NaN and invalid Dates as one non-finite order position`, () => { for (const foreign of [false, true]) { const invalid = dateInRealm(null, foreign) diff --git a/packages/db/tests/index-update-oracle.property.test.ts b/packages/db/tests/index-update-oracle.property.test.ts index 2f004948f5..8a2e01f3e2 100644 --- a/packages/db/tests/index-update-oracle.property.test.ts +++ b/packages/db/tests/index-update-oracle.property.test.ts @@ -1,5 +1,6 @@ import { describe, expect, expectTypeOf, test, vi } from 'vitest' import { fc } from '@fast-check/vitest' +import { Temporal } from 'temporal-polyfill' import { createCollection } from '../src/collection/index.js' import { CollectionInErrorStateError } from '../src/errors.js' import { BasicIndex } from '../src/indexes/basic-index.js' @@ -2469,3 +2470,234 @@ test(`ordered scan resolves inherited collation outside each comparison`, async expect(collection.indexes.size).toBe(0) }) }) + +/** + * Mixed string/number source values remain legal index inputs. A default + * comparator cannot put every string at the same position as a number while + * still ordering those strings: that would make comparator equality + * non-transitive and leave no stable ordered partition for a delete or update. + * A plain Map predicts exact membership and key conservation; the two strings + * still follow their configured collation. Their position relative to the + * number is left free. This lane covers two scalar types, three source keys, + * build and incremental admission, and returned direct actions. It does not + * claim a universal order over objects, Dates, or arbitrary custom callbacks. + */ +describe.each(indexTypes)( + `%s mixed scalar index refinement`, + (_name, IndexType) => { + test.each([ + [`build`, `forward`], + [`build`, `reverse`], + [`incremental`, `forward`], + [`incremental`, `reverse`], + ] as const)( + `%s %s keeps ordered positions and exact members through delete and update`, + (admission, sourceOrder) => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map([ + [`r0`, `b`], + [`r1`, 1], + [`r2`, `a`], + ]) + const sourceRows = + sourceOrder === `forward` ? [...rows] : [...rows].reverse() + const source: Array<[string, { value: string | number }]> = + sourceRows.map(([key, value]) => [key, { value }]) + if (admission === `build`) index.build(source) + else for (const [key, value] of source) index.add(key, value) + + const check = () => { + const ordered = index.takeFromStart(rows.size + 1) + expect(ordered).toHaveLength(rows.size) + expect(new Set(ordered)).toEqual(new Set(rows.keys())) + const stringKeys = [...rows] + .filter(([, value]) => typeof value === `string`) + .sort((left, right) => + (left[1] as string).localeCompare(right[1] as string), + ) + .map(([key]) => key) + expect(ordered.filter((key) => stringKeys.includes(key))).toEqual( + stringKeys, + ) + for (const value of [`a`, `b`, 1, 2]) { + expect(index.equalityLookup(value)).toEqual( + new Set( + [...rows] + .filter(([, candidate]) => candidate === value) + .map(([key]) => key), + ), + ) + } + } + + check() // source admission + index.remove(`r0`, { value: `b` }) + rows.delete(`r0`) + check() + index.update(`r1`, { value: 1 }, { value: `b` }) + rows.set(`r1`, `b`) + check() + index.update(`r1`, { value: `b` }, { value: 2 }) + rows.set(`r1`, 2) + check() + }, + ) + + test(`keeps a numeric string and its number as distinct live keys`, () => { + const index = new IndexType(1, new PropRef([`value`])) + index.add(`number`, { value: 1 }) + index.add(`text`, { value: `1` }) + expect(index.equalityLookup(1)).toEqual(new Set([`number`])) + expect(index.equalityLookup(`1`)).toEqual(new Set([`text`])) + expect(new Set(index.takeFromStart(2))).toEqual( + new Set([`text`, `number`]), + ) + }) + }, +) + +/** + * A default index must retain every key through legal mutations even when its + * field holds an array, Date, or Temporal value beside an ordinary object. + * Element, timestamp, and calendar order compare values within their own kind; + * this model makes no claim about their position relative to the ordinary + * object. Its source Map and supplied labels predict exact membership and keys in an unfiltered + * ordered take. This fixed history reaches representative replacement and a + * later update that exposed a stale ordered position in BasicIndex. + */ +describe.each(indexTypes)( + `%s mixed object index refinement`, + (_name, IndexType) => { + test.each([`Array`, `Date`, `PlainDate`] as const)( + `retains every key through %s and ordinary-object updates`, + (valueKind) => { + const values = + valueKind === `Array` + ? { high: [2], ordinary: {}, low: [1] } + : valueKind === `Date` + ? { high: new Date(2), ordinary: {}, low: new Date(1) } + : { + high: Temporal.PlainDate.from(`2024-01-02`), + ordinary: {}, + low: Temporal.PlainDate.from(`2024-01-01`), + } + type Label = keyof typeof values + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map([ + [`high`, `high`], + [`ordinary`, `ordinary`], + [`low`, `low`], + ]) + for (const [key, label] of rows) { + index.add(key, { value: values[label] }) + } + + const check = () => { + const orderedKeys = index.takeFromStart(rows.size + 1) + expect(orderedKeys).toHaveLength(rows.size) + expect(new Set(orderedKeys)).toEqual(new Set(rows.keys())) + for (const label of [`high`, `ordinary`, `low`] as const) { + expect(index.equalityLookup(values[label])).toEqual( + new Set( + [...rows] + .filter(([, candidate]) => candidate === label) + .map(([key]) => key), + ), + ) + } + const lowKeys = [...rows] + .filter(([, label]) => label === `low`) + .map(([key]) => key) + const highKeys = [...rows] + .filter(([, label]) => label === `high`) + .map(([key]) => key) + if (lowKeys.length > 0 && highKeys.length > 0) { + expect( + Math.max(...lowKeys.map((key) => orderedKeys.indexOf(key))), + ).toBeLessThan( + Math.min(...highKeys.map((key) => orderedKeys.indexOf(key))), + ) + } + } + + check() + index.update( + `ordinary`, + { value: values.ordinary }, + { value: values.high }, + ) + rows.set(`ordinary`, `high`) + check() + index.update(`ordinary`, { value: values.high }, { value: values.low }) + rows.set(`ordinary`, `low`) + check() + index.update(`low`, { value: values.low }, { value: values.high }) + rows.set(`low`, `high`) + check() + }, + ) + }, +) + +/** + * An index expression may return a function reference. The source Map predicts + * exact reference membership and one ordered occurrence per live key; it does + * not prescribe an order between functions and numbers. Two distinct function + * values beside a number exercise build, incremental admission, deletion, and + * replacement. Native relational comparison can otherwise tie each function + * to the number while ordering the two functions apart. + */ +describe.each(indexTypes)( + `%s callable index refinement`, + (_name, IndexType) => { + test.each([`build`, `incremental`] as const)( + `%s conserves live keys through function and number changes`, + (admission) => { + const values = { + a: function a() {}, + b: function b() {}, + one: 1, + two: 2, + } + type Label = keyof typeof values + const rows = new Map([ + [`b`, `b`], + [`number`, `one`], + [`a`, `a`], + ]) + const index = new IndexType(1, new PropRef([`value`])) + const source: Array<[string, { value: (typeof values)[Label] }]> = [ + ...rows, + ].map(([key, label]) => [key, { value: values[label] }]) + if (admission === `build`) index.build(source) + else for (const [key, value] of source) index.add(key, value) + + const check = () => { + const orderedKeys = index.takeFromStart(rows.size + 1) + expect(orderedKeys).toHaveLength(rows.size) + expect(new Set(orderedKeys)).toEqual(new Set(rows.keys())) + for (const label of [`a`, `b`, `one`, `two`] as const) { + expect(index.equalityLookup(values[label])).toEqual( + new Set( + [...rows] + .filter(([, candidate]) => candidate === label) + .map(([key]) => key), + ), + ) + } + } + + check() + index.remove(`b`, { value: values.b }) + rows.delete(`b`) + check() + index.update(`number`, { value: values.one }, { value: values.b }) + rows.set(`number`, `b`) + check() + index.update(`number`, { value: values.b }, { value: values.two }) + rows.set(`number`, `two`) + check() + }, + ) + }, +) diff --git a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts index 867e39f696..a14dcc24ac 100644 --- a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts +++ b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts @@ -1,3 +1,4 @@ +import { runInNewContext } from 'node:vm' import { describe, expect, it } from 'vitest' import fc from 'fast-check' import { Temporal } from 'temporal-polyfill' @@ -40,8 +41,9 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * of each rule: equal and unequal strings, a string that begins with the * internal normalization prefix, booleans, `0`, `-0`, `1`, `NaN`, `1n`, a * valid Date and its timestamp, an invalid Date, equal and unequal byte - * arrays, a Buffer, genuine bytes with an own display-tag shadow, a wide - * typed array with a spoofed byte tag, Temporal values including ZonedDateTime + * arrays, a Buffer, native bytes from another realm with equal and unequal + * content, genuine local and foreign bytes with own display-tag shadows, a + * wide typed array with a spoofed byte tag, Temporal values including ZonedDateTime * zone aliases, a forged tag, a borrowed prototype, shadowed display and * field members, * unequal values within each modeled Temporal kind, reference ISO year/day @@ -84,6 +86,13 @@ const shadowedTagBytes = new Uint8Array([1, 2]) Object.defineProperty(shadowedTagBytes, Symbol.toStringTag, { value: `ordinary`, }) +const foreignBytes = runInNewContext(`new Uint8Array([1, 2])`) as Uint8Array +const unequalForeignBytes = runInNewContext( + `new Uint8Array([1, 3])`, +) as Uint8Array +const shadowedForeignBytes = runInNewContext( + `Object.defineProperty(new Uint8Array([1, 2]), Symbol.toStringTag, { value: 'ordinary' })`, +) as Uint8Array const spoofedWideBytes = new Uint16Array([1, 2]) Object.defineProperty(spoofedWideBytes, Symbol.toStringTag, { value: `Uint8Array`, @@ -186,6 +195,9 @@ const pool: ReadonlyArray = [ new Uint8Array([1, 3]), Buffer.from([1, 2]), shadowedTagBytes, + foreignBytes, + unequalForeignBytes, + shadowedForeignBytes, spoofedWideBytes, otherSpoofedWideBytes, plainDate, @@ -522,6 +534,23 @@ describe(`in evaluator`, () => { checkEqualityPaths(spoofedWideBytes, otherSpoofedWideBytes, false) }) + it(`compares native bytes from another realm by content in every compiled equality path`, () => { + // The independent model's captured typed-array intrinsic accepts a native + // view across realms. An own display tag does not change its byte class. + expect(foreignBytes instanceof Uint8Array).toBe(false) + expect(ArrayBuffer.isView(foreignBytes)).toBe(true) + expect(Object.prototype.toString.call(shadowedForeignBytes)).toBe( + `[object ordinary]`, + ) + expect(isBytes(foreignBytes)).toBe(true) + expect(isBytes(shadowedForeignBytes)).toBe(true) + expect(isBytes(spoofedWideBytes)).toBe(false) + checkEqualityPaths(foreignBytes, new Uint8Array([1, 2]), true) + checkEqualityPaths(shadowedForeignBytes, shadowedTagBytes, true) + checkEqualityPaths(unequalForeignBytes, new Uint8Array([1, 2]), false) + checkEqualityPaths(foreignBytes, spoofedWideBytes, false) + }) + it(`distinguishes unequal intrinsic values within each Temporal kind`, () => { // Native equals supplies each kind's expected identity relation. The // public compiler must reject a kind-only equality key in all three paths. diff --git a/packages/db/tests/query/custom-index-window-selection-oracle.test.ts b/packages/db/tests/query/custom-index-window-selection-oracle.test.ts index 1b7025e098..d098649097 100644 --- a/packages/db/tests/query/custom-index-window-selection-oracle.test.ts +++ b/packages/db/tests/query/custom-index-window-selection-oracle.test.ts @@ -83,72 +83,79 @@ async function observeWindowMove( const replies: Array> = [] const writes: Array = [] const delivered = new Set([2, 4]) - const source = createCollection({ - id: `custom-window-source-${suffix}-${layout}`, - getKey: ({ id }) => id, - syncMode: `on-demand`, - startSync: true, - autoIndex: layout === `eager-custom-only` ? `eager` : `off`, - defaultIndexType: indexType, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of rows.filter(({ id }) => delivered.has(id))) { - write({ type: `insert`, value: row }) - } - commit() - markReady() - return { - loadSubset: (options: LoadSubsetOptions) => { - const reply = requestedRows(options) - replies.push(reply.map(({ id }) => id)) - begin() - for (const row of reply) { - if (delivered.has(row.id)) continue - delivered.add(row.id) - writes.push(row.id) - write({ type: `insert`, value: row }) - } - const receipt = commit(options.signal) - return receipt === true ? Promise.resolve() : receipt - }, - } + const createSource = () => + createCollection({ + id: `custom-window-source-${suffix}-${layout}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: layout === `eager-custom-only` ? `eager` : `off`, + defaultIndexType: indexType, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of rows.filter(({ id }) => delivered.has(id))) { + write({ type: `insert`, value: row }) + } + commit() + markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + const reply = requestedRows(options) + replies.push(reply.map(({ id }) => id)) + begin() + for (const row of reply) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + writes.push(row.id) + write({ type: `insert`, value: row }) + } + const receipt = commit(options.signal) + return receipt === true ? Promise.resolve() : receipt + }, + } + }, }, - }, - }) - await source.stateWhenReady() - const custom = () => - source.createIndex((row) => row.rank, { - indexType, - options: { compareFn: (a: number, b: number) => b - a }, }) - const ordinary = () => source.createIndex((row) => row.rank, { indexType }) - if (layout === `custom-first`) { - custom() - ordinary() - } else if (layout === `ordinary-first`) { - ordinary() - custom() - } else if (layout === `eager-custom-only`) { - custom() - } - - const live = createLiveQueryCollection((query) => - query - .from({ row: source }) - .orderBy(({ row }) => row.rank, `asc`) - .offset(0) - .limit(1) - .select(({ row }) => ({ id: row.id, rank: row.rank })), - ) const publications: Array> = [] - let subscription: ReturnType | undefined + let unsubscribe: (() => unknown) | undefined + let cleanupLive: (() => unknown) | undefined + let cleanupSource: (() => unknown) | undefined return withOracleCleanup(async () => { + const source = createSource() + cleanupSource = () => source.cleanup() + await source.stateWhenReady() + const custom = () => + source.createIndex((row) => row.rank, { + indexType, + options: { compareFn: (a: number, b: number) => b - a }, + }) + const ordinary = () => source.createIndex((row) => row.rank, { indexType }) + if (layout === `custom-first`) { + custom() + ordinary() + } else if (layout === `ordinary-first`) { + ordinary() + custom() + } else if (layout === `eager-custom-only`) { + custom() + } + + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .offset(0) + .limit(1) + .select(({ row }) => ({ id: row.id, rank: row.rank })), + ) + cleanupLive = () => live.cleanup() await live.preload() const initial = Array.from(live.values(), ({ id }) => id) - subscription = live.subscribeChanges(() => { + const subscription = live.subscribeChanges(() => { publications.push(Array.from(live.values(), ({ id }) => id)) }) + unsubscribe = () => subscription.unsubscribe() publications.length = 0 await live.utils.setWindow({ offset: 1, limit: 2 }) const moved = Array.from(live.values(), ({ id }) => id) @@ -163,11 +170,7 @@ async function observeWindowMove( if (layout === `eager-custom-only`) { expect(source.indexes.size).toBe(2) } - }, [ - () => subscription?.unsubscribe(), - () => live.cleanup(), - () => source.cleanup(), - ]) + }, [() => unsubscribe?.(), () => cleanupLive?.(), () => cleanupSource?.()]) } describe.each([ diff --git a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts index e927b16e49..1a63c0871f 100644 --- a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts +++ b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts @@ -70,7 +70,8 @@ * lookup checks the indexed equality key when an index is present. Fixed local * and foreign Date subclasses with inherited `getTime` join the same * timestamp classes. A forged object with a borrowed subclass prototype does - * not join a Date class. + * not join a Date class. A separate unindexed public query checks the local + * borrowed-prototype case against Date inequality and object identity. * * Calibration: pinned histories separate Kleene from two-valued logic, and a * checker test requires the two-valued answer to fail. Hostile production @@ -93,6 +94,11 @@ * prototypes, including own display tags, and fixed local and foreign Date * subclasses with inherited `getTime`. It excludes proxies, prototype swaps, * replaced `getTime` methods, other hosts, and generated subclass histories. + * The local borrowed-prototype query uses one row and observes keys after + * preload. It does not claim indexed publication for that forged object. + * A pinned mixed-scalar history checks an indexed source after deleting and + * replacing rows. Its WHERE result still follows `eq` truth; the index's + * private order must not lose an equality member or interrupt publication. */ import { runInNewContext } from 'node:vm' import { fc, test as fcTest } from '@fast-check/vitest' @@ -504,6 +510,18 @@ const pinnedPrefilteredSnapshot: History = { // Each change history moves rows across a predicate whose equality operand a // skip-if-no-match shortcut can misread. const pinnedChangeHistories: ReadonlyArray = [ + { + // The default ordered index must keep distinct string and number rows + // available to exact equality after a delete and two same-key updates. + kind: `changes`, + rows: [`b`, 1, `a`], + predicate: fieldEq(`a`), + transactions: [ + [{ type: `delete`, target: 0 }], + [{ type: `update`, target: 0, v: `b` }], + [{ type: `update`, target: 0, v: 2 }], + ], + }, { // `or` operands are alternatives, not conjuncts. kind: `changes`, @@ -1451,6 +1469,49 @@ describe(`WHERE predicate publication oracle`, () => { } }) + it(`keeps a forged local Date subclass prototype outside Date equality`, async () => { + class LocalDate extends Date {} + const forged = Object.create(LocalDate.prototype) as object + // The native-slot check establishes that this is an ordinary object, + // despite local instanceof reporting Date. + expect(forged instanceof Date).toBe(true) + expect(() => Date.prototype.getTime.call(forged)).toThrow(TypeError) + let collectionCleanup: (() => unknown) | undefined + let notDateCleanup: (() => unknown) | undefined + let sameObjectCleanup: (() => unknown) | undefined + await withOracleCleanup(async () => { + const collection = createCollection( + mockSyncCollectionOptions<{ id: string; v: object }>({ + id: `where-publication-forged-date-${collectionSerial++}`, + getKey: (row) => row.id, + autoIndex: `off`, + initialData: [{ id: `forged`, v: forged }], + }), + ) + collectionCleanup = () => collection.cleanup() + const notDate = createLiveQueryCollection((q) => + q + .from({ row: collection }) + .where(({ row }) => not(eq(row.v, new Date(1)))), + ) + notDateCleanup = () => notDate.cleanup() + const sameObject = createLiveQueryCollection((q) => + q.from({ row: collection }).where(({ row }) => eq(row.v, forged)), + ) + sameObjectCleanup = () => sameObject.cleanup() + await notDate.preload() + // A distinct Date is unequal to this object. The same object remains + // equal by identity at the public query publication checkpoint. + expect([...notDate.keys()]).toEqual([`forged`]) + await sameObject.preload() + expect([...sameObject.keys()]).toEqual([`forged`]) + }, [ + () => notDateCleanup?.(), + () => sameObjectCleanup?.(), + () => collectionCleanup?.(), + ]) + }) + for (const history of [ ...pinnedHistories, pinnedPrefilteredSnapshot, diff --git a/packages/db/tests/temporal-equality-alias-oracle.test.ts b/packages/db/tests/temporal-equality-alias-oracle.test.ts index dee3ce0c5e..89b1f1c99d 100644 --- a/packages/db/tests/temporal-equality-alias-oracle.test.ts +++ b/packages/db/tests/temporal-equality-alias-oracle.test.ts @@ -584,8 +584,13 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () const shadowed = Temporal.ZonedDateTime.from( `2024-01-15T09:00:00+00:00[+00:00]`, ) + const freshOffsetProbe = Temporal.ZonedDateTime.from( + `2024-01-15T09:00:00+00:00[+00:00]`, + ) Object.defineProperty(shadowed, `timeZoneId`, { value: `UTC` }) expect(aliasProbe.equals(shadowed)).toBe(false) + expect(shadowed).not.toBe(freshOffsetProbe) + expect(shadowed.equals(freshOffsetProbe)).toBe(true) expect(() => Reflect.apply(aliasProbe.equals, aliasProbe, [fake])).toThrow() const source = [ @@ -605,6 +610,12 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () expect(index.equalityLookup(row.when)).toEqual(new Set([row.id])) if (row.id === `forged`) { expect(index.equalityLookup(fakeCopy)).toEqual(new Set()) + } else { + // The own zone accessor must not turn a genuine fixed-offset value + // into reference-only equality with its native-equal fresh instance. + expect(index.equalityLookup(freshOffsetProbe)).toEqual( + new Set([row.id]), + ) } } } @@ -637,6 +648,7 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () ) } expect(keysFor(aliasProbe)).toEqual(new Set()) + expect(keysFor(freshOffsetProbe)).toEqual(new Set([`shadowed`])) expect(keysFor(fake)).toEqual(new Set([`forged`])) expect(keysFor(fakeCopy)).toEqual(new Set()) }, [() => collection.cleanup()]) From 210651d207eefe9bbc025c780f6259e470225c9e Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 20:32:29 -0600 Subject: [PATCH 18/46] fix(db): preserve native byte and Date range semantics --- .../fix-index-build-cross-realm-dates.md | 2 +- .../fix-mixed-index-and-foreign-bytes.md | 2 +- docs/contributing/oracle-coverage.md | 16 ++ .../borrowed-date-indexed-range.md | 39 ++++ .../oracle-reviews/cross-realm-date-order.md | 12 ++ .../oracle-reviews/mixed-value-index-order.md | 3 + .../oracle-reviews/temporal-equality-alias.md | 26 +++ packages/db/src/indexes/base-index.ts | 3 +- packages/db/src/utils/comparison.ts | 18 +- packages/db/src/utils/index-optimization.ts | 4 +- .../tests/comparison-oracle.property.test.ts | 64 ++++++- .../in-evaluator-oracle.property.test.ts | 98 +++++++++- .../query/index-path-collision-oracle.test.ts | 179 ++++++++++++++++++ 13 files changed, 444 insertions(+), 22 deletions(-) create mode 100644 docs/contributing/oracle-reviews/borrowed-date-indexed-range.md diff --git a/.changeset/fix-index-build-cross-realm-dates.md b/.changeset/fix-index-build-cross-realm-dates.md index 678f3676d3..3e5fa6d26b 100644 --- a/.changeset/fix-index-build-cross-realm-dates.md +++ b/.changeset/fix-index-build-cross-realm-dates.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Keep `undefined` values in nulls-first order when BasicIndex builds from existing rows. Compare native Dates from other JavaScript realms by timestamp in ordering and equality, including invalid Dates and subclasses. Treat an ordinary object with a borrowed Date prototype as an ordinary object. +Keep `undefined` values in nulls-first order when BasicIndex builds from existing rows. Compare native Dates from other JavaScript realms by timestamp in ordering and equality, including invalid Dates and subclasses. Treat an ordinary object with a borrowed Date prototype as an ordinary object, including when deciding whether a range index is safe. diff --git a/.changeset/fix-mixed-index-and-foreign-bytes.md b/.changeset/fix-mixed-index-and-foreign-bytes.md index ff73a9e85c..f5b44723ea 100644 --- a/.changeset/fix-mixed-index-and-foreign-bytes.md +++ b/.changeset/fix-mixed-index-and-foreign-bytes.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Keep mixed values in default ordered indexes through deletes and updates. Compare native Uint8Arrays from different JavaScript realms by byte content in compiled equality and membership predicates. +Keep mixed values in default ordered indexes through deletes and updates. Compare native Uint8Arrays from different JavaScript realms by byte content in compiled equality and membership predicates, even when an instance shadows its byte count. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 1f2641a6d4..367382fe48 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -524,6 +524,13 @@ equal and unequal content, including an own tag shadow, also reach compiled `eq`, constant `in`, and row-supplied `in` in both operand directions; the original classifier failed equal foreign/local byte pairs. The `in` property has a guarded direct replay selector that runs only its random/replay campaign. +Own `length` and `byteLength` shadows, including a foreign byte view, and a +shadowed iterator cannot redefine native byte content. The compiled owner +checks equal and unequal pairs in both operand directions through `eq` and +both `in` forms; the comparison owner checks direct normalized-key relations. +The old byte-count read failed both boolean equality and direct key +normalization. Public indexed publication, detached buffers, and proxies +need separate witnesses. The alias owner also checks a public unindexed `eq` scan with two native-equal PlainMonthDay rows and a distinct reference year. Direct BasicIndex/BTreeIndex add, build, removal, re-add, and public indexed `eq` cover the orderable @@ -594,6 +601,15 @@ A focused unindexed public query also keeps one local retaining its same-object match. Indexed publication for that local impostor remains open. +The [borrowed-Date range review](oracle-reviews/borrowed-date-indexed-range.md) +records a separate public range owner in `index-path-collision-oracle.test.ts`. +Fixed `gt` and `lt` histories compare public snapshots before and after +BasicIndex or BTreeIndex is installed. An ordinary object borrowing a local +Date subclass prototype must take the scan path whether it is a source value +or the predicate bound. A genuine foreign Date subclass is a positive indexed +range control. Arbitrary coercion methods, invalid Dates, later publication, +other range operators, and proxies remain open there. + The [mixed-value index order review](oracle-reviews/mixed-value-index-order.md) links the direct index and public WHERE owners. The direct owner checks BasicIndex and BTreeIndex key conservation, exact membership, and within-kind diff --git a/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md b/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md new file mode 100644 index 0000000000..ebaaea155f --- /dev/null +++ b/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md @@ -0,0 +1,39 @@ +# Borrowed Date prototypes in indexed ranges + +A Collection index may change the work used to answer a range predicate, but +it must not change the public key set. The live-query architecture requires a +range to fall back when indexed values do not share the bound's relational +domain. A native Date subclass has a Date timestamp even across JavaScript +realms; an ordinary object made with `Object.create(DateSubclass.prototype)` +does not. The latter cannot enter the index's Date range domain merely because +`instanceof Date` returns true. + +The receiving owner is `packages/db/tests/query/index-path-collision-oracle.test.ts`. +Its finite model applies JavaScript numeric coercion to supplied Date values +and treats a throwing borrowed-prototype coercion as a nonmatch. The driver +compares public `currentStateAsChanges` keys before and after installing +BasicIndex or BTreeIndex. Its lookup spy and `optimizedOnly` read establish +whether the index is eligible. The source is a controlled, ready in-memory +Collection, so no provider or scheduling claim follows. + +At `3668d1b88`, a hostile WHERE review found a public counterexample: +`gt(v, Date(1))` omitted a borrowed-prototype row in the unindexed snapshot +and included it after installing BasicIndex. The new oracle reproduces the +same result mismatch for both index types. A second history supplies the +borrowed object as a predicate bound; its scan result is empty, and an index +lookup would be unsafe. The positive control uses a genuine foreign Date +subclass and requires an optimized indexed result, so the repair cannot +disable all Date range optimization. + +Production now uses the comparator's guarded native Date timestamp check when +it classifies an indexed value's domain and a range bound's eligibility. The +ordinary borrowed object therefore takes the scan path; native local and +foreign Dates remain eligible. The original code failed the after-index key +set, and the repaired focused oracle passes all 16 cases. The Date comparison owner +separately checks direct timestamp ordering and borrowed-object identity. + +This witness covers the named `gt` and `lt` snapshots, two index types, one +local borrowed prototype, and one foreign native subclass. It does not prove +arbitrary coercion methods, invalid Dates, later publication histories, +other range operators, proxies, or provider reads. A fresh hostile review of +the committed repair is pending. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-order.md b/docs/contributing/oracle-reviews/cross-realm-date-order.md index da8445116b..38becb25c6 100644 --- a/docs/contributing/oracle-reviews/cross-realm-date-order.md +++ b/docs/contributing/oracle-reviews/cross-realm-date-order.md @@ -179,3 +179,15 @@ call, so this object falls through as an ordinary object. The direct witness failed before the repair and passed after it. The separate WHERE record owns the public key-set witness. Proxies, arbitrary prototype changes, and indexed behavior for the impostor remain outside this addition. + +At `3668d1b88`, the next hostile comparison review found two direct-order +sensitivity gaps. Valid local and foreign Date subclasses were compared with +an equal and a later Date, but not an earlier Date. The owner now requires +opposite finite signs in both directions against that earlier Date; an +injected subclass-first comparator fails the new assertion. A foreign +borrowed-prototype control now also requires `isUnorderable` to be false and +opposite nonzero signs, rather than accepting any result other than zero. +The focused comparison oracle passed 111 tests with no type, lint, formatting, +or diff-check errors. These additions do not establish indexed range behavior; +the separate [range-domain review](borrowed-date-indexed-range.md) owns that +public boundary. diff --git a/docs/contributing/oracle-reviews/mixed-value-index-order.md b/docs/contributing/oracle-reviews/mixed-value-index-order.md index 21c49ea541..609c10bf30 100644 --- a/docs/contributing/oracle-reviews/mixed-value-index-order.md +++ b/docs/contributing/oracle-reviews/mixed-value-index-order.md @@ -23,6 +23,9 @@ admission, reversed source order, exact lookup, and key conservation after deletion and replacement for both BasicIndex and BTreeIndex. A source Map predicts live keys and exact members. The test checks string collation only within the string family; it does not prescribe where the number appears. +The public WHERE check establishes that writes and publication complete with +the right keys. It does not prove that a public indexed lookup ran after each +update; the direct index owner supplies exact lookup observations there. The same comparison fault reached other legal values. A fixed PlainDate/ordinary-object history updated one row from an ordinary object to diff --git a/docs/contributing/oracle-reviews/temporal-equality-alias.md b/docs/contributing/oracle-reviews/temporal-equality-alias.md index b2a0d7ed7d..0a4560ddec 100644 --- a/docs/contributing/oracle-reviews/temporal-equality-alias.md +++ b/docs/contributing/oracle-reviews/temporal-equality-alias.md @@ -227,3 +227,29 @@ witness failed in twelve intended assertions before that repair and passed after it; 359 related tests and static checks passed. This proves compiled boolean equality in the `node:vm` realm, not browser iframe behavior or public WHERE publication. The null-only-list law question above remains open. + +## Native byte-content follow-up at `3668d1b88` + +The next compiled `in` reviewer found that two unequal native Uint8Arrays +could each define an own `byteLength = 0` property and compare equal. A +one-sided shadow could also make equal arrays compare unequal. The compiled +`eq`, constant-list `in`, and row-supplied `in` paths all reached the wrong +result. Direct `normalizeValue` also generated the empty-byte key for a +nonempty shadowed array. These are production defects under the native +byte-content law, beyond the retired Temporal and comparison examples. + +The receiving `in` reference now reads captured native length and iteration +intrinsics, while production compares indexed bytes. Fixed controls cross +equal and unequal native bytes with own `byteLength` or `length` shadows, a +foreign byte array with a count shadow, and an iterator that yields values +different from the native bytes. Every compiled equality path checks both +operand directions. The comparison owner separately checks normalized key +equality and inequality under count shadows. Before the source repair, the +three compiled paths failed at their boolean comparisons and direct +normalization collided with the empty-byte key. The repaired focused set +passed 362 tests with no type, lint, formatting, or diff-check errors. + +These witnesses cover native Uint8Array and Buffer content at the compiled +equality and direct-key boundaries. They do not establish public indexed +publication, detached buffers, proxies, or browser iframe behavior. A fresh +hostile review of the committed repair is pending. diff --git a/packages/db/src/indexes/base-index.ts b/packages/db/src/indexes/base-index.ts index d791ca759c..b3c0d4d78b 100644 --- a/packages/db/src/indexes/base-index.ts +++ b/packages/db/src/indexes/base-index.ts @@ -2,6 +2,7 @@ import { compileSingleRowExpression } from '../query/compiler/evaluators.js' import { comparisonFunctions } from '../query/builder/functions.js' import { DEFAULT_COMPARE_OPTIONS, deepEquals } from '../utils.js' import { + dateTimestamp, indexCustomComparatorValue, indexOrderValue, makeCheckedComparator, @@ -339,7 +340,7 @@ export abstract class BaseIndex< function rangeValueDomain(value: unknown): string | undefined { if (value == null) return undefined - if (value instanceof Date) return `date` + if (dateTimestamp(value) !== undefined) return `date` return typeof value } diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index 09650b1b7b..21d46a9eb2 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -22,7 +22,7 @@ function getObjectId(obj: object): number { // The Date intrinsic checks the native brand across realms. Cheap shape gates // avoid that throwing probe for ordinary objects and Temporal values. A Date // display tag cannot hide its timestamp, and a forged tag cannot supply one. -function dateTimestamp(value: unknown): number | undefined { +export function dateTimestamp(value: unknown): number | undefined { if (value === null || typeof value !== `object`) return undefined try { if (value instanceof Date) return Date.prototype.getTime.call(value) @@ -249,6 +249,14 @@ const typedArrayTag = Object.getOwnPropertyDescriptor( Object.getPrototypeOf(Uint8Array.prototype), Symbol.toStringTag, )!.get! +const typedArrayByteLength = Object.getOwnPropertyDescriptor( + Object.getPrototypeOf(Uint8Array.prototype), + `byteLength`, +)!.get! + +function nativeByteLength(value: Uint8Array): number { + return Reflect.apply(typedArrayByteLength, value, []) as number +} /** Recognize native Uint8Array values, including Buffers and foreign realms. */ export function isUint8Array(value: unknown): value is Uint8Array { @@ -260,10 +268,11 @@ export function isUint8Array(value: unknown): value is Uint8Array { /** Compare two Uint8Arrays for content equality. */ function areUint8ArraysEqual(a: Uint8Array, b: Uint8Array): boolean { - if (a.byteLength !== b.byteLength) { + const length = nativeByteLength(a) + if (length !== nativeByteLength(b)) { return false } - for (let i = 0; i < a.byteLength; i++) { + for (let i = 0; i < length; i++) { if (a[i] !== b[i]) { return false } @@ -297,7 +306,8 @@ function canonicalTemporalZone(zoneId: string): string { function normalizeBinary(value: Uint8Array): string { let bytes = `` - for (let index = 0; index < value.byteLength; index++) { + const length = nativeByteLength(value) + for (let index = 0; index < length; index++) { bytes += String.fromCharCode(value[index]!) } return normalizedKey(`binary`, bytes) diff --git a/packages/db/src/utils/index-optimization.ts b/packages/db/src/utils/index-optimization.ts index d26195154b..aeed21681d 100644 --- a/packages/db/src/utils/index-optimization.ts +++ b/packages/db/src/utils/index-optimization.ts @@ -18,7 +18,7 @@ import { DEFAULT_COMPARE_OPTIONS } from '../utils.js' import { ReverseIndex } from '../indexes/reverse-index.js' import { hasVirtualPropPath } from '../virtual-props.js' -import { makeComparator } from './comparison.js' +import { dateTimestamp, makeComparator } from './comparison.js' import type { CompareOptions } from '../query/builder/types.js' import type { IndexOperation, IndexReader } from '../indexes/base-index.js' import type { BasicExpression } from '../query/ir.js' @@ -173,7 +173,7 @@ function isRangeOrderingDivergent( if (value === null) return false // Dates order consistently with the evaluator: valid Dates by time, and // invalid Dates as the greatest value under PostgreSQL float semantics. - return !(value instanceof Date) + return dateTimestamp(value) === undefined } default: return false diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index 58842b58be..a268aa8640 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -53,7 +53,8 @@ import type { CompareOptions } from '../src/query/builder/types' * recorded in `packages/db/CHANGELOG.md` (#779). The bounded key-size and * indexed-byte checks retain the regression contract introduced with #1797; * they constrain this internal utility, not a public serialized format or a - * universal performance bound. The 32-character margin is the existing + * universal performance bound. Own `length` and `byteLength` properties do + * not change native byte content. The 32-character margin is the existing * constant-overhead budget for the sampled 0..200-byte keys. */ @@ -1093,6 +1094,18 @@ describe(`native Date order across JavaScript realms`, () => { ), ).toBeUndefined() + const assertEarlierDateOrder = ( + value: Date, + compare: typeof ascComparator = ascComparator, + ) => { + expect( + sign(compare(value, new Date(timestamp - 1), DEFAULT_COMPARE_OPTIONS)), + ).toBe(1) + expect( + sign(compare(new Date(timestamp - 1), value, DEFAULT_COMPARE_OPTIONS)), + ).toBe(-1) + } + for (const value of [localSubclass, foreignSubclass]) { expect(normalizeValue(value)).toBe(timestamp) expect( @@ -1101,6 +1114,9 @@ describe(`native Date order across JavaScript realms`, () => { expect( ascComparator(new Date(timestamp), value, DEFAULT_COMPARE_OPTIONS), ).toBe(0) + // A subclass-first comparator would put this value before both + // unequal Dates. The earlier Date must instead precede it. + assertEarlierDateOrder(value) expect( sign( ascComparator( @@ -1120,6 +1136,20 @@ describe(`native Date order across JavaScript realms`, () => { ), ).toBe(1) } + // Feed a subclass-first answer through the same earlier-Date comparison. + // It reaches the comparison and fails at the expected sign assertion. + let reachedWrongOrder = false + expect(() => + assertEarlierDateOrder(foreignSubclass, (left, right, options) => { + if (left === foreignSubclass && right instanceof Date) { + reachedWrongOrder = true + return -1 + } + return ascComparator(left, right, options) + }), + ).toThrowError() + expect(reachedWrongOrder).toBe(true) + expect(normalizeValue(foreignInvalid)).toBeNaN() expect(isUnorderable(foreignInvalid)).toBe(true) expect( @@ -1150,9 +1180,15 @@ describe(`native Date order across JavaScript realms`, () => { Object.defineProperty(forged, Symbol.toStringTag, { value: `Other` }) expect(() => Date.prototype.getTime.call(forged)).toThrow(TypeError) expect(normalizeValue(forged)).toBe(forged) - expect( + expect(isUnorderable(forged)).toBe(false) + const forward = sign( ascComparator(forged, foreignSubclass, DEFAULT_COMPARE_OPTIONS), - ).not.toBe(0) + ) + const reverse = sign( + ascComparator(foreignSubclass, forged, DEFAULT_COMPARE_OPTIONS), + ) + expect(forward).not.toBe(0) + expect(reverse).toBe(-forward) }) it(`rejects a local Date subclass prototype without a native Date slot`, () => { @@ -1352,6 +1388,28 @@ describe(`makeComparator property-based tests`, () => { }) describe(`normalizeValue property-based tests`, () => { + it(`binary keys ignore own byte-count shadows`, () => { + // Supplied native bytes, rather than public count properties, establish + // the key relation. The function return is the comparison checkpoint. + const equalKey = normalizeValue(new Uint8Array([1, 2])) + const unequalKey = normalizeValue(new Uint8Array([9, 9])) + expect(equalKey).not.toBe(unequalKey) + for (const field of [`length`, `byteLength`] as const) { + const equal = Object.defineProperty(new Uint8Array([1, 2]), field, { + value: 0, + }) + const unequal = Object.defineProperty(new Uint8Array([9, 9]), field, { + value: 0, + }) + expect(equal[field]).toBe(0) + expect(unequal[field]).toBe(0) + expect(Array.from(equal)).toEqual([1, 2]) + expect(Array.from(unequal)).toEqual([9, 9]) + expect(normalizeValue(equal)).toBe(equalKey) + expect(normalizeValue(unequal)).toBe(unequalKey) + } + }) + it(`binary key size and content laws reach the bounded length margins`, () => { for (const length of [0, 1, 128, 129, 200]) { const bytes = Uint8Array.from({ length }, (_, index) => index % 256) diff --git a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts index a14dcc24ac..4d2e46635c 100644 --- a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts +++ b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts @@ -22,7 +22,8 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * - `NaN` and an invalid Date equal each other and nothing else. * - A valid Date equals its millisecond timestamp, and another Date with the * same timestamp. - * - Two Uint8Arrays (Buffers included) are equal when their bytes are equal. + * - Two Uint8Arrays (Buffers included) are equal when their native bytes are + * equal. Own `length` or `byteLength` properties cannot change that result. * - Temporal kinds with an intrinsic `.equals()` use that relation. This * includes equivalent time-zone aliases for ZonedDateTime, but excludes a * distinct fixed-offset zone. PlainMonthDay and PlainYearMonth also retain @@ -42,11 +43,12 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * internal normalization prefix, booleans, `0`, `-0`, `1`, `NaN`, `1n`, a * valid Date and its timestamp, an invalid Date, equal and unequal byte * arrays, a Buffer, native bytes from another realm with equal and unequal - * content, genuine local and foreign bytes with own display-tag shadows, a - * wide typed array with a spoofed byte tag, Temporal values including ZonedDateTime - * zone aliases, a forged tag, a borrowed prototype, shadowed display and - * field members, - * unequal values within each modeled Temporal kind, reference ISO year/day + * content, genuine local and foreign bytes with own display-tag shadows, bytes + * with own `length` or `byteLength` shadows, and a wide typed array with a + * spoofed byte tag. A fixed control also shadows byte iteration. Temporal + * values include ZonedDateTime zone aliases, a forged tag, a borrowed + * prototype, shadowed display and field members, unequal values within each + * modeled Temporal kind, reference ISO year/day * differences hidden by default display, and Duration carry forms with equal * display but distinct fields, one shared object and a structurally equal * copy, `null` and `undefined`. Lists have zero to six @@ -93,6 +95,23 @@ const unequalForeignBytes = runInNewContext( const shadowedForeignBytes = runInNewContext( `Object.defineProperty(new Uint8Array([1, 2]), Symbol.toStringTag, { value: 'ordinary' })`, ) as Uint8Array +const byteLengthShadowedForeignBytes = Object.defineProperty( + runInNewContext(`new Uint8Array([1, 2])`) as Uint8Array, + `byteLength`, + { value: 0 }, +) +function shadowByteCount( + bytes: ReadonlyArray, + field: `length` | `byteLength`, +): Uint8Array { + return Object.defineProperty(Uint8Array.from(bytes), field, { value: 0 }) +} +const byteLengthShadowed = shadowByteCount([1, 2], `byteLength`) +const equalByteLengthShadowed = shadowByteCount([1, 2], `byteLength`) +const unequalByteLengthShadowed = shadowByteCount([9, 9], `byteLength`) +const lengthShadowed = shadowByteCount([1, 2], `length`) +const equalLengthShadowed = shadowByteCount([1, 2], `length`) +const unequalLengthShadowed = shadowByteCount([9, 9], `length`) const spoofedWideBytes = new Uint16Array([1, 2]) Object.defineProperty(spoofedWideBytes, Symbol.toStringTag, { value: `Uint8Array`, @@ -198,6 +217,13 @@ const pool: ReadonlyArray = [ foreignBytes, unequalForeignBytes, shadowedForeignBytes, + byteLengthShadowedForeignBytes, + byteLengthShadowed, + equalByteLengthShadowed, + unequalByteLengthShadowed, + lengthShadowed, + equalLengthShadowed, + unequalLengthShadowed, spoofedWideBytes, otherSpoofedWideBytes, plainDate, @@ -250,6 +276,11 @@ const typedArrayTag = Object.getOwnPropertyDescriptor( Object.getPrototypeOf(Uint8Array.prototype), Symbol.toStringTag, )!.get! +const typedArrayLength = Object.getOwnPropertyDescriptor( + Object.getPrototypeOf(Uint8Array.prototype), + `length`, +)!.get! +const typedArrayValues = Uint8Array.prototype.values function isBytes(value: unknown): value is Uint8Array { return ( @@ -333,11 +364,21 @@ function referenceEqual(left: unknown, right: unknown): boolean { return isNotANumber(left) && isNotANumber(right) } if (isBytes(left) || isBytes(right)) { + if (!isBytes(left) || !isBytes(right)) return false + // Captured native length and value iteration ignore own properties. The + // model materializes the two byte sequences before comparing them; + // production walks indexes directly. + const leftLength = Reflect.apply(typedArrayLength, left, []) as number + const rightLength = Reflect.apply(typedArrayLength, right, []) as number + const leftBytes = Array.from( + Reflect.apply(typedArrayValues, left, []) as Iterable, + ) + const rightBytes = Array.from( + Reflect.apply(typedArrayValues, right, []) as Iterable, + ) return ( - isBytes(left) && - isBytes(right) && - left.length === right.length && - Array.from(left).every((byte, index) => byte === right[index]) + leftLength === rightLength && + leftBytes.every((byte, index) => byte === rightBytes[index]) ) } const leftTemporal = temporalBrand(left) @@ -551,6 +592,43 @@ describe(`in evaluator`, () => { checkEqualityPaths(foreignBytes, spoofedWideBytes, false) }) + it(`ignores own byte-count shadows in every compiled equality path`, () => { + // Both shadowed arrays still hold two native bytes. Pairing two zero + // shadows catches a comparison that stops before reading their content; + // pairing one with an unshadowed array catches a false inequality. + expect(byteLengthShadowed.byteLength).toBe(0) + expect(lengthShadowed.length).toBe(0) + expect(Reflect.apply(typedArrayLength, byteLengthShadowed, [])).toBe(2) + expect(Reflect.apply(typedArrayLength, lengthShadowed, [])).toBe(2) + expect(byteLengthShadowedForeignBytes instanceof Uint8Array).toBe(false) + expect(byteLengthShadowedForeignBytes.byteLength).toBe(0) + checkEqualityPaths(byteLengthShadowed, new Uint8Array([1, 2]), true) + checkEqualityPaths(byteLengthShadowed, equalByteLengthShadowed, true) + checkEqualityPaths(byteLengthShadowed, unequalByteLengthShadowed, false) + checkEqualityPaths(byteLengthShadowedForeignBytes, byteLengthShadowed, true) + checkEqualityPaths( + byteLengthShadowedForeignBytes, + unequalByteLengthShadowed, + false, + ) + checkEqualityPaths(lengthShadowed, new Uint8Array([1, 2]), true) + checkEqualityPaths(lengthShadowed, equalLengthShadowed, true) + checkEqualityPaths(lengthShadowed, unequalLengthShadowed, false) + }) + + it(`reads native byte values when an array shadows its iterator`, () => { + const bytes = new Uint8Array([1, 2]) + Object.defineProperty(bytes, Symbol.iterator, { + value: function* () { + yield 9 + yield 9 + }, + }) + expect(Array.from(bytes)).toEqual([9, 9]) + checkEqualityPaths(bytes, new Uint8Array([1, 2]), true) + checkEqualityPaths(bytes, new Uint8Array([9, 9]), false) + }) + it(`distinguishes unequal intrinsic values within each Temporal kind`, () => { // Native equals supplies each kind's expected identity relation. The // public compiler must reject a kind-only equality key in all three paths. diff --git a/packages/db/tests/query/index-path-collision-oracle.test.ts b/packages/db/tests/query/index-path-collision-oracle.test.ts index e1932ab1e8..bb95e85a64 100644 --- a/packages/db/tests/query/index-path-collision-oracle.test.ts +++ b/packages/db/tests/query/index-path-collision-oracle.test.ts @@ -18,6 +18,13 @@ * argument order, bound inclusivity, reversed operands, and bound field. * A selected-field ordering reads both paths through the public `$selected` * callback proxy. + * A separate range history stores genuine Dates, a native Date subclass, and + * an ordinary object with a borrowed Date subclass prototype. Its public + * `gt` result must be the same before and after installing either index type. + * Another range history uses the borrowed object as the predicate bound. + * Both mixed domains require scan fallback after installation. + * A foreign Date subclass is the adjacent valid-domain control: its native + * timestamp keeps the range eligible for index lookup across realms. * * Production driver: `currentStateAsChanges`, Collection subscriptions, and * live-query Collections run the real predicates and callback compilers. @@ -34,9 +41,13 @@ * Known omissions: This bounded owner does not claim arbitrary path segments, * nullish values, custom collation, or incremental publication histories. * Lazy demand target deduplication has a separate compiler-boundary witness. + * The Date-domain history does not cover arbitrary coercion methods, invalid + * Dates, or publication after later writes. */ +import { runInNewContext } from 'node:vm' import { expect, it, vi } from 'vitest' import { createCollection } from '../../src/collection/index.js' +import { BasicIndex } from '../../src/indexes/basic-index.js' import { BTreeIndex } from '../../src/indexes/btree-index.js' import { add, and, gt, lt, lte } from '../../src/query/builder/functions.js' import { PropRef } from '../../src/query/ir.js' @@ -163,6 +174,174 @@ it.each(cases)( }, ) +it.each([ + { name: `BasicIndex`, indexType: BasicIndex }, + { name: `BTreeIndex`, indexType: BTreeIndex }, +])( + `keeps borrowed Date prototypes outside indexed range results: $name`, + async ({ name, indexType }) => { + await checkWithCleanup(async (register) => { + class NativeDate extends Date {} + const borrowed = Object.create(NativeDate.prototype) as object + const bound = new Date(1) + const dateRows: Array<{ id: string; v: object }> = [ + { id: `before`, v: new Date(0) }, + { id: `borrowed`, v: borrowed }, + { id: `after`, v: new Date(2) }, + { id: `subclass`, v: new NativeDate(3) }, + ] + const collection = createCollection( + mockSyncCollectionOptions({ + id: `index-borrowed-date-${name}`, + getKey: (row: (typeof dateRows)[number]) => row.id, + initialData: dateRows, + autoIndex: `off`, + }), + ) + register(() => collection.cleanup()) + await collection.stateWhenReady() + + // The reference uses JavaScript numeric coercion for this Date bound. + // Borrowing a Date prototype cannot supply its native timestamp, so the + // coercion fails and that source row cannot satisfy `v > bound`. + const expected = dateRows + .filter(({ v }) => { + try { + return Number(v) > bound.getTime() + } catch { + return false + } + }) + .map(({ id }) => id) + .sort() + expect(expected).toEqual([`after`, `subclass`]) + + const where = gt(new PropRef([`v`]), bound) + const snapshotKeys = () => + collection + .currentStateAsChanges({ where })! + .map((change) => String(change.key)) + .sort() + + // An unindexed public snapshot establishes the first checkpoint. The + // indexed snapshot must refine the same result; optimizedOnly may decline + // this mixed-domain index rather than claiming a complete answer. + expect(snapshotKeys(), `before ${name}`).toEqual(expected) + const index = collection.createIndex((row) => row.v, { indexType }) + const lookup = vi.spyOn(index, `lookup`) + expect(snapshotKeys(), `after ${name}`).toEqual(expected) + expect(lookup, `unsafe range index not used`).not.toHaveBeenCalled() + expect( + collection.currentStateAsChanges({ where, optimizedOnly: true }), + ).toBeUndefined() + }) + }, +) + +it.each([ + { name: `BasicIndex`, indexType: BasicIndex }, + { name: `BTreeIndex`, indexType: BTreeIndex }, +])( + `does not treat a borrowed Date prototype as a range bound: $name`, + async ({ name, indexType }) => { + await checkWithCleanup(async (register) => { + class NativeDate extends Date {} + const borrowed = Object.create(NativeDate.prototype) as Date + const dateRows = [ + { id: `before`, v: new Date(0) }, + { id: `after`, v: new NativeDate(2) }, + ] + const collection = createCollection( + mockSyncCollectionOptions({ + id: `index-borrowed-date-bound-${name}`, + getKey: (row: (typeof dateRows)[number]) => row.id, + initialData: dateRows, + autoIndex: `off`, + }), + ) + register(() => collection.cleanup()) + await collection.stateWhenReady() + + // Numeric coercion of the borrowed bound throws. No source row can + // satisfy the relational predicate, regardless of index availability. + const expected = dateRows + .filter(({ v }) => { + try { + return Number(v) < Number(borrowed) + } catch { + return false + } + }) + .map(({ id }) => id) + expect(expected).toEqual([]) + + const where = lt(new PropRef([`v`]), borrowed) + const snapshotKeys = () => + collection + .currentStateAsChanges({ where })! + .map((change) => String(change.key)) + .sort() + expect(snapshotKeys(), `before ${name}`).toEqual(expected) + const index = collection.createIndex((row) => row.v, { indexType }) + const lookup = vi.spyOn(index, `lookup`) + expect(snapshotKeys(), `after ${name}`).toEqual(expected) + expect(lookup, `unsafe range index not used`).not.toHaveBeenCalled() + expect( + collection.currentStateAsChanges({ where, optimizedOnly: true }), + ).toBeUndefined() + }) + }, +) + +it.each([ + { name: `BasicIndex`, indexType: BasicIndex }, + { name: `BTreeIndex`, indexType: BTreeIndex }, +])( + `keeps foreign Date subclasses in the indexed Date domain: $name`, + async ({ name, indexType }) => { + await checkWithCleanup(async (register) => { + const bound = new Date(1) + const dateRows = [ + { id: `before`, v: new Date(0) }, + { + id: `foreign-subclass`, + v: runInNewContext(`new (class extends Date {})(2)`) as Date, + }, + ] + const collection = createCollection( + mockSyncCollectionOptions({ + id: `index-foreign-date-${name}`, + getKey: (row: (typeof dateRows)[number]) => row.id, + initialData: dateRows, + autoIndex: `off`, + }), + ) + register(() => collection.cleanup()) + await collection.stateWhenReady() + + // A foreign subclass has a native Date timestamp, so the direct numeric + // relation predicts the same key on scans and both indexed paths. + const expected = dateRows + .filter(({ v }) => Number(v) > bound.getTime()) + .map(({ id }) => id) + expect(expected).toEqual([`foreign-subclass`]) + const where = gt(new PropRef([`v`]), bound) + const snapshotKeys = () => + collection.currentStateAsChanges({ where })!.map((change) => change.key) + expect(snapshotKeys(), `before ${name}`).toEqual(expected) + const index = collection.createIndex((row) => row.v, { indexType }) + const lookup = vi.spyOn(index, `lookup`) + expect(snapshotKeys(), `after ${name}`).toEqual(expected) + expect(lookup, `native Date range index reached`).toHaveBeenCalled() + expect( + collection + .currentStateAsChanges({ where, optimizedOnly: true }) + ?.map((change) => change.key), + ).toEqual(expected) + }) + }, +) + it.each([false, true])( `preserves distinct fields in a Collection subscription callback (indexed=%s)`, async (indexed) => { From d974d4a4f3076db9f9df1789cff17e4c073ed6d9 Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 20:52:58 -0600 Subject: [PATCH 19/46] fix(db): close hostile oracle gaps and indexed Date ordering --- .../fix-index-build-cross-realm-dates.md | 2 +- .../fix-mixed-index-and-foreign-bytes.md | 2 +- .changeset/fix-temporal-index-order.md | 2 +- docs/contributing/oracle-coverage.md | 52 ++-- .../borrowed-date-indexed-range.md | 23 ++ .../comparison-unit-retirement-round-5.md | 18 ++ .../comparison-unit-retirement-round-6.md | 15 + .../oracle-reviews/cross-realm-date-order.md | 21 ++ .../cross-realm-date-where-publication.md | 29 +- .../custom-index-order-selection.md | 26 ++ .../oracle-reviews/mixed-value-index-order.md | 24 ++ .../oracle-reviews/perf-in-constant-set.md | 20 ++ .../oracle-reviews/temporal-equality-alias.md | 22 ++ .../oracle-reviews/temporal-index-order.md | 21 ++ .../utils-unit-retirement-round-7.md | 9 + packages/db/src/utils/comparison.ts | 65 +++-- .../tests/comparison-oracle.property.test.ts | 67 +++-- ...ustom-index-order-selection-oracle.test.ts | 258 +++++++++++++----- .../in-evaluator-oracle.property.test.ts | 69 ++++- .../query/index-path-collision-oracle.test.ts | 161 ++++++++++- packages/db/tests/query/subset-dedupe.test.ts | 16 +- ...dicate-publication-oracle.property.test.ts | 15 +- .../temporal-equality-alias-oracle.test.ts | 103 ++++++- .../tests/temporal-index-order-oracle.test.ts | 21 +- 24 files changed, 884 insertions(+), 177 deletions(-) diff --git a/.changeset/fix-index-build-cross-realm-dates.md b/.changeset/fix-index-build-cross-realm-dates.md index 3e5fa6d26b..2903ec38da 100644 --- a/.changeset/fix-index-build-cross-realm-dates.md +++ b/.changeset/fix-index-build-cross-realm-dates.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Keep `undefined` values in nulls-first order when BasicIndex builds from existing rows. Compare native Dates from other JavaScript realms by timestamp in ordering and equality, including invalid Dates and subclasses. Treat an ordinary object with a borrowed Date prototype as an ordinary object, including when deciding whether a range index is safe. +Keep `undefined` values in nulls-first order when BasicIndex builds from existing rows. Compare native Dates from other JavaScript realms by timestamp in ordering, equality, and relational queries, including invalid Dates and subclasses that override `valueOf`. Treat an ordinary object with a borrowed Date prototype as an ordinary object, including when deciding whether a range index is safe. diff --git a/.changeset/fix-mixed-index-and-foreign-bytes.md b/.changeset/fix-mixed-index-and-foreign-bytes.md index f5b44723ea..de45b53ad6 100644 --- a/.changeset/fix-mixed-index-and-foreign-bytes.md +++ b/.changeset/fix-mixed-index-and-foreign-bytes.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Keep mixed values in default ordered indexes through deletes and updates. Compare native Uint8Arrays from different JavaScript realms by byte content in compiled equality and membership predicates, even when an instance shadows its byte count. +Keep mixed values in default ordered indexes through deletes and updates, and preserve public Date/number order after installing an index. Compare native Uint8Arrays from different JavaScript realms by byte content in compiled equality and membership predicates, even when an instance shadows its byte count. diff --git a/.changeset/fix-temporal-index-order.md b/.changeset/fix-temporal-index-order.md index e10846d874..2ccd9408af 100644 --- a/.changeset/fix-temporal-index-order.md +++ b/.changeset/fix-temporal-index-order.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Order Temporal values by their calendar or instant semantics in indexed reads. Match ZonedDateTime equality across equivalent time-zone aliases while keeping distinct zones separate. Preserve PlainMonthDay reference years in unindexed equality, and PlainYearMonth reference days and Duration fields in indexed equality, when default display strings collide. Keep Temporal equality when genuine values shadow display members, and reject ordinary objects with forged tags. Preserve custom index comparator operands and per-row order even when values share one query-equality class, including signed zero through updates and BTree ranges. Prevent one Temporal value from changing the comparator used for later values. +Order Temporal values by their calendar or instant semantics in indexed reads. Match ZonedDateTime equality across equivalent time-zone aliases while keeping distinct zones separate, including genuine subclasses. Preserve PlainMonthDay reference years in unindexed equality, and PlainYearMonth reference days and Duration fields in indexed equality, when default display strings collide. Keep Temporal equality when genuine values shadow display members, and reject ordinary objects with forged tags. Preserve custom index comparator operands and per-row order even when values share one query-equality class, including signed zero through updates and BTree ranges. Prevent one Temporal value from changing the comparator used for later values. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 367382fe48..d7f049eea5 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -498,11 +498,13 @@ live-query publication require separate witnesses. The [Temporal alias equality oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/temporal-equality-alias-oracle.test.ts) and its [review record](oracle-reviews/temporal-equality-alias.md) own the bounded `ZonedDateTime.equals()` partition for UTC/Etc/UTC aliases, a distinct -fixed-offset zone, adjacent nanoseconds, and a different calendar. Direct -BasicIndex/BTreeIndex add, build, and removal checks compare independent -fixture-assigned classes. Unindexed and optimized-only indexed Collection -snapshots check public `eq`; the `in` evaluator oracle checks constant and -row-supplied alias lists. A plain object with a forged Temporal tag and a +fixed-offset zone, adjacent nanoseconds, and a different calendar. One native +ZonedDateTime subclass with inherited Temporal methods belongs to the UTC +class, while an ordinary object borrowing its prototype retains reference +identity. Direct BasicIndex/BTreeIndex add, build, and removal checks compare +independent fixture-assigned classes. Unindexed and optimized-only indexed +Collection snapshots check public `eq`; the `in` evaluator oracle checks +constant and row-supplied alias lists. A plain object with a forged Temporal tag and a genuine value with a shadowed zone accessor must stay outside the UTC class. Forged and borrowed ordinary objects also retain reference self-identity in direct lookup and unindexed public `eq`; distinct structural copies do not @@ -524,6 +526,10 @@ equal and unequal content, including an own tag shadow, also reach compiled `eq`, constant `in`, and row-supplied `in` in both operand directions; the original classifier failed equal foreign/local byte pairs. The `in` property has a guarded direct replay selector that runs only its random/replay campaign. +Native local and `node:vm` Dates, including invalid Dates and timestamp +numbers, reach those three compiled equality paths in both operand directions. +Borrowed Date prototypes remain ordinary reference values. The native-Date +reference model uses a captured Date intrinsic rather than local `instanceof`. Own `length` and `byteLength` shadows, including a foreign byte view, and a shadowed iterator cannot redefine native byte content. The compiled owner checks equal and unequal pairs in both operand directions through `eq` and @@ -567,6 +573,10 @@ the compatible index; an eager range subscriber creates one when needed for field-left and value-left single or paired relational predicates. A custom-only index cannot serve an optimized range read. The fixed ready-state fixture covers BasicIndex and BTreeIndex, one numeric field, and limit one. +Positive spies observe the later ordinary index's ordered take and numeric +lookup/range calls. Inclusive value-left boundaries are fixed controls. +Another fixed ready-state fixture compares unindexed and optimized indexed +Date/number order, both full and limit one, with an index-take reach check. Mutation histories, arbitrary callbacks, and provider publication remain outside this direct snapshot owner. The separate [custom-index window selection oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/custom-index-window-selection-oracle.test.ts) @@ -589,7 +599,7 @@ For local and `node:vm` Dates with timestamps 1 and 2 or invalid timestamps, including tagged foreign Dates and fixed local/foreign subclasses with an own display tag, plus number 1, NaN, and a foreign borrowed-prototype impostor, it compares public live-query keys, filtered subscriber keys, and direct snapshot -keys after preload and a three-row sync transaction. +keys after preload and a four-row sync transaction. Auto-indexing is off in both modes; one mode adds BasicIndex explicitly and also checks its equality lookup and an optimized-only public snapshot. This proves a public indexed snapshot result but does not prove that the live-query @@ -598,7 +608,8 @@ Date prototype swaps or method replacement, public range predicates, and generated subclass histories need separate witnesses. A focused unindexed public query also keeps one local `Object.create(DateSubclass.prototype)` impostor outside Date equality while -retaining its same-object match. Indexed publication for that local impostor +retaining its same-object match. It also excludes the impostor from the +invalid-Date/NaN equality class. Indexed publication for that local impostor remains open. The [borrowed-Date range review](oracle-reviews/borrowed-date-indexed-range.md) @@ -607,21 +618,30 @@ Fixed `gt` and `lt` histories compare public snapshots before and after BasicIndex or BTreeIndex is installed. An ordinary object borrowing a local Date subclass prototype must take the scan path whether it is a source value or the predicate bound. A genuine foreign Date subclass is a positive indexed -range control. Arbitrary coercion methods, invalid Dates, later publication, -other range operators, and proxies remain open there. +range control. A later fixed history gives local and foreign native Date +subclasses misleading `valueOf` results. Public scan and indexed `gt`, `gte`, +`lt`, and `lte` snapshots use their intrinsic timestamps, with an +optimized-only index-path check. Date/number scan predicates run in both +operand orders; mixed-domain index eligibility is not claimed. Other coercion +methods, invalid Dates, later publication, and proxies remain open there. The [mixed-value index order review](oracle-reviews/mixed-value-index-order.md) links the direct index and public WHERE owners. The direct owner checks BasicIndex and BTreeIndex key conservation, exact membership, and within-kind -order across build, incremental admission, delete, and update for fixed -string/number, array/object, Date/object, PlainDate/object, and -callable/number histories. The WHERE owner pins a generated string/number -delete/update counterexample and compares public key sets after each commit. +order through build, incremental admission, delete, and update for fixed +string/number and callable/number histories. Array/object, Date/object, and +PlainDate/object histories check incremental admission and updates; they do +not check build or delete for those families. The WHERE owner pins a generated +string/number delete/update counterexample and compares public key sets after +each commit. The original default comparator failed these histories because cross-family ties and runtime-ID order conflicted with within-family order. Cross-family -relative positions remain unspecified. Longer mixed histories, proxies, -mixed Temporal kinds, and non-transitive user comparators need separate -witnesses before a universal sorted-index claim. +relative positions in these direct witnesses remain unspecified. A separate +public ordered-snapshot witness in the custom index selection owner checks +that adding either index preserves the scan's full and first-row order for +one Date/number fixture. It does not define a universal cross-family order. +Longer mixed histories, proxies, mixed Temporal kinds, and non-transitive user +comparators need separate witnesses before a universal sorted-index claim. The direct [index refinement oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update-oracle.property.test.ts) also owns exact nullish membership and direct `IN` set selection for BasicIndex diff --git a/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md b/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md index ebaaea155f..041c01cb89 100644 --- a/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md +++ b/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md @@ -37,3 +37,26 @@ local borrowed prototype, and one foreign native subclass. It does not prove arbitrary coercion methods, invalid Dates, later publication histories, other range operators, proxies, or provider reads. A fresh hostile review of the committed repair is pending. + +## Native timestamp law for subclass ranges + +The frozen-head review at `210651d207eefe9bbc025c780f6259e470225c9e` +found a separate production defect. A genuine Date subclass can override +`valueOf` so native relational operators place it away from its actual +timestamp. The working-tree oracle uses a local and a foreign native subclass +whose overrides reverse their apparent positions. An independent model calls +the captured native `Date.prototype.getTime` and checks public scan results +for `gt`, `gte`, `lt`, and `lte` against a Date bound. The old relational scan +failed at this public key-set comparison. After the production repair, both +BasicIndex and BTreeIndex preserve those sets in ordinary and +`optimizedOnly` snapshots; lookup spies prove the indexed path is reached. +Four mixed Date/number scan predicates check both operand positions without +claiming mixed-domain index eligibility. The focused receiving run passed +132 tests. + +This extends the range witness beyond the earlier borrowed-prototype `gt` +and `lt` histories. It covers one ready in-memory Collection, fixed rows, +four Date-bound operators, and both index types. Invalid Dates, +arbitrary coercion methods, later publication histories, provider cursors, +and generated subclass histories remain outside the evidence. A fresh +committed-head hostile review is pending. diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md index a05abbf730..b6ab97f0b4 100644 --- a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md @@ -88,3 +88,21 @@ Date detector. The new direct and public oracle controls failed against that published implementation. The repair and its fresh committed-head verdict are recorded in [the Date order review](cross-realm-date-order.md). This finding does not restore a unique assertion to any of the five retired blocks. + +## Frozen-head hostile recheck and adjacent byte-key sensitivity + +At `210651d207eefe9bbc025c780f6259e470225c9e`, the hostile review again +found no lost assertion among the five retired same-realm number/Date blocks. +It reconstructed the ledger operands and the direct comparator and sort +checkpoints. This verdict applies to the frozen commit, not to the later +uncommitted repairs. + +Source inspection found a separate sensitivity gap in the comparison owner's +native byte normalization law. Its changed-byte property reached the final +byte, but could miss a wrong implementation that ignored an interior byte. +The working-tree oracle now requires different normalized keys for +`[1, 0, 2]` and `[1, 9, 2]` and runs an independently supplied changed-byte +property at the direct `normalizeValue` return. No hostile mutant was run for +this addition, so the fixed pair and property are sensitivity evidence, not +an exact-commit kill. The [Temporal transfer](comparison-unit-retirement-round-6.md) +and [alias owner](temporal-equality-alias.md) record their separate domains. diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md index 2e243feb1e..8e0c5ff923 100644 --- a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md @@ -117,3 +117,18 @@ given a spoofed Temporal prototype. This is outside the declared custom-prototype domain and is not evidence of closure for that domain. The comparison owner needs a separate brand and mixed-value witness before claiming it. Fresh review of the final committed head remains pending. + +## Frozen-head hostile recheck + +At `210651d207eefe9bbc025c780f6259e470225c9e`, a hostile ledger/source +review found no lost assertion in the two retired PlainDate and ZonedDateTime +comparison blocks. Their direct helper checkpoints and supplied operand pairs +remain covered. The review does not extend that verdict to the subsequent +working-tree changes. + +A distinct counterexample involved a genuine `ZonedDateTime` subclass whose +inherited native `.equals()` accepts an ordinary UTC peer. The old classifier +treated it as an ordinary object. The [Temporal alias owner](temporal-equality-alias.md) +now supplies the subclass equality and borrowed-prototype controls and records +the RED/GREEN production repair. This issue is outside the two retired direct +comparison assertions. The custom-prototype question above remains open. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-order.md b/docs/contributing/oracle-reviews/cross-realm-date-order.md index 38becb25c6..18575753c6 100644 --- a/docs/contributing/oracle-reviews/cross-realm-date-order.md +++ b/docs/contributing/oracle-reviews/cross-realm-date-order.md @@ -191,3 +191,24 @@ The focused comparison oracle passed 111 tests with no type, lint, formatting, or diff-check errors. These additions do not establish indexed range behavior; the separate [range-domain review](borrowed-date-indexed-range.md) owns that public boundary. + +## Working-tree subclass and relational-order follow-up + +The hostile comparison review of frozen head +`210651d207eefe9bbc025c780f6259e470225c9e` found no loss from the five +retired same-realm blocks. It found that the direct invalid-Date control only +used a foreign subclass. The working-tree control now also creates a local +invalid subclass with an own display tag and requires native invalid-Date +normalization, unorderability, and both finite comparison directions. This +closes a fixture sensitivity gap; the review did not identify a new +production failure for that added local value. + +The separate [public range owner](borrowed-date-indexed-range.md) found a +production defect in relational scans: a genuine Date subclass can override +`valueOf` while its native timestamp remains unchanged. A `gt` or `lt` scan +could follow the override instead of the timestamp. The working-tree repair +uses the native Date timestamp for relational comparisons, including mixed +Date/number operands. The range owner records the original RED public +snapshots and the bounded scan/index GREEN evidence. Direct Date order and +public range order are separate observations; this record does not claim +arbitrary coercion methods, providers, or cursor continuation. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md index 364e069ed3..6bec75218b 100644 --- a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md +++ b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md @@ -24,7 +24,7 @@ compares public live-query Collection keys, filtered source subscriber keys, and direct Collection snapshot keys after preload. The indexed mode also compares `BasicIndex.lookup('eq')` and a public snapshot with `optimizedOnly: true`; that snapshot cannot silently use the scan fallback. -One sync transaction moves three source rows across equality classes, and the +One sync transaction moves four source rows across equality classes, and the same checks run after commit. The test does not prove that the public live query planner selected the index. The source and live-query Collections and subscription are released through the owner's cleanup helper. @@ -42,11 +42,11 @@ intrinsic. This is a product result mismatch at a public checkpoint. With the comparison utility repair, the complete WHERE publication oracle passed 18 tests with no type errors. The DB oracle suite passed 5,006 tests -across 73 files with no type errors. The scope is native local and `node:vm` -Dates, finite timestamps 1 and 2, invalid Dates and NaN, one three-row sync -transaction, and the named observations. Other realm hosts, proxies, -subclasses, spoofed tags, arbitrary timestamp histories, and planner proof of -live-query index selection remain outside this witness. +across 73 files with no type errors. At that first repair, the scope was native +local and `node:vm` Dates, finite timestamps 1 and 2, invalid Dates and NaN, +one three-row sync transaction, and the named observations. Other realm hosts, +proxies, subclasses, spoofed tags, arbitrary timestamp histories, and +planner proof of live-query index selection remain outside this witness. The full DB suite initially reported two failures in the existing `subset-dedupe.test.ts` Date transport test. It expected local `Date(2)` and @@ -149,3 +149,20 @@ preload. It failed at the public key set on the reviewed commit and passed after the guarded Date-classifier repair. The matrix still preserves the foreign borrowed-prototype control. This addition covers one unindexed row; indexed publication for the local impostor and other host realms remain open. + +## Invalid-Date exclusion at the public WHERE cut + +The hostile review of frozen head +`210651d207eefe9bbc025c780f6259e470225c9e` found one remaining +source-inspection gap in the local borrowed-prototype control. The existing +unindexed `not(eq(v, Date(1)))` and same-object `eq` checks could pass if +the borrowed object also joined the invalid-Date/NaN equality class. The +working-tree witness now preloads a public `eq(row.v, new Date(NaN))` query +and requires an empty key set for that same borrowed row. This added cut +did not demonstrate a new production failure. The coverage map now states +the bounded matrix's four post-commit updated rows accurately. + +The witness establishes this single local borrowed object at an unindexed +public checkpoint. It does not cover indexed publication for that object, +arbitrary prototype chains, or other host realms. A committed-head hostile +review of the working-tree addition remains pending. diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md index 27e1846184..c630efa878 100644 --- a/docs/contributing/oracle-reviews/custom-index-order-selection.md +++ b/docs/contributing/oracle-reviews/custom-index-order-selection.md @@ -147,3 +147,29 @@ an on-demand source*. The eight-case oracle proves the public result and the provider's cached-row interaction. It does not measure ordinary-index use or an eager source path. The architecture's index work law needs a separate index-use witness before this owner claims that property. + +## Frozen-head review and mixed Date/number order + +Review of `210651d207eefe9bbc025c780f6259e470225c9e` found three +source-inspection gaps in the ordered-selection oracle. Correct public rows +alone did not prove the later ordinary index supplied the ordered and range +reads. Inclusive value-left `gte`/`lte` boundaries were absent. Some +Collection construction still preceded cleanup protection. The working-tree +oracle now spies on the ordinary index's take, lookup, and range methods; +checks lower, upper, and paired inclusive value-left predicates against +independent row filters; and registers each Collection inside protected +construction. These gaps did not demonstrate an additional production bug. + +An adjacent public witness did expose a production order mismatch for one +Date/number fixture. Before the repair, the unindexed ordered snapshot was +`[equal-number, number, date]`, while BasicIndex and BTreeIndex optimized +snapshots returned `[date, equal-number, number]`. The two new index-type +cases failed at that public full-order comparison. They now compare both +full order and the first-row window before and after index installation, +check the Date/timestamp exact-equality class, and spy on the optimized +index take. The repair retains raw native Dates as index order values while +preserving their normalized exact lookup key. The focused index campaign +passed 497 tests, and the full `@tanstack/db` run passed 12,345 tests in 281 +files, with no reported type errors. These are working-tree runs, not a +committed-head review. The witness requires scan/index agreement for this +fixture; it does not define a universal order between value families. diff --git a/docs/contributing/oracle-reviews/mixed-value-index-order.md b/docs/contributing/oracle-reviews/mixed-value-index-order.md index 609c10bf30..321c24c8ad 100644 --- a/docs/contributing/oracle-reviews/mixed-value-index-order.md +++ b/docs/contributing/oracle-reviews/mixed-value-index-order.md @@ -63,3 +63,27 @@ transitivity can still corrupt an ordered index. The direct index and comparison owners need further generated family-mix histories before a universal sorted-index claim. A fresh exact-commit hostile review and final package verification follow after commit. + +## Frozen-head retirement audit and public-order neighbor + +The hostile reviewer inspected the ledgers, both index implementations, and +the receiving direct and public oracles at +`210651d207eefe9bbc025c780f6259e470225c9e`. It found no lost assertion +among the 18 retired BasicIndex/BTreeIndex blocks. The direct grammar's +build and delete observations include string/number and callable/number +histories. Array/object, Date/object, and PlainDate/object have incremental +add/update observations here, without a build/delete claim. The +[coverage map](../oracle-coverage.md) now states those boundaries explicitly. +This was source and ledger inspection, not an exhaustive generated proof. + +An adjacent public Date/number history exposed a separate order defect. +The unindexed ordered snapshot returned +`[equal-number, number, date]`; after either index type was installed, the +optimized snapshot returned `[date, equal-number, number]`. Two intended +index-type assertions failed before the repair. The +[custom-index selection owner](custom-index-order-selection.md) now compares +the full public order and limit-one result before and after index installation +and proves an index take ran. Retaining native Dates as order values while +keeping their numeric exact keys made the 497-test focused campaign and +12,345-test full DB campaign pass. This establishes scan/index agreement for +that fixture, not a universal mixed-family order or arbitrary histories. diff --git a/docs/contributing/oracle-reviews/perf-in-constant-set.md b/docs/contributing/oracle-reviews/perf-in-constant-set.md index fe09c2883d..fc99099167 100644 --- a/docs/contributing/oracle-reviews/perf-in-constant-set.md +++ b/docs/contributing/oracle-reviews/perf-in-constant-set.md @@ -106,3 +106,23 @@ times with no source index: Median write times did not change beyond noise. Two of the three comparisons were 1.00× and 0.98×, and a same-code comparison was 0.96×. + +## Frozen-head reference-model correction + +The hostile review of `210651d207eefe9bbc025c780f6259e470225c9e` +found a false-green risk in `referenceEqual`: realm-local `instanceof Date` +could classify a native foreign Date as an ordinary reference even when the +compiled evaluator correctly compared its timestamp. The working-tree model +now reads a captured native `Date.prototype.getTime`, uses valid and invalid +foreign Dates, and rejects a local object that only borrows a Date subclass +prototype. Fixed witnesses check compiled `eq`, constant-list `in`, and +row-supplied `in` in both operand directions. This was a reference-model gap, +not an observed production defect. The focused receiving run passed 72 tests. + +The transport controls for equal native Uint8Arrays from different realms +now expect `[true, true]` for the compiled equality paths; request-identity +assertions remain in place. This corrects old test expectations rather than +altering the native byte-content law. The open null-containing-list authority +question recorded by the [Temporal alias review](temporal-equality-alias.md) +is unchanged. The working-tree changes still need a fresh committed-head +hostile review. diff --git a/docs/contributing/oracle-reviews/temporal-equality-alias.md b/docs/contributing/oracle-reviews/temporal-equality-alias.md index 0a4560ddec..496b15704b 100644 --- a/docs/contributing/oracle-reviews/temporal-equality-alias.md +++ b/docs/contributing/oracle-reviews/temporal-equality-alias.md @@ -253,3 +253,25 @@ These witnesses cover native Uint8Array and Buffer content at the compiled equality and direct-key boundaries. They do not establish public indexed publication, detached buffers, proxies, or browser iframe behavior. A fresh hostile review of the committed repair is pending. + +## Genuine subclass counterexample after frozen-head review + +At frozen head `210651d207eefe9bbc025c780f6259e470225c9e`, a genuine +`Temporal.ZonedDateTime` subclass inherited the native slot-checking +`.equals()` relation but the production classifier treated it as an ordinary +reference. The alias owner now constructs that subclass from independently +supplied instant and zone facts, verifies native equality against a UTC peer, +and requires its class in direct BasicIndex/BTreeIndex lookups and unindexed +public `eq` reads. It also removes the subclass representative and checks the +remaining class. An object that merely borrows the subclass prototype and a +distinct borrowed copy must retain reference identity. Eight intended +comparisons failed before the classifier repair; the focused receiving run +passed 28 tests afterward. The repair recognizes an inherited Temporal +prototype whose method checks native slots. + +This is a production defect adjacent to the retired comparison assertions, +not a lost assertion from those blocks. The bounded claim covers this +polyfill's genuine subclass with inherited methods. The custom-prototype +fake described above, replaced subclass methods, mixed implementations, and +provider paths remain open. The working-tree repair still needs a fresh +committed-head hostile review. diff --git a/docs/contributing/oracle-reviews/temporal-index-order.md b/docs/contributing/oracle-reviews/temporal-index-order.md index 5cf9e0a0db..e9745d6848 100644 --- a/docs/contributing/oracle-reviews/temporal-index-order.md +++ b/docs/contributing/oracle-reviews/temporal-index-order.md @@ -116,3 +116,24 @@ independent bounded histories exercised both index types under both order rules, and a wrong-result probe confirmed that the optimized public snapshot uses the index result. Provider cursors, arbitrary callbacks, and live-query publication remain outside this verdict. + +## Frozen-head direct tie-order recheck + +The hostile review of `210651d207eefe9bbc025c780f6259e470225c9e` +found a source-inspection sensitivity gap, not a demonstrated production +defect. The direct oracle had allowed either source-key order for two +equal-instant peers. It now admits those peers in reverse key order through +both `add` and `build` and requires the direct `takeFromStart(2)` result to +follow the index interface's key order in BasicIndex and BTreeIndex. The +public limit-one tie remains permissive. This addition covers only the +named equal-instant fixture and direct index checkpoint. + +The maintainer confirmed the custom comparator contract: when a callback +distinguishes two `ZonedDateTime` rows that native `.equals()` groups +together, ordered index operations follow the callback for each row. The +existing oracle already checks that decision through add/build, exact +lookup, range and cursor reads, removal, and update for UTC/Etc-UTC peers. +This round retains those witnesses; it does not collapse their positions or +reject that comparator/value combination. The working-tree 497-test focused +campaign and 12,345-test DB campaign passed. A fresh exact-commit hostile +review of these additions is still pending. diff --git a/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md b/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md index 833bea4954..a4e40399b0 100644 --- a/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md +++ b/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md @@ -95,3 +95,12 @@ recorded properties, including selector 34 with seed `20260911` and path public change events, cross-realm values, RegExp subclasses, or arbitrary cross-type pairs. Ordinal selectors can shift if future registrations are inserted ahead of them. + +## Frozen-head ledger recheck + +At `210651d207eefe9bbc025c780f6259e470225c9e`, the hostile reviewer +read the six retired Date/RegExp assertions, their source, the direct +`deepEquals` implementation, and the receiving oracle. Source inspection +found no missing operand or expected result in the bounded transfer. This +is a frozen-head no-loss verdict by inspection; it is not a new mutant kill +or a fresh committed-head execution of subsequent working-tree changes. diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index 21d46a9eb2..54501c0887 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -326,26 +326,40 @@ const durationFields = [ `nanoseconds`, ] as const -/** Read a Temporal brand and display through its prototype, not own shadows. */ +/** Read a Temporal brand and display through its slot-checking prototype. */ type BrandedTemporal = { prototype: object; kind: string; text: string } function brandedTemporal(value: unknown): BrandedTemporal | undefined { if (value === null || typeof value !== `object`) return undefined - const prototype = Object.getPrototypeOf(value) + let prototype: object | null = Object.getPrototypeOf(value) if (!prototype || !isTemporal(prototype)) return undefined - const toString = Object.getOwnPropertyDescriptor(prototype, `toString`)?.value - if (typeof toString !== `function`) return undefined - try { - // The native/polyfill method checks internal slots. An own forged tag or - // borrowed Temporal prototype cannot claim an ordered position. Arbitrary - // custom prototypes that spoof these methods remain outside this guard. - const text = Reflect.apply(toString, value, []) - return typeof text === `string` - ? { prototype, kind: prototype[Symbol.toStringTag], text } - : undefined - } catch { - return undefined + while (prototype) { + const kind = Object.getOwnPropertyDescriptor( + prototype, + Symbol.toStringTag, + )?.value + const toString = Object.getOwnPropertyDescriptor( + prototype, + `toString`, + )?.value + if ( + typeof kind === `string` && + isTemporal(prototype) && + typeof toString === `function` + ) { + try { + // A genuine subclass inherits this method from the Temporal prototype. + // Its receiver check rejects forged tags and borrowed prototypes. + // A custom prototype supplying both members remains outside this check. + const text = Reflect.apply(toString, value, []) + return typeof text === `string` ? { prototype, kind, text } : undefined + } catch { + return undefined + } + } + prototype = Object.getPrototypeOf(prototype) } + return undefined } /** @@ -484,9 +498,11 @@ export function normalizeForBTree(value: any): any { return normalizeValue(value) } -/** Keep Temporal's native order separate from its exact index lookup key. */ +/** Keep native Date and Temporal order separate from exact lookup keys. */ export function indexOrderValue(value: any, exactKey: any): any { - return brandedTemporal(value) ? value : exactKey + return dateTimestamp(value) !== undefined || brandedTemporal(value) + ? value + : exactKey } /** Supply custom comparators the Temporal display key used before raw ordering. */ @@ -499,7 +515,9 @@ export function indexCustomComparatorValue(value: unknown): unknown { /** Recover the exact lookup key from a stored ordered value. */ export function indexExactKey(value: any): any { - return brandedTemporal(value) ? normalizeValue(value) : value + return dateTimestamp(value) !== undefined || brandedTemporal(value) + ? normalizeValue(value) + : value } /** @@ -573,8 +591,9 @@ function compareBrandedTemporalValues( * * Temporal types intentionally throw from `valueOf` to prevent silent * miscomparison via the native relational operators — delegate to - * `compareTemporalValues` for them. For everything else (numbers, strings, - * Dates via `valueOf`, etc.) the native operators do the right thing. + * `compareTemporalValues` for them. Native Dates compare by their intrinsic + * timestamps, including against numbers and when a subclass overrides + * `valueOf`. Other values retain the native relational comparison. * * Callers must handle null/undefined themselves — this helper assumes both * arguments are non-null. @@ -585,7 +604,13 @@ export function compareValues(a: unknown, b: unknown): number { if (aTemporal && bTemporal) { return compareBrandedTemporalValues(a, b, aTemporal, bTemporal) } - return (a as any) < (b as any) ? -1 : (a as any) > (b as any) ? 1 : 0 + const comparableA = dateTimestamp(a) ?? a + const comparableB = dateTimestamp(b) ?? b + return (comparableA as any) < (comparableB as any) + ? -1 + : (comparableA as any) > (comparableB as any) + ? 1 + : 0 } /** diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index a268aa8640..40db514881 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -1077,10 +1077,16 @@ describe(`native Date order across JavaScript realms`, () => { `class D extends Date {}; new D(timestamp)`, { timestamp }, ) as Date + const localInvalid = new LocalDate(Number.NaN) const foreignInvalid = runInNewContext( `class D extends Date {}; new D(Number.NaN)`, ) as Date - for (const value of [localSubclass, foreignSubclass, foreignInvalid]) { + for (const value of [ + localSubclass, + foreignSubclass, + localInvalid, + foreignInvalid, + ]) { Object.defineProperty(value, Symbol.toStringTag, { value: `Other` }) } // The supplied time and native slot establish the Date premise. The @@ -1150,29 +1156,23 @@ describe(`native Date order across JavaScript realms`, () => { ).toThrowError() expect(reachedWrongOrder).toBe(true) - expect(normalizeValue(foreignInvalid)).toBeNaN() - expect(isUnorderable(foreignInvalid)).toBe(true) - expect( - ascComparator(foreignInvalid, Number.NaN, DEFAULT_COMPARE_OPTIONS), - ).toBe(0) - expect( - sign( - ascComparator( - foreignInvalid, - new Date(timestamp), - DEFAULT_COMPARE_OPTIONS, + for (const invalid of [localInvalid, foreignInvalid]) { + expect(normalizeValue(invalid)).toBeNaN() + expect(isUnorderable(invalid)).toBe(true) + expect(ascComparator(invalid, Number.NaN, DEFAULT_COMPARE_OPTIONS)).toBe( + 0, + ) + expect( + sign( + ascComparator(invalid, new Date(timestamp), DEFAULT_COMPARE_OPTIONS), ), - ), - ).toBe(1) - expect( - sign( - ascComparator( - new Date(timestamp), - foreignInvalid, - DEFAULT_COMPARE_OPTIONS, + ).toBe(1) + expect( + sign( + ascComparator(new Date(timestamp), invalid, DEFAULT_COMPARE_OPTIONS), ), - ), - ).toBe(-1) + ).toBe(-1) + } // Borrowing that prototype and display tag cannot forge native Date // slots. The intrinsic rejects this adjacent non-Date control. @@ -1870,6 +1870,29 @@ describe(`Temporal default-order refinement`, () => { }) }) +describe(`native binary key content`, () => { + it(`distinguishes a changed interior byte at the normalized-key return`, () => { + // Equal lengths and endpoints cannot conceal an interior byte change. + // Native byte content, not the production encoding, defines inequality. + const left = Uint8Array.of(1, 0, 2) + const right = Uint8Array.of(1, 9, 2) + expect(normalizeValue(left)).not.toBe(normalizeValue(right)) + }) + + comparisonProperty([arbitraryChangedBytes])( + `a changed native byte changes the normalized key`, + (sample) => { + // Reuse the independently authored changed-byte history. This + // property has its own assertion site so a shrunk failure identifies + // key normalization rather than direct byte equality. + assertChangedBytes( + sample, + (left, right) => normalizeValue(left) === normalizeValue(right), + ) + }, + ) +}) + if (requestedProperty !== undefined) { it(`selects a registered comparison oracle property`, () => { expect(requestedProperty).toMatch(/^\d+$/) diff --git a/packages/db/tests/custom-index-order-selection-oracle.test.ts b/packages/db/tests/custom-index-order-selection-oracle.test.ts index 16f39789e6..45a92f1b36 100644 --- a/packages/db/tests/custom-index-order-selection-oracle.test.ts +++ b/packages/db/tests/custom-index-order-selection-oracle.test.ts @@ -22,8 +22,11 @@ import { withOracleCleanup } from './utils.js' * must trigger eager creation when only a custom index exists, whichever side * of a single or paired relational predicate holds the field. It covers * BasicIndex and BTreeIndex at the ready checkpoint, with one order term and - * limit one. It does not claim arbitrary custom comparator laws, mutation - * histories, provider ordering, or live-query publication. + * limit one. A neighboring mixed Date/number snapshot compares public scan + * order before index creation with optimized order afterward. It promises + * index installation preserves that order, without prescribing a general + * cross-family order. These checks do not claim arbitrary custom comparator + * laws, mutation histories, provider ordering, or live-query publication. */ type Row = { id: string; rank: number } @@ -38,6 +41,14 @@ const rows: ReadonlyArray = [ const expectedFirstKey = [...rows].sort((a, b) => a.rank - b.rank)[0]!.id const greaterThanTen = new Func(`gt`, [new PropRef([`rank`]), new Value(10)]) const tenLessThanRank = new Func(`lt`, [new Value(10), new PropRef([`rank`])]) +const twentyLessThanOrEqualRank = new Func(`lte`, [ + new Value(20), + new PropRef([`rank`]), +]) +const twentyGreaterThanOrEqualRank = new Func(`gte`, [ + new Value(20), + new PropRef([`rank`]), +]) const expectedGreaterThanTen = rows .filter(({ rank }) => rank > 10) .map(({ id }) => id) @@ -50,6 +61,18 @@ const expectedBetweenTenAndThirty = rows .filter(({ rank }) => 10 < rank && rank < 30) .map(({ id }) => id) .sort() +const expectedAtLeastTwenty = rows + .filter(({ rank }) => 20 <= rank) + .map(({ id }) => id) + .sort() +const expectedAtMostTwenty = rows + .filter(({ rank }) => 20 >= rank) + .map(({ id }) => id) + .sort() +const inclusiveReversedBetweenTwentyAndThirty = new Func(`and`, [ + twentyLessThanOrEqualRank, + new Func(`gte`, [new Value(30), new PropRef([`rank`])]), +]) const betweenTenAndThirty = new Func(`and`, [ greaterThanTen, new Func(`lt`, [new PropRef([`rank`]), new Value(30)]), @@ -97,6 +120,23 @@ function collectionWithRows( }) } +// Construction belongs inside the protected check. Register each Collection +// immediately so a later setup failure still releases the earlier ones. +async function withCreatedCollections( + check: (create: typeof collectionWithRows) => Promise, +): Promise { + const cleanups: Array<() => unknown> = [] + await withOracleCleanup( + () => + check((id, defaultIndexType) => { + const collection = collectionWithRows(id, defaultIndexType) + cleanups.unshift(() => collection.cleanup()) + return collection + }), + cleanups, + ) +} + const indexTypes = [ [`BasicIndex`, BasicIndex], [`BTreeIndex`, BTreeIndex], @@ -106,19 +146,24 @@ describe.each(indexTypes)( `%s custom comparator order selection`, (name, IndexType) => { it(`selects a later compatible index while exact equality keeps the first custom index`, async () => { - const collection = collectionWithRows(`custom-index-coexistence-${name}`) - await withOracleCleanup(async () => { + await withCreatedCollections(async (create) => { + const collection = create(`custom-index-coexistence-${name}`) await collection.stateWhenReady() const custom = collection.createIndex((row) => row.rank, { indexType: IndexType, options: { compareFn: (a: number, b: number) => b - a }, }) - collection.createIndex((row) => row.rank, { indexType: IndexType }) + const ordinary = collection.createIndex((row) => row.rank, { + indexType: IndexType, + }) // Legal declaration order matters to first-match index search. The // custom index is first and still serves exact equality; the ordinary // index is second and must serve the ordered read. const lookup = vi.spyOn(custom, `lookup`) + const ordinaryTake = vi.spyOn(ordinary, `takeFromStart`) + const ordinaryLookup = vi.spyOn(ordinary, `lookup`) + const ordinaryRange = vi.spyOn(ordinary, `rangeQuery`) expect( collection .currentStateAsChanges({ @@ -138,6 +183,7 @@ describe.each(indexTypes)( .currentStateAsChanges({ ...descendingFirst, optimizedOnly: true }) ?.map(({ key }) => key), ).toEqual([`last`]) + expect(ordinaryTake).toHaveBeenCalled() // Ordinary numeric predicates use relational order rather than the // custom index's descending walk. Both single and paired bounds must @@ -151,6 +197,7 @@ describe.each(indexTypes)( ?.map(({ key }) => key) .sort(), ).toEqual([`last`, `middle`]) + expect(ordinaryLookup).toHaveBeenCalledWith(`gt`, 10) expect( collection .currentStateAsChanges({ @@ -159,16 +206,14 @@ describe.each(indexTypes)( }) ?.map(({ key }) => key), ).toEqual([`middle`]) + expect(ordinaryRange).toHaveBeenCalled() expect(lookup).not.toHaveBeenCalledWith(`gt`, 10) - }, [() => collection.cleanup()]) + }) }) it(`creates a compatible eager index after an existing custom index`, async () => { - const collection = collectionWithRows( - `custom-index-eager-${name}`, - IndexType, - ) - await withOracleCleanup(async () => { + await withCreatedCollections(async (create) => { + const collection = create(`custom-index-eager-${name}`, IndexType) await collection.stateWhenReady() collection.createIndex((row) => row.rank, { indexType: IndexType, @@ -189,7 +234,7 @@ describe.each(indexTypes)( optimizedOnly: true, }) expect(collection.indexes.size).toBe(2) - }, [() => collection.cleanup()]) + }) }) it(`creates a compatible eager index for reversed single and paired ranges`, async () => { @@ -209,70 +254,78 @@ describe.each(indexTypes)( where: reversedBetweenTenAndThirty, expectedKeys: expectedBetweenTenAndThirty, }, + { + name: `field-right-inclusive-lower`, + where: twentyLessThanOrEqualRank, + expectedKeys: expectedAtLeastTwenty, + }, + { + name: `field-right-inclusive-upper`, + where: twentyGreaterThanOrEqualRank, + expectedKeys: expectedAtMostTwenty, + }, + { + name: `paired-field-right-inclusive`, + where: inclusiveReversedBetweenTwentyAndThirty, + expectedKeys: expectedAtLeastTwenty, + }, ] as const - const collections = forms.map(({ name: formName }) => - collectionWithRows( - `custom-index-eager-range-${name}-${formName}`, - IndexType, - ), - ) - await withOracleCleanup( - async () => { - await Promise.all( - collections.map((collection) => collection.stateWhenReady()), + await withCreatedCollections(async (create) => { + const collections = forms.map(({ name: formName }) => + create(`custom-index-eager-range-${name}-${formName}`, IndexType), + ) + await Promise.all( + collections.map((collection) => collection.stateWhenReady()), + ) + const observations = [] + for (const [index, form] of forms.entries()) { + const collection = collections[index]! + collection.createIndex((row) => row.rank, { + indexType: IndexType, + options: { compareFn: (a: number, b: number) => b - a }, + }) + const initialKeys: Array = [] + const subscription = collection.subscribeChanges( + (changes) => initialKeys.push(...changes.map(({ key }) => key)), + { whereExpression: form.where, includeInitialState: true }, ) - const observations = [] - for (const [index, form] of forms.entries()) { - const collection = collections[index]! - collection.createIndex((row) => row.rank, { - indexType: IndexType, - options: { compareFn: (a: number, b: number) => b - a }, + try { + observations.push({ + form: form.name, + initialKeys: initialKeys.sort(), + optimizedKeys: collection + .currentStateAsChanges({ + where: form.where, + optimizedOnly: true, + }) + ?.map(({ key }) => key) + .sort(), + indexCount: collection.indexes.size, }) - const initialKeys: Array = [] - const subscription = collection.subscribeChanges( - (changes) => initialKeys.push(...changes.map(({ key }) => key)), - { whereExpression: form.where, includeInitialState: true }, - ) - try { - observations.push({ - form: form.name, - initialKeys: initialKeys.sort(), - optimizedKeys: collection - .currentStateAsChanges({ - where: form.where, - optimizedOnly: true, - }) - ?.map(({ key }) => key) - .sort(), - indexCount: collection.indexes.size, - }) - } finally { - subscription.unsubscribe() - } + } finally { + subscription.unsubscribe() } - // The first two expressions denote the same numeric relation; the - // paired form narrows it. Initial publication and optimized-only - // snapshots must follow those independent row filters after the - // subscription's eager index construction. - expect(observations).toEqual( - forms.map(({ name: formName, expectedKeys }) => ({ - form: formName, - initialKeys: expectedKeys, - optimizedKeys: expectedKeys, - indexCount: 2, - })), - ) - }, - collections.map((collection) => () => collection.cleanup()), - ) + } + // The first two expressions denote the same numeric relation; the + // paired and inclusive forms exercise both bound directions. Initial + // publication and optimized-only snapshots must follow the independent + // row filters after each subscription's eager index construction. + expect(observations).toEqual( + forms.map(({ name: formName, expectedKeys }) => ({ + form: formName, + initialKeys: expectedKeys, + optimizedKeys: expectedKeys, + indexCount: 2, + })), + ) + }) }) it(`keeps a descending custom index out of an ascending public snapshot`, async () => { - const scanned = collectionWithRows(`custom-index-order-scan-${name}`) - const indexed = collectionWithRows(`custom-index-order-indexed-${name}`) - const ordinary = collectionWithRows(`custom-index-order-ordinary-${name}`) - - await withOracleCleanup(async () => { + await withCreatedCollections(async (create) => { + const scanned = create(`custom-index-order-scan-${name}`) + const indexed = create(`custom-index-order-indexed-${name}`) + const ordinary = create(`custom-index-order-ordinary-${name}`) await Promise.all([ scanned.stateWhenReady(), indexed.stateWhenReady(), @@ -362,11 +415,68 @@ describe.each(indexTypes)( optimizedOnly: true, }), ).toBeUndefined() - }, [ - () => indexed.cleanup(), - () => ordinary.cleanup(), - () => scanned.cleanup(), - ]) + }) }) }, ) + +describe.each(indexTypes)(`%s public mixed-value order`, (name, IndexType) => { + it(`keeps a Date and number in the same order after indexing`, async () => { + // The public cross-family order is not prescribed. Installing an index + // must preserve the scan's full order and first-row window for these + // immutable source facts. The index call proves the second read reaches + // the optimized path rather than another scan. + type MixedRow = { id: string; rank: number | Date } + const mixedRows: ReadonlyArray = [ + { id: `date`, rank: new Date(1) }, + { id: `equal-number`, rank: 1 }, + { id: `number`, rank: 2 }, + ] + let cleanup: (() => unknown) | undefined + await withOracleCleanup(async () => { + const collection = createCollection({ + id: `mixed-date-number-order-${name}`, + getKey: ({ id }) => id, + autoIndex: `off`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of mixedRows) write({ type: `insert`, value: row }) + commit() + markReady() + }, + }, + }) + cleanup = () => collection.cleanup() + await collection.stateWhenReady() + const orderBy = [ + { + expression: new PropRef([`rank`]), + compareOptions: DEFAULT_COMPARE_OPTIONS, + }, + ] + const keys = (limit?: number, optimizedOnly?: boolean) => + collection + .currentStateAsChanges({ orderBy, limit, optimizedOnly }) + ?.map(({ key }) => key) + + const scanned = keys() + const scannedFirst = keys(1) + expect(new Set(scanned)).toEqual( + new Set([`date`, `equal-number`, `number`]), + ) + expect(scannedFirst).toEqual(scanned?.slice(0, 1)) + const index = collection.createIndex((row) => row.rank, { + indexType: IndexType, + }) + const take = vi.spyOn(index, `takeFromStart`) + // Query equality puts the Date and its numeric timestamp in one exact + // class, although their public ordered positions are different. + expect(index.equalityLookup(1)).toEqual(new Set([`date`, `equal-number`])) + expect(keys(undefined, true)).toEqual(scanned) + expect(keys(1, true)).toEqual(scannedFirst) + expect(take).toHaveBeenCalled() + }, [() => cleanup?.()]) + }) +}) diff --git a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts index 4d2e46635c..6f75b5c774 100644 --- a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts +++ b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts @@ -20,8 +20,9 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * `src/query/compiler/evaluators.ts`: * * - `NaN` and an invalid Date equal each other and nothing else. - * - A valid Date equals its millisecond timestamp, and another Date with the - * same timestamp. + * - A native Date from either realm equals its millisecond timestamp, and + * another Date with the same timestamp. A borrowed Date prototype does not + * give an ordinary object native Date identity. * - Two Uint8Arrays (Buffers included) are equal when their native bytes are * equal. Own `length` or `byteLength` properties cannot change that result. * - Temporal kinds with an intrinsic `.equals()` use that relation. This @@ -41,7 +42,8 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * Domain: values and items come from a pool that contains the boundaries * of each rule: equal and unequal strings, a string that begins with the * internal normalization prefix, booleans, `0`, `-0`, `1`, `NaN`, `1n`, a - * valid Date and its timestamp, an invalid Date, equal and unequal byte + * local and foreign valid Dates and their timestamps, local and foreign + * invalid Dates, borrowed Date prototypes, equal and unequal byte * arrays, a Buffer, native bytes from another realm with equal and unequal * content, genuine local and foreign bytes with own display-tag shadows, bytes * with own `length` or `byteLength` shadows, and a wide typed array with a @@ -84,6 +86,13 @@ const otherPlainDateTime = Temporal.PlainDateTime.from(`2024-01-01T11:00`) const plainTime = Temporal.PlainTime.from(`10:00`) const samePlainTime = Temporal.PlainTime.from(`10:00`) const otherPlainTime = Temporal.PlainTime.from(`11:00`) +const localDate = new Date(1000) +const foreignDate = runInNewContext(`new Date(1000)`) as Date +const laterForeignDate = runInNewContext(`new Date(1001)`) as Date +const localInvalidDate = new Date(Number.NaN) +const foreignInvalidDate = runInNewContext(`new Date(NaN)`) as Date +const borrowedDate = Object.create(Date.prototype) as object +const otherBorrowedDate = Object.create(Date.prototype) as object const shadowedTagBytes = new Uint8Array([1, 2]) Object.defineProperty(shadowedTagBytes, Symbol.toStringTag, { value: `ordinary`, @@ -205,10 +214,15 @@ const pool: ReadonlyArray = [ 1, Number.NaN, 1n, + localDate, new Date(1000), - new Date(1000), + foreignDate, + laterForeignDate, 1000, - new Date(Number.NaN), + localInvalidDate, + foreignInvalidDate, + borrowedDate, + otherBorrowedDate, new Uint8Array([1, 2]), new Uint8Array([1, 2]), new Uint8Array([1, 3]), @@ -282,6 +296,20 @@ const typedArrayLength = Object.getOwnPropertyDescriptor( )!.get! const typedArrayValues = Uint8Array.prototype.values +// Date's native method reads the internal timestamp in either realm and +// rejects an object that only borrows the prototype. The reference captures +// that intrinsic directly; it does not import the production Date classifier. +const nativeDateGetTime = Date.prototype.getTime + +function nativeDateTimestamp(value: unknown): number | undefined { + if (typeof value !== `object` || value === null) return undefined + try { + return Reflect.apply(nativeDateGetTime, value, []) + } catch { + return undefined + } +} + function isBytes(value: unknown): value is Uint8Array { return ( ArrayBuffer.isView(value) && @@ -351,12 +379,12 @@ function temporalBrand(value: unknown): TemporalBrand | undefined { function isNotANumber(value: unknown): boolean { return ( (typeof value === `number` && Number.isNaN(value)) || - (value instanceof Date && Number.isNaN(value.getTime())) + Number.isNaN(nativeDateTimestamp(value)) ) } function asComparable(value: unknown): unknown { - return value instanceof Date ? value.getTime() : value + return nativeDateTimestamp(value) ?? value } function referenceEqual(left: unknown, right: unknown): boolean { @@ -506,12 +534,15 @@ function describeCase(testCase: InCase): string { if (value === shadowedTextPlainDate) return `PlainDate with shadowed text` if (value === shadowedHourDuration) return `Duration with shadowed text` if (value === forgedHourDuration) return `forged Duration tag` + if (value === borrowedDate) return `borrowed Date prototype` + if (value === otherBorrowedDate) return `other borrowed Date prototype` + const dateTime = nativeDateTimestamp(value) return typeof value === `bigint` ? `${value}n` : Object.is(value, -0) ? `-0` - : value instanceof Date - ? `Date(${value.getTime()})` + : dateTime !== undefined + ? `Date(${dateTime})` : isBytes(value) ? `bytes[${[...value].join(`,`)}]` : value === shared @@ -592,6 +623,26 @@ describe(`in evaluator`, () => { checkEqualityPaths(foreignBytes, spoofedWideBytes, false) }) + it(`compares foreign native Dates by timestamp and excludes borrowed prototypes in every compiled equality path`, () => { + expect(foreignDate instanceof Date).toBe(false) + expect(foreignInvalidDate instanceof Date).toBe(false) + expect(borrowedDate instanceof Date).toBe(true) + expect(nativeDateTimestamp(foreignDate)).toBe(1000) + expect(Number.isNaN(nativeDateTimestamp(foreignInvalidDate))).toBe(true) + expect(nativeDateTimestamp(borrowedDate)).toBeUndefined() + checkEqualityPaths(foreignDate, localDate, true) + checkEqualityPaths(foreignDate, 1000, true) + checkEqualityPaths(laterForeignDate, localDate, false) + checkEqualityPaths(laterForeignDate, 1000, false) + checkEqualityPaths(foreignInvalidDate, localInvalidDate, true) + checkEqualityPaths(foreignInvalidDate, Number.NaN, true) + checkEqualityPaths(foreignInvalidDate, localDate, false) + checkEqualityPaths(borrowedDate, localDate, false) + checkEqualityPaths(borrowedDate, 1000, false) + checkEqualityPaths(borrowedDate, borrowedDate, true) + checkEqualityPaths(borrowedDate, otherBorrowedDate, false) + }) + it(`ignores own byte-count shadows in every compiled equality path`, () => { // Both shadowed arrays still hold two native bytes. Pairing two zero // shadows catches a comparison that stops before reading their content; diff --git a/packages/db/tests/query/index-path-collision-oracle.test.ts b/packages/db/tests/query/index-path-collision-oracle.test.ts index bb95e85a64..fa5617b32d 100644 --- a/packages/db/tests/query/index-path-collision-oracle.test.ts +++ b/packages/db/tests/query/index-path-collision-oracle.test.ts @@ -25,6 +25,12 @@ * Both mixed domains require scan fallback after installation. * A foreign Date subclass is the adjacent valid-domain control: its native * timestamp keeps the range eligible for index lookup across realms. + * A further valid-Date history overrides `valueOf` in local and foreign + * subclasses while retaining native `getTime`. Its four relational predicates + * use intrinsic timestamps in the model, then compare public scan and indexed + * snapshots. Both index types must remain eligible for optimized-only reads. + * Date/number scan comparisons in both operand orders use the same model; + * mixed-domain range indexes make no eligibility claim here. * * Production driver: `currentStateAsChanges`, Collection subscriptions, and * live-query Collections run the real predicates and callback compilers. @@ -41,15 +47,22 @@ * Known omissions: This bounded owner does not claim arbitrary path segments, * nullish values, custom collation, or incremental publication histories. * Lazy demand target deduplication has a separate compiler-boundary witness. - * The Date-domain history does not cover arbitrary coercion methods, invalid - * Dates, or publication after later writes. + * The Date-domain histories cover a `valueOf` override but not other coercion + * methods, invalid Dates, or publication after later writes. */ import { runInNewContext } from 'node:vm' import { expect, it, vi } from 'vitest' import { createCollection } from '../../src/collection/index.js' import { BasicIndex } from '../../src/indexes/basic-index.js' import { BTreeIndex } from '../../src/indexes/btree-index.js' -import { add, and, gt, lt, lte } from '../../src/query/builder/functions.js' +import { + add, + and, + gt, + gte, + lt, + lte, +} from '../../src/query/builder/functions.js' import { PropRef } from '../../src/query/ir.js' import { createLiveQueryCollection } from '../../src/query/live-query-collection.js' import { mockSyncCollectionOptions } from '../utils.js' @@ -342,6 +355,148 @@ it.each([ }, ) +it.each([ + { name: `BasicIndex`, indexType: BasicIndex }, + { name: `BTreeIndex`, indexType: BTreeIndex }, +])( + `compares native Date subclasses by timestamp across scan and indexed ranges: $name`, + async ({ name, indexType }) => { + await checkWithCleanup(async (register) => { + class LaterDate extends Date { + override valueOf() { + return -100 + } + } + const earlierForeign = runInNewContext( + `new (class extends Date { valueOf() { return 100 } })(0)`, + ) as Date + const dateRows = [ + { id: `plain-before`, v: new Date(-1) }, + { id: `foreign-earlier`, v: earlierForeign }, + { id: `equal`, v: new Date(1) }, + { id: `local-later`, v: new LaterDate(2) }, + { id: `plain-after`, v: new Date(3) }, + ] + const bound = new Date(1) + const nativeGetTime = Date.prototype.getTime + const boundTime = nativeGetTime.call(bound) + const field = new PropRef([`v`]) + const predicates = [ + { + name: `gt`, + where: gt(field, bound), + accepts: (time: number) => time > boundTime, + }, + { + name: `gte`, + where: gte(field, bound), + accepts: (time: number) => time >= boundTime, + }, + { + name: `lt`, + where: lt(field, bound), + accepts: (time: number) => time < boundTime, + }, + { + name: `lte`, + where: lte(field, bound), + accepts: (time: number) => time <= boundTime, + }, + ] + const mixedPredicates = [ + { + name: `Date greater than number`, + where: gt(field, 1), + accepts: (time: number) => time > 1, + }, + { + name: `number less than Date`, + where: lt(1, field), + accepts: (time: number) => 1 < time, + }, + { + name: `Date less than number`, + where: lt(field, 1), + accepts: (time: number) => time < 1, + }, + { + name: `number greater than Date`, + where: gt(1, field), + accepts: (time: number) => 1 > time, + }, + ] + const collection = createCollection( + mockSyncCollectionOptions({ + id: `index-valueof-date-${name}`, + getKey: (row: (typeof dateRows)[number]) => row.id, + initialData: dateRows, + autoIndex: `off`, + }), + ) + register(() => collection.cleanup()) + await collection.stateWhenReady() + + // The model calls the native intrinsic, which ignores subclass valueOf. + // The two overrides deliberately reverse their apparent native-operator + // positions, while ordinary Dates control both sides and equality. + expect(Number(earlierForeign)).toBe(100) + expect(Number(dateRows[3]!.v)).toBe(-100) + const expectedDateKeys = (accepts: (time: number) => boolean) => + dateRows + .filter(({ v }) => accepts(nativeGetTime.call(v))) + .map(({ id }) => id) + .sort() + const snapshotKeys = ( + where: BasicExpression, + optimizedOnly = false, + ) => + collection + .currentStateAsChanges({ where, optimizedOnly }) + ?.map((change) => String(change.key)) + .sort() + + // A ready Collection's public snapshots are the comparison cuts. Scan + // first, then install one index and require both the ordinary and + // optimized-only path to refine the same timestamp relation. + for (const predicate of predicates) { + expect(snapshotKeys(predicate.where), `scan ${predicate.name}`).toEqual( + expectedDateKeys(predicate.accepts), + ) + expect( + snapshotKeys(predicate.where, true), + `unindexed optimized-only ${predicate.name}`, + ).toBeUndefined() + } + // Equality already normalizes native Dates to numbers. Relational scan + // results must use the same timestamp when either operand is a number. + for (const predicate of mixedPredicates) { + expect(snapshotKeys(predicate.where), `scan ${predicate.name}`).toEqual( + expectedDateKeys(predicate.accepts), + ) + } + const index = collection.createIndex((row) => row.v, { indexType }) + const lookup = vi.spyOn(index, `lookup`) + for (const predicate of predicates) { + expect( + snapshotKeys(predicate.where), + `indexed ${predicate.name}`, + ).toEqual(expectedDateKeys(predicate.accepts)) + expect(lookup, `indexed ${predicate.name} reached`).toHaveBeenCalled() + lookup.mockClear() + expect( + snapshotKeys(predicate.where, true), + `optimized-only ${predicate.name}`, + ).toEqual(expectedDateKeys(predicate.accepts)) + expect( + lookup, + `optimized-only ${predicate.name} reached`, + ).toHaveBeenCalled() + lookup.mockClear() + } + }) + }, +) + it.each([false, true])( `preserves distinct fields in a Collection subscription callback (indexed=%s)`, async (indexed) => { diff --git a/packages/db/tests/query/subset-dedupe.test.ts b/packages/db/tests/query/subset-dedupe.test.ts index 911d7cc030..10168b35e4 100644 --- a/packages/db/tests/query/subset-dedupe.test.ts +++ b/packages/db/tests/query/subset-dedupe.test.ts @@ -411,6 +411,14 @@ describe(`DeduplicatedLoadSubset`, () => { const foreign: unknown = runInNewContext( type === `Date` ? `new Date(2)` : `new Uint8Array([1, 2])`, ) + expect(foreign).not.toBe(local) + if (type === `Uint8Array`) { + // A genuine foreign view has the same native bytes despite having + // a different reference and failing this realm's instanceof check. + expect(foreign instanceof Uint8Array).toBe(false) + expect(ArrayBuffer.isView(foreign)).toBe(true) + expect(Array.from(foreign as Uint8Array)).toEqual([1, 2]) + } const value = realm === `local` ? local : foreign for (const where of [ eq(ref(`value`), val(value)), @@ -423,13 +431,7 @@ describe(`DeduplicatedLoadSubset`, () => { const matches = compileSingleRowExpression(request.where!) expect( [foreign, local].map((item) => matches({ value: item })), - ).toEqual( - type === `Date` - ? [true, true] - : realm === `foreign` - ? [true, false] - : [false, true], - ) + ).toEqual([true, true]) } }) }, diff --git a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts index 1a63c0871f..acf83abb40 100644 --- a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts +++ b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts @@ -599,7 +599,7 @@ const pinnedChangeHistories: ReadonlyArray = [ // Fixed subclass rows and literals exercise inherited getTime on both sides of // eq; the forged row controls against trusting only the borrowed prototype. // Number literals test the Date/number and invalid-Date/NaN equality classes. -// A later sync transaction moves three rows between those classes. +// A later sync transaction moves four rows between those classes. const taggedDateRows: ReadonlyArray = [ { id: `local-one`, tag: { kind: `date`, realm: `local`, timestamp: 1 } }, { id: `foreign-one`, tag: { kind: `date`, realm: `foreign`, timestamp: 1 } }, @@ -1479,6 +1479,7 @@ describe(`WHERE predicate publication oracle`, () => { let collectionCleanup: (() => unknown) | undefined let notDateCleanup: (() => unknown) | undefined let sameObjectCleanup: (() => unknown) | undefined + let invalidDateCleanup: (() => unknown) | undefined await withOracleCleanup(async () => { const collection = createCollection( mockSyncCollectionOptions<{ id: string; v: object }>({ @@ -1499,15 +1500,25 @@ describe(`WHERE predicate publication oracle`, () => { q.from({ row: collection }).where(({ row }) => eq(row.v, forged)), ) sameObjectCleanup = () => sameObject.cleanup() + const invalidDate = createLiveQueryCollection((q) => + q + .from({ row: collection }) + .where(({ row }) => eq(row.v, new Date(Number.NaN))), + ) + invalidDateCleanup = () => invalidDate.cleanup() await notDate.preload() // A distinct Date is unequal to this object. The same object remains - // equal by identity at the public query publication checkpoint. + // equal by identity at the public query publication checkpoint. It + // also cannot join the invalid-Date/NaN equality class. expect([...notDate.keys()]).toEqual([`forged`]) await sameObject.preload() expect([...sameObject.keys()]).toEqual([`forged`]) + await invalidDate.preload() + expect([...invalidDate.keys()]).toEqual([]) }, [ () => notDateCleanup?.(), () => sameObjectCleanup?.(), + () => invalidDateCleanup?.(), () => collectionCleanup?.(), ]) }) diff --git a/packages/db/tests/temporal-equality-alias-oracle.test.ts b/packages/db/tests/temporal-equality-alias-oracle.test.ts index 89b1f1c99d..324440950d 100644 --- a/packages/db/tests/temporal-equality-alias-oracle.test.ts +++ b/packages/db/tests/temporal-equality-alias-oracle.test.ts @@ -13,7 +13,8 @@ import { withOracleCleanup } from './utils.js' * the fixed-offset +00:00 zone remains a different identity. * * The ZonedDateTime lane covers values from one polyfill copy and one instant, - * including genuine instances with own display and intrinsic-getter shadows. A neighboring + * including a genuine subclass with inherited Temporal methods and instances + * with own display and intrinsic-getter shadows. A neighboring * PlainDate lane checks that native brand and equality survive those shadows * while forged tags retain ordinary object identity. Fixture-assigned classes, * independent of production normalization, predict exact key sets. Direct @@ -33,7 +34,8 @@ import { withOracleCleanup } from './utils.js' * Indexed PlainMonthDay equality remains open. YearMonth and Duration store * several classes with identical default display and check indexed add, * build, removal, re-add, and public reads. - * This oracle does not claim all IANA aliases, arbitrary spoofed prototypes, + * This oracle does not claim all IANA aliases, subclasses with replaced + * Temporal methods, arbitrary spoofed prototypes, * mixed Temporal implementations, joins, live-query publication, or provider * behavior. */ @@ -43,8 +45,11 @@ type Sample = { text: string identity: `utc` | `offset` | `later` | `calendar` ownShadow?: `tag` | `text` | `epoch` | `calendar` + subclass?: true } +class ZonedDateTimeSubclass extends Temporal.ZonedDateTime {} + const samples: ReadonlyArray = [ { id: `utc`, @@ -95,11 +100,24 @@ const samples: ReadonlyArray = [ identity: `utc`, ownShadow: `calendar`, }, + { + id: `utc-subclass`, + text: `2024-01-15T09:00:00+00:00[UTC]`, + identity: `utc`, + subclass: true, + }, ] function rows(): Array { - return samples.map(({ id, text, ownShadow }) => { - const when = Temporal.ZonedDateTime.from(text) + return samples.map(({ id, text, ownShadow, subclass }) => { + const base = Temporal.ZonedDateTime.from(text) + const when = subclass + ? new ZonedDateTimeSubclass( + base.epochNanoseconds, + base.timeZoneId, + base.calendarId, + ) + : base if (ownShadow === `tag`) { Object.defineProperty(when, Symbol.toStringTag, { value: `ordinary` }) } else if (ownShadow === `text`) { @@ -126,6 +144,10 @@ function expectedKeys(identity: Sample[`identity`]): Set { const aliasProbe = Temporal.ZonedDateTime.from( `2024-01-15T09:00:00+00:00[Etc/UTC]`, ) +const subclassProbe = new ZonedDateTimeSubclass( + aliasProbe.epochNanoseconds, + `UTC`, +) const offsetProbe = Temporal.ZonedDateTime.from( `2024-01-15T09:00:00+00:00[+00:00]`, ) @@ -265,6 +287,7 @@ it(`distinguishes equivalent aliases and own property shadows from a fixed-offse shadowedText, shadowedEpoch, shadowedCalendar, + subclass, ] = rows() // This checks the fixture against the external Temporal relation. Expected // index and query sets still come from the assigned identity classes. @@ -296,6 +319,20 @@ it(`distinguishes equivalent aliases and own property shadows from a fixed-offse alias!.when, ), ).toBe(true) + // A subclass inherits Temporal's slot-checking method. The fixture assigns + // it the UTC class independently of the production classifier. + expect(Object.getPrototypeOf(subclass!.when)).toBe( + ZonedDateTimeSubclass.prototype, + ) + expect( + Temporal.ZonedDateTime.prototype.equals.call(subclass!.when, alias!.when), + ).toBe(true) + expect( + Temporal.ZonedDateTime.prototype.equals.call(subclass!.when, fixed!.when), + ).toBe(false) + expect( + Temporal.ZonedDateTime.prototype.equals.call(subclassProbe, alias!.when), + ).toBe(true) expect(utc!.when.toString()).not.toBe(alias!.when.toString()) expect(expectedKeys(`utc`)).toEqual( new Set([ @@ -305,6 +342,7 @@ it(`distinguishes equivalent aliases and own property shadows from a fixed-offse `utc-shadowed-text`, `utc-shadowed-epoch`, `utc-shadowed-calendar`, + `utc-subclass`, ]), ) expect(expectedKeys(`offset`)).toEqual(new Set([`fixed-offset`])) @@ -312,7 +350,7 @@ it(`distinguishes equivalent aliases and own property shadows from a fixed-offse describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { it.each([`add`, `build`] as const)( - `finds the exact equality class after %s and one removal`, + `finds the exact equality class after %s and representative removals`, (construction) => { const source = rows() const index = new IndexType(1, new PropRef([`when`])) @@ -330,9 +368,12 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { // The query value is a fresh instance. A display-string key would omit // UTC; an instant-only key would also include fixed-offset. expect(index.equalityLookup(aliasProbe)).toEqual(expectedKeys(`utc`)) + expect(index.equalityLookup(subclassProbe)).toEqual(expectedKeys(`utc`)) expect(index.lookup(`eq`, aliasProbe)).toEqual(expectedKeys(`utc`)) + expect(index.lookup(`eq`, subclassProbe)).toEqual(expectedKeys(`utc`)) expect(index.inArrayLookup([aliasProbe])).toEqual(expectedKeys(`utc`)) expect(index.equalityLookup(offsetProbe)).toEqual(expectedKeys(`offset`)) + // The neighboring instant and calendar classes ensure that a repair // does not collapse every Temporal value sharing a named zone. expect(index.equalityLookup(source[3]!.when)).toEqual( @@ -344,6 +385,20 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { index.remove(source[0]!.id, source[0]!) expect(index.equalityLookup(aliasProbe)).toEqual( + new Set([ + `utc-alias`, + `utc-shadowed-tag`, + `utc-shadowed-text`, + `utc-shadowed-epoch`, + `utc-shadowed-calendar`, + `utc-subclass`, + ]), + ) + // Removing the subclass itself must retract one member of the class. + // Its native-equal probe still finds every remaining UTC alias. + const subclassRow = source.find((row) => row.id === `utc-subclass`)! + index.remove(subclassRow.id, subclassRow) + expect(index.equalityLookup(subclassProbe)).toEqual( new Set([ `utc-alias`, `utc-shadowed-tag`, @@ -374,16 +429,17 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { }, }, }) - const predicate = new Func(`eq`, [ - new PropRef([`when`]), - new Value(aliasProbe), - ]) - const readKeys = () => - new Set( + const readKeys = (probe: Temporal.ZonedDateTime) => { + const predicate = new Func(`eq`, [ + new PropRef([`when`]), + new Value(probe), + ]) + return new Set( collection .currentStateAsChanges({ where: predicate, optimizedOnly }) ?.map(({ value }) => value.id), ) + } await withOracleCleanup(async () => { await collection.stateWhenReady() @@ -396,9 +452,12 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { `lookup`, ) : undefined - const result = readKeys() - if (lookup) expect(lookup).toHaveBeenCalled() - expect(result).toEqual(expectedKeys(`utc`)) + for (const probe of [aliasProbe, subclassProbe]) { + lookup?.mockClear() + const result = readKeys(probe) + if (lookup) expect(lookup).toHaveBeenCalled() + expect(result).toEqual(expectedKeys(`utc`)) + } }, [() => collection.cleanup()]) }, ) @@ -587,15 +646,27 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () const freshOffsetProbe = Temporal.ZonedDateTime.from( `2024-01-15T09:00:00+00:00[+00:00]`, ) + const borrowedSubclass = Object.assign( + Object.create(ZonedDateTimeSubclass.prototype) as object, + { toString: () => `unrelated borrowed object` }, + ) + const borrowedSubclassCopy = Object.assign( + Object.create(ZonedDateTimeSubclass.prototype) as object, + { toString: () => `unrelated borrowed object` }, + ) Object.defineProperty(shadowed, `timeZoneId`, { value: `UTC` }) expect(aliasProbe.equals(shadowed)).toBe(false) expect(shadowed).not.toBe(freshOffsetProbe) expect(shadowed.equals(freshOffsetProbe)).toBe(true) expect(() => Reflect.apply(aliasProbe.equals, aliasProbe, [fake])).toThrow() + expect(() => + Temporal.ZonedDateTime.prototype.equals.call(borrowedSubclass, aliasProbe), + ).toThrow() const source = [ { id: `forged`, when: fake }, { id: `shadowed`, when: shadowed }, + { id: `borrowed-subclass`, when: borrowedSubclass }, ] const fakeCopy = { ...fake } for (const [, IndexType] of indexTypes) { @@ -610,6 +681,8 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () expect(index.equalityLookup(row.when)).toEqual(new Set([row.id])) if (row.id === `forged`) { expect(index.equalityLookup(fakeCopy)).toEqual(new Set()) + } else if (row.id === `borrowed-subclass`) { + expect(index.equalityLookup(borrowedSubclassCopy)).toEqual(new Set()) } else { // The own zone accessor must not turn a genuine fixed-offset value // into reference-only equality with its native-equal fresh instance. @@ -651,6 +724,8 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () expect(keysFor(freshOffsetProbe)).toEqual(new Set([`shadowed`])) expect(keysFor(fake)).toEqual(new Set([`forged`])) expect(keysFor(fakeCopy)).toEqual(new Set()) + expect(keysFor(borrowedSubclass)).toEqual(new Set([`borrowed-subclass`])) + expect(keysFor(borrowedSubclassCopy)).toEqual(new Set()) }, [() => collection.cleanup()]) }) diff --git a/packages/db/tests/temporal-index-order-oracle.test.ts b/packages/db/tests/temporal-index-order-oracle.test.ts index ac247931da..5aaa73780e 100644 --- a/packages/db/tests/temporal-index-order-oracle.test.ts +++ b/packages/db/tests/temporal-index-order-oracle.test.ts @@ -15,7 +15,8 @@ import { orderedEntriesArray, withOracleCleanup } from './utils.js' * * This bounded oracle covers one polyfill copy, two opposing offset zones, * one equal-instant pair, a one-nanosecond successor, and negative instants - * around the epoch. Equal-instant peers may appear in either key order. It + * around the epoch. Direct indexes order equal-instant peers by source key; + * a public Collection snapshot may choose either peer at a limit boundary. It * checks direct BasicIndex/BTreeIndex adds, builds, and value cursors, one * removal/insertion/update continuation, then the public ordered Collection * snapshot with limit 1. Direct custom-comparator witnesses preserve the @@ -204,6 +205,24 @@ describe.each(indexTypes)( }, ) + it.each([`add`, `build`] as const)( + `orders equal-instant source keys after reverse %s admission`, + (construction) => { + // The instant model ties these two rows; the direct IndexInterface + // contract orders keys inside a comparator group. Reverse admission + // distinguishes that law from insertion-order traversal. + const [plusFourteen, utc] = rowsFromFixtures() + const reversed = [utc!, plusFourteen!] + const index = new IndexType(1, new PropRef([`when`])) + if (construction === `add`) { + for (const row of reversed) index.add(row.id, row) + } else { + index.build(reversed.map((row): [string, Row] => [row.id, row])) + } + expect(index.takeFromStart(2)).toEqual([`b-plus-fourteen`, `c-utc`]) + }, + ) + it(`retains an equal-instant peer through removal, insertion, and zone update`, () => { const [plusFourteen, utc, nanosecondLater] = rowsFromFixtures() const index = new IndexType(1, new PropRef([`when`])) From 0c236f448157a76cc3c768bf710e3c455ac73cc4 Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 21:08:19 -0600 Subject: [PATCH 20/46] test(db): strengthen oracle review witnesses --- docs/contributing/oracle-coverage.md | 35 ++++-- .../borrowed-date-indexed-range.md | 10 ++ .../comparison-unit-retirement-round-5.md | 12 ++ .../comparison-unit-retirement-round-6.md | 13 ++ .../cross-realm-date-where-publication.md | 8 ++ .../custom-index-order-selection.md | 14 +++ .../oracle-reviews/perf-in-constant-set.md | 12 +- .../oracle-reviews/temporal-equality-alias.md | 25 ++++ .../oracle-reviews/temporal-index-order.md | 25 ++-- packages/db/src/utils/comparison.ts | 1 - .../tests/comparison-oracle.property.test.ts | 69 ++++++++++- ...ustom-index-order-selection-oracle.test.ts | 19 ++- .../in-evaluator-oracle.property.test.ts | 3 + .../query/index-path-collision-oracle.test.ts | 27 ++++- ...dicate-publication-oracle.property.test.ts | 8 +- .../temporal-equality-alias-oracle.test.ts | 112 ++++++++++++++++-- .../tests/temporal-index-order-oracle.test.ts | 47 +++++++- 17 files changed, 392 insertions(+), 48 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index d7f049eea5..ebb9c4e14a 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -472,23 +472,30 @@ through a Date prototype, checks valid and invalid subclass order in both directions against finite Dates, and rejects local and foreign borrowed-prototype impostors. The owner also checks an unindexed public ordered scan with a tagged foreign invalid -Date. Exact `normalizeValue` results are direct utility observations. Indexed +Date. A direct byte-key witness distinguishes two 129-byte native views that +differ only at byte 64; the generated changed-byte grammar spans lengths +1–200 and compares both `areValuesEqual` and `normalizeValue` returns. +Exact `normalizeValue` results are direct utility observations. Indexed order, public Date range predicates, cursor continuation, proxies, deeper subclass chains, replaced Date prototypes or `getTime` methods, and other Date-consuming paths remain outside that witness. -The comparison oracle also owns a bounded [Temporal order transfer](oracle-reviews/comparison-unit-retirement-round-6.md). Supplied ISO calendar fields and epoch-nanosecond offsets predict direct `ascComparator` signs and exact zeros for ordinary `PlainDate` and `ZonedDateTime` values from one polyfill copy. Fixed cases preserve the two retired unit blocks; generated cases vary calendar fields, offset zones, equal instants, and adjacent nanoseconds. Fresh-module histories reject a first-input constructor cache poisoned by a genuine instance or forged tag, including the direct `compareValues` path. Both direct helpers also receive a right-hand PlainDate with a constructor shadow. A genuine ZonedDateTime with an own shadowed display tag retains its instant position through both direct helpers. This lane does not establish public query or index behavior. Custom Temporal subclasses, spoofed or replaced instance prototypes, mixed native/polyfill implementations, other Temporal kinds, and public ordering/filter publication need separate witnesses. Spoofed Temporal prototypes can still cause asymmetric comparisons; the comparison owner needs a dedicated brand and mixed-value witness before that class can be called closed. +The comparison oracle also owns a bounded [Temporal order transfer](oracle-reviews/comparison-unit-retirement-round-6.md). Supplied ISO calendar fields and epoch-nanosecond offsets predict direct `ascComparator` signs and exact zeros for ordinary `PlainDate` and `ZonedDateTime` values from one polyfill copy. Fixed cases preserve the two retired unit blocks; generated cases vary calendar fields, offset zones, equal instants, and adjacent nanoseconds. Fresh-module histories reject a first-input constructor cache poisoned by a genuine instance or forged tag, including the direct `compareValues` path. Both direct helpers also receive a right-hand PlainDate with a constructor shadow. A genuine ZonedDateTime with an own shadowed display tag retains its instant position through both direct helpers. One genuine tagged ZonedDateTime subclass with inherited Temporal methods retains native-equal and native-order-equal results against a UTC peer; an object borrowing its prototype is an ordinary-object control. This lane does not establish public query or index behavior. Other Temporal subclasses, spoofed or replaced instance prototypes, mixed native/polyfill implementations, other Temporal kinds, and public ordering/filter publication need separate witnesses. Spoofed Temporal prototypes can still cause asymmetric comparisons; the comparison owner needs a dedicated brand and mixed-value witness before that class can be called closed. The [Temporal index order oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/temporal-index-order-oracle.test.ts) and its [review record](oracle-reviews/temporal-index-order.md) own direct BasicIndex and BTreeIndex ordered takes after add/build and one mutation continuation, plus an optimized-only public Collection snapshot with an explicit index. Supplied epoch nanoseconds establish order without promising -a key tie break for equal instants; distinct zones retain exact lookup -identities. A focused callback witness preserves the former string operands +a public key tie break for equal instants; direct index takes follow the +established source-key tie rule, and the public snapshot permits either +equal-instant peer. Distinct zones retain exact lookup identities. A focused +callback witness preserves the former string operands for a custom comparator over two distinct pre-epoch values. A second callback witness splits query-equal UTC aliases into per-row comparator positions after add/build and checks direct range, cursors, removal, and same-class update. +An opposite callback puts the UTC peer first even though display-text and +source-key order put the alias first. An add/build witness checks that a genuine value's own shadowed display tag cannot move its default ordered position or the value cursor. Direct removal supplies the exact prior row. Mixed value domains, other @@ -500,8 +507,9 @@ and its [review record](oracle-reviews/temporal-equality-alias.md) own the bounded `ZonedDateTime.equals()` partition for UTC/Etc/UTC aliases, a distinct fixed-offset zone, adjacent nanoseconds, and a different calendar. One native ZonedDateTime subclass with inherited Temporal methods belongs to the UTC -class, while an ordinary object borrowing its prototype retains reference -identity. Direct BasicIndex/BTreeIndex add, build, and removal checks compare +class. A second genuine subclass shadows its prototype's display tag and +remains in that class; ordinary objects borrowing either subclass prototype +retain reference identity. Direct BasicIndex/BTreeIndex add, build, and removal checks compare independent fixture-assigned classes. Unindexed and optimized-only indexed Collection snapshots check public `eq`; the `in` evaluator oracle checks constant and row-supplied alias lists. A plain object with a forged Temporal tag and a @@ -528,6 +536,9 @@ original classifier failed equal foreign/local byte pairs. The `in` property has direct replay selector that runs only its random/replay campaign. Native local and `node:vm` Dates, including invalid Dates and timestamp numbers, reach those three compiled equality paths in both operand directions. +An equal-timestamp Date from a second VM context is a positive compiled +equality control, so the reference cannot accept only the first foreign +prototype. Borrowed Date prototypes remain ordinary reference values. The native-Date reference model uses a captured Date intrinsic rather than local `instanceof`. Own `length` and `byteLength` shadows, including a foreign byte view, and a @@ -574,7 +585,10 @@ field-left and value-left single or paired relational predicates. A custom-only index cannot serve an optimized range read. The fixed ready-state fixture covers BasicIndex and BTreeIndex, one numeric field, and limit one. Positive spies observe the later ordinary index's ordered take and numeric -lookup/range calls. Inclusive value-left boundaries are fixed controls. +lookup/range calls, including a separate reverse take for descending order. +The paired inclusive range selects only rank 20 from ranks 10, 20, and 30, +so either misplaced bound changes the complete public key set. Inclusive +value-left boundaries are fixed controls. Another fixed ready-state fixture compares unindexed and optimized indexed Date/number order, both full and limit one, with an index-take reach check. Mutation histories, arbitrary callbacks, and provider publication remain @@ -600,6 +614,9 @@ including tagged foreign Dates and fixed local/foreign subclasses with an own display tag, plus number 1, NaN, and a foreign borrowed-prototype impostor, it compares public live-query keys, filtered subscriber keys, and direct snapshot keys after preload and a four-row sync transaction. +The generated WHERE histories check direct snapshots at attachment; the fixed +Date matrix also checks them after its commit. Subscriber batches reconstruct +subscriber state after each generated commit. Auto-indexing is off in both modes; one mode adds BasicIndex explicitly and also checks its equality lookup and an optimized-only public snapshot. This proves a public indexed snapshot result but does not prove that the live-query @@ -621,7 +638,9 @@ or the predicate bound. A genuine foreign Date subclass is a positive indexed range control. A later fixed history gives local and foreign native Date subclasses misleading `valueOf` results. Public scan and indexed `gt`, `gte`, `lt`, and `lte` snapshots use their intrinsic timestamps, with an -optimized-only index-path check. Date/number scan predicates run in both +optimized-only index-path check. A native subclass also serves as a bound on +both predicate sides while its `valueOf` reports a misleading number. +Date/number scan predicates run in both operand orders; mixed-domain index eligibility is not claimed. Other coercion methods, invalid Dates, later publication, and proxies remain open there. diff --git a/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md b/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md index 041c01cb89..6fb0617a38 100644 --- a/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md +++ b/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md @@ -60,3 +60,13 @@ four Date-bound operators, and both index types. Invalid Dates, arbitrary coercion methods, later publication histories, provider cursors, and generated subclass histories remain outside the evidence. A fresh committed-head hostile review is pending. + +Review of committed head `d974d4a4f` found that the misleading `valueOf` +subclasses were source values only. The fixed matrix now also uses a native +subclass as the predicate bound on both operand sides, where its override +returns a number opposite the native timestamp. Public scan and indexed +key sets still come from captured native `Date.prototype.getTime` values. +The focused owner passed 18 tests. This reaches the Date-bound compiler path; +it does not establish arbitrary coercion methods, generated subclass +histories, or other hosts. Fresh hostile review of the final committed +follow-up is pending. diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md index b6ab97f0b4..f3283a4eed 100644 --- a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md @@ -106,3 +106,15 @@ property at the direct `normalizeValue` return. No hostile mutant was run for this addition, so the fixed pair and property are sensitivity evidence, not an exact-commit kill. The [Temporal transfer](comparison-unit-retirement-round-6.md) and [alias owner](temporal-equality-alias.md) record their separate domains. + +Review of committed head `d974d4a4f` found that the changed-byte generator +reached only lengths 1–50 while the owner's bounded key-size claim reached +200. The follow-up expands independently supplied changed-byte pairs to +lengths 1–200 for both direct `areValuesEqual` and `normalizeValue` checks. +A fixed 129-byte pair differs only at byte 64. A first-64-byte comparison +fault fails the equality check at the intended direct return; the real +normalizer returns distinct keys. The focused comparison owner passed 117 +tests, and explicit test TypeScript, ESLint, and Prettier checks passed. This +is bounded sensitivity evidence, not a claim that every byte sequence or +all binary consumers were exercised. Final exact-commit hostile review of +the follow-up is pending. diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md index 8e0c5ff923..9fdb2c1644 100644 --- a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md @@ -132,3 +132,16 @@ treated it as an ordinary object. The [Temporal alias owner](temporal-equality-a now supplies the subclass equality and borrowed-prototype controls and records the RED/GREEN production repair. This issue is outside the two retired direct comparison assertions. The custom-prototype question above remains open. + +At committed head `d974d4a4f`, review exposed a second subclass boundary: +a genuine `ZonedDateTime` subclass with a shadowed prototype display tag was +classified as an ordinary object. The direct comparison owner now supplies +the instant and one-nanosecond successor independently, verifies the native +`Temporal.ZonedDateTime.compare` premise, and checks exact equal-instant zeros +and both signs through `ascComparator` and `compareValues`. The adjacent +[alias review](temporal-equality-alias.md#tagged-subclass-follow-up-after-d974d4a4f) +records the direct-index and public-`eq` RED/GREEN repair. This is an adjacent +product defect, not a recovered assertion from the two retired ordinary-value +blocks. Other subclasses, replaced methods, mixed implementations, and +arbitrary prototype chains remain open. A fresh hostile review of the final +committed follow-up is pending. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md index 6bec75218b..5564832f88 100644 --- a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md +++ b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md @@ -166,3 +166,11 @@ The witness establishes this single local borrowed object at an unindexed public checkpoint. It does not cover indexed publication for that object, arbitrary prototype chains, or other host realms. A committed-head hostile review of the working-tree addition remains pending. + +The follow-up review of `d974d4a4f` also found an overstatement in the +WHERE publication oracle's opening prose. Its generated histories compare +direct snapshots when subscribers attach, then reconstruct subscriber and +live-query key sets after each sync commit. The fixed Date matrix compares +direct snapshots both before and after its commit. The prose now names those +separate checkpoints. No behavior or product expectation changed in this +clarification. Final exact-commit hostile review remains pending. diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md index c630efa878..65c13519b8 100644 --- a/docs/contributing/oracle-reviews/custom-index-order-selection.md +++ b/docs/contributing/oracle-reviews/custom-index-order-selection.md @@ -173,3 +173,17 @@ passed 497 tests, and the full `@tanstack/db` run passed 12,345 tests in 281 files, with no reported type errors. These are working-tree runs, not a committed-head review. The witness requires scan/index agreement for this fixture; it does not define a universal order between value families. + +## Follow-up to committed-head review + +Review of `d974d4a4f` found that the earlier paired inclusive range +(`20 <= rank <= 30` over ranks 10, 20, and 30) did not make either bound +exclude a row. The paired fixture now uses an upper bound of 25, so its +independent model expects only rank 20. The ordinary index has separate +positive spies for ascending take, descending reverse take, numeric lookup, +and range traversal; the full mixed Date/number optimized read also proves +its take was called. These additions make a wrong selected index or a skipped +bound observable at the public snapshot and traversal checkpoints. The +focused owner passed. They do not prove arbitrary numeric histories, +providers, or the live-query compiler's index-use premise. A fresh hostile +review of the final committed follow-up is pending. diff --git a/docs/contributing/oracle-reviews/perf-in-constant-set.md b/docs/contributing/oracle-reviews/perf-in-constant-set.md index fc99099167..1a5f1b85d8 100644 --- a/docs/contributing/oracle-reviews/perf-in-constant-set.md +++ b/docs/contributing/oracle-reviews/perf-in-constant-set.md @@ -114,8 +114,8 @@ found a false-green risk in `referenceEqual`: realm-local `instanceof Date` could classify a native foreign Date as an ordinary reference even when the compiled evaluator correctly compared its timestamp. The working-tree model now reads a captured native `Date.prototype.getTime`, uses valid and invalid -foreign Dates, and rejects a local object that only borrows a Date subclass -prototype. Fixed witnesses check compiled `eq`, constant-list `in`, and +foreign Dates, and rejects a local object made with +`Object.create(Date.prototype)`. Fixed witnesses check compiled `eq`, constant-list `in`, and row-supplied `in` in both operand directions. This was a reference-model gap, not an observed production defect. The focused receiving run passed 72 tests. @@ -126,3 +126,11 @@ altering the native byte-content law. The open null-containing-list authority question recorded by the [Temporal alias review](temporal-equality-alias.md) is unchanged. The working-tree changes still need a fresh committed-head hostile review. + +The review of committed head `d974d4a4f` found that the positive foreign-Date +controls came from one VM context, while a second context supplied only a +different timestamp. The receiving oracle now requires that second context's +Date to equal its own timestamp through compiled `eq`, constant-list `in`, and +row-supplied `in`. The focused `in` owner passed after this addition. This +checks the compiled equality return for two VM contexts; browser iframe +behavior and a fresh exact-commit hostile review of the follow-up remain open. diff --git a/docs/contributing/oracle-reviews/temporal-equality-alias.md b/docs/contributing/oracle-reviews/temporal-equality-alias.md index 496b15704b..1aa8324cb2 100644 --- a/docs/contributing/oracle-reviews/temporal-equality-alias.md +++ b/docs/contributing/oracle-reviews/temporal-equality-alias.md @@ -275,3 +275,28 @@ polyfill's genuine subclass with inherited methods. The custom-prototype fake described above, replaced subclass methods, mixed implementations, and provider paths remain open. The working-tree repair still needs a fresh committed-head hostile review. + +## Tagged subclass follow-up after `d974d4a4f` + +The committed-head review found another native-brand hole. A genuine +`ZonedDateTime` subclass whose own prototype shadows `Symbol.toStringTag` +inherits Temporal's slot-checking methods. Native `.equals()` groups it with +a UTC alias, and native `compare` gives it the same instant position. The +classifier stopped at the immediate prototype, treated that genuine value as +an ordinary object, and missed it in direct BasicIndex/BTreeIndex lookups and +public `eq` snapshots. A distinct ordinary object borrowing that subclass +prototype must still retain reference identity. + +The alias owner now adds the tagged subclass to its independent UTC class, +checks direct add, build, lookup, removal, and unindexed and optimized-only +public `eq`, and retains the borrowed-prototype negative controls. The +comparison owner checks exact zeros and both signs through `ascComparator` +and `compareValues`. Ten intended comparisons were RED before the one-line +classifier repair. The repaired focused run passed 144 tests: 28 alias and +116 comparison, with explicit test TypeScript, ESLint, Prettier, and diff +checks passing. The repair removes the immediate-prototype tag gate and +continues to require a slot-checking inherited Temporal method. This evidence +covers one imported polyfill's subclass with inherited methods and one +shadowed prototype tag. Replaced intrinsic methods, other implementations, +arbitrary prototype chains, and provider paths remain open. A fresh hostile +review of the final committed change is pending. diff --git a/docs/contributing/oracle-reviews/temporal-index-order.md b/docs/contributing/oracle-reviews/temporal-index-order.md index e9745d6848..8d158afc7d 100644 --- a/docs/contributing/oracle-reviews/temporal-index-order.md +++ b/docs/contributing/oracle-reviews/temporal-index-order.md @@ -7,8 +7,10 @@ keeps zones distinct. The law comes from Temporal's `ZonedDateTime.compare` and `equals` semantics, changelog #1519, and the Collection's ordered-read and IndexInterface contracts. The live-query architecture also requires an ordered comparator group to keep a representative from a live exact-value -bucket. No public contract fixes the key order of two rows at the same -instant, so the oracle accepts either tie order. +bucket. The direct index's established source-key rule orders two rows at the +same instant. The public ordered snapshot does not promise which +equal-instant peer occupies a limit-one window, so that observation accepts +either peer. ## Original failure and repair @@ -124,16 +126,21 @@ found a source-inspection sensitivity gap, not a demonstrated production defect. The direct oracle had allowed either source-key order for two equal-instant peers. It now admits those peers in reverse key order through both `add` and `build` and requires the direct `takeFromStart(2)` result to -follow the index interface's key order in BasicIndex and BTreeIndex. The +follow the established source-key tie rule in BasicIndex and BTreeIndex. The public limit-one tie remains permissive. This addition covers only the named equal-instant fixture and direct index checkpoint. The maintainer confirmed the custom comparator contract: when a callback distinguishes two `ZonedDateTime` rows that native `.equals()` groups together, ordered index operations follow the callback for each row. The -existing oracle already checks that decision through add/build, exact -lookup, range and cursor reads, removal, and update for UTC/Etc-UTC peers. -This round retains those witnesses; it does not collapse their positions or -reject that comparator/value combination. The working-tree 497-test focused -campaign and 12,345-test DB campaign passed. A fresh exact-commit hostile -review of these additions is still pending. +existing oracle checks that decision through add/build, exact lookup, +range and cursor reads, removal, and update for UTC/Etc-UTC peers. Review of +committed head `d974d4a4f` found that those checks could accept display-text +order in place of callback order. A new opposite callback puts UTC first, +while both display-text and source-key order put the alias first; add/build +takes, range, and cursor results must follow that callback. Its focused run +passed 33 tests. This round does not collapse the peers' positions or reject +that comparator/value combination. The earlier 497-test focused campaign +and 12,345-test DB campaign describe the committed baseline, before this +opposite-callback witness. A fresh exact-commit hostile review of the +follow-up remains pending. diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index 54501c0887..c7840d95c9 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -332,7 +332,6 @@ type BrandedTemporal = { prototype: object; kind: string; text: string } function brandedTemporal(value: unknown): BrandedTemporal | undefined { if (value === null || typeof value !== `object`) return undefined let prototype: object | null = Object.getPrototypeOf(value) - if (!prototype || !isTemporal(prototype)) return undefined while (prototype) { const kind = Object.getOwnPropertyDescriptor( prototype, diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index 40db514881..9fa88c3ac0 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -55,7 +55,9 @@ import type { CompareOptions } from '../src/query/builder/types' * they constrain this internal utility, not a public serialized format or a * universal performance bound. Own `length` and `byteLength` properties do * not change native byte content. The 32-character margin is the existing - * constant-overhead budget for the sampled 0..200-byte keys. + * constant-overhead budget for the sampled 0..200-byte keys. Changed-byte + * equality and key histories cover lengths 1..200; a 129-byte interior change + * pins the first length beyond the 128-byte boundary. */ const defaultOpts: CompareOptions = { @@ -503,10 +505,13 @@ const arbitrarySameTypeTriple = fc.oneof( ), ) +// A legal changed-byte input retains length and all but one indexed byte. +// The 1..200 range spans both sides of the 128-byte key-size boundary; the +// empty input has no byte to change and is checked by the size-margin witness. const arbitraryChangedBytes = fc .tuple( - fc.uint8Array({ minLength: 1, maxLength: 50 }), - fc.integer({ min: 0, max: 49 }), + fc.uint8Array({ minLength: 1, maxLength: 200 }), + fc.integer({ min: 0, max: 199 }), fc.integer({ min: 1, max: 255 }), ) .map(([bytes, indexHint, delta]) => { @@ -1531,7 +1536,9 @@ describe(`areValuesEqual property-based tests`, () => { * ZonedDateTimes within two seconds of a fixed instant in four offset zones. * Equal-value branches construct separate objects, and the zoned branch forces * different zones at the same instant. A fixed genuine ZonedDateTime also - * shadows its own display tag; the supplied instant still determines order. + * shadows its own display tag; a genuine subclass changes its prototype's + * display tag while inheriting Temporal's slot-checking method. The supplied + * instant still determines order for both. * The production paths are the exported * `ascComparator` under the default options and `compareValues` in hostile * dispatch histories. Each return is the observation checkpoint; @@ -1770,6 +1777,38 @@ describe(`Temporal default-order refinement`, () => { expect(sign(ascComparator(utc, later, DEFAULT_COMPARE_OPTIONS))).toBe(-1) }) + it.each([ + [ + `ascComparator`, + (left: unknown, right: unknown) => + ascComparator(left, right, DEFAULT_COMPARE_OPTIONS), + ], + [`compareValues`, compareValues], + ] as const)( + `keeps a tagged ZonedDateTime subclass in instant order through %s`, + (_name, compare) => { + class TaggedZonedDateTime extends Temporal.ZonedDateTime {} + Object.defineProperty(TaggedZonedDateTime.prototype, Symbol.toStringTag, { + value: `Tagged`, + }) + const tagged = new TaggedZonedDateTime( + zonedBaseEpochNanoseconds, + `+00:00`, + ) + const peer = zonedValue(0, `+05:30`) + const later = zonedValue(1, `+00:00`) + + // Model positions come from supplied nanosecond offsets. Native compare + // confirms that the tagged receiver still has genuine Temporal slots. + expect(Temporal.ZonedDateTime.compare(tagged, peer)).toBe(0) + expect(Temporal.ZonedDateTime.compare(tagged, later)).toBe(-1) + expect(compare(tagged, peer)).toBe(0) + expect(compare(peer, tagged)).toBe(0) + expect(sign(compare(tagged, later))).toBe(-1) + expect(sign(compare(later, tagged))).toBe(1) + }, + ) + it(`keeps a PlainDate constructor override from poisoning later comparisons`, async () => { const { ascComparator: isolatedComparator, compareValues: isolatedValues } = await freshTemporalComparison() @@ -1879,6 +1918,28 @@ describe(`native binary key content`, () => { expect(normalizeValue(left)).not.toBe(normalizeValue(right)) }) + it(`distinguishes byte 64 in a 129-byte key`, () => { + const bytes = Uint8Array.from({ length: 129 }, (_, index) => index) + const changed = new Uint8Array(bytes) + changed[64] = (changed[64]! + 1) % 256 + const sample = { bytes, changed, index: 64 } + + // A key that reads only its first 64 bytes accepts this legal change. + // The normalizer must preserve it at the direct-return checkpoint. + expect(() => + assertChangedBytes( + sample, + (left, right) => + left.length === right.length && + left.subarray(0, 64).every((byte, index) => byte === right[index]), + ), + ).toThrow(`changed byte must be unequal`) + assertChangedBytes( + sample, + (left, right) => normalizeValue(left) === normalizeValue(right), + ) + }) + comparisonProperty([arbitraryChangedBytes])( `a changed native byte changes the normalized key`, (sample) => { diff --git a/packages/db/tests/custom-index-order-selection-oracle.test.ts b/packages/db/tests/custom-index-order-selection-oracle.test.ts index 45a92f1b36..9d98e7e6de 100644 --- a/packages/db/tests/custom-index-order-selection-oracle.test.ts +++ b/packages/db/tests/custom-index-order-selection-oracle.test.ts @@ -69,10 +69,14 @@ const expectedAtMostTwenty = rows .filter(({ rank }) => 20 >= rank) .map(({ id }) => id) .sort() -const inclusiveReversedBetweenTwentyAndThirty = new Func(`and`, [ +const inclusiveReversedBetweenTwentyAndTwentyFive = new Func(`and`, [ twentyLessThanOrEqualRank, - new Func(`gte`, [new Value(30), new PropRef([`rank`])]), + new Func(`gte`, [new Value(25), new PropRef([`rank`])]), ]) +const expectedBetweenTwentyAndTwentyFive = rows + .filter(({ rank }) => 20 <= rank && rank <= 25) + .map(({ id }) => id) + .sort() const betweenTenAndThirty = new Func(`and`, [ greaterThanTen, new Func(`lt`, [new PropRef([`rank`]), new Value(30)]), @@ -162,6 +166,7 @@ describe.each(indexTypes)( // index is second and must serve the ordered read. const lookup = vi.spyOn(custom, `lookup`) const ordinaryTake = vi.spyOn(ordinary, `takeFromStart`) + const ordinaryReverseTake = vi.spyOn(ordinary, `takeReversedFromEnd`) const ordinaryLookup = vi.spyOn(ordinary, `lookup`) const ordinaryRange = vi.spyOn(ordinary, `rangeQuery`) expect( @@ -178,12 +183,13 @@ describe.each(indexTypes)( .currentStateAsChanges({ ...ascendingFirst, optimizedOnly: true }) ?.map(({ key }) => key), ).toEqual([expectedFirstKey]) + expect(ordinaryTake).toHaveBeenCalled() expect( collection .currentStateAsChanges({ ...descendingFirst, optimizedOnly: true }) ?.map(({ key }) => key), ).toEqual([`last`]) - expect(ordinaryTake).toHaveBeenCalled() + expect(ordinaryReverseTake).toHaveBeenCalled() // Ordinary numeric predicates use relational order rather than the // custom index's descending walk. Both single and paired bounds must @@ -198,6 +204,7 @@ describe.each(indexTypes)( .sort(), ).toEqual([`last`, `middle`]) expect(ordinaryLookup).toHaveBeenCalledWith(`gt`, 10) + ordinaryRange.mockClear() expect( collection .currentStateAsChanges({ @@ -266,8 +273,8 @@ describe.each(indexTypes)( }, { name: `paired-field-right-inclusive`, - where: inclusiveReversedBetweenTwentyAndThirty, - expectedKeys: expectedAtLeastTwenty, + where: inclusiveReversedBetweenTwentyAndTwentyFive, + expectedKeys: expectedBetweenTwentyAndTwentyFive, }, ] as const await withCreatedCollections(async (create) => { @@ -475,6 +482,8 @@ describe.each(indexTypes)(`%s public mixed-value order`, (name, IndexType) => { // class, although their public ordered positions are different. expect(index.equalityLookup(1)).toEqual(new Set([`date`, `equal-number`])) expect(keys(undefined, true)).toEqual(scanned) + expect(take).toHaveBeenCalled() + take.mockClear() expect(keys(1, true)).toEqual(scannedFirst) expect(take).toHaveBeenCalled() }, [() => cleanup?.()]) diff --git a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts index 6f75b5c774..2ce03d0b18 100644 --- a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts +++ b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts @@ -634,6 +634,9 @@ describe(`in evaluator`, () => { checkEqualityPaths(foreignDate, 1000, true) checkEqualityPaths(laterForeignDate, localDate, false) checkEqualityPaths(laterForeignDate, 1000, false) + // This Date comes from a second vm context. Its own timestamp must also + // match, so a classifier restricted to the first foreign prototype fails. + checkEqualityPaths(laterForeignDate, 1001, true) checkEqualityPaths(foreignInvalidDate, localInvalidDate, true) checkEqualityPaths(foreignInvalidDate, Number.NaN, true) checkEqualityPaths(foreignInvalidDate, localDate, false) diff --git a/packages/db/tests/query/index-path-collision-oracle.test.ts b/packages/db/tests/query/index-path-collision-oracle.test.ts index fa5617b32d..e9e0caf36a 100644 --- a/packages/db/tests/query/index-path-collision-oracle.test.ts +++ b/packages/db/tests/query/index-path-collision-oracle.test.ts @@ -29,8 +29,9 @@ * subclasses while retaining native `getTime`. Its four relational predicates * use intrinsic timestamps in the model, then compare public scan and indexed * snapshots. Both index types must remain eligible for optimized-only reads. - * Date/number scan comparisons in both operand orders use the same model; - * mixed-domain range indexes make no eligibility claim here. + * A subclass bound also reaches both operand positions. Date/number scan + * comparisons in both operand orders use the same model; mixed-domain range + * indexes make no eligibility claim here. * * Production driver: `currentStateAsChanges`, Collection subscriptions, and * live-query Collections run the real predicates and callback compilers. @@ -378,6 +379,7 @@ it.each([ { id: `plain-after`, v: new Date(3) }, ] const bound = new Date(1) + const subclassBound = new LaterDate(1) const nativeGetTime = Date.prototype.getTime const boundTime = nativeGetTime.call(bound) const field = new PropRef([`v`]) @@ -402,6 +404,26 @@ it.each([ where: lte(field, bound), accepts: (time: number) => time <= boundTime, }, + { + name: `gt with subclass bound`, + where: gt(field, subclassBound), + accepts: (time: number) => time > boundTime, + }, + { + name: `lte with subclass bound`, + where: lte(field, subclassBound), + accepts: (time: number) => time <= boundTime, + }, + { + name: `subclass bound less than field`, + where: lt(subclassBound, field), + accepts: (time: number) => boundTime < time, + }, + { + name: `subclass bound greater than or equal to field`, + where: gte(subclassBound, field), + accepts: (time: number) => boundTime >= time, + }, ] const mixedPredicates = [ { @@ -441,6 +463,7 @@ it.each([ // positions, while ordinary Dates control both sides and equality. expect(Number(earlierForeign)).toBe(100) expect(Number(dateRows[3]!.v)).toBe(-100) + expect(Number(subclassBound)).toBe(-100) const expectedDateKeys = (accepts: (time: number) => boolean) => dateRows .filter(({ v }) => accepts(nativeGetTime.call(v))) diff --git a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts index acf83abb40..ea937fe552 100644 --- a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts +++ b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts @@ -56,9 +56,11 @@ * must be routed to exactly the subscribers they can satisfy; and the public * `Collection.currentStateAsChanges` snapshot. * - * Refinement check: each consumer's exact key set equals the model after the - * subscribers attach and after each sync transaction commits. Direct - * subscribers are reconstructed from their callback batches. Two fixed + * Refinement check: live-query and subscriber key sets equal the model after + * attachment and after each sync transaction commits. The generated history + * checks a direct snapshot at attachment. Direct subscribers are reconstructed + * from their callback batches. The fixed Date matrix checks direct snapshots + * before and after its sync transaction. Two fixed * witnesses check boundaries outside the generated grammar: Collection * readiness reaches a filtered subscriber as one empty batch, and an eager * source restarted after cleanup retracts a vanished row even when its first diff --git a/packages/db/tests/temporal-equality-alias-oracle.test.ts b/packages/db/tests/temporal-equality-alias-oracle.test.ts index 324440950d..b5727a5c30 100644 --- a/packages/db/tests/temporal-equality-alias-oracle.test.ts +++ b/packages/db/tests/temporal-equality-alias-oracle.test.ts @@ -13,8 +13,9 @@ import { withOracleCleanup } from './utils.js' * the fixed-offset +00:00 zone remains a different identity. * * The ZonedDateTime lane covers values from one polyfill copy and one instant, - * including a genuine subclass with inherited Temporal methods and instances - * with own display and intrinsic-getter shadows. A neighboring + * including genuine subclasses with inherited Temporal methods. One subclass + * prototype has an unrelated display tag. Other genuine instances have own + * display and intrinsic-getter shadows. A neighboring * PlainDate lane checks that native brand and equality survive those shadows * while forged tags retain ordinary object identity. Fixture-assigned classes, * independent of production normalization, predict exact key sets. Direct @@ -46,9 +47,18 @@ type Sample = { identity: `utc` | `offset` | `later` | `calendar` ownShadow?: `tag` | `text` | `epoch` | `calendar` subclass?: true + taggedSubclass?: true } class ZonedDateTimeSubclass extends Temporal.ZonedDateTime {} +class TaggedZonedDateTimeSubclass extends Temporal.ZonedDateTime {} +Object.defineProperty( + TaggedZonedDateTimeSubclass.prototype, + Symbol.toStringTag, + { + value: `Tagged`, + }, +) const samples: ReadonlyArray = [ { @@ -106,18 +116,24 @@ const samples: ReadonlyArray = [ identity: `utc`, subclass: true, }, + { + id: `utc-tagged-subclass`, + text: `2024-01-15T09:00:00+00:00[UTC]`, + identity: `utc`, + taggedSubclass: true, + }, ] function rows(): Array { - return samples.map(({ id, text, ownShadow, subclass }) => { + return samples.map(({ id, text, ownShadow, subclass, taggedSubclass }) => { const base = Temporal.ZonedDateTime.from(text) - const when = subclass - ? new ZonedDateTimeSubclass( - base.epochNanoseconds, - base.timeZoneId, - base.calendarId, - ) - : base + const Subclass = taggedSubclass + ? TaggedZonedDateTimeSubclass + : ZonedDateTimeSubclass + const when = + subclass || taggedSubclass + ? new Subclass(base.epochNanoseconds, base.timeZoneId, base.calendarId) + : base if (ownShadow === `tag`) { Object.defineProperty(when, Symbol.toStringTag, { value: `ordinary` }) } else if (ownShadow === `text`) { @@ -148,6 +164,10 @@ const subclassProbe = new ZonedDateTimeSubclass( aliasProbe.epochNanoseconds, `UTC`, ) +const taggedSubclassProbe = new TaggedZonedDateTimeSubclass( + aliasProbe.epochNanoseconds, + `UTC`, +) const offsetProbe = Temporal.ZonedDateTime.from( `2024-01-15T09:00:00+00:00[+00:00]`, ) @@ -288,6 +308,7 @@ it(`distinguishes equivalent aliases and own property shadows from a fixed-offse shadowedEpoch, shadowedCalendar, subclass, + taggedSubclass, ] = rows() // This checks the fixture against the external Temporal relation. Expected // index and query sets still come from the assigned identity classes. @@ -333,6 +354,32 @@ it(`distinguishes equivalent aliases and own property shadows from a fixed-offse expect( Temporal.ZonedDateTime.prototype.equals.call(subclassProbe, alias!.when), ).toBe(true) + // The subclass prototype's tag is presentation. The inherited intrinsic + // still sees genuine ZonedDateTime slots and the same assigned UTC class. + expect(Object.getPrototypeOf(taggedSubclass!.when)).toBe( + TaggedZonedDateTimeSubclass.prototype, + ) + expect(Object.getPrototypeOf(taggedSubclass!.when)[Symbol.toStringTag]).toBe( + `Tagged`, + ) + expect( + Object.getOwnPropertyDescriptor( + TaggedZonedDateTimeSubclass.prototype, + `toString`, + ), + ).toBeUndefined() + expect( + Temporal.ZonedDateTime.prototype.equals.call( + taggedSubclass!.when, + alias!.when, + ), + ).toBe(true) + expect( + Temporal.ZonedDateTime.prototype.equals.call( + taggedSubclassProbe, + alias!.when, + ), + ).toBe(true) expect(utc!.when.toString()).not.toBe(alias!.when.toString()) expect(expectedKeys(`utc`)).toEqual( new Set([ @@ -343,6 +390,7 @@ it(`distinguishes equivalent aliases and own property shadows from a fixed-offse `utc-shadowed-epoch`, `utc-shadowed-calendar`, `utc-subclass`, + `utc-tagged-subclass`, ]), ) expect(expectedKeys(`offset`)).toEqual(new Set([`fixed-offset`])) @@ -369,8 +417,14 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { // UTC; an instant-only key would also include fixed-offset. expect(index.equalityLookup(aliasProbe)).toEqual(expectedKeys(`utc`)) expect(index.equalityLookup(subclassProbe)).toEqual(expectedKeys(`utc`)) + expect(index.equalityLookup(taggedSubclassProbe)).toEqual( + expectedKeys(`utc`), + ) expect(index.lookup(`eq`, aliasProbe)).toEqual(expectedKeys(`utc`)) expect(index.lookup(`eq`, subclassProbe)).toEqual(expectedKeys(`utc`)) + expect(index.lookup(`eq`, taggedSubclassProbe)).toEqual( + expectedKeys(`utc`), + ) expect(index.inArrayLookup([aliasProbe])).toEqual(expectedKeys(`utc`)) expect(index.equalityLookup(offsetProbe)).toEqual(expectedKeys(`offset`)) @@ -392,6 +446,7 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { `utc-shadowed-epoch`, `utc-shadowed-calendar`, `utc-subclass`, + `utc-tagged-subclass`, ]), ) // Removing the subclass itself must retract one member of the class. @@ -399,6 +454,18 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { const subclassRow = source.find((row) => row.id === `utc-subclass`)! index.remove(subclassRow.id, subclassRow) expect(index.equalityLookup(subclassProbe)).toEqual( + new Set([ + `utc-alias`, + `utc-shadowed-tag`, + `utc-shadowed-text`, + `utc-shadowed-epoch`, + `utc-shadowed-calendar`, + `utc-tagged-subclass`, + ]), + ) + const taggedRow = source.find((row) => row.id === `utc-tagged-subclass`)! + index.remove(taggedRow.id, taggedRow) + expect(index.equalityLookup(taggedSubclassProbe)).toEqual( new Set([ `utc-alias`, `utc-shadowed-tag`, @@ -452,7 +519,7 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { `lookup`, ) : undefined - for (const probe of [aliasProbe, subclassProbe]) { + for (const probe of [aliasProbe, subclassProbe, taggedSubclassProbe]) { lookup?.mockClear() const result = readKeys(probe) if (lookup) expect(lookup).toHaveBeenCalled() @@ -654,6 +721,14 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () Object.create(ZonedDateTimeSubclass.prototype) as object, { toString: () => `unrelated borrowed object` }, ) + const borrowedTaggedSubclass = Object.assign( + Object.create(TaggedZonedDateTimeSubclass.prototype) as object, + { toString: () => `unrelated borrowed object` }, + ) + const borrowedTaggedSubclassCopy = Object.assign( + Object.create(TaggedZonedDateTimeSubclass.prototype) as object, + { toString: () => `unrelated borrowed object` }, + ) Object.defineProperty(shadowed, `timeZoneId`, { value: `UTC` }) expect(aliasProbe.equals(shadowed)).toBe(false) expect(shadowed).not.toBe(freshOffsetProbe) @@ -662,11 +737,18 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () expect(() => Temporal.ZonedDateTime.prototype.equals.call(borrowedSubclass, aliasProbe), ).toThrow() + expect(() => + Temporal.ZonedDateTime.prototype.equals.call( + borrowedTaggedSubclass, + aliasProbe, + ), + ).toThrow() const source = [ { id: `forged`, when: fake }, { id: `shadowed`, when: shadowed }, { id: `borrowed-subclass`, when: borrowedSubclass }, + { id: `borrowed-tagged-subclass`, when: borrowedTaggedSubclass }, ] const fakeCopy = { ...fake } for (const [, IndexType] of indexTypes) { @@ -683,6 +765,10 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () expect(index.equalityLookup(fakeCopy)).toEqual(new Set()) } else if (row.id === `borrowed-subclass`) { expect(index.equalityLookup(borrowedSubclassCopy)).toEqual(new Set()) + } else if (row.id === `borrowed-tagged-subclass`) { + expect(index.equalityLookup(borrowedTaggedSubclassCopy)).toEqual( + new Set(), + ) } else { // The own zone accessor must not turn a genuine fixed-offset value // into reference-only equality with its native-equal fresh instance. @@ -726,6 +812,10 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () expect(keysFor(fakeCopy)).toEqual(new Set()) expect(keysFor(borrowedSubclass)).toEqual(new Set([`borrowed-subclass`])) expect(keysFor(borrowedSubclassCopy)).toEqual(new Set()) + expect(keysFor(borrowedTaggedSubclass)).toEqual( + new Set([`borrowed-tagged-subclass`]), + ) + expect(keysFor(borrowedTaggedSubclassCopy)).toEqual(new Set()) }, [() => collection.cleanup()]) }) diff --git a/packages/db/tests/temporal-index-order-oracle.test.ts b/packages/db/tests/temporal-index-order-oracle.test.ts index 5aaa73780e..ab51b83181 100644 --- a/packages/db/tests/temporal-index-order-oracle.test.ts +++ b/packages/db/tests/temporal-index-order-oracle.test.ts @@ -10,8 +10,9 @@ import { orderedEntriesArray, withOracleCleanup } from './utils.js' * A ZonedDateTime's ordered position is its instant. Two zones at the same * instant share an ordered position but remain distinct equality values. * Authority: Temporal.ZonedDateTime.compare/equals, the established WHERE - * evaluator contract in query/compiler/evaluators-oracle.test.ts, and the - * ordered Collection and IndexInterface contracts in docs/reference. + * evaluator contract in query/compiler/evaluators-oracle.test.ts, the + * ordered Collection contract, and the established direct key-tie rule in + * deterministic-ordering.test.ts. IndexInterface names the direct methods. * * This bounded oracle covers one polyfill copy, two opposing offset zones, * one equal-instant pair, a one-nanosecond successor, and negative instants @@ -21,7 +22,8 @@ import { orderedEntriesArray, withOracleCleanup } from './utils.js' * removal/insertion/update continuation, then the public ordered Collection * snapshot with limit 1. Direct custom-comparator witnesses preserve the * former display-string operands and let the callback order two rows whose - * values share one query-equality class. They check add/build, exact lookup, + * values share one query-equality class. An opposite callback distinguishes + * callback-result order from plain display-text order. They check add/build, exact lookup, * range and cursor boundaries, representative removal, and an update that * changes callback order without changing query equality. Fixed add/build * histories also shadow a genuine value's display tag; the value cursor must @@ -403,6 +405,45 @@ describe.each(indexTypes)( }, ) + it.each([`add`, `build`] as const)( + `follows an opposite callback for equal UTC aliases after %s`, + (construction) => { + const source = [ + { + id: `z-utc`, + when: Temporal.ZonedDateTime.from(`2024-01-15T09:00:00+00:00[UTC]`), + }, + { + id: `a-alias`, + when: Temporal.ZonedDateTime.from( + `2024-01-15T09:00:00+00:00[Etc/UTC]`, + ), + }, + ] + const [utc, alias] = source + expect(utc!.when.equals(alias!.when)).toBe(true) + expect( + alias!.when.toString().localeCompare(utc!.when.toString()), + ).toBeLessThan(0) + const index = new IndexType(1, new PropRef([`when`]), undefined, { + compareFn: (left: string, right: string) => right.localeCompare(left), + }) + // The callback puts UTC first. Both display-text order and source-key + // order put the alias first, so either shortcut fails this observation. + if (construction === `add`) { + for (const row of source) index.add(row.id, row) + } else { + index.build(source.map((row): [string, Row] => [row.id, row])) + } + expect(index.equalityLookup(utc!.when)).toEqual( + new Set([`z-utc`, `a-alias`]), + ) + expect(index.takeFromStart(2)).toEqual([`z-utc`, `a-alias`]) + expect(index.rangeQuery({ to: utc!.when })).toEqual(new Set([`z-utc`])) + expect(index.take(1, utc!.when)).toEqual([`a-alias`]) + }, + ) + it(`moves one row within its exact class when its custom order changes`, () => { const utc = { id: `a-utc`, From d917ba4c960ead6ee7f7f8e45df3cbc025ba7251 Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 23:33:50 -0600 Subject: [PATCH 21/46] fix(db): refresh captured inArray lists across live queries --- .changeset/refresh-captured-inarray-lists.md | 6 + docs/contributing/oracle-coverage.md | 102 +- .../borrowed-date-indexed-range.md | 20 + .../captured-inarray-publication.md | 196 ++++ .../comparison-unit-retirement-round-6.md | 10 + .../cross-realm-date-where-publication.md | 14 + .../custom-index-order-selection.md | 9 + .../oracle-reviews/temporal-equality-alias.md | 17 + .../oracle-reviews/temporal-index-order.md | 43 +- docs/errors.md | 10 + docs/guides/live-queries.md | 9 + packages/db-ivm/src/operators/groupBy.ts | 46 +- packages/db/package.json | 2 +- packages/db/src/collection/state.ts | 4 + packages/db/src/query/builder/functions.ts | 2 +- packages/db/src/query/compiler/evaluators.ts | 63 +- packages/db/src/query/compiler/group-by.ts | 217 ++-- packages/db/src/query/compiler/index.ts | 247 ++++- packages/db/src/query/compiler/joins.ts | 67 +- .../query/compiler/mutable-list-refresh.ts | 62 ++ packages/db/src/query/compiler/order-by.ts | 32 +- packages/db/src/query/compiler/select.ts | 24 +- packages/db/src/query/live/ARCHITECTURE.md | 118 ++- .../query/live/subset-demand-controller.ts | 7 +- packages/db/src/query/pooled-live-query.ts | 10 +- .../tests/fixtures/error-site-messages.json | 5 + .../captured-inarray-stages-oracle.test.ts | 510 ++++++++++ .../in-evaluator-oracle.property.test.ts | 59 +- .../query/index-path-collision-oracle.test.ts | 67 +- packages/db/tests/query/indexes.test.ts | 4 +- .../tests/query/load-subset-subquery.test.ts | 6 +- .../pooled-live-query-oracle.property.test.ts | 55 +- ...dicate-publication-oracle.property.test.ts | 945 +++++++++++++++++- .../temporal-equality-alias-oracle.test.ts | 175 ++-- .../tests/temporal-index-order-oracle.test.ts | 135 ++- 35 files changed, 2930 insertions(+), 368 deletions(-) create mode 100644 .changeset/refresh-captured-inarray-lists.md create mode 100644 docs/contributing/oracle-reviews/captured-inarray-publication.md create mode 100644 packages/db/src/query/compiler/mutable-list-refresh.ts create mode 100644 packages/db/tests/query/captured-inarray-stages-oracle.test.ts diff --git a/.changeset/refresh-captured-inarray-lists.md b/.changeset/refresh-captured-inarray-lists.md new file mode 100644 index 0000000000..55c4e14958 --- /dev/null +++ b/.changeset/refresh-captured-inarray-lists.md @@ -0,0 +1,6 @@ +--- +'@tanstack/db': patch +'@tanstack/db-ivm': patch +--- + +Read the current members of a caller-owned array captured by `inArray` when a compiled expression evaluates. After the array or one of its items changes, refresh the existing live-query result on the next source event, including rows untouched by that event. Let grouped queries retain their prior key and aggregate input values while they replace affected results. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index ebb9c4e14a..c009ed03b1 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -444,6 +444,7 @@ generic visitor behavior outside the named collectors. | Infinite-query publication | [shared publication laws](https://github.com/TanStack/db/blob/main/packages/db/tests/conformance/infinite-suite-oracle.ts), [React render cuts](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-render-cuts-oracle.test.tsx), [Svelte hook tests](https://github.com/TanStack/db/blob/main/packages/svelte-db/tests/useLiveInfiniteQuery.svelte.test.ts), [review record](oracle-reviews/2026-10-05-infinite-first-paint.md) | Each driver records every value its framework published: React layout commits, Vue pre-flushes, and the Svelte construction value plus pre-effect runs. The shared suite checks that a synchronous source shows its ready first page on the first value, for a query callback and for a supplied collection that has not started, that every ready value in a fixed-source, fixed-query scenario equals the source prefix for its own page count, including an equal dependency replacement that keeps page depth, and that a mount requests an on-demand first window once, for a query callback and for a supplied collection whose finite window is wider than or equal to the one the hook needs, judged by the rows the on-demand source holds after mount. Under deferred acquisition, an on-demand first value is either the ready first page or a value with no rows, by framework: React and Svelte publish before the hook subscribes, Vue after. `preloaded-collection-first-paint` checks the documented route-loader pattern: a limited live query preloaded before mount shows its ready first page on the first value in every framework and asks the source for nothing more; a preload that does not count as demand never settles there. Pre-commit work has no shared form, because Vue setup is its commit. The React driver owns renders that never commit, StrictMode double renders (one first-window request on React 19), retries after GC, a mounted replacement whose startup error must reach the error boundary, and a collection kept from an abandoned render that enters terminal error before its query is selected again, each as under `useLiveQuery`. The React source ID reuse oracle drives the infinite hook too. Svelte owns construction and superseded recomputes before the subscribing effect. In both, the hook acquires an on-demand source as often as `useLiveQuery` does under the same history, for either input form, direct or behind a live-query Collection. On-demand requests from renders that never commit are now owned by the live-query deferred acquisition oracle; the React render-cut oracle checks that an abandoned render makes no acquisition. Open cells, owned here: React 18 StrictMode, where neither hook keeps refs across the double render; a React commit held after the 50 ms unsubscribed GC floor, where a controlled-premise controller witness shows no non-ready value but no React scheduling witness exists; ready values in scenarios that write the source or replace the query, which need a per-step source model; a join of eager and on-demand sources, which needs a zero-acquisition check before commit; a supplied collection with no `.limit`, which sends one unbounded request on `main` too, because the controller cannot narrow it before the source request; and Svelte `state_unsafe_mutation` when a start inside `$derived.by` synchronously writes rows a peer is waiting for, shared with Svelte's `useLiveQuery`, which a two-component probe reproduces and the deferred-acquisition follow-up removes. | | Structural values and ordered primitives | [hash values](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-oracle.property.test.ts), [hash identity](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-identity-oracle.property.test.ts), [MultiSet consolidation](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/multiset-consolidate-oracle.property.test.ts), [hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-graph-oracle.property.test.ts), [mixed hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-mixed-graph-oracle.property.test.ts), [hash retry](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-failure-retry-oracle.property.test.ts), [murmur stream](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/murmur-stream-oracle.property.test.ts), [comparison](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts), [deep equality](https://github.com/TanStack/db/blob/main/packages/db/tests/utils-oracle.property.test.ts), [cursor](https://github.com/TanStack/db/blob/main/packages/db/tests/cursor-oracle.property.test.ts), [indexes](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update-oracle.property.test.ts), [BTree Map](https://github.com/TanStack/db/blob/main/packages/db/tests/btree-map-oracle.test.ts), [query identity](https://github.com/TanStack/db/blob/main/packages/db/tests/query/identity-output-shape-oracle.test.ts), [LIKE semantics](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/evaluators-oracle.test.ts) | Independent flat values, graph topology, algebraic laws, structural object keys in code-unit order (the flat-value, identity and consolidation grammars include keys that tie under locale collation), murmur byte framing against a byte-at-a-time reference (block mixing, the finalizer and symbol framing have no owner there), Map/group/sort recomputation, expression denotation, custom comparator dispatch/reference identity/index compatibility, LIKE wildcard refinement, compiled output bags, declared value-pair agreement between D2 equality keys and IR equality operands, and Collection collation with exact public scan/index order and one-index reuse across repeated reads. The auto-index cells cross BasicIndex and BTreeIndex; scan cells omit indexes. Both paths cover lexical and numeric `en-US` locale defaults plus an explicit ordinary locale override of a numeric default. An unindexed six-row work cell bounds Collection collation resolution to at most two reads per ordered scan: one index-path check and one scan clause. Other locales and ordered histories are outside this owner. The identity pair grammar covers primitives, signed zero, NaN/invalid Date, valid Date/timestamp, binary/Buffer, Temporal, nested references, symbols, and functions; fixed and sampled pairs do not prove every JavaScript value or hash collision freedom. Ref proxies are expressions rather than literal values in this lane. The deep-equality owner also pins that `deepEquals` ignores Map and Set insertion order, RegExp `lastIndex`, array holes, and typed-array class, the rules draft equality keeps. It also pins that objects of another class differ, that plain and null-prototype objects are one class, that a class instance without enumerable keys equals only itself, that URLs compare by `href`, and that typed-array elements compare like numbers. Its [Date/RegExp unit transfer](oracle-reviews/utils-unit-retirement-round-7.md) adds fixed and generated direct-return witnesses that Date differs from its ISO text and epoch number, and RegExp differs from its source text, while preserving the original copy, changed-time, changed-flag, and changed-source operands. Other cross-type pairs and public change-event traces remain outside that transfer. The hash-values owner checks equal-hash consequences for independently constructed flat values. Distinct type identities are checked by the hash-identity owner through `equalHashValues`, including forced digest collisions; distinct 32-bit digests are not promised. The hash-identity owner checks `hash` and `equalHashValues` against one spec-level identity model: primitives with signed zero, NaN, and bigint; reference leaves (functions, registered handles, `File`, binary over 128 bytes); Date, binary, Temporal, and RegExp headers; array length and holes; Map and Set insertion order; and object keys in any order, for acyclic values and for cycles through arrays, Map values, Sets, and plain objects. Map/Set order sensitivity is pinned current behavior, not a promise. Arrays from another realm, RegExps with a `NaN` `lastIndex`, getters, proxies, other cycle carriers, and pairs that differ only in a back-edge target are outside its grammar; pinned cases keep equality's cross-realm length check and strict RegExp field comparison. `hash-work.test.ts` pins how visited values and array/RegExp headers count toward the structural work cap, and that equality copies a shared Map's entries once per distinct pair ([code-weight hash dispatch review](oracle-reviews/code-weight-hash-dispatch.md)). Custom string indexes do not optimize ordinary ranges, and custom cursors remain unsupported. The LIKE owner covers boolean string matching and bounded work, not nullish three-valued logic or a general Unicode collation contract. Unsupported composite cursors reject. The MultiSet consolidation owner checks keyed, single-type, and structural identity, signed zero and NaN in keyed and single-number data, the retained first record, zero-sum removal, and input record contents. It does not assert output order. Its keyed grammar includes cross-type key and value collisions, non-finite numeric keys, the `\|` delimiter, symbol and function identity, and a direct-object/object-tuple delimiter control from #1948. Eight named collision witnesses and both generated campaigns are RED on `c879ba6d` and GREEN with the repair ([#1948 review record](oracle-reviews/issue-1948-keyed-consolidation.md)); the [code-weight consolidation review](oracle-reviews/code-weight-multiset-consolidation.md) records the earlier exclusion. The keyed reference uses direct value/reference equality, including SameValueZero for numeric keys. The unkeyed fallback keeps its original key and value domains. The direct `MultiSet` owner does not prove which `@tanstack/db` query shapes produce primitive keyed values or mixed-type source keys; that needs a live-query production witness. The index owner checks lower-bound presence on bounded numeric histories after each add, update, delete, and build: an upper-only range ignores an unused `fromInclusive: false`, while an explicit exclusive `from` excludes its boundary. A source Map supplies expected sets for BasicIndex and BTreeIndex; wrong-output controls reject both boundary mistakes. It also checks exact `equalityLookup` and `lookup('eq')` after singleton case-insensitive `A → a → A` replacements and populated comparator-group updates, plus fresh-object same-group replacements in fixed and generated histories. These direct-index checks do not cover every JavaScript value, arbitrary comparators, or long Collection histories. The index owner enforces one invalid-comparator law across both BasicIndex and BTreeIndex: for each comparator, accepted numeric prefix (including the empty prefix), and probe operation (add, update, eq/range lookup, take, build), the operation throws exactly when it receives a `NaN` or non-number result and never when every comparison is valid. A `signed infinity` comparator is the valid control, and a custom collation `compare` supplied through `compareOptions` is checked the same way. Two BTree Map split histories check inserted-key reads, exact whole-range payloads/order, size, and extrema, proving split placement follows the insertion index without a post-mutation comparison. A rejected index add or remove leaves the index refining its accepted rows; update and build are not atomic. At the Collection boundary, both index types and both write paths (optimistic insert, sync commit) crash the collection: status becomes `error` and the next mutation throws, so a usable collection never holds rows its subscribers were not told about. Open: a sync source can still commit writes on a collection in `error` state, which stores unpublished rows; this is existing `markError` behavior and needs a lifecycle-owner witness. Comparator transitivity is outside this evidence. | | WHERE predicate publication | [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts), [`in` evaluator](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts) ([review](oracle-reviews/perf-in-constant-set.md)) | An independent Kleene evaluator judges `eq`/`not`/`and`/`or` predicates over strings, a normalization-prefixed string, booleans, numbers, `NaN`, a valid Date, `null`, a missing field, and virtual fields. Snapshot histories with pending optimistic inserts compare a live query, direct subscribers with and without initial state, and `currentStateAsChanges`; change histories apply multi-key sync transactions, including reinsertion of a key deleted earlier, and compare every consumer's key set after each commit, on scan and `BasicIndex` paths. Peer subscribers share one field with different literals, plus one on a virtual field, so every published change must reach exactly the subscribers whose predicate it can satisfy; routing mutants that ignored the previous value, ignored the field, delivered a change twice, or routed while stale published rows awaited reconciliation fail here. Fixed witnesses cover a layout-only publication reaching a filtered subscriber as one empty batch, a filtered subscriber's empty Collection-readiness batch, retraction of a vanished row after eager cleanup and restart, and the new row plus exact subscriber update payload when a same-key row stays TRUE. A stale-live-value mutant survives the generated membership checks but fails the payload witness. A focused [property-visibility test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-prefilter-property-visibility.test.ts) compares the public unindexed snapshot with the enriched-row contract for inherited, non-enumerable, enumerable-own, and nested getter paths; the pre-fix stored-row shortcut failed three of its four controls. Other property-descriptor and stateful-getter histories remain outside those fixed controls. Hostile mutants for FALSE-for-UNKNOWN `eq`, `or` or number-literal subscription prefilters, a prefilter that ignores the previous value, a skipped readiness batch, a skip while stale published rows await reconciliation, and an unindexed snapshot scan that reads only synced rows while an optimistic insert or delete changes visibility passed the prior `@tanstack/db` suite and fail here. Change routing withholds a dropped row from an `eq` subscription before its sent key could be recorded, so a mutant that records dropped rows as sent is equivalent for routed predicates in this oracle. A focused `collection-subscription.test.ts` witness checks that a dropped insert or update, alone or beside a matching change, cannot advance a limited subscription's page offset under a routed `eq` and an unrouted `or` predicate; the unrouted cases beside a matching change kill that mutant, and the unrouted `alone` case does not. A loose-equality prefilter is an equivalent mutant: it only skips less. Skipping during truncate replay also survives; the replay's baseline diff re-derives the same retraction, so the guard keeps the prior dataflow without its own witness. Generated cleanup and restart histories for filtered subscribers remain open for the lifecycle publication owner. No-op updates, other comparison operators, Temporal and binary operands, joins, ordering, optimistic updates, and truncate are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. Change routing was later removed in favor of pooled live queries; the routing mutants above are historical, and the same peer laws now check plain subscription dispatch. The unindexed snapshot prefilter was restored for compiled queries as one stored-row `eq` test on a string or boolean literal. Its property-visibility test also covers the reverse direction: a stored row with an inherited or non-enumerable field must still match `isUndefined` on that field, which a scan that evaluated the full predicate on stored rows fails. The `in` evaluator owner checks `in` against an independent linear reference over Dates, `NaN`, `-0`, BigInt, bytes, Temporal values, objects and nullish items, for constant and row-supplied lists. Its one-polyfill Temporal model uses captured intrinsic methods to reject forged and borrowed brands, accepts genuine own display shadows, and models Duration by ten fields, including PT1H versus PT60M. Arbitrary custom prototype forgery, mixed Temporal implementations, WHERE publication, and indexed mixed ordinary-object/Temporal range order remain outside that owner. | +| Captured `inArray` query stages | [stage oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/captured-inarray-stages-oracle.test.ts) ([review](oracle-reviews/captured-inarray-publication.md)) | Twelve fixed public cuts include nine numeric stage and acquisition witnesses after one unrelated source transaction, a limited ORDER uniqueness counterexample, grouped SUM/min/max arguments in an untouched group, and an aggregate argument wrapped by a selected addition. The controlled on-demand cut checks an unfiltered, unordered, unlimited first request. The nine initial stage cuts, ORDER duplicate, and wrapped aggregate were RED→GREEN in the working tree; grouped SUM/min/max passes in workspace Vitest. The final DB suite passes 12,383 tests across 282 files; DB-IVM passes 693 across 45. Arbitrary values and histories, real providers, and exact-head review remain open. | | Joined result keys | [Joined result key oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/join-result-key-oracle.property.test.ts) | A nested-loop model recomputes the pair set of an inner, left, or full join over source keys drawn from plain, comma-bearing, bracket-bearing, and quoted strings, numbers beside the strings that print the same, both infinities, and `NaN`. After preload and each synced group change, the published rows must equal the model's pairs and the result key count must equal the row count. The comma-joined key encoding fails its two pinned histories and both campaigns; plain `JSON.stringify`, which prints infinities as the missing-side `null`, fails the pinned infinity history and both campaigns. A pinned history in which a `NaN`-keyed pair leaves and re-forms fails when the join index compares source-key prefixes with `===`. Joins over subqueries, more than two sources, custom `getKey`, and optimistic mutations are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. | | D2 Index storage | [Index refinement oracle](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/index-refinement-oracle.property.test.ts) | A plain-`Map` model sums multiplicities under a type-and-value identity in which `NaN` equals `NaN` and `-0` equals `0`. Histories of up to twelve additions to two keys mix prefixed arrays with `NaN`, `0`, `-0`, `1`, `'1'`, `1n`, and `'a'` prefixes and unprefixed values, with cancellation and reappearance. After each addition, `get` and `has` must equal the model. Comparing prefixes with `===`, which disagrees with the `Map` that holds them, fails three pinned histories and both campaigns; the `-0` control passes under both. Compaction, presence tracking, and structural payloads are outside this owner; the join operator tests and incrementalization law own operator behavior. | | Pooled live queries | [pooled live query oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/pooled-live-query-oracle.property.test.ts) | An independent `eq` model (nullish UNKNOWN, Date as timestamp, `NaN` equal to `NaN`, `-0` equal to `0`) gives membership; a live-query Collection compiled for the same query gives order, values, and status. Histories mix sync transactions, optimistic writes confirmed or rolled back, peer mounts and unmounts, and source cleanup and restart, including cleanup while a write is pending and a mount between cleanup and restart. Wholesale observers must match after every step, granular changes must match a reference observer by type, key, value, and previous value and replay to the model's rows, the layout revision must advance when keys reorder, and observing must never build the live-query Collection. Peers may add a residual `not(eq(...))` conjunct, which the model evaluates itself; a view that ignores it, reports an unfiltered `entries()`, or delivers a row entering or leaving it as an update fails it. Other residual operators use the same compiler evaluator but are not generated. Peers may also order their rows by `id` or by a `g` that can be null, with explicit `nulls`; order comes from the reference. A comparator that ignores direction or `nulls`, an order left out of the partition key, and key order instead of the query's order each fail the pinned reorder history and both campaigns. A partition that ignores a row's previous group, orders by arrival, skips normalization, skips the initial snapshot, follows the source's status after cleanup, or never terminates fails it. Two shared conformance scenarios (`eq-filter-rows`, `eq-filter-peers`) run the pooled path under React, Vue, Solid, Svelte, and Angular; a partition that ignores a row's previous group fails both under every adapter, and `eq-filter-peers` checks through `subscriberCount` that each adapter shares one source subscription per partition. A focused [release-timing test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/pooled-live-query-gc-oracle.test.ts) compares when a pooled query and a live-query Collection release their sources across `gcTime` 1, 100, 0, and `Infinity`, with and without a subscriber, and with no `gcTime`, where both use the live-query Collection default, checks that a partition waits for its views' longest `gcTime`, checks that a view resubscribed after its partition released follows later source writes as its live-query Collection does, including a view with a residual conjunct, and checks that a partition that subscribes again still serves new mounts from one source subscription. Further focused witnesses check that a partition keeps groups only for values that have rows or watchers, that a detached reader never sees a dropped group's revision reused, keep a pooled query's public Collection live while the view is observed, accept it as a query source, and check that a pooled query turns terminal when source cleanup starts while the adapter's cleanup is still pending, as a live-query Collection does. A pinned oracle witness checks that an `eq` path whose getter throws excludes the row on both paths; a pooled path that rethrows fails it. The pooled oracle's grammar never releases a partition, so release and resubscribe histories rely on these fixed witnesses; a fixed delay, a missing unsubscribed floor, a last-`gcTime`-wins rule, and releasing at `gcTime` 0 each fail it. The [2026-10-01 review](oracle-reviews/2026-10-01-pooled-live-queries.md) records each ORC outcome. | @@ -480,22 +481,24 @@ order, public Date range predicates, cursor continuation, proxies, deeper subclass chains, replaced Date prototypes or `getTime` methods, and other Date-consuming paths remain outside that witness. -The comparison oracle also owns a bounded [Temporal order transfer](oracle-reviews/comparison-unit-retirement-round-6.md). Supplied ISO calendar fields and epoch-nanosecond offsets predict direct `ascComparator` signs and exact zeros for ordinary `PlainDate` and `ZonedDateTime` values from one polyfill copy. Fixed cases preserve the two retired unit blocks; generated cases vary calendar fields, offset zones, equal instants, and adjacent nanoseconds. Fresh-module histories reject a first-input constructor cache poisoned by a genuine instance or forged tag, including the direct `compareValues` path. Both direct helpers also receive a right-hand PlainDate with a constructor shadow. A genuine ZonedDateTime with an own shadowed display tag retains its instant position through both direct helpers. One genuine tagged ZonedDateTime subclass with inherited Temporal methods retains native-equal and native-order-equal results against a UTC peer; an object borrowing its prototype is an ordinary-object control. This lane does not establish public query or index behavior. Other Temporal subclasses, spoofed or replaced instance prototypes, mixed native/polyfill implementations, other Temporal kinds, and public ordering/filter publication need separate witnesses. Spoofed Temporal prototypes can still cause asymmetric comparisons; the comparison owner needs a dedicated brand and mixed-value witness before that class can be called closed. +The comparison oracle also owns a bounded [Temporal order transfer](oracle-reviews/comparison-unit-retirement-round-6.md). Supplied ISO calendar fields and epoch-nanosecond offsets predict direct `ascComparator` signs and exact zeros for ordinary `PlainDate` and `ZonedDateTime` values from one polyfill copy. Fixed cases preserve the two retired unit blocks; generated cases vary calendar fields, offset zones, equal instants, and adjacent nanoseconds. Fresh-module histories reject a first-input constructor cache poisoned by a genuine instance or forged tag, including the direct `compareValues` path. Both direct helpers also receive a right-hand PlainDate with a constructor shadow. A genuine ZonedDateTime with an own shadowed display tag retains its instant position through both direct helpers. One genuine tagged ZonedDateTime subclass with inherited Temporal methods compares equal in instant position to a +05:30 peer through both direct helpers. The alias equality owner below checks native `.equals()` against a UTC peer and the borrowed-prototype control. This direct comparison lane does not establish public query or index behavior. Other Temporal subclasses, spoofed or replaced instance prototypes, mixed native/polyfill implementations, other Temporal kinds, and public ordering/filter publication need separate witnesses. Spoofed Temporal prototypes can still cause asymmetric comparisons; the comparison owner needs a dedicated brand and mixed-value witness before that class can be called closed. The [Temporal index order oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/temporal-index-order-oracle.test.ts) and its [review record](oracle-reviews/temporal-index-order.md) own direct BasicIndex and BTreeIndex ordered takes after add/build and one mutation continuation, plus an optimized-only public Collection snapshot with an -explicit index. Supplied epoch nanoseconds establish order without promising -a public key tie break for equal instants; direct index takes follow the -established source-key tie rule, and the public snapshot permits either -equal-instant peer. Distinct zones retain exact lookup identities. A focused +explicit index. Supplied epoch nanoseconds and source keys establish the +order of equal-instant peers in direct indexes and public snapshots under +changelog #958; reverse admission makes a wrong public limit-one tie visible. +Distinct zones retain exact lookup identities. A focused callback witness preserves the former string operands for a custom comparator over two distinct pre-epoch values. A second callback witness splits query-equal UTC aliases into per-row comparator positions after add/build and checks direct range, cursors, removal, and same-class update. An opposite callback puts the UTC peer first even though display-text and -source-key order put the alias first. +source-key order put the alias first. Same-equality-class updates test both +transitions between a callback split and a source-key tie, including ranges +and cursors after each transition. An add/build witness checks that a genuine value's own shadowed display tag cannot move its default ordered position or the value cursor. Direct removal supplies the exact prior row. Mixed value domains, other @@ -507,9 +510,11 @@ and its [review record](oracle-reviews/temporal-equality-alias.md) own the bounded `ZonedDateTime.equals()` partition for UTC/Etc/UTC aliases, a distinct fixed-offset zone, adjacent nanoseconds, and a different calendar. One native ZonedDateTime subclass with inherited Temporal methods belongs to the UTC -class. A second genuine subclass shadows its prototype's display tag and -remains in that class; ordinary objects borrowing either subclass prototype -retain reference identity. Direct BasicIndex/BTreeIndex add, build, and removal checks compare +class. Another genuine subclass shadows its prototype's display tag with an +unrecognized name, and a third uses the recognized but misleading +`Temporal.PlainDate` tag. Native slots keep both in the UTC class; ordinary +objects borrowing these subclass prototypes retain reference identity. +Direct BasicIndex/BTreeIndex add, build, and removal checks compare independent fixture-assigned classes. Unindexed and optimized-only indexed Collection snapshots check public `eq`; the `in` evaluator oracle checks constant and row-supplied alias lists. A plain object with a forged Temporal tag and a @@ -587,7 +592,7 @@ fixture covers BasicIndex and BTreeIndex, one numeric field, and limit one. Positive spies observe the later ordinary index's ordered take and numeric lookup/range calls, including a separate reverse take for descending order. The paired inclusive range selects only rank 20 from ranks 10, 20, and 30, -so either misplaced bound changes the complete public key set. Inclusive +so omitting either bound changes the complete public key set. Inclusive value-left boundaries are fixed controls. Another fixed ready-state fixture compares unindexed and optimized indexed Date/number order, both full and limit one, with an index-take reach check. @@ -626,8 +631,74 @@ generated subclass histories need separate witnesses. A focused unindexed public query also keeps one local `Object.create(DateSubclass.prototype)` impostor outside Date equality while retaining its same-object match. It also excludes the impostor from the -invalid-Date/NaN equality class. Indexed publication for that local impostor -remains open. +invalid-Date/NaN and valid zero Date/number equality classes at public +checkpoints. Indexed publication for that local impostor remains open. + +The [captured `inArray` publication review](oracle-reviews/captured-inarray-publication.md) +records the caller-owned list law: changing its members alone does not +publish, but the next source event must reconcile the complete existing +live-query result. Fixed direct-snapshot, retained live-query, and ordered +limit-one cuts compare current list membership with public keys and, where +applicable, exact insert/delete events for untouched rows. A second ordered +cut deletes a row outside the formerly acquired prefix after the list changes. +It failed before the ordered plan required full-source acquisition and passes +after that repair; the plan now pays full-source transfer to observe unseen +source events. A pooled query with an equality group and residual mutable +list uses the live-query Collection path; a source event in another group +must still refresh its result. Controlled on-demand acquisition, no-delta events on both joined +source sides, an outer QueryRef filter, a correlated include, and mutable Date +and byte-list items have fixed working-tree receivers. One 40-row direct-query +witness rejects an all-candidate membership rescan on an ordinary +unchanged-list event while requiring a revisit after a list change. These are bounded +in-memory receiving cuts. A controlled on-demand ordered receiver checks that +its first request omits `where`, `orderBy`, and `limit`. An on-demand LEFT-join +receiver was RED when a delete outside keyed lazy demand failed to refresh +membership, then GREEN after dynamic-list compilation disabled that demand. +An on-demand correlated include receiver now checks that an unacquired spare +child deletion refreshes membership and that its first child request omits +`where`; it passes after the same graph-wide lazy-demand gate. The earlier +WHERE-focused working-tree and neighboring run passed 988 tests across 11 +suites with no type errors; exact committed-head review remains pending. The +graph-wide gate can increase full-source transfer for on-demand joins and includes. +Source inspection shows list snapshots cost work proportional to list length +per delivered graph event, with candidate reevaluation after list changes. +The existing index-optimization and subquery-forwarding positive tests now +use frozen `inArray` lists: immutable membership can retain index pushdown +and finite predicate forwarding, while mutable captured membership requires +full-source reach at the covered boundaries. Their 34 focused tests pass. +`ExpressionLike` accepts these `ReadonlyArray` inputs, and a separate missing +error-catalog entry, code 243, is recorded. The full DB suite passes 12,383 +tests across 282 files; DB-IVM passes 693 tests across 45 files. Both builds +and explicit source and test typechecks pass. +Real providers, broader schedules and values, arbitrary list histories, and +general work bounds remain open for these WHERE paths. + +The [captured `inArray` stage oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/captured-inarray-stages-oracle.test.ts) +extends the same law to SELECT, ORDER, GROUP keys, aggregate arguments, +HAVING, either JOIN operand, and an inner QueryRef SELECT. Twelve fixed public +witnesses use plain arrays and current membership to predict results. Nine +initial stage and acquisition cuts change `[1]` to `[2]` before one unrelated +source transaction. SELECT also checks one coherent subscriber batch, and an +on-demand projected ordered limit checks that the first source request has no +`where`, `orderBy`, or `limit`. These nine were RED against the earlier stage +implementation and GREEN after shared raw-source list revision and retained +stage-output repair. A hostile review then found a limited ORDER duplicate: +a separate four-row, limit-three witness was RED before deferred Collection +order maintenance and GREEN afterward. A grouped SUM/min/max receiver checks +an untouched group after an unrelated group update and passes in workspace +Vitest; a contrary standalone probe imported stale DB-IVM output and did not +reproduce there. A twelfth witness wraps a mutable aggregate argument in +selected `add(sum(caseWhen(inArray(...))), 10)`. The prior implementation +kept total 11 after the list changed, while the plain-array model required +12; recursive aggregate-input detection makes this public comparison pass. +Hostile nullish grouped/public and projected-sort probes were bounded GREEN +controls, not confirmed defects. Before the wrapped witness, 11 stage and +33 WHERE/publication cases passed together; the final DB suite includes all +twelve stage witnesses and the WHERE/publication receivers. This owner does +not yet establish arbitrary values or source histories, real-provider +behavior, or exact committed-head review. +The [review record](oracle-reviews/captured-inarray-publication.md) tracks the +ORC-012 applicability audit and unresolved cells. The [borrowed-Date range review](oracle-reviews/borrowed-date-indexed-range.md) records a separate public range owner in `index-path-collision-oracle.test.ts`. @@ -638,8 +709,11 @@ or the predicate bound. A genuine foreign Date subclass is a positive indexed range control. A later fixed history gives local and foreign native Date subclasses misleading `valueOf` results. Public scan and indexed `gt`, `gte`, `lt`, and `lte` snapshots use their intrinsic timestamps, with an -optimized-only index-path check. A native subclass also serves as a bound on -both predicate sides while its `valueOf` reports a misleading number. +optimized-only index-path check. The model freezes each source timestamp +before Collection admission, so later changes to a source Date cannot alter +the expected answer. A native subclass also serves as a bound for all four +effective relations in both operand positions while its `valueOf` reports a +misleading number. Date/number scan predicates run in both operand orders; mixed-domain index eligibility is not claimed. Other coercion methods, invalid Dates, later publication, and proxies remain open there. diff --git a/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md b/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md index 6fb0617a38..e095e95ebf 100644 --- a/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md +++ b/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md @@ -70,3 +70,23 @@ The focused owner passed 18 tests. This reaches the Date-bound compiler path; it does not establish arbitrary coercion methods, generated subclass histories, or other hosts. Fresh hostile review of the final committed follow-up is pending. + +## Read-only review of `0c236f448` + +The hostile reviewer ran 18 focused cases and found a false-green reference +model. The earlier expected-key function called the native Date intrinsic on +source objects after those objects had entered the Collection. A source Date +changed after admission could therefore change the model's expected answer +along with production. A read-only probe demonstrated that risk; it did not +report a production range failure. + +The model now freezes each source row's native timestamp before Collection +admission and derives every expected public key set from those numbers. A +post-check Date mutation confirms the recorded expectation stays fixed. The +subclass bound now reaches all four effective relations in both operand +positions; source values include local and foreign subclasses whose +`valueOf` disagrees with native `getTime`. Public scan and optimized indexed +snapshots still compare against the same frozen model. The focused owner +passes 18 tests after the change. This is a fixed in-memory matrix, not +evidence for later publication, other realms, arbitrary coercion, or provider +cursors. The new witnesses await hostile review at a committed head. diff --git a/docs/contributing/oracle-reviews/captured-inarray-publication.md b/docs/contributing/oracle-reviews/captured-inarray-publication.md new file mode 100644 index 0000000000..f6bab6c4bd --- /dev/null +++ b/docs/contributing/oracle-reviews/captured-inarray-publication.md @@ -0,0 +1,196 @@ +# Captured `inArray` membership at source publication + +The accepted contract for a live query that captures a caller-owned `inArray` +list is that its current members govern the whole existing result when the +next source event is processed. Changing the list alone publishes nothing. +An untouched row can enter or leave after an unrelated source update. This +query input is distinct from an already submitted `LoadSubsetOptions` request: +the latter's membership data remains immutable for its lifetime under +`packages/db/src/query/live/ARCHITECTURE.md`. + +The primary receiving owner is +`packages/db/tests/query/where-predicate-publication-oracle.property.test.ts`. +Its fixed model filters authored source facts with the list's current +members; it does not use compiled membership or source-index decisions to +predict expected keys. A direct Collection snapshot checks current members +after a source transaction. A retained live-query Collection starts with +members `[1]`, changes them to `[2]` and then `[2, 1]`, and delivers an +unrelated source update each time. It compares the complete public key set +and exact insert/delete events for untouched rows, then checks that an +unchanged list does not republish those rows. A separate ordered limit-one +history admits a previously excluded row ahead of the old top and then +removes all prefix matches, requiring refill to the next current member. +A design review then found that this green history missed an unseen source +event. A second ordered history changes `[1]` to `[0]` and deletes rank 4, +outside the acquired finite prefix. Before repair, its public result stayed +`['1']` instead of `['0']`. The repaired dynamic-list ordered plan requires a +full-source acquisition, so that delete reaches the query and the receiver +passes. This costs a full-source acquisition for such ordered plans; the +in-memory receiver does not measure a real provider's bandwidth or latency. +A controlled on-demand ordered receiver checks that its first `loadSubset` +request has no `where`, `orderBy`, or `limit`, then checks the public result +after an unseen delete and list change. + +The controlled on-demand receiver deliberately returns only its first row +when a source request contains a `where` predicate. The query must acquire +unfiltered rows so a later list member is available after an unrelated +update. An inner join receiver changes the list, then updates an unmatched +row on either source side. Neither write yields a joined-row delta, but each +must publish one coherent reverse insert/delete batch for the already joined +rows. A QueryRef receiver puts mutable membership in the outer query while +the inner filter suppresses the triggering source change. A correlated +include receiver changes an unrelated child row and checks the parent row's +child Collection against current membership. Fixed Date and byte-array +items mutate in place inside their captured lists and change the result at +the next unrelated source event. + +The pooled-query receiver is +`packages/db/tests/query/pooled-live-query-oracle.property.test.ts`. A query +with `eq(group, 'g')` and a residual `inArray` list changes membership, then +updates a source row in another equality group. A pooled group would not +notify that view, so the fixed receiver requires the live-query Collection +path and checks the complete result after the event. The unguarded pooled +path failed this public snapshot before the guard; the receiving cut passes +after fallback. This does not assert that the fallback has the same work cost +as a pooled view. + +A 40-row work witness observes indexed reads of one captured list slot. An +ordinary source update with unchanged membership reads that slot fewer than +ten times; changing the list before the next update reads it at least 40 +times and refreshes the result. This rejects an all-candidate membership +rescan for every ordinary event in that fixed direct-query path. It does not +establish a general time bound, provider work bound, or cost for the other +query shapes above. Source inspection shows that each delivered graph event +snapshots the captured list in work proportional to its length; a list change +also revisits retained candidates. The fixed counter observes one list slot, +not an asymptotic cost bound for all list values and query shapes. + +At committed head `0c236f448`, the evaluator's compiled equality owner passed +249 focused tests for its declared value grammar. That result did not test +the retained live-query result after a caller changed a captured list. The +new fixed receiving histories address that separate publication boundary. +The revised source and receiver still await an exact committed-head hostile +review. A further design review found an on-demand LEFT-join gap: +main rows `one(v=1,g)` and `two(v=2,g)` join group `g`, while group `h` is +outside the lazy joined-side demand. After the list changes from `[1]` to +`[2]`, deleting `h` left the public result at `one` instead of changing it +to `two`. The joined source event was real, but keyed lazy demand did not +deliver this unacquired row to the query. The new public receiver was RED +before the compiler disabled keyed lazy joined-side demand for dynamic-list +compilation and GREEN afterward. This controlled source does not establish +real-provider behavior or the transfer cost of full acquisition. A parallel +on-demand include receiver has parent `g`, children `one(v=1,g)` and +`two(v=2,g)`, and spare child `(v=3,h)` outside the former lazy child demand. +After `[1]` changes to `[2]`, deleting the spare must change the materialized +children from `[one]` to `[two]`. The working-tree receiver passes and checks +that the first child `loadSubset` request has no `where` predicate. A separate +public probe saw `[one]` before this gate and `[two]` afterward; the oracle +suite's combined run establishes the receiving check after the gate. + +An earlier working-tree focused and neighboring run passed 988 tests across +11 suites, with no type errors. It includes the direct, pooled, ordered, +on-demand, joined, QueryRef, include, Date/byte, and 40-row work receivers. +The later full DB run exposed two older positive fixtures that assumed an +ordinary mutable `inArray` literal could still use index pushdown or forward +a finite subset predicate. Those assumptions conflict with the accepted +captured-list law: a changed member can admit an existing row outside the +former subset. The index-optimization and subquery-forwarding tests now use +`Object.freeze` lists, so they continue to check those optimizations within +the immutable-list domain. Their 34 focused tests pass. This fixture change +does not retire either test. `ExpressionLike` now accepts `ReadonlyArray` so +the frozen expressions typecheck; a separate error-catalog failure was fixed +by recording code 243. The final DB suite passes 12,383 tests across 282 files, +and DB-IVM passes 693 tests across 45 files. Both suites have no test type +errors. The DB and DB-IVM builds and explicit source and test typechecks pass. +Real providers, broader source schedules, arbitrary list values and histories, +and general work bounds remain outside these fixed receivers. Exact +committed-head hostile review remains pending. + +## Query-stage transfer + +The new primary stage owner is +`packages/db/tests/query/captured-inarray-stages-oracle.test.ts`. Its plain +array model checks numeric list membership after `[1]` changes to `[2]` and +one unrelated source transaction updates value 3. Values 1 and 2 remain +untouched. The initial nine fixed public witnesses cover selected membership, +including one coherent update batch and a callback that reads the complete +new result. The other cuts cover a retained ordered limit, an on-demand +selected value under an ordered limit, group-key moves, an aggregate argument, +HAVING, membership on each JOIN operand side, and an inner QueryRef SELECT +consumed by an outer filter. The +controlled on-demand witness also requires its first `loadSubset` request to +omit `where`, `orderBy`, and `limit`. Three later fixed cuts check a distinct +four-row limited ORDER result, grouped SUM/min/max arguments while a +different group receives the source update, and a mutable aggregate argument +wrapped by a selected addition. These are stage and acquisition +cuts of the same accepted source-publication law, not a generated grammar. + +The nine initial witnesses were RED against the earlier stage implementation +and GREEN after the working-tree repair; each also compares public rows. +Source inspection shows one list revision driven by raw source events across +the compiled graph. A revision +revisits retained stage inputs; each stage retains its prior output for +retraction before evaluating current membership. The repair applies this at +JOIN, WHERE, SELECT, GROUP and aggregate input, HAVING, and ORDER, with an +optional mapped-input hook in the D2 groupBy operator. The fixed witnesses +establish the named observations, not all combinations of these stages. + +Hostile design review found one further public counterexample: after +membership changed, a limited ordered result duplicated a source row. A +four-row, limit-three receiver sorts each source identity once by current +membership and key. It was RED before Collection deferred order maintenance +for a layout change and GREEN afterward. A separate grouped receiver checks +SUM, min, and max arguments for an untouched group after a different group +updates; it passes in workspace Vitest. The standalone probe that suggested a +min/max failure imported stale DB-IVM output, so it did not establish a +production defect. The wrapped-aggregate receiver selects +`add(sum(caseWhen(inArray(row.v, members), row.v, 0)), 10)`. After `[1]` changes +to `[2]` and an unrelated row updates, the plain-array model requires total +12; the earlier implementation retained 11. Recursive aggregate-input +detection makes the twelfth public witness GREEN. Hostile probes of nullish +grouped/public output and projected sorting were bounded GREEN controls, not +confirmed production defects. The earlier working-tree run passed 11 stage +tests and 33 WHERE/publication cases; the final DB suite includes all twelve +stage witnesses and the WHERE/publication receivers. The exact committed-head +hostile verdict remains pending. Real-provider event delivery, arbitrary list +values or histories, mixed stage combinations, and general work bounds remain +outside this fixed stage owner. + +### Provisional oracle-guide applicability + +- **ORC-001–003:** The accepted next-source-event law and fixed numeric limits + are stated in the owner; plain-array `includes` computes expected results + without compiler or D2 outputs; opening prose distinguishes model, source + history, public observations, and comparison cuts. Exact-head review of + that prose remains pending. +- **ORC-004, ORC-007, ORC-008, ORC-009:** No generated history, seed/path + campaign, stateful reference model, or model-only subsystem vocabulary is + introduced by these fixed recomputations. +- **ORC-005–006:** The real live-query Collection is compared immediately + after preload and one source commit. SELECT additionally checks one public + subscriber batch; the on-demand case checks request shape before comparing + public rows. The nine initial pre-repair runs failed their receiving checks + and passed after repair in the working tree. The separate limited ORDER + duplicate was also RED at its public comparison and GREEN after its repair. + The wrapped aggregate was RED at total 11 against expected 12 and GREEN + after recursive detection. The grouped SUM/min/max receiver passes in + workspace Vitest. This + calibrates these fixed cuts without claiming all legal histories. +- **ORC-010:** `withOracleCleanup` releases each source, live-query Collection, + and subscription where present while preserving the primary assertion + failure. No shrink or replay capture occurs. +- **ORC-011:** The limited ORDER duplicate and top-level-only aggregate + detection name additional wrong designs; plain-array ordering and wrapped + sum predictions reject them. The apparent min/max + failure came from a stale standalone module import; the workspace receiver + passes. No general stage-law closure is claimed. +- **ORC-013:** This owner claims twelve fixed cuts and checkpoints, not all + legal values, histories, or stage combinations. New reachable cases remain + within this stage owner's coverage responsibility. +- **ORC-014:** The on-demand source is controlled. Its unfiltered request and + public result establish the Collection path; real-provider adherence and + bandwidth remain open. +- **ORC-012:** This applicability record is provisional. Verdict-critical + evidence needs a fresh hostile review tied to the exact committed head. The + full DB and DB-IVM suites, builds, and source/test typechecks pass at the + working-tree checkpoint above. diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md index 9fdb2c1644..932662f5dd 100644 --- a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md @@ -145,3 +145,13 @@ product defect, not a recovered assertion from the two retired ordinary-value blocks. Other subclasses, replaced methods, mixed implementations, and arbitrary prototype chains remain open. A fresh hostile review of the final committed follow-up is pending. + +Read-only hostile review of `0c236f448` ran 122 comparison tests and found +no lost assertion from the two retired ordinary Temporal blocks. It corrected +the coverage-map attribution for the tagged subclass: direct +`ascComparator` and `compareValues` compare the subclass with a +05:30 peer +at the same instant, while the alias owner separately checks native +`.equals()` against a UTC peer and the borrowed-prototype control. Those +observations support different boundaries; the direct comparison owner does +not itself prove the public equality result. The final committed follow-up +still needs a fresh hostile review. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md index 5564832f88..9f15ddacf5 100644 --- a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md +++ b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md @@ -174,3 +174,17 @@ live-query key sets after each sync commit. The fixed Date matrix compares direct snapshots both before and after its commit. The prose now names those separate checkpoints. No behavior or product expectation changed in this clarification. Final exact-commit hostile review remains pending. + +## Zero timestamp false-green control at `0c236f448` + +A read-only hostile review found that the local borrowed-prototype object's +existing checks could still pass if it erroneously joined the valid zero +Date/number equality class. The owner now asks two additional public live +queries for `eq(row.v, new Date(0))` and `eq(row.v, 0)` after preload. Both +must return no rows for that ordinary object. These controls use public +publication rather than a direct comparison helper and distinguish object +identity from timestamp zero and numeric zero. The focused 81-test run +including them passed with no type errors. The later working-tree focused and +neighboring run passed 985 tests across 11 suites. This does not establish +indexed publication for the forged object, other borrowed prototypes, or +other host realms. Exact committed-head review of these controls is pending. diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md index 65c13519b8..e3e3c09d69 100644 --- a/docs/contributing/oracle-reviews/custom-index-order-selection.md +++ b/docs/contributing/oracle-reviews/custom-index-order-selection.md @@ -187,3 +187,12 @@ bound observable at the public snapshot and traversal checkpoints. The focused owner passed. They do not prove arbitrary numeric histories, providers, or the live-query compiler's index-use premise. A fresh hostile review of the final committed follow-up is pending. + +The read-only hostile review of `0c236f448` ran all 10 owner tests and found +no blocker in the fixed selection domain. It corrected one claim about the +paired inclusive bound: with ranks 10, 20, and 30 and predicate +`20 <= rank <= 25`, omitting either bound changes the result. Moving either +bound slightly does not necessarily change it, so the coverage map now says +"omitting" rather than "misplacing." This is a documentation correction, +not a new production bug or an added witness. Final review of the next +committed head remains pending. diff --git a/docs/contributing/oracle-reviews/temporal-equality-alias.md b/docs/contributing/oracle-reviews/temporal-equality-alias.md index 1aa8324cb2..deef723500 100644 --- a/docs/contributing/oracle-reviews/temporal-equality-alias.md +++ b/docs/contributing/oracle-reviews/temporal-equality-alias.md @@ -300,3 +300,20 @@ covers one imported polyfill's subclass with inherited methods and one shadowed prototype tag. Replaced intrinsic methods, other implementations, arbitrary prototype chains, and provider paths remain open. A fresh hostile review of the final committed change is pending. + +## Read-only review of `0c236f448` + +The reviewer ran 28 alias tests and found no blocker in the then-declared +subclass domain. It identified a neighboring sensitivity gap: the tagged +subclass used an unrecognized tag, so a classifier that trusted a recognized +but misleading `Temporal.PlainDate` tag could still put a genuine +`ZonedDateTime` in the wrong equality kind. The fixture now gives another +genuine subclass that tag while leaving its inherited native methods and +slots intact. Native `ZonedDateTime.prototype.equals` confirms its UTC alias +class. Both direct index types check add, build, exact lookup, and removal; +unindexed and optimized-only public `eq` check the same class. An ordinary +object borrowing that subclass prototype and a distinct copy remain +reference-identity controls. The revised owner passes 28 tests. This adds +one recognized wrong-kind tag to the fixed polyfill grammar; arbitrary +prototype impersonation and mixed implementations remain open. Fresh hostile +review of the next committed head is pending. diff --git a/docs/contributing/oracle-reviews/temporal-index-order.md b/docs/contributing/oracle-reviews/temporal-index-order.md index 8d158afc7d..68afccfeb7 100644 --- a/docs/contributing/oracle-reviews/temporal-index-order.md +++ b/docs/contributing/oracle-reviews/temporal-index-order.md @@ -7,10 +7,10 @@ keeps zones distinct. The law comes from Temporal's `ZonedDateTime.compare` and `equals` semantics, changelog #1519, and the Collection's ordered-read and IndexInterface contracts. The live-query architecture also requires an ordered comparator group to keep a representative from a live exact-value -bucket. The direct index's established source-key rule orders two rows at the -same instant. The public ordered snapshot does not promise which -equal-instant peer occupies a limit-one window, so that observation accepts -either peer. +bucket. Changelog #958 promises deterministic key-based tie breaking for +Collection and index iteration, including live queries with `orderBy` and +`limit`. Both direct indexes and the public ordered snapshot therefore use +the source-key tie rule for two rows at the same instant. ## Original failure and repair @@ -95,19 +95,19 @@ tag or mixed Temporal implementations. | Requirement | Evidence and limit | | --- | --- | -| ORC-001 | The opening names instant order, zone-sensitive equality, live representatives, authority, and the bounded domain. Equal-instant tie order is deliberately unspecified. | +| ORC-001 | Temporal comparison and changelog #1519 authorize instant order and zone-sensitive equality. Changelog #958 authorizes source-key ties in direct indexes and public ordered snapshots. The fixed domain remains bounded. | | ORC-002 | Supplied BigInt instants and zone classes determine expected order and exact matches independently of normalization and comparison code. | | ORC-003 | The oracle describes its law, fixture model, legal actions, direct and public drivers, observations, and checkpoints beside executable code. | | ORC-004 | Not triggered: the bounded fixture matrix and one continuation are fixed histories, not a generated-history claim. The nearby invalid mixed-kind domain is excluded. | | ORC-005 | Direct takes, ranges, cursors, and equality lookups observe both index implementations. An optimized-only Collection snapshot invokes the explicit index's `takeFromStart` after readiness. | -| ORC-006 | The original normalized-string design failed at direct ordered takes and the public `limit: 1` snapshot. A stable wrong-result control rejects that later-instant first row while allowing both legitimate equal-instant ties. The first BTree repair failed the live-bucket representative check; a tuple-fed custom comparator failed the pre-epoch take check; the merged-exact-key design failed six alias/callback takes. Own-tag shadows failed all four value-cursor checks before the branded dispatch repair. | +| ORC-006 | The original normalized-string design failed at direct ordered takes and the public `limit: 1` snapshot. A wrong-result control rejects both a later instant and the wrong source-key peer at the equal-instant limit. The first BTree repair failed the live-bucket representative check; a tuple-fed custom comparator failed the pre-epoch take check; the merged-exact-key design failed six alias/callback takes. Own-tag shadows failed all four value-cursor checks before the branded dispatch repair. | | ORC-007 | Not triggered: this is a bounded fixed oracle, not an important generated property. The existing index refinement property keeps its own campaigns. | | ORC-008 | Not triggered: the fixture's independent model has no mutable reference state. | | ORC-009 | The model's instant label represents default order only; its zone class represents exact equality. One comparator position can contain two exact values, and a custom callback can split one exact value into two positions. The callback-argument witness checks exact former input text. | | ORC-010 | `withOracleCleanup` preserves a primary public mismatch and releases the Collection. Direct indexes own no external resources. | | ORC-011 | The unindexed public scan is a second production formulation for the ordered result. The numeric BigInt model also distinguishes a shared display-string mistake. | | ORC-012 | This record contains RED/GREEN evidence, a hostile review, and scoped unresolved cells. An exact-commit verdict follows below. | -| ORC-013 | The equal-instant zones and one-nanosecond neighbor distinguish collapse and truncation; the later instant with earlier display text distinguishes lexical order. | +| ORC-013 | Equal-instant zones admitted in reverse key order distinguish source-key ties from admission order. A one-nanosecond neighbor distinguishes collapse and truncation; a later instant with earlier display text distinguishes lexical order. | | ORC-014 | Not triggered: the Collection has controlled sync and the claim ends at its in-memory ordered snapshot, with no real-provider claim. | ## Exact committed-head review @@ -127,8 +127,9 @@ defect. The direct oracle had allowed either source-key order for two equal-instant peers. It now admits those peers in reverse key order through both `add` and `build` and requires the direct `takeFromStart(2)` result to follow the established source-key tie rule in BasicIndex and BTreeIndex. The -public limit-one tie remains permissive. This addition covers only the -named equal-instant fixture and direct index checkpoint. +earlier public limit-one allowance was too broad under changelog #958. This +addition initially covered only the named direct-index checkpoint; the public +receiving cut is described below. The maintainer confirmed the custom comparator contract: when a callback distinguishes two `ZonedDateTime` rows that native `.equals()` groups @@ -144,3 +145,27 @@ that comparator/value combination. The earlier 497-test focused campaign and 12,345-test DB campaign describe the committed baseline, before this opposite-callback witness. A fresh exact-commit hostile review of the follow-up remains pending. + +## Read-only review of `0c236f448` + +The reviewer ran 54 focused and neighboring tests and found two enforcement +gaps, with no reported production mismatch. First, the public limit-one +checker allowed either equal-instant peer even though changelog #958 promises +source-key tie breaking for Collection and index iteration. The model now +sorts supplied instant labels and then source keys. A wrong-tie control fails +at the exact prefix comparison. The public witness admits the larger-key UTC +row first, then compares unindexed and optimized indexed limit-one snapshots +with the smaller-key +14:00 row; the index take spy confirms the optimized +path ran. + +Second, the opposite custom callback had distinguished display order on +add/build, but a later same-equality-class update could still leave a stale +comparator position. The new continuation changes Etc/UTC to UTC and checks +the resulting source-key tie, inclusive ranges, and empty cursors. A second +history begins with a callback tie, updates one row to UTC, then requires +the callback to put it first despite both display text and source keys +preferring the other row. Both BasicIndex and BTreeIndex run these histories. +The revised owner passes 35 tests. The law remains confined to this polyfill, +the named UTC aliases, fixed histories, and direct/indexed Collection cuts; +provider cursors and longer histories remain open. Fresh hostile review of +the next committed head is pending. diff --git a/docs/errors.md b/docs/errors.md index f33e22aa84..678dae164d 100644 --- a/docs/errors.md +++ b/docs/errors.md @@ -2449,3 +2449,13 @@ Development builds show this message, with `${...}` replaced by values: ```text LocalStorage sync belongs to its options Collection. ``` + + + +## Error 243: `query/compiler/mutable-list-refresh.ts` + +Development builds show this message, with `${...}` replaced by values: + +```text +An inner mutable-list revision lost its candidate +``` diff --git a/docs/guides/live-queries.md b/docs/guides/live-queries.md index b28a6aeb9c..8c3097460f 100644 --- a/docs/guides/live-queries.md +++ b/docs/guides/live-queries.md @@ -2647,6 +2647,15 @@ inArray(user.id, [1, 2, 3]) inArray(user.role, ['admin', 'moderator']) ``` +If a query captures a caller-owned array, it reads that array's current +members when the next source change is processed. That source change updates +the whole existing query result, including rows that did not change. This +applies when `inArray` filters rows, computes selected values, forms a join or +group, supplies an aggregate even inside another selected expression, filters +with `having`, or determines order. The +resulting values and order are published together. Editing the array alone +does not publish a new result. + #### `like(value, pattern)`, `ilike(value, pattern)` String pattern matching: ```ts diff --git a/packages/db-ivm/src/operators/groupBy.ts b/packages/db-ivm/src/operators/groupBy.ts index 5fa657d0fc..90e938163b 100644 --- a/packages/db-ivm/src/operators/groupBy.ts +++ b/packages/db-ivm/src/operators/groupBy.ts @@ -40,7 +40,14 @@ export function groupBy< T, K extends GroupKey, A extends Record>, ->(keyExtractor: (data: T) => K, aggregates: A = {} as A) { +>( + keyExtractor: (data: T) => K, + aggregates: A = {} as A, + mapInput?: ( + stream: IStreamBuilder, + project: (data: T) => KeyValue>, + ) => IStreamBuilder>>, +) { type ResultType = K & AggregatesReturnType const basicAggregates = Object.fromEntries( @@ -64,28 +71,29 @@ export function groupBy< const KEY_SENTINEL = Symbol(`original_group_key`) // First map to extract keys and pre-aggregate values - const withKeysAndValues = stream.pipe( - map((data) => { - const key = keyExtractor(data) - const keyString = serializeValue(key) + const project = (data: T) => { + const key = keyExtractor(data) + const keyString = serializeValue(key) - // Create values object with pre-aggregated values - const values: Record = {} + // Create values object with pre-aggregated values + const values: Record = {} - // Store the original key object - values[KEY_SENTINEL] = key + // Store the original key object + values[KEY_SENTINEL] = key - // Add pre-aggregated values - for (const [name, aggregate] of Object.entries(basicAggregates)) { - values[name] = aggregate.preMap(data) - } + // Add pre-aggregated values + for (const [name, aggregate] of Object.entries(basicAggregates)) { + values[name] = aggregate.preMap(data) + } - return [keyString, values] as KeyValue< - string, - Record - > - }), - ) + return [keyString, values] as KeyValue< + string, + Record + > + } + const withKeysAndValues = mapInput + ? mapInput(stream, project) + : stream.pipe(map(project)) // Then reduce to compute aggregates const reduced = withKeysAndValues.pipe( diff --git a/packages/db/package.json b/packages/db/package.json index 31a5f6eb61..e05430595f 100644 --- a/packages/db/package.json +++ b/packages/db/package.json @@ -23,7 +23,7 @@ "test": "vitest --run", "test:dist": "vitest run --config vitest.dist.config.ts", "test:facade-retention": "node --expose-gc --import tsx tests/facade-retention.probe.ts", - "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/temporal-index-order-oracle.test.ts tests/temporal-equality-alias-oracle.test.ts tests/custom-index-order-selection-oracle.test.ts tests/query/custom-index-window-selection-oracle.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", + "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/temporal-index-order-oracle.test.ts tests/temporal-equality-alias-oracle.test.ts tests/custom-index-order-selection-oracle.test.ts tests/query/custom-index-window-selection-oracle.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/captured-inarray-stages-oracle.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", "bench:nested-includes": "vitest bench tests/query/includes-performance.bench.ts --run", "test:indexed-db:package": "pnpm --filter @tanstack/db... build && vitest run --config vitest.indexed-db.package.config.ts", "test:indexed-db:browser": "playwright test --config playwright.indexed-db.config.ts", diff --git a/packages/db/src/collection/state.ts b/packages/db/src/collection/state.ts index 9f47f0261a..4962742bac 100644 --- a/packages/db/src/collection/state.ts +++ b/packages/db/src/collection/state.ts @@ -1129,7 +1129,11 @@ export class CollectionStateManager< // Application is now the point of no return. Event listeners run before // the receipts resolve, so a signal aborted from one of those listeners // must not cancel writes that are already becoming visible. + // A layout-only update can change the comparator's fractional index + // before these writes run. Rebuild after the batch rather than searching + // the old order with a comparator that already sees new positions. const deferOrder = + layoutChanged || committedSyncedTransactions.reduce( (count, transaction) => count + transaction.operations.length, 0, diff --git a/packages/db/src/query/builder/functions.ts b/packages/db/src/query/builder/functions.ts index c2095a5cb6..d5490a0837 100644 --- a/packages/db/src/query/builder/functions.ts +++ b/packages/db/src/query/builder/functions.ts @@ -42,7 +42,7 @@ type ExpressionLike = | Date | null | undefined - | Array + | ReadonlyArray export type CaseWhenValue = | ExpressionLike diff --git a/packages/db/src/query/compiler/evaluators.ts b/packages/db/src/query/compiler/evaluators.ts index cf4d5df51c..6b93b044c7 100644 --- a/packages/db/src/query/compiler/evaluators.ts +++ b/packages/db/src/query/compiler/evaluators.ts @@ -17,7 +17,7 @@ import { getParentContextProjectedValue, } from '../equality-value-identity.js' import { getNamespacedRouteMetadata } from './route-metadata.js' -import type { BasicExpression, Func, PropRef } from '../ir.js' +import type { Aggregate, BasicExpression, Func, PropRef } from '../ir.js' import type { NamespacedRow } from '../../types.js' /** @@ -32,6 +32,42 @@ function normalizeEqualityOperand(value: unknown): unknown { return isUint8Array(value) ? value : normalizeValue(value) } +export function isStablePrimitiveList(list: ReadonlyArray): boolean { + if (!Object.isFrozen(list)) return false + for (let index = 0; index < list.length; index++) { + const slot = Object.getOwnPropertyDescriptor(list, index) + // Frozen arrays can still have getters or mutable objects. A missing slot + // can also resolve through the prototype, so none may be cached. + if (!slot || !(`value` in slot)) return false + const item = slot.value + if ( + (typeof item === `object` && item !== null) || + typeof item === `function` + ) + return false + } + return true +} + +export function hasMutableInList( + expression: BasicExpression | Aggregate, + lists?: Set>, +): boolean { + if (expression.type !== `func` && expression.type !== `agg`) return false + const list = expression.args[1] + const values = list?.type === `val` ? list.value : undefined + let found = + expression.type === `func` && + expression.name === `in` && + Array.isArray(values) && + !isStablePrimitiveList(values) + if (found) lists?.add(values as Array) + for (const arg of expression.args) { + found = hasMutableInList(arg, lists) || found + } + return found +} + /** * Equality that follows PostgreSQL float semantics for `NaN`/invalid Dates: * such values are equal to one another and unequal to anything else. For all @@ -445,23 +481,22 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { const valueEvaluator = compiledArgs[0]! const arrayEvaluator = compiledArgs[1]! const list = func.args[1]! - if (list.type === `val` && Array.isArray(list.value)) { - // A constant list, such as join demand's key list, becomes a Set of - // normalized keys. Set membership uses SameValueZero, which matches - // `valuesEqual` on normalized operands: NaN equals NaN and -0 equals - // 0. Byte arrays stay out of the Set and compare by content, so no - // byte array is encoded as a string. + if ( + list.type === `val` && + Array.isArray(list.value) && + isStablePrimitiveList(list.value) + ) { + // An immutable primitive list, such as internally owned join demand, + // can be indexed once. Caller-owned arrays and mutable members must + // instead be read at each evaluation. SameValueZero on normalized + // keys preserves NaN and -0 equality. Byte values cannot match a + // primitive item and are never encoded here. const items = list.value.filter((item) => !isUnknown(item)) - const bytes = items.filter(isUint8Array) - const keys = new Set( - items.filter((item) => !isUint8Array(item)).map(normalizeValue), - ) + const keys = new Set(items.map(normalizeValue)) return (data) => { const value = valueEvaluator(data) if (isUnknown(value)) return null - return isUint8Array(value) - ? bytes.some((item) => areValuesEqual(item, value)) - : keys.has(normalizeValue(value)) + return isUint8Array(value) ? false : keys.has(normalizeValue(value)) } } return (data) => { diff --git a/packages/db/src/query/compiler/group-by.ts b/packages/db/src/query/compiler/group-by.ts index 0662ca9284..3e6be19b21 100644 --- a/packages/db/src/query/compiler/group-by.ts +++ b/packages/db/src/query/compiler/group-by.ts @@ -27,9 +27,11 @@ import { } from '../equality-value-identity.js' import { compileExpression, + hasMutableInList, isCaseWhenConditionTrue, toBooleanPredicate, } from './evaluators.js' +import { refreshMap } from './mutable-list-refresh.js' import { INCLUDES_PUBLIC_KEY, attachRouteMetadata, @@ -37,6 +39,7 @@ import { stripInternalCallbackMetadata, } from './route-metadata.js' import type { ValueIdentity } from '../equality-value-identity.js' +import type { MutableListRevision } from './mutable-list-refresh.js' import type { RouteMetadata } from './route-metadata.js' import type { Aggregate, @@ -304,6 +307,9 @@ export function processGroupBy( aggregateCollectionId?: string, mainSource?: string, sanitizeCallbackRows = false, + mutableRevision?: MutableListRevision, + mutableSelect = false, + mutableAggregateInput = false, ): NamespacedAndKeyedStream { const fields = createInternalGroupFields(groupByClause.length, selectClause) const virtualAggregates: Record = { @@ -461,92 +467,105 @@ export function processGroupBy( } // Apply the groupBy operator - pipeline = pipeline.pipe(groupBy(keyExtractor, aggregates)) + const refreshInputs = + mutableRevision !== undefined && + (mutableAggregateInput || + groupByClause.some((expr) => hasMutableInList(expr))) + pipeline = pipeline.pipe( + groupBy( + keyExtractor, + aggregates, + refreshInputs + ? (stream, project) => refreshMap(stream, mutableRevision, project) + : undefined, + ), + ) // Update $selected to handle GROUP BY results - pipeline = pipeline.pipe( - map(([, aggregatedRow]) => { - const finalResults: Record = {} - - if (selectClause) { - // Every field takes its place in select order. Plain aggregates are - // filled first; every other value reads them, the group keys, and - // the route's parent context. - for (const [alias, expr] of Object.entries(selectClause)) { - finalResults[alias] = - expr.type === `agg` ? aggregatedRow[alias] : undefined - } - evaluateWrappedAggregates( - finalResults, - aggregatedRow as Record, - wrappedAggExprs, - fields, - ) - } else { - // No SELECT clause - just use the group keys - for (let i = 0; i < groupByClause.length; i++) { - finalResults[`__key_${i}`] = aggregatedRow[fields.groupValues[i]!] - } + const projectGroup = ([, aggregatedRow]: [unknown, Record]) => { + const finalResults: Record = {} + + if (selectClause) { + // Every field takes its place in select order. Plain aggregates are + // filled first; every other value reads them, the group keys, and + // the route's parent context. + for (const [alias, expr] of Object.entries(selectClause)) { + finalResults[alias] = + expr.type === `agg` ? aggregatedRow[alias] : undefined } - - // Generate a simple key for the live collection using group values. - // In includes mode, add the complete route so correlated groups do not - // collide. - const route = mainSource - ? getGroupRoute(aggregatedRow, fields) - : undefined - const correlationKey = route?.correlationKey - const correlationRoute = mainSource - ? getCorrelationRouteIdentity(aggregatedRow, fields) - : undefined - const keyParts: Array = [] - const publicKeyParts: Array = [] + evaluateWrappedAggregates( + finalResults, + aggregatedRow, + wrappedAggExprs, + fields, + ) + } else { + // No SELECT clause - just use the group keys for (let i = 0; i < groupByClause.length; i++) { - keyParts.push(aggregatedRow[fields.groupKeys[i]!]) - publicKeyParts.push(aggregatedRow[fields.groupValues[i]!]) - } - if (correlationRoute !== undefined) { - keyParts.push(correlationRoute) + finalResults[`__key_${i}`] = aggregatedRow[fields.groupValues[i]!] } - const finalKey = singleGroup - ? correlationRoute !== undefined - ? `single_group_${serializeValue(correlationRoute)}` - : `single_group` - : keyParts.length === 1 - ? keyParts[0] - : serializeValue(keyParts) - const publicKey = singleGroup - ? `single_group` - : createPublicGroupKey(publicKeyParts) - - // When in includes mode, restore route metadata for output routing. - const resultRow: Record = { - ...(aggregatedRow as Record), - $selected: finalResults, - } - const virtual = (aggregatedRow as Record)[fields.virtual] as - RowVirtualMetadata | undefined - resultRow.$synced = virtual?.synced ?? true - resultRow.$hasPendingWrites = !resultRow.$synced - resultRow.$origin = ( - virtual?.hasLocal ? `local` : `remote` - ) satisfies VirtualOrigin - resultRow.$key = publicKey - resultRow.$collectionId = aggregateCollectionId ?? resultRow.$collectionId - if (mainSource && correlationKey !== undefined) { - attachPublicGroupKey(resultRow, publicKey) - attachRouteMetadata( - resultRow, - correlationKey, - route?.parentContext ?? null, - ) - } - return [mainSource ? finalKey : publicKey, resultRow] as [ - unknown, - Record, - ] - }), - ) + } + + // Generate a simple key for the live collection using group values. + // In includes mode, add the complete route so correlated groups do not + // collide. + const route = mainSource ? getGroupRoute(aggregatedRow, fields) : undefined + const correlationKey = route?.correlationKey + const correlationRoute = mainSource + ? getCorrelationRouteIdentity(aggregatedRow, fields) + : undefined + const keyParts: Array = [] + const publicKeyParts: Array = [] + for (let i = 0; i < groupByClause.length; i++) { + keyParts.push(aggregatedRow[fields.groupKeys[i]!]) + publicKeyParts.push(aggregatedRow[fields.groupValues[i]!]) + } + if (correlationRoute !== undefined) { + keyParts.push(correlationRoute) + } + const finalKey = singleGroup + ? correlationRoute !== undefined + ? `single_group_${serializeValue(correlationRoute)}` + : `single_group` + : keyParts.length === 1 + ? keyParts[0] + : serializeValue(keyParts) + const publicKey = singleGroup + ? `single_group` + : createPublicGroupKey(publicKeyParts) + + // When in includes mode, restore route metadata for output routing. + const resultRow: Record = { + ...aggregatedRow, + $selected: finalResults, + } + const virtual = aggregatedRow[fields.virtual] as + RowVirtualMetadata | undefined + resultRow.$synced = virtual?.synced ?? true + resultRow.$hasPendingWrites = !resultRow.$synced + resultRow.$origin = ( + virtual?.hasLocal ? `local` : `remote` + ) satisfies VirtualOrigin + resultRow.$key = publicKey + resultRow.$collectionId = aggregateCollectionId ?? resultRow.$collectionId + if (mainSource && correlationKey !== undefined) { + attachPublicGroupKey(resultRow, publicKey) + attachRouteMetadata( + resultRow, + correlationKey, + route?.parentContext ?? null, + ) + } + return [mainSource ? finalKey : publicKey, resultRow] as [ + unknown, + Record, + ] + } + pipeline = ( + mutableRevision && mutableSelect + ? refreshMap(pipeline, mutableRevision, projectGroup) + : pipeline.pipe(map(projectGroup)) + ) as NamespacedAndKeyedStream // Apply HAVING clauses if present if (havingClauses && havingClauses.length > 0) { @@ -562,22 +581,26 @@ export function processGroupBy( ) const compiledHaving = compileExpression(transformedHavingClause) - pipeline = pipeline.pipe( - filter(([, row]) => { - let selected = row.$selected as Record - if (fields.groupKeyRefs.length > 0) { - const withKeys: Record = { ...selected } - fields.groupKeyRefs.forEach((ref, i) => { - withKeys[ref] = row[fields.groupValues[i]!] - }) - selected = withKeys - } - const namespacedRow = getGroupEvaluationRow(row, fields, selected) - const result = compiledHaving(namespacedRow) - // Preserve each path's coercion for unchecked nonboolean IR values. - return singleGroup ? toBooleanPredicate(result) : result - }), - ) + const passesHaving = ([, row]: [unknown, Record]) => { + let selected = row.$selected as Record + if (fields.groupKeyRefs.length > 0) { + const withKeys: Record = { ...selected } + fields.groupKeyRefs.forEach((ref, i) => { + withKeys[ref] = row[fields.groupValues[i]!] + }) + selected = withKeys + } + const namespacedRow = getGroupEvaluationRow(row, fields, selected) + const result = compiledHaving(namespacedRow) + // Preserve each path's coercion for unchecked nonboolean IR values. + return singleGroup ? toBooleanPredicate(result) : result + } + pipeline = + mutableRevision && hasMutableInList(havingExpression) + ? (refreshMap(pipeline, mutableRevision, (entry) => + passesHaving(entry) ? entry : undefined, + ) as NamespacedAndKeyedStream) + : pipeline.pipe(filter(passesHaving)) } } diff --git a/packages/db/src/query/compiler/index.ts b/packages/db/src/query/compiler/index.ts index 91e125f364..7883deb4c4 100644 --- a/packages/db/src/query/compiler/index.ts +++ b/packages/db/src/query/compiler/index.ts @@ -63,6 +63,7 @@ import { normalizeValue } from '../../utils/comparison.js' import { codedMessage, devBuild } from '../../error-message.js' import { compileExpression, + hasMutableInList, isCaseWhenConditionTrue, toBooleanPredicate, } from './evaluators.js' @@ -70,6 +71,7 @@ import { processJoins, registerLazyDemandPlan } from './joins.js' import { containsAggregate, processGroupBy } from './group-by.js' import { getLazyLoadTargets } from './lazy-targets.js' import { processOrderBy } from './order-by.js' +import { refreshMap } from './mutable-list-refresh.js' import { crossJoinParentRoutes } from './parent-routes.js' import { queriesMatchForCaching } from './query-equivalence.js' import { @@ -88,6 +90,7 @@ import { processSelect } from './select.js' import type { ValueIdentity } from '../equality-value-identity.js' import type { CollectionSubscription } from '../../collection/subscription.js' import type { OrderByOptimizationInfo } from './order-by.js' +import type { MutableListRevision } from './mutable-list-refresh.js' import type { Aggregate, BasicExpression, @@ -367,11 +370,143 @@ export interface CompilationResult { } const valueIdentitiesByCache = new WeakMap() +const unfilteredQueryCaches = new WeakSet() +const mutableListRevisions = new WeakMap() function getCompilationValueIdentity(cache: QueryCache): ValueIdentity { return getOrCreate(valueIdentitiesByCache, cache, createValueIdentity) } +function collectMutableSelectLists( + value: unknown, + lists: Set>, + includeSubqueries = true, +): void { + if (value instanceof ConditionalSelect) { + for (const branch of value.branches) { + hasMutableInList(branch.condition, lists) + collectMutableSelectLists(branch.value, lists, includeSubqueries) + } + collectMutableSelectLists(value.defaultValue, lists, includeSubqueries) + } else if (value instanceof IncludesSubquery) { + if (!includeSubqueries) return + for (const where of value.parentFilters ?? []) { + hasMutableInList(getWhereExpression(where), lists) + } + collectMutableQueryLists(value.query, lists) + } else if (isBasicOrAggregateExpression(value)) { + if (value.type === `agg`) { + for (const arg of value.args) hasMutableInList(arg, lists) + } else { + hasMutableInList(value, lists) + } + } else if (value && typeof value === `object`) { + for (const child of Object.values(value)) { + collectMutableSelectLists(child, lists, includeSubqueries) + } + } +} + +function selectHasMutableAggregateInput(value: unknown): boolean { + if (value instanceof ConditionalSelect) { + return ( + value.branches.some( + (branch) => + selectHasMutableAggregateInput(branch.condition) || + selectHasMutableAggregateInput(branch.value), + ) || selectHasMutableAggregateInput(value.defaultValue) + ) + } + if (value instanceof IncludesSubquery) return false + if (isBasicOrAggregateExpression(value)) { + if (value.type !== `agg` && value.type !== `func`) return false + return ( + (value.type === `agg` && + value.args.some((arg) => hasMutableInList(arg))) || + value.args.some(selectHasMutableAggregateInput) + ) + } + return ( + value !== null && + typeof value === `object` && + Object.values(value).some(selectHasMutableAggregateInput) + ) +} + +function collectMutableQueryLists( + query: QueryIR, + lists: Set>, +): void { + for (const where of query.where ?? []) { + hasMutableInList(getWhereExpression(where), lists) + } + for (const join of query.join ?? []) { + hasMutableInList(join.on, lists) + if (join.from.type === `queryRef`) { + collectMutableQueryLists(join.from.query, lists) + } + } + for (const expression of query.groupBy ?? []) { + hasMutableInList(expression, lists) + } + for (const having of query.having ?? []) { + hasMutableInList(getHavingExpression(having), lists) + } + for (const { expression } of query.orderBy ?? []) { + hasMutableInList(expression, lists) + } + if (query.select) collectMutableSelectLists(query.select, lists) + if (query.from.type === `unionAll`) { + for (const branch of query.from.queries) { + collectMutableQueryLists(branch, lists) + } + } + for (const source of getFromSources(query.from)) { + if (source.type === `queryRef`) { + collectMutableQueryLists(source.query, lists) + } + } +} + +function selectHasMutableInList(select: QueryIR[`select`]): boolean { + const lists = new Set>() + collectMutableSelectLists(select, lists, false) + return lists.size > 0 +} + +const missingListSlot = Symbol(`missing inArray list slot`) + +function snapshotList(list: ReadonlyArray): { + values: Array + volatile: boolean +} { + const values: Array = [] + let volatile = false + for (let index = 0; index < list.length; index++) { + let slot: PropertyDescriptor | undefined + try { + slot = Object.getOwnPropertyDescriptor(list, index) + } catch { + volatile = true + } + if (!slot || !(`value` in slot)) volatile = true + const item = index in list ? list[index] : missingListSlot + values.push(normalizeValue(item)) + } + return { values, volatile } +} + +function sameListSnapshot( + left: ReturnType, + right: ReturnType, +): boolean { + if (left.volatile || right.volatile) return false + if (left.values.length !== right.values.length) return false + return left.values.every((value, index) => + Object.is(value, right.values[index]), + ) +} + /** * Compiles a query IR into a D2 pipeline * @param rawQuery The query IR to compile @@ -401,7 +536,51 @@ type CompileQueryArgs = [ ] export function compileQuery(...args: CompileQueryArgs): CompilationResult { - return compileQueryInternal(args, false) + // A mutable-list plan cannot reuse a subquery compiled earlier with source + // pushdown. Give this graph its own cache and share it only recursively. + const mutableLists = new Set>() + collectMutableQueryLists(args[0], mutableLists) + const dynamic = mutableLists.size > 0 + const cache = dynamic ? new WeakMap() : (args[8] ?? new WeakMap()) + args[8] = cache + if (dynamic) { + unfilteredQueryCaches.add(cache) + const lists = [...mutableLists] + let snapshots = lists.map(snapshotList) + let revision = 0 + // Raw source events clock every expression stage, including events that + // a join, filter, or nested QueryRef suppresses before that stage. + const inputs = [...new Set(Object.values(args[1]))] + let pulses = inputs[0]!.pipe(map(() => [0, 0] as [number, number])) + for (const input of inputs.slice(1)) { + pulses = pulses.pipe( + concatOperator(input.pipe(map(() => [0, 0] as [number, number]))), + ) + } + mutableListRevisions.set( + cache, + pulses.pipe( + reduce(() => { + const next = lists.map(snapshotList) + if ( + next.some( + (snapshot, index) => + !sameListSnapshot(snapshot, snapshots[index]!), + ) + ) { + revision++ + } + snapshots = next + return [[revision, 1]] + }), + ), + ) + } + try { + return compileQueryInternal(args, false) + } finally { + unfilteredQueryCaches.delete(cache) + } } function compileQueryInternal( @@ -435,9 +614,23 @@ function compileQueryInternal( if (!scopeValidated) validateQueryStructure(rawQuery) const rawSources = collectCollectionSources(rawQuery) + // A captured mutable membership list can change between source events. + // A mutable membership list needs the complete pre-WHERE relation. On-demand + // sources acquire unfiltered rows rather than only the current list's subset. + const refreshMutableWhere = + rawQuery.where?.some((where) => + hasMutableInList(getWhereExpression(where)), + ) ?? false + const mutableRevision = mutableListRevisions.get(cache) + // Optimize the query before compilation const { optimizedQuery, sourceWhereClauses: aliasWhereClauses } = - optimizeQuery(rawQuery) + unfilteredQueryCaches.has(cache) + ? { + optimizedQuery: rawQuery, + sourceWhereClauses: new Map>(), + } + : optimizeQuery(rawQuery) const sourceWhereClauses = keyWhereClausesBySource( rawQuery, aliasWhereClauses, @@ -612,6 +805,8 @@ function compileQueryInternal( joinsParentDirectly, valueIdentity, parentKeyStream, + unfilteredQueryCaches.has(cache), + mutableRevision, ) } @@ -639,15 +834,26 @@ function compileQueryInternal( // Process the WHERE clause if it exists if (query.where && query.where.length > 0) { - // Apply each WHERE condition as a filter (they are ANDed together) - for (const where of query.where) { - const whereExpression = getWhereExpression(where) - const compiledWhere = compileExpression(whereExpression) - pipeline = pipeline.pipe( - filter(([_key, namespacedRow]) => { - return toBooleanPredicate(compiledWhere(namespacedRow)) - }), + if (refreshMutableWhere && mutableRevision) { + const predicates = query.where.map((where) => + compileExpression(getWhereExpression(where)), ) + pipeline = refreshMap(pipeline, mutableRevision, (pair) => + predicates.every((predicate) => toBooleanPredicate(predicate(pair[1]))) + ? pair + : undefined, + ) as NamespacedAndKeyedStream + } else { + // Apply each WHERE condition as a filter (they are ANDed together). + for (const where of query.where) { + const whereExpression = getWhereExpression(where) + const compiledWhere = compileExpression(whereExpression) + pipeline = pipeline.pipe( + filter(([_key, namespacedRow]) => { + return toBooleanPredicate(compiledWhere(namespacedRow)) + }), + ) + } } } @@ -826,7 +1032,9 @@ function compileQueryInternal( aliasRemapping, ) - if (lazyTargets.length > 0) { + // Every child source event can refresh a mutable membership list, even + // when the child does not match the current parent route. + if (!unfilteredQueryCaches.has(cache) && lazyTargets.length > 0) { // 1. Mark child source as lazy so CollectionSubscriber skips initial full load for (const target of lazyTargets) { lazySources.add(target.sourceId) @@ -1057,7 +1265,12 @@ function compileQueryInternal( // An aggregate query computes its select values from each group, so it // does not project rows first. if (!selectHasAggregates && !(query.groupBy && query.groupBy.length > 0)) - pipeline = processSelect(pipeline, query.select, allInputs) + pipeline = processSelect( + pipeline, + query.select, + allInputs, + selectHasMutableInList(query.select) ? mutableRevision : undefined, + ) } else { // If no SELECT clause, create $selected with the main table data pipeline = pipeline.pipe( @@ -1126,6 +1339,9 @@ function compileQueryInternal( mainCollectionId, groupByMainSource, sourceCarriesInternalRouteState || includesRoutingFns.length > 0, + mutableRevision, + query.select ? selectHasMutableInList(query.select) : false, + query.select ? selectHasMutableAggregateInput(query.select) : false, ) } else if (selectHasAggregates) { // SELECT contains aggregates but no GROUP BY: implicit single-group aggregation @@ -1140,6 +1356,9 @@ function compileQueryInternal( mainCollectionId, groupByMainSource, sourceCarriesInternalRouteState || includesRoutingFns.length > 0, + mutableRevision, + query.select ? selectHasMutableInList(query.select) : false, + query.select ? selectHasMutableAggregateInput(query.select) : false, ) } @@ -1272,6 +1491,10 @@ function compileQueryInternal( resultLimit, query.offset, includesGroupKeyFn, + unfilteredQueryCaches.has(cache), + query.orderBy?.some(({ expression }) => hasMutableInList(expression)) + ? mutableRevision + : undefined, ).pipe( map(([key, [row, orderByIndex]]) => finalizeRow(key, row, orderByIndex)), ) as ResultStream diff --git a/packages/db/src/query/compiler/joins.ts b/packages/db/src/query/compiler/joins.ts index 65f1e38092..55b1f36460 100644 --- a/packages/db/src/query/compiler/joins.ts +++ b/packages/db/src/query/compiler/joins.ts @@ -27,7 +27,8 @@ import { getPropRefSourceAlias, requireCollectionSource, } from '../ir.js' -import { compileExpression } from './evaluators.js' +import { compileExpression, hasMutableInList } from './evaluators.js' +import { refreshMap } from './mutable-list-refresh.js' import { getLazyLoadTargets } from './lazy-targets.js' import { crossJoinParentRoutes } from './parent-routes.js' import { queriesMatchForCaching } from './query-equivalence.js' @@ -43,6 +44,7 @@ import { import type { ValueIdentity } from '../equality-value-identity.js' import type { CompileQueryFn } from './index.js' import type { OrderByOptimizationInfo } from './order-by.js' +import type { MutableListRevision } from './mutable-list-refresh.js' import type { BasicExpression, CollectionSourceRef, @@ -235,6 +237,8 @@ export function processJoins( mainSourceIsParentFiltered: boolean, valueIdentity: ValueIdentity, parentKeyStream?: KeyedStream, + disableLazyDemand = false, + mutableRevision?: MutableListRevision, ): NamespacedAndKeyedStream { let resultPipeline = pipeline const localBindings = new Set( @@ -266,6 +270,8 @@ export function processJoins( valueIdentity, localBindings, parentKeyStream, + disableLazyDemand, + mutableRevision, ) localBindings.add(joinClause.from.bindingId) } @@ -301,6 +307,8 @@ function processJoin( valueIdentity: ValueIdentity, localBindings: ReadonlySet, parentKeyStream?: KeyedStream, + disableLazyDemand = false, + mutableRevision?: MutableListRevision, ): NamespacedAndKeyedStream { const isCollectionRef = joinClause.from.type === `collectionRef` @@ -406,33 +414,34 @@ function processJoin( ) // Prepare both sides with the same key and raw-demand rules. - let mainPipeline = pipeline.pipe( - map(([currentKey, namespacedRow]) => - keyJoinInput( - currentKey, - namespacedRow, - mainSource, - `main`, - compiledMainExpressions, - routeJoinedSource, - valueIdentity, - ), - ), - ) - - let joinedPipeline = joinedInput.pipe( - map(([currentKey, row]) => - keyJoinInput( - currentKey, - wrapJoinedInputRow(joinedSource, row), - joinedSource, - `joined`, - compiledJoinedExpressions, - routeJoinedSource, - valueIdentity, - ), - ), - ) + const keyMain = ([currentKey, namespacedRow]: [unknown, NamespacedRow]) => + keyJoinInput( + currentKey, + namespacedRow, + mainSource, + `main`, + compiledMainExpressions, + routeJoinedSource, + valueIdentity, + ) + const keyJoined = ([currentKey, row]: [unknown, any]) => + keyJoinInput( + currentKey, + wrapJoinedInputRow(joinedSource, row), + joinedSource, + `joined`, + compiledJoinedExpressions, + routeJoinedSource, + valueIdentity, + ) + const refreshJoin = + mutableRevision !== undefined && hasMutableInList(joinClause.on) + let mainPipeline = refreshJoin + ? refreshMap(pipeline, mutableRevision, keyMain) + : pipeline.pipe(map(keyMain)) + let joinedPipeline = refreshJoin + ? refreshMap(joinedInput, mutableRevision, keyJoined) + : joinedInput.pipe(map(keyJoined)) // Apply the join operation if (![`inner`, `left`, `right`, `full`].includes(joinClause.type)) { @@ -465,7 +474,7 @@ function processJoin( aliasRemapping, ) - if (!limitedSubquery && lazyTargets.length > 0) { + if (!disableLazyDemand && !limitedSubquery && lazyTargets.length > 0) { // This join can be optimized by having the active collection // dynamically load keys into the lazy collection // based on the value of the joinKey and by looking up diff --git a/packages/db/src/query/compiler/mutable-list-refresh.ts b/packages/db/src/query/compiler/mutable-list-refresh.ts new file mode 100644 index 0000000000..0ef5562277 --- /dev/null +++ b/packages/db/src/query/compiler/mutable-list-refresh.ts @@ -0,0 +1,62 @@ +import { join, map, reduce, unkey } from '@tanstack/db-ivm' +import { codedMessage, devBuild } from '../../error-message.js' +import type { IStreamBuilder, KeyValue } from '@tanstack/db-ivm' + +/** A singleton relation whose value changes only when a captured list changes. */ +export type MutableListRevision = IStreamBuilder> + +/** + * Retain the output of an expression-bearing stage by its input key. A list + * revision visits every retained input, while an ordinary delta visits only + * its own key. The reduction owns the exact prior output: a retraction never + * reevaluates an old input against the newly edited list. + * + * A missing projection drops an input, which also makes this a retained filter. + */ +export function refreshMap( + input: IStreamBuilder>, + revision: MutableListRevision, + project: (entry: KeyValue) => O | undefined, + equivalent?: (previous: O, next: O) => boolean, +): IStreamBuilder { + const priorOutput = new WeakMap() + const projectRetained = (entry: KeyValue): O | undefined => { + const next = project(entry) + if (next === undefined) { + priorOutput.delete(entry) + return undefined + } + const previous = priorOutput.get(entry) + if (previous !== undefined && equivalent?.(previous, next)) { + return previous + } + priorOutput.set(entry, next) + return next + } + const candidates = input.pipe( + map((entry) => [0, entry] as KeyValue>), + ) + return candidates.pipe( + join(revision, `inner`), + map(([, [entry]]) => { + if (entry === null) { + throw new Error( + devBuild() && process.env.NODE_ENV !== `production` + ? `An inner mutable-list revision lost its candidate` + : codedMessage(243), + ) + } + return [entry[0], entry] as KeyValue> + }), + reduce((values) => + values.flatMap(([entry, multiplicity]) => { + if (multiplicity <= 0) return [] + const output = projectRetained(entry) + return output === undefined + ? [] + : [[output, multiplicity] as [O, number]] + }), + ), + unkey(), + ) +} diff --git a/packages/db/src/query/compiler/order-by.ts b/packages/db/src/query/compiler/order-by.ts index 3f3b304f06..778c3e73fa 100644 --- a/packages/db/src/query/compiler/order-by.ts +++ b/packages/db/src/query/compiler/order-by.ts @@ -14,11 +14,13 @@ import { ensureIndexForField } from '../../indexes/auto-index.js' import { findIndexForField } from '../../utils/index-optimization.js' import { devBuild } from '../../error-message.js' import { compileExpression } from './evaluators.js' +import { refreshMap } from './mutable-list-refresh.js' import { getSourceAliasesFromExpression } from './expressions.js' import { replaceAggregatesByRefs } from './group-by.js' import type { CompareOptions } from '../builder/types.js' import type { WindowOptions } from './types.js' import type { CompiledSingleRowExpression } from './evaluators.js' +import type { MutableListRevision } from './mutable-list-refresh.js' import type { OrderBy, OrderByClause, QueryIR, Select } from '../ir.js' import type { CollectionLike, @@ -57,6 +59,8 @@ export type OrderByOptimizationInfo = { joinedFilterSourceId?: string } +const CAPTURED_ORDER_VALUE = Symbol(`captured_order_value`) + /** * Processes the ORDER BY clause * Works with the new structure that has both namespaced row data and $selected @@ -73,6 +77,8 @@ export function processOrderBy( limit?: number, offset?: number, groupKeyFn?: (key: unknown, value: unknown) => unknown, + requiresFullSourceForMutableList = false, + mutableRevision?: MutableListRevision, ): IStreamBuilder> { // Pre-compile all order by expressions const compiledOrderBy = orderByClause.map((clause) => { @@ -111,6 +117,21 @@ export function processOrderBy( // Default case - no ordering return null } + // topK must retract the old sort key before a changed list supplies the + // new one. Retain the evaluated key beside each candidate, not in topK's + // callback over a raw row that can see only current list members. + if (mutableRevision) { + pipeline = refreshMap(pipeline, mutableRevision, ([key, row]) => [ + key, + { ...row, [CAPTURED_ORDER_VALUE]: valueExtractor(row) }, + ]) as NamespacedAndKeyedStream + } + const readOrderValue = mutableRevision + ? (row: NamespacedRow) => + (row as NamespacedRow & { [CAPTURED_ORDER_VALUE]: unknown })[ + CAPTURED_ORDER_VALUE + ] + : valueExtractor // Create a multi-property comparator that respects the order and direction of each property const compare = (a: unknown, b: unknown) => { @@ -146,7 +167,9 @@ export function processOrderBy( // When there's a limit, create orderByOptimizationInfo for top-K source // loading. Unbounded queries use subscription hints instead. A plan whose // local semantics cannot be established from a provider prefix records - // requiresFullSource and issues one filtered full-source acquisition. + // requiresFullSource and issues one full-source acquisition. A mutable + // captured list needs that acquisition without a source WHERE filter so + // later changes to previously unseen rows can trigger membership refresh. // We try to use an index on the FIRST orderBy column for lazy loading, // even for multi-column orderBy (using wider bounds on first column). // Skip this optimization when using grouped ordering (includes with limit), @@ -276,6 +299,9 @@ export function processOrderBy( sourceHoldsAllRows: followRefResult.collection.config.syncMode !== `on-demand`, requiresFullSource: + // An unseen source delete must still trigger a captured mutable + // membership list, so a finite prefix cannot own this subscription. + requiresFullSourceForMutableList || sourceOrderBy.some( ({ compareOptions }) => compareOptions.stringSort === `custom`, ) || @@ -321,7 +347,7 @@ export function processOrderBy( // otherwise use the standard global ordering operator. if (groupKeyFn) { return pipeline.pipe( - groupedOrderByWithFractionalIndex(valueExtractor, { + groupedOrderByWithFractionalIndex(readOrderValue, { limit, offset, comparator: compare, @@ -346,7 +372,7 @@ export function processOrderBy( // Use fractional indexing and return the tuple [value, index] return pipeline.pipe( - orderByWithFractionalIndex(valueExtractor, { + orderByWithFractionalIndex(readOrderValue, { limit, offset, comparator: compare, diff --git a/packages/db/src/query/compiler/select.ts b/packages/db/src/query/compiler/select.ts index f9db0515be..88f0ef799b 100644 --- a/packages/db/src/query/compiler/select.ts +++ b/packages/db/src/query/compiler/select.ts @@ -7,7 +7,9 @@ import { } from '../ir.js' import { UnsafeAliasPathError } from '../../errors.js' import { compileExpression, isCaseWhenConditionTrue } from './evaluators.js' +import { refreshMap } from './mutable-list-refresh.js' import type { BasicExpression, Select, SelectValueExpression } from '../ir.js' +import type { MutableListRevision } from './mutable-list-refresh.js' import type { KeyedStream, NamespacedAndKeyedStream, @@ -146,14 +148,19 @@ export function processSelect( pipeline: NamespacedAndKeyedStream, select: Select, _allInputs: Record, + mutableRevision?: MutableListRevision, ): NamespacedAndKeyedStream { if (!isNestedSelectObject(select)) { const compiled = compileSelectValue(select as SelectValueExpression) - return pipeline.pipe( - map(([key, namespacedRow]) => [ - key, - { ...namespacedRow, $selected: compiled(namespacedRow) }, - ]), + const project = ([key, namespacedRow]: [unknown, NamespacedRow]) => + [key, { ...namespacedRow, $selected: compiled(namespacedRow) }] as [ + unknown, + NamespacedRow, + ] + return ( + mutableRevision + ? refreshMap(pipeline, mutableRevision, project) + : pipeline.pipe(map(project)) ) as NamespacedAndKeyedStream } @@ -162,7 +169,12 @@ export function processSelect( addFromObject([], select, ops) - return pipeline.pipe(map((row) => processRow(row, ops))) + const project = (row: [unknown, NamespacedRow]) => processRow(row, ops) + return ( + mutableRevision + ? refreshMap(pipeline, mutableRevision, project) + : pipeline.pipe(map(project)) + ) as NamespacedAndKeyedStream } function compileSelectObject( diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 310a6df4ed..e6bee3aa73 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -744,6 +744,45 @@ lifecycle remains live. Cancellation and release are not data mutations. The immutable-demand boundary matrix checks direct and deferred sync startup, adapter return and asynchronous settlement, cancellation, and release identity. +A caller-owned array captured by a query's `inArray` expression is a query +input, not permission to mutate submitted request data. Editing that array +alone publishes no query change. At the next source event, the query must +evaluate its current members and bring its whole existing result into line, +including rows untouched by that event. The rule applies when membership is +read in `where`, a join operand, `select`, a group key or aggregate argument, +`having`, or an order expression. One source transaction publishes the new +result coherently; changing a projected value, group, or order cannot leave +the old stage output behind or expose an intermediate public result. + +The compiler derives one list revision from raw source events and shares it +across the compiled query graph. A list change makes each affected expression +stage revisit its retained inputs; an ordinary delta with an unchanged list +visits only the inputs it changes. Each stage retains its prior output for +retraction before it evaluates the new membership. JOIN, WHERE, SELECT, +GROUP BY and aggregate input, HAVING, and ORDER use this rule. The groupBy +operator accepts an optional mapped-input hook for the same revision. These +mechanics explain the receiving paths; they do not prove a general work bound. +Selected-expression inspection must descend through wrappers to aggregate +arguments. For example, `add(sum(caseWhen(inArray(...))), 10)` still needs +the groupBy input refresh when membership changes. + +At the public Collection boundary, a layout-only order change defers order +maintenance until the sync batch has applied. Otherwise a comparator reading +new fractional positions can search an old order and duplicate a source row +in a limited result. + +Any source request derived from the query still obeys the immutable +request-data boundary above. The compiler and pooled-query paths preserve +the captured-list law when they choose where to filter or retain source +rows. For a graph with captured mutable membership, on-demand source +acquisition is unfiltered. The compiler disables predicate pushdown and +keyed lazy join and include demand for that graph. An ordered plan also +acquires the full source without order or limit hints. An event on a source +row outside the current match, join key, child route, or ordered prefix must +still clock the list's current members. This can increase source transfer +and retained rows; controlled in-memory receivers establish the path, not +a real provider's bandwidth. + A Collection subscription installs each logical subset owner before it calls the source adapter. Reentrant release during `loadSubset` therefore retires the logical owner at once, but physical release waits until the adapter returns and @@ -937,7 +976,9 @@ if it rejects, an explicit retry must also start at offset zero without a cursor. The same holds after any later ordered request fails or is canceled: the adapter may already have written only part of its response, so those rows cannot establish a continuation boundary. The next explicit retry starts from -the source as one authoritative filtered full-source request. Core cannot know +the source as one authoritative full-source request. The request retains the +ordinary plan's source predicate, but a captured mutable-list graph requests +unfiltered source rows. Core cannot know which rows a failed request wrote, and a successful limited request proves neither how many authoritative rows it applied nor source exhaustion. Recovery therefore does not infer a safe finite prefix from local row count or boundary @@ -976,8 +1017,9 @@ before adapter code runs, and reentrant truncate or disposal stops the current retirement pass. No copied rows or additional cursor history are retained. A visible delete or source-order change after an otherwise settled, -expressible window uses a narrower authoritative path. Core reissues the -ordered prefix from the start with the query predicate, order terms, and +expressible window that does not require full-source acquisition uses a +narrower authoritative path. Core reissues the ordered prefix from the start +with the query predicate, order terms, and `limit = offset + limit`; the transport `offset` is omitted because this acquisition starts at the provider's source prefix. It then reacquires the first-column tie boundary and any required refill before publishing. Only the @@ -1024,7 +1066,8 @@ remains bounded by `orderBy` and `limit`; local delivery cardinality is a separate observation. Bounded repair is available only when a provider prefix is sufficient for the -local plan. Core requires full-source recovery for an indirect order +local plan. A captured mutable `inArray` list excludes bounded repair. Core +requires full-source recovery for an indirect order expression, a non-root ordered source, an inner or right join, a residual or cross-alias predicate, a functional predicate, grouping, `having`, functional `having`, or `distinct`. These cases can discard or reorder an otherwise valid @@ -1043,9 +1086,10 @@ prefix retains the established full-source fallback. A custom string comparator is a local ordering contract. If any resolved source order term uses `stringSort: 'custom'`, bounded and unbounded queries -use one authoritative filtered full-source request for the first acquisition -and every replacement: the source predicate remains, while `orderBy`, `limit`, -`offset`, and cursor are omitted. Core then sorts locally and uses ascending +use one authoritative full-source request for the first acquisition +and every replacement: the ordinary source predicate remains, while `orderBy`, +`limit`, `offset`, and cursor are omitted. A captured mutable-list graph uses +no source predicate, as above. Core then sorts locally and uses ascending public-key order for comparator-equal rows, including when the primary term is descending. It never interprets the comparator function as a stable backend collation identity or @@ -1317,7 +1361,9 @@ preload. An initial ordered load also has a synchronous observation cut. The initial load is an ordinary ordered request or, for a plan that requires the full -source, its first filtered full-source request. The cut holds when the first +source, its first full-source request. An ordinary plan retains its source +predicate; a captured mutable-list graph acquires unfiltered rows. The cut +holds when the first subscriber or preload starts the live-query Collection's sync run, or when its source Collections are eager. A sync run that started earlier resumes deferred on-demand acquisition through the subscription's restart path, which settles @@ -1514,6 +1560,12 @@ create recursive Collection machinery. counts as a preload. A deferring live query keeps its sources from garbage collection but is not counted in their `subscriberCount` or `subscribers:change`, which adapters read as a request for live data. +16. **Captured membership:** editing a caller-owned `inArray` list does not + publish by itself. After the next source event, the complete existing + live-query result reflects the list's current members at every expression + stage, including rows untouched by that event. Value, group, join, and + order changes publish coherently. Submitted source request data remains + immutable. ## Glossary @@ -1547,30 +1599,32 @@ keep the meanings defined there. ## Executable contracts -| Contract | Test suite | -| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | -| State equivalence, route lifecycle, transition history, and batch partition | `packages/db/tests/query/includes-oracle.property.test.ts` | -| Joined multiplicity, alias identity, and null-key normalization | `packages/db/tests/query/includes-query-shape-oracle.test.ts` | -| Alias reuse, captured bindings, public keys, and `SourceId` preservation | (`packages/db/tests/query/includes-oracle.property.test.ts`, `packages/db/tests/query/includes-alias-shadowing-oracle.test.ts`) | -| Same-scope and union alias rejection | `packages/db/tests/query/validate-aliases.test.ts` | -| Demand, cancellation, and progressive timing | `packages/db/tests/query/includes-temporal-oracle.test.ts` | -| Optimistic confirmation, rollback, and later reactivity | `packages/db/tests/query/includes-optimistic-oracle.property.test.ts` | -| Coherent layered publication | `packages/db/tests/query/includes-publication-oracle.test.ts` | -| Collection facades, event coherence, and route activation | `packages/db/tests/query/includes-collection-oracle.property.test.ts` | -| Correlated physical work | `packages/db/tests/query/includes-work-counter-oracle.test.ts` | -| Constructed and retained facades in a nested Collection tree | `packages/db/tests/query/includes-space-oracle.test.ts` | -| Route-context discovery and transport across recursive and join boundaries | `packages/db/tests/query/includes-context-transport-oracle.test.ts` | -| Functional projection input boundaries, timing, and output preservation | `packages/db/tests/query/includes-functional-projection-oracle.test.ts` | -| Functional input rejection and inline alternatives | `packages/db/tests/query/includes-functional-input-boundary.test.ts` | -| Public-container descriptors and reference-key matches across internal query stages | `packages/db/tests/query/public-container-copy.test.ts` | -| Cross-formulation equivalence and reference-sensitive route identity | `packages/db/tests/query/includes-cross-formulation-oracle.property.test.ts` | -| Query-db ownership | `packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts` | -| Failed replay retention, peer isolation, and explicit consumer-only recovery | `packages/db/tests/query/replay-failure-boundary.test.ts` | -| Replay lease balance, reference-counted peers, and failed-start recovery | `packages/db/tests/replay-adapter-ownership.test.ts` | -| Reachable nested shape | `packages/query-db-collection/tests/includes-work-counter-oracle.test.ts` | -| Cleanup-start invalidation, settlement, and restart admission | `packages/db/tests/collection-cleanup-restart-oracle.test.ts` | -| Pooled live queries match their live-query Collection, including cleanup | `packages/db/tests/query/pooled-live-query-oracle.property.test.ts` | -| Deferred acquisition before a subscriber or preload, readiness, and the synchronous cut | `packages/db/tests/live-query-deferred-acquisition-oracle.test.ts` | +| Contract | Test suite | +| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | +| State equivalence, route lifecycle, transition history, and batch partition | `packages/db/tests/query/includes-oracle.property.test.ts` | +| Joined multiplicity, alias identity, and null-key normalization | `packages/db/tests/query/includes-query-shape-oracle.test.ts` | +| Alias reuse, captured bindings, public keys, and `SourceId` preservation | (`packages/db/tests/query/includes-oracle.property.test.ts`, `packages/db/tests/query/includes-alias-shadowing-oracle.test.ts`) | +| Same-scope and union alias rejection | `packages/db/tests/query/validate-aliases.test.ts` | +| Demand, cancellation, and progressive timing | `packages/db/tests/query/includes-temporal-oracle.test.ts` | +| Optimistic confirmation, rollback, and later reactivity | `packages/db/tests/query/includes-optimistic-oracle.property.test.ts` | +| Coherent layered publication | `packages/db/tests/query/includes-publication-oracle.test.ts` | +| Collection facades, event coherence, and route activation | `packages/db/tests/query/includes-collection-oracle.property.test.ts` | +| Correlated physical work | `packages/db/tests/query/includes-work-counter-oracle.test.ts` | +| Constructed and retained facades in a nested Collection tree | `packages/db/tests/query/includes-space-oracle.test.ts` | +| Route-context discovery and transport across recursive and join boundaries | `packages/db/tests/query/includes-context-transport-oracle.test.ts` | +| Functional projection input boundaries, timing, and output preservation | `packages/db/tests/query/includes-functional-projection-oracle.test.ts` | +| Functional input rejection and inline alternatives | `packages/db/tests/query/includes-functional-input-boundary.test.ts` | +| Public-container descriptors and reference-key matches across internal query stages | `packages/db/tests/query/public-container-copy.test.ts` | +| Cross-formulation equivalence and reference-sensitive route identity | `packages/db/tests/query/includes-cross-formulation-oracle.property.test.ts` | +| Query-db ownership | `packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts` | +| Failed replay retention, peer isolation, and explicit consumer-only recovery | `packages/db/tests/query/replay-failure-boundary.test.ts` | +| Replay lease balance, reference-counted peers, and failed-start recovery | `packages/db/tests/replay-adapter-ownership.test.ts` | +| Reachable nested shape | `packages/query-db-collection/tests/includes-work-counter-oracle.test.ts` | +| Cleanup-start invalidation, settlement, and restart admission | `packages/db/tests/collection-cleanup-restart-oracle.test.ts` | +| Captured `inArray` membership at source publication, including ordered, join, and include cuts | `packages/db/tests/query/where-predicate-publication-oracle.property.test.ts` | +| Captured `inArray` membership in SELECT, ORDER, GROUP, aggregates, HAVING, JOIN, and QueryRef stages | `packages/db/tests/query/captured-inarray-stages-oracle.test.ts` | +| Pooled live queries match their live-query Collection, including mutable residual admission and cleanup | `packages/db/tests/query/pooled-live-query-oracle.property.test.ts` | +| Deferred acquisition before a subscriber or preload, readiness, and the synchronous cut | `packages/db/tests/live-query-deferred-acquisition-oracle.test.ts` | Each oracle identifies the first divergent checkpoint and compares either the whole result or one exact structural difference. Correlated-materialization diff --git a/packages/db/src/query/live/subset-demand-controller.ts b/packages/db/src/query/live/subset-demand-controller.ts index 9d80b9047b..2f09ea7ad0 100644 --- a/packages/db/src/query/live/subset-demand-controller.ts +++ b/packages/db/src/query/live/subset-demand-controller.ts @@ -237,7 +237,12 @@ function createSegment( plan: LazyDemandPlan, keys: Map, ): DemandSegment { - const where = inArray(new PropRef(plan.path), [...keys.values()]) + // This controller owns the request's array snapshot. Freezing it lets the + // evaluator index primitive demand keys once without freezing the values or + // changing the caller-owned array semantics of public inArray expressions. + const keyList = [...keys.values()] + Object.freeze(keyList) + const where = inArray(new PropRef(plan.path), keyList) return { keys, where, diff --git a/packages/db/src/query/pooled-live-query.ts b/packages/db/src/query/pooled-live-query.ts index 9f87605ff3..32bfe51db6 100644 --- a/packages/db/src/query/pooled-live-query.ts +++ b/packages/db/src/query/pooled-live-query.ts @@ -6,7 +6,11 @@ import { isVirtualPropName } from '../virtual-props.js' import { getPersistedReadinessSource } from '../persisted-readiness.js' import { getWhereExpression } from './ir.js' import { equalityConjunct, equalityKey, readPath } from './equality-conjunct.js' -import { compileExpression, toBooleanPredicate } from './compiler/evaluators.js' +import { + compileExpression, + hasMutableInList, + toBooleanPredicate, +} from './compiler/evaluators.js' import { buildCompareOptions } from './compiler/order-by.js' import { createLiveQueryCollection } from './live-query-collection.js' import type { BasicExpression, OrderBy, QueryIR } from './ir.js' @@ -396,6 +400,10 @@ function poolableShape(query: QueryIR): PoolableShape | undefined { return undefined } } + // A source event in another partition must still refresh membership when a + // captured list changes. The shared group does not notify this view there. + if (residual.some((expression) => hasMutableInList(expression))) + return undefined // A partition needs at least one equality to group by. if (conjuncts.length === 0) return undefined const orderBy = query.orderBy?.length ? query.orderBy : undefined diff --git a/packages/db/tests/fixtures/error-site-messages.json b/packages/db/tests/fixtures/error-site-messages.json index 38d4b408a1..375dc16bc6 100644 --- a/packages/db/tests/fixtures/error-site-messages.json +++ b/packages/db/tests/fixtures/error-site-messages.json @@ -923,5 +923,10 @@ "file": "local-storage.ts", "template": "LocalStorage sync belongs to its options Collection.", "literals": ["LocalStorage sync belongs to its options Collection."] + }, + "243": { + "file": "query/compiler/mutable-list-refresh.ts", + "template": "An inner mutable-list revision lost its candidate", + "literals": ["An inner mutable-list revision lost its candidate"] } } diff --git a/packages/db/tests/query/captured-inarray-stages-oracle.test.ts b/packages/db/tests/query/captured-inarray-stages-oracle.test.ts new file mode 100644 index 0000000000..db6e1f2c26 --- /dev/null +++ b/packages/db/tests/query/captured-inarray-stages-oracle.test.ts @@ -0,0 +1,510 @@ +/** + * # Where must captured membership be refreshed? + * + * Law: a caller-owned array passed to `inArray` remains query input after + * compilation. Editing it alone need not publish. After the next delivered + * source transaction, every existing public result uses its current members, + * including rows that the transaction did not touch. The fixed histories here + * exercise numeric membership in projection, ordering, grouping, plain and + * wrapped aggregate arguments, HAVING, and either operand of a join equality. + * They establish these placements for one unrelated source update, not all + * legal histories. + * + * Model: the small plain arrays below calculate membership with JavaScript + * `includes` at each checkpoint. Their expected values do not use compiled + * expressions or D2 output. The source grammar has three rows: values 1 and 2 + * are untouched, while an update to value 3 clocks the change. The joined + * grammar adds one TRUE flag. Public live-query rows and one subscriber batch + * are the observations, immediately after preload and the source commit. + * A compiler that only reevaluates the changed row fails at the second cut. + */ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { BasicIndex } from '../../src/indexes/basic-index.js' +import { + add, + caseWhen, + count, + eq, + inArray, + max, + min, + sum, +} from '../../src/query/builder/functions.js' +import { createLiveQueryCollection } from '../../src/query/index.js' +import { mockSyncCollectionOptions, withOracleCleanup } from '../utils.js' + +type Row = { id: string; v: number; revision: number } +const originalRows: Array = [ + { id: `one`, v: 1, revision: 0 }, + { id: `two`, v: 2, revision: 0 }, + { id: `trigger`, v: 3, revision: 0 }, +] +let collectionSerial = 0 + +function makeRows() { + return createCollection( + mockSyncCollectionOptions({ + id: `captured-inarray-stages-${collectionSerial++}`, + getKey: (row) => row.id, + initialData: originalRows, + }), + ) +} + +function clockRows(source: ReturnType): void { + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `trigger`, v: 3, revision: 1 }, + }) + source.utils.commit() +} + +describe(`captured inArray query-stage oracle`, () => { + it(`refreshes every projected member in one public transaction`, async () => { + const source = makeRows() + const members = [1] + const live = createLiveQueryCollection((q) => + q.from({ row: source }).select(({ row }) => ({ + id: row.id, + member: inArray(row.v, members), + })), + ) + const batches: Array> = + [] + let unsubscribe: (() => void) | undefined + await withOracleCleanup(async () => { + await live.preload() + const subscription = live.subscribeChanges( + (changes) => { + // Publication is one coherent cut: the callback can read the whole + // new result while it receives the changed projected rows. + expect( + live.toArray + .map((row) => ({ id: row.id, member: row.member })) + .sort((a, b) => a.id.localeCompare(b.id)), + ).toEqual( + originalRows + .map((row) => ({ id: row.id, member: members.includes(row.v) })) + .sort((a, b) => a.id.localeCompare(b.id)), + ) + batches.push( + changes.map((change) => ({ + type: change.type, + id: change.value.id, + member: change.value.member, + })), + ) + }, + { includeInitialState: false }, + ) + unsubscribe = () => subscription.unsubscribe() + const expected = () => + originalRows.map((row) => ({ + id: row.id, + member: members.includes(row.v), + })) + const actual = () => + live.toArray + .map((row) => ({ id: row.id, member: row.member })) + .sort((a, b) => a.id.localeCompare(b.id)) + expect(actual()).toEqual( + expected().sort((a, b) => a.id.localeCompare(b.id)), + ) + + members[0] = 2 + clockRows(source) + expect(actual()).toEqual( + expected().sort((a, b) => a.id.localeCompare(b.id)), + ) + expect(batches).toHaveLength(1) + expect( + batches[0] + ?.filter((change) => change.id !== `trigger`) + .sort((a, b) => a.id.localeCompare(b.id)), + ).toEqual([ + { type: `update`, id: `one`, member: false }, + { type: `update`, id: `two`, member: true }, + ]) + }, [() => unsubscribe?.(), () => live.cleanup(), () => source.cleanup()]) + }) + + it(`reorders a retained limited result`, async () => { + const source = makeRows() + const members = [1] + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => inArray(row.v, members), `desc`) + .limit(1), + ) + await withOracleCleanup(async () => { + await live.preload() + const expected = () => + originalRows.find((row) => members.includes(row.v))?.id + expect(live.toArray.map((row) => row.id)).toEqual([expected()]) + members[0] = 2 + clockRows(source) + expect(live.toArray.map((row) => row.id)).toEqual([expected()]) + }, [() => live.cleanup(), () => source.cleanup()]) + }) + + it(`keeps a limited order unique when captured membership changes`, async () => { + const rows: Array = [ + { id: `a`, v: 1, revision: 0 }, + { id: `b`, v: 2, revision: 0 }, + { id: `c`, v: 3, revision: 0 }, + { id: `trigger`, v: 4, revision: 0 }, + ] + const source = createCollection( + mockSyncCollectionOptions({ + id: `captured-inarray-order-unique-${collectionSerial++}`, + getKey: (row) => row.id, + initialData: rows, + }), + ) + const members = [1] + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .select(({ row }) => ({ id: row.id })) + .orderBy(({ row }) => inArray(row.v, members), `desc`) + .limit(3), + ) + await withOracleCleanup(async () => { + await live.preload() + // The plain-array model sorts each row once by current membership, + // then key. The public limit must contain distinct source identities. + const expected = () => + [...rows] + .sort( + (a, b) => + Number(members.includes(b.v)) - Number(members.includes(a.v)) || + a.id.localeCompare(b.id), + ) + .slice(0, 3) + .map((row) => row.id) + expect(live.toArray.map((row) => row.id)).toEqual(expected()) + members[0] = 2 + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `trigger`, v: 4, revision: 1 }, + }) + source.utils.commit() + expect(live.toArray.map((row) => row.id)).toEqual(expected()) + }, [() => live.cleanup(), () => source.cleanup()]) + }) + + it(`requests an unbounded source for projected membership under an ordered limit`, async () => { + type RankedRow = { id: string; v: number; rank: number } + const rows: Array = [0, 1, 2, 3, 4].map((v) => ({ + id: String(v), + v, + rank: v, + })) + const requests: Array<{ + hasWhere: boolean + hasOrder: boolean + hasLimit: boolean + }> = [] + let begin!: () => void + let write!: (message: { type: `delete`; value: RankedRow }) => void + let commit!: () => unknown + const source = createCollection({ + id: `captured-inarray-projected-demand-${collectionSerial++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { + loadSubset: (options) => { + requests.push({ + hasWhere: options.where !== undefined, + hasOrder: options.orderBy !== undefined, + hasLimit: options.limit !== undefined, + }) + params.begin() + for (const row of rows) { + params.write({ type: `insert`, value: row }) + } + params.commit() + return true + }, + } + }, + }, + }) + const members = [1] + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .select(({ row }) => ({ + id: row.id, + member: inArray(row.v, members), + })) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + await withOracleCleanup(async () => { + await live.preload() + expect(requests[0]).toEqual({ + hasWhere: false, + hasOrder: false, + hasLimit: false, + }) + const expected = () => ({ id: `0`, member: members.includes(0) }) + expect( + live.toArray.map((row) => ({ id: row.id, member: row.member })), + ).toEqual([expected()]) + + members[0] = 0 + begin() + write({ type: `delete`, value: rows[4]! }) + commit() + expect( + live.toArray.map((row) => ({ id: row.id, member: row.member })), + ).toEqual([expected()]) + }, [() => live.cleanup(), () => source.cleanup()]) + }) + + it(`moves untouched rows between group keys`, async () => { + const source = makeRows() + const members = [1] + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .groupBy(({ row }) => inArray(row.v, members)) + .select(({ row }) => ({ total: sum(row.v) })), + ) + await withOracleCleanup(async () => { + await live.preload() + const expected = () => { + const totals = new Map() + for (const row of originalRows) { + const key = members.includes(row.v) + totals.set(key, (totals.get(key) ?? 0) + row.v) + } + return [...totals.values()].sort((a, b) => a - b) + } + const actual = () => + live.toArray.map((row) => row.total).sort((a, b) => a - b) + expect(actual()).toEqual(expected()) + members[0] = 2 + clockRows(source) + expect(actual()).toEqual(expected()) + }, [() => live.cleanup(), () => source.cleanup()]) + }) + + it(`recomputes an aggregate argument for untouched rows`, async () => { + const source = makeRows() + const members = [1] + const live = createLiveQueryCollection((q) => + q.from({ row: source }).select(({ row }) => ({ + total: sum(caseWhen(inArray(row.v, members), row.v, 0)), + })), + ) + await withOracleCleanup(async () => { + await live.preload() + const expected = () => + originalRows.reduce( + (total, row) => total + (members.includes(row.v) ? row.v : 0), + 0, + ) + expect(live.toArray.map((row) => row.total)).toEqual([expected()]) + members[0] = 2 + clockRows(source) + expect(live.toArray.map((row) => row.total)).toEqual([expected()]) + }, [() => live.cleanup(), () => source.cleanup()]) + }) + + it(`recomputes a mutable aggregate argument inside a selected expression`, async () => { + const source = makeRows() + const members = [1] + const live = createLiveQueryCollection((q) => + q.from({ row: source }).select(({ row }) => ({ + total: add(sum(caseWhen(inArray(row.v, members), row.v, 0)), 10), + })), + ) + await withOracleCleanup(async () => { + await live.preload() + // The outer addition changes no grouping rule: its aggregate argument + // still sums every current member of the retained source relation. + const expected = () => + 10 + + originalRows.reduce( + (total, row) => total + (members.includes(row.v) ? row.v : 0), + 0, + ) + expect(live.toArray.map((row) => row.total)).toEqual([expected()]) + members[0] = 2 + clockRows(source) + expect(live.toArray.map((row) => row.total)).toEqual([expected()]) + }, [() => live.cleanup(), () => source.cleanup()]) + }) + + it(`recomputes grouped SUM and min/max arguments in an untouched group`, async () => { + type GroupRow = Row & { group: string } + const rows: Array = [ + { id: `a1`, group: `a`, v: 1, revision: 0 }, + { id: `a2`, group: `a`, v: 2, revision: 0 }, + { id: `b1`, group: `b`, v: 3, revision: 0 }, + ] + const source = createCollection( + mockSyncCollectionOptions({ + id: `captured-inarray-group-aggregate-${collectionSerial++}`, + getKey: (row) => row.id, + initialData: rows, + }), + ) + const members = [1] + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .groupBy(({ row }) => row.group) + .select(({ row }) => ({ + group: row.group, + total: sum(caseWhen(inArray(row.v, members), row.v, 0)), + lowest: min(caseWhen(inArray(row.v, members), row.v, 100)), + highest: max(caseWhen(inArray(row.v, members), row.v, 0)), + })), + ) + await withOracleCleanup(async () => { + await live.preload() + // Each aggregate reads current membership over all source rows in its + // group, including group a when the source transaction touches group b. + const expected = () => + [`a`, `b`].map((group) => { + const groupRows = rows.filter((row) => row.group === group) + return { + group, + total: groupRows.reduce( + (total, row) => total + (members.includes(row.v) ? row.v : 0), + 0, + ), + lowest: Math.min( + ...groupRows.map((row) => + members.includes(row.v) ? row.v : 100, + ), + ), + highest: Math.max( + ...groupRows.map((row) => (members.includes(row.v) ? row.v : 0)), + ), + } + }) + const actual = () => + live.toArray + .map((row) => ({ + group: row.group, + total: row.total, + lowest: row.lowest, + highest: row.highest, + })) + .sort((a, b) => a.group.localeCompare(b.group)) + expect(actual()).toEqual(expected()) + members[0] = 2 + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `b1`, group: `b`, v: 3, revision: 1 }, + }) + source.utils.commit() + expect(actual()).toEqual(expected()) + }, [() => live.cleanup(), () => source.cleanup()]) + }) + + it(`refreshes HAVING after an unrelated group receives an update`, async () => { + const source = makeRows() + const members = [1] + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .groupBy(({ row }) => row.v) + .select(({ row }) => ({ v: row.v, n: count(row.id) })) + .having(({ $selected }) => inArray($selected.v, members)), + ) + await withOracleCleanup(async () => { + await live.preload() + const expected = () => + originalRows + .filter((row) => members.includes(row.v)) + .map((row) => row.v) + expect(live.toArray.map((row) => row.v)).toEqual(expected()) + members[0] = 2 + clockRows(source) + expect(live.toArray.map((row) => row.v)).toEqual(expected()) + }, [() => live.cleanup(), () => source.cleanup()]) + }) + + for (const sourceSide of [`main`, `joined`] as const) { + it(`refreshes membership in the ${sourceSide} JOIN operand`, async () => { + const source = makeRows() + const flags = createCollection( + mockSyncCollectionOptions<{ id: string; flag: boolean }>({ + id: `captured-inarray-join-flags-${collectionSerial++}`, + getKey: (row) => row.id, + initialData: [{ id: `true`, flag: true }], + }), + ) + const members = [1] + const live = createLiveQueryCollection((q) => + sourceSide === `main` + ? q + .from({ row: source }) + .innerJoin({ flag: flags }, ({ row, flag }) => + eq(inArray(row.v, members), flag.flag), + ) + : q + .from({ flag: flags }) + .innerJoin({ row: source }, ({ flag, row }) => + eq(flag.flag, inArray(row.v, members)), + ), + ) + await withOracleCleanup(async () => { + await live.preload() + const expected = () => + originalRows + .filter((row) => members.includes(row.v)) + .map((row) => row.id) + expect(live.toArray.map((row) => row.row.id)).toEqual(expected()) + members[0] = 2 + clockRows(source) + expect(live.toArray.map((row) => row.row.id)).toEqual(expected()) + }, [() => live.cleanup(), () => flags.cleanup(), () => source.cleanup()]) + }) + } + + it(`refreshes projected membership through a nested QueryRef`, async () => { + const source = makeRows() + const members = [1] + const live = createLiveQueryCollection((q) => { + const inner = q.from({ item: source }).select(({ item }) => ({ + id: item.id, + member: inArray(item.v, members), + })) + return q + .from({ row: inner }) + .where(({ row }) => eq(row.member, true)) + .select(({ row }) => ({ id: row.id })) + }) + await withOracleCleanup(async () => { + await live.preload() + const expected = () => + originalRows + .filter((row) => members.includes(row.v)) + .map((row) => row.id) + expect(live.toArray.map((row) => row.id)).toEqual(expected()) + members[0] = 2 + clockRows(source) + expect(live.toArray.map((row) => row.id)).toEqual(expected()) + }, [() => live.cleanup(), () => source.cleanup()]) + }) +}) diff --git a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts index 2ce03d0b18..947a6db157 100644 --- a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts +++ b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts @@ -9,7 +9,10 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' /** * # Which values does `in` find in a list? * - * Transfer law: The public `inArray` API describes array membership. The + * Transfer law: The public `inArray` API describes array membership. A + * compiled expression holding a caller-owned array reads that array's current + * members whenever it evaluates, even if the array or one of its items has + * changed since compilation. The * current evaluator returns UNKNOWN (`null`) for a null or undefined value * and FALSE for a list that is not an array. * Otherwise the result is TRUE when some list item equals the value under the @@ -55,13 +58,15 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * display but distinct fields, one shared object and a structurally equal * copy, `null` and `undefined`. Lists have zero to six * items and may repeat. The list is a constant `Value`, the shape that join - * demand builds, or a row field, so changing lists are also checked. + * demand builds, or a row field. Fixed histories also compile a caller-owned + * list, then replace a slot or mutate a Date or byte item before reevaluating. * * Production driver: compile `in` with the public IR and evaluate it for * each row, as a WHERE clause does. * * Refinement check: production and the model return the same value for every - * case. Each property runs a fixed campaign and a random or replayed campaign. + * case at each evaluation checkpoint, including after a list edit. Each + * property runs a fixed campaign and a random or replayed campaign. * A calibration test feeds the checker an evaluator that compares raw values * without normalization and requires the checker to reject it. * @@ -578,6 +583,54 @@ describe(`in evaluator`, () => { await fc.assert(property, oraclePropertyOptions(2000, PROPERTY)) }) + it(`reads replaced scalar list members after compilation`, () => { + const list: Array = [1] + const evaluate = compileExpression( + new Func(`in`, [new PropRef([`row`, `value`]), new Value(list)]), + ) + const checkCurrent = (value: unknown) => + expect(evaluate({ row: { value } })).toBe(referenceIn(value, list)) + + checkCurrent(1) + list[0] = 2 + checkCurrent(1) + checkCurrent(2) + list.push(3) + checkCurrent(3) + list.splice(0, 1) + checkCurrent(2) + }) + + it(`reads a Date item's current timestamp after compilation`, () => { + const dateItem = new Date(1) + const list = [dateItem] + const evaluate = compileExpression( + new Func(`in`, [new PropRef([`row`, `value`]), new Value(list)]), + ) + const checkCurrent = (value: unknown) => + expect(evaluate({ row: { value } })).toBe(referenceIn(value, list)) + + checkCurrent(1) + dateItem.setTime(2) + checkCurrent(1) + checkCurrent(2) + }) + + it(`reads a byte item's current content after compilation`, () => { + const byteItem = new Uint8Array([1, 2]) + const list = [byteItem] + const evaluate = compileExpression( + new Func(`in`, [new PropRef([`row`, `value`]), new Value(list)]), + ) + const checkCurrent = (value: unknown) => + expect(evaluate({ row: { value } })).toBe(referenceIn(value, list)) + + checkCurrent(new Uint8Array([1, 2])) + byteItem[1] = 3 + checkCurrent(new Uint8Array([1, 2])) + checkCurrent(new Uint8Array([1, 3])) + }) + it.each([ [`a Date and its timestamp`, 1000, [new Date(1000)], true], [`NaN and an invalid Date`, Number.NaN, [new Date(Number.NaN)], true], diff --git a/packages/db/tests/query/index-path-collision-oracle.test.ts b/packages/db/tests/query/index-path-collision-oracle.test.ts index e9e0caf36a..30e03b64c4 100644 --- a/packages/db/tests/query/index-path-collision-oracle.test.ts +++ b/packages/db/tests/query/index-path-collision-oracle.test.ts @@ -26,12 +26,15 @@ * A foreign Date subclass is the adjacent valid-domain control: its native * timestamp keeps the range eligible for index lookup across realms. * A further valid-Date history overrides `valueOf` in local and foreign - * subclasses while retaining native `getTime`. Its four relational predicates - * use intrinsic timestamps in the model, then compare public scan and indexed - * snapshots. Both index types must remain eligible for optimized-only reads. - * A subclass bound also reaches both operand positions. Date/number scan - * comparisons in both operand orders use the same model; mixed-domain range - * indexes make no eligibility claim here. + * subclasses while retaining native `getTime`. An ordinary Date bound and a + * subclass bound each reach all four effective relations; the subclass bound + * occupies both operand positions. The intrinsic-timestamp model compares + * public scan and indexed snapshots, and both index types must remain eligible + * for optimized-only reads. The reference captures each source timestamp + * before the rows enter the Collection, so later source-Date changes cannot + * change the expected answer. Date/number scan comparisons in both operand + * orders use the same model; mixed-domain range indexes make no eligibility + * claim here. * * Production driver: `currentStateAsChanges`, Collection subscriptions, and * live-query Collections run the real predicates and callback compilers. @@ -382,6 +385,15 @@ it.each([ const subclassBound = new LaterDate(1) const nativeGetTime = Date.prototype.getTime const boundTime = nativeGetTime.call(bound) + const subclassBoundTime = nativeGetTime.call(subclassBound) + // Capture the model before Collection admission. Neither the scan nor + // the indexed checkpoint may rederive its answer from mutable source + // Date objects handed to production. + const referenceRows = Object.freeze( + dateRows.map(({ id, v }) => + Object.freeze({ id, time: nativeGetTime.call(v) }), + ), + ) const field = new PropRef([`v`]) const predicates = [ { @@ -407,22 +419,42 @@ it.each([ { name: `gt with subclass bound`, where: gt(field, subclassBound), - accepts: (time: number) => time > boundTime, + accepts: (time: number) => time > subclassBoundTime, + }, + { + name: `gte with subclass bound`, + where: gte(field, subclassBound), + accepts: (time: number) => time >= subclassBoundTime, + }, + { + name: `lt with subclass bound`, + where: lt(field, subclassBound), + accepts: (time: number) => time < subclassBoundTime, }, { name: `lte with subclass bound`, where: lte(field, subclassBound), - accepts: (time: number) => time <= boundTime, + accepts: (time: number) => time <= subclassBoundTime, }, { name: `subclass bound less than field`, where: lt(subclassBound, field), - accepts: (time: number) => boundTime < time, + accepts: (time: number) => subclassBoundTime < time, + }, + { + name: `subclass bound less than or equal to field`, + where: lte(subclassBound, field), + accepts: (time: number) => subclassBoundTime <= time, + }, + { + name: `subclass bound greater than field`, + where: gt(subclassBound, field), + accepts: (time: number) => subclassBoundTime > time, }, { name: `subclass bound greater than or equal to field`, where: gte(subclassBound, field), - accepts: (time: number) => boundTime >= time, + accepts: (time: number) => subclassBoundTime >= time, }, ] const mixedPredicates = [ @@ -458,15 +490,15 @@ it.each([ register(() => collection.cleanup()) await collection.stateWhenReady() - // The model calls the native intrinsic, which ignores subclass valueOf. + // The model captured native timestamps, which ignore subclass valueOf. // The two overrides deliberately reverse their apparent native-operator // positions, while ordinary Dates control both sides and equality. expect(Number(earlierForeign)).toBe(100) expect(Number(dateRows[3]!.v)).toBe(-100) expect(Number(subclassBound)).toBe(-100) const expectedDateKeys = (accepts: (time: number) => boolean) => - dateRows - .filter(({ v }) => accepts(nativeGetTime.call(v))) + referenceRows + .filter(({ time }) => accepts(time)) .map(({ id }) => id) .sort() const snapshotKeys = ( @@ -516,6 +548,15 @@ it.each([ ).toHaveBeenCalled() lookup.mockClear() } + + // This hostile mutation crosses the bound after every public check. + // The expected keys stay fixed because the model recorded the original + // timestamps before production received the source rows. + const greaterKeys = expectedDateKeys((time) => time > boundTime) + expect(greaterKeys).toContain(`local-later`) + dateRows[3]!.v.setTime(-2) + expect(nativeGetTime.call(dateRows[3]!.v)).toBe(-2) + expect(expectedDateKeys((time) => time > boundTime)).toEqual(greaterKeys) }) }, ) diff --git a/packages/db/tests/query/indexes.test.ts b/packages/db/tests/query/indexes.test.ts index 9c47c0a890..169187b55e 100644 --- a/packages/db/tests/query/indexes.test.ts +++ b/packages/db/tests/query/indexes.test.ts @@ -286,14 +286,14 @@ describe(`Query Index Optimization`, () => { }) }) - it(`should use indexes for live queries with inArray conditions`, async () => { + it(`uses indexes for live queries with immutable inArray lists`, async () => { await withIndexTracking(collection, async (tracker) => { const liveQuery = createLiveQueryCollection({ query: (q: any) => q .from({ item: collection }) .where(({ item }: any) => - inArray(item.status, [`active`, `pending`]), + inArray(item.status, Object.freeze([`active`, `pending`])), ) .select(({ item }: any) => ({ id: item.id, diff --git a/packages/db/tests/query/load-subset-subquery.test.ts b/packages/db/tests/query/load-subset-subquery.test.ts index ead6b921d3..ee85a23cdd 100644 --- a/packages/db/tests/query/load-subset-subquery.test.ts +++ b/packages/db/tests/query/load-subset-subquery.test.ts @@ -117,8 +117,10 @@ const forwardingCases: ReadonlyArray = [ }, { name: `IN`, - build: (order) => inArray(order.id, [1, 2, 3]), - expected: inArray(new PropRef([`id`]), [1, 2, 3]), + // An immutable list can be forwarded once. A mutable captured list needs + // the full source so later edits can refresh existing query rows. + build: (order) => inArray(order.id, Object.freeze([1, 2, 3])), + expected: inArray(new PropRef([`id`]), Object.freeze([1, 2, 3])), }, { name: `NOT`, diff --git a/packages/db/tests/query/pooled-live-query-oracle.property.test.ts b/packages/db/tests/query/pooled-live-query-oracle.property.test.ts index 694f1ab8a1..f70d1a1152 100644 --- a/packages/db/tests/query/pooled-live-query-oracle.property.test.ts +++ b/packages/db/tests/query/pooled-live-query-oracle.property.test.ts @@ -63,17 +63,22 @@ * * Known omissions: on-demand and persisted sources, `DbClient` hydration, * Suspense, `select`, and every other clause keep the live-query Collection - * and are outside this owner. + * and are outside this owner. A captured mutable `inArray` list in a residual + * predicate also keeps the live-query Collection: a pooled group would not + * notify the view about a source event in another group, although that event + * must refresh the whole result using the list's current members. */ import { fc, test as fcTest } from '@fast-check/vitest' import { describe, expect, it } from 'vitest' import { createCollection } from '../../src/collection/index.js' import { createLiveQueryObserver } from '../../src/live-query-observer.js' +import { resolveLiveQueryValue } from '../../src/live-query-options.js' import { Query } from '../../src/query/builder/index.js' import { and, createLiveQueryCollection, eq, + inArray, not, } from '../../src/query/index.js' import { createPooledLiveQuery } from '../../src/query/pooled-live-query.js' @@ -897,6 +902,54 @@ async function runHistory(history: History): Promise { describe(`pooled live query oracle`, () => { if (requestedReplayProperty === undefined) { + it(`routes a mutable residual inArray list to a live-query Collection`, async () => { + type PooledMembershipRow = { + id: string + g: string + v: number + revision: number + } + const source = createCollection( + mockSyncCollectionOptions({ + id: `pooled-mutable-in-${serial++}`, + getKey: (row) => row.id, + initialData: [ + { id: `one`, g: `g`, v: 1, revision: 0 }, + { id: `two`, g: `g`, v: 2, revision: 0 }, + { id: `trigger`, g: `other`, v: 3, revision: 0 }, + ], + }), + ) + const members = [1] + const query = new Query() + .from({ r: source }) + .where(({ r }) => and(eq(r.g, `g`), inArray(r.v, members))) + const resolved = resolveLiveQueryValue(query)! + const observer = createLiveQueryObserver(resolved, { mode: `wholesale` }) + const stop = observer.subscribe(() => {}) + await withOracleCleanup(async () => { + await resolved.preload() + const expected = () => + [ + { id: `one`, v: 1 }, + { id: `two`, v: 2 }, + ] + .filter((row) => members.includes(row.v)) + .map((row) => row.id) + const check = () => + expect([...observer.getSnapshot().state!.keys()]).toEqual(expected()) + check() + members[0] = 2 + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `trigger`, g: `other`, v: 3, revision: 1 }, + }) + source.utils.commit() + check() + }, [stop, () => resolved.cleanup(), () => source.cleanup()]) + }) + // Generated fields are plain values; this pins a field whose getter // throws, which both paths treat as a row that does not match. it(`matches the live-query Collection when an eq path getter throws`, async () => { diff --git a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts index ea937fe552..6d37a1bad8 100644 --- a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts +++ b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts @@ -73,7 +73,30 @@ * and foreign Date subclasses with inherited `getTime` join the same * timestamp classes. A forged object with a borrowed subclass prototype does * not join a Date class. A separate unindexed public query checks the local - * borrowed-prototype case against Date inequality and object identity. + * borrowed-prototype case against Date inequality and object identity, + * including the zero timestamp class. + * + * Captured-list law: a caller-owned array passed to `inArray` remains live + * input. Editing that array alone need not publish. At the next delivered + * source event, the entire retained result, including untouched rows, uses + * its current members. The fixed model starts with authored source rows and + * filters them against the list at that checkpoint; it does not reuse the + * compiled predicate, source index, or emitted delta. The direct Collection + * snapshot, retained live query, ordered limit, on-demand source, join, + * filtered QueryRef, and correlated include witnesses compare public results + * with that model. On-demand ordered, joined, and include receivers check that + * the first acquisition is unfiltered when the captured list is mutable. + * The retained direct query also checks exact insert/delete + * events and no duplicate event when the list did not change. Separate fixed + * Date and byte-item cases change contents inside the captured list, then + * check the next source event. An ordered limit witness deletes a source row + * outside the old loaded prefix after a list edit; it requires the newly + * matching row to appear, so ordered acquisition must retain all source rows + * while this live list is captured. Joined and included on-demand sources use + * an unrelated source delete to prove that keyed lazy demand cannot hide the + * event. A 40-row work witness requires an unchanged + * list to inspect fewer than ten list slots and a changed list to revisit + * retained candidates; a constant-key all-row rescan fails the first bound. * * Calibration: pinned histories separate Kleene from two-valued logic, and a * checker test requires the two-valued answer to fail. Hostile production @@ -84,10 +107,16 @@ * reconciliation each passed the pre-existing `@tanstack/db` suite and fail * here. * - * Known omissions: comparison operators other than `eq`, `in`, `like`, - * Temporal and binary operands, joins, ordering, optimistic updates and - * deletes, truncate, and failed replay are outside this owner. The structural - * comparison, cold-join, and subscription replay oracles own those boundaries. + * Known omissions: the generated WHERE grammar still covers `eq`, `not`, + * `and`, and `or`, not other comparison operators, `like`, or arbitrary + * `inArray` histories. The fixed captured-list witnesses cover the named + * scalar, Date, byte, ordered (including unseen deletion), on-demand, joined, + * QueryRef, and include cuts; + * they do not establish those paths for all list values, source schedules, + * optimizers, or real providers. Temporal operands, general join and ordering + * laws, optimistic updates and deletes, truncate, and failed replay remain + * with their structural comparison, cold-join, ordered, and subscription + * replay owners. * Generated cleanup and restart histories for filtered subscribers remain * open; the lifecycle publication owner needs a predicate dimension to reach * them. The mock sync source is a controlled provider; this oracle claims only @@ -107,7 +136,14 @@ import { fc, test as fcTest } from '@fast-check/vitest' import { describe, expect, it } from 'vitest' import { createCollection } from '../../src/collection/index.js' import { BasicIndex } from '../../src/indexes/basic-index.js' -import { and, eq, not, or } from '../../src/query/builder/functions.js' +import { + and, + eq, + inArray, + materialize, + not, + or, +} from '../../src/query/builder/functions.js' import { Func, PropRef, Value } from '../../src/query/ir.js' import { createLiveQueryCollection } from '../../src/query/index.js' import { @@ -1325,6 +1361,881 @@ describe(`WHERE predicate publication oracle`, () => { ]) }) + it(`direct snapshots read current inArray members after a source update`, async () => { + type MembershipRow = { id: string; v: number; revision: number } + const source = createCollection( + mockSyncCollectionOptions({ + id: `where-publication-snapshot-in-${collectionSerial++}`, + getKey: (row) => row.id, + initialData: [ + { id: `one`, v: 1, revision: 0 }, + { id: `two`, v: 2, revision: 0 }, + ], + }), + ) + const members = [1] + const where = inArray(new PropRef([`v`]), members) + await withOracleCleanup(async () => { + await source.stateWhenReady() + const expected = (rows: ReadonlyArray) => + rows.filter((row) => members.includes(row.v)).map((row) => row.id) + const initialRows = [ + { id: `one`, v: 1, revision: 0 }, + { id: `two`, v: 2, revision: 0 }, + ] + const snapshotKeys = () => + source + .currentStateAsChanges({ where }) + ?.map((change) => change.key) + .sort() + expect(snapshotKeys()).toEqual(expected(initialRows)) + + members[0] = 2 + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `one`, v: 1, revision: 1 }, + }) + source.utils.write({ + type: `update`, + value: { id: `two`, v: 2, revision: 1 }, + }) + source.utils.commit() + expect(snapshotKeys()).toEqual( + expected(initialRows.map((row) => ({ ...row, revision: 1 }))), + ) + }, [() => source.cleanup()]) + }) + + it(`a source event refreshes untouched rows after a captured inArray list changes`, async () => { + type MembershipRow = { id: string; v: number; revision: number } + const source = createCollection( + mockSyncCollectionOptions({ + id: `where-publication-live-in-${collectionSerial++}`, + getKey: (row) => row.id, + autoIndex: `off`, + initialData: [ + { id: `one`, v: 1, revision: 0 }, + { id: `two`, v: 2, revision: 0 }, + { id: `trigger`, v: 3, revision: 0 }, + ], + }), + ) + const members = [1] + const modelRows = [ + { id: `one`, v: 1 }, + { id: `two`, v: 2 }, + { id: `trigger`, v: 3 }, + ] + const live = createLiveQueryCollection((q) => + q.from({ row: source }).where(({ row }) => inArray(row.v, members)), + ) + const events: Array<{ type: string; key: string }> = [] + let unsubscribe: (() => void) | undefined + await withOracleCleanup(async () => { + await live.preload() + const subscription = live.subscribeChanges( + (changes) => { + events.push( + ...changes.map((change) => ({ + type: change.type, + key: String(change.key), + })), + ) + }, + { includeInitialState: false }, + ) + unsubscribe = () => subscription.unsubscribe() + const expected = () => + modelRows + .filter((row) => members.includes(row.v)) + .map((row) => row.id) + .sort() + const check = () => expect([...live.keys()].sort()).toEqual(expected()) + check() + + members[0] = 2 + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `trigger`, v: 3, revision: 1 }, + }) + source.utils.commit() + check() + expect( + events.sort((left, right) => left.key.localeCompare(right.key)), + ).toEqual([ + { type: `delete`, key: `one` }, + { type: `insert`, key: `two` }, + ]) + events.length = 0 + + members.push(1) + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `trigger`, v: 3, revision: 2 }, + }) + source.utils.commit() + check() + expect(events).toEqual([{ type: `insert`, key: `one` }]) + events.length = 0 + + // A source update with no further list edit cannot republish the + // unchanged result merely because the reduction revisits every row. + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `trigger`, v: 3, revision: 3 }, + }) + source.utils.commit() + check() + expect(events).toEqual([]) + + source.utils.begin() + source.utils.write({ type: `delete`, key: `two` }) + source.utils.commit() + modelRows.splice( + modelRows.findIndex((row) => row.id === `two`), + 1, + ) + check() + expect(events).toEqual([{ type: `delete`, key: `two` }]) + }, [() => unsubscribe?.(), () => live.cleanup(), () => source.cleanup()]) + }) + + it(`a source event reads current Date and byte list-item contents`, async () => { + type MembershipRow = { + id: string + date: Date + bytes: Uint8Array + revision: number + } + const source = createCollection( + mockSyncCollectionOptions({ + id: `where-publication-mutable-items-${collectionSerial++}`, + getKey: (row) => row.id, + initialData: [1, 2, 3].map((value) => ({ + id: String(value), + date: new Date(value), + bytes: new Uint8Array([value]), + revision: 0, + })), + }), + ) + const dateMembers = [new Date(1)] + const byteMembers = [new Uint8Array([1])] + const dates = createLiveQueryCollection((q) => + q + .from({ row: source }) + .where(({ row }) => inArray(row.date, dateMembers)), + ) + const bytes = createLiveQueryCollection((q) => + q + .from({ row: source }) + .where(({ row }) => inArray(row.bytes, byteMembers)), + ) + await withOracleCleanup(async () => { + await Promise.all([dates.preload(), bytes.preload()]) + expect([...dates.keys()]).toEqual([`1`]) + expect([...bytes.keys()]).toEqual([`1`]) + + dateMembers[0]!.setTime(2) + byteMembers[0]![0] = 2 + source.utils.begin() + source.utils.write({ + type: `update`, + value: { + id: `3`, + date: new Date(3), + bytes: new Uint8Array([3]), + revision: 1, + }, + }) + source.utils.commit() + expect([...dates.keys()]).toEqual([`2`]) + expect([...bytes.keys()]).toEqual([`2`]) + }, [() => dates.cleanup(), () => bytes.cleanup(), () => source.cleanup()]) + }) + + it(`an ordered limit can admit an untouched row from a changed inArray list`, async () => { + type MembershipRow = { + id: string + v: number + rank: number + revision: number + } + const source = createCollection( + mockSyncCollectionOptions({ + id: `where-publication-ordered-in-${collectionSerial++}`, + getKey: (row) => row.id, + autoIndex: `eager`, + initialData: [0, 1, 2, 3, 4].map((v) => ({ + id: String(v), + v, + rank: v, + revision: 0, + })), + }), + ) + const members = [1] + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .where(({ row }) => inArray(row.v, members)) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + await withOracleCleanup(async () => { + await live.preload() + const expected = () => + [0, 1, 2, 3, 4] + .filter((value) => members.includes(value)) + .sort((left, right) => left - right) + .slice(0, 1) + .map(String) + const check = () => expect([...live.keys()]).toEqual(expected()) + check() + // The previously excluded row 0 is before the old top row. An + // unfiltered ordered prefix must already hold it. + members.push(0) + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `2`, v: 2, rank: 2, revision: 1 }, + }) + source.utils.commit() + check() + // Removing every prefix match requires the loader to continue until + // it finds the next current member. + members.splice(0, members.length, 4) + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `2`, v: 2, rank: 2, revision: 2 }, + }) + source.utils.commit() + check() + }, [() => live.cleanup(), () => source.cleanup()]) + }) + + it(`an unseen ordered-source delete refreshes changed inArray members`, async () => { + type MembershipRow = { id: string; v: number; rank: number } + const source = createCollection( + mockSyncCollectionOptions({ + id: `where-publication-unseen-ordered-delete-${collectionSerial++}`, + getKey: (row) => row.id, + autoIndex: `eager`, + initialData: [0, 1, 2, 3, 4].map((v) => ({ + id: String(v), + v, + rank: v, + })), + }), + ) + const members = [1] + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .where(({ row }) => inArray(row.v, members)) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + await withOracleCleanup(async () => { + await live.preload() + expect([...live.keys()]).toEqual([`1`]) + members[0] = 0 + // The deleted rank-4 row is outside the acquired prefix. The event + // still clocks the complete result to the list's current membership. + source.utils.begin() + source.utils.write({ type: `delete`, key: `4` }) + source.utils.commit() + expect([...live.keys()]).toEqual([`0`]) + }, [() => live.cleanup(), () => source.cleanup()]) + }) + + it(`an ordered on-demand query requests the full source for a mutable inArray list`, async () => { + type MembershipRow = { id: string; v: number; rank: number } + const rows = [0, 1, 2, 3, 4].map((v) => ({ + id: String(v), + v, + rank: v, + })) + const requests: Array<{ + hasWhere: boolean + hasOrder: boolean + hasLimit: boolean + }> = [] + let begin!: () => void + let write!: (message: { type: `delete`; key: string }) => void + let commit!: () => unknown + const source = createCollection({ + id: `where-publication-ordered-demand-in-${collectionSerial++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { + loadSubset: (options) => { + requests.push({ + hasWhere: options.where !== undefined, + hasOrder: options.orderBy !== undefined, + hasLimit: options.limit !== undefined, + }) + params.begin() + for (const row of rows) { + params.write({ type: `insert`, value: row }) + } + params.commit() + return true + }, + } + }, + }, + }) + const members = [1] + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .where(({ row }) => inArray(row.v, members)) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + await withOracleCleanup(async () => { + await live.preload() + expect(requests[0]).toEqual({ + hasWhere: false, + hasOrder: false, + hasLimit: false, + }) + expect([...live.keys()]).toEqual([`1`]) + + members[0] = 0 + begin() + write({ type: `delete`, key: `4` }) + commit() + expect([...live.keys()]).toEqual([`0`]) + }, [() => live.cleanup(), () => source.cleanup()]) + }) + + it(`an on-demand query acquires rows for a captured inArray list's future members`, async () => { + type MembershipRow = { id: string; v: number; revision: number } + const rows: Array = [ + { id: `one`, v: 1, revision: 0 }, + { id: `two`, v: 2, revision: 0 }, + { id: `trigger`, v: 3, revision: 0 }, + ] + let begin!: () => void + let write!: (message: { type: `update`; value: MembershipRow }) => void + let commit!: () => unknown + const source = createCollection({ + id: `where-publication-demand-in-${collectionSerial++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { + loadSubset: ({ where }) => { + params.begin() + for (const row of where ? rows.slice(0, 1) : rows) { + params.write({ type: `insert`, value: row }) + } + params.commit() + return true + }, + } + }, + }, + }) + const members = [1] + const live = createLiveQueryCollection((q) => + q.from({ row: source }).where(({ row }) => inArray(row.v, members)), + ) + await withOracleCleanup(async () => { + await live.preload() + const expected = () => + rows.filter((row) => members.includes(row.v)).map((row) => row.id) + const check = () => expect([...live.keys()].sort()).toEqual(expected()) + check() + + members[0] = 2 + begin() + write({ + type: `update`, + value: { id: `trigger`, v: 3, revision: 1 }, + }) + commit() + check() + }, [() => live.cleanup(), () => source.cleanup()]) + }) + + it(`a source event with no joined-row delta refreshes a changed inArray list`, async () => { + type MemberRow = { + id: string + v: number + groupId: string + revision: number + } + const rows = createCollection( + mockSyncCollectionOptions({ + id: `where-publication-join-in-rows-${collectionSerial++}`, + getKey: (row) => row.id, + autoIndex: `eager`, + initialData: [ + { id: `one`, v: 1, groupId: `g`, revision: 0 }, + { id: `two`, v: 2, groupId: `g`, revision: 0 }, + { id: `trigger`, v: 3, groupId: `missing`, revision: 0 }, + ], + }), + ) + const groups = createCollection( + mockSyncCollectionOptions<{ id: string; revision: number }>({ + id: `where-publication-join-in-groups-${collectionSerial++}`, + getKey: (row) => row.id, + initialData: [ + { id: `g`, revision: 0 }, + { id: `spare`, revision: 0 }, + ], + }), + ) + const members = [1] + const live = createLiveQueryCollection((q) => + q + .from({ row: rows }) + .innerJoin({ group: groups }, ({ row, group }) => + eq(row.groupId, group.id), + ) + .where(({ row }) => inArray(row.v, members)), + ) + const batches: Array> = [] + let unsubscribe: (() => void) | undefined + await withOracleCleanup(async () => { + await live.preload() + const subscription = live.subscribeChanges( + (changes) => + batches.push( + changes.map((change) => ({ + type: change.type, + id: change.value.row.id, + })), + ), + { includeInitialState: false }, + ) + unsubscribe = () => subscription.unsubscribe() + const expected = () => + [ + { id: `one`, v: 1 }, + { id: `two`, v: 2 }, + ] + .filter((row) => members.includes(row.v)) + .map((row) => row.id) + const check = () => + expect(live.toArray.map((result) => result.row.id).sort()).toEqual( + expected(), + ) + check() + + members[0] = 2 + rows.utils.begin() + rows.utils.write({ + type: `update`, + value: { id: `trigger`, v: 3, groupId: `missing`, revision: 1 }, + }) + rows.utils.commit() + check() + expect(batches).toHaveLength(1) + expect(batches[0]!.sort((a, b) => a.id.localeCompare(b.id))).toEqual([ + { type: `delete`, id: `one` }, + { type: `insert`, id: `two` }, + ]) + batches.length = 0 + + rows.utils.begin() + rows.utils.write({ + type: `update`, + value: { id: `trigger`, v: 3, groupId: `missing`, revision: 2 }, + }) + rows.utils.commit() + check() + expect(batches).toEqual([]) + + // A pulse must also come from the joined source. The spare group has + // no joined row, so this write cannot reach the WHERE through the + // joined relation itself. + members[0] = 1 + groups.utils.begin() + groups.utils.write({ + type: `update`, + value: { id: `spare`, revision: 1 }, + }) + groups.utils.commit() + check() + expect(batches).toHaveLength(1) + expect(batches[0]!.sort((a, b) => a.id.localeCompare(b.id))).toEqual([ + { type: `insert`, id: `one` }, + { type: `delete`, id: `two` }, + ]) + }, [ + () => unsubscribe?.(), + () => live.cleanup(), + () => rows.cleanup(), + () => groups.cleanup(), + ]) + }) + + it(`a lazy joined source must deliver unrelated events to refresh inArray membership`, async () => { + type MemberRow = { id: string; groupId: string; v: number } + type GroupRow = { id: string; revision: number } + const membersSource = createCollection( + mockSyncCollectionOptions({ + id: `where-publication-lazy-join-main-${collectionSerial++}`, + getKey: (row) => row.id, + initialData: [ + { id: `one`, groupId: `g`, v: 1 }, + { id: `two`, groupId: `g`, v: 2 }, + ], + }), + ) + const requests: Array = [] + let begin!: () => void + let write!: (message: { type: `delete`; key: string }) => void + let commit!: () => unknown + const groups = createCollection({ + id: `where-publication-lazy-join-groups-${collectionSerial++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.begin() + params.write({ type: `insert`, value: { id: `g`, revision: 0 } }) + params.write({ type: `insert`, value: { id: `h`, revision: 0 } }) + params.commit() + params.markReady() + return { + loadSubset: ({ where }) => { + requests.push(where !== undefined) + return true + }, + } + }, + }, + }) + const members = [1] + const live = createLiveQueryCollection((q) => + q + .from({ row: membersSource }) + .leftJoin({ group: groups }, ({ row, group }) => + eq(row.groupId, group.id), + ) + .where(({ row }) => inArray(row.v, members)), + ) + await withOracleCleanup(async () => { + await live.preload() + expect(requests[0]).toBe(false) + const expected = () => + [ + { id: `one`, v: 1 }, + { id: `two`, v: 2 }, + ] + .filter((row) => members.includes(row.v)) + .map((row) => row.id) + const check = () => + expect(live.toArray.map((result) => result.row.id)).toEqual( + expected(), + ) + check() + + members[0] = 2 + begin() + write({ type: `delete`, key: `h` }) + commit() + check() + }, [ + () => live.cleanup(), + () => groups.cleanup(), + () => membersSource.cleanup(), + ]) + }) + + it(`an outer QueryRef filter refreshes after an inner filter suppresses a source event`, async () => { + type MembershipRow = { + id: string + v: number + groupId: string + revision: number + } + const source = createCollection( + mockSyncCollectionOptions({ + id: `where-publication-queryref-in-${collectionSerial++}`, + getKey: (row) => row.id, + initialData: [ + { id: `one`, v: 1, groupId: `g`, revision: 0 }, + { id: `two`, v: 2, groupId: `g`, revision: 0 }, + { id: `trigger`, v: 3, groupId: `other`, revision: 0 }, + ], + }), + ) + const members = [1] + const live = createLiveQueryCollection((q) => { + const inner = q + .from({ item: source }) + .where(({ item }) => eq(item.groupId, `g`)) + .select(({ item }) => ({ id: item.id, v: item.v })) + return q + .from({ row: inner }) + .where(({ row }) => inArray(row.v, members)) + .select(({ row }) => ({ id: row.id })) + }) + await withOracleCleanup(async () => { + await live.preload() + const expected = () => + [ + { id: `one`, v: 1 }, + { id: `two`, v: 2 }, + ] + .filter((row) => members.includes(row.v)) + .map((row) => row.id) + const check = () => + expect(live.toArray.map((row) => row.id).sort()).toEqual(expected()) + check() + members[0] = 2 + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `trigger`, v: 3, groupId: `other`, revision: 1 }, + }) + source.utils.commit() + check() + }, [() => live.cleanup(), () => source.cleanup()]) + }) + + it(`a correlated include reads current inArray members after an unrelated child event`, async () => { + type ParentRow = { id: string; groupId: string } + type ChildRow = { + id: string + groupId: string + v: number + revision: number + } + const parents = createCollection( + mockSyncCollectionOptions({ + id: `where-publication-include-parents-${collectionSerial++}`, + getKey: (row) => row.id, + initialData: [{ id: `parent`, groupId: `g` }], + }), + ) + const children = createCollection( + mockSyncCollectionOptions({ + id: `where-publication-include-children-${collectionSerial++}`, + getKey: (row) => row.id, + initialData: [ + { id: `one`, groupId: `g`, v: 1, revision: 0 }, + { id: `two`, groupId: `g`, v: 2, revision: 0 }, + { id: `trigger`, groupId: `other`, v: 3, revision: 0 }, + ], + }), + ) + const members = [1] + const live = createLiveQueryCollection((q) => + q.from({ parent: parents }).select(({ parent }) => ({ + id: parent.id, + children: materialize( + q + .from({ child: children }) + .where(({ child }) => + and( + eq(child.groupId, parent.groupId), + inArray(child.v, members), + ), + ) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ id: child.id })), + ), + })), + ) + await withOracleCleanup(async () => { + await live.preload() + const expected = () => + [ + { id: `one`, v: 1 }, + { id: `two`, v: 2 }, + ] + .filter((row) => members.includes(row.v)) + .map((row) => row.id) + const check = () => + expect(live.toArray[0]?.children.map((child) => child.id)).toEqual( + expected(), + ) + check() + + members[0] = 2 + children.utils.begin() + children.utils.write({ + type: `update`, + value: { id: `trigger`, groupId: `other`, v: 3, revision: 1 }, + }) + children.utils.commit() + check() + }, [ + () => live.cleanup(), + () => children.cleanup(), + () => parents.cleanup(), + ]) + }) + + it(`an on-demand include acquires unfiltered children for mutable inArray membership`, async () => { + type ParentRow = { id: string; groupId: string } + type ChildRow = { id: string; groupId: string; v: number } + const parents = createCollection( + mockSyncCollectionOptions({ + id: `where-publication-demand-include-parents-${collectionSerial++}`, + getKey: (row) => row.id, + initialData: [{ id: `parent`, groupId: `g` }], + }), + ) + const requests: Array = [] + let begin!: () => void + let write!: (message: { type: `delete`; key: string }) => void + let commit!: () => unknown + const children = createCollection({ + id: `where-publication-demand-include-children-${collectionSerial++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.begin() + params.write({ + type: `insert`, + value: { id: `one`, groupId: `g`, v: 1 }, + }) + params.write({ + type: `insert`, + value: { id: `two`, groupId: `g`, v: 2 }, + }) + params.write({ + type: `insert`, + value: { id: `spare`, groupId: `h`, v: 3 }, + }) + params.commit() + params.markReady() + return { + loadSubset: ({ where }) => { + requests.push(where !== undefined) + return true + }, + } + }, + }, + }) + const members = [1] + const live = createLiveQueryCollection((q) => + q.from({ parent: parents }).select(({ parent }) => ({ + id: parent.id, + children: materialize( + q + .from({ child: children }) + .where(({ child }) => + and( + eq(child.groupId, parent.groupId), + inArray(child.v, members), + ), + ) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ id: child.id })), + ), + })), + ) + await withOracleCleanup(async () => { + await live.preload() + expect(requests[0]).toBe(false) + const expected = () => + [ + { id: `one`, v: 1 }, + { id: `two`, v: 2 }, + ] + .filter((row) => members.includes(row.v)) + .map((row) => row.id) + const check = () => + expect(live.toArray[0]?.children.map((child) => child.id)).toEqual( + expected(), + ) + check() + + members[0] = 2 + begin() + write({ type: `delete`, key: `spare` }) + commit() + check() + }, [ + () => live.cleanup(), + () => children.cleanup(), + () => parents.cleanup(), + ]) + }) + + it(`checks only changed rows until captured inArray members change`, async () => { + type MembershipRow = { id: string; v: number; revision: number } + let listReads = 0 + const row = (v: number, revision: number): MembershipRow => ({ + id: String(v), + v, + revision, + }) + const source = createCollection( + mockSyncCollectionOptions({ + id: `where-publication-in-work-${collectionSerial++}`, + getKey: (item) => item.id, + autoIndex: `off`, + initialData: Array.from({ length: 40 }, (_, v) => row(v, 0)), + }), + ) + const members = new Proxy([1], { + get(target, slotKey, receiver) { + if (slotKey === `0`) listReads++ + return Reflect.get(target, slotKey, receiver) + }, + }) + const live = createLiveQueryCollection((q) => + q.from({ item: source }).where(({ item }) => inArray(item.v, members)), + ) + await withOracleCleanup(async () => { + await live.preload() + expect([...live.keys()]).toEqual([`1`]) + + listReads = 0 + source.utils.begin() + source.utils.write({ type: `update`, value: row(39, 1) }) + source.utils.commit() + expect([...live.keys()]).toEqual([`1`]) + expect(listReads).toBeLessThan(10) + + members[0] = 2 + listReads = 0 + source.utils.begin() + source.utils.write({ type: `update`, value: row(39, 2) }) + source.utils.commit() + expect([...live.keys()]).toEqual([`2`]) + expect(listReads).toBeGreaterThanOrEqual(40) + }, [() => live.cleanup(), () => source.cleanup()]) + }) + it(`a pending optimistic delete hides a row from a prefiltered unindexed scan`, async () => { // The scan may read synced rows directly only when no optimistic state // changes visibility. A pending delete removes a synced row. @@ -1482,6 +2393,8 @@ describe(`WHERE predicate publication oracle`, () => { let notDateCleanup: (() => unknown) | undefined let sameObjectCleanup: (() => unknown) | undefined let invalidDateCleanup: (() => unknown) | undefined + let zeroDateCleanup: (() => unknown) | undefined + let zeroNumberCleanup: (() => unknown) | undefined await withOracleCleanup(async () => { const collection = createCollection( mockSyncCollectionOptions<{ id: string; v: object }>({ @@ -1508,6 +2421,20 @@ describe(`WHERE predicate publication oracle`, () => { .where(({ row }) => eq(row.v, new Date(Number.NaN))), ) invalidDateCleanup = () => invalidDate.cleanup() + const zeroDate = createLiveQueryCollection((q) => + q + .from({ row: collection }) + .where(({ row }) => eq(row.v, new Date(0))), + ) + zeroDateCleanup = () => zeroDate.cleanup() + const zeroNumber = createLiveQueryCollection((q) => + q + .from({ row: collection }) + .where( + () => new Func(`eq`, [new PropRef([`row`, `v`]), new Value(0)]), + ), + ) + zeroNumberCleanup = () => zeroNumber.cleanup() await notDate.preload() // A distinct Date is unequal to this object. The same object remains // equal by identity at the public query publication checkpoint. It @@ -1517,10 +2444,16 @@ describe(`WHERE predicate publication oracle`, () => { expect([...sameObject.keys()]).toEqual([`forged`]) await invalidDate.preload() expect([...invalidDate.keys()]).toEqual([]) + await zeroDate.preload() + expect([...zeroDate.keys()]).toEqual([]) + await zeroNumber.preload() + expect([...zeroNumber.keys()]).toEqual([]) }, [ () => notDateCleanup?.(), () => sameObjectCleanup?.(), () => invalidDateCleanup?.(), + () => zeroDateCleanup?.(), + () => zeroNumberCleanup?.(), () => collectionCleanup?.(), ]) }) diff --git a/packages/db/tests/temporal-equality-alias-oracle.test.ts b/packages/db/tests/temporal-equality-alias-oracle.test.ts index b5727a5c30..f16c84f6c6 100644 --- a/packages/db/tests/temporal-equality-alias-oracle.test.ts +++ b/packages/db/tests/temporal-equality-alias-oracle.test.ts @@ -13,8 +13,9 @@ import { withOracleCleanup } from './utils.js' * the fixed-offset +00:00 zone remains a different identity. * * The ZonedDateTime lane covers values from one polyfill copy and one instant, - * including genuine subclasses with inherited Temporal methods. One subclass - * prototype has an unrelated display tag. Other genuine instances have own + * including genuine subclasses with inherited Temporal methods. Subclass + * prototypes have unrelated and misleading Temporal-kind display tags; their + * native slots still determine equality. Other genuine instances have own * display and intrinsic-getter shadows. A neighboring * PlainDate lane checks that native brand and equality survive those shadows * while forged tags retain ordinary object identity. Fixture-assigned classes, @@ -46,8 +47,7 @@ type Sample = { text: string identity: `utc` | `offset` | `later` | `calendar` ownShadow?: `tag` | `text` | `epoch` | `calendar` - subclass?: true - taggedSubclass?: true + subclass?: `untagged` | `tagged` | `plain-date-tagged` } class ZonedDateTimeSubclass extends Temporal.ZonedDateTime {} @@ -59,6 +59,17 @@ Object.defineProperty( value: `Tagged`, }, ) +class PlainDateTaggedZonedDateTimeSubclass extends Temporal.ZonedDateTime {} +Object.defineProperty( + PlainDateTaggedZonedDateTimeSubclass.prototype, + Symbol.toStringTag, + { value: `Temporal.PlainDate` }, +) +const subclassConstructors = { + untagged: ZonedDateTimeSubclass, + tagged: TaggedZonedDateTimeSubclass, + 'plain-date-tagged': PlainDateTaggedZonedDateTimeSubclass, +} as const const samples: ReadonlyArray = [ { @@ -114,26 +125,29 @@ const samples: ReadonlyArray = [ id: `utc-subclass`, text: `2024-01-15T09:00:00+00:00[UTC]`, identity: `utc`, - subclass: true, + subclass: `untagged`, }, { id: `utc-tagged-subclass`, text: `2024-01-15T09:00:00+00:00[UTC]`, identity: `utc`, - taggedSubclass: true, + subclass: `tagged`, + }, + { + id: `utc-plain-date-tagged-subclass`, + text: `2024-01-15T09:00:00+00:00[UTC]`, + identity: `utc`, + subclass: `plain-date-tagged`, }, ] function rows(): Array { - return samples.map(({ id, text, ownShadow, subclass, taggedSubclass }) => { + return samples.map(({ id, text, ownShadow, subclass }) => { const base = Temporal.ZonedDateTime.from(text) - const Subclass = taggedSubclass - ? TaggedZonedDateTimeSubclass - : ZonedDateTimeSubclass - const when = - subclass || taggedSubclass - ? new Subclass(base.epochNanoseconds, base.timeZoneId, base.calendarId) - : base + const Subclass = subclass && subclassConstructors[subclass] + const when = Subclass + ? new Subclass(base.epochNanoseconds, base.timeZoneId, base.calendarId) + : base if (ownShadow === `tag`) { Object.defineProperty(when, Symbol.toStringTag, { value: `ordinary` }) } else if (ownShadow === `text`) { @@ -168,6 +182,10 @@ const taggedSubclassProbe = new TaggedZonedDateTimeSubclass( aliasProbe.epochNanoseconds, `UTC`, ) +const plainDateTaggedSubclassProbe = new PlainDateTaggedZonedDateTimeSubclass( + aliasProbe.epochNanoseconds, + `UTC`, +) const offsetProbe = Temporal.ZonedDateTime.from( `2024-01-15T09:00:00+00:00[+00:00]`, ) @@ -309,6 +327,7 @@ it(`distinguishes equivalent aliases and own property shadows from a fixed-offse shadowedCalendar, subclass, taggedSubclass, + plainDateTaggedSubclass, ] = rows() // This checks the fixture against the external Temporal relation. Expected // index and query sets still come from the assigned identity classes. @@ -380,6 +399,32 @@ it(`distinguishes equivalent aliases and own property shadows from a fixed-offse alias!.when, ), ).toBe(true) + // A tag naming another Temporal kind is still only presentation. The + // inherited ZonedDateTime intrinsic checks the actual subclass slots. + expect(Object.getPrototypeOf(plainDateTaggedSubclass!.when)).toBe( + PlainDateTaggedZonedDateTimeSubclass.prototype, + ) + expect( + Object.getPrototypeOf(plainDateTaggedSubclass!.when)[Symbol.toStringTag], + ).toBe(`Temporal.PlainDate`) + expect( + Object.getOwnPropertyDescriptor( + PlainDateTaggedZonedDateTimeSubclass.prototype, + `equals`, + ), + ).toBeUndefined() + expect( + Temporal.ZonedDateTime.prototype.equals.call( + plainDateTaggedSubclass!.when, + alias!.when, + ), + ).toBe(true) + expect( + Temporal.ZonedDateTime.prototype.equals.call( + plainDateTaggedSubclassProbe, + alias!.when, + ), + ).toBe(true) expect(utc!.when.toString()).not.toBe(alias!.when.toString()) expect(expectedKeys(`utc`)).toEqual( new Set([ @@ -391,6 +436,7 @@ it(`distinguishes equivalent aliases and own property shadows from a fixed-offse `utc-shadowed-calendar`, `utc-subclass`, `utc-tagged-subclass`, + `utc-plain-date-tagged-subclass`, ]), ) expect(expectedKeys(`offset`)).toEqual(new Set([`fixed-offset`])) @@ -415,16 +461,15 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { // The query value is a fresh instance. A display-string key would omit // UTC; an instant-only key would also include fixed-offset. - expect(index.equalityLookup(aliasProbe)).toEqual(expectedKeys(`utc`)) - expect(index.equalityLookup(subclassProbe)).toEqual(expectedKeys(`utc`)) - expect(index.equalityLookup(taggedSubclassProbe)).toEqual( - expectedKeys(`utc`), - ) - expect(index.lookup(`eq`, aliasProbe)).toEqual(expectedKeys(`utc`)) - expect(index.lookup(`eq`, subclassProbe)).toEqual(expectedKeys(`utc`)) - expect(index.lookup(`eq`, taggedSubclassProbe)).toEqual( - expectedKeys(`utc`), - ) + for (const probe of [ + aliasProbe, + subclassProbe, + taggedSubclassProbe, + plainDateTaggedSubclassProbe, + ]) { + expect(index.equalityLookup(probe)).toEqual(expectedKeys(`utc`)) + expect(index.lookup(`eq`, probe)).toEqual(expectedKeys(`utc`)) + } expect(index.inArrayLookup([aliasProbe])).toEqual(expectedKeys(`utc`)) expect(index.equalityLookup(offsetProbe)).toEqual(expectedKeys(`offset`)) @@ -437,43 +482,20 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { expectedKeys(`calendar`), ) - index.remove(source[0]!.id, source[0]!) - expect(index.equalityLookup(aliasProbe)).toEqual( - new Set([ - `utc-alias`, - `utc-shadowed-tag`, - `utc-shadowed-text`, - `utc-shadowed-epoch`, - `utc-shadowed-calendar`, - `utc-subclass`, - `utc-tagged-subclass`, - ]), - ) // Removing the subclass itself must retract one member of the class. - // Its native-equal probe still finds every remaining UTC alias. - const subclassRow = source.find((row) => row.id === `utc-subclass`)! - index.remove(subclassRow.id, subclassRow) - expect(index.equalityLookup(subclassProbe)).toEqual( - new Set([ - `utc-alias`, - `utc-shadowed-tag`, - `utc-shadowed-text`, - `utc-shadowed-epoch`, - `utc-shadowed-calendar`, - `utc-tagged-subclass`, - ]), - ) - const taggedRow = source.find((row) => row.id === `utc-tagged-subclass`)! - index.remove(taggedRow.id, taggedRow) - expect(index.equalityLookup(taggedSubclassProbe)).toEqual( - new Set([ - `utc-alias`, - `utc-shadowed-tag`, - `utc-shadowed-text`, - `utc-shadowed-epoch`, - `utc-shadowed-calendar`, - ]), - ) + // Each native-equal probe still finds every remaining UTC alias. + const remainingUtc = expectedKeys(`utc`) + for (const [id, probe] of [ + [`utc`, aliasProbe], + [`utc-subclass`, subclassProbe], + [`utc-tagged-subclass`, taggedSubclassProbe], + [`utc-plain-date-tagged-subclass`, plainDateTaggedSubclassProbe], + ] as const) { + const row = source.find((candidate) => candidate.id === id)! + index.remove(row.id, row) + remainingUtc.delete(id) + expect(index.equalityLookup(probe)).toEqual(remainingUtc) + } expect(index.equalityLookup(offsetProbe)).toEqual(expectedKeys(`offset`)) }, ) @@ -519,7 +541,12 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { `lookup`, ) : undefined - for (const probe of [aliasProbe, subclassProbe, taggedSubclassProbe]) { + for (const probe of [ + aliasProbe, + subclassProbe, + taggedSubclassProbe, + plainDateTaggedSubclassProbe, + ]) { lookup?.mockClear() const result = readKeys(probe) if (lookup) expect(lookup).toHaveBeenCalled() @@ -729,6 +756,14 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () Object.create(TaggedZonedDateTimeSubclass.prototype) as object, { toString: () => `unrelated borrowed object` }, ) + const borrowedPlainDateTaggedSubclass = Object.assign( + Object.create(PlainDateTaggedZonedDateTimeSubclass.prototype) as object, + { toString: () => `unrelated borrowed object` }, + ) + const borrowedPlainDateTaggedSubclassCopy = Object.assign( + Object.create(PlainDateTaggedZonedDateTimeSubclass.prototype) as object, + { toString: () => `unrelated borrowed object` }, + ) Object.defineProperty(shadowed, `timeZoneId`, { value: `UTC` }) expect(aliasProbe.equals(shadowed)).toBe(false) expect(shadowed).not.toBe(freshOffsetProbe) @@ -743,12 +778,22 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () aliasProbe, ), ).toThrow() + expect(() => + Temporal.ZonedDateTime.prototype.equals.call( + borrowedPlainDateTaggedSubclass, + aliasProbe, + ), + ).toThrow() const source = [ { id: `forged`, when: fake }, { id: `shadowed`, when: shadowed }, { id: `borrowed-subclass`, when: borrowedSubclass }, { id: `borrowed-tagged-subclass`, when: borrowedTaggedSubclass }, + { + id: `borrowed-plain-date-tagged-subclass`, + when: borrowedPlainDateTaggedSubclass, + }, ] const fakeCopy = { ...fake } for (const [, IndexType] of indexTypes) { @@ -769,6 +814,10 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () expect(index.equalityLookup(borrowedTaggedSubclassCopy)).toEqual( new Set(), ) + } else if (row.id === `borrowed-plain-date-tagged-subclass`) { + expect( + index.equalityLookup(borrowedPlainDateTaggedSubclassCopy), + ).toEqual(new Set()) } else { // The own zone accessor must not turn a genuine fixed-offset value // into reference-only equality with its native-equal fresh instance. @@ -816,6 +865,10 @@ it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () new Set([`borrowed-tagged-subclass`]), ) expect(keysFor(borrowedTaggedSubclassCopy)).toEqual(new Set()) + expect(keysFor(borrowedPlainDateTaggedSubclass)).toEqual( + new Set([`borrowed-plain-date-tagged-subclass`]), + ) + expect(keysFor(borrowedPlainDateTaggedSubclassCopy)).toEqual(new Set()) }, [() => collection.cleanup()]) }) diff --git a/packages/db/tests/temporal-index-order-oracle.test.ts b/packages/db/tests/temporal-index-order-oracle.test.ts index ab51b83181..f62c4563c7 100644 --- a/packages/db/tests/temporal-index-order-oracle.test.ts +++ b/packages/db/tests/temporal-index-order-oracle.test.ts @@ -16,16 +16,16 @@ import { orderedEntriesArray, withOracleCleanup } from './utils.js' * * This bounded oracle covers one polyfill copy, two opposing offset zones, * one equal-instant pair, a one-nanosecond successor, and negative instants - * around the epoch. Direct indexes order equal-instant peers by source key; - * a public Collection snapshot may choose either peer at a limit boundary. It + * around the epoch. Direct indexes and public Collection snapshots order + * equal-instant peers by source key, including at a limit boundary. It * checks direct BasicIndex/BTreeIndex adds, builds, and value cursors, one * removal/insertion/update continuation, then the public ordered Collection * snapshot with limit 1. Direct custom-comparator witnesses preserve the * former display-string operands and let the callback order two rows whose * values share one query-equality class. An opposite callback distinguishes - * callback-result order from plain display-text order. They check add/build, exact lookup, - * range and cursor boundaries, representative removal, and an update that - * changes callback order without changing query equality. Fixed add/build + * callback-result order from plain display-text order. They check add/build, + * exact lookup, range and cursor boundaries, representative removal, and + * updates that change callback order without changing query equality. Fixed add/build * histories also shadow a genuine value's display tag; the value cursor must * still cross its native instant position. These checks do not claim arbitrary * Temporal kinds, time-zone transitions, longer histories, @@ -107,37 +107,29 @@ function rowsFromFixtures( }) } -// The order contract fixes instant order, but gives no public tie break for -// different zones at the same instant. A prefix may choose any members at its -// boundary, provided it includes every row at an earlier instant. +// The model sorts supplied instant labels, then source keys at a tied instant. +// It has no access to index normalization or production comparison. An exact +// prefix comparison detects omissions, duplicates, order, and a wrong limit tie. function expectOrderedPrefix( actual: Array | undefined, count: number, samples: ReadonlyArray = fixtures, ): void { - expect(actual).toHaveLength(count) - const byId = new Map(samples.map((sample) => [sample.id, sample])) - const instants = actual!.map((id) => { - if (typeof id !== `string`) - throw new Error(`index returned a non-string key`) - const sample = byId.get(id) - expect(sample).toBeDefined() - return sample!.epochNanoseconds - }) - expect(new Set(actual).size).toBe(count) - const sortedInstants = samples - .map(({ epochNanoseconds }) => epochNanoseconds) - .sort((left, right) => (left < right ? -1 : left > right ? 1 : 0)) - const boundary = sortedInstants[count - 1]! - for (const sample of samples) { - if (sample.epochNanoseconds < boundary) { - expect(actual).toContain(sample.id) - } - } - for (let index = 0; index < instants.length; index++) { - expect(instants[index]! <= boundary).toBe(true) - if (index > 0) expect(instants[index - 1]! <= instants[index]!).toBe(true) - } + const expected = [...samples] + .sort((left, right) => + left.epochNanoseconds < right.epochNanoseconds + ? -1 + : left.epochNanoseconds > right.epochNanoseconds + ? 1 + : left.id < right.id + ? -1 + : left.id > right.id + ? 1 + : 0, + ) + .slice(0, count) + .map(({ id }) => id) + expect(actual).toEqual(expected) } const indexTypes = [ @@ -145,11 +137,11 @@ const indexTypes = [ [`BTreeIndex`, BTreeIndex], ] as const -it(`accepts either equal-instant boundary row but rejects a later instant`, () => { - // A lexical index chose d-minus-five at the first-row checkpoint. The - // checker must reject that result while allowing either earliest zone. +it(`rejects a later instant and the wrong equal-instant boundary key`, () => { + // A lexical index chose d-minus-five at the first-row checkpoint. A tie + // resolved by admission order can choose c-utc. Both violate the model. expect(() => expectOrderedPrefix([`b-plus-fourteen`], 1)).not.toThrow() - expect(() => expectOrderedPrefix([`c-utc`], 1)).not.toThrow() + expect(() => expectOrderedPrefix([`c-utc`], 1)).toThrow() expect(() => expectOrderedPrefix([`d-minus-five`], 1)).toThrow() }) @@ -194,7 +186,7 @@ describe.each(indexTypes)( // Returned key arrays preserve instant order and multiplicity at the // direct IndexInterface checkpoint. Limit 1 detects a later instant - // before the equal-instant first group, without imposing a tie break. + // or the wrong source-key tie inside the equal-instant first group. expectOrderedPrefix(index.takeFromStart(rows.length), rows.length) expectOrderedPrefix(index.takeFromStart(1), 1) // A cursor at the shared instant must skip both zones, while retaining @@ -406,7 +398,7 @@ describe.each(indexTypes)( ) it.each([`add`, `build`] as const)( - `follows an opposite callback for equal UTC aliases after %s`, + `follows an opposite callback for equal UTC aliases after %s and update`, (construction) => { const source = [ { @@ -441,9 +433,66 @@ describe.each(indexTypes)( expect(index.takeFromStart(2)).toEqual([`z-utc`, `a-alias`]) expect(index.rangeQuery({ to: utc!.when })).toEqual(new Set([`z-utc`])) expect(index.take(1, utc!.when)).toEqual([`a-alias`]) + + // Updating Etc/UTC to UTC preserves query equality but makes the + // callback operands identical. Both rows must now occupy the same + // callback position and use the source-key tie rule. An index that + // retains the old per-row position would leak a stale row through a + // bound or cursor at UTC. + const moved = { + id: alias!.id, + when: Temporal.ZonedDateTime.from(utc!.when.toString()), + } + expect(moved.when.equals(alias!.when)).toBe(true) + index.update(alias!.id, alias, moved) + expect(index.equalityLookup(utc!.when)).toEqual( + new Set([`z-utc`, `a-alias`]), + ) + expect(index.takeFromStart(2)).toEqual([`a-alias`, `z-utc`]) + expect(index.rangeQuery({ to: utc!.when, toInclusive: true })).toEqual( + new Set([`a-alias`, `z-utc`]), + ) + expect( + index.rangeQuery({ from: utc!.when, fromInclusive: true }), + ).toEqual(new Set([`a-alias`, `z-utc`])) + expect(index.take(2, utc!.when)).toEqual([]) + expect(index.takeReversed(2, utc!.when)).toEqual([]) }, ) + it(`reorders equal UTC aliases by the callback after a same-class update`, () => { + const alias = Temporal.ZonedDateTime.from( + `2024-01-15T09:00:00+00:00[Etc/UTC]`, + ) + const utc = Temporal.ZonedDateTime.from(`2024-01-15T09:00:00+00:00[UTC]`) + expect(alias.equals(utc)).toBe(true) + expect(alias.toString().localeCompare(utc.toString())).toBeLessThan(0) + const first = { id: `z-utc`, when: alias } + const second = { + id: `a-alias`, + when: Temporal.ZonedDateTime.from(alias.toString()), + } + const index = new IndexType(1, new PropRef([`when`]), undefined, { + compareFn: (left: string, right: string) => right.localeCompare(left), + }) + + // The initial callback operands are identical, so source keys order + // the two rows. The update changes one callback operand to UTC while + // keeping its query-equality class. Reverse locale order then puts + // z-utc first even though display text and source keys put a-alias first. + index.add(first.id, first) + index.add(second.id, second) + expect(index.takeFromStart(2)).toEqual([`a-alias`, `z-utc`]) + const moved = { id: first.id, when: utc } + index.update(first.id, first, moved) + expect(index.equalityLookup(utc)).toEqual(new Set([`z-utc`, `a-alias`])) + expect(index.takeFromStart(2)).toEqual([`z-utc`, `a-alias`]) + expect(index.rangeQuery({ to: utc, toInclusive: true })).toEqual( + new Set([`z-utc`]), + ) + expect(index.take(2, utc)).toEqual([`a-alias`]) + }) + it(`moves one row within its exact class when its custom order changes`, () => { const utc = { id: `a-utc`, @@ -471,8 +520,12 @@ describe.each(indexTypes)( expect(index.take(2, alias.when)).toEqual([]) }) - it(`publishes the earliest instant from an indexed ordered Collection snapshot`, async () => { + it(`publishes the source-key first equal-instant row from an ordered Collection snapshot`, async () => { const rows = rowsFromFixtures() + // Admit the UTC peer before the smaller-key +14:00 peer. Changelog #958 + // establishes key order for comparator ties; this reversal distinguishes + // that public limit-one result from admission order on both read paths. + const admission = [rows[1]!, rows[0]!, ...rows.slice(2)] const collection = createCollection({ id: `temporal-index-order-${_name}`, getKey: (row) => row.id, @@ -481,7 +534,7 @@ describe.each(indexTypes)( sync: { sync: ({ begin, write, commit, markReady }) => { begin() - for (const row of rows) write({ type: `insert`, value: row }) + for (const row of admission) write({ type: `insert`, value: row }) commit() markReady() }, @@ -507,7 +560,7 @@ describe.each(indexTypes)( expect(collection.indexes.size).toBe(0) // The unindexed public scan checks the same ordering law through a // separate production path before index creation. - expectOrderedPrefix(readFirst(false), 1) + expect(readFirst(false)).toEqual([`b-plus-fourteen`]) const index = collection.createIndex((row) => row.when, { indexType: IndexType, @@ -517,7 +570,7 @@ describe.each(indexTypes)( // index path supplied this public snapshot after the build. const indexed = readFirst(true) expect(take).toHaveBeenCalled() - expectOrderedPrefix(indexed, 1) + expect(indexed).toEqual([`b-plus-fourteen`]) }, [() => collection.cleanup()]) }) }, From 5c97f18dcd30d83101bafa850ab44598e5ec4c4a Mon Sep 17 00:00:00 2001 From: Isaac Date: Mon, 12 Oct 2026 23:54:27 -0600 Subject: [PATCH 22/46] fix(db): preserve public order ties and strengthen oracle witnesses --- docs/contributing/oracle-coverage.md | 50 ++- .../borrowed-date-indexed-range.md | 51 +++ .../comparison-unit-retirement-round-5.md | 27 +- .../custom-index-order-selection.md | 49 +++ .../oracle-reviews/perf-in-constant-set.md | 14 + .../oracle-reviews/temporal-equality-alias.md | 15 + packages/db/src/collection/change-events.ts | 72 +++- packages/db/src/collection/subscription.ts | 65 ++-- packages/db/src/query/compiler/evaluators.ts | 2 +- .../tests/comparison-oracle.property.test.ts | 55 +-- ...ustom-index-order-selection-oracle.test.ts | 323 +++++++++++++++++- .../in-evaluator-oracle.property.test.ts | 125 ++++++- ...stom-index-window-selection-oracle.test.ts | 30 +- .../query/index-path-collision-oracle.test.ts | 55 ++- ...dicate-publication-oracle.property.test.ts | 71 ++-- .../temporal-equality-alias-oracle.test.ts | 142 +++++++- 16 files changed, 1015 insertions(+), 131 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index c009ed03b1..c91f789e32 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -444,7 +444,7 @@ generic visitor behavior outside the named collectors. | Infinite-query publication | [shared publication laws](https://github.com/TanStack/db/blob/main/packages/db/tests/conformance/infinite-suite-oracle.ts), [React render cuts](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-render-cuts-oracle.test.tsx), [Svelte hook tests](https://github.com/TanStack/db/blob/main/packages/svelte-db/tests/useLiveInfiniteQuery.svelte.test.ts), [review record](oracle-reviews/2026-10-05-infinite-first-paint.md) | Each driver records every value its framework published: React layout commits, Vue pre-flushes, and the Svelte construction value plus pre-effect runs. The shared suite checks that a synchronous source shows its ready first page on the first value, for a query callback and for a supplied collection that has not started, that every ready value in a fixed-source, fixed-query scenario equals the source prefix for its own page count, including an equal dependency replacement that keeps page depth, and that a mount requests an on-demand first window once, for a query callback and for a supplied collection whose finite window is wider than or equal to the one the hook needs, judged by the rows the on-demand source holds after mount. Under deferred acquisition, an on-demand first value is either the ready first page or a value with no rows, by framework: React and Svelte publish before the hook subscribes, Vue after. `preloaded-collection-first-paint` checks the documented route-loader pattern: a limited live query preloaded before mount shows its ready first page on the first value in every framework and asks the source for nothing more; a preload that does not count as demand never settles there. Pre-commit work has no shared form, because Vue setup is its commit. The React driver owns renders that never commit, StrictMode double renders (one first-window request on React 19), retries after GC, a mounted replacement whose startup error must reach the error boundary, and a collection kept from an abandoned render that enters terminal error before its query is selected again, each as under `useLiveQuery`. The React source ID reuse oracle drives the infinite hook too. Svelte owns construction and superseded recomputes before the subscribing effect. In both, the hook acquires an on-demand source as often as `useLiveQuery` does under the same history, for either input form, direct or behind a live-query Collection. On-demand requests from renders that never commit are now owned by the live-query deferred acquisition oracle; the React render-cut oracle checks that an abandoned render makes no acquisition. Open cells, owned here: React 18 StrictMode, where neither hook keeps refs across the double render; a React commit held after the 50 ms unsubscribed GC floor, where a controlled-premise controller witness shows no non-ready value but no React scheduling witness exists; ready values in scenarios that write the source or replace the query, which need a per-step source model; a join of eager and on-demand sources, which needs a zero-acquisition check before commit; a supplied collection with no `.limit`, which sends one unbounded request on `main` too, because the controller cannot narrow it before the source request; and Svelte `state_unsafe_mutation` when a start inside `$derived.by` synchronously writes rows a peer is waiting for, shared with Svelte's `useLiveQuery`, which a two-component probe reproduces and the deferred-acquisition follow-up removes. | | Structural values and ordered primitives | [hash values](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-oracle.property.test.ts), [hash identity](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-identity-oracle.property.test.ts), [MultiSet consolidation](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/multiset-consolidate-oracle.property.test.ts), [hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-graph-oracle.property.test.ts), [mixed hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-mixed-graph-oracle.property.test.ts), [hash retry](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-failure-retry-oracle.property.test.ts), [murmur stream](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/murmur-stream-oracle.property.test.ts), [comparison](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts), [deep equality](https://github.com/TanStack/db/blob/main/packages/db/tests/utils-oracle.property.test.ts), [cursor](https://github.com/TanStack/db/blob/main/packages/db/tests/cursor-oracle.property.test.ts), [indexes](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update-oracle.property.test.ts), [BTree Map](https://github.com/TanStack/db/blob/main/packages/db/tests/btree-map-oracle.test.ts), [query identity](https://github.com/TanStack/db/blob/main/packages/db/tests/query/identity-output-shape-oracle.test.ts), [LIKE semantics](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/evaluators-oracle.test.ts) | Independent flat values, graph topology, algebraic laws, structural object keys in code-unit order (the flat-value, identity and consolidation grammars include keys that tie under locale collation), murmur byte framing against a byte-at-a-time reference (block mixing, the finalizer and symbol framing have no owner there), Map/group/sort recomputation, expression denotation, custom comparator dispatch/reference identity/index compatibility, LIKE wildcard refinement, compiled output bags, declared value-pair agreement between D2 equality keys and IR equality operands, and Collection collation with exact public scan/index order and one-index reuse across repeated reads. The auto-index cells cross BasicIndex and BTreeIndex; scan cells omit indexes. Both paths cover lexical and numeric `en-US` locale defaults plus an explicit ordinary locale override of a numeric default. An unindexed six-row work cell bounds Collection collation resolution to at most two reads per ordered scan: one index-path check and one scan clause. Other locales and ordered histories are outside this owner. The identity pair grammar covers primitives, signed zero, NaN/invalid Date, valid Date/timestamp, binary/Buffer, Temporal, nested references, symbols, and functions; fixed and sampled pairs do not prove every JavaScript value or hash collision freedom. Ref proxies are expressions rather than literal values in this lane. The deep-equality owner also pins that `deepEquals` ignores Map and Set insertion order, RegExp `lastIndex`, array holes, and typed-array class, the rules draft equality keeps. It also pins that objects of another class differ, that plain and null-prototype objects are one class, that a class instance without enumerable keys equals only itself, that URLs compare by `href`, and that typed-array elements compare like numbers. Its [Date/RegExp unit transfer](oracle-reviews/utils-unit-retirement-round-7.md) adds fixed and generated direct-return witnesses that Date differs from its ISO text and epoch number, and RegExp differs from its source text, while preserving the original copy, changed-time, changed-flag, and changed-source operands. Other cross-type pairs and public change-event traces remain outside that transfer. The hash-values owner checks equal-hash consequences for independently constructed flat values. Distinct type identities are checked by the hash-identity owner through `equalHashValues`, including forced digest collisions; distinct 32-bit digests are not promised. The hash-identity owner checks `hash` and `equalHashValues` against one spec-level identity model: primitives with signed zero, NaN, and bigint; reference leaves (functions, registered handles, `File`, binary over 128 bytes); Date, binary, Temporal, and RegExp headers; array length and holes; Map and Set insertion order; and object keys in any order, for acyclic values and for cycles through arrays, Map values, Sets, and plain objects. Map/Set order sensitivity is pinned current behavior, not a promise. Arrays from another realm, RegExps with a `NaN` `lastIndex`, getters, proxies, other cycle carriers, and pairs that differ only in a back-edge target are outside its grammar; pinned cases keep equality's cross-realm length check and strict RegExp field comparison. `hash-work.test.ts` pins how visited values and array/RegExp headers count toward the structural work cap, and that equality copies a shared Map's entries once per distinct pair ([code-weight hash dispatch review](oracle-reviews/code-weight-hash-dispatch.md)). Custom string indexes do not optimize ordinary ranges, and custom cursors remain unsupported. The LIKE owner covers boolean string matching and bounded work, not nullish three-valued logic or a general Unicode collation contract. Unsupported composite cursors reject. The MultiSet consolidation owner checks keyed, single-type, and structural identity, signed zero and NaN in keyed and single-number data, the retained first record, zero-sum removal, and input record contents. It does not assert output order. Its keyed grammar includes cross-type key and value collisions, non-finite numeric keys, the `\|` delimiter, symbol and function identity, and a direct-object/object-tuple delimiter control from #1948. Eight named collision witnesses and both generated campaigns are RED on `c879ba6d` and GREEN with the repair ([#1948 review record](oracle-reviews/issue-1948-keyed-consolidation.md)); the [code-weight consolidation review](oracle-reviews/code-weight-multiset-consolidation.md) records the earlier exclusion. The keyed reference uses direct value/reference equality, including SameValueZero for numeric keys. The unkeyed fallback keeps its original key and value domains. The direct `MultiSet` owner does not prove which `@tanstack/db` query shapes produce primitive keyed values or mixed-type source keys; that needs a live-query production witness. The index owner checks lower-bound presence on bounded numeric histories after each add, update, delete, and build: an upper-only range ignores an unused `fromInclusive: false`, while an explicit exclusive `from` excludes its boundary. A source Map supplies expected sets for BasicIndex and BTreeIndex; wrong-output controls reject both boundary mistakes. It also checks exact `equalityLookup` and `lookup('eq')` after singleton case-insensitive `A → a → A` replacements and populated comparator-group updates, plus fresh-object same-group replacements in fixed and generated histories. These direct-index checks do not cover every JavaScript value, arbitrary comparators, or long Collection histories. The index owner enforces one invalid-comparator law across both BasicIndex and BTreeIndex: for each comparator, accepted numeric prefix (including the empty prefix), and probe operation (add, update, eq/range lookup, take, build), the operation throws exactly when it receives a `NaN` or non-number result and never when every comparison is valid. A `signed infinity` comparator is the valid control, and a custom collation `compare` supplied through `compareOptions` is checked the same way. Two BTree Map split histories check inserted-key reads, exact whole-range payloads/order, size, and extrema, proving split placement follows the insertion index without a post-mutation comparison. A rejected index add or remove leaves the index refining its accepted rows; update and build are not atomic. At the Collection boundary, both index types and both write paths (optimistic insert, sync commit) crash the collection: status becomes `error` and the next mutation throws, so a usable collection never holds rows its subscribers were not told about. Open: a sync source can still commit writes on a collection in `error` state, which stores unpublished rows; this is existing `markError` behavior and needs a lifecycle-owner witness. Comparator transitivity is outside this evidence. | | WHERE predicate publication | [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts), [`in` evaluator](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts) ([review](oracle-reviews/perf-in-constant-set.md)) | An independent Kleene evaluator judges `eq`/`not`/`and`/`or` predicates over strings, a normalization-prefixed string, booleans, numbers, `NaN`, a valid Date, `null`, a missing field, and virtual fields. Snapshot histories with pending optimistic inserts compare a live query, direct subscribers with and without initial state, and `currentStateAsChanges`; change histories apply multi-key sync transactions, including reinsertion of a key deleted earlier, and compare every consumer's key set after each commit, on scan and `BasicIndex` paths. Peer subscribers share one field with different literals, plus one on a virtual field, so every published change must reach exactly the subscribers whose predicate it can satisfy; routing mutants that ignored the previous value, ignored the field, delivered a change twice, or routed while stale published rows awaited reconciliation fail here. Fixed witnesses cover a layout-only publication reaching a filtered subscriber as one empty batch, a filtered subscriber's empty Collection-readiness batch, retraction of a vanished row after eager cleanup and restart, and the new row plus exact subscriber update payload when a same-key row stays TRUE. A stale-live-value mutant survives the generated membership checks but fails the payload witness. A focused [property-visibility test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-prefilter-property-visibility.test.ts) compares the public unindexed snapshot with the enriched-row contract for inherited, non-enumerable, enumerable-own, and nested getter paths; the pre-fix stored-row shortcut failed three of its four controls. Other property-descriptor and stateful-getter histories remain outside those fixed controls. Hostile mutants for FALSE-for-UNKNOWN `eq`, `or` or number-literal subscription prefilters, a prefilter that ignores the previous value, a skipped readiness batch, a skip while stale published rows await reconciliation, and an unindexed snapshot scan that reads only synced rows while an optimistic insert or delete changes visibility passed the prior `@tanstack/db` suite and fail here. Change routing withholds a dropped row from an `eq` subscription before its sent key could be recorded, so a mutant that records dropped rows as sent is equivalent for routed predicates in this oracle. A focused `collection-subscription.test.ts` witness checks that a dropped insert or update, alone or beside a matching change, cannot advance a limited subscription's page offset under a routed `eq` and an unrouted `or` predicate; the unrouted cases beside a matching change kill that mutant, and the unrouted `alone` case does not. A loose-equality prefilter is an equivalent mutant: it only skips less. Skipping during truncate replay also survives; the replay's baseline diff re-derives the same retraction, so the guard keeps the prior dataflow without its own witness. Generated cleanup and restart histories for filtered subscribers remain open for the lifecycle publication owner. No-op updates, other comparison operators, Temporal and binary operands, joins, ordering, optimistic updates, and truncate are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. Change routing was later removed in favor of pooled live queries; the routing mutants above are historical, and the same peer laws now check plain subscription dispatch. The unindexed snapshot prefilter was restored for compiled queries as one stored-row `eq` test on a string or boolean literal. Its property-visibility test also covers the reverse direction: a stored row with an inherited or non-enumerable field must still match `isUndefined` on that field, which a scan that evaluated the full predicate on stored rows fails. The `in` evaluator owner checks `in` against an independent linear reference over Dates, `NaN`, `-0`, BigInt, bytes, Temporal values, objects and nullish items, for constant and row-supplied lists. Its one-polyfill Temporal model uses captured intrinsic methods to reject forged and borrowed brands, accepts genuine own display shadows, and models Duration by ten fields, including PT1H versus PT60M. Arbitrary custom prototype forgery, mixed Temporal implementations, WHERE publication, and indexed mixed ordinary-object/Temporal range order remain outside that owner. | -| Captured `inArray` query stages | [stage oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/captured-inarray-stages-oracle.test.ts) ([review](oracle-reviews/captured-inarray-publication.md)) | Twelve fixed public cuts include nine numeric stage and acquisition witnesses after one unrelated source transaction, a limited ORDER uniqueness counterexample, grouped SUM/min/max arguments in an untouched group, and an aggregate argument wrapped by a selected addition. The controlled on-demand cut checks an unfiltered, unordered, unlimited first request. The nine initial stage cuts, ORDER duplicate, and wrapped aggregate were RED→GREEN in the working tree; grouped SUM/min/max passes in workspace Vitest. The final DB suite passes 12,383 tests across 282 files; DB-IVM passes 693 across 45. Arbitrary values and histories, real providers, and exact-head review remain open. | +| Captured `inArray` query stages | [stage oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/captured-inarray-stages-oracle.test.ts) ([review](oracle-reviews/captured-inarray-publication.md)) | Twelve fixed public cuts include nine numeric stage and acquisition witnesses after one unrelated source transaction, a limited ORDER uniqueness counterexample, grouped SUM/min/max arguments in an untouched group, and an aggregate argument wrapped by a selected addition. The controlled on-demand cut checks an unfiltered, unordered, unlimited first request. The nine initial stage cuts, ORDER duplicate, and wrapped aggregate were RED→GREEN in the working tree; grouped SUM/min/max passes in workspace Vitest. The final DB suite passes 12,396 tests across 282 files; DB-IVM passes 693 across 45. Arbitrary values and histories, real providers, and exact-head review remain open. | | Joined result keys | [Joined result key oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/join-result-key-oracle.property.test.ts) | A nested-loop model recomputes the pair set of an inner, left, or full join over source keys drawn from plain, comma-bearing, bracket-bearing, and quoted strings, numbers beside the strings that print the same, both infinities, and `NaN`. After preload and each synced group change, the published rows must equal the model's pairs and the result key count must equal the row count. The comma-joined key encoding fails its two pinned histories and both campaigns; plain `JSON.stringify`, which prints infinities as the missing-side `null`, fails the pinned infinity history and both campaigns. A pinned history in which a `NaN`-keyed pair leaves and re-forms fails when the join index compares source-key prefixes with `===`. Joins over subqueries, more than two sources, custom `getKey`, and optimistic mutations are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. | | D2 Index storage | [Index refinement oracle](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/index-refinement-oracle.property.test.ts) | A plain-`Map` model sums multiplicities under a type-and-value identity in which `NaN` equals `NaN` and `-0` equals `0`. Histories of up to twelve additions to two keys mix prefixed arrays with `NaN`, `0`, `-0`, `1`, `'1'`, `1n`, and `'a'` prefixes and unprefixed values, with cancellation and reappearance. After each addition, `get` and `has` must equal the model. Comparing prefixes with `===`, which disagrees with the `Map` that holds them, fails three pinned histories and both campaigns; the `-0` control passes under both. Compaction, presence tracking, and structural payloads are outside this owner; the join operator tests and incrementalization law own operator behavior. | | Pooled live queries | [pooled live query oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/pooled-live-query-oracle.property.test.ts) | An independent `eq` model (nullish UNKNOWN, Date as timestamp, `NaN` equal to `NaN`, `-0` equal to `0`) gives membership; a live-query Collection compiled for the same query gives order, values, and status. Histories mix sync transactions, optimistic writes confirmed or rolled back, peer mounts and unmounts, and source cleanup and restart, including cleanup while a write is pending and a mount between cleanup and restart. Wholesale observers must match after every step, granular changes must match a reference observer by type, key, value, and previous value and replay to the model's rows, the layout revision must advance when keys reorder, and observing must never build the live-query Collection. Peers may add a residual `not(eq(...))` conjunct, which the model evaluates itself; a view that ignores it, reports an unfiltered `entries()`, or delivers a row entering or leaving it as an update fails it. Other residual operators use the same compiler evaluator but are not generated. Peers may also order their rows by `id` or by a `g` that can be null, with explicit `nulls`; order comes from the reference. A comparator that ignores direction or `nulls`, an order left out of the partition key, and key order instead of the query's order each fail the pinned reorder history and both campaigns. A partition that ignores a row's previous group, orders by arrival, skips normalization, skips the initial snapshot, follows the source's status after cleanup, or never terminates fails it. Two shared conformance scenarios (`eq-filter-rows`, `eq-filter-peers`) run the pooled path under React, Vue, Solid, Svelte, and Angular; a partition that ignores a row's previous group fails both under every adapter, and `eq-filter-peers` checks through `subscriberCount` that each adapter shares one source subscription per partition. A focused [release-timing test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/pooled-live-query-gc-oracle.test.ts) compares when a pooled query and a live-query Collection release their sources across `gcTime` 1, 100, 0, and `Infinity`, with and without a subscriber, and with no `gcTime`, where both use the live-query Collection default, checks that a partition waits for its views' longest `gcTime`, checks that a view resubscribed after its partition released follows later source writes as its live-query Collection does, including a view with a residual conjunct, and checks that a partition that subscribes again still serves new mounts from one source subscription. Further focused witnesses check that a partition keeps groups only for values that have rows or watchers, that a detached reader never sees a dropped group's revision reused, keep a pooled query's public Collection live while the view is observed, accept it as a query source, and check that a pooled query turns terminal when source cleanup starts while the adapter's cleanup is still pending, as a live-query Collection does. A pinned oracle witness checks that an `eq` path whose getter throws excludes the row on both paths; a pooled path that rethrows fails it. The pooled oracle's grammar never releases a partition, so release and resubscribe histories rely on these fixed witnesses; a fixed delay, a missing unsubscribed floor, a last-`gcTime`-wins rule, and releasing at `gcTime` 0 each fail it. The [2026-10-01 review](oracle-reviews/2026-10-01-pooled-live-queries.md) records each ORC outcome. | @@ -473,9 +473,12 @@ through a Date prototype, checks valid and invalid subclass order in both directions against finite Dates, and rejects local and foreign borrowed-prototype impostors. The owner also checks an unindexed public ordered scan with a tagged foreign invalid -Date. A direct byte-key witness distinguishes two 129-byte native views that -differ only at byte 64; the generated changed-byte grammar spans lengths -1–200 and compares both `areValuesEqual` and `normalizeValue` returns. +Date. A fixed 129-byte witness changes interior byte 64, and a 200-byte +witness changes the last byte. Both compare direct `areValuesEqual` returns +in each direction and distinct `normalizeValue` keys. The generated +changed-byte grammar spans lengths 1–200, but its fixed 100-run campaign did +not reach those long positions; the fixed witnesses reject a first-64-byte +comparator. Exact `normalizeValue` results are direct utility observations. Indexed order, public Date range predicates, cursor continuation, proxies, deeper subclass chains, replaced Date prototypes or `getTime` methods, and other @@ -531,6 +534,12 @@ evaluator owner distinguishes PlainMonthDay reference years, PlainYearMonth reference days, and Duration field vectors that share default display text. Fixed controls reach compiled `eq`, constant-list `in`, and row-supplied `in` in both operand directions. +Five further Duration variants isolate `years`, `months`, `weeks`, `days`, and +`nanoseconds`. Each omitted-field mutant fails the compiled membership +comparison. The alias owner checks these exact classes in an unindexed public +scan and BasicIndex/BTreeIndex equality lookup with an explicit total +comparator; default ordering of calendar-unit Durations needs `relativeTo` +and is outside this equality claim. Equal and unequal values within Instant, PlainDate, PlainDateTime, and PlainTime reject a kind-only Temporal equality key at those same paths. A genuine Uint8Array with a shadowed display tag and a spoof-tagged Uint16Array @@ -539,6 +548,12 @@ equal and unequal content, including an own tag shadow, also reach compiled `eq`, constant `in`, and row-supplied `in` in both operand directions; the original classifier failed equal foreign/local byte pairs. The `in` property has a guarded direct replay selector that runs only its random/replay campaign. +For caller-owned arrays, the compiled `in` owner now walks current indexed +members independently of an own `some` method. Constant and row-supplied +lists check false rejection and false acceptance after compilation and a list +slot edit. The production evaluator calls native Array `some` on the current +list; sparse and nullish controls retain native membership behavior. This +direct evaluation does not by itself establish live-query publication. Native local and `node:vm` Dates, including invalid Dates and timestamp numbers, reach those three compiled equality paths in both operand directions. An equal-timestamp Date from a second VM context is a positive compiled @@ -596,12 +611,22 @@ so omitting either bound changes the complete public key set. Inclusive value-left boundaries are fixed controls. Another fixed ready-state fixture compares unindexed and optimized indexed Date/number order, both full and limit one, with an index-take reach check. -Mutation histories, arbitrary callbacks, and provider publication remain -outside this direct snapshot owner. The separate +Separate public tie witnesses compare unindexed and optimized indexed +snapshots for ascending and descending numeric ties and custom-collation and +nullish ties. Source-key order selects finite windows, including a window cut +through a comparator group, and unbounded results; both index types receive +the same public rule. A direct on-demand ordered subscription also checks an +initial tied prefix and continuation from a unique rank into a tied rank. +A custom-collation cursor includes an unsent string from the cursor's +comparator-equal group. Mutation histories, arbitrary callbacks, provider +ordering, and full live-query publication remain outside this owner. The separate [custom-index window selection oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/custom-index-window-selection-oracle.test.ts) checks the live-query compiler's receiving path. A controlled on-demand source starts with ranks 20 and 40 cached, then serves exact ordered requests from -ranks 10, 20, 30, and 40. Moving a public window from offset 0/limit 1 to +ranks 10, 20, 30, and 40 in physical order distinct from rank order. The +controlled provider sorts only when the request carries `orderBy`, and each +finite request must ask for ascending rank. Moving a public window from offset +0/limit 1 to offset 1/limit 2 must publish ranks 20 and 30 once, even though the provider does not rewrite cached rank 20. Custom-first, ordinary-first, no-index, and eager-custom-only layouts cross BasicIndex and BTreeIndex. Provider reply and @@ -660,6 +685,11 @@ child deletion refreshes membership and that its first child request omits WHERE-focused working-tree and neighboring run passed 988 tests across 11 suites with no type errors; exact committed-head review remains pending. The graph-wide gate can increase full-source transfer for on-demand joins and includes. +The retained direct receiver also checks the interval after a list edit and +before a source transaction: public keys stay unchanged and no subscriber +callback runs. At the next transaction, one callback contains the complete +delete/insert batch and reads the new whole public key set inside that +callback. A split-batch recorder mutant fails at this checkpoint. Source inspection shows list snapshots cost work proportional to list length per delivered graph event, with candidate reevaluation after list changes. The existing index-optimization and subquery-forwarding positive tests now @@ -667,7 +697,7 @@ use frozen `inArray` lists: immutable membership can retain index pushdown and finite predicate forwarding, while mutable captured membership requires full-source reach at the covered boundaries. Their 34 focused tests pass. `ExpressionLike` accepts these `ReadonlyArray` inputs, and a separate missing -error-catalog entry, code 243, is recorded. The full DB suite passes 12,383 +error-catalog entry, code 243, is recorded. The full DB suite passes 12,396 tests across 282 files; DB-IVM passes 693 tests across 45 files. Both builds and explicit source and test typechecks pass. Real providers, broader schedules and values, arbitrary list histories, and @@ -714,6 +744,10 @@ before Collection admission, so later changes to a source Date cannot alter the expected answer. A native subclass also serves as a bound for all four effective relations in both operand positions while its `valueOf` reports a misleading number. +Plain, local-subclass, and foreign-subclass rows also share the bound's exact +timestamp. Inclusive and exclusive ranges compare complete key sets across +the scan and both indexed paths; an omitted tied subclass fails the exact-key +assertion. Date/number scan predicates run in both operand orders; mixed-domain index eligibility is not claimed. Other coercion methods, invalid Dates, later publication, and proxies remain open there. diff --git a/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md b/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md index e095e95ebf..f06c4adf41 100644 --- a/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md +++ b/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md @@ -90,3 +90,54 @@ snapshots still compare against the same frozen model. The focused owner passes 18 tests after the change. This is a fixed in-memory matrix, not evidence for later publication, other realms, arbitrary coercion, or provider cursors. The new witnesses await hostile review at a committed head. + +## Read-only review of `d917ba4c9` + +The exact-head hostile review found an enforcement gap in the fixed Date-range +matrix. Its only source row at the bound's intrinsic timestamp was a plain +`Date`. A wrong comparator that sent a native subclass tied with the bound to +the strict side of a range could pass every subclass assertion: the local and +foreign subclass rows were previously below or above the bound. This was an +oracle-history gap, not a demonstrated production defect. The repair belongs +in the source-row grammar and exact-key comparison; changing the accepted +timestamp law or production comparator was not justified by this finding. + +The working-tree owner now admits a local and a foreign native Date subclass +at the bound's exact millisecond timestamp. Their `valueOf` overrides report +`-100` and `100`, respectively, while the bound's native timestamp is `1`. +The model freezes all source timestamps through `Date.prototype.getTime` +before Collection admission. Thus `gt` and `lt` exclude all three tied rows, +while `gte` and `lte` include all three, including the plain-Date control. +The same exact expected key sets judge the unindexed scan, ordinary indexed +snapshot, and `optimizedOnly` indexed snapshot for BasicIndex and BTreeIndex; +the lookup spy confirms both indexed paths. A wrong-result control removes +`foreign-equal` from a real inclusive scan result and reaches the same +exact-key `toEqual` comparison, which rejects it as an assertion failure. +The focused Vitest run passed 18 tests with no type errors. These results are +working-tree evidence after the `d917ba4c9` review, not a fresh committed-head +hostile review. + +### Oracle-guide applicability for this repair + +| Requirement | Outcome at this working tree | +| --- | --- | +| ORC-001 | The live-query guide's comparison semantics promise native Date timestamp ordering, including foreign Dates; the owner limits its claim to fixed ready-Collection snapshots and the named Date domain. | +| ORC-002 | Expected keys come from native `getTime` numbers frozen before Collection admission and simple numeric relations; no production comparator or index classifier computes the reference. | +| ORC-003 | Opening prose states the law and limits. Adjacent comments identify the model, finite row and predicate grammar, Collection driver, public key observation, and scan/index checkpoints. | +| ORC-004 | Not triggered: this is a fixed bounded matrix, not a generated-history property. | +| ORC-005 | Public `currentStateAsChanges` keys are compared exactly before indexing and after BasicIndex or BTreeIndex installation; `optimizedOnly` and lookup spies witness the indexed path. | +| ORC-006 | Removing `foreign-equal` from a real `gte` scan result fails the exact-key assertion at its intended checkpoint. The control is an assertion failure, not a setup or runtime failure. | +| ORC-007 | Not triggered: the fixed matrix makes no important generated-property claim. | +| ORC-008 | Not triggered: the reference recomputes from frozen rows and introduces no stateful transition model. | +| ORC-009 | The reference maps plain, local-subclass, and foreign-subclass Date values to one intrinsic millisecond timestamp domain. The Collection retains the source values; the model intentionally does not represent their overridden coercion methods as ordering state. | +| ORC-010 | The owner's cleanup wrapper preserves a primary comparison failure as the `AggregateError` cause if release also fails; its separate cleanup witness checks both errors and release order. | +| ORC-011 | Not triggered by this review: the named fault was a missing tied-subclass history, not a plausible semantic error shared by production and the native-timestamp reference for which a second formulation was identified. | +| ORC-012 | This table records the other applicable requirements and explains non-applicable triggers. It does not substitute for a new exact-head hostile review. | +| ORC-013 | `local-equal` and `foreign-equal` reach the boundary premise. Inclusive and exclusive relations on both sides distinguish correct timestamp ties from a wrong strict-side rule; the wrong-result control rejects an omitted tie. | +| ORC-014 | Not triggered: the claim is limited to the controlled ready in-memory Collection and makes no real-provider or host handoff claim. | + +The bounded claim remains: a ready in-memory Collection, fixed rows, these +four range operators and operand arrangements, two index types, and public +snapshot keys. Later publication, generated subclass histories, invalid Dates, +arbitrary coercion methods, provider cursors, and other hosts still need their +own witnesses before broader closure. diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md index f3283a4eed..27aa7df308 100644 --- a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md @@ -111,10 +111,23 @@ Review of committed head `d974d4a4f` found that the changed-byte generator reached only lengths 1–50 while the owner's bounded key-size claim reached 200. The follow-up expands independently supplied changed-byte pairs to lengths 1–200 for both direct `areValuesEqual` and `normalizeValue` checks. -A fixed 129-byte pair differs only at byte 64. A first-64-byte comparison -fault fails the equality check at the intended direct return; the real -normalizer returns distinct keys. The focused comparison owner passed 117 -tests, and explicit test TypeScript, ESLint, and Prettier checks passed. This -is bounded sensitivity evidence, not a claim that every byte sequence or -all binary consumers were exercised. Final exact-commit hostile review of -the follow-up is pending. +A fixed 129-byte pair differs only at byte 64. The former fixed assertion +checked `normalizeValue` through `assertChangedBytes`, but did not call +production `areValuesEqual`; the 100-run generated campaign declared lengths +through 200 without guaranteeing a long changed-byte sample. Replaying that +fixed-seed generator and budget reached at most length 12 and changed index +10; an executed first-64-byte comparator fault survived the generated lane. +Exact-head hostile review of `d917ba4c9` found this direct-equality gap. + +The next fixed witness supplies a 129-byte pair changed at interior byte 64 +and a 200-byte pair changed at its last byte, 199. For each pair, it calls +production `areValuesEqual` in both directions at the direct-return +checkpoint, checks an unchanged copy, and compares the normalized keys. An +executed first-64-byte comparator fault fails the same changed-byte checker +used by the generated equality property for both pairs. The generated grammar +still spans lengths 1–200; these pins guarantee upper-position reach within +the normal 100-run budget. This is bounded sensitivity evidence, not a claim +that every byte sequence or all binary consumers were exercised. The focused +comparison owner passed 117 tests with no Vitest type errors; explicit test +TypeScript, ESLint, Prettier, and whitespace checks passed. Final exact-commit +hostile review of this repair is pending. diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md index e3e3c09d69..954f74a51f 100644 --- a/docs/contributing/oracle-reviews/custom-index-order-selection.md +++ b/docs/contributing/oracle-reviews/custom-index-order-selection.md @@ -196,3 +196,52 @@ bound slightly does not necessarily change it, so the coverage map now says "omitting" rather than "misplacing." This is a documentation correction, not a new production bug or an added witness. Final review of the next committed head remains pending. + +## Finite provider order handoff + +Review of `d917ba4c9` found that the window oracle's controlled provider read +its source rows from the rank-sorted expected result. Removing `orderBy` from +every request therefore left all eight cases green: the fixture still served +the right prefix without an ordering hint. The source facts now have physical +rank order 30, 10, 40, 20. The independent public model still sorts by numeric +rank. The adapter sorts its filtered source rows only when the request carries +`orderBy`, and it checks that every finite request asks for one ascending +`rank` term. This keeps provider work at the request boundary; the public +preload and moved-window comparisons keep their original law. + +All eight fixed cases passed after this oracle repair. A temporary test-only +mutant stripped `orderBy` as the adapter received it. All eight cases failed +at the finite-request assertion. With that assertion temporarily disabled, +the BasicIndex/custom-first case reached the initial public snapshot and +failed with `[2]` instead of `[1]`. Both mutations were removed. These are +assertion failures at the intended request and public checkpoints, not +setup failures. The controlled provider still does not establish how a remote +provider interprets an ordered request. + +## Public comparator ties at `d917ba4c9` + +The public `currentStateAsChanges` path returned `[b]` for descending +`orderBy rank, limit: 1` over tied source keys, while an unindexed scan +returned `[a]`. BasicIndex and BTreeIndex both exposed the +failure through their reverse views. The low-level reverse index may reverse +tie order; the public Collection rule from changelog #958 requires source keys +ascending within comparator-equal positions. New numeric, mixed-key, +custom-collation, and nullish fixtures compare independently sorted source +facts with unindexed and optimized indexed snapshots. Finite prefixes cut +through tied groups, and index-take spies establish the optimized path. + +The same defect reached direct on-demand `requestLimitedSnapshot`: a source +with rank-1 keys `c`, `b`, `a` published `[c]` for a descending limit-one +request. That was a separate RED public receiving witness. A neighboring +history first publishes a unique rank-2 row, then continues at a cursor into +the rank-1 tie. A custom string-length comparator checks that a cursor at +`xx` still includes the unsent `yy` peer in the same comparator position. +The repair shares indexed prefix selection across ready snapshots and direct +subscriptions, completes only a finite prefix's boundary group, and sorts +source keys within each comparator group. It does not change the low-level +ReverseIndex contract. The repaired focused ordering, lifecycle, pagination, +and work suites passed 748 tests, plus TypeScript, lint, formatting, and diff +checks. The evidence covers these two public snapshot paths and fixed +histories; provider ordering, source mutations, and full live-query +publication require separate receiving witnesses. Fresh hostile review of +the committed repair remains pending. diff --git a/docs/contributing/oracle-reviews/perf-in-constant-set.md b/docs/contributing/oracle-reviews/perf-in-constant-set.md index 1a5f1b85d8..be1de639fb 100644 --- a/docs/contributing/oracle-reviews/perf-in-constant-set.md +++ b/docs/contributing/oracle-reviews/perf-in-constant-set.md @@ -134,3 +134,17 @@ Date to equal its own timestamp through compiled `eq`, constant-list `in`, and row-supplied `in`. The focused `in` owner passed after this addition. This checks the compiled equality return for two VM contexts; browser iframe behavior and a fresh exact-commit hostile review of the follow-up remain open. + +## Own-method membership gap at `d917ba4c9` + +The compiled `in` evaluator and its reference both called the caller array's +overridable `some` method. A compiled `[1]` with an own `some` returning false +rejected present member `1`; after the list changed to `[2]`, an own `some` +returning true accepted absent member `1`. Constant and row-supplied lists +were both false green. The reference now walks present integer indices through +the length captured at evaluation, independently of the array's methods. +Production calls the native Array `some` with the current list as receiver. +Both false rejection and false acceptance failed at the compiled return before +the repair; the sparse-hole and nullish controls pass afterward. The focused +`in` owner passed 44 tests. This proves the named array-method boundary at +evaluation, not every possible proxy or mutation during one evaluation. diff --git a/docs/contributing/oracle-reviews/temporal-equality-alias.md b/docs/contributing/oracle-reviews/temporal-equality-alias.md index deef723500..27082b7cc6 100644 --- a/docs/contributing/oracle-reviews/temporal-equality-alias.md +++ b/docs/contributing/oracle-reviews/temporal-equality-alias.md @@ -317,3 +317,18 @@ reference-identity controls. The revised owner passes 28 tests. This adds one recognized wrong-kind tag to the fixed polyfill grammar; arbitrary prototype impersonation and mixed implementations remain open. Fresh hostile review of the next committed head is pending. + +## Duration field sensitivity after `d917ba4c9` + +Read-only review found that the alias and compiled `in` owners claimed all +ten intrinsic Duration fields while their fixtures did not vary `years`, +`months`, `weeks`, `days`, or `nanoseconds`. Five new variants each change one +of those fields from an equal baseline. The compiled owner checks `eq` and +both `in` forms in both operand directions; its omit-one-field mutant fails at +the membership comparison for every omitted field. The alias owner checks +the same independent equality classes in an unindexed public scan and exact +BasicIndex/BTreeIndex lookups. Those indexes use an explicit total comparator: +default ordering of calendar-unit Durations requires `relativeTo`, so this +evidence does not claim their default ordered-index behavior. The focused +compiled and alias owners passed 75 tests. A fresh committed-head hostile +review remains pending. diff --git a/packages/db/src/collection/change-events.ts b/packages/db/src/collection/change-events.ts index 621eb179b4..78656282f5 100644 --- a/packages/db/src/collection/change-events.ts +++ b/packages/db/src/collection/change-events.ts @@ -1,3 +1,4 @@ +import { compareKeys } from '@tanstack/db-ivm' import { compileSingleRowExpression, toBooleanPredicate, @@ -16,6 +17,7 @@ import { import { makeComparator } from '../utils/comparison.js' import { buildCompareOptions } from '../query/compiler/order-by' import { codedMessage, codedWarning, devBuild } from '../error-message.js' +import type { IndexReader } from '../indexes/base-index.js' import type { ChangeMessage, CollectionLike, @@ -23,7 +25,7 @@ import type { SubscribeChangesOptions, } from '../types' import type { CollectionImpl } from './index.js' -import type { BasicExpression, OrderBy } from '../query/ir.js' +import type { BasicExpression, OrderBy, OrderByClause } from '../query/ir.js' import type { WithVirtualProps } from '../virtual-props.js' /** @@ -371,10 +373,7 @@ function getOrderedKeys( return whereFilter?.(value) ?? true } - // Take the keys that match the filter and limit - // if no limit is provided `index.keyCount` is used, - // i.e. we will take all keys that match the filter - return index.takeFromStart(limit ?? index.keyCount, filterFn) + return takeOrderedIndexKeys(collection, index, clause, limit, filterFn) } } } @@ -397,18 +396,13 @@ function getOrderedKeys( })) const compare = (a: { key: TKey; value: T }, b: { key: TKey; value: T }) => { for (const clause of clauses) { - // Extract values for comparison const aValue = extractValueFromItem(a.value, clause.expression) const bValue = extractValueFromItem(b.value, clause.expression) - const result = clause.compare(aValue, bValue) - if (result !== 0) { - return result - } + if (result !== 0) return result } - return 0 + return compareKeys(a.key, b.key) } - allItems.sort(compare) const sortedKeys = allItems.map((item) => item.key) @@ -421,6 +415,60 @@ function getOrderedKeys( return sortedKeys } +/** + * Takes an indexed ordered prefix, completing its comparator-equal boundary. + * Public snapshots keep source keys ascending within ties even when a reversed + * index walks them backward. A finite read visits the indexed prefix and its + * boundary group; an unbounded read traverses all indexed keys. `from` marks + * an exclusive comparator position, including an explicit nullish value. + */ +export function takeOrderedIndexKeys< + T extends object, + TKey extends string | number, +>( + collection: CollectionLike, + index: IndexReader, + clause: OrderByClause, + limit: number | undefined, + filterFn: (key: TKey) => boolean, + from?: { value: unknown }, +): Array { + if (limit !== undefined && limit <= 0) return [] + const count = limit ?? index.keyCount + const taken = from + ? index.take(count, from.value, filterFn) + : index.takeFromStart(count, filterFn) + if (taken.length === 0) return taken + + const valueFor = (key: TKey) => + extractValueFromItem(collection.get(key)!, clause.expression) + let keys = taken + if (limit !== undefined && taken.length === limit) { + const boundaryValue = valueFor(taken[taken.length - 1]!) + const boundaryKeys = [ + ...index.rangeQuery({ from: boundaryValue, to: boundaryValue }), + ].filter(filterFn) + keys = [...new Set([...taken, ...boundaryKeys])] + } + + // The index already orders distinct comparator positions. Sort only the + // source keys inside each equal position, including a completed boundary. + const values = keys.map(valueFor) + const compare = makeComparator(buildCompareOptions(clause, collection)) + const ordered: Array = [] + for (let start = 0; start < keys.length;) { + let end = start + 1 + while (end < keys.length && compare(values[start], values[end]) === 0) { + end++ + } + for (const key of keys.slice(start, end).sort(compareKeys)) { + ordered.push(key) + } + start = end + } + return limit === undefined ? ordered : ordered.slice(0, limit) +} + /** * Helper function to extract a value from an item based on an expression */ diff --git a/packages/db/src/collection/subscription.ts b/packages/db/src/collection/subscription.ts index cd5b62bfea..31529a4793 100644 --- a/packages/db/src/collection/subscription.ts +++ b/packages/db/src/collection/subscription.ts @@ -1,6 +1,7 @@ +import { compareKeys } from '@tanstack/db-ivm' import { ensureIndexForExpression } from '../indexes/auto-index.js' -import { and, eq } from '../query/builder/functions.js' -import { PropRef, Value, getPropRefPropertyPath } from '../query/ir.js' +import { and } from '../query/builder/functions.js' +import { PropRef, getPropRefPropertyPath } from '../query/ir.js' import { EventEmitter } from '../event-emitter.js' import { compileExpression } from '../query/compiler/evaluators.js' import { buildCursor, buildCursorCurrent } from '../utils/cursor.js' @@ -13,6 +14,7 @@ import { codedMessage, devBuild } from '../error-message.js' import { createFilterFunctionFromExpression, createFilteredCallback, + takeOrderedIndexKeys, } from './change-events.js' import type { BasicExpression, OrderBy } from '../query/ir.js' import type { IndexReader } from '../indexes/base-index.js' @@ -1477,34 +1479,36 @@ export class CollectionSubscription // so we load all rows equal to minValue first, to be sure we don't skip any duplicate values let keys: Array = [] if (hasMinValue) { - // First, get all items with the same FIRST COLUMN value as minValue - // This provides wide bounds for the local index - const { expression } = orderBy[0]! - const allRowsWithMinValue = this.collection.currentStateAsChanges({ - where: eq(expression, new Value(minValueForIndex)), - }) - - if (allRowsWithMinValue) { - const keysWithMinValue = allRowsWithMinValue - .map((change) => change.key) - .filter(filterFn) - - // Add items with the minValue first - keys.push(...keysWithMinValue) - - // Then get items greater than minValue - const keysGreaterThanMin = index.take( + // A cursor names a comparator position, which can contain several + // unequal values (or both null and undefined). Keep that whole group + // before walking later positions, with public keys ascending. + keys = [ + ...index.rangeQuery({ + from: minValueForIndex, + to: minValueForIndex, + }), + ] + .filter(filterFn) + .sort(compareKeys) + keys.push( + ...takeOrderedIndexKeys( + this.collection, + index, + orderBy[0]!, limit - keys.length, - minValueForIndex!, filterFn, - ) - keys.push(...keysGreaterThanMin) - } else { - keys = index.take(limit, minValueForIndex!, filterFn) - } + { value: minValueForIndex }, + ), + ) } else { // No min value provided, start from the beginning - keys = index.takeFromStart(limit, filterFn) + keys = takeOrderedIndexKeys( + this.collection, + index, + orderBy[0]!, + limit, + filterFn, + ) } const valuesNeeded = () => Math.max(limit - changes.length, 0) @@ -1533,7 +1537,14 @@ export class CollectionSubscription biggestObservedValue = valueExtractor ? valueExtractor(value) : value } - keys = index.take(valuesNeeded(), biggestObservedValue!, filterFn) + keys = takeOrderedIndexKeys( + this.collection, + index, + orderBy[0]!, + valuesNeeded(), + filterFn, + { value: biggestObservedValue }, + ) } // Reserve rows before publication. A direct truncate replay keeps them in diff --git a/packages/db/src/query/compiler/evaluators.ts b/packages/db/src/query/compiler/evaluators.ts index 6b93b044c7..b61d7ac32f 100644 --- a/packages/db/src/query/compiler/evaluators.ts +++ b/packages/db/src/query/compiler/evaluators.ts @@ -509,7 +509,7 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { if (!Array.isArray(array)) { return false } - return array.some((item) => + return Array.prototype.some.call(array, (item: unknown) => valuesEqual(normalizeEqualityOperand(item), value), ) } diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index 9fa88c3ac0..9df8ddb5a2 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -56,8 +56,9 @@ import type { CompareOptions } from '../src/query/builder/types' * universal performance bound. Own `length` and `byteLength` properties do * not change native byte content. The 32-character margin is the existing * constant-overhead budget for the sampled 0..200-byte keys. Changed-byte - * equality and key histories cover lengths 1..200; a 129-byte interior change - * pins the first length beyond the 128-byte boundary. + * equality and key histories cover lengths 1..200. Fixed changes at byte 64 + * of a 129-byte view and the last byte of a 200-byte view pin positions that + * a short generated campaign need not sample. */ const defaultOpts: CompareOptions = { @@ -1918,26 +1919,38 @@ describe(`native binary key content`, () => { expect(normalizeValue(left)).not.toBe(normalizeValue(right)) }) - it(`distinguishes byte 64 in a 129-byte key`, () => { - const bytes = Uint8Array.from({ length: 129 }, (_, index) => index) - const changed = new Uint8Array(bytes) - changed[64] = (changed[64]! + 1) % 256 - const sample = { bytes, changed, index: 64 } + it(`distinguishes interior and last bytes of long native views`, () => { + // The supplied byte at each named position changes while length and all + // other bytes stay equal. At each function return, equality must reject + // the pair in both directions and normalization must produce distinct keys. + for (const { length, index } of [ + { length: 129, index: 64 }, + { length: 200, index: 199 }, + ]) { + const bytes = Uint8Array.from({ length }, (_, byteIndex) => byteIndex) + const changed = new Uint8Array(bytes) + changed[index] = (changed[index]! + 1) % 256 + const sample = { bytes, changed, index } + + // A plausible first-64-byte comparator passes every short pair but + // fails the same changed-byte checker used by the generated equality + // property. This fault injection reaches the intended return check. + expect(() => + assertChangedBytes( + sample, + (left, right) => + left.length === right.length && + left + .subarray(0, 64) + .every((byte, byteIndex) => byte === right[byteIndex]), + ), + ).toThrow(`changed byte must be unequal`) - // A key that reads only its first 64 bytes accepts this legal change. - // The normalizer must preserve it at the direct-return checkpoint. - expect(() => - assertChangedBytes( - sample, - (left, right) => - left.length === right.length && - left.subarray(0, 64).every((byte, index) => byte === right[index]), - ), - ).toThrow(`changed byte must be unequal`) - assertChangedBytes( - sample, - (left, right) => normalizeValue(left) === normalizeValue(right), - ) + expect(areValuesEqual(bytes, new Uint8Array(bytes))).toBe(true) + expect(areValuesEqual(bytes, changed)).toBe(false) + expect(areValuesEqual(changed, bytes)).toBe(false) + expect(normalizeValue(bytes)).not.toBe(normalizeValue(changed)) + } }) comparisonProperty([arbitraryChangedBytes])( diff --git a/packages/db/tests/custom-index-order-selection-oracle.test.ts b/packages/db/tests/custom-index-order-selection-oracle.test.ts index 9d98e7e6de..13e92fc10f 100644 --- a/packages/db/tests/custom-index-order-selection-oracle.test.ts +++ b/packages/db/tests/custom-index-order-selection-oracle.test.ts @@ -2,6 +2,7 @@ import { describe, expect, it, vi } from 'vitest' import { createCollection } from '../src/collection/index.js' import { BasicIndex } from '../src/indexes/basic-index.js' import { BTreeIndex } from '../src/indexes/btree-index.js' +import { ReverseIndex } from '../src/indexes/reverse-index.js' import { Func, PropRef, Value } from '../src/query/ir.js' import { DEFAULT_COMPARE_OPTIONS } from '../src/utils.js' import { withOracleCleanup } from './utils.js' @@ -25,8 +26,12 @@ import { withOracleCleanup } from './utils.js' * limit one. A neighboring mixed Date/number snapshot compares public scan * order before index creation with optimized order afterward. It promises * index installation preserves that order, without prescribing a general - * cross-family order. These checks do not claim arbitrary custom comparator - * laws, mutation histories, provider ordering, or live-query publication. + * cross-family order. Changelog #958 and the temporal index order oracle + * establish key-ascending ties in public ordered snapshots. The witnesses + * below add equal numeric and custom-collation values, nullish values, finite + * windows, and unbounded reads. A direct on-demand ordered subscription + * checks initial and cursor snapshots of an equal-rank group. They do not + * claim mutation histories, provider ordering, or live-query publication. */ type Row = { id: string; rank: number } @@ -489,3 +494,317 @@ describe.each(indexTypes)(`%s public mixed-value order`, (name, IndexType) => { }, [() => cleanup?.()]) }) }) + +describe.each(indexTypes)(`%s public key ties`, (name, IndexType) => { + it(`publishes key-first ties through limited ordered subscriptions`, async () => { + // A direct ordered subscription is a second public receiving path. The + // first history begins inside a tie; the second first emits a unique rank + // and then continues into a tied rank. The provider accepts each request + // immediately, so callbacks are observed after each local snapshot. + const observe = async ( + suffix: string, + source: ReadonlyArray, + requests: ReadonlyArray<{ minValues?: Array }>, + ) => { + const batches: Array> = [] + const collection = createCollection({ + id: `subscription-public-ties-${name}-${suffix}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of source) write({ type: `insert`, value: row }) + commit() + markReady() + return { loadSubset: () => true } + }, + }, + }) + await withOracleCleanup(async () => { + await collection.stateWhenReady() + const index = collection.createIndex((row) => row.rank, { + indexType: IndexType, + }) + const subscription = collection.subscribeChanges( + (changes) => { + if (changes.length) batches.push(changes.map(({ key }) => key)) + }, + { includeInitialState: false }, + ) + try { + subscription.setOrderByIndex(new ReverseIndex(index, true)) + for (const request of requests) { + subscription.requestLimitedSnapshot({ + orderBy: descendingFirst.orderBy, + limit: 1, + ...request, + }) + } + } finally { + subscription.unsubscribe() + } + }, [() => collection.cleanup()]) + return batches + } + + // Numeric rank descending, then source key ascending, predicts `a` in + // the initial tie. The unique rank-2 row is an adjacent non-tie control; + // a cursor at rank 2 then reaches the complete rank-1 tie group. + expect( + await observe( + `initial`, + [ + { id: `c`, rank: 1 }, + { id: `b`, rank: 1 }, + { id: `a`, rank: 1 }, + ], + [{}], + ), + ).toEqual([[`a`]]) + expect( + await observe( + `cursor`, + [ + { id: `z`, rank: 2 }, + { id: `c`, rank: 1 }, + { id: `b`, rank: 1 }, + { id: `a`, rank: 1 }, + ], + [{}, { minValues: [2] }], + ), + ).toEqual([[`z`], [`a`]]) + }) + it(`uses ascending keys within numeric ties in both order directions`, async () => { + type TiedRow = { id: string | number; rank: number } + const source: ReadonlyArray = [ + { id: 2, rank: 1 }, + { id: 1, rank: 1 }, + { id: `c`, rank: 1 }, + { id: `b`, rank: 1 }, + { id: `a`, rank: 1 }, + { id: `z`, rank: 0 }, + { id: `y`, rank: 0 }, + ] + let cleanup: (() => unknown) | undefined + await withOracleCleanup(async () => { + const collection = createCollection({ + id: `numeric-public-ties-${name}`, + getKey: ({ id }) => id, + autoIndex: `off`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of source) write({ type: `insert`, value: row }) + commit() + markReady() + }, + }, + }) + cleanup = () => collection.cleanup() + await collection.stateWhenReady() + + // The independent reference sorts source facts by numeric rank, then + // by source key: strings before numbers, each in ascending order. + // Reverse admission distinguishes that law from a stable scan. A + // descending window cuts through the rank-1 group and the string-to- + // number boundary within it. + const keyOrder = (left: string | number, right: string | number) => + typeof left !== typeof right + ? typeof left === `string` + ? -1 + : 1 + : left < right + ? -1 + : left > right + ? 1 + : 0 + const expected = (direction: `asc` | `desc`, limit?: number) => + [...source] + .sort( + (left, right) => + (direction === `asc` + ? left.rank - right.rank + : right.rank - left.rank) || keyOrder(left.id, right.id), + ) + .slice(0, limit) + .map(({ id }) => id) + const read = ( + direction: `asc` | `desc`, + limit?: number, + optimizedOnly?: boolean, + ) => + collection + .currentStateAsChanges({ + orderBy: [ + { + expression: new PropRef([`rank`]), + compareOptions: { ...DEFAULT_COMPARE_OPTIONS, direction }, + }, + ], + limit, + optimizedOnly, + }) + ?.map(({ key }) => key) + const cases = [ + { direction: `asc`, limit: 1 }, + { direction: `asc`, limit: 3 }, + { direction: `desc`, limit: 1 }, + { direction: `desc`, limit: 2 }, + { direction: `desc`, limit: 4 }, + { direction: `desc`, limit: undefined }, + ] as const + + // Before index creation, the public scan is one receiving formulation. + for (const { direction, limit } of cases) { + expect(read(direction, limit)).toEqual(expected(direction, limit)) + } + const index = collection.createIndex((row) => row.rank, { + indexType: IndexType, + }) + const take = vi.spyOn(index, `takeFromStart`) + const reverseTake = vi.spyOn(index, `takeReversedFromEnd`) + // optimizedOnly forbids scan fallback. The take spies show that the + // index path supplies the same public observation after readiness. + for (const { direction, limit } of cases) { + expect(read(direction, limit, true)).toEqual(expected(direction, limit)) + } + expect(take).toHaveBeenCalled() + expect(reverseTake).toHaveBeenCalled() + }, [() => cleanup?.()]) + }) + + it(`uses source keys within custom-collation and nullish ties`, async () => { + type LabelRow = { id: string; label: string | null | undefined } + const source: ReadonlyArray = [ + { id: `b`, label: `yy` }, + { id: `a`, label: `xx` }, + { id: `n`, label: null }, + { id: `m`, label: undefined }, + { id: `c`, label: `z` }, + ] + const compare = (left: string, right: string) => left.length - right.length + let cleanup: (() => unknown) | undefined + await withOracleCleanup(async () => { + const collection = createCollection({ + id: `custom-public-ties-${name}`, + getKey: ({ id }) => id, + autoIndex: `off`, + defaultStringCollation: { stringSort: `custom`, compare }, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of source) write({ type: `insert`, value: row }) + commit() + markReady() + }, + }, + }) + cleanup = () => collection.cleanup() + await collection.stateWhenReady() + + // Null and undefined share one position. The callback compares only + // non-null strings by length. The independent reference applies the + // public key rule to both comparator-equal groups. + const expected = (direction: `asc` | `desc`, limit?: number) => + [...source] + .sort((left, right) => { + const order = + left.label == null + ? right.label == null + ? 0 + : -1 + : right.label == null + ? 1 + : direction === `asc` + ? left.label.length - right.label.length + : right.label.length - left.label.length + return ( + order || (left.id < right.id ? -1 : left.id > right.id ? 1 : 0) + ) + }) + .slice(0, limit) + .map(({ id }) => id) + const read = ( + direction: `asc` | `desc`, + limit?: number, + optimizedOnly?: boolean, + ) => + collection + .currentStateAsChanges({ + orderBy: [ + { + expression: new PropRef([`label`]), + compareOptions: { direction, nulls: `first` }, + }, + ], + limit, + optimizedOnly, + }) + ?.map(({ key }) => key) + const cases = [ + { direction: `asc`, limit: 1 }, + { direction: `asc`, limit: 4 }, + { direction: `desc`, limit: 3 }, + { direction: `desc`, limit: undefined }, + ] as const + + for (const { direction, limit } of cases) { + expect(read(direction, limit)).toEqual(expected(direction, limit)) + } + const index = collection.createIndex((row) => row.label, { + indexType: IndexType, + options: { + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `custom`, + compare, + }, + }, + }) + const take = vi.spyOn(index, `takeFromStart`) + const reverseTake = vi.spyOn(index, `takeReversedFromEnd`) + for (const { direction, limit } of cases) { + expect(read(direction, limit, true)).toEqual(expected(direction, limit)) + } + expect(take).toHaveBeenCalled() + expect(reverseTake).toHaveBeenCalled() + + // A cursor at `xx` names the callback's length-two position. Its + // comparator-equal `yy` peer remains eligible even though exact string + // equality would find only the already-sent `xx` row. + const batches: Array> = [] + const subscription = collection.subscribeChanges( + (changes) => { + if (changes.length) batches.push(changes.map(({ key }) => key)) + }, + { includeInitialState: false }, + ) + try { + subscription.setOrderByIndex(new ReverseIndex(index, true)) + const orderBy = [ + { + expression: new PropRef([`label`]), + compareOptions: { + direction: `desc` as const, + nulls: `first` as const, + }, + }, + ] + subscription.requestLimitedSnapshot({ orderBy, limit: 3 }) + subscription.requestLimitedSnapshot({ + orderBy, + limit: 1, + minValues: [`xx`], + }) + expect(batches).toEqual([[`m`, `n`, `a`], [`b`]]) + } finally { + subscription.unsubscribe() + } + }, [() => cleanup?.()]) + }) +}) diff --git a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts index 947a6db157..fb4db2c06d 100644 --- a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts +++ b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts @@ -12,9 +12,9 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * Transfer law: The public `inArray` API describes array membership. A * compiled expression holding a caller-owned array reads that array's current * members whenever it evaluates, even if the array or one of its items has - * changed since compilation. The - * current evaluator returns UNKNOWN (`null`) for a null or undefined value - * and FALSE for a list that is not an array. + * changed since compilation. An own `some` method is not a list member and + * cannot decide membership. The current evaluator returns UNKNOWN (`null`) + * for a null or undefined value and FALSE for a list that is not an array. * Otherwise the result is TRUE when some list item equals the value under the * evaluator's equality, and FALSE when none does. A null or undefined item * never equals a value under that membership interpretation. The public @@ -38,9 +38,10 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * - Every other pair compares with `===`, so `-0` equals `0`, `1n` does not * equal `1`, and objects compare by reference. * - * Model: `referenceIn` walks the list and applies `referenceEqual`, which - * restates the rules above. It imports no production comparison or - * normalization helper. + * Model: `referenceIn` walks present indices through the length captured at + * the start of evaluation and applies `referenceEqual`, which restates the + * rules above. It does not call the array's overridable methods and imports + * no production comparison or normalization helper. * * Domain: values and items come from a pool that contains the boundaries * of each rule: equal and unequal strings, a string that begins with the @@ -60,6 +61,7 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * items and may repeat. The list is a constant `Value`, the shape that join * demand builds, or a row field. Fixed histories also compile a caller-owned * list, then replace a slot or mutate a Date or byte item before reevaluating. + * A fixed history also compiles, shadows `some`, and edits a list slot. * * Production driver: compile `in` with the public IR and evaluate it for * each row, as a WHERE clause does. @@ -437,10 +439,14 @@ function referenceEqual(left: unknown, right: unknown): boolean { function referenceIn(value: unknown, list: unknown): boolean | null { if (value === null || value === undefined) return null if (!Array.isArray(list)) return false - return list.some( - (item) => - item !== null && item !== undefined && referenceEqual(item, value), - ) + const length = list.length + for (let index = 0; index < length; index++) { + if (!(index in list)) continue + const item = list[index] + if (item !== null && item !== undefined && referenceEqual(item, value)) + return true + } + return false } type InCase = { value: unknown; list: unknown; listFromRow: boolean } @@ -601,6 +607,56 @@ describe(`in evaluator`, () => { checkCurrent(2) }) + it.each([ + { listFromRow: false, source: `constant` }, + { listFromRow: true, source: `row-supplied` }, + ])( + `ignores an own some method after compiling a $source list and editing its member`, + ({ listFromRow }) => { + const list: Array = [1] + const listExpression = listFromRow + ? new PropRef([`row`, `list`]) + : new Value(list) + const evaluate = compileExpression( + new Func(`in`, [new PropRef([`row`, `value`]), listExpression]), + ) + const checkCurrent = (value: unknown) => + expect + .soft(evaluate({ row: { value, list } })) + .toBe(referenceIn(value, list)) + + // A hostile own method can lie, but indexed members remain the input. + checkCurrent(1) + Object.defineProperty(list, `some`, { + configurable: true, + value: () => false, + }) + checkCurrent(1) + list[0] = 2 + Object.defineProperty(list, `some`, { + configurable: true, + value: () => true, + }) + checkCurrent(1) + checkCurrent(2) + checkCurrent(null) + }, + ) + + it(`skips sparse holes and nullish items while retaining present members`, () => { + const list = Array(4) + list[1] = null + list[2] = undefined + list[3] = 2 + expect(0 in list).toBe(false) + const evaluate = compileExpression( + new Func(`in`, [new PropRef([`row`, `value`]), new Value(list)]), + ) + for (const value of [1, 2, null, undefined]) { + expect(evaluate({ row: { value } })).toBe(referenceIn(value, list)) + } + }) + it(`reads a Date item's current timestamp after compilation`, () => { const dateItem = new Date(1) const list = [dateItem] @@ -937,6 +993,55 @@ describe(`in evaluator`, () => { ) }) + it(`reads years, months, weeks, days, and nanoseconds in Duration equality`, () => { + const variedFields = [ + `years`, + `months`, + `weeks`, + `days`, + `nanoseconds`, + ] as const + const fields = { + years: 1, + months: 1, + weeks: 1, + days: 1, + nanoseconds: 1, + } + const baseline = Temporal.Duration.from(fields) + const equalCopy = Temporal.Duration.from(fields) + checkEqualityPaths(baseline, equalCopy, true) + + // Every variant differs in one intrinsic field. A comparator that drops + // that field predicts a false match, which the referenceIn check rejects. + for (const omitted of variedFields) { + const variant = Temporal.Duration.from({ ...fields, [omitted]: 2 }) + const equalWithoutField = ( + left: Temporal.Duration, + right: Temporal.Duration, + ) => + durationFields.every( + (field, index) => + field === omitted || + Reflect.apply(durationGetters[index]!, left, []) === + Reflect.apply(durationGetters[index]!, right, []), + ) + expect(equalWithoutField(baseline, variant)).toBe(true) + const wrong: Evaluate = ({ value, list }) => + Array.isArray(list) && + value instanceof Temporal.Duration && + list.some( + (candidate) => + candidate instanceof Temporal.Duration && + equalWithoutField(value, candidate), + ) + expect(() => + check(wrong, { value: variant, list: [baseline], listFromRow: false }), + ).toThrow(/in mismatch/) + checkEqualityPaths(baseline, variant, false) + } + }) + it(`rejects display-string equality for equivalent zone aliases`, () => { const lexical: Evaluate = ({ value, list }) => Array.isArray(list) && diff --git a/packages/db/tests/query/custom-index-window-selection-oracle.test.ts b/packages/db/tests/query/custom-index-window-selection-oracle.test.ts index d098649097..d87415a3d2 100644 --- a/packages/db/tests/query/custom-index-window-selection-oracle.test.ts +++ b/packages/db/tests/query/custom-index-window-selection-oracle.test.ts @@ -16,8 +16,9 @@ import type { LoadSubsetOptions } from '../../src/types.js' * * The bounded history caches ranks 20 and 40, installs indexes in one of four * legal layouts, preloads a one-row window, then moves to offset one with two - * rows. A controlled provider fulfills each exact request from four immutable - * source rows; it never rewrites cached rows. A plain numeric sort and slice + * rows. A controlled provider holds four immutable source rows in physical + * order that differs from rank order. It sorts only when the request carries + * `orderBy`, and it never rewrites cached rows. A plain numeric sort and slice * predicts both public checkpoints independently of index selection. Public * snapshots and publication batches are compared after preload and after the * window promise settles. This checks BasicIndex and BTreeIndex; it does not @@ -29,11 +30,12 @@ type Layout = `custom-first` | `ordinary-first` | `no-index` | `eager-custom-only` const rows: ReadonlyArray = [ - { id: 1, rank: 10 }, - { id: 2, rank: 20 }, { id: 3, rank: 30 }, + { id: 1, rank: 10 }, { id: 4, rank: 40 }, + { id: 2, rank: 20 }, ] +// The public order follows the query's rank expression, not source storage. const expected = [...rows].sort((a, b) => a.rank - b.rank || a.id - b.id) const expectedWindow = (offset: number, limit: number) => expected.slice(offset, offset + limit).map(({ id }) => id) @@ -41,7 +43,12 @@ const expectedWindow = (offset: number, limit: number) => // This adapter models only the request grammar exercised here: a whole-source // request, a finite prefix, or a cursor's current tie plus following rows. // Its answers come from source facts and the request predicate, never indexes. +// Finite ordered acquisition needs a rank hint: without it, a provider slices +// physical order and cannot establish the requested prefix. function requestedRows(options: LoadSubsetOptions): Array { + if (options.limit !== undefined) { + expect(options.orderBy).toBeDefined() + } if (options.orderBy) { expect(options.orderBy).toHaveLength(1) expect(options.orderBy[0]!.expression).toMatchObject({ @@ -51,21 +58,24 @@ function requestedRows(options: LoadSubsetOptions): Array { expect(options.orderBy[0]!.compareOptions.direction).toBe(`asc`) } const matching = options.where - ? expected.filter( + ? rows.filter( (row) => evaluateReferenceExpression(options.where!, row) === true, ) - : expected + : rows + const providerRows = options.orderBy + ? [...matching].sort((a, b) => a.rank - b.rank || a.id - b.id) + : matching if (!options.cursor) { const start = options.offset ?? 0 - return matching.slice( + return providerRows.slice( start, options.limit === undefined ? undefined : start + options.limit, ) } - const current = matching.filter((row) => + const current = providerRows.filter((row) => Boolean(evaluateReferenceExpression(options.cursor!.whereCurrent, row)), ) - const following = matching + const following = providerRows .filter((row) => Boolean(evaluateReferenceExpression(options.cursor!.whereFrom, row)), ) @@ -152,6 +162,7 @@ async function observeWindowMove( cleanupLive = () => live.cleanup() await live.preload() const initial = Array.from(live.values(), ({ id }) => id) + expect(initial).toEqual(expectedWindow(0, 1)) const subscription = live.subscribeChanges(() => { publications.push(Array.from(live.values(), ({ id }) => id)) }) @@ -164,7 +175,6 @@ async function observeWindowMove( // its write stream does not. This reaches the cached-row merge path. expect(replies.some((reply) => reply.join(`,`) === `1,2,3`)).toBe(true) expect(writes).toEqual([1, 3]) - expect(initial).toEqual(expectedWindow(0, 1)) expect(moved).toEqual(expectedWindow(1, 2)) expect(publications).toEqual([expectedWindow(1, 2)]) if (layout === `eager-custom-only`) { diff --git a/packages/db/tests/query/index-path-collision-oracle.test.ts b/packages/db/tests/query/index-path-collision-oracle.test.ts index 30e03b64c4..c33ddc27fb 100644 --- a/packages/db/tests/query/index-path-collision-oracle.test.ts +++ b/packages/db/tests/query/index-path-collision-oracle.test.ts @@ -28,13 +28,15 @@ * A further valid-Date history overrides `valueOf` in local and foreign * subclasses while retaining native `getTime`. An ordinary Date bound and a * subclass bound each reach all four effective relations; the subclass bound - * occupies both operand positions. The intrinsic-timestamp model compares - * public scan and indexed snapshots, and both index types must remain eligible - * for optimized-only reads. The reference captures each source timestamp - * before the rows enter the Collection, so later source-Date changes cannot - * change the expected answer. Date/number scan comparisons in both operand - * orders use the same model; mixed-domain range indexes make no eligibility - * claim here. + * occupies both operand positions. Plain, local-subclass, and foreign-subclass + * rows share the bound's exact timestamp, so inclusive and exclusive ranges + * must respectively keep or exclude every tied key. The intrinsic-timestamp + * model compares public scan and indexed snapshots, and both index types must + * remain eligible for optimized-only reads. The reference captures each source + * timestamp before the rows enter the Collection, so later source-Date changes + * cannot change the expected answer. Date/number scan comparisons in both + * operand orders use the same model; mixed-domain range indexes make no + * eligibility claim here. * * Production driver: `currentStateAsChanges`, Collection subscriptions, and * live-query Collections run the real predicates and callback compilers. @@ -374,11 +376,18 @@ it.each([ const earlierForeign = runInNewContext( `new (class extends Date { valueOf() { return 100 } })(0)`, ) as Date + const equalForeign = runInNewContext( + `new (class extends Date { valueOf() { return 100 } })(1)`, + ) as Date + const equalLocal = new LaterDate(1) + const localLater = new LaterDate(2) const dateRows = [ { id: `plain-before`, v: new Date(-1) }, { id: `foreign-earlier`, v: earlierForeign }, { id: `equal`, v: new Date(1) }, - { id: `local-later`, v: new LaterDate(2) }, + { id: `local-equal`, v: equalLocal }, + { id: `foreign-equal`, v: equalForeign }, + { id: `local-later`, v: localLater }, { id: `plain-after`, v: new Date(3) }, ] const bound = new Date(1) @@ -491,10 +500,12 @@ it.each([ await collection.stateWhenReady() // The model captured native timestamps, which ignore subclass valueOf. - // The two overrides deliberately reverse their apparent native-operator - // positions, while ordinary Dates control both sides and equality. + // The overrides deliberately move their native-operator positions, + // including at the exact bound, while ordinary Dates control both sides. expect(Number(earlierForeign)).toBe(100) - expect(Number(dateRows[3]!.v)).toBe(-100) + expect(Number(equalForeign)).toBe(100) + expect(Number(equalLocal)).toBe(-100) + expect(Number(localLater)).toBe(-100) expect(Number(subclassBound)).toBe(-100) const expectedDateKeys = (accepts: (time: number) => boolean) => referenceRows @@ -509,6 +520,13 @@ it.each([ .currentStateAsChanges({ where, optimizedOnly }) ?.map((change) => String(change.key)) .sort() + expect( + referenceRows + .filter(({ time }) => time === boundTime) + .map(({ id }) => id) + .sort(), + `plain and native subclass rows at the range bound`, + ).toEqual([`equal`, `foreign-equal`, `local-equal`]) // A ready Collection's public snapshots are the comparison cuts. Scan // first, then install one index and require both the ordinary and @@ -522,6 +540,17 @@ it.each([ `unindexed optimized-only ${predicate.name}`, ).toBeUndefined() } + // Calibrate the exact-key refinement check with a plausible wrong + // result: dropping one tied subclass from an inclusive public scan. + const inclusive = predicates[1]! + const fullScan = snapshotKeys(inclusive.where)! + expect(fullScan).toContain(`foreign-equal`) + expect(() => + expect( + fullScan.filter((key) => key !== `foreign-equal`), + `wrong-result control: omitted foreign Date tie`, + ).toEqual(expectedDateKeys(inclusive.accepts)), + ).toThrow() // Equality already normalizes native Dates to numbers. Relational scan // results must use the same timestamp when either operand is a number. for (const predicate of mixedPredicates) { @@ -554,8 +583,8 @@ it.each([ // timestamps before production received the source rows. const greaterKeys = expectedDateKeys((time) => time > boundTime) expect(greaterKeys).toContain(`local-later`) - dateRows[3]!.v.setTime(-2) - expect(nativeGetTime.call(dateRows[3]!.v)).toBe(-2) + localLater.setTime(-2) + expect(nativeGetTime.call(localLater)).toBe(-2) expect(expectedDateKeys((time) => time > boundTime)).toEqual(greaterKeys) }) }, diff --git a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts index 6d37a1bad8..79e2dcc9f3 100644 --- a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts +++ b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts @@ -86,8 +86,11 @@ * filtered QueryRef, and correlated include witnesses compare public results * with that model. On-demand ordered, joined, and include receivers check that * the first acquisition is unfiltered when the captured list is mutable. - * The retained direct query also checks exact insert/delete - * events and no duplicate event when the list did not change. Separate fixed + * The retained direct query also checks that a list edit alone preserves the + * old public snapshot and sends no callback. At the next source transaction, + * one subscriber callback contains the complete insert/delete batch and reads + * the new whole result. It checks no duplicate event when the list did not + * change. Separate fixed * Date and byte-item cases change contents inside the captured list, then * check the next source event. An ordered limit witness deletes a source row * outside the old loaded prefix after a list edit; it requires the newly @@ -1430,18 +1433,28 @@ describe(`WHERE predicate publication oracle`, () => { const live = createLiveQueryCollection((q) => q.from({ row: source }).where(({ row }) => inArray(row.v, members)), ) - const events: Array<{ type: string; key: string }> = [] + // Retain callback boundaries and read the Collection while each callback + // runs. Flattening changes would accept two partial publications with the + // right eventual keys, even though one source transaction owes one + // coherent public result. + const batches: Array<{ + changes: Array<{ type: string; key: string }> + visibleKeys: Array + }> = [] let unsubscribe: (() => void) | undefined await withOracleCleanup(async () => { await live.preload() const subscription = live.subscribeChanges( (changes) => { - events.push( - ...changes.map((change) => ({ - type: change.type, - key: String(change.key), - })), - ) + batches.push({ + changes: changes + .map((change) => ({ + type: change.type, + key: String(change.key), + })) + .sort((left, right) => left.key.localeCompare(right.key)), + visibleKeys: [...live.keys()].map(String).sort(), + }) }, { includeInitialState: false }, ) @@ -1453,8 +1466,13 @@ describe(`WHERE predicate publication oracle`, () => { .sort() const check = () => expect([...live.keys()].sort()).toEqual(expected()) check() + expect(batches).toEqual([]) members[0] = 2 + // A list edit is only an input change. The previous public snapshot + // remains in place until a source event clocks the retained query. + expect([...live.keys()].sort()).toEqual([`one`]) + expect(batches).toEqual([]) source.utils.begin() source.utils.write({ type: `update`, @@ -1462,15 +1480,20 @@ describe(`WHERE predicate publication oracle`, () => { }) source.utils.commit() check() - expect( - events.sort((left, right) => left.key.localeCompare(right.key)), - ).toEqual([ - { type: `delete`, key: `one` }, - { type: `insert`, key: `two` }, + expect(batches).toEqual([ + { + changes: [ + { type: `delete`, key: `one` }, + { type: `insert`, key: `two` }, + ], + visibleKeys: [`two`], + }, ]) - events.length = 0 + batches.length = 0 members.push(1) + expect([...live.keys()].sort()).toEqual([`two`]) + expect(batches).toEqual([]) source.utils.begin() source.utils.write({ type: `update`, @@ -1478,8 +1501,13 @@ describe(`WHERE predicate publication oracle`, () => { }) source.utils.commit() check() - expect(events).toEqual([{ type: `insert`, key: `one` }]) - events.length = 0 + expect(batches).toEqual([ + { + changes: [{ type: `insert`, key: `one` }], + visibleKeys: [`one`, `two`], + }, + ]) + batches.length = 0 // A source update with no further list edit cannot republish the // unchanged result merely because the reduction revisits every row. @@ -1490,7 +1518,7 @@ describe(`WHERE predicate publication oracle`, () => { }) source.utils.commit() check() - expect(events).toEqual([]) + expect(batches).toEqual([]) source.utils.begin() source.utils.write({ type: `delete`, key: `two` }) @@ -1500,7 +1528,12 @@ describe(`WHERE predicate publication oracle`, () => { 1, ) check() - expect(events).toEqual([{ type: `delete`, key: `two` }]) + expect(batches).toEqual([ + { + changes: [{ type: `delete`, key: `two` }], + visibleKeys: [`one`], + }, + ]) }, [() => unsubscribe?.(), () => live.cleanup(), () => source.cleanup()]) }) diff --git a/packages/db/tests/temporal-equality-alias-oracle.test.ts b/packages/db/tests/temporal-equality-alias-oracle.test.ts index f16c84f6c6..85f8fcc8a1 100644 --- a/packages/db/tests/temporal-equality-alias-oracle.test.ts +++ b/packages/db/tests/temporal-equality-alias-oracle.test.ts @@ -35,7 +35,9 @@ import { withOracleCleanup } from './utils.js' * this file checks its public Collection equality through an unindexed scan. * Indexed PlainMonthDay equality remains open. YearMonth and Duration store * several classes with identical default display and check indexed add, - * build, removal, re-add, and public reads. + * build, removal, re-add, and public reads. Duration also varies years, + * months, weeks, days, and nanoseconds one field at a time; a key that omits + * any one of those fields would merge its distinct fixture classes. * This oracle does not claim all IANA aliases, subclasses with replaced * Temporal methods, arbitrary spoofed prototypes, * mixed Temporal implementations, joins, live-query publication, or provider @@ -305,6 +307,79 @@ function intrinsicFixture(kind: IntrinsicKind): IntrinsicFixture { } } +function variedDurationFixture(): IntrinsicFixture { + const fields = { + years: 1, + months: 1, + weeks: 1, + days: 1, + nanoseconds: 1, + } + const baseline = Temporal.Duration.from(fields) + const variedFields = [ + `years`, + `months`, + `weeks`, + `days`, + `nanoseconds`, + ] as const + const variants = variedFields.map((field) => ({ + field, + when: Temporal.Duration.from({ ...fields, [field]: 2 }), + })) + const allFields = [ + `years`, + `months`, + `weeks`, + `days`, + `hours`, + `minutes`, + `seconds`, + `milliseconds`, + `microseconds`, + `nanoseconds`, + ] as const + const getters = allFields.map( + (field) => + Object.getOwnPropertyDescriptor(Temporal.Duration.prototype, field)!.get!, + ) + return { + rows: [ + { id: `mixed`, when: baseline, identity: `mixed` }, + { + id: `mixed-copy`, + when: Temporal.Duration.from(fields), + identity: `mixed`, + }, + ...variants.map(({ field, when }) => ({ + id: `mixed-${field}`, + when, + identity: `mixed-${field}`, + })), + ], + probes: [ + { when: Temporal.Duration.from(fields), identity: `mixed` }, + ...variants.map(({ field }) => ({ + when: Temporal.Duration.from({ ...fields, [field]: 2 }), + identity: `mixed-${field}`, + })), + ], + checkNativeRelation: () => { + // Each class differs from mixed in exactly one intrinsic field. Dropping + // that field from an exact key would merge the two classes. + for (const { field, when } of variants) { + expect( + allFields.filter( + (_, index) => + Reflect.apply(getters[index]!, baseline, []) !== + Reflect.apply(getters[index]!, when, []), + ), + ).toEqual([field]) + } + }, + } +} + function intrinsicExpectedKeys( source: ReadonlyArray, identity: string, @@ -597,6 +672,71 @@ it(`keeps PlainMonthDay reference years distinct in an unindexed public equality }, [() => collection.cleanup()]) }) +it(`keeps five additional Duration fields distinct in an unindexed public equality scan`, async () => { + const fixture = variedDurationFixture() + fixture.checkNativeRelation() + const collection = createCollection({ + id: `temporal-duration-varied-unindexed-equality`, + getKey: (row) => row.id, + autoIndex: `off`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of fixture.rows) write({ type: `insert`, value: row }) + commit() + markReady() + }, + }, + }) + + // The scan reads exact classes without asking Temporal to order calendar + // units, which would require relativeTo. + await withOracleCleanup(async () => { + await collection.stateWhenReady() + expect(collection.indexes.size).toBe(0) + for (const probe of fixture.probes) { + const predicate = new Func(`eq`, [ + new PropRef([`when`]), + new Value(probe.when), + ]) + const keys = new Set( + collection + .currentStateAsChanges({ where: predicate, optimizedOnly: false }) + ?.map(({ value }) => value.id), + ) + expect(keys).toEqual(intrinsicExpectedKeys(fixture.rows, probe.identity)) + } + }, [() => collection.cleanup()]) +}) + +it.each(indexTypes)( + `%s keeps five additional Duration fields in exact index equality`, + (_name, IndexType) => { + const fixture = variedDurationFixture() + fixture.checkNativeRelation() + // A lexical display comparator supplies only the index's required total + // order. The independently assigned classes judge exact equality. + const compareText = (left: unknown, right: unknown) => { + const a = String(left) + const b = String(right) + return a < b ? -1 : a > b ? 1 : 0 + } + const index = new IndexType(1, new PropRef([`when`]), undefined, { + compareFn: compareText, + }) + index.build( + fixture.rows.map((row): [string, IntrinsicRow] => [row.id, row]), + ) + for (const probe of fixture.probes) { + const expected = intrinsicExpectedKeys(fixture.rows, probe.identity) + expect(index.equalityLookup(probe.when)).toEqual(expected) + expect(index.lookup(`eq`, probe.when)).toEqual(expected) + expect(index.inArrayLookup([probe.when])).toEqual(expected) + } + }, +) + describe.each(indexTypes)( `%s Temporal intrinsic field equality`, (indexName, IndexType) => { From d93ae1dd6d20706735bc104cb652dfbaa2b6174c Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 00:09:06 -0600 Subject: [PATCH 23/46] fix(db): order indexed values before equality encoding --- docs/contributing/oracle-coverage.md | 19 +- .../borrowed-date-indexed-range.md | 10 + .../captured-inarray-publication.md | 23 ++ .../comparison-unit-retirement-round-5.md | 10 + .../custom-index-order-selection.md | 52 ++++ .../oracle-reviews/perf-in-constant-set.md | 10 + .../oracle-reviews/temporal-equality-alias.md | 12 + packages/db/src/collection/change-events.ts | 7 +- packages/db/src/indexes/base-index.ts | 2 +- packages/db/src/utils/comparison.ts | 12 +- ...ustom-index-order-selection-oracle.test.ts | 274 +++++++++++++++++- ...dicate-publication-oracle.property.test.ts | 14 +- 12 files changed, 421 insertions(+), 24 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index c91f789e32..6c4929878d 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -444,7 +444,7 @@ generic visitor behavior outside the named collectors. | Infinite-query publication | [shared publication laws](https://github.com/TanStack/db/blob/main/packages/db/tests/conformance/infinite-suite-oracle.ts), [React render cuts](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-render-cuts-oracle.test.tsx), [Svelte hook tests](https://github.com/TanStack/db/blob/main/packages/svelte-db/tests/useLiveInfiniteQuery.svelte.test.ts), [review record](oracle-reviews/2026-10-05-infinite-first-paint.md) | Each driver records every value its framework published: React layout commits, Vue pre-flushes, and the Svelte construction value plus pre-effect runs. The shared suite checks that a synchronous source shows its ready first page on the first value, for a query callback and for a supplied collection that has not started, that every ready value in a fixed-source, fixed-query scenario equals the source prefix for its own page count, including an equal dependency replacement that keeps page depth, and that a mount requests an on-demand first window once, for a query callback and for a supplied collection whose finite window is wider than or equal to the one the hook needs, judged by the rows the on-demand source holds after mount. Under deferred acquisition, an on-demand first value is either the ready first page or a value with no rows, by framework: React and Svelte publish before the hook subscribes, Vue after. `preloaded-collection-first-paint` checks the documented route-loader pattern: a limited live query preloaded before mount shows its ready first page on the first value in every framework and asks the source for nothing more; a preload that does not count as demand never settles there. Pre-commit work has no shared form, because Vue setup is its commit. The React driver owns renders that never commit, StrictMode double renders (one first-window request on React 19), retries after GC, a mounted replacement whose startup error must reach the error boundary, and a collection kept from an abandoned render that enters terminal error before its query is selected again, each as under `useLiveQuery`. The React source ID reuse oracle drives the infinite hook too. Svelte owns construction and superseded recomputes before the subscribing effect. In both, the hook acquires an on-demand source as often as `useLiveQuery` does under the same history, for either input form, direct or behind a live-query Collection. On-demand requests from renders that never commit are now owned by the live-query deferred acquisition oracle; the React render-cut oracle checks that an abandoned render makes no acquisition. Open cells, owned here: React 18 StrictMode, where neither hook keeps refs across the double render; a React commit held after the 50 ms unsubscribed GC floor, where a controlled-premise controller witness shows no non-ready value but no React scheduling witness exists; ready values in scenarios that write the source or replace the query, which need a per-step source model; a join of eager and on-demand sources, which needs a zero-acquisition check before commit; a supplied collection with no `.limit`, which sends one unbounded request on `main` too, because the controller cannot narrow it before the source request; and Svelte `state_unsafe_mutation` when a start inside `$derived.by` synchronously writes rows a peer is waiting for, shared with Svelte's `useLiveQuery`, which a two-component probe reproduces and the deferred-acquisition follow-up removes. | | Structural values and ordered primitives | [hash values](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-oracle.property.test.ts), [hash identity](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-identity-oracle.property.test.ts), [MultiSet consolidation](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/multiset-consolidate-oracle.property.test.ts), [hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-graph-oracle.property.test.ts), [mixed hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-mixed-graph-oracle.property.test.ts), [hash retry](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-failure-retry-oracle.property.test.ts), [murmur stream](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/murmur-stream-oracle.property.test.ts), [comparison](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts), [deep equality](https://github.com/TanStack/db/blob/main/packages/db/tests/utils-oracle.property.test.ts), [cursor](https://github.com/TanStack/db/blob/main/packages/db/tests/cursor-oracle.property.test.ts), [indexes](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update-oracle.property.test.ts), [BTree Map](https://github.com/TanStack/db/blob/main/packages/db/tests/btree-map-oracle.test.ts), [query identity](https://github.com/TanStack/db/blob/main/packages/db/tests/query/identity-output-shape-oracle.test.ts), [LIKE semantics](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/evaluators-oracle.test.ts) | Independent flat values, graph topology, algebraic laws, structural object keys in code-unit order (the flat-value, identity and consolidation grammars include keys that tie under locale collation), murmur byte framing against a byte-at-a-time reference (block mixing, the finalizer and symbol framing have no owner there), Map/group/sort recomputation, expression denotation, custom comparator dispatch/reference identity/index compatibility, LIKE wildcard refinement, compiled output bags, declared value-pair agreement between D2 equality keys and IR equality operands, and Collection collation with exact public scan/index order and one-index reuse across repeated reads. The auto-index cells cross BasicIndex and BTreeIndex; scan cells omit indexes. Both paths cover lexical and numeric `en-US` locale defaults plus an explicit ordinary locale override of a numeric default. An unindexed six-row work cell bounds Collection collation resolution to at most two reads per ordered scan: one index-path check and one scan clause. Other locales and ordered histories are outside this owner. The identity pair grammar covers primitives, signed zero, NaN/invalid Date, valid Date/timestamp, binary/Buffer, Temporal, nested references, symbols, and functions; fixed and sampled pairs do not prove every JavaScript value or hash collision freedom. Ref proxies are expressions rather than literal values in this lane. The deep-equality owner also pins that `deepEquals` ignores Map and Set insertion order, RegExp `lastIndex`, array holes, and typed-array class, the rules draft equality keeps. It also pins that objects of another class differ, that plain and null-prototype objects are one class, that a class instance without enumerable keys equals only itself, that URLs compare by `href`, and that typed-array elements compare like numbers. Its [Date/RegExp unit transfer](oracle-reviews/utils-unit-retirement-round-7.md) adds fixed and generated direct-return witnesses that Date differs from its ISO text and epoch number, and RegExp differs from its source text, while preserving the original copy, changed-time, changed-flag, and changed-source operands. Other cross-type pairs and public change-event traces remain outside that transfer. The hash-values owner checks equal-hash consequences for independently constructed flat values. Distinct type identities are checked by the hash-identity owner through `equalHashValues`, including forced digest collisions; distinct 32-bit digests are not promised. The hash-identity owner checks `hash` and `equalHashValues` against one spec-level identity model: primitives with signed zero, NaN, and bigint; reference leaves (functions, registered handles, `File`, binary over 128 bytes); Date, binary, Temporal, and RegExp headers; array length and holes; Map and Set insertion order; and object keys in any order, for acyclic values and for cycles through arrays, Map values, Sets, and plain objects. Map/Set order sensitivity is pinned current behavior, not a promise. Arrays from another realm, RegExps with a `NaN` `lastIndex`, getters, proxies, other cycle carriers, and pairs that differ only in a back-edge target are outside its grammar; pinned cases keep equality's cross-realm length check and strict RegExp field comparison. `hash-work.test.ts` pins how visited values and array/RegExp headers count toward the structural work cap, and that equality copies a shared Map's entries once per distinct pair ([code-weight hash dispatch review](oracle-reviews/code-weight-hash-dispatch.md)). Custom string indexes do not optimize ordinary ranges, and custom cursors remain unsupported. The LIKE owner covers boolean string matching and bounded work, not nullish three-valued logic or a general Unicode collation contract. Unsupported composite cursors reject. The MultiSet consolidation owner checks keyed, single-type, and structural identity, signed zero and NaN in keyed and single-number data, the retained first record, zero-sum removal, and input record contents. It does not assert output order. Its keyed grammar includes cross-type key and value collisions, non-finite numeric keys, the `\|` delimiter, symbol and function identity, and a direct-object/object-tuple delimiter control from #1948. Eight named collision witnesses and both generated campaigns are RED on `c879ba6d` and GREEN with the repair ([#1948 review record](oracle-reviews/issue-1948-keyed-consolidation.md)); the [code-weight consolidation review](oracle-reviews/code-weight-multiset-consolidation.md) records the earlier exclusion. The keyed reference uses direct value/reference equality, including SameValueZero for numeric keys. The unkeyed fallback keeps its original key and value domains. The direct `MultiSet` owner does not prove which `@tanstack/db` query shapes produce primitive keyed values or mixed-type source keys; that needs a live-query production witness. The index owner checks lower-bound presence on bounded numeric histories after each add, update, delete, and build: an upper-only range ignores an unused `fromInclusive: false`, while an explicit exclusive `from` excludes its boundary. A source Map supplies expected sets for BasicIndex and BTreeIndex; wrong-output controls reject both boundary mistakes. It also checks exact `equalityLookup` and `lookup('eq')` after singleton case-insensitive `A → a → A` replacements and populated comparator-group updates, plus fresh-object same-group replacements in fixed and generated histories. These direct-index checks do not cover every JavaScript value, arbitrary comparators, or long Collection histories. The index owner enforces one invalid-comparator law across both BasicIndex and BTreeIndex: for each comparator, accepted numeric prefix (including the empty prefix), and probe operation (add, update, eq/range lookup, take, build), the operation throws exactly when it receives a `NaN` or non-number result and never when every comparison is valid. A `signed infinity` comparator is the valid control, and a custom collation `compare` supplied through `compareOptions` is checked the same way. Two BTree Map split histories check inserted-key reads, exact whole-range payloads/order, size, and extrema, proving split placement follows the insertion index without a post-mutation comparison. A rejected index add or remove leaves the index refining its accepted rows; update and build are not atomic. At the Collection boundary, both index types and both write paths (optimistic insert, sync commit) crash the collection: status becomes `error` and the next mutation throws, so a usable collection never holds rows its subscribers were not told about. Open: a sync source can still commit writes on a collection in `error` state, which stores unpublished rows; this is existing `markError` behavior and needs a lifecycle-owner witness. Comparator transitivity is outside this evidence. | | WHERE predicate publication | [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts), [`in` evaluator](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts) ([review](oracle-reviews/perf-in-constant-set.md)) | An independent Kleene evaluator judges `eq`/`not`/`and`/`or` predicates over strings, a normalization-prefixed string, booleans, numbers, `NaN`, a valid Date, `null`, a missing field, and virtual fields. Snapshot histories with pending optimistic inserts compare a live query, direct subscribers with and without initial state, and `currentStateAsChanges`; change histories apply multi-key sync transactions, including reinsertion of a key deleted earlier, and compare every consumer's key set after each commit, on scan and `BasicIndex` paths. Peer subscribers share one field with different literals, plus one on a virtual field, so every published change must reach exactly the subscribers whose predicate it can satisfy; routing mutants that ignored the previous value, ignored the field, delivered a change twice, or routed while stale published rows awaited reconciliation fail here. Fixed witnesses cover a layout-only publication reaching a filtered subscriber as one empty batch, a filtered subscriber's empty Collection-readiness batch, retraction of a vanished row after eager cleanup and restart, and the new row plus exact subscriber update payload when a same-key row stays TRUE. A stale-live-value mutant survives the generated membership checks but fails the payload witness. A focused [property-visibility test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-prefilter-property-visibility.test.ts) compares the public unindexed snapshot with the enriched-row contract for inherited, non-enumerable, enumerable-own, and nested getter paths; the pre-fix stored-row shortcut failed three of its four controls. Other property-descriptor and stateful-getter histories remain outside those fixed controls. Hostile mutants for FALSE-for-UNKNOWN `eq`, `or` or number-literal subscription prefilters, a prefilter that ignores the previous value, a skipped readiness batch, a skip while stale published rows await reconciliation, and an unindexed snapshot scan that reads only synced rows while an optimistic insert or delete changes visibility passed the prior `@tanstack/db` suite and fail here. Change routing withholds a dropped row from an `eq` subscription before its sent key could be recorded, so a mutant that records dropped rows as sent is equivalent for routed predicates in this oracle. A focused `collection-subscription.test.ts` witness checks that a dropped insert or update, alone or beside a matching change, cannot advance a limited subscription's page offset under a routed `eq` and an unrouted `or` predicate; the unrouted cases beside a matching change kill that mutant, and the unrouted `alone` case does not. A loose-equality prefilter is an equivalent mutant: it only skips less. Skipping during truncate replay also survives; the replay's baseline diff re-derives the same retraction, so the guard keeps the prior dataflow without its own witness. Generated cleanup and restart histories for filtered subscribers remain open for the lifecycle publication owner. No-op updates, other comparison operators, Temporal and binary operands, joins, ordering, optimistic updates, and truncate are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. Change routing was later removed in favor of pooled live queries; the routing mutants above are historical, and the same peer laws now check plain subscription dispatch. The unindexed snapshot prefilter was restored for compiled queries as one stored-row `eq` test on a string or boolean literal. Its property-visibility test also covers the reverse direction: a stored row with an inherited or non-enumerable field must still match `isUndefined` on that field, which a scan that evaluated the full predicate on stored rows fails. The `in` evaluator owner checks `in` against an independent linear reference over Dates, `NaN`, `-0`, BigInt, bytes, Temporal values, objects and nullish items, for constant and row-supplied lists. Its one-polyfill Temporal model uses captured intrinsic methods to reject forged and borrowed brands, accepts genuine own display shadows, and models Duration by ten fields, including PT1H versus PT60M. Arbitrary custom prototype forgery, mixed Temporal implementations, WHERE publication, and indexed mixed ordinary-object/Temporal range order remain outside that owner. | -| Captured `inArray` query stages | [stage oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/captured-inarray-stages-oracle.test.ts) ([review](oracle-reviews/captured-inarray-publication.md)) | Twelve fixed public cuts include nine numeric stage and acquisition witnesses after one unrelated source transaction, a limited ORDER uniqueness counterexample, grouped SUM/min/max arguments in an untouched group, and an aggregate argument wrapped by a selected addition. The controlled on-demand cut checks an unfiltered, unordered, unlimited first request. The nine initial stage cuts, ORDER duplicate, and wrapped aggregate were RED→GREEN in the working tree; grouped SUM/min/max passes in workspace Vitest. The final DB suite passes 12,396 tests across 282 files; DB-IVM passes 693 across 45. Arbitrary values and histories, real providers, and exact-head review remain open. | +| Captured `inArray` query stages | [stage oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/captured-inarray-stages-oracle.test.ts) ([review](oracle-reviews/captured-inarray-publication.md)) | Twelve fixed public cuts include nine numeric stage and acquisition witnesses after one unrelated source transaction, a limited ORDER uniqueness counterexample, grouped SUM/min/max arguments in an untouched group, and an aggregate argument wrapped by a selected addition. The controlled on-demand cut checks an unfiltered, unordered, unlimited first request. The nine initial stage cuts, ORDER duplicate, and wrapped aggregate were RED→GREEN in the working tree; grouped SUM/min/max passes in workspace Vitest. The final DB suite passes 12,406 tests across 282 files; DB-IVM passes 693 across 45. Arbitrary values and histories, real providers, and exact-head review remain open. | | Joined result keys | [Joined result key oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/join-result-key-oracle.property.test.ts) | A nested-loop model recomputes the pair set of an inner, left, or full join over source keys drawn from plain, comma-bearing, bracket-bearing, and quoted strings, numbers beside the strings that print the same, both infinities, and `NaN`. After preload and each synced group change, the published rows must equal the model's pairs and the result key count must equal the row count. The comma-joined key encoding fails its two pinned histories and both campaigns; plain `JSON.stringify`, which prints infinities as the missing-side `null`, fails the pinned infinity history and both campaigns. A pinned history in which a `NaN`-keyed pair leaves and re-forms fails when the join index compares source-key prefixes with `===`. Joins over subqueries, more than two sources, custom `getKey`, and optimistic mutations are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. | | D2 Index storage | [Index refinement oracle](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/index-refinement-oracle.property.test.ts) | A plain-`Map` model sums multiplicities under a type-and-value identity in which `NaN` equals `NaN` and `-0` equals `0`. Histories of up to twelve additions to two keys mix prefixed arrays with `NaN`, `0`, `-0`, `1`, `'1'`, `1n`, and `'a'` prefixes and unprefixed values, with cancellation and reappearance. After each addition, `get` and `has` must equal the model. Comparing prefixes with `===`, which disagrees with the `Map` that holds them, fails three pinned histories and both campaigns; the `-0` control passes under both. Compaction, presence tracking, and structural payloads are outside this owner; the join operator tests and incrementalization law own operator behavior. | | Pooled live queries | [pooled live query oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/pooled-live-query-oracle.property.test.ts) | An independent `eq` model (nullish UNKNOWN, Date as timestamp, `NaN` equal to `NaN`, `-0` equal to `0`) gives membership; a live-query Collection compiled for the same query gives order, values, and status. Histories mix sync transactions, optimistic writes confirmed or rolled back, peer mounts and unmounts, and source cleanup and restart, including cleanup while a write is pending and a mount between cleanup and restart. Wholesale observers must match after every step, granular changes must match a reference observer by type, key, value, and previous value and replay to the model's rows, the layout revision must advance when keys reorder, and observing must never build the live-query Collection. Peers may add a residual `not(eq(...))` conjunct, which the model evaluates itself; a view that ignores it, reports an unfiltered `entries()`, or delivers a row entering or leaving it as an update fails it. Other residual operators use the same compiler evaluator but are not generated. Peers may also order their rows by `id` or by a `g` that can be null, with explicit `nulls`; order comes from the reference. A comparator that ignores direction or `nulls`, an order left out of the partition key, and key order instead of the query's order each fail the pinned reorder history and both campaigns. A partition that ignores a row's previous group, orders by arrival, skips normalization, skips the initial snapshot, follows the source's status after cleanup, or never terminates fails it. Two shared conformance scenarios (`eq-filter-rows`, `eq-filter-peers`) run the pooled path under React, Vue, Solid, Svelte, and Angular; a partition that ignores a row's previous group fails both under every adapter, and `eq-filter-peers` checks through `subscriberCount` that each adapter shares one source subscription per partition. A focused [release-timing test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/pooled-live-query-gc-oracle.test.ts) compares when a pooled query and a live-query Collection release their sources across `gcTime` 1, 100, 0, and `Infinity`, with and without a subscriber, and with no `gcTime`, where both use the live-query Collection default, checks that a partition waits for its views' longest `gcTime`, checks that a view resubscribed after its partition released follows later source writes as its live-query Collection does, including a view with a residual conjunct, and checks that a partition that subscribes again still serves new mounts from one source subscription. Further focused witnesses check that a partition keeps groups only for values that have rows or watchers, that a detached reader never sees a dropped group's revision reused, keep a pooled query's public Collection live while the view is observed, accept it as a query source, and check that a pooled query turns terminal when source cleanup starts while the adapter's cleanup is still pending, as a live-query Collection does. A pinned oracle witness checks that an `eq` path whose getter throws excludes the row on both paths; a pooled path that rethrows fails it. The pooled oracle's grammar never releases a partition, so release and resubscribe histories rely on these fixed witnesses; a fixed delay, a missing unsubscribed floor, a last-`gcTime`-wins rule, and releasing at `gcTime` 0 each fail it. The [2026-10-01 review](oracle-reviews/2026-10-01-pooled-live-queries.md) records each ORC outcome. | @@ -618,8 +618,15 @@ through a comparator group, and unbounded results; both index types receive the same public rule. A direct on-demand ordered subscription also checks an initial tied prefix and continuation from a unique rank into a tied rank. A custom-collation cursor includes an unsent string from the cursor's -comparator-equal group. Mutation histories, arbitrary callbacks, provider -ordering, and full live-query publication remain outside this owner. The separate +comparator-equal group. Escaped strings reach direct custom callbacks and +Collection custom collation as raw values across ordered snapshots, direct +range/take operations, and one subscription cursor; normalized exact lookup +still distinguishes their string keys. A mixed string/number/two-byte-view +snapshot preserves public scan order after indexing while equal byte content +shares one exact lookup key. An escaped literal remains distinct from the +BTree undefined traversal marker. Mutation histories, arbitrary callbacks, +provider ordering, other binary values, and full live-query publication remain +outside this owner. The separate [custom-index window selection oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/custom-index-window-selection-oracle.test.ts) checks the live-query compiler's receiving path. A controlled on-demand source starts with ranks 20 and 40 cached, then serves exact ordered requests from @@ -685,8 +692,8 @@ child deletion refreshes membership and that its first child request omits WHERE-focused working-tree and neighboring run passed 988 tests across 11 suites with no type errors; exact committed-head review remains pending. The graph-wide gate can increase full-source transfer for on-demand joins and includes. -The retained direct receiver also checks the interval after a list edit and -before a source transaction: public keys stay unchanged and no subscriber +The retained direct receiver also checks one microtask turn after a list edit +and before a source transaction: public keys stay unchanged and no subscriber callback runs. At the next transaction, one callback contains the complete delete/insert batch and reads the new whole public key set inside that callback. A split-batch recorder mutant fails at this checkpoint. @@ -697,7 +704,7 @@ use frozen `inArray` lists: immutable membership can retain index pushdown and finite predicate forwarding, while mutable captured membership requires full-source reach at the covered boundaries. Their 34 focused tests pass. `ExpressionLike` accepts these `ReadonlyArray` inputs, and a separate missing -error-catalog entry, code 243, is recorded. The full DB suite passes 12,396 +error-catalog entry, code 243, is recorded. The full DB suite passes 12,406 tests across 282 files; DB-IVM passes 693 tests across 45 files. Both builds and explicit source and test typechecks pass. Real providers, broader schedules and values, arbitrary list histories, and diff --git a/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md b/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md index f06c4adf41..3c5ddcf2fb 100644 --- a/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md +++ b/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md @@ -141,3 +141,13 @@ four range operators and operand arrangements, two index types, and public snapshot keys. Later publication, generated subclass histories, invalid Dates, arbitrary coercion methods, provider cursors, and other hosts still need their own witnesses before broader closure. + +## Exact-commit review of `5c97f18dc` + +A fresh read-only reviewer found no blocker in the tied-subclass Date-range +matrix. The frozen native timestamps predict strict exclusion and inclusive +inclusion of all three tied keys. Exact public key arrays, index lookup spies, +and the omitted-key wrong-result control reach the intended scan and indexed +checkpoints for both index types. The focused owner passed 18 tests with no +type errors. This verdict remains limited to ready in-memory snapshots and +the fixed rows, operators, and bounds described above. diff --git a/docs/contributing/oracle-reviews/captured-inarray-publication.md b/docs/contributing/oracle-reviews/captured-inarray-publication.md index f6bab6c4bd..df8dc2c9a1 100644 --- a/docs/contributing/oracle-reviews/captured-inarray-publication.md +++ b/docs/contributing/oracle-reviews/captured-inarray-publication.md @@ -194,3 +194,26 @@ outside this fixed stage owner. evidence needs a fresh hostile review tied to the exact committed head. The full DB and DB-IVM suites, builds, and source/test typechecks pass at the working-tree checkpoint above. + +## Exact-commit publication review of `5c97f18dc` + +A fresh read-only reviewer found no blocker in the retained direct WHERE +receiver's callback and pre-event checks. The accepted law requires a whole +current result after the next source transaction. The fixture changes +`[1]` to `[2]` and then `[2,1]`, while an excluded source row supplies each +event. The independent source-row model predicts all public keys. The +recorder preserves callback boundaries, checks one complete delete/insert +batch, and reads the whole public result during delivery. The focused +checkpoint passed with no type errors. The earlier pre-event assertion +covered only the synchronous call stack. The next repair waits one microtask +turn after each edit, then checks the old keys and zero callbacks again. The +33-test focused WHERE owner passes. Later timers, arbitrary asynchronous +schedules, and real providers remain outside this fixed history. A fresh +review of the committed microtask addition remains pending. + +The captured-list stage owner received a separate read-only review at +`d917ba4c9`. All twelve fixed checks passed. Four temporary public probes +also passed for repeated list edits, combined stages, an unmatched LEFT JOIN, +and a UNION refreshed by another source. Those probes were removed and do +not extend the committed owner's claim. Broader callback coherence and real +provider behavior remain open for that owner. diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md index 27aa7df308..67ffc7efde 100644 --- a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md @@ -131,3 +131,13 @@ that every byte sequence or all binary consumers were exercised. The focused comparison owner passed 117 tests with no Vitest type errors; explicit test TypeScript, ESLint, Prettier, and whitespace checks passed. Final exact-commit hostile review of this repair is pending. + +## Exact-commit review of `5c97f18dc` + +A fresh read-only reviewer found no blocker for the five retired same-realm +comparison blocks or the two fixed byte positions. It checked the old unit +assertions, the independent tagged model, production comparison code, and the +new direct equality and normalization returns. The first-64-byte fault fails +at the intended changed-byte assertion. The focused comparison owner passed +117 tests with no type errors. This verdict does not establish every byte +position, offset view, or public binary query path. diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md index 954f74a51f..f71a1b30a1 100644 --- a/docs/contributing/oracle-reviews/custom-index-order-selection.md +++ b/docs/contributing/oracle-reviews/custom-index-order-selection.md @@ -245,3 +245,55 @@ checks. The evidence covers these two public snapshot paths and fixed histories; provider ordering, source mutations, and full live-query publication require separate receiving witnesses. Fresh hostile review of the committed repair remains pending. + +## Exact-commit hostile finding at `5c97f18dc` + +The fresh reviewer confirmed that ordered indexes could compare encoded +equality keys instead of source values. A custom string-length callback saw +an escaped representation of `"\u0000tanstack-db:x"`, so an indexed ascending +limit-one read selected a 17-character string before that 14-character +string. The unindexed public read selected the 14-character row. Both +BasicIndex and BTreeIndex failed. Direct take, range, and cursor reads also +misordered or omitted it, and a direct subscription continued to the wrong +row. The previous tie fixtures used ordinary strings, so they never crossed +the encoding boundary. + +The same encoded-order error moved native byte views into the string order +class. A bounded 40-case hostile sweep crossed five value groups, both index +types, both null placements, and both order directions. String/byte, +number/byte, and equal-content distinct byte-view groups disagreed with the +public scan in their corresponding eight-cell lanes. String/Date and +nullish/byte controls agreed. Exact byte lookups stayed correct. This was an +order-operand bug, not an equality-key bug. + +The repaired oracle supplies raw string lengths for the custom callback, +checks direct ordered take/range/cursor and one subscription continuation, +then compares ready public scan and indexed snapshots. A mixed-value fixture +checks primitive-before-byte order and preserves the scan order of two +distinct equal-content byte views after indexing. Exact lookup still groups +those byte views. A literal string that resembles the old BTree undefined +marker stays distinct from undefined in both order and exact lookup. The +production repair keeps raw order operands and normalized exact keys, with a +unique undefined traversal marker for BTree. Six new public assertions and +four direct comparisons per index type were RED on `5c97f18dc`; the focused +649-test campaign passed afterward. A normalized-byte mutant fails both +public indexed comparisons, and an old string marker fails the BTree ordered +read. Both faults were restored. + +The earlier prefix helper's take spies establish that an index supplied the +keys. They do not prove a strict work bound: ReverseIndex can enumerate its +whole nullish group inside one finite take. The helper comment now limits +its claim to the requested prefix and boundary group at its own boundary. +The fixed rows and controlled subscription do not establish arbitrary value +mixtures, source mutations, remote provider order, or all live-query +publication schedules. A fresh hostile review of the next committed code +remains necessary. + +## Exact-commit window review of `5c97f18dc` + +A separate fresh reviewer found no blocker in the finite provider order-hint +repair. Physical provider order differs from rank order. Missing `orderBy` +fails the finite-request guard; without that guard, it fails the initial +public snapshot. All eight fixed window cases passed with no type errors. +This controlled provider does not prove real-provider collation, index-use +efficiency, ties, mutation, or replay. diff --git a/docs/contributing/oracle-reviews/perf-in-constant-set.md b/docs/contributing/oracle-reviews/perf-in-constant-set.md index be1de639fb..b9c4b8da97 100644 --- a/docs/contributing/oracle-reviews/perf-in-constant-set.md +++ b/docs/contributing/oracle-reviews/perf-in-constant-set.md @@ -148,3 +148,13 @@ Both false rejection and false acceptance failed at the compiled return before the repair; the sparse-hole and nullish controls pass afterward. The focused `in` owner passed 44 tests. This proves the named array-method boundary at evaluation, not every possible proxy or mutation during one evaluation. + +## Exact-commit review of `5c97f18dc` + +A fresh read-only reviewer found no blocker in the own-method repair. The +model walks present indices independently of production's native `some` +call. Fixed constant and row-supplied cases reject both lying-false and +lying-true methods after a slot edit; sparse and nullish controls preserve the +stated behavior. The compiled `in` and byte-work owners passed 120 tests with +no type errors. Generated lists remain dense and static. Arbitrary getters, +proxies, and mutation during one evaluation remain outside this verdict. diff --git a/docs/contributing/oracle-reviews/temporal-equality-alias.md b/docs/contributing/oracle-reviews/temporal-equality-alias.md index 27082b7cc6..022c61ecbc 100644 --- a/docs/contributing/oracle-reviews/temporal-equality-alias.md +++ b/docs/contributing/oracle-reviews/temporal-equality-alias.md @@ -332,3 +332,15 @@ default ordering of calendar-unit Durations requires `relativeTo`, so this evidence does not claim their default ordered-index behavior. The focused compiled and alias owners passed 75 tests. A fresh committed-head hostile review remains pending. + +## Exact-commit Duration review of `5c97f18dc` + +A fresh read-only reviewer found no blocker for the five new intrinsic-field +classes. Each fixture variant differs from its baseline in one captured +Duration field. Compiled `eq` and both `in` forms check both operand +directions, and an omit-one-field wrong answer fails the independent +membership comparison. The alias owner checks a public unindexed scan and +both built index types' exact lookups. The explicit text comparator supplies +only the index's required order. The two focused owners passed 75 tests with +no type errors. The five variants do not themselves exercise indexed public +reads, index removal, or default ordering of calendar-unit Durations. diff --git a/packages/db/src/collection/change-events.ts b/packages/db/src/collection/change-events.ts index 78656282f5..365288869e 100644 --- a/packages/db/src/collection/change-events.ts +++ b/packages/db/src/collection/change-events.ts @@ -418,9 +418,10 @@ function getOrderedKeys( /** * Takes an indexed ordered prefix, completing its comparator-equal boundary. * Public snapshots keep source keys ascending within ties even when a reversed - * index walks them backward. A finite read visits the indexed prefix and its - * boundary group; an unbounded read traverses all indexed keys. `from` marks - * an exclusive comparator position, including an explicit nullish value. + * index walks them backward. A finite read asks for an indexed prefix and + * completes its boundary group; it does not bound work inside the index. + * An unbounded read asks for all indexed keys. `from` marks an exclusive + * comparator position, including an explicit nullish value. */ export function takeOrderedIndexKeys< T extends object, diff --git a/packages/db/src/indexes/base-index.ts b/packages/db/src/indexes/base-index.ts index b3c0d4d78b..a38b70ad1e 100644 --- a/packages/db/src/indexes/base-index.ts +++ b/packages/db/src/indexes/base-index.ts @@ -222,7 +222,7 @@ export abstract class BaseIndex< return !this.hasCustomComparator } - /** Temporal uses raw values for order while retaining normalized exact keys. */ + /** Ordering sees raw values; exact lookup keeps its normalized key. */ protected orderValue(value: unknown, exactKey: unknown): unknown { return indexOrderValue(value, exactKey) } diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index c7840d95c9..78cd1b2b72 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -366,7 +366,7 @@ function brandedTemporal(value: unknown): BrandedTemporal | undefined { * This allows distinguishing between "start from beginning" (undefined parameter) * and "start from the key undefined" (actual undefined value in the tree). */ -export const UNDEFINED_SENTINEL = normalizedKey(`undefined`, ``) +export const UNDEFINED_SENTINEL = Symbol(`undefined index value`) /** * Normalize a value for comparison and Map key usage @@ -497,11 +497,9 @@ export function normalizeForBTree(value: any): any { return normalizeValue(value) } -/** Keep native Date and Temporal order separate from exact lookup keys. */ +/** Order raw values while keeping BTree's undefined position distinct from its traversal marker. */ export function indexOrderValue(value: any, exactKey: any): any { - return dateTimestamp(value) !== undefined || brandedTemporal(value) - ? value - : exactKey + return value === undefined ? exactKey : value } /** Supply custom comparators the Temporal display key used before raw ordering. */ @@ -514,9 +512,7 @@ export function indexCustomComparatorValue(value: unknown): unknown { /** Recover the exact lookup key from a stored ordered value. */ export function indexExactKey(value: any): any { - return dateTimestamp(value) !== undefined || brandedTemporal(value) - ? normalizeValue(value) - : value + return normalizeValue(value) } /** diff --git a/packages/db/tests/custom-index-order-selection-oracle.test.ts b/packages/db/tests/custom-index-order-selection-oracle.test.ts index 13e92fc10f..eaac34ee24 100644 --- a/packages/db/tests/custom-index-order-selection-oracle.test.ts +++ b/packages/db/tests/custom-index-order-selection-oracle.test.ts @@ -30,8 +30,13 @@ import { withOracleCleanup } from './utils.js' * establish key-ascending ties in public ordered snapshots. The witnesses * below add equal numeric and custom-collation values, nullish values, finite * windows, and unbounded reads. A direct on-demand ordered subscription - * checks initial and cursor snapshots of an equal-rank group. They do not - * claim mutation histories, provider ordering, or live-query publication. + * checks initial and cursor snapshots of an equal-rank group. Raw escaped + * strings must reach custom-collation callbacks without their exact-key + * encoding; direct range and cursor operations use those same positions. + * A mixed string, number, and byte fixture checks that opaque byte order + * still matches the public scan while byte content retains exact equality. + * They do not claim mutation histories, provider ordering, arbitrary binary + * values, or live-query publication. */ type Row = { id: string; rank: number } @@ -807,4 +812,269 @@ describe.each(indexTypes)(`%s public key ties`, (name, IndexType) => { } }, [() => cleanup?.()]) }) + + it(`passes raw escaped strings to a direct custom comparator`, () => { + const source = [ + { id: `a`, label: `abcdefghijklmnopq` }, + { id: `b`, label: `\u0000tanstack-db:x` }, + { id: `c`, label: `abcdefghij` }, + ] + const index = new IndexType(1, new PropRef([`label`]), undefined, { + compareFn: (left: string, right: string) => left.length - right.length, + }) + for (const row of source) index.add(row.id, row) + + // This callback's law is raw length: 10 < 14 < 17. Exact equality has + // its own normalized key, while every direct traversal boundary uses the + // callback position. Soft comparisons record all boundary failures on + // the original implementation before any production edit. + expect.soft(index.takeFromStart(3)).toEqual([`c`, `b`, `a`]) + expect.soft(index.take(3, source[2]!.label)).toEqual([`b`, `a`]) + expect.soft(index.take(3, source[0]!.label)).toEqual([]) + expect + .soft(index.rangeQuery({ from: source[2]!.label, to: source[0]!.label })) + .toEqual(new Set([`c`, `b`, `a`])) + expect(index.equalityLookup(source[1]!.label)).toEqual(new Set([`b`])) + }) + + it(`orders escaped strings by their raw custom-collation values`, async () => { + type LabelRow = { id: string; label: string } + const source: ReadonlyArray = [ + { id: `a`, label: `abcdefghijklmnopq` }, + { id: `b`, label: `\u0000tanstack-db:x` }, + { id: `c`, label: `abcdefghij` }, + ] + const compare = (left: string, right: string) => left.length - right.length + const model = [...source] + .sort( + (left, right) => + left.label.length - right.label.length || + (left.id < right.id ? -1 : left.id > right.id ? 1 : 0), + ) + .map(({ id }) => id) + let cleanup: (() => unknown) | undefined + await withOracleCleanup(async () => { + const collection = createCollection({ + id: `escaped-custom-collation-${name}`, + getKey: ({ id }) => id, + autoIndex: `off`, + defaultStringCollation: { stringSort: `custom`, compare }, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of source) write({ type: `insert`, value: row }) + commit() + markReady() + }, + }, + }) + cleanup = () => collection.cleanup() + await collection.stateWhenReady() + const orderBy = [ + { + expression: new PropRef([`label`]), + compareOptions: { + direction: `asc` as const, + nulls: `first` as const, + }, + }, + ] + const read = (limit?: number, optimizedOnly?: boolean) => + collection + .currentStateAsChanges({ orderBy, limit, optimizedOnly }) + ?.map(({ key }) => key) + + // The model reads the supplied string lengths, not an index key. The + // leading zero marks an ordinary user string; it has length 14, between + // c's 10 and a's 17. A finite prefix exposes the wrong encoded order. + expect(model).toEqual([`c`, `b`, `a`]) + expect(read()).toEqual(model) + expect(read(2)).toEqual(model.slice(0, 2)) + const index = collection.createIndex((row) => row.label, { + indexType: IndexType, + options: { + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `custom`, + compare, + }, + }, + }) + const take = vi.spyOn(index, `takeFromStart`) + expect(read(undefined, true)).toEqual(model) + expect(read(2, true)).toEqual(model.slice(0, 2)) + expect(take).toHaveBeenCalled() + + // Direct ordered traversal, both boundaries, and exact equality are + // separate observations of the same installed index. A cursor after a + // excludes every earlier callback position; equality still uses its + // normalized key, so another spelling cannot alias b. + expect(index.takeFromStart(3)).toEqual(model) + expect(index.take(3, source[2]!.label)).toEqual([`b`, `a`]) + expect(index.take(3, source[0]!.label)).toEqual([]) + expect( + index.rangeQuery({ from: source[2]!.label, to: source[0]!.label }), + ).toEqual(new Set(model)) + expect(index.equalityLookup(source[1]!.label)).toEqual(new Set([`b`])) + expect( + index.equalityLookup(`\u0000tanstack-db:string:\u0000tanstack-db:x`), + ).toEqual(new Set()) + }, [() => cleanup?.()]) + }) + + it(`continues a custom-collation subscription from the raw string position`, async () => { + type LabelRow = { id: string; label: string } + const source: ReadonlyArray = [ + { id: `a`, label: `abcdefghijklmnopq` }, + { id: `b`, label: `\u0000tanstack-db:x` }, + { id: `c`, label: `abcdefghij` }, + ] + const compare = (left: string, right: string) => left.length - right.length + let cleanup: (() => unknown) | undefined + await withOracleCleanup(async () => { + const collection = createCollection({ + id: `escaped-custom-cursor-${name}`, + getKey: ({ id }) => id, + autoIndex: `off`, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of source) write({ type: `insert`, value: row }) + commit() + markReady() + return { loadSubset: () => true } + }, + }, + }) + cleanup = () => collection.cleanup() + await collection.stateWhenReady() + const index = collection.createIndex((row) => row.label, { + indexType: IndexType, + options: { + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `custom`, + compare, + }, + }, + }) + const batches: Array> = [] + const subscription = collection.subscribeChanges( + (changes) => { + if (changes.length) batches.push(changes.map(({ key }) => key)) + }, + { includeInitialState: false }, + ) + try { + subscription.setOrderByIndex(index) + const orderBy = [ + { + expression: new PropRef([`label`]), + compareOptions: { + direction: `asc` as const, + nulls: `first` as const, + }, + }, + ] + subscription.requestLimitedSnapshot({ orderBy, limit: 1 }) + subscription.requestLimitedSnapshot({ + orderBy, + limit: 1, + minValues: [source[2]!.label], + }) + expect(batches).toEqual([[`c`], [`b`]]) + } finally { + subscription.unsubscribe() + } + }, [() => cleanup?.()]) + }) + + it(`keeps an escaped user string distinct from the undefined order marker`, () => { + const literal = `\u0000tanstack-db:undefined:` + const index = new IndexType(1, new PropRef([`value`]), undefined, { + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `lexical`, + }, + }) + index.add(`z`, { value: `z` }) + index.add(`literal`, { value: literal }) + index.add(`missing`, { value: undefined }) + + // BTree uses a traversal marker for undefined, but this exact string is + // ordinary user data. The two have different order and equality classes. + expect(index.takeFromStart(3)).toEqual([`missing`, `literal`, `z`]) + expect(index.rangeQuery({ from: literal, to: literal })).toEqual( + new Set([`literal`]), + ) + expect(index.equalityLookup(undefined)).toEqual(new Set([`missing`])) + expect(index.equalityLookup(literal)).toEqual(new Set([`literal`])) + }) + + it(`keeps raw byte order distinct from content-based equality`, async () => { + type MixedRow = { id: string; value: string | number | Uint8Array } + const firstByte = new Uint8Array([1]) + const secondByte = new Uint8Array([1]) + const source: ReadonlyArray = [ + { id: `s`, value: `z` }, + { id: `n`, value: 1 }, + { id: `z-byte`, value: firstByte }, + { id: `a-byte`, value: secondByte }, + ] + let cleanup: (() => unknown) | undefined + await withOracleCleanup(async () => { + const collection = createCollection({ + id: `mixed-byte-order-${name}`, + getKey: ({ id }) => id, + autoIndex: `off`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of source) write({ type: `insert`, value: row }) + commit() + markReady() + }, + }, + }) + cleanup = () => collection.cleanup() + await collection.stateWhenReady() + const orderBy = [ + { + expression: new PropRef([`value`]), + compareOptions: DEFAULT_COMPARE_OPTIONS, + }, + ] + const read = (optimizedOnly?: boolean) => + collection + .currentStateAsChanges({ orderBy, optimizedOnly }) + ?.map(({ key }) => key) + + // Public scan order is the independent receiving path for opaque byte + // references. The value-kind law puts string and number before bytes; + // the scan also records the runtime order of these two distinct views. + const scanned = read() + expect(scanned?.slice(0, 2)).toEqual([`s`, `n`]) + expect(scanned?.slice(2)).toEqual([`z-byte`, `a-byte`]) + expect(new Set(scanned)).toEqual(new Set([`s`, `n`, `z-byte`, `a-byte`])) + const index = collection.createIndex((row) => row.value, { + indexType: IndexType, + }) + const take = vi.spyOn(index, `takeFromStart`) + expect(read(true)).toEqual(scanned) + expect(take).toHaveBeenCalled() + expect(index.equalityLookup(new Uint8Array([1]))).toEqual( + new Set([`z-byte`, `a-byte`]), + ) + expect(index.equalityLookup(`\u0000tanstack-db:binary:\u0001`)).toEqual( + new Set(), + ) + }, [() => cleanup?.()]) + }) }) diff --git a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts index 79e2dcc9f3..59d2f40cde 100644 --- a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts +++ b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts @@ -87,10 +87,10 @@ * with that model. On-demand ordered, joined, and include receivers check that * the first acquisition is unfiltered when the captured list is mutable. * The retained direct query also checks that a list edit alone preserves the - * old public snapshot and sends no callback. At the next source transaction, - * one subscriber callback contains the complete insert/delete batch and reads - * the new whole result. It checks no duplicate event when the list did not - * change. Separate fixed + * old public snapshot and sends no callback, including after a microtask turn. + * At the next source transaction, one subscriber callback contains the + * complete insert/delete batch and reads the new whole result. It checks no + * duplicate event when the list did not change. Separate fixed * Date and byte-item cases change contents inside the captured list, then * check the next source event. An ordered limit witness deletes a source row * outside the old loaded prefix after a list edit; it requires the newly @@ -1473,6 +1473,9 @@ describe(`WHERE predicate publication oracle`, () => { // remains in place until a source event clocks the retained query. expect([...live.keys()].sort()).toEqual([`one`]) expect(batches).toEqual([]) + await Promise.resolve() + expect([...live.keys()].sort()).toEqual([`one`]) + expect(batches).toEqual([]) source.utils.begin() source.utils.write({ type: `update`, @@ -1494,6 +1497,9 @@ describe(`WHERE predicate publication oracle`, () => { members.push(1) expect([...live.keys()].sort()).toEqual([`two`]) expect(batches).toEqual([]) + await Promise.resolve() + expect([...live.keys()].sort()).toEqual([`two`]) + expect(batches).toEqual([]) source.utils.begin() source.utils.write({ type: `update`, From ea569cc2a451ce3eb8650daca8349286bdc33206 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 00:42:25 -0600 Subject: [PATCH 24/46] fix(db): align byte ordering and publication with native content --- .../fix-custom-index-order-selection.md | 2 +- docs/contributing/oracle-coverage.md | 39 +- .../captured-inarray-publication.md | 17 + .../custom-index-order-selection.md | 89 +++ packages/db/src/collection/state.ts | 17 +- packages/db/src/utils.ts | 16 +- packages/db/src/utils/comparison.ts | 45 +- .../tests/change-event-history-oracle.test.ts | 101 +++- .../tests/comparison-oracle.property.test.ts | 113 ++++ ...ustom-index-order-selection-oracle.test.ts | 515 +++++++++++++++++- ...dicate-publication-oracle.property.test.ts | 2 +- .../db/tests/utils-oracle.property.test.ts | 23 + 12 files changed, 936 insertions(+), 43 deletions(-) diff --git a/.changeset/fix-custom-index-order-selection.md b/.changeset/fix-custom-index-order-selection.md index a7a2251cf0..deb7c070ab 100644 --- a/.changeset/fix-custom-index-order-selection.md +++ b/.changeset/fix-custom-index-order-selection.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Use a custom-comparator index for exact lookup without selecting its order for an ordinary Collection `orderBy` read. Select a compatible ordinary index for ordered reads, including live-query window moves with cached rows. Eager indexing creates a compatible index when needed, including for equivalent value-left range predicates. +Use a custom-comparator index for exact lookup without selecting its order for an ordinary Collection `orderBy` read. Select a compatible ordinary index for ordered reads, including live-query window moves with cached rows. Eager indexing creates a compatible index when needed, including for equivalent value-left range predicates. Ordered indexes compare raw row values while exact lookup keeps normalized keys. This preserves custom collation of escaped strings. Default ordering compares native byte views by unsigned content and source-key ties. Equal-content replacements refresh custom index positions without publishing a value change. Changed byte content still publishes when a view shadows its own `length` property. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 6c4929878d..caf1a110cb 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -479,6 +479,18 @@ in each direction and distinct `normalizeValue` keys. The generated changed-byte grammar spans lengths 1–200, but its fixed 100-run campaign did not reach those long positions; the fixed witnesses reject a first-64-byte comparator. +The proposed default native-byte order is unsigned lexicographic content, +with shorter prefixes first and equal contents at one comparator position. +Direct signs cover fresh equal views, the 127/128 boundary, prefixes, +Buffer/Uint8Array pairs, a foreign native view, and an own `length` shadow. +A byte/object/byte triple checks transitivity without prescribing which kind +comes first. These are immediate comparator observations, not relational +`WHERE` comparisons or live-query publication. The deep-equality owner also +checks that an own `length` shadow cannot hide a changed intrinsic byte in +direct equality, including a nested row field, over fixed and random 1–20 +byte inputs. Other typed-array class transitions remain under its existing +class-agnostic equality law; whether such transitions must publish a source +change needs a separate contract witness. Exact `normalizeValue` results are direct utility observations. Indexed order, public Date range predicates, cursor continuation, proxies, deeper subclass chains, replaced Date prototypes or `getTime` methods, and other @@ -622,11 +634,20 @@ comparator-equal group. Escaped strings reach direct custom callbacks and Collection custom collation as raw values across ordered snapshots, direct range/take operations, and one subscription cursor; normalized exact lookup still distinguishes their string keys. A mixed string/number/two-byte-view -snapshot preserves public scan order after indexing while equal byte content -shares one exact lookup key. An escaped literal remains distinct from the -BTree undefined traversal marker. Mutation histories, arbitrary callbacks, -provider ordering, other binary values, and full live-query publication remain -outside this owner. The separate +snapshot and bounded native-byte fixtures compare unsigned content order and +source-key ties with the unindexed scan and both index types. They include the +127/128 boundary, prefixes, Buffer, a foreign view, and an own `length` +shadow; exact lookup still groups equal byte content. An escaped literal +remains distinct from the BTree undefined traversal marker. One source +history silently replaces an equal-content byte view and checks scan/index +order and key count. Another adopts a peer view, deletes, and reinserts its +key, checking ordered arrays and exact sets after every commit. A custom +callback history moves an equal-content view to a different callback rank and +checks direct take, range, and cursor positions. The working-tree version of +these histories rejected stale index positions and duplicate keys on +`d93ae1dd6`. Arbitrary callbacks and mutation histories, provider ordering, +other binary values, and full live-query publication remain outside this +owner. The separate [custom-index window selection oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/custom-index-window-selection-oracle.test.ts) checks the live-query compiler's receiving path. A controlled on-demand source starts with ranks 20 and 40 cached, then serves exact ordered requests from @@ -1453,6 +1474,14 @@ new returned-mutation check. The direct index refinement owner does not prove this Collection mutation path. Callback-time index agreement, the reported insert-seeded string-key setup, adapter cancellation, and non-local-only persistence remain outside this lane. +Its native-byte sync witness adds a one-key `[1]` to `[2]` replacement in which +both genuine views shadow own `length` as zero. After commit, one update with +the previous and current intrinsic bytes must reach the subscriber while +public state and BasicIndex or BTreeIndex exact lookup agree with `[2]`. A +fresh equal-content view then remains silent. The pre-fix implementation +updated state and indexes but emitted no message for the changed bytes. This +bounded sync cut does not cover optimistic draft admission, other typed-array +classes, or live-query notification. The [issue #1912 review record](oracle-reviews/issue-1912-eager-index-history.md) records the grammar controls, exact reviewed commit, and ORC audit. diff --git a/docs/contributing/oracle-reviews/captured-inarray-publication.md b/docs/contributing/oracle-reviews/captured-inarray-publication.md index df8dc2c9a1..3111d5e96c 100644 --- a/docs/contributing/oracle-reviews/captured-inarray-publication.md +++ b/docs/contributing/oracle-reviews/captured-inarray-publication.md @@ -217,3 +217,20 @@ also passed for repeated list edits, combined stages, an unmatched LEFT JOIN, and a UNION refreshed by another source. Those probes were removed and do not extend the committed owner's claim. Broader callback coherence and real provider behavior remain open for that owner. + +## Exact-commit microtask review of `d93ae1dd6` + +A fresh read-only reviewer found no behavior defect in the retained direct +WHERE history. The selected test passed with no type errors. The independent +source-row model predicts the current whole key set. The direct receiver checks +old keys and zero callbacks through one microtask after each list edit. After +the next source transaction, it checks one complete callback batch and reads +the current whole result during that callback. The compiler schedules list +refreshes from source input events; the reviewer found no list-only scheduling +path. The review found one contract-wording gap: the oracle said a list edit +"need not publish," while the architecture says it does not publish. The +follow-up changes that prose to the accepted law. The existing split-batch +control calibrates event coherence, but no separate delayed-schedule mutant +calibrates the new microtask checkpoint. Nested microtasks, timers, arbitrary +list/event histories, and real providers remain outside this witness. It does +not retire an additional legacy test block. diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md index f71a1b30a1..a040af0cdf 100644 --- a/docs/contributing/oracle-reviews/custom-index-order-selection.md +++ b/docs/contributing/oracle-reviews/custom-index-order-selection.md @@ -289,6 +289,95 @@ mixtures, source mutations, remote provider order, or all live-query publication schedules. A fresh hostile review of the next committed code remains necessary. +## Exact-commit hostile finding at `d93ae1dd6` + +Two independent read-only reviewers found a legal source-update history that +the ready-state byte witness missed. The public comparator orders distinct +native byte views by runtime reference ID, while change-event equality groups +views with equal contents. Replacing one row's byte view with a fresh +equal-content view therefore changed the public scan order but published no +update to its index. Both built-in index types kept the old order. A later +delete could remove the exact equality owner while leaving its old ordered +entry; reinsertion then produced a duplicate public key. The two-row +replacement and a four-row update/delete/reinsert history failed at exact +public indexed arrays and direct index traversals. The latter retained a +deleted key even while `keyCount` and exact lookup appeared correct. + +A third test-owned callback ranks three equal-content views at distinct +positions. After Collection replaces a row with another view, the direct +custom index must use the new callback position while exact lookup retains +content equality. This witness failed for BasicIndex and BTreeIndex at take, +range, and cursor comparisons. Direct index update histories had passed +because their callers supplied both raw values; Collection suppressed the +structurally equal source update before it reached the index. The repair must +refresh indexed raw positions from the visible before/after rows even when +change-event equality suppresses an ordinary change message. + +These witnesses establish indexed-versus-scan parity under the current default +comparator. They do not, by themselves, make runtime-reference ordering of +native bytes the permanent product contract. The default-byte order and its +live-query publication consequence need an explicit contract choice before +claiming this bug class closed. + +## Proposed native-byte order and hostile review + +This PR proposes unsigned lexicographic content order for genuine native +Uint8Arrays and Buffers in default ordered reads. The old comparator put fresh +equal-content views at distinct runtime-reference positions: replacing one +could move the public scan without a source event or index refresh. The +proposed comparator puts equal contents at one position and uses the public +source key to break row ties, so an equal-content replacement preserves the +ordinary ordered result and remains silent. A changed first byte orders by +unsigned value; a shorter equal prefix comes first. Relational `WHERE` +comparison remains a separate contract and was not changed to follow this +order. A user custom index comparator still reads each current raw view, so +Collection must refresh its index even when ordinary change-event equality +suppresses a source message. These rival predictions distinguish the design +choice from the original production behavior. + +The first content-order tests failed on `d93ae1dd6` at direct comparator, +unindexed scan, and indexed snapshot checkpoints: 11 tests failed and four +passed in the selected run after separating the `WHERE` assertion. With the +new default comparator and silent index refresh, the comparison and custom +index owners passed 162 tests. A fresh read-only hostile reviewer then found +that the proposed unsigned law could pass with signed-byte ordering; the +revised pair crosses 127/128. It also found missing cross-realm and mixed +ordinary-object distinctions. The revised direct owner uses a foreign native +view, an own `length` shadow, and a byte/object/byte transitivity triple. The +public fixture adds the 127/128, foreign-view, and length-shadow cases for +both index types. These owners now pass 172 tests. The mixed-kind check +protects total order without fixing whether byte values precede ordinary +objects. Full live-query `ORDER BY` materialization and publication after byte +updates remain outside these ready-snapshot checks. + +A second fresh hostile reviewer found two reference-model errors. Byte ties +used host locale collation rather than the public source key's code-unit order; +the equal-content replacement now uses `A` and `a` so that error would fail. +The byte model also called an instance's overrideable `.values()` method; it +now invokes the native typed-array iterator on each view. It confirmed the +index refresh reaches both built-in index types after a sync replacement and +found no further mismatch within those bounded histories. It did not find an +optimistic same-content replacement reaching Collection state: draft tracking +discards that assignment before index maintenance. Arbitrary optimistic +histories remain unclaimed. + +The same review found a separate public publication defect. A native `[1]` +view and a native `[2]` view that each shadow own `length` as zero were treated +as equal by `deepEquals`, although Collection state and exact index lookup +changed. Both index types emitted no source update; an unshadowed control did. +The direct deep-equality property and Collection change-event witness were RED +on the uncorrected implementation: two generated equality lanes and both +index-type publication cases failed at their intended comparisons. Reading +the native typed-array element count repaired this boundary; the four focused +owners then passed 693 tests with no type errors. A class transition between +Uint8Array and Uint16Array with equal elements presents a neighboring +deep-equality/query-equality tension. Existing general equality deliberately +ignores typed-array class, so that contract needs separate review before +changing publication for that transition. + +These review results are for the working tree. Exact-commit hostile review and +the final full-suite run remain pending. + ## Exact-commit window review of `5c97f18dc` A separate fresh reviewer found no blocker in the finite provider order-hint diff --git a/packages/db/src/collection/state.ts b/packages/db/src/collection/state.ts index 4962742bac..74721a2fe9 100644 --- a/packages/db/src/collection/state.ts +++ b/packages/db/src/collection/state.ts @@ -1184,6 +1184,7 @@ export class CollectionStateManager< } const events: Array> = [] + const indexOnlyChanges: Array> = [] let reappliedKeys: ReadonlySet | undefined if (hasTruncateSync) { // All queued transactions publish as one batch. Its clear prefix must @@ -1403,6 +1404,18 @@ export class CollectionStateManager< value: newVisibleValue, previousValue: withPreviousVirtualProps(previousVisibleValue), }) + } else if ( + previousVisibleValue !== newVisibleValue && + this.indexes.indexes.size > 0 + ) { + // Change-event equality can group rows whose raw index order differs. + // Refresh their index positions without publishing a value change. + indexOnlyChanges.push({ + type: `update`, + key, + value: newVisibleValue, + previousValue: withPreviousVirtualProps(previousVisibleValue), + }) } } @@ -1410,8 +1423,8 @@ export class CollectionStateManager< this.size = this.calculateSize() // Update indexes for all events before emitting - if (events.length > 0) { - this.indexes.updateIndexes(events) + if (events.length > 0 || indexOnlyChanges.length > 0) { + this.indexes.updateIndexes([...events, ...indexOnlyChanges]) } // End batching and emit all events (combines any batched events with sync events) diff --git a/packages/db/src/utils.ts b/packages/db/src/utils.ts index e472470ed2..1d2b15d6b3 100644 --- a/packages/db/src/utils.ts +++ b/packages/db/src/utils.ts @@ -9,6 +9,16 @@ interface TypedArray { [index: number]: number } +const typedArrayLength = Object.getOwnPropertyDescriptor( + Object.getPrototypeOf(Uint8Array.prototype), + `length`, +)!.get! + +/** Read the native element count even when a view shadows its length property. */ +export function nativeTypedArrayLength(value: ArrayBufferView): number { + return Reflect.apply(typedArrayLength, value, []) as number +} + /** * Deep equality function that compares two values recursively * Handles primitives, objects, arrays, Date, RegExp, Map, Set, TypedArrays, and Temporal objects @@ -229,15 +239,17 @@ export function deepEqualsInternal( ) { const typedA = a as unknown as TypedArray const typedB = b as unknown as TypedArray + const lengthA = nativeTypedArrayLength(a) + const lengthB = nativeTypedArrayLength(b) // Elements compare like numbers: -0 equals 0, NaN equals NaN. Only a // draft treats a change of typed-array class as a change. if ( (draft && Object.getPrototypeOf(a) !== Object.getPrototypeOf(b)) || - typedA.length !== typedB.length + lengthA !== lengthB ) return false - for (let i = 0; i < typedA.length; i++) { + for (let i = 0; i < lengthA; i++) { const x = typedA[i]! const y = typedB[i]! if (x !== y && !(x !== x && y !== y)) return false diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index 78cd1b2b72..4378a16a44 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -1,4 +1,4 @@ -import { isTemporal } from '../utils' +import { isTemporal, nativeTypedArrayLength } from '../utils' import { getRuntimeReferenceIdentity } from '../query/runtime-reference-identity' import { codedMessage, devBuild } from '../error-message' import type { CompareOptions } from '../query/builder/types' @@ -135,6 +135,21 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { return compareBrandedTemporalValues(a, b, aTemporal, bTemporal) } + // Native byte views use their unsigned contents for default order. This + // agrees with their content-based equality and gives fresh equal views the + // same position; source keys break ties between rows. + const aIsBytes = isUint8Array(a) + const bIsBytes = isUint8Array(b) + if (aIsBytes && bIsBytes) { + const aLength = nativeTypedArrayLength(a) + const bLength = nativeTypedArrayLength(b) + for (let index = 0; index < Math.min(aLength, bLength); index++) { + const difference = a[index]! - b[index]! + if (difference !== 0) return difference + } + return aLength - bLength + } + // Symbols have identity but no built-in order: relational comparison throws. // A stable runtime ID gives tree indexes a total order while preserving // equality only for the same symbol. @@ -164,14 +179,18 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { ? 1 : aTemporal ? 2 - : 3 + : aIsBytes + ? 3 + : 4 const bKind = Array.isArray(b) ? 0 : bDateTime !== undefined ? 1 : bTemporal ? 2 - : 3 + : bIsBytes + ? 3 + : 4 if (aKind !== bKind) return aKind - bKind const aId = getObjectId(a) const bId = getObjectId(b) @@ -249,15 +268,6 @@ const typedArrayTag = Object.getOwnPropertyDescriptor( Object.getPrototypeOf(Uint8Array.prototype), Symbol.toStringTag, )!.get! -const typedArrayByteLength = Object.getOwnPropertyDescriptor( - Object.getPrototypeOf(Uint8Array.prototype), - `byteLength`, -)!.get! - -function nativeByteLength(value: Uint8Array): number { - return Reflect.apply(typedArrayByteLength, value, []) as number -} - /** Recognize native Uint8Array values, including Buffers and foreign realms. */ export function isUint8Array(value: unknown): value is Uint8Array { return ( @@ -268,8 +278,8 @@ export function isUint8Array(value: unknown): value is Uint8Array { /** Compare two Uint8Arrays for content equality. */ function areUint8ArraysEqual(a: Uint8Array, b: Uint8Array): boolean { - const length = nativeByteLength(a) - if (length !== nativeByteLength(b)) { + const length = nativeTypedArrayLength(a) + if (length !== nativeTypedArrayLength(b)) { return false } for (let i = 0; i < length; i++) { @@ -306,7 +316,7 @@ function canonicalTemporalZone(zoneId: string): string { function normalizeBinary(value: Uint8Array): string { let bytes = `` - const length = nativeByteLength(value) + const length = nativeTypedArrayLength(value) for (let index = 0; index < length; index++) { bytes += String.fromCharCode(value[index]!) } @@ -510,11 +520,6 @@ export function indexCustomComparatorValue(value: unknown): unknown { : value } -/** Recover the exact lookup key from a stored ordered value. */ -export function indexExactKey(value: any): any { - return normalizeValue(value) -} - /** * Compare values using the equality semantics used by Map keys. */ diff --git a/packages/db/tests/change-event-history-oracle.test.ts b/packages/db/tests/change-event-history-oracle.test.ts index 005b450e22..b7222e9981 100644 --- a/packages/db/tests/change-event-history-oracle.test.ts +++ b/packages/db/tests/change-event-history-oracle.test.ts @@ -10,7 +10,7 @@ import { import { localOnlyCollectionOptions } from '../src/local-only' import { PropRef } from '../src/query/ir' import { oraclePropertyOptions, oracleRuns } from './oracle-config' -import { captureCreatedTransactions } from './utils' +import { captureCreatedTransactions, withOracleCleanup } from './utils' import type { LocalOnlyCollectionUtils } from '../src/local-only' import type { Collection } from '../src/index' import type { BaseIndex } from '../src/indexes/base-index' @@ -874,3 +874,102 @@ describe(`change-event history oracle`, () => { verifyGeneratedHistory, ) }) + +describe(`native-byte change publication`, () => { + /** + * A sync replacement with different intrinsic bytes changes the visible + * row, so one update must reach a subscriber after commit. An own `length` + * shadow is display state, not a way to erase the byte difference. The + * source's one-key before/after sequence supplies the expected payload; + * exact index lookup is a separate receiving path. An equal-byte copy then + * remains silent. This bounded history does not claim optimistic writes or + * other typed-array classes. + */ + it.each([ + [`BasicIndex`, BasicIndex], + [`BTreeIndex`, BTreeIndex], + ] as const)( + `publishes changed intrinsic bytes with %s`, + async (name, IndexType) => { + type ByteRow = { id: string; value: Uint8Array } + type ByteSync = Parameters[`sync`]>[0] + const shadow = (value: number) => + Object.defineProperty(Uint8Array.of(value), `length`, { value: 0 }) + const bytes = (view: Uint8Array) => + Array.from(Uint8Array.prototype.values.call(view)) + const before = shadow(1) + const after = shadow(2) + const equalCopy = shadow(2) + let sync!: ByteSync + let unsubscribe: (() => unknown) | undefined + let cleanup: (() => unknown) | undefined + + await withOracleCleanup(async () => { + const collection = createCollection({ + id: `native-byte-publication-${name}`, + getKey: ({ id }) => id, + autoIndex: `off`, + startSync: true, + sync: { + sync: (actions) => { + sync = actions + actions.begin() + actions.write({ + type: `insert`, + value: { id: `a`, value: before }, + }) + actions.commit() + actions.markReady() + }, + }, + }) + cleanup = () => collection.cleanup() + await collection.stateWhenReady() + const index = collection.createIndex((row) => row.value, { + indexType: IndexType, + }) + const messages: Array<{ + type: string + key: string + previous: Array | undefined + value: Array + }> = [] + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + messages.push({ + type: change.type, + key: change.key, + previous: + change.type === `update` + ? bytes(change.previousValue!.value) + : undefined, + value: bytes(change.value.value), + }) + } + }, + { includeInitialState: false }, + ) + unsubscribe = () => subscription.unsubscribe() + expect(messages).toEqual([]) + + sync.begin() + sync.write({ type: `update`, value: { id: `a`, value: after } }) + await sync.commit() + expect(bytes(collection.get(`a`)!.value)).toEqual([2]) + expect(index.equalityLookup(Uint8Array.of(1))).toEqual(new Set()) + expect(index.equalityLookup(Uint8Array.of(2))).toEqual(new Set([`a`])) + expect(messages).toEqual([ + { type: `update`, key: `a`, previous: [1], value: [2] }, + ]) + + sync.begin() + sync.write({ type: `update`, value: { id: `a`, value: equalCopy } }) + await sync.commit() + expect(bytes(collection.get(`a`)!.value)).toEqual([2]) + expect(index.equalityLookup(Uint8Array.of(2))).toEqual(new Set([`a`])) + expect(messages).toHaveLength(1) + }, [() => unsubscribe?.(), () => cleanup?.()]) + }, + ) +}) diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index 9df8ddb5a2..a9525d5fdc 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -1,3 +1,4 @@ +import { Buffer } from 'node:buffer' import { runInNewContext } from 'node:vm' import { describe, expect, it, vi } from 'vitest' import { fc, test as fcTest } from '@fast-check/vitest' @@ -59,6 +60,14 @@ import type { CompareOptions } from '../src/query/builder/types' * equality and key histories cover lengths 1..200. Fixed changes at byte 64 * of a 129-byte view and the last byte of a 200-byte view pin positions that * a short generated campaign need not sample. + * + * The proposed native-byte ORDER BY law compares genuine Uint8Arrays and + * Buffers by unsigned byte content, with a shorter equal prefix first. Equal + * contents compare at one position. Genuine foreign-realm views and own + * length shadows retain their native bytes. The mixed-object witness checks + * total-order transitivity without choosing which kind comes first. This + * bounded direct-return witness does not claim relational WHERE comparison, + * other typed-array classes, arbitrary object order, or mutation. */ const defaultOpts: CompareOptions = { @@ -1967,6 +1976,110 @@ describe(`native binary key content`, () => { ) }) +describe(`native byte content order`, () => { + // The independent rule reads only the supplied unsigned elements. It does + // not use the production object-identity allocator, comparator, or exact-key + // normalization. Each fresh pair exercises immediate comparator returns. + const model = (left: Uint8Array, right: Uint8Array): number => { + // Native iteration reads the byte slots even when an own `length` + // property shadows the public getter. + const leftBytes = Array.from(Uint8Array.prototype.values.call(left)) + const rightBytes = Array.from(Uint8Array.prototype.values.call(right)) + for ( + let index = 0; + index < Math.min(leftBytes.length, rightBytes.length); + index++ + ) { + const difference = leftBytes[index]! - rightBytes[index]! + if (difference !== 0) return difference + } + return leftBytes.length - rightBytes.length + } + + it.each([ + { + caseId: `equal-content distinct views`, + makePair: () => [Uint8Array.of(1), Uint8Array.of(1)] as const, + expectedSign: 0, + }, + { + caseId: `first differing unsigned byte`, + makePair: () => [Uint8Array.of(1, 255), Uint8Array.of(2, 0)] as const, + expectedSign: -1, + }, + { + caseId: `unsigned 127/128 boundary`, + makePair: () => [Uint8Array.of(127), Uint8Array.of(128)] as const, + expectedSign: -1, + }, + { + caseId: `foreign native byte view`, + makePair: () => + [ + runInNewContext(`new Uint8Array([1])`) as Uint8Array, + Uint8Array.of(2), + ] as const, + expectedSign: -1, + }, + { + caseId: `own length shadow`, + makePair: () => + [ + Uint8Array.of(1), + Object.defineProperty(Uint8Array.of(2), `length`, { value: 0 }), + ] as const, + expectedSign: -1, + }, + { + caseId: `shorter equal prefix`, + makePair: () => [Uint8Array.of(1), Uint8Array.of(1, 0)] as const, + expectedSign: -1, + }, + { + caseId: `Buffer and Uint8Array with equal content`, + makePair: () => [Uint8Array.of(1), Buffer.from([1])] as const, + expectedSign: 0, + }, + { + caseId: `Buffer and Uint8Array with differing content`, + makePair: () => [Uint8Array.of(1), Buffer.from([2])] as const, + expectedSign: -1, + }, + ])(`compares $caseId`, ({ makePair, expectedSign }) => { + const [lessOrEqual, greaterOrEqual] = makePair() + expect(sign(model(lessOrEqual, greaterOrEqual))).toBe(expectedSign) + const reverseSign = expectedSign === 0 ? 0 : -expectedSign + + // Call the reverse direction first. An identity-order implementation + // assigns its first runtime ID to the greater operand here; content order + // must still return the opposite sign. Both directions and descending + // have the same immediate checkpoint. + expect + .soft(sign(ascComparator(greaterOrEqual, lessOrEqual, defaultOpts))) + .toBe(reverseSign) + expect + .soft(sign(ascComparator(lessOrEqual, greaterOrEqual, defaultOpts))) + .toBe(expectedSign) + expect + .soft(sign(descComparator(lessOrEqual, greaterOrEqual, defaultOpts))) + .toBe(reverseSign) + }) + + it(`keeps byte content order transitive through ordinary objects`, () => { + const high = Uint8Array.of(2) + const ordinary = {} + const low = Uint8Array.of(1) + // These two mixed comparisons assign identity IDs in an implementation + // that wrongly groups bytes with ordinary objects. The content edge then + // closes a strict cycle: high < ordinary < low < high. + const highToOrdinary = ascComparator(high, ordinary, defaultOpts) + const ordinaryToLow = ascComparator(ordinary, low, defaultOpts) + const highToLow = ascComparator(high, low, defaultOpts) + expect(highToLow).toBeGreaterThan(0) + assertTransitive(highToOrdinary, ordinaryToLow, highToLow) + }) +}) + if (requestedProperty !== undefined) { it(`selects a registered comparison oracle property`, () => { expect(requestedProperty).toMatch(/^\d+$/) diff --git a/packages/db/tests/custom-index-order-selection-oracle.test.ts b/packages/db/tests/custom-index-order-selection-oracle.test.ts index eaac34ee24..60c3097394 100644 --- a/packages/db/tests/custom-index-order-selection-oracle.test.ts +++ b/packages/db/tests/custom-index-order-selection-oracle.test.ts @@ -1,3 +1,5 @@ +import { Buffer } from 'node:buffer' +import { runInNewContext } from 'node:vm' import { describe, expect, it, vi } from 'vitest' import { createCollection } from '../src/collection/index.js' import { BasicIndex } from '../src/indexes/basic-index.js' @@ -6,6 +8,7 @@ import { ReverseIndex } from '../src/indexes/reverse-index.js' import { Func, PropRef, Value } from '../src/query/ir.js' import { DEFAULT_COMPARE_OPTIONS } from '../src/utils.js' import { withOracleCleanup } from './utils.js' +import type { SyncConfig } from '../src/types.js' /** * An ordered Collection snapshot may use an index only when the index executes @@ -33,10 +36,19 @@ import { withOracleCleanup } from './utils.js' * checks initial and cursor snapshots of an equal-rank group. Raw escaped * strings must reach custom-collation callbacks without their exact-key * encoding; direct range and cursor operations use those same positions. - * A mixed string, number, and byte fixture checks that opaque byte order - * still matches the public scan while byte content retains exact equality. - * They do not claim mutation histories, provider ordering, arbitrary binary - * values, or live-query publication. + * This PR's proposed default native-byte order is unsigned lexicographic by + * content, with shorter prefixes first and source keys ascending at equal + * content. A mixed string, number, and byte fixture checks that order beside + * content-based exact equality. + * Bounded source histories replace one byte view with a fresh equal-content + * view, or another row's existing view, then delete and reinsert its key. + * They compare indexed and unindexed order with a reference based on source + * facts; exact membership comes only from byte content. Equal-content + * replacement keeps ordinary public order and publishes no source change. + * The maintainer's custom-comparator decision also requires each current row + * to follow the callback when equal-content views have distinct callback ranks. + * They do not claim arbitrary mutation histories, provider ordering, arbitrary + * binary values, or live-query publication. */ type Row = { id: string; rank: number } @@ -156,6 +168,27 @@ const indexTypes = [ [`BTreeIndex`, BTreeIndex], ] as const +// For genuine native byte views in this bounded domain, the first differing +// unsigned byte determines order; an exhausted prefix comes first. This model +// reads the supplied bytes and never calls the production comparator or index. +function compareByteContent(left: Uint8Array, right: Uint8Array): number { + const leftBytes = Array.from(Uint8Array.prototype.values.call(left)) + const rightBytes = Array.from(Uint8Array.prototype.values.call(right)) + for ( + let index = 0; + index < Math.min(leftBytes.length, rightBytes.length); + index++ + ) { + const difference = leftBytes[index]! - rightBytes[index]! + if (difference !== 0) return difference + } + return leftBytes.length - rightBytes.length +} + +function compareStringKeys(left: string, right: string): number { + return left < right ? -1 : left > right ? 1 : 0 +} + describe.each(indexTypes)( `%s custom comparator order selection`, (name, IndexType) => { @@ -1017,7 +1050,166 @@ describe.each(indexTypes)(`%s public key ties`, (name, IndexType) => { expect(index.equalityLookup(literal)).toEqual(new Set([`literal`])) }) - it(`keeps raw byte order distinct from content-based equality`, async () => { + it.each([ + { + caseId: `equal-content-key-tie`, + makeRows: () => { + const z = new Uint8Array([1]) + const a = new Uint8Array([1]) + return [ + { id: `a`, value: a }, + { id: `z`, value: z }, + ] + }, + expected: [`a`, `z`], + }, + { + caseId: `first-differing-byte`, + makeRows: () => { + const high = new Uint8Array([2, 0]) + const low = new Uint8Array([1, 255]) + return [ + { id: `high`, value: high }, + { id: `low`, value: low }, + ] + }, + expected: [`low`, `high`], + }, + { + caseId: `unsigned-127-128-boundary`, + makeRows: () => { + const low = Uint8Array.of(127) + const high = Uint8Array.of(128) + return [ + { id: `low`, value: low }, + { id: `high`, value: high }, + ] + }, + expected: [`low`, `high`], + }, + { + caseId: `foreign-native-byte-view`, + makeRows: () => { + const low = runInNewContext(`new Uint8Array([1])`) as Uint8Array + const high = Uint8Array.of(2) + return [ + { id: `foreign-low`, value: low }, + { id: `local-high`, value: high }, + ] + }, + expected: [`foreign-low`, `local-high`], + }, + { + caseId: `own-length-shadow`, + makeRows: () => { + const low = Uint8Array.of(1) + const high = Object.defineProperty(Uint8Array.of(2), `length`, { + value: 0, + }) + return [ + { id: `low`, value: low }, + { id: `high`, value: high }, + ] + }, + expected: [`low`, `high`], + }, + { + caseId: `shorter-prefix`, + makeRows: () => { + const long = new Uint8Array([1, 0]) + const short = new Uint8Array([1]) + return [ + { id: `long`, value: long }, + { id: `short`, value: short }, + ] + }, + expected: [`short`, `long`], + }, + { + caseId: `buffer-and-uint8array`, + makeRows: () => { + const buffer = Buffer.from([1]) + const view = new Uint8Array([1]) + return [ + { id: `a-view`, value: view }, + { id: `z-buffer`, value: buffer }, + ] + }, + expected: [`a-view`, `z-buffer`], + }, + ])(`orders native bytes by content / $caseId`, async (example) => { + type ByteRow = { id: string; value: Uint8Array } + const source: ReadonlyArray = example.makeRows() + // The independent model sorts supplied unsigned content and uses the + // source key only for content-equal values. The fixtures include orders + // that disagree with reference identity, signed bytes, and host collation. + const model = [...source] + .sort( + (left, right) => + compareByteContent(left.value, right.value) || + compareStringKeys(left.id, right.id), + ) + .map(({ id }) => id) + expect(model).toEqual(example.expected) + let cleanup: (() => unknown) | undefined + await withOracleCleanup(async () => { + const collection = createCollection({ + id: `byte-content-${name}-${example.caseId}`, + getKey: ({ id }) => id, + autoIndex: `off`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of source) write({ type: `insert`, value: row }) + commit() + markReady() + }, + }, + }) + cleanup = () => collection.cleanup() + await collection.stateWhenReady() + const orderBy = [ + { + expression: new PropRef([`value`]), + compareOptions: DEFAULT_COMPARE_OPTIONS, + }, + ] + const read = (optimizedOnly?: boolean) => + collection + .currentStateAsChanges({ orderBy, optimizedOnly }) + ?.map(({ key }) => key) + + // Read before index installation, then force the installed index path. + // Exact arrays preserve both order and multiplicity at readiness. + expect.soft(read(), `unindexed ${example.caseId}`).toEqual(model) + const index = collection.createIndex((row) => row.value, { + indexType: IndexType, + }) + const take = vi.spyOn(index, `takeFromStart`) + expect.soft(read(true), `indexed ${example.caseId}`).toEqual(model) + expect(take).toHaveBeenCalled() + expect.soft(index.takeFromStart(source.length)).toEqual(model) + expect(index.keyCount).toBe(source.length) + for (const row of source) { + expect( + index.equalityLookup( + Uint8Array.from(Uint8Array.prototype.values.call(row.value)), + ), + ).toEqual( + new Set( + source + .filter( + (other) => compareByteContent(other.value, row.value) === 0, + ) + .map(({ id }) => id), + ), + ) + } + }, [() => cleanup?.()]) + }) + + it(`keeps mixed-value byte order and content equality aligned`, async () => { type MixedRow = { id: string; value: string | number | Uint8Array } const firstByte = new Uint8Array([1]) const secondByte = new Uint8Array([1]) @@ -1056,13 +1248,11 @@ describe.each(indexTypes)(`%s public key ties`, (name, IndexType) => { .currentStateAsChanges({ orderBy, optimizedOnly }) ?.map(({ key }) => key) - // Public scan order is the independent receiving path for opaque byte - // references. The value-kind law puts string and number before bytes; - // the scan also records the runtime order of these two distinct views. + // The value-kind law puts string and number before native bytes. The + // two byte views have equal content, so public source keys break their + // tie even though source admission put z-byte first. const scanned = read() - expect(scanned?.slice(0, 2)).toEqual([`s`, `n`]) - expect(scanned?.slice(2)).toEqual([`z-byte`, `a-byte`]) - expect(new Set(scanned)).toEqual(new Set([`s`, `n`, `z-byte`, `a-byte`])) + expect(scanned).toEqual([`s`, `n`, `a-byte`, `z-byte`]) const index = collection.createIndex((row) => row.value, { indexType: IndexType, }) @@ -1077,4 +1267,307 @@ describe.each(indexTypes)(`%s public key ties`, (name, IndexType) => { ) }, [() => cleanup?.()]) }) + + it(`keeps a content-equal replacement in key order without a source event`, async () => { + type ByteRow = { id: string; value: Uint8Array } + type Sync = Parameters[`sync`]>[0] + const firstKey = `A` + const peerKey = `a` + const first = new Uint8Array([1]) + const peer = new Uint8Array([1]) + const replacement = new Uint8Array([1]) + const source = new Map([ + [peerKey, peer], + [firstKey, first], + ]) + let sync!: Sync + let cleanup: (() => unknown) | undefined + let unsubscribe: (() => unknown) | undefined + await withOracleCleanup(async () => { + const collection = createCollection({ + id: `byte-replacement-order-${name}`, + getKey: ({ id }) => id, + autoIndex: `off`, + startSync: true, + sync: { + sync: (actions) => { + sync = actions + actions.begin() + for (const [id, value] of source) + actions.write({ type: `insert`, value: { id, value } }) + actions.commit() + actions.markReady() + }, + }, + }) + cleanup = () => collection.cleanup() + await collection.stateWhenReady() + const orderBy = [ + { + expression: new PropRef([`value`]), + compareOptions: DEFAULT_COMPARE_OPTIONS, + }, + ] + const read = (optimizedOnly?: boolean) => + collection + .currentStateAsChanges({ orderBy, optimizedOnly }) + ?.map(({ key }) => key) + + // The plain source Map is the model. Every live value has byte content + // [1], so the public order is the source-key tie regardless of which + // native view supplies that content. The change-event equality law + // suppresses a source event for the equal-content replacement. + const model = () => + [...source] + .sort( + ([leftKey, leftValue], [rightKey, rightValue]) => + compareByteContent(leftValue, rightValue) || + compareStringKeys(leftKey, rightKey), + ) + .map(([key]) => key) + expect(model()).toEqual([firstKey, peerKey]) + expect.soft(read()).toEqual(model()) + const index = collection.createIndex((row) => row.value, { + indexType: IndexType, + }) + const take = vi.spyOn(index, `takeFromStart`) + expect.soft(read(true)).toEqual(model()) + expect(take).toHaveBeenCalled() + expect.soft(index.takeFromStart(2)).toEqual(model()) + expect(index.keyCount).toBe(source.size) + expect(index.equalityLookup(new Uint8Array([1]))).toEqual( + new Set(source.keys()), + ) + const events: Array<{ type: string; key: string }> = [] + const subscription = collection.subscribeChanges( + (changes) => { + events.push(...changes.map(({ type, key }) => ({ type, key }))) + }, + { includeInitialState: false }, + ) + unsubscribe = () => subscription.unsubscribe() + expect(events).toEqual([]) + + // Commit changes the retained source instance while preserving its + // content. The indexed read and scan must both retain source-key order; + // neither a duplicate key nor a value event is allowed at this cut. + sync.begin() + sync.write({ + type: `update`, + value: { id: firstKey, value: replacement }, + }) + source.set(firstKey, replacement) + await sync.commit() + expect(collection.state.get(firstKey)?.value).toBe(replacement) + expect(events).toEqual([]) + const indexed = read(true) + const direct = index.takeFromStart(2) + expect(index.keyCount).toBe(source.size) + expect(index.equalityLookup(new Uint8Array([1]))).toEqual( + new Set(source.keys()), + ) + expect(collection.removeIndex(index)).toBe(true) + const scanned = read() + expect.soft(scanned).toEqual(model()) + expect.soft(indexed).toEqual(model()) + expect.soft(direct).toEqual(model()) + }, [() => unsubscribe?.(), () => cleanup?.()]) + }) + + it(`conserves keys when a byte view adopts a peer value and is reinserted`, async () => { + type ByteRow = { id: string; value: Uint8Array } + type Sync = Parameters[`sync`]>[0] + const source = new Map([ + [`0`, new Uint8Array([1])], + [`1`, new Uint8Array([1])], + [`2`, new Uint8Array([2])], + [`3`, Buffer.from([1])], + ]) + let sync!: Sync + let cleanup: (() => unknown) | undefined + await withOracleCleanup(async () => { + const collection = createCollection({ + id: `byte-reinsertion-order-${name}`, + getKey: ({ id }) => id, + autoIndex: `off`, + startSync: true, + sync: { + sync: (actions) => { + sync = actions + actions.begin() + for (const [id, value] of source) + actions.write({ type: `insert`, value: { id, value } }) + actions.commit() + actions.markReady() + }, + }, + }) + cleanup = () => collection.cleanup() + await collection.stateWhenReady() + const orderBy = [ + { + expression: new PropRef([`value`]), + compareOptions: DEFAULT_COMPARE_OPTIONS, + }, + ] + const read = (optimizedOnly?: boolean) => + collection + .currentStateAsChanges({ orderBy, optimizedOnly }) + ?.map(({ key }) => key) + + // A plain Map models source row ownership. Unsigned content puts every + // [1] view before [2], regardless of its native class or identity; + // source keys break the [1] tie. Adopting the peer value, deleting, and + // reinserting key 0 change only current membership. This reference never + // reads an index bucket or the production comparator. + const expectedOrder = () => + [...source] + .sort( + ([leftKey, leftValue], [rightKey, rightValue]) => + compareByteContent(leftValue, rightValue) || + compareStringKeys(leftKey, rightKey), + ) + .map(([key]) => key) + expect(expectedOrder()).toEqual([`0`, `1`, `3`, `2`]) + expect.soft(read(), `initial unindexed order`).toEqual(expectedOrder()) + const expectedExact = (byte: number) => + new Set( + [...source] + .filter(([, value]) => value.length === 1 && value[0] === byte) + .map(([key]) => key), + ) + const index = collection.createIndex((row) => row.value, { + indexType: IndexType, + }) + const check = (phase: string) => { + const expected = expectedOrder() + expect + .soft(read(true), `${phase} public indexed order`) + .toEqual(expected) + expect + .soft( + index.takeFromStart(source.size + 1), + `${phase} direct index order`, + ) + .toEqual(expected) + expect(index.keyCount, `${phase} key count`).toBe(source.size) + expect(index.equalityLookup(new Uint8Array([1]))).toEqual( + expectedExact(1), + ) + expect(index.equalityLookup(new Uint8Array([2]))).toEqual( + expectedExact(2), + ) + } + + check(`initial`) + const peerValue = source.get(`1`)! + sync.begin() + sync.write({ type: `update`, value: { id: `0`, value: peerValue } }) + source.set(`0`, peerValue) + await sync.commit() + expect(collection.state.get(`0`)?.value).toBe(peerValue) + check(`after update`) + + sync.begin() + sync.write({ type: `delete`, key: `0` }) + source.delete(`0`) + await sync.commit() + expect(collection.state.has(`0`)).toBe(false) + check(`after delete`) + + sync.begin() + sync.write({ type: `insert`, value: { id: `0`, value: peerValue } }) + source.set(`0`, peerValue) + await sync.commit() + expect(collection.state.get(`0`)?.value).toBe(peerValue) + check(`after reinsert`) + + // Removing the index makes the same public ordered read use its scan. + // Its final keys are an independent receiving control for the history. + expect(collection.removeIndex(index)).toBe(true) + expect(read()).toEqual(expectedOrder()) + }, [() => cleanup?.()]) + }) + + it(`orders current byte views by a custom callback after equal-content replacement`, async () => { + type ByteRow = { id: string; value: Uint8Array } + type Sync = Parameters[`sync`]>[0] + const first = new Uint8Array([1]) + const peer = new Uint8Array([1]) + const replacement = new Uint8Array([1]) + const callbackRanks = new WeakMap([ + [first, 1], + [peer, 2], + [replacement, 3], + ]) + const source = new Map([ + [`first`, { value: first, rank: 1 }], + [`peer`, { value: peer, rank: 2 }], + ]) + let sync!: Sync + let cleanup: (() => unknown) | undefined + await withOracleCleanup(async () => { + const collection = createCollection({ + id: `custom-byte-replacement-${name}`, + getKey: ({ id }) => id, + autoIndex: `off`, + startSync: true, + sync: { + sync: (actions) => { + sync = actions + actions.begin() + for (const [id, { value }] of source) + actions.write({ type: `insert`, value: { id, value } }) + actions.commit() + actions.markReady() + }, + }, + }) + cleanup = () => collection.cleanup() + await collection.stateWhenReady() + const index = collection.createIndex((row) => row.value, { + indexType: IndexType, + options: { + compareFn: (left: Uint8Array, right: Uint8Array) => + callbackRanks.get(left)! - callbackRanks.get(right)!, + }, + }) + + // Source facts carry numeric ranks independently of index storage. + // The callback receives the raw view for each current row; exact lookup + // instead groups these three one-byte views by content. A same-content + // update changes the first row's callback rank from 1 to 3. + const ordered = () => + [...source] + .sort(([, left], [, right]) => left.rank - right.rank) + .map(([id]) => id) + const atOrAbovePeer = () => + new Set( + [...source].filter(([, { rank }]) => rank >= 2).map(([id]) => id), + ) + const afterPeer = () => ordered().filter((id) => source.get(id)!.rank > 2) + const exact = () => + new Set( + [...source] + .filter(([, { value }]) => value.length === 1 && value[0] === 1) + .map(([id]) => id), + ) + + expect(index.takeFromStart(2)).toEqual(ordered()) + expect(index.rangeQuery({ from: peer })).toEqual(atOrAbovePeer()) + expect(index.take(2, peer)).toEqual(afterPeer()) + expect(index.equalityLookup(new Uint8Array([1]))).toEqual(exact()) + + sync.begin() + sync.write({ type: `update`, value: { id: `first`, value: replacement } }) + source.set(`first`, { value: replacement, rank: 3 }) + await sync.commit() + expect(collection.state.get(`first`)?.value).toBe(replacement) + expect(index.keyCount).toBe(source.size) + expect(index.equalityLookup(new Uint8Array([1]))).toEqual(exact()) + expect.soft(index.takeFromStart(2)).toEqual(ordered()) + expect.soft(index.rangeQuery({ from: peer })).toEqual(atOrAbovePeer()) + expect.soft(index.take(2, peer)).toEqual(afterPeer()) + }, [() => cleanup?.()]) + }) }) diff --git a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts index 59d2f40cde..012ee17398 100644 --- a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts +++ b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts @@ -77,7 +77,7 @@ * including the zero timestamp class. * * Captured-list law: a caller-owned array passed to `inArray` remains live - * input. Editing that array alone need not publish. At the next delivered + * input. Editing that array alone does not publish. At the next delivered * source event, the entire retained result, including untouched rows, uses * its current members. The fixed model starts with authored source rows and * filters them against the list at that checkpoint; it does not reuse the diff --git a/packages/db/tests/utils-oracle.property.test.ts b/packages/db/tests/utils-oracle.property.test.ts index 23c943460d..c040778c3d 100644 --- a/packages/db/tests/utils-oracle.property.test.ts +++ b/packages/db/tests/utils-oracle.property.test.ts @@ -15,6 +15,9 @@ import { deepEquals, equalPersistedSnapshotValues } from '../src/utils' * pairs remain unequal, even when a primitive displays or encodes a Date or * RegExp value. Separate bounded graph laws construct corresponding rings * and acyclic copies without reusing the production walk. + * A native typed-array length shadow cannot change how many intrinsic + * elements participate in change-event equality. A supplied changed byte + * must remain unequal even when both views report an own `length` of zero. * * Arbitrary different cycle topologies are outside this model. Naming that * exclusion matters: more random examples cannot establish semantics the @@ -1131,6 +1134,26 @@ describe(`deepEquals property-based tests`, () => { }, ) + utilsProperty([fc.uint8Array({ minLength: 1, maxLength: 20 }), fc.nat(20)])( + `native bytes ignore own length shadows`, + (bytes, indexHint) => { + const changedIndex = indexHint % bytes.length + const changed = Uint8Array.from(bytes) + changed[changedIndex] = (changed[changedIndex]! + 1) % 256 + const shadow = (view: Uint8Array) => + Object.defineProperty(view, `length`, { value: 0 }) + const original = shadow(Uint8Array.from(bytes)) + const copy = shadow(Uint8Array.from(bytes)) + const different = shadow(changed) + + // The supplied byte sequence, rather than either public `length`, + // determines both expected outcomes at the direct equality return. + expectEqualityPair(original, copy, true) + expectEqualityPair(original, different, false) + expectEqualityPair({ value: original }, { value: different }, false) + }, + ) + utilsProperty([ fc.array(fc.oneof(fc.integer(), fc.constant(`hole`)), { minLength: 1, From 23a478e78f5e3c2f2927d225bbbe9ad86e49ff51 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 00:54:10 -0600 Subject: [PATCH 25/46] fix(db): keep foreign DataView sync updates coherent --- .../fix-custom-index-order-selection.md | 2 +- docs/contributing/oracle-coverage.md | 20 ++- .../custom-index-order-selection.md | 48 +++++- packages/db/src/utils.ts | 25 ++-- packages/db/src/utils/comparison.ts | 18 +-- .../tests/change-event-history-oracle.test.ts | 140 ++++++++++++++++-- .../tests/comparison-oracle.property.test.ts | 14 ++ ...ustom-index-order-selection-oracle.test.ts | 21 +++ .../db/tests/utils-oracle.property.test.ts | 23 +++ 9 files changed, 268 insertions(+), 43 deletions(-) diff --git a/.changeset/fix-custom-index-order-selection.md b/.changeset/fix-custom-index-order-selection.md index deb7c070ab..c4f381a442 100644 --- a/.changeset/fix-custom-index-order-selection.md +++ b/.changeset/fix-custom-index-order-selection.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Use a custom-comparator index for exact lookup without selecting its order for an ordinary Collection `orderBy` read. Select a compatible ordinary index for ordered reads, including live-query window moves with cached rows. Eager indexing creates a compatible index when needed, including for equivalent value-left range predicates. Ordered indexes compare raw row values while exact lookup keeps normalized keys. This preserves custom collation of escaped strings. Default ordering compares native byte views by unsigned content and source-key ties. Equal-content replacements refresh custom index positions without publishing a value change. Changed byte content still publishes when a view shadows its own `length` property. +Use a custom-comparator index for exact lookup without selecting its order for an ordinary Collection `orderBy` read. Select a compatible ordinary index for ordered reads, including live-query window moves with cached rows. Eager indexing creates a compatible index when needed, including for equivalent value-left range predicates. Ordered indexes compare raw row values while exact lookup keeps normalized keys. This preserves custom collation of escaped strings. Default ordering compares native byte views by unsigned content and source-key ties. Equal-content replacements refresh custom index positions without publishing a value change. Changed byte content still publishes when a view shadows its own `length` property. A sync update to a foreign-realm DataView now commits and publishes without leaving its index on the prior value. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index caf1a110cb..bce258e271 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -482,7 +482,11 @@ comparator. The proposed default native-byte order is unsigned lexicographic content, with shorter prefixes first and equal contents at one comparator position. Direct signs cover fresh equal views, the 127/128 boundary, prefixes, -Buffer/Uint8Array pairs, a foreign native view, and an own `length` shadow. +an equal first byte followed by different second bytes, Buffer/Uint8Array +pairs, a foreign native view, and own `length` and `byteLength` shadows. The +later-byte case and its ready public scan/index counterpart reject a +first-byte-only order +mutant at their direct comparison and exact ordered-array checkpoints. A byte/object/byte triple checks transitivity without prescribing which kind comes first. These are immediate comparator observations, not relational `WHERE` comparisons or live-query publication. The deep-equality owner also @@ -491,6 +495,15 @@ direct equality, including a nested row field, over fixed and random 1–20 byte inputs. Other typed-array class transitions remain under its existing class-agnostic equality law; whether such transitions must publish a source change needs a separate contract witness. +The same deep-equality owner checks two distinct keyless DataViews from one +foreign realm against the keyless-instance rule, including a forged display +tag, plus a mixed typed-array/DataView +pair. The Collection change-event owner receives that boundary through a +typed-array-to-foreign-DataView update and a same-realm foreign-DataView +replacement. Both commits must settle with public state, BasicIndex or +BTreeIndex exact lookup, and whole subscriber batches aligned. This rejects +the getter failure that changed the row while leaving its index and subscriber +behind. It does not assert content equality for DataViews. Exact `normalizeValue` results are direct utility observations. Indexed order, public Date range predicates, cursor continuation, proxies, deeper subclass chains, replaced Date prototypes or `getTime` methods, and other @@ -1480,8 +1493,9 @@ the previous and current intrinsic bytes must reach the subscriber while public state and BasicIndex or BTreeIndex exact lookup agree with `[2]`. A fresh equal-content view then remains silent. The pre-fix implementation updated state and indexes but emitted no message for the changed bytes. This -bounded sync cut does not cover optimistic draft admission, other typed-array -classes, or live-query notification. +bounded sync cut records whole callback batches, so even an empty callback +after the equal copy fails the silence check. It does not cover optimistic +draft admission, other typed-array classes, or live-query notification. The [issue #1912 review record](oracle-reviews/issue-1912-eager-index-history.md) records the grammar controls, exact reviewed commit, and ORC audit. diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md index a040af0cdf..a067282121 100644 --- a/docs/contributing/oracle-reviews/custom-index-order-selection.md +++ b/docs/contributing/oracle-reviews/custom-index-order-selection.md @@ -375,8 +375,7 @@ deep-equality/query-equality tension. Existing general equality deliberately ignores typed-array class, so that contract needs separate review before changing publication for that transition. -These review results are for the working tree. Exact-commit hostile review and -the final full-suite run remain pending. +These results preceded the exact-commit review recorded below. ## Exact-commit window review of `5c97f18dc` @@ -386,3 +385,48 @@ fails the finite-request guard; without that guard, it fails the initial public snapshot. All eight fixed window cases passed with no type errors. This controlled provider does not prove real-provider collation, index-use efficiency, ties, mutation, or replay. + +## Exact-commit hostile review of `ea569cc2a` + +Three read-only agents reviewed the committed byte/order and Collection +publication changes against the source and retirement ledgers. One found a +foreign-realm DataView regression: `ArrayBuffer.isView` accepted it, a local +`instanceof DataView` failed to exclude it, and the new native typed-array +length getter threw. A public sync commit had already changed the row, but +left BasicIndex and BTreeIndex on the old value and emitted no update. Direct +foreign-DataView equality and public typed-array-to-DataView then +DataView-to-DataView histories failed before the repair at the getter. The +native typed-array brand check now excludes DataViews across realms. Both +commits settle, with exact index lookup and subscriber batches matching the +public row. Distinct DataViews from one foreign realm have equal prototypes +but remain unequal by the established opaque-instance law. + +The publication reviewer also found an enforcement gap: flattening messages +would miss an extra empty callback after an equal-byte replacement. The +native-byte sync witness now records whole callback batches and rejects that +callback. Current production already suppresses it. This is a test repair, +not an observed product failure. + +The order reviewer put a first-byte/length comparator mutant in an isolated +test copy. All 172 prior comparison and custom-index tests passed, although +the mutant ignored every later byte. New `[1, 2]` versus `[1, 3]` direct +signs and a ready public `a:[1,3], z:[1,2]` order fail that mutant at both +unindexed and indexed comparisons for BasicIndex and BTreeIndex. The same +witnesses pass against production. A second mutant read an own `byteLength` +shadow instead of the native element count and passed those revised tests. +Direct and public byte-length-shadow fixtures now distinguish it. These +fixtures close the named finite sensitivity gaps; they do not prove arbitrary +byte lengths or live-query publication. + +Content equality and source-key tie behavior predate this PR; unsigned native +byte order is a proposed policy. It makes equal-content replacement stable +under default ordering, but byte relational `WHERE` comparisons remain a +separate rule. The current direct and ready-snapshot checks do not establish +how all byte range predicates, providers, or live queries should interact with +that ordering. + +The index reviewer found no additional mismatch in bounded 250-action sweeps +per built-in index for native-byte order, key conservation, and custom +callback positions. These sweeps are diagnostic evidence, not a replacement +for the committed oracle. A fresh review of the follow-up committed head is +still required. diff --git a/packages/db/src/utils.ts b/packages/db/src/utils.ts index 1d2b15d6b3..ac92fe4f78 100644 --- a/packages/db/src/utils.ts +++ b/packages/db/src/utils.ts @@ -13,6 +13,16 @@ const typedArrayLength = Object.getOwnPropertyDescriptor( Object.getPrototypeOf(Uint8Array.prototype), `length`, )!.get! +const typedArrayTag = Object.getOwnPropertyDescriptor( + Object.getPrototypeOf(Uint8Array.prototype), + Symbol.toStringTag, +)!.get! + +/** Read a native typed-array class across realms without trusting display tags. */ +export function nativeTypedArrayTag(value: unknown): string | undefined { + if (!ArrayBuffer.isView(value)) return undefined + return Reflect.apply(typedArrayTag, value, []) as string | undefined +} /** Read the native element count even when a view shadows its length property. */ export function nativeTypedArrayLength(value: ArrayBufferView): number { @@ -231,12 +241,9 @@ export function deepEqualsInternal( if (b instanceof Set) return false // Handle TypedArrays - if ( - ArrayBuffer.isView(a) && - ArrayBuffer.isView(b) && - !(a instanceof DataView) && - !(b instanceof DataView) - ) { + const typedArrayTagA = nativeTypedArrayTag(a) + const typedArrayTagB = nativeTypedArrayTag(b) + if (typedArrayTagA !== undefined && typedArrayTagB !== undefined) { const typedA = a as unknown as TypedArray const typedB = b as unknown as TypedArray const lengthA = nativeTypedArrayLength(a) @@ -258,11 +265,7 @@ export function deepEqualsInternal( return true } // Symmetric check: if b is TypedArray but a is not, they're not equal - if ( - ArrayBuffer.isView(b) && - !(b instanceof DataView) && - !ArrayBuffer.isView(a) - ) { + if (typedArrayTagB !== undefined && !ArrayBuffer.isView(a)) { return false } diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index 4378a16a44..7880ef75d0 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -1,4 +1,8 @@ -import { isTemporal, nativeTypedArrayLength } from '../utils' +import { + isTemporal, + nativeTypedArrayLength, + nativeTypedArrayTag, +} from '../utils' import { getRuntimeReferenceIdentity } from '../query/runtime-reference-identity' import { codedMessage, devBuild } from '../error-message' import type { CompareOptions } from '../query/builder/types' @@ -261,19 +265,9 @@ export const defaultComparator = makeComparator({ stringSort: `locale`, }) -// The native getter reads a view's typed-array class across realms even when -// its own display tag shadows Symbol.toStringTag. Buffer has the same native -// byte class as Uint8Array. -const typedArrayTag = Object.getOwnPropertyDescriptor( - Object.getPrototypeOf(Uint8Array.prototype), - Symbol.toStringTag, -)!.get! /** Recognize native Uint8Array values, including Buffers and foreign realms. */ export function isUint8Array(value: unknown): value is Uint8Array { - return ( - ArrayBuffer.isView(value) && - Reflect.apply(typedArrayTag, value, []) === `Uint8Array` - ) + return nativeTypedArrayTag(value) === `Uint8Array` } /** Compare two Uint8Arrays for content equality. */ diff --git a/packages/db/tests/change-event-history-oracle.test.ts b/packages/db/tests/change-event-history-oracle.test.ts index b7222e9981..3d9a4ae0b8 100644 --- a/packages/db/tests/change-event-history-oracle.test.ts +++ b/packages/db/tests/change-event-history-oracle.test.ts @@ -1,3 +1,4 @@ +import { runInNewContext } from 'node:vm' import { fc, test as fcTest } from '@fast-check/vitest' import { describe, expect, it, vi } from 'vitest' import { @@ -928,16 +929,18 @@ describe(`native-byte change publication`, () => { const index = collection.createIndex((row) => row.value, { indexType: IndexType, }) - const messages: Array<{ - type: string - key: string - previous: Array | undefined - value: Array - }> = [] + const batches: Array< + Array<{ + type: string + key: string + previous: Array | undefined + value: Array + }> + > = [] const subscription = collection.subscribeChanges( (changes) => { - for (const change of changes) { - messages.push({ + batches.push( + changes.map((change) => ({ type: change.type, key: change.key, previous: @@ -945,13 +948,13 @@ describe(`native-byte change publication`, () => { ? bytes(change.previousValue!.value) : undefined, value: bytes(change.value.value), - }) - } + })), + ) }, { includeInitialState: false }, ) unsubscribe = () => subscription.unsubscribe() - expect(messages).toEqual([]) + expect(batches).toEqual([]) sync.begin() sync.write({ type: `update`, value: { id: `a`, value: after } }) @@ -959,8 +962,8 @@ describe(`native-byte change publication`, () => { expect(bytes(collection.get(`a`)!.value)).toEqual([2]) expect(index.equalityLookup(Uint8Array.of(1))).toEqual(new Set()) expect(index.equalityLookup(Uint8Array.of(2))).toEqual(new Set([`a`])) - expect(messages).toEqual([ - { type: `update`, key: `a`, previous: [1], value: [2] }, + expect(batches).toEqual([ + [{ type: `update`, key: `a`, previous: [1], value: [2] }], ]) sync.begin() @@ -968,7 +971,116 @@ describe(`native-byte change publication`, () => { await sync.commit() expect(bytes(collection.get(`a`)!.value)).toEqual([2]) expect(index.equalityLookup(Uint8Array.of(2))).toEqual(new Set([`a`])) - expect(messages).toHaveLength(1) + expect(batches).toEqual([ + [{ type: `update`, key: `a`, previous: [1], value: [2] }], + ]) + }, [() => unsubscribe?.(), () => cleanup?.()]) + }, + ) +}) + +describe(`opaque DataView change publication`, () => { + /** + * A keyless DataView's bytes live outside enumerable row properties, so two + * distinct keyless DataViews remain distinct even when their bytes match. + * One sync history crosses from a typed array to a foreign DataView, then + * replaces that DataView with a fresh foreign one. The independent expected + * sequence is two updates. After each commit, public rows, exact index + * lookup, and each complete subscriber callback must agree on that step. + */ + it.each([ + [`BasicIndex`, BasicIndex], + [`BTreeIndex`, BTreeIndex], + ] as const)( + `publishes foreign DataView replacements with %s`, + async (name, IndexType) => { + type ViewRow = { id: string; value: Uint8Array | DataView } + type ViewSync = Parameters[`sync`]>[0] + const before = Uint8Array.of(1) + const [first, second] = runInNewContext( + `[new DataView(new Uint8Array([1]).buffer), + new DataView(new Uint8Array([1]).buffer)]`, + ) as [DataView, DataView] + const labels = new Map([ + [before, `bytes`], + [first, `first`], + [second, `second`], + ]) + let sync!: ViewSync + let unsubscribe: (() => unknown) | undefined + let cleanup: (() => unknown) | undefined + + await withOracleCleanup(async () => { + const collection = createCollection({ + id: `foreign-dataview-publication-${name}`, + getKey: ({ id }) => id, + autoIndex: `off`, + startSync: true, + sync: { + sync: (actions) => { + sync = actions + actions.begin() + actions.write({ + type: `insert`, + value: { id: `a`, value: before }, + }) + actions.commit() + actions.markReady() + }, + }, + }) + cleanup = () => collection.cleanup() + await collection.stateWhenReady() + const index = collection.createIndex((row) => row.value, { + indexType: IndexType, + }) + const batches: Array< + Array<{ + type: string + key: string + previous: string | undefined + value: string | undefined + }> + > = [] + const subscription = collection.subscribeChanges( + (changes) => { + batches.push( + changes.map((change) => ({ + type: change.type, + key: String(change.key), + previous: + change.type === `update` + ? labels.get(change.previousValue!.value) + : undefined, + value: labels.get(change.value.value), + })), + ) + }, + { includeInitialState: false }, + ) + unsubscribe = () => subscription.unsubscribe() + expect(batches).toEqual([]) + + sync.begin() + sync.write({ type: `update`, value: { id: `a`, value: first } }) + await sync.commit() + expect(collection.get(`a`)!.value).toBe(first) + expect(index.equalityLookup(before)).toEqual(new Set()) + expect(index.equalityLookup(first)).toEqual(new Set([`a`])) + expect(batches).toEqual([ + [{ type: `update`, key: `a`, previous: `bytes`, value: `first` }], + ]) + + sync.begin() + sync.write({ type: `update`, value: { id: `a`, value: second } }) + await sync.commit() + expect(collection.get(`a`)!.value).toBe(second) + expect(index.equalityLookup(first)).toEqual(new Set()) + expect(index.equalityLookup(second)).toEqual(new Set([`a`])) + expect(batches).toEqual([ + [{ type: `update`, key: `a`, previous: `bytes`, value: `first` }], + [{ type: `update`, key: `a`, previous: `first`, value: `second` }], + ]) }, [() => unsubscribe?.(), () => cleanup?.()]) }, ) diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index a9525d5fdc..64afc36f48 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -2007,6 +2007,11 @@ describe(`native byte content order`, () => { makePair: () => [Uint8Array.of(1, 255), Uint8Array.of(2, 0)] as const, expectedSign: -1, }, + { + caseId: `later differing byte with equal first byte`, + makePair: () => [Uint8Array.of(1, 2), Uint8Array.of(1, 3)] as const, + expectedSign: -1, + }, { caseId: `unsigned 127/128 boundary`, makePair: () => [Uint8Array.of(127), Uint8Array.of(128)] as const, @@ -2030,6 +2035,15 @@ describe(`native byte content order`, () => { ] as const, expectedSign: -1, }, + { + caseId: `own byteLength shadow`, + makePair: () => + [ + Uint8Array.of(1), + Object.defineProperty(Uint8Array.of(2), `byteLength`, { value: 0 }), + ] as const, + expectedSign: -1, + }, { caseId: `shorter equal prefix`, makePair: () => [Uint8Array.of(1), Uint8Array.of(1, 0)] as const, diff --git a/packages/db/tests/custom-index-order-selection-oracle.test.ts b/packages/db/tests/custom-index-order-selection-oracle.test.ts index 60c3097394..e25f248d26 100644 --- a/packages/db/tests/custom-index-order-selection-oracle.test.ts +++ b/packages/db/tests/custom-index-order-selection-oracle.test.ts @@ -1075,6 +1075,14 @@ describe.each(indexTypes)(`%s public key ties`, (name, IndexType) => { }, expected: [`low`, `high`], }, + { + caseId: `later-differing-byte`, + makeRows: () => [ + { id: `a`, value: Uint8Array.of(1, 3) }, + { id: `z`, value: Uint8Array.of(1, 2) }, + ], + expected: [`z`, `a`], + }, { caseId: `unsigned-127-128-boundary`, makeRows: () => { @@ -1113,6 +1121,19 @@ describe.each(indexTypes)(`%s public key ties`, (name, IndexType) => { }, expected: [`low`, `high`], }, + { + caseId: `own-byte-length-shadow`, + makeRows: () => [ + { id: `low`, value: Uint8Array.of(1) }, + { + id: `high`, + value: Object.defineProperty(Uint8Array.of(2), `byteLength`, { + value: 0, + }), + }, + ], + expected: [`low`, `high`], + }, { caseId: `shorter-prefix`, makeRows: () => { diff --git a/packages/db/tests/utils-oracle.property.test.ts b/packages/db/tests/utils-oracle.property.test.ts index c040778c3d..5febc6c4a5 100644 --- a/packages/db/tests/utils-oracle.property.test.ts +++ b/packages/db/tests/utils-oracle.property.test.ts @@ -1,3 +1,4 @@ +import { runInNewContext } from 'node:vm' import { describe, expect, it } from 'vitest' import { fc, test as fcTest } from '@fast-check/vitest' import { Temporal } from 'temporal-polyfill' @@ -18,6 +19,10 @@ import { deepEquals, equalPersistedSnapshotValues } from '../src/utils' * A native typed-array length shadow cannot change how many intrinsic * elements participate in change-event equality. A supplied changed byte * must remain unequal even when both views report an own `length` of zero. + * Keyless DataViews have hidden state but no typed-array elements. Distinct + * views from one foreign realm remain unequal by the keyless-instance law; + * pairing one with a typed array must return false without reading a typed- + * array length from the DataView, even if it spoofs a display tag. * * Arbitrary different cycle topologies are outside this model. Naming that * exclusion matters: more random examples cannot establish semantics the @@ -1154,6 +1159,24 @@ describe(`deepEquals property-based tests`, () => { }, ) + it(`keeps foreign DataViews opaque and distinct from typed arrays`, () => { + const [first, second] = runInNewContext( + `[new DataView(new Uint8Array([1]).buffer), + new DataView(new Uint8Array([1]).buffer)]`, + ) as [DataView, DataView] + const local = new DataView(Uint8Array.of(1).buffer) + const bytes = Uint8Array.of(1) + Object.defineProperty(first, Symbol.toStringTag, { + value: `Uint8Array`, + }) + + // Neither equal bytes nor a display tag supplies typed-array elements. + expectEqualityPair(first, first, true) + expectEqualityPair(first, second, false) + expectEqualityPair(local, first, false) + expectEqualityPair(bytes, first, false) + }) + utilsProperty([ fc.array(fc.oneof(fc.integer(), fc.constant(`hole`)), { minLength: 1, From 9f4b417f7bccad7f8ce433da98928167b3e43e01 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 01:02:38 -0600 Subject: [PATCH 26/46] test(db): close byte order and publication oracle gaps --- docs/contributing/oracle-coverage.md | 7 ++++- .../custom-index-order-selection.md | 31 +++++++++++++++++-- .../tests/change-event-history-oracle.test.ts | 21 +++++++++++-- .../tests/comparison-oracle.property.test.ts | 22 +++++++++++++ ...ustom-index-order-selection-oracle.test.ts | 8 +++++ 5 files changed, 84 insertions(+), 5 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index bce258e271..ff523b9be1 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -487,6 +487,9 @@ pairs, a foreign native view, and own `length` and `byteLength` shadows. The later-byte case and its ready public scan/index counterpart reject a first-byte-only order mutant at their direct comparison and exact ordered-array checkpoints. +Fixed and random direct samples vary a common prefix of 2–24 bytes before +the first differing byte; a third-byte ready public case checks both index +types and rejects a comparator that reads only the first two bytes. A byte/object/byte triple checks transitivity without prescribing which kind comes first. These are immediate comparator observations, not relational `WHERE` comparisons or live-query publication. The deep-equality owner also @@ -501,7 +504,9 @@ tag, plus a mixed typed-array/DataView pair. The Collection change-event owner receives that boundary through a typed-array-to-foreign-DataView update and a same-realm foreign-DataView replacement. Both commits must settle with public state, BasicIndex or -BTreeIndex exact lookup, and whole subscriber batches aligned. This rejects +BTreeIndex exact lookup, and whole subscriber batches aligned. The callback +also records the current public row and exact index keys, so a late index +refresh fails at the publication cut. This rejects the getter failure that changed the row while leaving its index and subscriber behind. It does not assert content equality for DataViews. Exact `normalizeValue` results are direct utility observations. Indexed diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md index a067282121..4c8f11f834 100644 --- a/docs/contributing/oracle-reviews/custom-index-order-selection.md +++ b/docs/contributing/oracle-reviews/custom-index-order-selection.md @@ -428,5 +428,32 @@ that ordering. The index reviewer found no additional mismatch in bounded 250-action sweeps per built-in index for native-byte order, key conservation, and custom callback positions. These sweeps are diagnostic evidence, not a replacement -for the committed oracle. A fresh review of the follow-up committed head is -still required. +for the committed oracle. + +## Exact-commit hostile review of `23a478e78` + +Two fresh read-only reviewers challenged the repaired byte and DataView +oracles. Neither found a further production defect in the paths exercised. +The byte reviewer showed that a comparator reading only the first two bytes +passed the prior direct and ready-Collection owners. A third-byte fixture +failed that mutant at the direct comparator, unindexed scan, and both built-in +index paths; it passed unchanged production. The direct owner now generates a +common prefix of 2–24 bytes before a differing byte, and the ready public +owner includes a third-byte witness with source keys opposed to content order. +The fixed and random direct runs vary the first-difference position; the +public witness checks one later position. + +The DataView reviewer moved index maintenance after subscriber notification +in an isolated copy. The prior two-step Collection oracle stayed green even +though the callback could read a new public row with an old index. Recording +the row and exact index lookup inside each callback failed that mutant and +passed unchanged production for both index types. The existing after-commit +checks remain. This strengthens the atomic-publication observation for one +key and two sync commits; it does not establish every callback schedule. + +The byte reviewer confirmed that relational `WHERE` still uses JavaScript +coercion for byte comparisons, while the proposed default ordered-read rule +uses unsigned byte content. The planner declined byte range optimization in +the reviewed public path, so indexed and unindexed results agreed there. The +two comparison rules remain a product-design question beyond this PR's byte +equality repair. diff --git a/packages/db/tests/change-event-history-oracle.test.ts b/packages/db/tests/change-event-history-oracle.test.ts index 3d9a4ae0b8..4a91f6089e 100644 --- a/packages/db/tests/change-event-history-oracle.test.ts +++ b/packages/db/tests/change-event-history-oracle.test.ts @@ -985,8 +985,8 @@ describe(`opaque DataView change publication`, () => { * distinct keyless DataViews remain distinct even when their bytes match. * One sync history crosses from a typed array to a foreign DataView, then * replaces that DataView with a fresh foreign one. The independent expected - * sequence is two updates. After each commit, public rows, exact index - * lookup, and each complete subscriber callback must agree on that step. + * sequence is two updates. At each subscriber callback, and again after + * commit, public rows and exact index lookup must agree on that step. */ it.each([ [`BasicIndex`, BasicIndex], @@ -1042,8 +1042,20 @@ describe(`opaque DataView change publication`, () => { value: string | undefined }> > = [] + const callbackSnapshots: Array<{ + row: string | undefined + indexKeys: Array + }> = [] const subscription = collection.subscribeChanges( (changes) => { + // Publication must expose the new row and index before invoking + // the subscriber. Recording inside the callback distinguishes a + // torn publication that repairs its index only after notification. + const published = collection.get(`a`)?.value + callbackSnapshots.push({ + row: labels.get(published), + indexKeys: published ? [...index.equalityLookup(published)] : [], + }) batches.push( changes.map((change) => ({ type: change.type, @@ -1070,6 +1082,7 @@ describe(`opaque DataView change publication`, () => { expect(batches).toEqual([ [{ type: `update`, key: `a`, previous: `bytes`, value: `first` }], ]) + expect(callbackSnapshots).toEqual([{ row: `first`, indexKeys: [`a`] }]) sync.begin() sync.write({ type: `update`, value: { id: `a`, value: second } }) @@ -1081,6 +1094,10 @@ describe(`opaque DataView change publication`, () => { [{ type: `update`, key: `a`, previous: `bytes`, value: `first` }], [{ type: `update`, key: `a`, previous: `first`, value: `second` }], ]) + expect(callbackSnapshots).toEqual([ + { row: `first`, indexKeys: [`a`] }, + { row: `second`, indexKeys: [`a`] }, + ]) }, [() => unsubscribe?.(), () => cleanup?.()]) }, ) diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index 64afc36f48..45e02ded3b 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -2079,6 +2079,28 @@ describe(`native byte content order`, () => { .toBe(reverseSign) }) + comparisonProperty([ + fc.array(fc.integer({ min: 0, max: 255 }), { + minLength: 2, + maxLength: 24, + }), + fc.integer({ min: 0, max: 254 }), + fc.array(fc.integer({ min: 0, max: 255 }), { maxLength: 8 }), + ])( + `compares the first differing byte after a common prefix`, + (prefix, lowerByte, suffix) => { + // The prefix varies the first-difference position. A comparator that + // inspects only one or two leading bytes agrees on the pinned examples + // but disagrees with this independent content rule. + const lower = Uint8Array.from([...prefix, lowerByte, ...suffix]) + const higher = Uint8Array.from([...prefix, lowerByte + 1, ...suffix]) + expect(sign(model(lower, higher))).toBe(-1) + expect(sign(ascComparator(higher, lower, defaultOpts))).toBe(1) + expect(sign(ascComparator(lower, higher, defaultOpts))).toBe(-1) + expect(sign(descComparator(lower, higher, defaultOpts))).toBe(1) + }, + ) + it(`keeps byte content order transitive through ordinary objects`, () => { const high = Uint8Array.of(2) const ordinary = {} diff --git a/packages/db/tests/custom-index-order-selection-oracle.test.ts b/packages/db/tests/custom-index-order-selection-oracle.test.ts index e25f248d26..7e2711f955 100644 --- a/packages/db/tests/custom-index-order-selection-oracle.test.ts +++ b/packages/db/tests/custom-index-order-selection-oracle.test.ts @@ -1083,6 +1083,14 @@ describe.each(indexTypes)(`%s public key ties`, (name, IndexType) => { ], expected: [`z`, `a`], }, + { + caseId: `third-differing-byte`, + makeRows: () => [ + { id: `a`, value: Uint8Array.of(1, 1, 3) }, + { id: `z`, value: Uint8Array.of(1, 1, 2) }, + ], + expected: [`z`, `a`], + }, { caseId: `unsigned-127-128-boundary`, makeRows: () => { From 429c0ea921aa9368bff11af6cec5589d109cafa9 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 07:35:05 -0600 Subject: [PATCH 27/46] test(db): check old index membership during publication --- docs/contributing/oracle-coverage.md | 5 +++-- .../captured-inarray-publication.md | 12 +++++++++++ .../custom-index-order-selection.md | 18 ++++++++++++++++ .../tests/change-event-history-oracle.test.ts | 21 ++++++++++++++----- 4 files changed, 49 insertions(+), 7 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 0c4efb9559..5491696b2e 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -505,8 +505,9 @@ pair. The Collection change-event owner receives that boundary through a typed-array-to-foreign-DataView update and a same-realm foreign-DataView replacement. Both commits must settle with public state, BasicIndex or BTreeIndex exact lookup, and whole subscriber batches aligned. The callback -also records the current public row and exact index keys, so a late index -refresh fails at the publication cut. This rejects +also records the current public row, new exact index keys, and absence of the +previous exact membership. A late index refresh or transient dual membership +fails at the publication cut. This rejects the getter failure that changed the row while leaving its index and subscriber behind. It does not assert content equality for DataViews. Exact `normalizeValue` results are direct utility observations. Indexed diff --git a/docs/contributing/oracle-reviews/captured-inarray-publication.md b/docs/contributing/oracle-reviews/captured-inarray-publication.md index d756eb02bb..723546a7a7 100644 --- a/docs/contributing/oracle-reviews/captured-inarray-publication.md +++ b/docs/contributing/oracle-reviews/captured-inarray-publication.md @@ -280,3 +280,15 @@ whole-result refresh. An Effect with `skipInitial` and a lagging on-demand source is another untested intersection. These are coverage limits, not observed production failures. The execution-based hostile review found no further defect in the two new witnesses. + +## Exact pushed-head review at `b51f0c79d` + +A read-only hostile reviewer reran the unbounded Collection and Effect +witnesses on the clean pushed head. Both passed with no type errors. Three +isolated mutants failed at their intended observations. Sending `orderBy` in +either first request failed the recorded no-hints request shape. Treating the +second graph-input revision as readiness failed the pending-status assertion +before subset settlement. Returning equality for every local comparison in +the unbounded captured-list path failed the opposed key/value order check. +The reviewer found no further defect in those two witnesses. Their stated +composition and provider limits still apply. diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md index 4c8f11f834..64c8437f38 100644 --- a/docs/contributing/oracle-reviews/custom-index-order-selection.md +++ b/docs/contributing/oracle-reviews/custom-index-order-selection.md @@ -457,3 +457,21 @@ uses unsigned byte content. The planner declined byte range optimization in the reviewed public path, so indexed and unindexed results agreed there. The two comparison rules remain a product-design question beyond this PR's byte equality repair. + +## Pushed-head byte and DataView recheck at `b51f0c79d` + +A fresh hostile reviewer checked the byte comparison, direct index, and +Collection publication paths against the retirement ledgers at the clean +pushed head. Ten receiving and retained files passed 1,246 tests. An isolated +comparator that read only the first two bytes failed the generated direct +comparison and both ready indexed order checks. No production defect appeared +in those paths. + +The reviewer found another callback-time enforcement gap. An isolated index +mutant kept both the old and new exact memberships during notification, then +removed the old membership before `commit()` settled. Both existing DataView +cases passed because they checked the new membership during notification and +the old membership only afterward. The revised callback snapshot also checks +that the old membership is absent. It fails the mutant in both index types and +passes unchanged production. This checks one-key, two-commit publication; it +does not prove every callback schedule or broader byte live-query behavior. diff --git a/packages/db/tests/change-event-history-oracle.test.ts b/packages/db/tests/change-event-history-oracle.test.ts index 4a91f6089e..9288e025ba 100644 --- a/packages/db/tests/change-event-history-oracle.test.ts +++ b/packages/db/tests/change-event-history-oracle.test.ts @@ -1045,16 +1045,25 @@ describe(`opaque DataView change publication`, () => { const callbackSnapshots: Array<{ row: string | undefined indexKeys: Array + previousIndexKeys: Array }> = [] const subscription = collection.subscribeChanges( (changes) => { // Publication must expose the new row and index before invoking - // the subscriber. Recording inside the callback distinguishes a - // torn publication that repairs its index only after notification. + // the subscriber, without retaining the old index membership. + // Recording inside the callback distinguishes a torn publication + // that repairs either side only after notification. const published = collection.get(`a`)?.value + const previous = + changes[0]?.type === `update` + ? changes[0].previousValue!.value + : undefined callbackSnapshots.push({ row: labels.get(published), indexKeys: published ? [...index.equalityLookup(published)] : [], + previousIndexKeys: previous + ? [...index.equalityLookup(previous)] + : [], }) batches.push( changes.map((change) => ({ @@ -1082,7 +1091,9 @@ describe(`opaque DataView change publication`, () => { expect(batches).toEqual([ [{ type: `update`, key: `a`, previous: `bytes`, value: `first` }], ]) - expect(callbackSnapshots).toEqual([{ row: `first`, indexKeys: [`a`] }]) + expect(callbackSnapshots).toEqual([ + { row: `first`, indexKeys: [`a`], previousIndexKeys: [] }, + ]) sync.begin() sync.write({ type: `update`, value: { id: `a`, value: second } }) @@ -1095,8 +1106,8 @@ describe(`opaque DataView change publication`, () => { [{ type: `update`, key: `a`, previous: `first`, value: `second` }], ]) expect(callbackSnapshots).toEqual([ - { row: `first`, indexKeys: [`a`] }, - { row: `second`, indexKeys: [`a`] }, + { row: `first`, indexKeys: [`a`], previousIndexKeys: [] }, + { row: `second`, indexKeys: [`a`], previousIndexKeys: [] }, ]) }, [() => unsubscribe?.(), () => cleanup?.()]) }, From ba474b66beb107345d655af6c6f67c014d7de291 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 07:39:29 -0600 Subject: [PATCH 28/46] test(query-db): keep fixed subset fixtures immutable --- .../query-db-collection/tests/query.test.ts | 32 +++++++++++++------ 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/packages/query-db-collection/tests/query.test.ts b/packages/query-db-collection/tests/query.test.ts index 1dbd38b2df..0f97550a27 100644 --- a/packages/query-db-collection/tests/query.test.ts +++ b/packages/query-db-collection/tests/query.test.ts @@ -3141,18 +3141,21 @@ describe(`QueryCollection`, () => { syncMode: `on-demand`, }), ) + // These are fixed provider demands; mutable lists acquire the full source. + const firstIds = Object.freeze([`1`, `2`]) + const secondIds = Object.freeze([`2`, `3`]) const firstLiveQuery = createLiveQueryCollection({ query: (q) => q .from({ item: collection }) - .where(({ item }) => inArray(item.id, [`1`, `2`])) + .where(({ item }) => inArray(item.id, firstIds)) .select(({ item }) => ({ id: item.id, name: item.name })), }) const secondLiveQuery = createLiveQueryCollection({ query: (q) => q .from({ item: collection }) - .where(({ item }) => inArray(item.id, [`2`, `3`])) + .where(({ item }) => inArray(item.id, secondIds)) .select(({ item }) => ({ id: item.id, name: item.name })), }) @@ -7165,17 +7168,20 @@ describe(`QueryCollection`, () => { syncMode: `on-demand`, }) const collection = createCollection(options) + // Separate fixed demands make shared row ownership observable. + const firstIds = Object.freeze([`1`, `2`]) + const secondIds = Object.freeze([`2`, `3`]) const firstSubset = createLiveQueryCollection({ query: (q) => q .from({ item: collection }) - .where(({ item }) => inArray(item.id, [`1`, `2`])), + .where(({ item }) => inArray(item.id, firstIds)), }) const secondSubset = createLiveQueryCollection({ query: (q) => q .from({ item: collection }) - .where(({ item }) => inArray(item.id, [`2`, `3`])), + .where(({ item }) => inArray(item.id, secondIds)), }) await firstSubset.preload() @@ -11202,6 +11208,12 @@ describe(`QueryCollection`, () => { }), ) + // These fixed lists retain distinct subset demands while writes are held. + const taskIds = Object.freeze([10]) + const allResources = Object.freeze([1, 2, 3, 4]) + const postInsertResources = Object.freeze([1, 2, 4]) + const firstTwoResources = Object.freeze([1, 2]) + // Complete server population is deliberately returned for every demand: // this fixture tests cache writeback/ownership, not backend filtering. const world = new Map([ @@ -11246,7 +11258,7 @@ describe(`QueryCollection`, () => { query: (q) => q .from({ assignment: collection }) - .where(({ assignment }) => inArray(assignment.task_id, [10])), + .where(({ assignment }) => inArray(assignment.task_id, taskIds)), }) cleanups.push(() => taskQuery.cleanup()) try { @@ -11271,7 +11283,7 @@ describe(`QueryCollection`, () => { q .from({ assignment: collection }) .where(({ assignment }) => - inArray(assignment.resource_id, [1, 2, 3, 4]), + inArray(assignment.resource_id, allResources), ), }) cleanups.push(() => workloadQuery.cleanup()) @@ -11335,7 +11347,7 @@ describe(`QueryCollection`, () => { q .from({ assignment: collection }) .where(({ assignment }) => - inArray(assignment.resource_id, [1, 2, 4]), + inArray(assignment.resource_id, postInsertResources), ), }) const firstLoad = startHeldWorkloadLoad() @@ -11376,7 +11388,7 @@ describe(`QueryCollection`, () => { q .from({ assignment: collection }) .where(({ assignment }) => - inArray(assignment.resource_id, [1, 2]), + inArray(assignment.resource_id, firstTwoResources), ), }) const deleteLoad = startHeldWorkloadLoad() @@ -11400,7 +11412,7 @@ describe(`QueryCollection`, () => { q .from({ assignment: collection }) .where(({ assignment }) => - inArray(assignment.resource_id, [1, 2, 3, 4]), + inArray(assignment.resource_id, allResources), ), }) await workloadQuery.preload() @@ -11431,7 +11443,7 @@ describe(`QueryCollection`, () => { q .from({ assignment: collection }) .where(({ assignment }) => - inArray(assignment.resource_id, [1, 2, 4]), + inArray(assignment.resource_id, postInsertResources), ), }) const secondLoad = startHeldWorkloadLoad() From da00b034b667528fe29e289f3d17fd4ea9a32b25 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 07:59:51 -0600 Subject: [PATCH 29/46] refactor(db): remove redundant index and live-list state --- packages/db/src/indexes/basic-index.ts | 46 +++++-------------- packages/db/src/indexes/btree-index.ts | 33 ++++++------- packages/db/src/query/compiler/index.ts | 20 +++----- .../query/compiler/mutable-list-refresh.ts | 17 +------ packages/db/tests/utils.ts | 25 ++++++---- 5 files changed, 51 insertions(+), 90 deletions(-) diff --git a/packages/db/src/indexes/basic-index.ts b/packages/db/src/indexes/basic-index.ts index 2dc74330e6..f2ff90b2e4 100644 --- a/packages/db/src/indexes/basic-index.ts +++ b/packages/db/src/indexes/basic-index.ts @@ -13,13 +13,6 @@ import { import type { CompareOptions } from '../query/builder/types.js' import type { IndexOperation } from './base-index.js' -// Map collapses signed zero, while a user comparator may order them apart. -const NEGATIVE_ZERO_ORDER_KEY = Symbol(`negative zero order key`) - -function orderedBucketKey(value: unknown): unknown { - return Object.is(value, -0) ? NEGATIVE_ZERO_ORDER_KEY : value -} - /** * Options for range queries */ @@ -65,7 +58,8 @@ export class BasicIndex< // Equality and comparator positions are independent: a custom comparator // may distinguish values that have one exact query-equality key. private sortedValues: Array = [] - private orderedBuckets = new Map>() + // Each bucket stays at the same array position as its order representative. + private orderedBuckets: Array> = [] // Set of all indexed PKs private indexedKeys = new Set() @@ -112,15 +106,10 @@ export class BasicIndex< insertIdx < this.sortedValues.length && this.compareFn(this.sortedValues[insertIdx], orderValue) === 0 ) { - this.orderedBuckets - .get(orderedBucketKey(this.sortedValues[insertIdx]))! - .set(key, orderValue) + this.orderedBuckets[insertIdx]!.set(key, orderValue) } else { this.sortedValues.splice(insertIdx, 0, orderValue) - this.orderedBuckets.set( - orderedBucketKey(orderValue), - new Map([[key, orderValue]]), - ) + this.orderedBuckets.splice(insertIdx, 0, new Map([[key, orderValue]])) } } @@ -170,11 +159,11 @@ export class BasicIndex< if (index >= this.sortedValues.length) return const representative = this.sortedValues[index] if (this.compareFn(representative, orderValue) !== 0) return - const members = this.orderedBuckets.get(orderedBucketKey(representative)) - if (!members?.has(key)) return + const members = this.orderedBuckets[index]! + if (!members.has(key)) return if (members.size === 1) { this.sortedValues.splice(index, 1) - this.orderedBuckets.delete(orderedBucketKey(representative)) + this.orderedBuckets.splice(index, 1) return } // Keep a live representative when its row leaves the comparator group. @@ -188,8 +177,6 @@ export class BasicIndex< if (index + 1 < this.sortedValues.length) this.compareFn(successor, this.sortedValues[index + 1]) this.sortedValues[index] = successor - this.orderedBuckets.delete(orderedBucketKey(representative)) - this.orderedBuckets.set(orderedBucketKey(successor), members) } members.delete(key) } @@ -270,15 +257,10 @@ export class BasicIndex< this.sortedValues.length > 0 && this.compareFn(representative, entry.orderValue) === 0 ) { - this.orderedBuckets - .get(orderedBucketKey(representative))! - .set(entry.key, entry.orderValue) + this.orderedBuckets.at(-1)!.set(entry.key, entry.orderValue) } else { this.sortedValues.push(entry.orderValue) - this.orderedBuckets.set( - orderedBucketKey(entry.orderValue), - new Map([[entry.key, entry.orderValue]]), - ) + this.orderedBuckets.push(new Map([[entry.key, entry.orderValue]])) } } } @@ -289,7 +271,7 @@ export class BasicIndex< clear(): void { this.valueMap.clear() this.sortedValues = [] - this.orderedBuckets.clear() + this.orderedBuckets = [] this.indexedKeys.clear() this.clearRangeValues() } @@ -401,9 +383,7 @@ export class BasicIndex< // Collect all keys in range for (let i = startIdx; i < endIdx; i++) { - for (const key of this.orderedBuckets - .get(orderedBucketKey(this.sortedValues[i]))! - .keys()) { + for (const key of this.orderedBuckets[i]!.keys()) { result.add(key) } } @@ -490,9 +470,7 @@ export class BasicIndex< const groupValue = this.sortedValues[index] const groupKeys: Array = [] do { - for (const key of this.orderedBuckets - .get(orderedBucketKey(this.sortedValues[index])) - ?.keys() ?? []) { + for (const key of this.orderedBuckets[index]!.keys()) { groupKeys.push(key) } index += step diff --git a/packages/db/src/indexes/btree-index.ts b/packages/db/src/indexes/btree-index.ts index 69cddcc202..d77de16de4 100644 --- a/packages/db/src/indexes/btree-index.ts +++ b/packages/db/src/indexes/btree-index.ts @@ -39,7 +39,6 @@ type OrderedBucket = { // One comparator position may contain several exact values, while one // exact value may occupy several custom comparator positions. values: Map - keys: Set } // Map uses SameValueZero, but a custom callback may order -0 apart from +0. @@ -67,7 +66,7 @@ export class BTreeIndex< private orderedEntries = new BTree>((a, b) => this.compareStored(a, b), ) - private valueMap = new Map }>() + private valueMap = new Map>() // Repeated identical callback operands belong to the same comparator // position. This avoids another tree search for every row in a hot bucket. private orderOperands = new Map< @@ -117,15 +116,14 @@ export class BTreeIndex< ): void { this.addToOrder(key, orderValue) const exact = this.valueMap.get(normalizedValue) - if (exact) exact.keys.add(key) - else this.valueMap.set(normalizedValue, { keys: new Set([key]) }) + if (exact) exact.add(key) + else this.valueMap.set(normalizedValue, new Set([key])) } private addToOrder(key: TKey, orderValue: unknown): void { const operand = this.orderOperand(orderValue) const cached = this.orderOperands.get(operand) if (cached) { - cached.bucket.keys.add(key) cached.bucket.values.set(key, orderValue) cached.owners++ return @@ -133,13 +131,11 @@ export class BTreeIndex< let orderedBucket = this.orderedEntries.get(orderValue) if (orderedBucket) { - orderedBucket.keys.add(key) orderedBucket.values.set(key, orderValue) } else { orderedBucket = { representative: orderValue, values: new Map([[key, orderValue]]), - keys: new Set([key]), } this.orderedEntries.set(orderValue, orderedBucket) } @@ -177,19 +173,19 @@ export class BTreeIndex< orderValue: unknown, ): void { const exact = this.valueMap.get(normalizedValue) - if (!exact?.keys.has(key)) return + if (!exact?.has(key)) return this.removeFromOrder(key, orderValue) - exact.keys.delete(key) - if (exact.keys.size === 0) this.valueMap.delete(normalizedValue) + exact.delete(key) + if (exact.size === 0) this.valueMap.delete(normalizedValue) } private removeFromOrder(key: TKey, orderValue: unknown): void { const operand = this.orderOperand(orderValue) const cached = this.orderOperands.get(operand) const orderedBucket = cached?.bucket ?? this.orderedEntries.get(orderValue) - if (!orderedBucket?.keys.has(key)) return + if (!orderedBucket?.values.has(key)) return const replacesRepresentative = - orderedBucket.keys.size > 1 && + orderedBucket.values.size > 1 && areSameValueZeroEqual( orderedBucket.representative, orderedBucket.values.get(key), @@ -197,7 +193,7 @@ export class BTreeIndex< // Tree writes compare first, so a comparator failure throws before the // index changes. A comparator-equal `set` overwrites the entry's key. - if (orderedBucket.keys.size === 1) { + if (orderedBucket.values.size === 1) { this.orderedEntries.delete(orderedBucket.representative) } else if (replacesRepresentative) { const values = orderedBucket.values.entries() @@ -208,7 +204,6 @@ export class BTreeIndex< this.orderedEntries.set(representative, orderedBucket) orderedBucket.representative = representative } - orderedBucket.keys.delete(key) orderedBucket.values.delete(key) if (cached && --cached.owners === 0) this.orderOperands.delete(operand) } @@ -234,7 +229,7 @@ export class BTreeIndex< const newOrder = this.orderValue(newIndexedValue, newValue) if ( areSameValueZeroEqual(oldValue, newValue) && - this.valueMap.get(newValue)?.keys.has(key) + this.valueMap.get(newValue)?.has(key) ) { if (!Object.is(oldOrder, newOrder)) { this.removeFromOrder(key, oldOrder) @@ -323,7 +318,7 @@ export class BTreeIndex< */ equalityLookup(value: any): Set { const normalizedValue = normalizeForBTree(value) - return new Set(this.valueMap.get(normalizedValue)?.keys ?? []) + return new Set(this.valueMap.get(normalizedValue) ?? []) } /** @@ -366,7 +361,7 @@ export class BTreeIndex< return } - bucket.keys.forEach((key) => result.add(key)) + for (const key of bucket.values.keys()) result.add(key) }, ) @@ -396,7 +391,7 @@ export class BTreeIndex< while ((pair = nextPair(key)) !== undefined && result.length < n) { key = pair[0] // Sort keys for deterministic order within a comparator position. - const sorted = Array.from(pair[1].keys).sort( + const sorted = Array.from(pair[1].values.keys()).sort( reversed ? compareKeysReversed : compareKeys, ) for (const ks of sorted) { @@ -473,7 +468,7 @@ export class BTreeIndex< for (const value of values) { const normalizedValue = normalizeForBTree(value) - const keys = this.valueMap.get(normalizedValue)?.keys + const keys = this.valueMap.get(normalizedValue) if (keys) { keys.forEach((key) => result.add(key)) } diff --git a/packages/db/src/query/compiler/index.ts b/packages/db/src/query/compiler/index.ts index 8dbf8a71b3..7a2a6a5324 100644 --- a/packages/db/src/query/compiler/index.ts +++ b/packages/db/src/query/compiler/index.ts @@ -373,7 +373,6 @@ export interface CompilationResult { } const valueIdentitiesByCache = new WeakMap() -const unfilteredQueryCaches = new WeakSet() const mutableListRevisions = new WeakMap() function getCompilationValueIdentity(cache: QueryCache): ValueIdentity { @@ -547,7 +546,6 @@ export function compileQuery(...args: CompileQueryArgs): CompilationResult { const cache = dynamic ? new WeakMap() : (args[8] ?? new WeakMap()) args[8] = cache if (dynamic) { - unfilteredQueryCaches.add(cache) const lists = [...mutableLists] let snapshots = lists.map(snapshotList) let revision = 0 @@ -579,11 +577,7 @@ export function compileQuery(...args: CompileQueryArgs): CompilationResult { ), ) } - try { - return compileQueryInternal(args, false) - } finally { - unfilteredQueryCaches.delete(cache) - } + return compileQueryInternal(args, false) } function compileQueryInternal( @@ -628,7 +622,7 @@ function compileQueryInternal( // Optimize the query before compilation const { optimizedQuery, sourceWhereClauses: aliasWhereClauses } = - unfilteredQueryCaches.has(cache) + mutableListRevisions.has(cache) ? { optimizedQuery: rawQuery, sourceWhereClauses: new Map>(), @@ -808,7 +802,7 @@ function compileQueryInternal( joinsParentDirectly, valueIdentity, parentKeyStream, - unfilteredQueryCaches.has(cache), + mutableListRevisions.has(cache), mutableRevision, ) } @@ -1037,7 +1031,7 @@ function compileQueryInternal( // Every child source event can refresh a mutable membership list, even // when the child does not match the current parent route. - if (!unfilteredQueryCaches.has(cache) && lazyTargets.length > 0) { + if (!mutableListRevisions.has(cache) && lazyTargets.length > 0) { // 1. Mark child source as lazy so CollectionSubscriber skips initial full load for (const target of lazyTargets) { lazySources.add(target.sourceId) @@ -1216,7 +1210,7 @@ function compileQueryInternal( valueIdentity, collectionId: mainCollectionId, sourceWhereClauses, - hasCapturedMutableList: unfilteredQueryCaches.has(cache), + hasCapturedMutableList: mutableListRevisions.has(cache), aliasToCollectionId, aliasRemapping, }) @@ -1495,7 +1489,7 @@ function compileQueryInternal( resultLimit, query.offset, includesGroupKeyFn, - unfilteredQueryCaches.has(cache), + mutableListRevisions.has(cache), query.orderBy?.some(({ expression }) => hasMutableInList(expression)) ? mutableRevision : undefined, @@ -1514,7 +1508,7 @@ function compileQueryInternal( pipeline: resultPipeline, valueIdentity, sourceWhereClauses, - hasCapturedMutableList: unfilteredQueryCaches.has(cache), + hasCapturedMutableList: mutableListRevisions.has(cache), aliasToCollectionId, aliasRemapping, includes: includesResults.length > 0 ? includesResults : undefined, diff --git a/packages/db/src/query/compiler/mutable-list-refresh.ts b/packages/db/src/query/compiler/mutable-list-refresh.ts index 0ef5562277..2d00da3c0c 100644 --- a/packages/db/src/query/compiler/mutable-list-refresh.ts +++ b/packages/db/src/query/compiler/mutable-list-refresh.ts @@ -17,22 +17,7 @@ export function refreshMap( input: IStreamBuilder>, revision: MutableListRevision, project: (entry: KeyValue) => O | undefined, - equivalent?: (previous: O, next: O) => boolean, ): IStreamBuilder { - const priorOutput = new WeakMap() - const projectRetained = (entry: KeyValue): O | undefined => { - const next = project(entry) - if (next === undefined) { - priorOutput.delete(entry) - return undefined - } - const previous = priorOutput.get(entry) - if (previous !== undefined && equivalent?.(previous, next)) { - return previous - } - priorOutput.set(entry, next) - return next - } const candidates = input.pipe( map((entry) => [0, entry] as KeyValue>), ) @@ -51,7 +36,7 @@ export function refreshMap( reduce((values) => values.flatMap(([entry, multiplicity]) => { if (multiplicity <= 0) return [] - const output = projectRetained(entry) + const output = project(entry) return output === undefined ? [] : [[output, multiplicity] as [O, number]] diff --git a/packages/db/tests/utils.ts b/packages/db/tests/utils.ts index c952b7674e..2927bc9bd9 100644 --- a/packages/db/tests/utils.ts +++ b/packages/db/tests/utils.ts @@ -599,9 +599,13 @@ export function withExpectedRejection( } type IndexInternals = { indexedKeys: Set } & ( - | { sortedValues: Array; valueMap: Map> } | { - valueMap: Map }> + sortedValues: Array + valueMap: Map> + orderedBuckets: Array> + } + | { + valueMap: Map> orderedEntries: { size: number minKey: () => unknown @@ -610,7 +614,10 @@ type IndexInternals = { indexedKeys: Set } & ( low: unknown, high: unknown, includeHigh: boolean, - onFound: (key: unknown, bucket: { keys: Set }) => void, + onFound: ( + key: unknown, + bucket: { values: Map }, + ) => void, ) => void } } @@ -636,8 +643,8 @@ export function valueMapData(index: object): Map> { const [internals] = indexInternals(index) if (`sortedValues` in internals) return internals.valueMap const result = new Map>() - for (const [key, bucket] of internals.valueMap) { - result.set(denormalizeUndefined(key), bucket.keys) + for (const [key, keys] of internals.valueMap) { + result.set(denormalizeUndefined(key), keys) } return result } @@ -649,16 +656,18 @@ export function orderedEntriesArray( const [internals, reversed] = indexInternals(index) let entries: Array<[unknown, Set]> if (`sortedValues` in internals) { - entries = internals.sortedValues.map((value) => [ + if (internals.sortedValues.length !== internals.orderedBuckets.length) + throw new Error(`BasicIndex order buckets are misaligned`) + entries = internals.sortedValues.map((value, position) => [ value, - internals.valueMap.get(value) ?? new Set(), + new Set(internals.orderedBuckets[position]!.keys()), ]) } else { const tree = internals.orderedEntries entries = [] if (tree.size > 0) { tree.forRange(tree.minKey(), tree.maxKey(), true, (key, bucket) => { - entries.push([denormalizeUndefined(key), bucket.keys]) + entries.push([denormalizeUndefined(key), new Set(bucket.values.keys())]) }) } } From 4ed5242dbef0e7dbf87b31ca97c2fd2bc7901cad Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 08:51:32 -0600 Subject: [PATCH 30/46] fix(db): close oracle review correctness gaps --- docs/contributing/oracle-coverage.md | 112 ++-- packages/db/src/indexes/base-index.ts | 23 +- packages/db/src/indexes/basic-index.ts | 29 +- packages/db/src/indexes/btree-index.ts | 39 +- packages/db/src/utils.ts | 71 ++- packages/db/src/utils/comparison.ts | 274 ++++------ .../tests/change-event-history-oracle.test.ts | 342 ++++++++++++ .../tests/comparison-oracle.property.test.ts | 153 +----- .../index-update-oracle.property.test.ts | 171 ++++++ .../in-evaluator-oracle.property.test.ts | 125 +---- .../temporal-equality-alias-oracle.test.ts | 498 +----------------- .../tests/temporal-index-order-oracle.test.ts | 271 +++++++++- .../db/tests/utils-oracle.property.test.ts | 359 +++++++++++++ packages/db/tests/utils.ts | 4 +- 14 files changed, 1438 insertions(+), 1033 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 5491696b2e..8f89487234 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -443,7 +443,7 @@ generic visitor behavior outside the named collectors. | Window-controller overlap | [shared infinite-query conformance](https://github.com/TanStack/db/blob/main/packages/db/tests/conformance/infinite-suite-oracle.ts), [React driver](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-conformance.test.tsx), [Vue driver](https://github.com/TanStack/db/blob/main/packages/vue-db/tests/infinite-query-conformance.test.ts), [Svelte driver](https://github.com/TanStack/db/blob/main/packages/svelte-db/tests/infinite-query-conformance.svelte.test.ts), [review record](oracle-reviews/dec-03-window-overlap.md) | Four or five ordered source rows, two window-settlement orders, and public snapshots after each settlement, a subscribed observer update, a detached controller read, and explicit recovery. Insertion and removal of the fifth row distinguish both continuation directions while an earlier error remains visible. The detached read follows a source change with no controller subscriber; its preloaded Collection retains the five-row window. The exact overlap uses the exported DB controller in each package realm. Framework hooks expose no controller preload, so this cell does not establish a hook scheduling path; direct hook paging remains in the adjacent shared scenarios. An unsubscribed controller has no notification claim. | | Infinite-query publication | [shared publication laws](https://github.com/TanStack/db/blob/main/packages/db/tests/conformance/infinite-suite-oracle.ts), [React render cuts](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-render-cuts-oracle.test.tsx), [Svelte hook tests](https://github.com/TanStack/db/blob/main/packages/svelte-db/tests/useLiveInfiniteQuery.svelte.test.ts), [review record](oracle-reviews/2026-10-05-infinite-first-paint.md) | Each driver records every value its framework published: React layout commits, Vue pre-flushes, and the Svelte construction value plus pre-effect runs. The shared suite checks that a synchronous source shows its ready first page on the first value, for a query callback and for a supplied collection that has not started, that every ready value in a fixed-source, fixed-query scenario equals the source prefix for its own page count, including an equal dependency replacement that keeps page depth, and that a mount requests an on-demand first window once, for a query callback and for a supplied collection whose finite window is wider than or equal to the one the hook needs, judged by the rows the on-demand source holds after mount. Under deferred acquisition, an on-demand first value is either the ready first page or a value with no rows, by framework: React and Svelte publish before the hook subscribes, Vue after. `preloaded-collection-first-paint` checks the documented route-loader pattern: a limited live query preloaded before mount shows its ready first page on the first value in every framework and asks the source for nothing more; a preload that does not count as demand never settles there. Pre-commit work has no shared form, because Vue setup is its commit. The React driver owns renders that never commit, StrictMode double renders (one first-window request on React 19), retries after GC, a mounted replacement whose startup error must reach the error boundary, and a collection kept from an abandoned render that enters terminal error before its query is selected again, each as under `useLiveQuery`. The React source ID reuse oracle drives the infinite hook too. Svelte owns construction and superseded recomputes before the subscribing effect. In both, the hook acquires an on-demand source as often as `useLiveQuery` does under the same history, for either input form, direct or behind a live-query Collection. On-demand requests from renders that never commit are now owned by the live-query deferred acquisition oracle; the React render-cut oracle checks that an abandoned render makes no acquisition. Open cells, owned here: React 18 StrictMode, where neither hook keeps refs across the double render; a React commit held after the 50 ms unsubscribed GC floor, where a controlled-premise controller witness shows no non-ready value but no React scheduling witness exists; ready values in scenarios that write the source or replace the query, which need a per-step source model; a join of eager and on-demand sources, which needs a zero-acquisition check before commit; a supplied collection with no `.limit`, which sends one unbounded request on `main` too, because the controller cannot narrow it before the source request; and Svelte `state_unsafe_mutation` when a start inside `$derived.by` synchronously writes rows a peer is waiting for, shared with Svelte's `useLiveQuery`, which a two-component probe reproduces and the deferred-acquisition follow-up removes. | | Structural values and ordered primitives | [hash values](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-oracle.property.test.ts), [hash identity](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-identity-oracle.property.test.ts), [MultiSet consolidation](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/multiset-consolidate-oracle.property.test.ts), [hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-graph-oracle.property.test.ts), [mixed hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-mixed-graph-oracle.property.test.ts), [hash retry](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-failure-retry-oracle.property.test.ts), [murmur stream](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/murmur-stream-oracle.property.test.ts), [comparison](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts), [deep equality](https://github.com/TanStack/db/blob/main/packages/db/tests/utils-oracle.property.test.ts), [cursor](https://github.com/TanStack/db/blob/main/packages/db/tests/cursor-oracle.property.test.ts), [indexes](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update-oracle.property.test.ts), [BTree Map](https://github.com/TanStack/db/blob/main/packages/db/tests/btree-map-oracle.test.ts), [query identity](https://github.com/TanStack/db/blob/main/packages/db/tests/query/identity-output-shape-oracle.test.ts), [LIKE semantics](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/evaluators-oracle.test.ts) | Independent flat values, graph topology, algebraic laws, structural object keys in code-unit order (the flat-value, identity and consolidation grammars include keys that tie under locale collation), murmur byte framing against a byte-at-a-time reference (block mixing, the finalizer and symbol framing have no owner there), Map/group/sort recomputation, expression denotation, custom comparator dispatch/reference identity/index compatibility, LIKE wildcard refinement, compiled output bags, declared value-pair agreement between D2 equality keys and IR equality operands, and Collection collation with exact public scan/index order and one-index reuse across repeated reads. The auto-index cells cross BasicIndex and BTreeIndex; scan cells omit indexes. Both paths cover lexical and numeric `en-US` locale defaults plus an explicit ordinary locale override of a numeric default. An unindexed six-row work cell bounds Collection collation resolution to at most two reads per ordered scan: one index-path check and one scan clause. Other locales and ordered histories are outside this owner. The identity pair grammar covers primitives, signed zero, NaN/invalid Date, valid Date/timestamp, binary/Buffer, Temporal, nested references, symbols, and functions; fixed and sampled pairs do not prove every JavaScript value or hash collision freedom. Ref proxies are expressions rather than literal values in this lane. The deep-equality owner also pins that `deepEquals` ignores Map and Set insertion order, RegExp `lastIndex`, array holes, and typed-array class, the rules draft equality keeps. It also pins that objects of another class differ, that plain and null-prototype objects are one class, that a class instance without enumerable keys equals only itself, that URLs compare by `href`, and that typed-array elements compare like numbers. Its [Date/RegExp unit transfer](oracle-reviews/utils-unit-retirement-round-7.md) adds fixed and generated direct-return witnesses that Date differs from its ISO text and epoch number, and RegExp differs from its source text, while preserving the original copy, changed-time, changed-flag, and changed-source operands. Other cross-type pairs and public change-event traces remain outside that transfer. The hash-values owner checks equal-hash consequences for independently constructed flat values. Distinct type identities are checked by the hash-identity owner through `equalHashValues`, including forced digest collisions; distinct 32-bit digests are not promised. The hash-identity owner checks `hash` and `equalHashValues` against one spec-level identity model: primitives with signed zero, NaN, and bigint; reference leaves (functions, registered handles, `File`, binary over 128 bytes); Date, binary, Temporal, and RegExp headers; array length and holes; Map and Set insertion order; and object keys in any order, for acyclic values and for cycles through arrays, Map values, Sets, and plain objects. Map/Set order sensitivity is pinned current behavior, not a promise. Arrays from another realm, RegExps with a `NaN` `lastIndex`, getters, proxies, other cycle carriers, and pairs that differ only in a back-edge target are outside its grammar; pinned cases keep equality's cross-realm length check and strict RegExp field comparison. `hash-work.test.ts` pins how visited values and array/RegExp headers count toward the structural work cap, and that equality copies a shared Map's entries once per distinct pair ([code-weight hash dispatch review](oracle-reviews/code-weight-hash-dispatch.md)). Custom string indexes do not optimize ordinary ranges, and custom cursors remain unsupported. The LIKE owner covers boolean string matching and bounded work, not nullish three-valued logic or a general Unicode collation contract. Unsupported composite cursors reject. The MultiSet consolidation owner checks keyed, single-type, and structural identity, signed zero and NaN in keyed and single-number data, the retained first record, zero-sum removal, and input record contents. It does not assert output order. Its keyed grammar includes cross-type key and value collisions, non-finite numeric keys, the `\|` delimiter, symbol and function identity, and a direct-object/object-tuple delimiter control from #1948. Eight named collision witnesses and both generated campaigns are RED on `c879ba6d` and GREEN with the repair ([#1948 review record](oracle-reviews/issue-1948-keyed-consolidation.md)); the [code-weight consolidation review](oracle-reviews/code-weight-multiset-consolidation.md) records the earlier exclusion. The keyed reference uses direct value/reference equality, including SameValueZero for numeric keys. The unkeyed fallback keeps its original key and value domains. The direct `MultiSet` owner does not prove which `@tanstack/db` query shapes produce primitive keyed values or mixed-type source keys; that needs a live-query production witness. The index owner checks lower-bound presence on bounded numeric histories after each add, update, delete, and build: an upper-only range ignores an unused `fromInclusive: false`, while an explicit exclusive `from` excludes its boundary. A source Map supplies expected sets for BasicIndex and BTreeIndex; wrong-output controls reject both boundary mistakes. It also checks exact `equalityLookup` and `lookup('eq')` after singleton case-insensitive `A → a → A` replacements and populated comparator-group updates, plus fresh-object same-group replacements in fixed and generated histories. These direct-index checks do not cover every JavaScript value, arbitrary comparators, or long Collection histories. The index owner enforces one invalid-comparator law across both BasicIndex and BTreeIndex: for each comparator, accepted numeric prefix (including the empty prefix), and probe operation (add, update, eq/range lookup, take, build), the operation throws exactly when it receives a `NaN` or non-number result and never when every comparison is valid. A `signed infinity` comparator is the valid control, and a custom collation `compare` supplied through `compareOptions` is checked the same way. Two BTree Map split histories check inserted-key reads, exact whole-range payloads/order, size, and extrema, proving split placement follows the insertion index without a post-mutation comparison. A rejected index add or remove leaves the index refining its accepted rows; update and build are not atomic. At the Collection boundary, both index types and both write paths (optimistic insert, sync commit) crash the collection: status becomes `error` and the next mutation throws, so a usable collection never holds rows its subscribers were not told about. Open: a sync source can still commit writes on a collection in `error` state, which stores unpublished rows; this is existing `markError` behavior and needs a lifecycle-owner witness. Comparator transitivity is outside this evidence. | -| WHERE predicate publication | [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts), [`in` evaluator](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts) ([review](oracle-reviews/perf-in-constant-set.md)) | An independent Kleene evaluator judges `eq`/`not`/`and`/`or` predicates over strings, a normalization-prefixed string, booleans, numbers, `NaN`, a valid Date, `null`, a missing field, and virtual fields. Snapshot histories with pending optimistic inserts compare a live query, direct subscribers with and without initial state, and `currentStateAsChanges`; change histories apply multi-key sync transactions, including reinsertion of a key deleted earlier, and compare every consumer's key set after each commit, on scan and `BasicIndex` paths. Peer subscribers share one field with different literals, plus one on a virtual field, so every published change must reach exactly the subscribers whose predicate it can satisfy; routing mutants that ignored the previous value, ignored the field, delivered a change twice, or routed while stale published rows awaited reconciliation fail here. Fixed witnesses cover a layout-only publication reaching a filtered subscriber as one empty batch, a filtered subscriber's empty Collection-readiness batch, retraction of a vanished row after eager cleanup and restart, and the new row plus exact subscriber update payload when a same-key row stays TRUE. A stale-live-value mutant survives the generated membership checks but fails the payload witness. A focused [property-visibility test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-prefilter-property-visibility.test.ts) compares the public unindexed snapshot with the enriched-row contract for inherited, non-enumerable, enumerable-own, and nested getter paths; the pre-fix stored-row shortcut failed three of its four controls. Other property-descriptor and stateful-getter histories remain outside those fixed controls. Hostile mutants for FALSE-for-UNKNOWN `eq`, `or` or number-literal subscription prefilters, a prefilter that ignores the previous value, a skipped readiness batch, a skip while stale published rows await reconciliation, and an unindexed snapshot scan that reads only synced rows while an optimistic insert or delete changes visibility passed the prior `@tanstack/db` suite and fail here. Change routing withholds a dropped row from an `eq` subscription before its sent key could be recorded, so a mutant that records dropped rows as sent is equivalent for routed predicates in this oracle. A focused `collection-subscription.test.ts` witness checks that a dropped insert or update, alone or beside a matching change, cannot advance a limited subscription's page offset under a routed `eq` and an unrouted `or` predicate; the unrouted cases beside a matching change kill that mutant, and the unrouted `alone` case does not. A loose-equality prefilter is an equivalent mutant: it only skips less. Skipping during truncate replay also survives; the replay's baseline diff re-derives the same retraction, so the guard keeps the prior dataflow without its own witness. Generated cleanup and restart histories for filtered subscribers remain open for the lifecycle publication owner. No-op updates, other comparison operators, Temporal and binary operands, joins, ordering, optimistic updates, and truncate are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. Change routing was later removed in favor of pooled live queries; the routing mutants above are historical, and the same peer laws now check plain subscription dispatch. The unindexed snapshot prefilter was restored for compiled queries as one stored-row `eq` test on a string or boolean literal. Its property-visibility test also covers the reverse direction: a stored row with an inherited or non-enumerable field must still match `isUndefined` on that field, which a scan that evaluated the full predicate on stored rows fails. The `in` evaluator owner checks `in` against an independent linear reference over Dates, `NaN`, `-0`, BigInt, bytes, Temporal values, objects and nullish items, for constant and row-supplied lists. Its one-polyfill Temporal model uses captured intrinsic methods to reject forged and borrowed brands, accepts genuine own display shadows, and models Duration by ten fields, including PT1H versus PT60M. Arbitrary custom prototype forgery, mixed Temporal implementations, WHERE publication, and indexed mixed ordinary-object/Temporal range order remain outside that owner. | +| WHERE predicate publication | [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts), [`in` evaluator](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts) ([review](oracle-reviews/perf-in-constant-set.md)) | An independent Kleene evaluator judges `eq`/`not`/`and`/`or` predicates over strings, a normalization-prefixed string, booleans, numbers, `NaN`, a valid Date, `null`, a missing field, and virtual fields. Snapshot histories with pending optimistic inserts compare a live query, direct subscribers with and without initial state, and `currentStateAsChanges`; change histories apply multi-key sync transactions, including reinsertion of a key deleted earlier, and compare every consumer's key set after each commit, on scan and `BasicIndex` paths. Peer subscribers share one field with different literals, plus one on a virtual field, so every published change must reach exactly the subscribers whose predicate it can satisfy; routing mutants that ignored the previous value, ignored the field, delivered a change twice, or routed while stale published rows awaited reconciliation fail here. Fixed witnesses cover a layout-only publication reaching a filtered subscriber as one empty batch, a filtered subscriber's empty Collection-readiness batch, retraction of a vanished row after eager cleanup and restart, and the new row plus exact subscriber update payload when a same-key row stays TRUE. A stale-live-value mutant survives the generated membership checks but fails the payload witness. A focused [property-visibility test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-prefilter-property-visibility.test.ts) compares the public unindexed snapshot with the enriched-row contract for inherited, non-enumerable, enumerable-own, and nested getter paths; the pre-fix stored-row shortcut failed three of its four controls. Other property-descriptor and stateful-getter histories remain outside those fixed controls. Hostile mutants for FALSE-for-UNKNOWN `eq`, `or` or number-literal subscription prefilters, a prefilter that ignores the previous value, a skipped readiness batch, a skip while stale published rows await reconciliation, and an unindexed snapshot scan that reads only synced rows while an optimistic insert or delete changes visibility passed the prior `@tanstack/db` suite and fail here. Change routing withholds a dropped row from an `eq` subscription before its sent key could be recorded, so a mutant that records dropped rows as sent is equivalent for routed predicates in this oracle. A focused `collection-subscription.test.ts` witness checks that a dropped insert or update, alone or beside a matching change, cannot advance a limited subscription's page offset under a routed `eq` and an unrouted `or` predicate; the unrouted cases beside a matching change kill that mutant, and the unrouted `alone` case does not. A loose-equality prefilter is an equivalent mutant: it only skips less. Skipping during truncate replay also survives; the replay's baseline diff re-derives the same retraction, so the guard keeps the prior dataflow without its own witness. Generated cleanup and restart histories for filtered subscribers remain open for the lifecycle publication owner. No-op updates, other comparison operators, Temporal and binary operands, joins, ordering, optimistic updates, and truncate are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. Change routing was later removed in favor of pooled live queries; the routing mutants above are historical, and the same peer laws now check plain subscription dispatch. The unindexed snapshot prefilter was restored for compiled queries as one stored-row `eq` test on a string or boolean literal. Its property-visibility test also covers the reverse direction: a stored row with an inherited or non-enumerable field must still match `isUndefined` on that field, which a scan that evaluated the full predicate on stored rows fails. The `in` evaluator owner checks `in` against an independent linear reference over Dates, `NaN`, `-0`, BigInt, bytes, Temporal values, objects and nullish items, for constant and row-supplied lists. Its one-polyfill Temporal model uses captured intrinsic methods for native equality and models Duration by ten fields, including PT1H versus PT60M. Malformed Temporal lookalikes and deliberately shadowed Temporal members are outside that owner, along with mixed Temporal implementations, WHERE publication, and indexed mixed ordinary-object/Temporal range order. | | Captured `inArray` query stages | [stage oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/captured-inarray-stages-oracle.test.ts) ([review](oracle-reviews/captured-inarray-publication.md)) | Twelve fixed public cuts include nine numeric stage and acquisition witnesses after one unrelated source transaction, a limited ORDER uniqueness counterexample, grouped SUM/min/max arguments in an untouched group, and an aggregate argument wrapped by a selected addition. The controlled on-demand cut checks an unfiltered, unordered, unlimited first request. The nine initial stage cuts, ORDER duplicate, and wrapped aggregate were RED→GREEN in the working tree; grouped SUM/min/max passes in workspace Vitest. The final DB suite passes 12,406 tests across 282 files; DB-IVM passes 693 across 45. Arbitrary values and histories, real providers, and exact-head review remain open. | | Joined result keys | [Joined result key oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/join-result-key-oracle.property.test.ts) | A nested-loop model recomputes the pair set of an inner, left, or full join over source keys drawn from plain, comma-bearing, bracket-bearing, and quoted strings, numbers beside the strings that print the same, both infinities, and `NaN`. After preload and each synced group change, the published rows must equal the model's pairs and the result key count must equal the row count. The comma-joined key encoding fails its two pinned histories and both campaigns; plain `JSON.stringify`, which prints infinities as the missing-side `null`, fails the pinned infinity history and both campaigns. A pinned history in which a `NaN`-keyed pair leaves and re-forms fails when the join index compares source-key prefixes with `===`. Joins over subqueries, more than two sources, custom `getKey`, and optimistic mutations are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. | | D2 Index storage | [Index refinement oracle](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/index-refinement-oracle.property.test.ts) | A plain-`Map` model sums multiplicities under a type-and-value identity in which `NaN` equals `NaN` and `-0` equals `0`. Histories of up to twelve additions to two keys mix prefixed arrays with `NaN`, `0`, `-0`, `1`, `'1'`, `1n`, and `'a'` prefixes and unprefixed values, with cancellation and reappearance. After each addition, `get` and `has` must equal the model. Comparing prefixes with `===`, which disagrees with the `Map` that holds them, fails three pinned histories and both campaigns; the `-0` control passes under both. Every history also runs with `prefixIdentity`, where values under one key with the same prefix are one value; mutants that merge unprefixed values or ignore the option in a prefix map fail. The [structural identity oracle](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/structural-identity-oracle.property.test.ts) checks that `structuralIdentity` agrees with D2 value identity for equal representations (array subclasses, class instances, null prototypes, key order, non-enumerable symbols) and independent values, including separator-bearing strings and keys, enumerable symbol keys, Dates, RegExps, Maps, Sets, binary values, holes, deep nesting, and a cyclic value; an independent normal-form model decides equality and is checked against `equalHashValues`. Mutants that drop a string's or key's length, an array's length, or the Date and RegExp headers fail it; lowering the depth cap is equivalent, because the cap only bounds work. groupBy's use of it is owned by the incrementalization law: its string-extrema lane fails a string identity without a length prefix, and a focused groupBy witness fails when the group key is left out of the identity. Compaction, presence tracking, and structural payloads are outside this owner; the join operator tests and incrementalization law own operator behavior. | @@ -510,12 +510,51 @@ previous exact membership. A late index refresh or transient dual membership fails at the publication cut. This rejects the getter failure that changed the row while leaving its index and subscriber behind. It does not assert content equality for DataViews. +The deep-equality owner also models ordinary native Date equality by supplied +time across local and `node:vm` realms. A fixed matrix crosses both operand +realms and equal or changed valid and invalid times. Fixed-seed and random +campaigns sample finite times from -1e12 to 1e12, invalid time, equal or +adjacent changed time, and both realm placements. Direct results in both +operand orders reject a local-only Date classifier. One fixed equal-label +foreign Date pair rejects an enumerable-key answer that ignores changed time; +another keeps a labeled foreign Date distinct from an equal-keyed plain record. +These fixed controls do not specify a general policy for user-added Date +properties. The Collection change-event owner receives the ordinary Date rule +through six one-row sync updates, including invalid-to-valid and +valid-to-invalid transitions. A seventh fixed equal-label update preserves the +original missed-event counterexample. Each checks exact subscriber batches, +callback-time public time, and the change-message mirror against public time +after commit. These witnesses cover one `node:vm` host, unindexed sync +publication, and native Dates with unchanged prototypes and methods. +Optimistic publication, indexed Date updates, other hosts, proxies, user-added +properties beyond the fixed controls, and replaced Date prototypes or methods +remain open. +The same deep-equality owner models ordinary Temporal.Duration equality by an +independent ten-field vector. Fixed copies and one-field changes reconstruct +each field under both signs. Fixed opposite-sign seconds reject absolute-field +comparison. Eight fixed-factor unit carries from years/months through +microseconds/nanoseconds under both signs reject pairwise balancing, including +the `PT1H` versus `PT60M` distinction promised by +`packages/db/CHANGELOG.md`. Fixed-seed and seedless campaigns vary field +magnitudes from 0 to 3 with one shared positive or negative sign, one selected +field change, and three equal-display subsecond representations with signed +multipliers of magnitude 1 to 100. A display-only comparator fails the fixed +wrong-design control, and the original implementation fails +the equal-display property. The Collection change-event owner checks ready, +unindexed sync replacements in both directions between one second and one +thousand milliseconds under both signs, plus fresh equal-field replacements +for both signs. A one-hour to 60-minutes replacement reaches the same public +checkpoint, with different default displays but equivalent elapsed length. +Exact subscriber batches, callback-time public rows, and the mirror must agree +at commit. +Mixed-sign Duration input, optimistic writes, indexed publication, other +Temporal kinds, and arbitrary ten-field Collection histories remain open. Exact `normalizeValue` results are direct utility observations. Indexed order, public Date range predicates, cursor continuation, proxies, deeper subclass chains, replaced Date prototypes or `getTime` methods, and other Date-consuming paths remain outside that witness. -The comparison oracle also owns a bounded [Temporal order transfer](oracle-reviews/comparison-unit-retirement-round-6.md). Supplied ISO calendar fields and epoch-nanosecond offsets predict direct `ascComparator` signs and exact zeros for ordinary `PlainDate` and `ZonedDateTime` values from one polyfill copy. Fixed cases preserve the two retired unit blocks; generated cases vary calendar fields, offset zones, equal instants, and adjacent nanoseconds. Fresh-module histories reject a first-input constructor cache poisoned by a genuine instance or forged tag, including the direct `compareValues` path. Both direct helpers also receive a right-hand PlainDate with a constructor shadow. A genuine ZonedDateTime with an own shadowed display tag retains its instant position through both direct helpers. One genuine tagged ZonedDateTime subclass with inherited Temporal methods compares equal in instant position to a +05:30 peer through both direct helpers. The alias equality owner below checks native `.equals()` against a UTC peer and the borrowed-prototype control. This direct comparison lane does not establish public query or index behavior. Other Temporal subclasses, spoofed or replaced instance prototypes, mixed native/polyfill implementations, other Temporal kinds, and public ordering/filter publication need separate witnesses. Spoofed Temporal prototypes can still cause asymmetric comparisons; the comparison owner needs a dedicated brand and mixed-value witness before that class can be called closed. +The comparison oracle also owns a bounded [Temporal order transfer](oracle-reviews/comparison-unit-retirement-round-6.md). Supplied ISO calendar fields and epoch-nanosecond offsets predict direct `ascComparator` signs and exact zeros for ordinary `PlainDate` and `ZonedDateTime` values from one polyfill copy. Fixed cases preserve the two retired unit blocks; generated cases vary calendar fields, offset zones, equal instants, and adjacent nanoseconds. A normal ZonedDateTime subclass with inherited Temporal methods compares equal in instant position to a +05:30 peer through both direct helpers. This direct comparison lane does not establish public query or index behavior. Other Temporal subclasses, malformed Temporal lookalikes, deliberately shadowed members, mixed native/polyfill implementations, other Temporal kinds, and public ordering/filter publication need separate witnesses if those behaviors become product guarantees. The [Temporal index order oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/temporal-index-order-oracle.test.ts) and its [review record](oracle-reviews/temporal-index-order.md) own direct @@ -533,35 +572,35 @@ An opposite callback puts the UTC peer first even though display-text and source-key order put the alias first. Same-equality-class updates test both transitions between a callback split and a source-key tie, including ranges and cursors after each transition. -An add/build witness checks that a genuine value's own shadowed display tag -cannot move its default ordered position or the value cursor. -Direct removal supplies the exact prior row. Mixed value domains, other -Temporal kinds, public WHERE ranges, provider cursors, longer histories, and -live-query publication require separate witnesses. +The valid-value admission lane checks default BasicIndex/BTreeIndex add and +build for Duration with calendar units, PlainMonthDay with distinct reference +years, and mixed genuine Temporal kinds. It compares exact membership and +complete key sets across forward, reverse, and build histories without +assigning those values a public sort order. A source Collection commits two +rows separately and checks its stored row and exact index membership inside +each change callback, then checks ready status. This rejects a second valid +row that leaves the Collection stored but unpublished after an index ordering +error. A public limit-one `orderBy` then rejects each pair through the strict +Temporal comparison, so the private index order cannot replace the public +one. Changelog #1519 requires public `orderBy` to reject PlainMonthDay and +mixed Temporal kinds. Arbitrary kind mixtures, public ordered reads beyond +these rejection witnesses, provider paths, and longer publication histories +remain outside this lane. +Direct removal supplies the exact prior row. Mixed Temporal/ordinary value +domains, other Temporal kinds beyond this bounded admission lane, public WHERE ranges, +provider cursors, longer histories, and live-query publication require +separate witnesses. The [Temporal alias equality oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/temporal-equality-alias-oracle.test.ts) and its [review record](oracle-reviews/temporal-equality-alias.md) own the bounded `ZonedDateTime.equals()` partition for UTC/Etc/UTC aliases, a distinct -fixed-offset zone, adjacent nanoseconds, and a different calendar. One native +fixed-offset zone, adjacent nanoseconds, and a different calendar. One normal ZonedDateTime subclass with inherited Temporal methods belongs to the UTC -class. Another genuine subclass shadows its prototype's display tag with an -unrecognized name, and a third uses the recognized but misleading -`Temporal.PlainDate` tag. Native slots keep both in the UTC class; ordinary -objects borrowing these subclass prototypes retain reference identity. +class. Direct BasicIndex/BTreeIndex add, build, and removal checks compare independent fixture-assigned classes. Unindexed and optimized-only indexed Collection snapshots check public `eq`; the `in` evaluator oracle checks -constant and row-supplied alias lists. A plain object with a forged Temporal tag and a -genuine value with a shadowed zone accessor must stay outside the UTC class. -Forged and borrowed ordinary objects also retain reference self-identity in -direct lookup and unindexed public `eq`; distinct structural copies do not -match them. -The neighboring PlainDate lane checks forged tags, borrowed prototypes, and -genuine own-tag or display shadows at direct lookup and unindexed public `eq`. -The ZonedDateTime fixture also contains genuine UTC rows with own display-tag, -`toString`, `epochNanoseconds`, and `calendarId` shadows; direct indexes and -both public `eq` snapshot paths retain their alias equality. The `in` -evaluator owner distinguishes +constant and row-supplied alias lists. The `in` evaluator owner distinguishes PlainMonthDay reference years, PlainYearMonth reference days, and Duration field vectors that share default display text. Fixed controls reach compiled `eq`, constant-list `in`, and row-supplied `in` in both operand directions. @@ -611,15 +650,13 @@ value tested against a null-only list as FALSE, while the live-query guide's broad SQL/PostgreSQL comparison statement suggests UNKNOWN. The compiled `in` owner tracks this contract-authority question; a product decision is needed before changing its expected result or claiming SQL-style null-list behavior. -Other zone aliases, custom spoofed prototypes, mixed Temporal implementations, +Other zone aliases, malformed Temporal lookalikes, deliberately shadowed +Temporal members, mixed Temporal implementations, mixed ordinary-object/Temporal ordered indexes, joins, grouping, query identity, live-query publication, and provider paths remain unproved. Direct D2 `groupBy` has its separately approved kind-and-string identity policy. -The alias owner records a concrete custom-prototype fake that currently -matches a genuine PlainDate and a distinct fake copy in direct indexes and -public `eq`. Its stated grammar excludes that boundary. A future contract -decision about non-branded object identity and a trusted Temporal intrinsic -witness are needed before it can be claimed closed. +The earlier spoofed-prototype finding is outside the valid Temporal input +domain. It needs a separate product decision before it becomes a guarantee. The [custom index order selection oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/custom-index-order-selection-oracle.test.ts) and its [review record](oracle-reviews/custom-index-order-selection.md) check @@ -849,9 +886,20 @@ Fixed build histories use both `undefined`/`1` insertion orders and a larger ownership. The [BasicIndex build bug record](oracle-reviews/basic-index-nullish-build.md) contains the original `takeFromStart(1)` failure and repair evidence. Skipped-update, wrong equality, lost-owner, dropped-operand, wrong-dispatch, -wrong-cursor, and wrong-range controls fail the check. This direct-index -evidence does not establish mixed-type ordering, custom comparator behavior, -all forward bound combinations, reverse nullish ranges, filtered takes, +wrong-cursor, and wrong-range controls fail the check. A genuine-Date history +also removes a row with an equal but distinct Date object, re-adds its original +object, retains a same-timestamp peer, and moves a row by update. After every +direct action a source Map of supplied epochs predicts exact membership, +ordered takes, and lower-bound sets. A separate callback assigns two order +positions to Dates with the same timestamp; source labels predict their order +and bounded ranges through removal, re-addition, and update. A fresh old item +can share the timestamp while its callback position differs from the stored +row's position; both index types left a stale ordered row in that history. +The stale BTree operand cache also failed the same-object re-add take before +repair. These fixed histories do not cover cross-realm Dates, arbitrary object +equivalence, or Collection publication. This direct-index evidence does not +establish mixed-type ordering, arbitrary custom comparator behavior, all +forward bound combinations, reverse nullish ranges, filtered takes, private ordered-entry representation, live-query behavior, bucket identity, or instrumentation. The [nullish transfer ledger](oracle-reviews/index-unit-retirement-round-2.md) records the first five retired legacy blocks; the [IN transfer ledger](oracle-reviews/index-unit-retirement-round-3.md) diff --git a/packages/db/src/indexes/base-index.ts b/packages/db/src/indexes/base-index.ts index a38b70ad1e..967dae74e9 100644 --- a/packages/db/src/indexes/base-index.ts +++ b/packages/db/src/indexes/base-index.ts @@ -5,8 +5,9 @@ import { dateTimestamp, indexCustomComparatorValue, indexOrderValue, + indexTemporalKind, makeCheckedComparator, - makeComparator, + makeIndexComparator, } from '../utils/comparison.js' import { codedMessage, codedWarning, devBuild } from '../error-message.js' import type { CompiledSingleRowExpression } from '../query/compiler/evaluators.js' @@ -158,7 +159,7 @@ export abstract class BaseIndex< const compareFn = options?.compareFn this.compareFn = makeCheckedComparator( compareFn == null - ? makeComparator(this.compareOptions) + ? makeIndexComparator(this.compareOptions) : (a, b) => compareFn( indexCustomComparatorValue(a), @@ -219,7 +220,21 @@ export abstract class BaseIndex< } get supportsRangeOptimization(): boolean { - return !this.hasCustomComparator + if (this.hasCustomComparator) return false + // The default index can retain these genuine values for exact lookup, + // but its internal total order does not stand in for public `orderBy`. + let temporalKind: string | undefined + for (const domain of this.rangeValueDomains.keys()) { + if (!domain.startsWith(`Temporal.`)) continue + if ( + domain === `Temporal.Duration` || + domain === `Temporal.PlainMonthDay` || + (temporalKind !== undefined && temporalKind !== domain) + ) + return false + temporalKind = domain + } + return true } /** Ordering sees raw values; exact lookup keeps its normalized key. */ @@ -341,7 +356,7 @@ export abstract class BaseIndex< function rangeValueDomain(value: unknown): string | undefined { if (value == null) return undefined if (dateTimestamp(value) !== undefined) return `date` - return typeof value + return indexTemporalKind(value) ?? typeof value } function isNativeRangeDomain(domain: string): boolean { diff --git a/packages/db/src/indexes/basic-index.ts b/packages/db/src/indexes/basic-index.ts index f2ff90b2e4..7182ad698e 100644 --- a/packages/db/src/indexes/basic-index.ts +++ b/packages/db/src/indexes/basic-index.ts @@ -60,8 +60,8 @@ export class BasicIndex< private sortedValues: Array = [] // Each bucket stays at the same array position as its order representative. private orderedBuckets: Array> = [] - // Set of all indexed PKs - private indexedKeys = new Set() + // Each indexed key retains the order value used for its current position. + private indexedKeys = new Map() protected initialize(_options?: BasicIndexOptions): void {} @@ -71,15 +71,12 @@ export class BasicIndex< add(key: TKey, item: any): void { const indexedValue = this.evaluateAddedValue(key, item) const normalizedValue = normalizeValue(indexedValue) + const orderValue = this.orderValue(indexedValue, normalizedValue) - this.addToBucket( - key, - normalizedValue, - this.orderValue(indexedValue, normalizedValue), - ) + this.addToBucket(key, normalizedValue, orderValue) this.addRangeValue(indexedValue) - this.indexedKeys.add(key) + this.indexedKeys.set(key, orderValue) } private addToBucket( @@ -128,11 +125,7 @@ export class BasicIndex< const normalizedValue = normalizeValue(indexedValue) - this.removeFromBucket( - key, - normalizedValue, - this.orderValue(indexedValue, normalizedValue), - ) + this.removeFromBucket(key, normalizedValue, this.indexedKeys.get(key)) this.removeRangeValue(indexedValue) this.indexedKeys.delete(key) @@ -198,7 +191,7 @@ export class BasicIndex< const oldValue = normalizeValue(oldIndexedValue) const newValue = normalizeValue(newIndexedValue) - const oldOrder = this.orderValue(oldIndexedValue, oldValue) + const oldOrder = this.indexedKeys.get(key) const newOrder = this.orderValue(newIndexedValue, newValue) if ( areSameValueZeroEqual(oldValue, newValue) && @@ -211,6 +204,7 @@ export class BasicIndex< } this.removeRangeValue(oldIndexedValue) this.addRangeValue(newIndexedValue) + this.indexedKeys.set(key, newOrder) return } @@ -218,7 +212,7 @@ export class BasicIndex< this.removeRangeValue(oldIndexedValue) this.addToBucket(key, newValue, newOrder) this.addRangeValue(newIndexedValue) - this.indexedKeys.add(key) + this.indexedKeys.set(key, newOrder) } /** @@ -232,13 +226,14 @@ export class BasicIndex< for (const [key, item] of entries) { const indexedValue = this.evaluateAddedValue(key, item) const value = normalizeValue(indexedValue) + const orderValue = this.orderValue(indexedValue, value) entriesArray.push({ key, value, - orderValue: this.orderValue(indexedValue, value), + orderValue, }) this.addRangeValue(indexedValue) - this.indexedKeys.add(key) + this.indexedKeys.set(key, orderValue) } // Native Array.sort moves raw undefined values last without comparing them. diff --git a/packages/db/src/indexes/btree-index.ts b/packages/db/src/indexes/btree-index.ts index d77de16de4..33cffb5ceb 100644 --- a/packages/db/src/indexes/btree-index.ts +++ b/packages/db/src/indexes/btree-index.ts @@ -73,7 +73,7 @@ export class BTreeIndex< unknown, { bucket: OrderedBucket; owners: number } >() - private indexedKeys = new Set() + private indexedKeys = new Map() // Stored keys use UNDEFINED_SENTINEL; the comparator, which may be // user-supplied, expects undefined. @@ -98,15 +98,12 @@ export class BTreeIndex< // Normalize the value for Map key usage const normalizedValue = normalizeForBTree(indexedValue) + const orderValue = this.orderValue(indexedValue, normalizedValue) - this.addToBucket( - key, - normalizedValue, - this.orderValue(indexedValue, normalizedValue), - ) + this.addToBucket(key, normalizedValue, orderValue) this.addRangeValue(indexedValue) - this.indexedKeys.add(key) + this.indexedKeys.set(key, orderValue) } private addToBucket( @@ -157,11 +154,7 @@ export class BTreeIndex< // Normalize the value for Map key usage const normalizedValue = normalizeForBTree(indexedValue) - this.removeFromBucket( - key, - normalizedValue, - this.orderValue(indexedValue, normalizedValue), - ) + this.removeFromBucket(key, normalizedValue, this.indexedKeys.get(key)) this.removeRangeValue(indexedValue) this.indexedKeys.delete(key) @@ -225,7 +218,7 @@ export class BTreeIndex< const oldValue = normalizeForBTree(oldIndexedValue) const newValue = normalizeForBTree(newIndexedValue) - const oldOrder = this.orderValue(oldIndexedValue, oldValue) + const oldOrder = this.indexedKeys.get(key) const newOrder = this.orderValue(newIndexedValue, newValue) if ( areSameValueZeroEqual(oldValue, newValue) && @@ -237,6 +230,7 @@ export class BTreeIndex< } this.removeRangeValue(oldIndexedValue) this.addRangeValue(newIndexedValue) + this.indexedKeys.set(key, newOrder) return } @@ -244,7 +238,7 @@ export class BTreeIndex< this.removeRangeValue(oldIndexedValue) this.addToBucket(key, newValue, newOrder) this.addRangeValue(newIndexedValue) - this.indexedKeys.add(key) + this.indexedKeys.set(key, newOrder) } /** @@ -349,8 +343,7 @@ export class BTreeIndex< // Only exclude the boundary when an exclusive lower bound was // actually provided. Without a `from` bound, `fromKey` defaults to // the minimum key and must not be dropped. Compare against the - // normalized key since indexed values are stored normalized - // (e.g. dates as timestamps), so the raw `from` would never match. + // stored order value; exact lookup uses a separate normalized key. if ( hasFrom && !fromInclusive && @@ -372,7 +365,7 @@ export class BTreeIndex< * Internal method for taking items from the index. * @param n - The number of items to return * @param nextPair - Function to get the next pair from the BTree - * @param from - Already normalized! undefined means "start from beginning/end", sentinel means "start from the key undefined" + * @param from - Stored order value; undefined means "start from beginning/end", sentinel means "start from the key undefined" * @param filterFn - Optional filter function * @param reversed - Whether to reverse the order of keys within each value */ @@ -385,7 +378,7 @@ export class BTreeIndex< ): Array { const result: Array = [] let pair: [any, OrderedBucket] | undefined - let key = from // Use as-is - it's already normalized by the caller + let key = from // Every key owns exactly one bucket, so the walk never repeats a key. while ((pair = nextPair(key)) !== undefined && result.length < n) { @@ -411,9 +404,8 @@ export class BTreeIndex< */ take(n: number, from: any, filterFn?: (key: TKey) => boolean): Array { const nextPair = (k?: any) => this.orderedEntries.nextHigherPair(k) - // Normalize the from value - const normalizedFrom = this.orderValue(from, normalizeForBTree(from)) - return this.takeInternal(n, nextPair, normalizedFrom, filterFn) + const orderFrom = this.orderValue(from, normalizeForBTree(from)) + return this.takeInternal(n, nextPair, orderFrom, filterFn) } /** @@ -440,9 +432,8 @@ export class BTreeIndex< filterFn?: (key: TKey) => boolean, ): Array { const nextPair = (k?: any) => this.orderedEntries.nextLowerPair(k) - // Normalize the from value - const normalizedFrom = this.orderValue(from, normalizeForBTree(from)) - return this.takeInternal(n, nextPair, normalizedFrom, filterFn, true) + const orderFrom = this.orderValue(from, normalizeForBTree(from)) + return this.takeInternal(n, nextPair, orderFrom, filterFn, true) } /** diff --git a/packages/db/src/utils.ts b/packages/db/src/utils.ts index ac92fe4f78..721bbc06bb 100644 --- a/packages/db/src/utils.ts +++ b/packages/db/src/utils.ts @@ -29,6 +29,51 @@ export function nativeTypedArrayLength(value: ArrayBufferView): number { return Reflect.apply(typedArrayLength, value, []) as number } +// Duration equality retains the exact units, including subsecond units that +// share a default display. Query normalization uses the same field order. +export const durationFields = [ + `years`, + `months`, + `weeks`, + `days`, + `hours`, + `minutes`, + `seconds`, + `milliseconds`, + `microseconds`, + `nanoseconds`, +] as const + +// The Date intrinsic recognizes native Dates across realms. A display tag +// cannot hide a Date's timestamp or make an ordinary object into a Date. +export function dateTimestamp(value: unknown): number | undefined { + if (value === null || typeof value !== `object`) return undefined + try { + if (value instanceof Date) return Date.prototype.getTime.call(value) + if (Symbol.toStringTag in value) { + // A Date subclass can inherit getTime from its Date prototype. Walk + // descriptors without reading a user display-tag or getTime getter; + // the intrinsic below still checks for the native Date slot. + let prototype = Object.getPrototypeOf(value) + while (prototype !== null) { + const descriptor = Object.getOwnPropertyDescriptor(prototype, `getTime`) + if (descriptor) { + if (typeof descriptor.value !== `function`) return undefined + break + } + prototype = Object.getPrototypeOf(prototype) + } + if (prototype === null) return undefined + } else if (Object.prototype.toString.call(value) !== `[object Date]`) { + return undefined + } + return Date.prototype.getTime.call(value) + } catch { + // An ordinary object or proxy has no native Date timestamp. + return undefined + } +} + /** * Deep equality function that compares two values recursively * Handles primitives, objects, arrays, Date, RegExp, Map, Set, TypedArrays, and Temporal objects @@ -103,13 +148,13 @@ export function deepEqualsInternal( return false } - // Handle Date objects - if (a instanceof Date) { - if (!(b instanceof Date)) return false - return Object.is(a.getTime(), b.getTime()) + // Native Dates compare by time even when their constructors come from + // different realms. An invalid Date has a NaN time, distinct from no Date. + const aDateTime = dateTimestamp(a) + const bDateTime = dateTimestamp(b) + if (aDateTime !== undefined || bDateTime !== undefined) { + return Object.is(aDateTime, bDateTime) } - // Symmetric check: if b is Date but a is not, they're not equal - if (b instanceof Date) return false // Handle RegExp objects if (a instanceof RegExp) { @@ -278,13 +323,15 @@ export function deepEqualsInternal( // If they're different Temporal types, they're not equal if (aTag !== bTag) return false - // Use Temporal's built-in equals method if available - if (typeof a.equals === `function`) { - return a.equals(b) + if (aTag === `Temporal.Duration`) { + const left = a as typeof a & + Record<(typeof durationFields)[number], number> + const right = b as typeof b & + Record<(typeof durationFields)[number], number> + return durationFields.every((field) => left[field] === right[field]) } - // Fallback to toString comparison for other types - return a.toString() === b.toString() + return a.equals!(b) } // Symmetric check: if b is Temporal but a is not, they're not equal if (isTemporal(b)) return false @@ -382,7 +429,7 @@ export interface TemporalLike { equals?: (other: unknown) => boolean } -/** Checks if the value is a Temporal object by checking for the Temporal brand */ +/** Recognizes a supported Temporal tag; callers assume tagged values are valid. */ export function isTemporal(a: unknown): a is TemporalLike { if (a == null || typeof a !== `object`) return false const tag = (a as Record)[Symbol.toStringTag] diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index 7880ef75d0..fbe20320a0 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -1,4 +1,6 @@ import { + dateTimestamp, + durationFields, isTemporal, nativeTypedArrayLength, nativeTypedArrayTag, @@ -7,6 +9,8 @@ import { getRuntimeReferenceIdentity } from '../query/runtime-reference-identity import { codedMessage, devBuild } from '../error-message' import type { CompareOptions } from '../query/builder/types' +export { dateTimestamp } from '../utils' + // WeakMap to store stable IDs for objects const objectIds = new WeakMap() let nextObjectId = 1 @@ -23,37 +27,6 @@ function getObjectId(obj: object): number { return id } -// The Date intrinsic checks the native brand across realms. Cheap shape gates -// avoid that throwing probe for ordinary objects and Temporal values. A Date -// display tag cannot hide its timestamp, and a forged tag cannot supply one. -export function dateTimestamp(value: unknown): number | undefined { - if (value === null || typeof value !== `object`) return undefined - try { - if (value instanceof Date) return Date.prototype.getTime.call(value) - if (Symbol.toStringTag in value) { - // A Date subclass can inherit getTime from its Date prototype. Walk - // descriptors without reading a user display-tag or getTime getter; - // the intrinsic below still checks for the native Date slot. - let prototype = Object.getPrototypeOf(value) - while (prototype !== null) { - const descriptor = Object.getOwnPropertyDescriptor(prototype, `getTime`) - if (descriptor) { - if (typeof descriptor.value !== `function`) return undefined - break - } - prototype = Object.getPrototypeOf(prototype) - } - if (prototype === null) return undefined - } else if (Object.prototype.toString.call(value) !== `[object Date]`) { - return undefined - } - return Date.prototype.getTime.call(value) - } catch { - // An ordinary object or proxy has no native Date timestamp. - return undefined - } -} - function hasUnorderableValue( value: unknown, timestamp: number | undefined, @@ -133,10 +106,10 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { } // If both are Temporal objects, use compareTemporalValues for correct semantic ordering - const aTemporal = brandedTemporal(a) - const bTemporal = brandedTemporal(b) + const aTemporal = isTemporal(a) + const bTemporal = isTemporal(b) if (aTemporal && bTemporal) { - return compareBrandedTemporalValues(a, b, aTemporal, bTemporal) + return compareTemporalValues(a, b) } // Native byte views use their unsigned contents for default order. This @@ -239,6 +212,45 @@ export function makeComparator( } } +/** + * Index storage needs a total order even for Temporal values that have no + * public `orderBy` relation. Keep kinds in separate groups, and consistently + * order every Duration and PlainMonthDay by its exact equality key. Public + * comparisons still use `makeComparator` and retain their strict semantics. + */ +export function makeIndexComparator( + opts: CompareOptions, +): (a: any, b: any) => number { + const compare = makeComparator(opts) + return (a, b) => { + const aTemporal = isTemporal(a) + const bTemporal = isTemporal(b) + if (!aTemporal || !bTemporal) return compare(a, b) + + const aKind = a[Symbol.toStringTag] + const bKind = b[Symbol.toStringTag] + let order: number + if (aKind !== bKind) { + order = aKind < bKind ? -1 : 1 + } else if ( + aKind === `Temporal.Duration` || + aKind === `Temporal.PlainMonthDay` + ) { + const aKey = normalizeValue(a) as string + const bKey = normalizeValue(b) as string + order = aKey < bKey ? -1 : aKey > bKey ? 1 : 0 + } else { + return compare(a, b) + } + return opts.direction === `desc` ? -order : order + } +} + +/** Identify a supported Temporal tag for index ordering. */ +export function indexTemporalKind(value: unknown): string | undefined { + return isTemporal(value) ? value[Symbol.toStringTag] : undefined +} + /** * Wraps a user comparator so an index crashes on a result that cannot order * its storage, instead of silently misplacing rows. @@ -317,54 +329,6 @@ function normalizeBinary(value: Uint8Array): string { return normalizedKey(`binary`, bytes) } -const durationFields = [ - `years`, - `months`, - `weeks`, - `days`, - `hours`, - `minutes`, - `seconds`, - `milliseconds`, - `microseconds`, - `nanoseconds`, -] as const - -/** Read a Temporal brand and display through its slot-checking prototype. */ -type BrandedTemporal = { prototype: object; kind: string; text: string } - -function brandedTemporal(value: unknown): BrandedTemporal | undefined { - if (value === null || typeof value !== `object`) return undefined - let prototype: object | null = Object.getPrototypeOf(value) - while (prototype) { - const kind = Object.getOwnPropertyDescriptor( - prototype, - Symbol.toStringTag, - )?.value - const toString = Object.getOwnPropertyDescriptor( - prototype, - `toString`, - )?.value - if ( - typeof kind === `string` && - isTemporal(prototype) && - typeof toString === `function` - ) { - try { - // A genuine subclass inherits this method from the Temporal prototype. - // Its receiver check rejects forged tags and borrowed prototypes. - // A custom prototype supplying both members remains outside this check. - const text = Reflect.apply(toString, value, []) - return typeof text === `string` ? { prototype, kind, text } : undefined - } catch { - return undefined - } - } - prototype = Object.getPrototypeOf(prototype) - } - return undefined -} - /** * Sentinel value representing undefined in normalized form. * This allows distinguishing between "start from beginning" (undefined parameter) @@ -394,89 +358,46 @@ export function normalizeValue(value: any): any { const timestamp = dateTimestamp(value) if (timestamp !== undefined) return timestamp - // Query equality comes from the branded value, never an own display shadow. - const temporal = brandedTemporal(value) - if (temporal) { - const { prototype, kind, text } = temporal + if (isTemporal(value)) { + const kind = value[Symbol.toStringTag] if (kind === `Temporal.ZonedDateTime`) { - // Read Temporal's internal slots through its prototype getters. Own - // properties can shadow the public accessors without changing equals(). - let epochNanoseconds: unknown - let timeZoneId: unknown - let calendarId: unknown - try { - epochNanoseconds = Object.getOwnPropertyDescriptor( - prototype, - `epochNanoseconds`, - )?.get?.call(value) - timeZoneId = Object.getOwnPropertyDescriptor( - prototype, - `timeZoneId`, - )?.get?.call(value) - calendarId = Object.getOwnPropertyDescriptor( - prototype, - `calendarId`, - )?.get?.call(value) - } catch { - // A borrowed prototype is not a branded Temporal instance. - return value - } - if ( - typeof epochNanoseconds === `bigint` && - typeof timeZoneId === `string` && - typeof calendarId === `string` - ) { - // Equality retains the instant, calendar, and canonical zone. A fixed - // +00:00 zone stays distinct from named UTC, while UTC aliases merge. - return normalizedKey( - `temporal`, - JSON.stringify([ - kind, - String(epochNanoseconds), - canonicalTemporalZone(timeZoneId), - calendarId, - ]), - ) + const zoned = value as typeof value & { + epochNanoseconds: bigint + timeZoneId: string + calendarId: string } - return value + // Equality retains the instant, calendar, and canonical zone. A fixed + // +00:00 zone stays distinct from named UTC, while UTC aliases merge. + return normalizedKey( + `temporal`, + JSON.stringify([ + kind, + String(zoned.epochNanoseconds), + canonicalTemporalZone(zoned.timeZoneId), + zoned.calendarId, + ]), + ) } if ( kind === `Temporal.PlainMonthDay` || kind === `Temporal.PlainYearMonth` ) { // Default display omits the reference ISO year/day that native equals() - // compares. The intrinsic string form with the calendar shown includes - // those fields and ignores an instance's own toString shadow. - const toString = Object.getOwnPropertyDescriptor( - prototype, - `toString`, - )?.value - try { - const fullText = Reflect.apply(toString, value, [ - { calendarName: `always` }, - ]) - return normalizedKey(`temporal`, `${kind}:${fullText}`) - } catch { - return value - } + // compares. The calendar-inclusive display preserves those fields. + const fullText = ( + value.toString as (options: { calendarName: `always` }) => string + )({ calendarName: `always` }) + return normalizedKey(`temporal`, `${kind}:${fullText}`) } if (kind === `Temporal.Duration`) { // Duration's default display balances subsecond units. Equality under - // the #1519 contract instead preserves all ten intrinsic fields. - try { - const fields = durationFields.map((field) => { - const get = Object.getOwnPropertyDescriptor(prototype, field)?.get - return Reflect.apply(get!, value, []) as number - }) - if (fields.every((field) => typeof field === `number`)) { - return normalizedKey(`temporal`, JSON.stringify([kind, ...fields])) - } - } catch { - // A borrowed prototype has no Temporal internal slots. - } - return value + // the #1519 contract instead preserves all ten fields. + const duration = value as typeof value & + Record<(typeof durationFields)[number], number> + const fields = durationFields.map((field) => duration[field]) + return normalizedKey(`temporal`, JSON.stringify([kind, ...fields])) } - return normalizedKey(`temporal`, `${kind}:${text}`) + return normalizedKey(`temporal`, `${kind}:${value.toString()}`) } // Normalize Uint8Arrays/Buffers to a string representation for Map key usage @@ -508,9 +429,11 @@ export function indexOrderValue(value: any, exactKey: any): any { /** Supply custom comparators the Temporal display key used before raw ordering. */ export function indexCustomComparatorValue(value: unknown): unknown { - const temporal = brandedTemporal(value) - return temporal - ? normalizedKey(`temporal`, `${temporal.kind}:${temporal.text}`) + return isTemporal(value) + ? normalizedKey( + `temporal`, + `${value[Symbol.toStringTag]}:${value.toString()}`, + ) : value } @@ -535,41 +458,24 @@ export function denormalizeUndefined(value: any): any { /** * Compare two Temporal values of the same type, returning -1, 0, or 1. * - * The tag checks type agreement; the prototype supplies the static compare - * function. An instance's own `constructor` can be shadowed, and a claimed tag - * must not install a comparator that later genuine values use. Types without - * a static `.compare` (e.g. `PlainMonthDay`) throw instead of using text order. + * The tags check type agreement. Types without a static `.compare` (e.g. + * `PlainMonthDay`) throw instead of using text order. * * Callers must ensure both arguments are Temporal objects; mixed types throw. */ export function compareTemporalValues(a: unknown, b: unknown): number { - return compareBrandedTemporalValues( - a, - b, - brandedTemporal(a), - brandedTemporal(b), - ) -} - -function compareBrandedTemporalValues( - a: unknown, - b: unknown, - aTemporal: BrandedTemporal | undefined, - bTemporal: BrandedTemporal | undefined, -): number { - const aTag = aTemporal?.kind ?? `unbranded` - const bTag = bTemporal?.kind ?? `unbranded` - if (!aTemporal || !bTemporal || aTag !== bTag) { + const aTag = (a as Record)[Symbol.toStringTag] + const bTag = (b as Record)[Symbol.toStringTag] + if (aTag !== bTag) { throw new TypeError( devBuild() && process.env.NODE_ENV !== `production` ? `Cannot order Temporal values of different types: ${aTag} vs ${bTag}` : codedMessage(167, { aTag, bTag }), ) } - const prototype = aTemporal.prototype as { - constructor?: { compare?: (x: unknown, y: unknown) => number } - } | null - const compare = prototype?.constructor?.compare + const compare = ( + a as { constructor: { compare?: (x: unknown, y: unknown) => number } } + ).constructor.compare if (typeof compare !== `function`) { throw new TypeError( devBuild() && process.env.NODE_ENV !== `production` @@ -593,10 +499,8 @@ function compareBrandedTemporalValues( * arguments are non-null. */ export function compareValues(a: unknown, b: unknown): number { - const aTemporal = brandedTemporal(a) - const bTemporal = brandedTemporal(b) - if (aTemporal && bTemporal) { - return compareBrandedTemporalValues(a, b, aTemporal, bTemporal) + if (isTemporal(a) && isTemporal(b)) { + return compareTemporalValues(a, b) } const comparableA = dateTimestamp(a) ?? a const comparableB = dateTimestamp(b) ?? b diff --git a/packages/db/tests/change-event-history-oracle.test.ts b/packages/db/tests/change-event-history-oracle.test.ts index 9288e025ba..38943d8cdd 100644 --- a/packages/db/tests/change-event-history-oracle.test.ts +++ b/packages/db/tests/change-event-history-oracle.test.ts @@ -1,5 +1,6 @@ import { runInNewContext } from 'node:vm' import { fc, test as fcTest } from '@fast-check/vitest' +import { Temporal } from 'temporal-polyfill' import { describe, expect, it, vi } from 'vitest' import { BTreeIndex, @@ -1113,3 +1114,344 @@ describe(`opaque DataView change publication`, () => { }, ) }) + +describe(`native Date change publication across realms`, () => { + /** + * A settled sync replacement must leave a change-message mirror equal to the + * Collection's public row, as the Collection change-message contract above + * states. The native Date timestamp rule in `docs/guides/live-queries.md` + * includes Dates from another JavaScript realm and invalid Dates. The model + * knows the two supplied times and predicts one update exactly when they + * differ. One fixed pair has the same own string label on both Dates; it + * preserves a missed-update counterexample without defining behavior for + * differently labeled Dates. The driver uses real sync writes and a public + * subscriber; it checks the callback and the settled commit. These fixed + * histories do not claim optimistic writes, indexes, or other hosts. + */ + it.each([ + { + name: `changed equally labeled Dates in one foreign realm`, + beforeTime: 1, + afterTime: 2, + beforeRealm: `foreign`, + afterRealm: `foreign`, + ownLabel: true, + }, + { + name: `equal Dates in one foreign realm`, + beforeTime: 1, + afterTime: 1, + beforeRealm: `foreign`, + afterRealm: `foreign`, + ownLabel: false, + }, + { + name: `equal local-to-foreign Dates`, + beforeTime: 1, + afterTime: 1, + beforeRealm: `local`, + afterRealm: `foreign`, + ownLabel: false, + }, + { + name: `equal foreign-to-local Dates`, + beforeTime: 1, + afterTime: 1, + beforeRealm: `foreign`, + afterRealm: `local`, + ownLabel: false, + }, + { + name: `equal invalid local-to-foreign Dates`, + beforeTime: null, + afterTime: null, + beforeRealm: `local`, + afterRealm: `foreign`, + ownLabel: false, + }, + { + name: `foreign invalid-to-valid Dates`, + beforeTime: null, + afterTime: 1, + beforeRealm: `foreign`, + afterRealm: `foreign`, + ownLabel: false, + }, + { + name: `foreign valid-to-invalid Dates`, + beforeTime: 1, + afterTime: null, + beforeRealm: `foreign`, + afterRealm: `foreign`, + ownLabel: false, + }, + ] as const)(`keeps the public mirror for $name`, async (sample) => { + type DateRow = { id: string; value: Date } + type DateSync = Parameters[`sync`]>[0] + const beforeTime = sample.beforeTime ?? Number.NaN + const afterTime = sample.afterTime ?? Number.NaN + const [foreignBefore, foreignAfter] = runInNewContext( + `[new Date(beforeTime), new Date(afterTime)]`, + { beforeTime, afterTime }, + ) as [Date, Date] + const before = + sample.beforeRealm === `foreign` ? foreignBefore : new Date(beforeTime) + const after = + sample.afterRealm === `foreign` ? foreignAfter : new Date(afterTime) + if (sample.ownLabel) { + for (const date of [before, after]) { + Object.defineProperty(date, `label`, { + value: `same`, + enumerable: true, + }) + } + } + // The native intrinsic verifies the host supplied real Dates. The model + // itself uses the constructor inputs, never the production classifier. + expect(Date.prototype.getTime.call(before)).toEqual(beforeTime) + expect(Date.prototype.getTime.call(after)).toEqual(afterTime) + + let sync!: DateSync + let unsubscribe: (() => unknown) | undefined + let cleanup: (() => unknown) | undefined + await withOracleCleanup(async () => { + const collection = createCollection({ + id: `native-date-publication-${sample.name}`, + getKey: ({ id }) => id, + autoIndex: `off`, + startSync: true, + sync: { + sync: (actions) => { + sync = actions + actions.begin() + actions.write({ type: `insert`, value: { id: `a`, value: before } }) + actions.commit() + actions.markReady() + }, + }, + }) + cleanup = () => collection.cleanup() + await collection.stateWhenReady() + + let mirrorTime = beforeTime + const batches: Array< + Array<{ + type: string + previousTime: number | undefined + time: number + }> + > = [] + const callbackTimes: Array = [] + const subscription = collection.subscribeChanges( + (changes) => { + batches.push( + changes.map((change) => ({ + type: change.type, + previousTime: + change.type === `update` + ? Date.prototype.getTime.call(change.previousValue!.value) + : undefined, + time: Date.prototype.getTime.call(change.value.value), + })), + ) + for (const change of changes) { + mirrorTime = Date.prototype.getTime.call(change.value.value) + } + callbackTimes.push( + Date.prototype.getTime.call(collection.get(`a`)!.value), + ) + }, + { includeInitialState: false }, + ) + unsubscribe = () => subscription.unsubscribe() + + sync.begin() + sync.write({ type: `update`, value: { id: `a`, value: after } }) + await sync.commit() + + const changed = !Object.is(beforeTime, afterTime) + expect(batches).toEqual( + changed + ? [[{ type: `update`, previousTime: beforeTime, time: afterTime }]] + : [], + ) + expect(callbackTimes).toEqual(changed ? [afterTime] : []) + expect(mirrorTime).toEqual(afterTime) + expect(Date.prototype.getTime.call(collection.get(`a`)!.value)).toEqual( + afterTime, + ) + }, [() => unsubscribe?.(), () => cleanup?.()]) + }) +}) + +describe(`Temporal.Duration change publication`, () => { + /** + * The Collection change-message contract above requires a mirror built from + * delivered messages to equal the public row after sync commit. The query + * equality contract already distinguishes all ten Duration fields, even + * when default display text agrees. The model supplies fresh ordinary + * Durations with either one second or one thousand milliseconds under a + * shared positive or negative sign, plus one hour versus 60 minutes. It + * predicts an update for each field change and no event for a fresh copy. + * The production driver writes through ready sync Collections; subscriber + * batches, callback-time public rows, the mirror, and settled public rows + * are the observations. These fixed histories do not cover optimistic + * writes, other Temporal kinds, or every possible ten-field transition. + */ + const fields = [ + `years`, + `months`, + `weeks`, + `days`, + `hours`, + `minutes`, + `seconds`, + `milliseconds`, + `microseconds`, + `nanoseconds`, + ] as const + const snapshot = (value: Temporal.Duration): Array => + fields.map((field) => value[field]) + + it.each([ + { + name: `seconds to milliseconds`, + before: { seconds: 1 }, + after: { milliseconds: 1000 }, + changed: true, + sameDisplay: true, + }, + { + name: `milliseconds to seconds`, + before: { milliseconds: 1000 }, + after: { seconds: 1 }, + changed: true, + sameDisplay: true, + }, + { + name: `fresh equal seconds`, + before: { seconds: 1 }, + after: { seconds: 1 }, + changed: false, + sameDisplay: true, + }, + { + name: `fresh equal milliseconds`, + before: { milliseconds: 1000 }, + after: { milliseconds: 1000 }, + changed: false, + sameDisplay: true, + }, + { + name: `negative seconds to milliseconds`, + before: { seconds: -1 }, + after: { milliseconds: -1000 }, + changed: true, + sameDisplay: true, + }, + { + name: `negative milliseconds to seconds`, + before: { milliseconds: -1000 }, + after: { seconds: -1 }, + changed: true, + sameDisplay: true, + }, + { + name: `fresh equal negative seconds`, + before: { seconds: -1 }, + after: { seconds: -1 }, + changed: false, + sameDisplay: true, + }, + { + name: `fresh equal negative milliseconds`, + before: { milliseconds: -1000 }, + after: { milliseconds: -1000 }, + changed: false, + sameDisplay: true, + }, + { + name: `hours to 60 minutes`, + before: { hours: 1 }, + after: { minutes: 60 }, + changed: true, + sameDisplay: false, + }, + ])(`keeps the public mirror for $name`, async (sample) => { + type DurationRow = { id: string; value: Temporal.Duration } + type DurationSync = Parameters[`sync`]>[0] + const before = Temporal.Duration.from(sample.before) + const after = Temporal.Duration.from(sample.after) + expect(before).not.toBe(after) + expect(before.toString() === after.toString()).toBe(sample.sameDisplay) + const beforeFields = snapshot(before) + const afterFields = snapshot(after) + expect( + beforeFields.every((field, index) => field === afterFields[index]), + ).toBe(!sample.changed) + + let sync!: DurationSync + let unsubscribe: (() => unknown) | undefined + let cleanup: (() => unknown) | undefined + await withOracleCleanup(async () => { + const collection = createCollection({ + id: `duration-publication-${sample.name}`, + getKey: ({ id }) => id, + autoIndex: `off`, + startSync: true, + sync: { + sync: (actions) => { + sync = actions + actions.begin() + actions.write({ type: `insert`, value: { id: `a`, value: before } }) + actions.commit() + actions.markReady() + }, + }, + }) + cleanup = () => collection.cleanup() + await collection.stateWhenReady() + + let mirror = beforeFields + const batches: Array< + Array<{ + type: string + previous: Array | undefined + value: Array + }> + > = [] + const callbackRows: Array> = [] + const subscription = collection.subscribeChanges( + (changes) => { + batches.push( + changes.map((change) => ({ + type: change.type, + previous: + change.type === `update` + ? snapshot(change.previousValue!.value) + : undefined, + value: snapshot(change.value.value), + })), + ) + for (const change of changes) mirror = snapshot(change.value.value) + callbackRows.push(snapshot(collection.get(`a`)!.value)) + }, + { includeInitialState: false }, + ) + unsubscribe = () => subscription.unsubscribe() + + sync.begin() + sync.write({ type: `update`, value: { id: `a`, value: after } }) + await sync.commit() + + expect(batches).toEqual( + sample.changed + ? [[{ type: `update`, previous: beforeFields, value: afterFields }]] + : [], + ) + expect(callbackRows).toEqual(sample.changed ? [afterFields] : []) + expect(mirror).toEqual(afterFields) + expect(snapshot(collection.get(`a`)!.value)).toEqual(afterFields) + }, [() => unsubscribe?.(), () => cleanup?.()]) + }) +}) diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index 45e02ded3b..fdbce68577 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -1,6 +1,6 @@ import { Buffer } from 'node:buffer' import { runInNewContext } from 'node:vm' -import { describe, expect, it, vi } from 'vitest' +import { describe, expect, it } from 'vitest' import { fc, test as fcTest } from '@fast-check/vitest' import { Temporal } from 'temporal-polyfill' import { createCollection } from '../src/collection/index' @@ -1545,10 +1545,8 @@ describe(`areValuesEqual property-based tests`, () => { * Legal inputs are ISO PlainDates from years 2000..2030 with days 1..28 and * ZonedDateTimes within two seconds of a fixed instant in four offset zones. * Equal-value branches construct separate objects, and the zoned branch forces - * different zones at the same instant. A fixed genuine ZonedDateTime also - * shadows its own display tag; a genuine subclass changes its prototype's - * display tag while inheriting Temporal's slot-checking method. The supplied - * instant still determines order for both. + * different zones at the same instant. A normal ZonedDateTime subclass + * inherits its Temporal methods and retains the same instant order. * The production paths are the exported * `ascComparator` under the default options and `compareValues` in hostile * dispatch histories. Each return is the observation checkpoint; @@ -1699,14 +1697,6 @@ const arbitraryZonedDateTimeOrderInput: fc.Arbitrary = })), ) -// Dispatch previously cached the first comparison function seen for a tag. -// Reload before each hostile history so its first dispatch, rather than test -// execution order, determines whether a bad function can poison later values. -async function freshTemporalComparison() { - vi.resetModules() - return import('../src/utils/comparison') -} - describe(`Temporal default-order refinement`, () => { it(`reconstructs the three PlainDate and one ZonedDateTime assertions`, () => { const jan = { year: 2024, month: 1, day: 1 } @@ -1773,20 +1763,6 @@ describe(`Temporal default-order refinement`, () => { expect(() => assertZonedDateTimeOrder(sample, truncated)).toThrow() }) - it(`keeps a genuine ZonedDateTime's instant order when its own tag is shadowed`, () => { - const utc = zonedValue(0, `+00:00`) - const peer = zonedValue(0, `+05:30`) - const later = zonedValue(1, `+05:30`) - Object.defineProperty(utc, Symbol.toStringTag, { value: `ordinary` }) - - // The supplied offsets, not the instance's display property, define the - // model positions. The direct comparator and helper are separate paths. - expect(ascComparator(utc, peer, DEFAULT_COMPARE_OPTIONS)).toBe(0) - expect(ascComparator(peer, utc, DEFAULT_COMPARE_OPTIONS)).toBe(0) - expect(compareValues(utc, peer)).toBe(0) - expect(sign(ascComparator(utc, later, DEFAULT_COMPARE_OPTIONS))).toBe(-1) - }) - it.each([ [ `ascComparator`, @@ -1795,128 +1771,25 @@ describe(`Temporal default-order refinement`, () => { ], [`compareValues`, compareValues], ] as const)( - `keeps a tagged ZonedDateTime subclass in instant order through %s`, + `keeps a ZonedDateTime subclass in instant order through %s`, (_name, compare) => { - class TaggedZonedDateTime extends Temporal.ZonedDateTime {} - Object.defineProperty(TaggedZonedDateTime.prototype, Symbol.toStringTag, { - value: `Tagged`, - }) - const tagged = new TaggedZonedDateTime( + class SubclassZonedDateTime extends Temporal.ZonedDateTime {} + const subclass = new SubclassZonedDateTime( zonedBaseEpochNanoseconds, `+00:00`, ) const peer = zonedValue(0, `+05:30`) const later = zonedValue(1, `+00:00`) - // Model positions come from supplied nanosecond offsets. Native compare - // confirms that the tagged receiver still has genuine Temporal slots. - expect(Temporal.ZonedDateTime.compare(tagged, peer)).toBe(0) - expect(Temporal.ZonedDateTime.compare(tagged, later)).toBe(-1) - expect(compare(tagged, peer)).toBe(0) - expect(compare(peer, tagged)).toBe(0) - expect(sign(compare(tagged, later))).toBe(-1) - expect(sign(compare(later, tagged))).toBe(1) + // The model position comes from the supplied nanosecond offset. + expect(Temporal.ZonedDateTime.compare(subclass, peer)).toBe(0) + expect(Temporal.ZonedDateTime.compare(subclass, later)).toBe(-1) + expect(compare(subclass, peer)).toBe(0) + expect(compare(peer, subclass)).toBe(0) + expect(sign(compare(subclass, later))).toBe(-1) + expect(sign(compare(later, subclass))).toBe(1) }, ) - - it(`keeps a PlainDate constructor override from poisoning later comparisons`, async () => { - const { ascComparator: isolatedComparator, compareValues: isolatedValues } = - await freshTemporalComparison() - // A genuine Temporal instance may have an own data property; its calendar - // value is unchanged by that property. It is the first dispatch here. - const shadowed = new Temporal.PlainDate(2024, 1, 1) - Object.defineProperty(shadowed, `constructor`, { - value: { compare: () => 0 }, - }) - const later = new Temporal.PlainDate(2024, 6, 1) - const ordinary = new Temporal.PlainDate(2024, 1, 1) - const expected = modelPlainDateSign( - { year: 2024, month: 1, day: 1 }, - { year: 2024, month: 6, day: 1 }, - ) - const observed = [ - isolatedComparator(shadowed, later, DEFAULT_COMPARE_OPTIONS), - isolatedComparator(ordinary, later, DEFAULT_COMPARE_OPTIONS), - isolatedValues(ordinary, later), - ] - expect(observed.map(sign)).toEqual([expected, expected, expected]) - }) - - it(`keeps a right-hand constructor override from changing PlainDate order`, async () => { - const { ascComparator: isolatedComparator, compareValues: isolatedValues } = - await freshTemporalComparison() - const january = new Temporal.PlainDate(2024, 1, 1) - const shadowedJune = new Temporal.PlainDate(2024, 6, 1) - Object.defineProperty(shadowedJune, `constructor`, { - value: { compare: () => 0 }, - }) - const ordinaryJanuary = new Temporal.PlainDate(2024, 1, 1) - const ordinaryJune = new Temporal.PlainDate(2024, 6, 1) - const expected = modelPlainDateSign( - { year: 2024, month: 1, day: 1 }, - { year: 2024, month: 6, day: 1 }, - ) - const observed = [ - isolatedComparator(january, shadowedJune, DEFAULT_COMPARE_OPTIONS), - isolatedValues(january, shadowedJune), - isolatedComparator( - ordinaryJanuary, - ordinaryJune, - DEFAULT_COMPARE_OPTIONS, - ), - isolatedValues(ordinaryJanuary, ordinaryJune), - ] - expect(observed.map(sign)).toEqual([expected, expected, expected, expected]) - }) - - it(`keeps a compareValues-first constructor override from poisoning later comparisons`, async () => { - const { ascComparator: isolatedComparator, compareValues: isolatedValues } = - await freshTemporalComparison() - const shadowedJanuary = new Temporal.PlainDate(2024, 1, 1) - Object.defineProperty(shadowedJanuary, `constructor`, { - value: { compare: () => 0 }, - }) - const june = new Temporal.PlainDate(2024, 6, 1) - const ordinaryJanuary = new Temporal.PlainDate(2024, 1, 1) - const expected = modelPlainDateSign( - { year: 2024, month: 1, day: 1 }, - { year: 2024, month: 6, day: 1 }, - ) - const observed = [ - isolatedValues(shadowedJanuary, june), - isolatedValues(ordinaryJanuary, june), - isolatedComparator(ordinaryJanuary, june, DEFAULT_COMPARE_OPTIONS), - ] - expect(observed.map(sign)).toEqual([expected, expected, expected]) - }) - - it(`keeps a forged Temporal tag from poisoning genuine PlainDate order`, async () => { - // A claimed display tag may enter the dispatch path, but must not change - // the result of a later comparison between genuine Temporal values. - const { ascComparator: isolatedComparator, compareValues: isolatedValues } = - await freshTemporalComparison() - const forged = { - [Symbol.toStringTag]: `Temporal.PlainDate`, - constructor: { compare: () => 0 }, - } - const june = new Temporal.PlainDate(2024, 6, 1) - try { - isolatedComparator(forged, june, DEFAULT_COMPARE_OPTIONS) - } catch { - // Rejecting the forged value is a valid outcome for this history. - } - const january = new Temporal.PlainDate(2024, 1, 1) - const expected = modelPlainDateSign( - { year: 2024, month: 1, day: 1 }, - { year: 2024, month: 6, day: 1 }, - ) - expect( - [ - isolatedComparator(january, june, DEFAULT_COMPARE_OPTIONS), - isolatedValues(january, june), - ].map(sign), - ).toEqual([expected, expected]) - }) }) describe(`native binary key content`, () => { diff --git a/packages/db/tests/index-update-oracle.property.test.ts b/packages/db/tests/index-update-oracle.property.test.ts index 8a2e01f3e2..8d95709b2b 100644 --- a/packages/db/tests/index-update-oracle.property.test.ts +++ b/packages/db/tests/index-update-oracle.property.test.ts @@ -476,6 +476,177 @@ describe.each(indexTypes)(`%s update properties`, (_indexName, IndexType) => { }) }) +/** + * Equal native Dates identify one exact index value by timestamp, even when + * an add and its later remove receive different Date objects. Authority: the + * IndexInterface equality and ordered-read methods, together with the native + * Date timestamp comparison law in the comparison oracle. Both direct-index + * histories use valid Dates: one follows default timestamp order, while the + * other lets a custom callback separate rows with equal timestamps. Neither + * claims arbitrary object equivalence or Collection publication. + * + * The default lane's independent source Map stores supplied epoch numbers, + * not the objects or an index-normalized value. Its fresh sort predicts the + * public order; filtering its rows predicts exact and lower-bound membership. + * The callback lane's Map stores supplied order positions separately from + * exact timestamp membership. A same-object re-add after an equal-but-distinct + * removal distinguishes a live comparator position from an obsolete one. A + * second row and neighboring epochs exercise promotion and bounded traversal. + * Each add, remove, and update uses the direct index API and is compared after + * return. + */ +function expectDateIndexMatchesEpochModel( + index: BaseIndex, + rows: ReadonlyMap, +): void { + const ordered = [...rows].sort( + ([leftKey, leftEpoch], [rightKey, rightEpoch]) => + leftEpoch - rightEpoch || + (leftKey < rightKey ? -1 : leftKey > rightKey ? 1 : 0), + ) + expect(index.keyCount).toBe(rows.size) + expect(index.takeFromStart(rows.size + 1)).toEqual( + ordered.map(([key]) => key), + ) + expect(index.takeReversedFromEnd(rows.size + 1)).toEqual( + ordered.map(([key]) => key).reverse(), + ) + + for (const epoch of [0, 100, 200]) { + const exact = new Set( + [...rows].filter(([, value]) => value === epoch).map(([key]) => key), + ) + const atOrAbove = new Set( + [...rows].filter(([, value]) => value >= epoch).map(([key]) => key), + ) + expect(index.equalityLookup(new Date(epoch))).toEqual(exact) + expect(index.lookup(`eq`, new Date(epoch))).toEqual(exact) + expect(index.rangeQuery({ from: new Date(epoch) })).toEqual(atOrAbove) + expect(index.lookup(`gte`, new Date(epoch))).toEqual(atOrAbove) + } +} + +describe.each(indexTypes)(`%s equivalent Date removal`, (_name, IndexType) => { + test(`keeps re-added rows in ordered reads after removal by an equal Date`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + const original = new Date(100) + const peer = new Date(100) + + const add = (key: string, value: Date, epoch: number) => { + index.add(key, { value }) + rows.set(key, epoch) + expectDateIndexMatchesEpochModel(index, rows) + } + const remove = (key: string, value: Date) => { + index.remove(key, { value }) + rows.delete(key) + expectDateIndexMatchesEpochModel(index, rows) + } + const update = ( + key: string, + oldValue: Date, + newValue: Date, + newEpoch: number, + ) => { + index.update(key, { value: oldValue }, { value: newValue }) + rows.set(key, newEpoch) + expectDateIndexMatchesEpochModel(index, rows) + } + + expectDateIndexMatchesEpochModel(index, rows) + add(`first`, original, 100) + remove(`first`, new Date(100)) + add(`second`, original, 100) + remove(`second`, new Date(100)) + add(`before`, new Date(0), 0) + add(`first`, original, 100) + add(`peer`, peer, 100) + remove(`first`, new Date(100)) + add(`after`, new Date(200), 200) + update(`peer`, new Date(100), new Date(200), 200) + add(`first`, original, 100) + }) + + test(`keeps callback-distinct Date positions through equivalent removal and update`, () => { + // The callback is a declared ordering fact. Two Dates with one exact + // timestamp can occupy different positions; the source model records each + // row's supplied position separately from its timestamp. A fresh old-item + // Date at position zero is legal during removal and update. + const first = new Date(100) + const second = new Date(100) + const replacement = new Date(100) + const positions = new WeakMap([ + [first, 0], + [second, 1], + [replacement, 1], + ]) + const index = new IndexType(1, new PropRef([`value`]), undefined, { + compareFn: (left: unknown, right: unknown) => + (left as Date).getTime() - (right as Date).getTime() || + (positions.get(left as Date) ?? 0) - + (positions.get(right as Date) ?? 0), + }) + const rows = new Map() + const check = () => { + const ordered = [...rows].sort( + ([leftKey, leftPosition], [rightKey, rightPosition]) => + leftPosition - rightPosition || + (leftKey < rightKey ? -1 : leftKey > rightKey ? 1 : 0), + ) + expect(index.keyCount).toBe(rows.size) + expect(index.equalityLookup(new Date(100))).toEqual(new Set(rows.keys())) + expect(index.takeFromStart(rows.size + 1)).toEqual( + ordered.map(([key]) => key), + ) + for (const [position, bound] of [ + [0, first], + [1, second], + ] as const) { + expect(index.rangeQuery({ from: bound, to: bound })).toEqual( + new Set( + [...rows] + .filter(([, rowPosition]) => rowPosition === position) + .map(([key]) => key), + ), + ) + } + } + + check() + index.add(`z`, { value: first }) + rows.set(`z`, 0) + check() + index.add(`a`, { value: second }) + rows.set(`a`, 1) + check() + index.remove(`z`, { value: new Date(100) }) + rows.delete(`z`) + check() + index.add(`z`, { value: first }) + rows.set(`z`, 0) + check() + index.update(`z`, { value: new Date(100) }, { value: replacement }) + rows.set(`z`, 1) + check() + index.add(`b`, { value: first }) + rows.set(`b`, 0) + check() + // The old item has the same exact timestamp as `a`, but its callback + // position is zero while `a` was stored at position one. `b` occupies + // position zero, so searching by the old item would find the wrong bucket. + index.remove(`a`, { value: new Date(100) }) + rows.delete(`a`) + check() + index.update(`z`, { value: new Date(100) }, { value: first }) + rows.set(`z`, 0) + check() + index.add(`a`, { value: second }) + rows.set(`a`, 1) + check() + }) +}) + /** * Direct index equality keeps `undefined` and `null` in distinct exact-value * buckets, even though the default ordering comparator places both at one diff --git a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts index fb4db2c06d..914e04a94c 100644 --- a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts +++ b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts @@ -52,8 +52,7 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * content, genuine local and foreign bytes with own display-tag shadows, bytes * with own `length` or `byteLength` shadows, and a wide typed array with a * spoofed byte tag. A fixed control also shadows byte iteration. Temporal - * values include ZonedDateTime zone aliases, a forged tag, a borrowed - * prototype, shadowed display and field members, unequal values within each + * values include ZonedDateTime zone aliases, unequal values within each * modeled Temporal kind, reference ISO year/day * differences hidden by default display, and Duration carry forms with equal * display but distinct fields, one shared object and a structurally equal @@ -73,9 +72,9 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * without normalization and requires the checker to reject it. * * Limits: this owner checks the boolean result of `in` for ordinary values - * from one Temporal polyfill copy and the listed display shadows. It does not - * claim mixed Temporal implementations, replaced genuine prototypes, or - * arbitrary custom prototypes that imitate Temporal intrinsics. WHERE + * from one Temporal polyfill copy. It does not claim mixed Temporal + * implementations, malformed Temporal lookalikes, deliberately shadowed + * Temporal members, or replaced genuine prototypes. WHERE * publication, indexes, and subset loading have other owners. */ @@ -136,22 +135,7 @@ const otherSpoofedWideBytes = new Uint16Array([1, 2]) Object.defineProperty(otherSpoofedWideBytes, Symbol.toStringTag, { value: `Uint8Array`, }) -const forgedPlainDate = { - [Symbol.toStringTag]: `Temporal.PlainDate`, - toString: () => `2024-01-01`, -} -const borrowedPlainDate = Object.assign( - Object.create(Temporal.PlainDate.prototype) as object, - { toString: () => `2024-01-01` }, -) -const shadowedTagPlainDate = Temporal.PlainDate.from(`2024-01-01`) -Object.defineProperty(shadowedTagPlainDate, Symbol.toStringTag, { - value: `ordinary`, -}) -const shadowedTextPlainDate = Temporal.PlainDate.from(`2024-01-01`) -Object.defineProperty(shadowedTextPlainDate, `toString`, { - value: () => `unrelated text`, -}) +const samePlainDate = Temporal.PlainDate.from(`2024-01-01`) const leapMonthDay = new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972) const sameLeapMonthDay = new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972) const otherReferenceMonthDay = new Temporal.PlainMonthDay( @@ -173,8 +157,6 @@ const sameHourDuration = Temporal.Duration.from(`PT1H`) const sixtyMinuteDuration = Temporal.Duration.from(`PT60M`) const secondDuration = Temporal.Duration.from({ seconds: 1 }) const sameSecondDuration = Temporal.Duration.from({ seconds: 1 }) -const shadowedSecondDuration = Temporal.Duration.from({ seconds: 1 }) -Object.defineProperty(shadowedSecondDuration, `seconds`, { value: 2 }) const thousandMillisecondDuration = Temporal.Duration.from({ milliseconds: 1000, }) @@ -184,14 +166,6 @@ const sameThousandMillisecondDuration = Temporal.Duration.from({ const millionMicrosecondDuration = Temporal.Duration.from({ microseconds: 1_000_000, }) -const shadowedHourDuration = Temporal.Duration.from(`PT1H`) -Object.defineProperty(shadowedHourDuration, `toString`, { - value: () => `PT2H`, -}) -const forgedHourDuration = { - [Symbol.toStringTag]: `Temporal.Duration`, - toString: () => `PT1H`, -} const utcZoned = Temporal.ZonedDateTime.from(`2024-01-15T09:00:00+00:00[UTC]`) const aliasZoned = Temporal.ZonedDateTime.from( `2024-01-15T09:00:00+00:00[Etc/UTC]`, @@ -199,17 +173,6 @@ const aliasZoned = Temporal.ZonedDateTime.from( const offsetZoned = Temporal.ZonedDateTime.from( `2024-01-15T09:00:00+00:00[+00:00]`, ) -const shadowedOffsetZoned = Temporal.ZonedDateTime.from( - `2024-01-15T09:00:00+00:00[+00:00]`, -) -Object.defineProperty(shadowedOffsetZoned, `timeZoneId`, { value: `UTC` }) -const forgedZoned = { - [Symbol.toStringTag]: `Temporal.ZonedDateTime`, - epochNanoseconds: utcZoned.epochNanoseconds, - timeZoneId: `UTC`, - calendarId: `iso8601`, - toString: () => `unrelated plain object`, -} const pool: ReadonlyArray = [ `a`, `b`, @@ -259,10 +222,7 @@ const pool: ReadonlyArray = [ plainTime, samePlainTime, otherPlainTime, - forgedPlainDate, - borrowedPlainDate, - shadowedTagPlainDate, - shadowedTextPlainDate, + samePlainDate, leapMonthDay, sameLeapMonthDay, otherReferenceMonthDay, @@ -274,17 +234,12 @@ const pool: ReadonlyArray = [ sixtyMinuteDuration, secondDuration, sameSecondDuration, - shadowedSecondDuration, thousandMillisecondDuration, sameThousandMillisecondDuration, millionMicrosecondDuration, - shadowedHourDuration, - forgedHourDuration, utcZoned, aliasZoned, offsetZoned, - shadowedOffsetZoned, - forgedZoned, shared, { id: 1 }, null, @@ -324,10 +279,8 @@ function isBytes(value: unknown): value is Uint8Array { ) } -// The trusted polyfill methods inspect Temporal's internal slots. A user can -// shadow an instance's tag, equals, or toString, and can borrow its prototype; -// none of those visible properties establishes the brand. The bounded model -// tests the trusted methods directly and never imports production's classifier. +// The polyfill methods supply the reference equality relation for the +// supported Temporal values. The model never imports production's classifier. const temporalEqualities = [ [`Instant`, Temporal.Instant.prototype.equals], [`PlainDate`, Temporal.PlainDate.prototype.equals], @@ -370,7 +323,7 @@ function temporalBrand(value: unknown): TemporalBrand | undefined { } } } catch { - // A borrowed or forged prototype has no Temporal internal slots. + // Ordinary non-Temporal objects fail these intrinsic calls. } } try { @@ -539,12 +492,6 @@ function evaluateWithKindOnlyTemporalKey(testCase: InCase): boolean | null { function describeCase(testCase: InCase): string { const show = (value: unknown): string => { - if (value === forgedPlainDate) return `forged PlainDate tag` - if (value === borrowedPlainDate) return `borrowed PlainDate prototype` - if (value === shadowedTagPlainDate) return `PlainDate with shadowed tag` - if (value === shadowedTextPlainDate) return `PlainDate with shadowed text` - if (value === shadowedHourDuration) return `Duration with shadowed text` - if (value === forgedHourDuration) return `forged Duration tag` if (value === borrowedDate) return `borrowed Date prototype` if (value === otherBorrowedDate) return `other borrowed Date prototype` const dateTime = nativeDateTimestamp(value) @@ -563,7 +510,7 @@ function describeCase(testCase: InCase): string { try { return String(value) } catch { - return `object with borrowed Temporal prototype` + return `object` } })() : typeof value === `string` @@ -803,7 +750,7 @@ describe(`in evaluator`, () => { expect(plainTime.equals(samePlainTime)).toBe(true) expect(plainTime.equals(otherPlainTime)).toBe(false) for (const [left, equal, other] of [ - [plainDate, shadowedTagPlainDate, otherPlainDate], + [plainDate, samePlainDate, otherPlainDate], [instant, sameInstant, otherInstant], [plainDateTime, samePlainDateTime, otherPlainDateTime], [plainTime, samePlainTime, otherPlainTime], @@ -818,7 +765,7 @@ describe(`in evaluator`, () => { it(`rejects a kind-only Temporal list key at the membership comparison`, () => { for (const [value, equal, unequal] of [ - [plainDate, shadowedTagPlainDate, otherPlainDate], + [plainDate, samePlainDate, otherPlainDate], [instant, sameInstant, otherInstant], [plainDateTime, samePlainDateTime, otherPlainDateTime], [plainTime, samePlainTime, otherPlainTime], @@ -869,40 +816,6 @@ describe(`in evaluator`, () => { }, ) - it.each([false, true])( - `rejects a forged tag and a shadowed zone in a %s list`, - (listFromRow) => { - expect(utcZoned.equals(shadowedOffsetZoned)).toBe(false) - for (const value of [forgedZoned, shadowedOffsetZoned]) { - check(evaluateWithProduction, { - value, - list: [utcZoned], - listFromRow, - }) - } - }, - ) - - it.each( - [false, true].flatMap((listFromRow) => - [ - [`forged own tag`, forgedPlainDate], - [`borrowed prototype`, borrowedPlainDate], - [`shadowed own tag`, shadowedTagPlainDate], - [`shadowed toString`, shadowedTextPlainDate], - ].map(([name, value]) => ({ name, value, listFromRow })), - ), - )( - `uses the native brand for $name with listFromRow=$listFromRow`, - ({ value, listFromRow }) => { - check(evaluateWithProduction, { - value, - list: [plainDate], - listFromRow, - }) - }, - ) - it.each([false, true])( `matches Duration fields, while keeping equivalent representations distinct in a %s list`, (listFromRow) => { @@ -916,16 +829,6 @@ describe(`in evaluator`, () => { list: [hourDuration], listFromRow, }) - check(evaluateWithProduction, { - value: shadowedHourDuration, - list: [hourDuration], - listFromRow, - }) - check(evaluateWithProduction, { - value: forgedHourDuration, - list: [hourDuration], - listFromRow, - }) }, ) @@ -968,7 +871,6 @@ describe(`in evaluator`, () => { for (const value of [ secondDuration, sameSecondDuration, - shadowedSecondDuration, thousandMillisecondDuration, sameThousandMillisecondDuration, millionMicrosecondDuration, @@ -977,9 +879,6 @@ describe(`in evaluator`, () => { } checkEqualityPaths(secondDuration, sameSecondDuration, true) - // An own getter shadow cannot change the intrinsic Duration fields. - expect(shadowedSecondDuration.seconds).toBe(2) - checkEqualityPaths(secondDuration, shadowedSecondDuration, true) checkEqualityPaths( thousandMillisecondDuration, sameThousandMillisecondDuration, diff --git a/packages/db/tests/temporal-equality-alias-oracle.test.ts b/packages/db/tests/temporal-equality-alias-oracle.test.ts index 85f8fcc8a1..da82b9244e 100644 --- a/packages/db/tests/temporal-equality-alias-oracle.test.ts +++ b/packages/db/tests/temporal-equality-alias-oracle.test.ts @@ -13,18 +13,13 @@ import { withOracleCleanup } from './utils.js' * the fixed-offset +00:00 zone remains a different identity. * * The ZonedDateTime lane covers values from one polyfill copy and one instant, - * including genuine subclasses with inherited Temporal methods. Subclass - * prototypes have unrelated and misleading Temporal-kind display tags; their - * native slots still determine equality. Other genuine instances have own - * display and intrinsic-getter shadows. A neighboring - * PlainDate lane checks that native brand and equality survive those shadows - * while forged tags retain ordinary object identity. Fixture-assigned classes, + * including a normal subclass with inherited Temporal methods. + * Fixture-assigned classes, * independent of production normalization, predict exact key sets. Direct * BasicIndex/BTreeIndex add, build, and removal histories compare those sets * after each returned action. Public Collection * reads compare an unindexed scan and an optimized-only indexed snapshot after - * readiness. The mixed forged/genuine PlainDate Collection has no index: - * default range order across an ordinary object and Temporal is unspecified. + * readiness. * A second lane uses one polyfill copy to check `PlainMonthDay` and * `PlainYearMonth` reference ISO fields, and Duration's ten fields, under * the same exact query equality. Changelog #1519 explicitly keeps equivalent @@ -39,7 +34,8 @@ import { withOracleCleanup } from './utils.js' * months, weeks, days, and nanoseconds one field at a time; a key that omits * any one of those fields would merge its distinct fixture classes. * This oracle does not claim all IANA aliases, subclasses with replaced - * Temporal methods, arbitrary spoofed prototypes, + * Temporal methods, malformed Temporal lookalikes or deliberately shadowed + * Temporal members, * mixed Temporal implementations, joins, live-query publication, or provider * behavior. */ @@ -48,30 +44,10 @@ type Sample = { id: string text: string identity: `utc` | `offset` | `later` | `calendar` - ownShadow?: `tag` | `text` | `epoch` | `calendar` - subclass?: `untagged` | `tagged` | `plain-date-tagged` + subclass?: boolean } class ZonedDateTimeSubclass extends Temporal.ZonedDateTime {} -class TaggedZonedDateTimeSubclass extends Temporal.ZonedDateTime {} -Object.defineProperty( - TaggedZonedDateTimeSubclass.prototype, - Symbol.toStringTag, - { - value: `Tagged`, - }, -) -class PlainDateTaggedZonedDateTimeSubclass extends Temporal.ZonedDateTime {} -Object.defineProperty( - PlainDateTaggedZonedDateTimeSubclass.prototype, - Symbol.toStringTag, - { value: `Temporal.PlainDate` }, -) -const subclassConstructors = { - untagged: ZonedDateTimeSubclass, - tagged: TaggedZonedDateTimeSubclass, - 'plain-date-tagged': PlainDateTaggedZonedDateTimeSubclass, -} as const const samples: ReadonlyArray = [ { @@ -99,68 +75,24 @@ const samples: ReadonlyArray = [ text: `2024-01-15T09:00:00+00:00[UTC][u-ca=gregory]`, identity: `calendar`, }, - { - id: `utc-shadowed-tag`, - text: `2024-01-15T09:00:00+00:00[UTC]`, - identity: `utc`, - ownShadow: `tag`, - }, - { - id: `utc-shadowed-text`, - text: `2024-01-15T09:00:00+00:00[UTC]`, - identity: `utc`, - ownShadow: `text`, - }, - { - id: `utc-shadowed-epoch`, - text: `2024-01-15T09:00:00+00:00[UTC]`, - identity: `utc`, - ownShadow: `epoch`, - }, - { - id: `utc-shadowed-calendar`, - text: `2024-01-15T09:00:00+00:00[UTC]`, - identity: `utc`, - ownShadow: `calendar`, - }, { id: `utc-subclass`, text: `2024-01-15T09:00:00+00:00[UTC]`, identity: `utc`, - subclass: `untagged`, - }, - { - id: `utc-tagged-subclass`, - text: `2024-01-15T09:00:00+00:00[UTC]`, - identity: `utc`, - subclass: `tagged`, - }, - { - id: `utc-plain-date-tagged-subclass`, - text: `2024-01-15T09:00:00+00:00[UTC]`, - identity: `utc`, - subclass: `plain-date-tagged`, + subclass: true, }, ] function rows(): Array { - return samples.map(({ id, text, ownShadow, subclass }) => { + return samples.map(({ id, text, subclass }) => { const base = Temporal.ZonedDateTime.from(text) - const Subclass = subclass && subclassConstructors[subclass] - const when = Subclass - ? new Subclass(base.epochNanoseconds, base.timeZoneId, base.calendarId) + const when = subclass + ? new ZonedDateTimeSubclass( + base.epochNanoseconds, + base.timeZoneId, + base.calendarId, + ) : base - if (ownShadow === `tag`) { - Object.defineProperty(when, Symbol.toStringTag, { value: `ordinary` }) - } else if (ownShadow === `text`) { - Object.defineProperty(when, `toString`, { - value: () => `unrelated text`, - }) - } else if (ownShadow === `epoch`) { - Object.defineProperty(when, `epochNanoseconds`, { value: 0n }) - } else if (ownShadow === `calendar`) { - Object.defineProperty(when, `calendarId`, { value: `gregory` }) - } return { id, when } }) } @@ -180,14 +112,6 @@ const subclassProbe = new ZonedDateTimeSubclass( aliasProbe.epochNanoseconds, `UTC`, ) -const taggedSubclassProbe = new TaggedZonedDateTimeSubclass( - aliasProbe.epochNanoseconds, - `UTC`, -) -const plainDateTaggedSubclassProbe = new PlainDateTaggedZonedDateTimeSubclass( - aliasProbe.epochNanoseconds, - `UTC`, -) const offsetProbe = Temporal.ZonedDateTime.from( `2024-01-15T09:00:00+00:00[+00:00]`, ) @@ -268,19 +192,12 @@ function intrinsicFixture(kind: IntrinsicKind): IntrinsicFixture { const seconds = Temporal.Duration.from({ seconds: 1 }) const secondsCopy = Temporal.Duration.from({ seconds: 1 }) - const shadowedSeconds = Temporal.Duration.from({ seconds: 1 }) - Object.defineProperty(shadowedSeconds, `seconds`, { value: 2 }) const milliseconds = Temporal.Duration.from({ milliseconds: 1000 }) const microseconds = Temporal.Duration.from({ microseconds: 1_000_000 }) - const intrinsicSeconds = Object.getOwnPropertyDescriptor( - Temporal.Duration.prototype, - `seconds`, - )!.get! return { rows: [ { id: `seconds`, when: seconds, identity: `seconds` }, { id: `seconds-copy`, when: secondsCopy, identity: `seconds` }, - { id: `seconds-shadow`, when: shadowedSeconds, identity: `seconds` }, { id: `milliseconds`, when: milliseconds, identity: `milliseconds` }, { id: `microseconds`, when: microseconds, identity: `microseconds` }, ], @@ -298,8 +215,6 @@ function intrinsicFixture(kind: IntrinsicKind): IntrinsicFixture { checkNativeRelation: () => { expect(seconds.toString()).toBe(milliseconds.toString()) expect(milliseconds.toString()).toBe(microseconds.toString()) - expect(intrinsicSeconds.call(shadowedSeconds)).toBe(1) - expect(shadowedSeconds.seconds).toBe(2) expect(seconds.seconds).toBe(1) expect(milliseconds.milliseconds).toBe(1000) expect(microseconds.microseconds).toBe(1_000_000) @@ -389,51 +304,14 @@ function intrinsicExpectedKeys( ) } -it(`distinguishes equivalent aliases and own property shadows from a fixed-offset zone`, () => { - const [ - utc, - alias, - fixed, - later, - gregory, - shadowedTag, - shadowedText, - shadowedEpoch, - shadowedCalendar, - subclass, - taggedSubclass, - plainDateTaggedSubclass, - ] = rows() +it(`distinguishes equivalent aliases from a fixed-offset zone`, () => { + const [utc, alias, fixed, later, gregory, subclass] = rows() // This checks the fixture against the external Temporal relation. Expected // index and query sets still come from the assigned identity classes. expect(utc!.when.equals(alias!.when)).toBe(true) expect(utc!.when.equals(fixed!.when)).toBe(false) expect(utc!.when.equals(later!.when)).toBe(false) expect(utc!.when.equals(gregory!.when)).toBe(false) - expect( - Temporal.ZonedDateTime.prototype.equals.call( - shadowedTag!.when, - alias!.when, - ), - ).toBe(true) - expect( - Temporal.ZonedDateTime.prototype.equals.call( - shadowedText!.when, - alias!.when, - ), - ).toBe(true) - expect( - Temporal.ZonedDateTime.prototype.equals.call( - shadowedEpoch!.when, - alias!.when, - ), - ).toBe(true) - expect( - Temporal.ZonedDateTime.prototype.equals.call( - shadowedCalendar!.when, - alias!.when, - ), - ).toBe(true) // A subclass inherits Temporal's slot-checking method. The fixture assigns // it the UTC class independently of the production classifier. expect(Object.getPrototypeOf(subclass!.when)).toBe( @@ -448,71 +326,9 @@ it(`distinguishes equivalent aliases and own property shadows from a fixed-offse expect( Temporal.ZonedDateTime.prototype.equals.call(subclassProbe, alias!.when), ).toBe(true) - // The subclass prototype's tag is presentation. The inherited intrinsic - // still sees genuine ZonedDateTime slots and the same assigned UTC class. - expect(Object.getPrototypeOf(taggedSubclass!.when)).toBe( - TaggedZonedDateTimeSubclass.prototype, - ) - expect(Object.getPrototypeOf(taggedSubclass!.when)[Symbol.toStringTag]).toBe( - `Tagged`, - ) - expect( - Object.getOwnPropertyDescriptor( - TaggedZonedDateTimeSubclass.prototype, - `toString`, - ), - ).toBeUndefined() - expect( - Temporal.ZonedDateTime.prototype.equals.call( - taggedSubclass!.when, - alias!.when, - ), - ).toBe(true) - expect( - Temporal.ZonedDateTime.prototype.equals.call( - taggedSubclassProbe, - alias!.when, - ), - ).toBe(true) - // A tag naming another Temporal kind is still only presentation. The - // inherited ZonedDateTime intrinsic checks the actual subclass slots. - expect(Object.getPrototypeOf(plainDateTaggedSubclass!.when)).toBe( - PlainDateTaggedZonedDateTimeSubclass.prototype, - ) - expect( - Object.getPrototypeOf(plainDateTaggedSubclass!.when)[Symbol.toStringTag], - ).toBe(`Temporal.PlainDate`) - expect( - Object.getOwnPropertyDescriptor( - PlainDateTaggedZonedDateTimeSubclass.prototype, - `equals`, - ), - ).toBeUndefined() - expect( - Temporal.ZonedDateTime.prototype.equals.call( - plainDateTaggedSubclass!.when, - alias!.when, - ), - ).toBe(true) - expect( - Temporal.ZonedDateTime.prototype.equals.call( - plainDateTaggedSubclassProbe, - alias!.when, - ), - ).toBe(true) expect(utc!.when.toString()).not.toBe(alias!.when.toString()) expect(expectedKeys(`utc`)).toEqual( - new Set([ - `utc`, - `utc-alias`, - `utc-shadowed-tag`, - `utc-shadowed-text`, - `utc-shadowed-epoch`, - `utc-shadowed-calendar`, - `utc-subclass`, - `utc-tagged-subclass`, - `utc-plain-date-tagged-subclass`, - ]), + new Set([`utc`, `utc-alias`, `utc-subclass`]), ) expect(expectedKeys(`offset`)).toEqual(new Set([`fixed-offset`])) }) @@ -536,12 +352,7 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { // The query value is a fresh instance. A display-string key would omit // UTC; an instant-only key would also include fixed-offset. - for (const probe of [ - aliasProbe, - subclassProbe, - taggedSubclassProbe, - plainDateTaggedSubclassProbe, - ]) { + for (const probe of [aliasProbe, subclassProbe]) { expect(index.equalityLookup(probe)).toEqual(expectedKeys(`utc`)) expect(index.lookup(`eq`, probe)).toEqual(expectedKeys(`utc`)) } @@ -563,8 +374,6 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { for (const [id, probe] of [ [`utc`, aliasProbe], [`utc-subclass`, subclassProbe], - [`utc-tagged-subclass`, taggedSubclassProbe], - [`utc-plain-date-tagged-subclass`, plainDateTaggedSubclassProbe], ] as const) { const row = source.find((candidate) => candidate.id === id)! index.remove(row.id, row) @@ -616,12 +425,7 @@ describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { `lookup`, ) : undefined - for (const probe of [ - aliasProbe, - subclassProbe, - taggedSubclassProbe, - plainDateTaggedSubclassProbe, - ]) { + for (const probe of [aliasProbe, subclassProbe]) { lookup?.mockClear() const result = readKeys(probe) if (lookup) expect(lookup).toHaveBeenCalled() @@ -865,267 +669,3 @@ describe.each(indexTypes)( ) }, ) - -it(`keeps a forged tag and shadowed zone outside genuine UTC equality`, async () => { - const fake = { - [Symbol.toStringTag]: `Temporal.ZonedDateTime`, - epochNanoseconds: aliasProbe.epochNanoseconds, - timeZoneId: `UTC`, - calendarId: `iso8601`, - toString: () => `unrelated plain object`, - } - const shadowed = Temporal.ZonedDateTime.from( - `2024-01-15T09:00:00+00:00[+00:00]`, - ) - const freshOffsetProbe = Temporal.ZonedDateTime.from( - `2024-01-15T09:00:00+00:00[+00:00]`, - ) - const borrowedSubclass = Object.assign( - Object.create(ZonedDateTimeSubclass.prototype) as object, - { toString: () => `unrelated borrowed object` }, - ) - const borrowedSubclassCopy = Object.assign( - Object.create(ZonedDateTimeSubclass.prototype) as object, - { toString: () => `unrelated borrowed object` }, - ) - const borrowedTaggedSubclass = Object.assign( - Object.create(TaggedZonedDateTimeSubclass.prototype) as object, - { toString: () => `unrelated borrowed object` }, - ) - const borrowedTaggedSubclassCopy = Object.assign( - Object.create(TaggedZonedDateTimeSubclass.prototype) as object, - { toString: () => `unrelated borrowed object` }, - ) - const borrowedPlainDateTaggedSubclass = Object.assign( - Object.create(PlainDateTaggedZonedDateTimeSubclass.prototype) as object, - { toString: () => `unrelated borrowed object` }, - ) - const borrowedPlainDateTaggedSubclassCopy = Object.assign( - Object.create(PlainDateTaggedZonedDateTimeSubclass.prototype) as object, - { toString: () => `unrelated borrowed object` }, - ) - Object.defineProperty(shadowed, `timeZoneId`, { value: `UTC` }) - expect(aliasProbe.equals(shadowed)).toBe(false) - expect(shadowed).not.toBe(freshOffsetProbe) - expect(shadowed.equals(freshOffsetProbe)).toBe(true) - expect(() => Reflect.apply(aliasProbe.equals, aliasProbe, [fake])).toThrow() - expect(() => - Temporal.ZonedDateTime.prototype.equals.call(borrowedSubclass, aliasProbe), - ).toThrow() - expect(() => - Temporal.ZonedDateTime.prototype.equals.call( - borrowedTaggedSubclass, - aliasProbe, - ), - ).toThrow() - expect(() => - Temporal.ZonedDateTime.prototype.equals.call( - borrowedPlainDateTaggedSubclass, - aliasProbe, - ), - ).toThrow() - - const source = [ - { id: `forged`, when: fake }, - { id: `shadowed`, when: shadowed }, - { id: `borrowed-subclass`, when: borrowedSubclass }, - { id: `borrowed-tagged-subclass`, when: borrowedTaggedSubclass }, - { - id: `borrowed-plain-date-tagged-subclass`, - when: borrowedPlainDateTaggedSubclass, - }, - ] - const fakeCopy = { ...fake } - for (const [, IndexType] of indexTypes) { - for (const row of source) { - // One row avoids asking the default ordering comparator to order a - // forged object and a genuine Temporal value against each other. - const index = new IndexType(1, new PropRef([`when`])) - index.add(row.id, row) - expect(index.equalityLookup(aliasProbe)).toEqual(new Set()) - // Rejecting the forged brand must not discard the ordinary object's - // exact self-identity. A distinct structural copy stays outside it. - expect(index.equalityLookup(row.when)).toEqual(new Set([row.id])) - if (row.id === `forged`) { - expect(index.equalityLookup(fakeCopy)).toEqual(new Set()) - } else if (row.id === `borrowed-subclass`) { - expect(index.equalityLookup(borrowedSubclassCopy)).toEqual(new Set()) - } else if (row.id === `borrowed-tagged-subclass`) { - expect(index.equalityLookup(borrowedTaggedSubclassCopy)).toEqual( - new Set(), - ) - } else if (row.id === `borrowed-plain-date-tagged-subclass`) { - expect( - index.equalityLookup(borrowedPlainDateTaggedSubclassCopy), - ).toEqual(new Set()) - } else { - // The own zone accessor must not turn a genuine fixed-offset value - // into reference-only equality with its native-equal fresh instance. - expect(index.equalityLookup(freshOffsetProbe)).toEqual( - new Set([row.id]), - ) - } - } - } - - const collection = createCollection<{ id: string; when: unknown }, string>({ - id: `temporal-alias-spoof-control`, - getKey: (row) => row.id, - autoIndex: `off`, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of source) write({ type: `insert`, value: row }) - commit() - markReady() - }, - }, - }) - await withOracleCleanup(async () => { - await collection.stateWhenReady() - const keysFor = (probe: unknown) => { - const predicate = new Func(`eq`, [ - new PropRef([`when`]), - new Value(probe), - ]) - return new Set( - collection - .currentStateAsChanges({ where: predicate }) - ?.map(({ value }) => value.id), - ) - } - expect(keysFor(aliasProbe)).toEqual(new Set()) - expect(keysFor(freshOffsetProbe)).toEqual(new Set([`shadowed`])) - expect(keysFor(fake)).toEqual(new Set([`forged`])) - expect(keysFor(fakeCopy)).toEqual(new Set()) - expect(keysFor(borrowedSubclass)).toEqual(new Set([`borrowed-subclass`])) - expect(keysFor(borrowedSubclassCopy)).toEqual(new Set()) - expect(keysFor(borrowedTaggedSubclass)).toEqual( - new Set([`borrowed-tagged-subclass`]), - ) - expect(keysFor(borrowedTaggedSubclassCopy)).toEqual(new Set()) - expect(keysFor(borrowedPlainDateTaggedSubclass)).toEqual( - new Set([`borrowed-plain-date-tagged-subclass`]), - ) - expect(keysFor(borrowedPlainDateTaggedSubclassCopy)).toEqual(new Set()) - }, [() => collection.cleanup()]) -}) - -it(`uses the PlainDate brand for direct lookups and a public unindexed snapshot`, async () => { - const genuine = Temporal.PlainDate.from(`2024-01-01`) - const fake = { - [Symbol.toStringTag]: `Temporal.PlainDate`, - toString: () => `2024-01-01`, - } - const borrowed = Object.assign( - Object.create(Temporal.PlainDate.prototype) as object, - { toString: () => `2024-01-01` }, - ) - const shadowedTag = Temporal.PlainDate.from(`2024-01-01`) - Object.defineProperty(shadowedTag, Symbol.toStringTag, { - value: `ordinary`, - }) - const shadowedText = Temporal.PlainDate.from(`2024-01-01`) - Object.defineProperty(shadowedText, `toString`, { - value: () => `unrelated text`, - }) - expect(Temporal.PlainDate.prototype.equals.call(shadowedTag, genuine)).toBe( - true, - ) - expect(Temporal.PlainDate.prototype.equals.call(shadowedText, genuine)).toBe( - true, - ) - - const brandSamples: ReadonlyArray<{ - id: string - when: unknown - matches: boolean - }> = [ - { id: `genuine`, when: genuine, matches: true }, - { id: `forged-tag`, when: fake, matches: false }, - { id: `borrowed-prototype`, when: borrowed, matches: false }, - { id: `shadowed-tag`, when: shadowedTag, matches: true }, - { id: `shadowed-text`, when: shadowedText, matches: true }, - ] - const probe = Temporal.PlainDate.from(`2024-01-01`) - expect(() => - Reflect.apply(Temporal.PlainDate.prototype.equals, fake, [probe]), - ).toThrow() - expect(() => - Reflect.apply(Temporal.PlainDate.prototype.equals, borrowed, [probe]), - ).toThrow() - - // A one-row index tests each exact equality class without ordering mixed - // ordinary objects and Temporal values, whose range order is unspecified. - for (const [, IndexType] of indexTypes) { - for (const sample of brandSamples) { - const index = new IndexType(1, new PropRef([`when`])) - index.add(sample.id, sample) - expect(index.equalityLookup(probe)).toEqual( - sample.matches ? new Set([sample.id]) : new Set(), - ) - expect(index.equalityLookup(sample.when)).toEqual(new Set([sample.id])) - } - // Both insertion orders must retain the same equality class when a live - // representative is removed. The forged objects are intentionally kept - // out of this ordered index: the default comparator has no promised order - // for a mixed ordinary-object and Temporal domain. - for (const first of [genuine, shadowedTag]) { - const second = first === genuine ? shadowedTag : genuine - const index = new IndexType(1, new PropRef([`when`])) - index.add(`first`, { when: first }) - index.add(`second`, { when: second }) - expect(index.equalityLookup(probe)).toEqual(new Set([`first`, `second`])) - index.remove(`first`, { when: first }) - expect(index.equalityLookup(probe)).toEqual(new Set([`second`])) - } - } - - const collection = createCollection<{ id: string; when: unknown }, string>({ - id: `temporal-plain-date-brand`, - getKey: (row) => row.id, - autoIndex: `off`, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const { id, when } of brandSamples) { - write({ type: `insert`, value: { id, when } }) - } - commit() - markReady() - }, - }, - }) - await withOracleCleanup(async () => { - await collection.stateWhenReady() - const keysFor = (probeValue: unknown) => { - const predicate = new Func(`eq`, [ - new PropRef([`when`]), - new Value(probeValue), - ]) - return new Set( - collection - .currentStateAsChanges({ where: predicate }) - ?.map(({ value }) => value.id), - ) - } - expect(keysFor(probe)).toEqual( - new Set([`genuine`, `shadowed-tag`, `shadowed-text`]), - ) - // An impostor remains an ordinary object: querying by its own reference - // finds it, while a distinct object with the same display cannot. - const borrowedCopy = Object.assign( - Object.create(Temporal.PlainDate.prototype) as object, - { toString: () => `2024-01-01` }, - ) - for (const [impostor, id, copy] of [ - [fake, `forged-tag`, { ...fake }], - [borrowed, `borrowed-prototype`, borrowedCopy], - ] as const) { - expect(keysFor(impostor)).toEqual(new Set([id])) - expect(keysFor(copy)).toEqual(new Set()) - } - }, [() => collection.cleanup()]) -}) diff --git a/packages/db/tests/temporal-index-order-oracle.test.ts b/packages/db/tests/temporal-index-order-oracle.test.ts index f62c4563c7..77094500f4 100644 --- a/packages/db/tests/temporal-index-order-oracle.test.ts +++ b/packages/db/tests/temporal-index-order-oracle.test.ts @@ -25,11 +25,16 @@ import { orderedEntriesArray, withOracleCleanup } from './utils.js' * values share one query-equality class. An opposite callback distinguishes * callback-result order from plain display-text order. They check add/build, * exact lookup, range and cursor boundaries, representative removal, and - * updates that change callback order without changing query equality. Fixed add/build - * histories also shadow a genuine value's display tag; the value cursor must - * still cross its native instant position. These checks do not claim arbitrary + * updates that change callback order without changing query equality. These + * checks do not claim arbitrary * Temporal kinds, time-zone transitions, longer histories, * public/provider cursor continuation, or live-query publication. + * The separate admission lane below protects default equality indexes for + * genuine Temporal values that lack a public order. It does not assign those + * values a public `orderBy` result: changelog #1519 requires `orderBy` to + * reject PlainMonthDay and mixed Temporal kinds. Its admission and exact + * membership authority is the public createIndex contract and #1519's + * unchanged Temporal `eq` relation; #1370 established Temporal index keys. */ type Row = { id: string; when: Temporal.ZonedDateTime } @@ -293,28 +298,6 @@ describe.each(indexTypes)( }, ) - it.each([`add`, `build`] as const)( - `keeps instant order when a genuine value shadows its display tag after %s`, - (construction) => { - const rows = rowsFromFixtures().slice(0, 3) - Object.defineProperty(rows[0]!.when, Symbol.toStringTag, { - value: `ordinary`, - }) - const index = new IndexType(1, new PropRef([`when`])) - if (construction === `add`) { - for (const row of rows) index.add(row.id, row) - } else { - index.build(rows.map((row): [string, Row] => [row.id, row])) - } - - // The source fixture supplies the instant labels. An instance display - // property cannot move its row out of the shared earliest position. - expectOrderedPrefix(index.takeFromStart(3), 3, fixtures.slice(0, 3)) - expectOrderedPrefix(index.takeFromStart(1), 1, fixtures.slice(0, 3)) - expect(index.take(1, rows[0]!.when)).toEqual([`a-one-nanosecond-later`]) - }, - ) - it(`retains a string-compatible custom comparator across negative instants`, () => { const [minusTen, minusHundred] = rowsFromFixtures(negativeFixtures) const index = new IndexType(1, new PropRef([`when`]), undefined, { @@ -575,3 +558,241 @@ describe.each(indexTypes)( }) }, ) + +type AdmissionRow = { id: string; when: unknown; equalityClass: string } +type AdmissionProbe = { when: unknown; equalityClass: string } + +// These are valid values from one Temporal implementation. The model's class +// labels come from constructor fields and the native equality distinction, +// independent of index normalization or the index's internal sort order. +const admissionFixtures = [ + { + name: `Duration with calendar units`, + orderErrorType: RangeError, + orderErrorMessage: `Missing relativeTo`, + rows: [ + { + id: `a`, + when: Temporal.Duration.from({ months: 1 }), + equalityClass: `one-month`, + }, + { + id: `b`, + when: Temporal.Duration.from({ months: 2 }), + equalityClass: `two-months`, + }, + { + id: `c`, + when: Temporal.Duration.from({ hours: 1 }), + equalityClass: `one-hour`, + }, + ], + probes: [ + { + when: Temporal.Duration.from({ months: 1 }), + equalityClass: `one-month`, + }, + { + when: Temporal.Duration.from({ months: 2 }), + equalityClass: `two-months`, + }, + { when: Temporal.Duration.from({ hours: 1 }), equalityClass: `one-hour` }, + ], + }, + { + name: `PlainMonthDay with distinct reference years`, + orderErrorType: TypeError, + orderErrorMessage: `Temporal.PlainMonthDay has no defined ordering`, + rows: [ + { + id: `a`, + when: new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972), + equalityClass: `1972`, + }, + { + id: `b`, + when: new Temporal.PlainMonthDay(2, 29, `iso8601`, 2004), + equalityClass: `2004`, + }, + { + id: `c`, + when: new Temporal.PlainMonthDay(3, 1, `iso8601`, 1972), + equalityClass: `march`, + }, + ], + probes: [ + { + when: new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972), + equalityClass: `1972`, + }, + { + when: new Temporal.PlainMonthDay(2, 29, `iso8601`, 2004), + equalityClass: `2004`, + }, + { + when: new Temporal.PlainMonthDay(3, 1, `iso8601`, 1972), + equalityClass: `march`, + }, + ], + }, + { + name: `mixed Temporal kinds`, + orderErrorType: TypeError, + orderErrorMessage: `Cannot order Temporal values of different types`, + rows: [ + { + id: `a`, + when: Temporal.Instant.from(`2024-01-01T00:00:00Z`), + equalityClass: `instant-one`, + }, + { + id: `b`, + when: Temporal.PlainDate.from(`2024-01-02`), + equalityClass: `date`, + }, + { + id: `c`, + when: Temporal.Instant.from(`2024-01-03T00:00:00Z`), + equalityClass: `instant-two`, + }, + ], + probes: [ + { + when: Temporal.Instant.from(`2024-01-01T00:00:00Z`), + equalityClass: `instant-one`, + }, + { when: Temporal.PlainDate.from(`2024-01-02`), equalityClass: `date` }, + { + when: Temporal.Instant.from(`2024-01-03T00:00:00Z`), + equalityClass: `instant-two`, + }, + ], + }, +] satisfies Array<{ + name: string + orderErrorType: typeof Error + orderErrorMessage: string + rows: Array + probes: Array +}> + +describe.each(indexTypes)(`%s valid Temporal admission`, (_name, IndexType) => { + it.each(admissionFixtures)( + `retains exact membership across add/build order for $name`, + ({ rows, probes }) => { + const expectedKeys = new Set(rows.map(({ id }) => id)) + // Admission order is not a public order contract for these values. + // Every construction must retain every key in its internal traversal. + for (const construction of [`forward`, `reverse`, `build`] as const) { + const index = new IndexType(1, new PropRef([`when`])) + const admission = + construction === `reverse` ? [...rows].reverse() : rows + if (construction === `build`) { + index.build( + admission.map((row): [string, AdmissionRow] => [row.id, row]), + ) + } else { + for (const row of admission) index.add(row.id, row) + } + for (const row of rows) { + expect(index.equalityLookup(row.when)).toEqual(new Set([row.id])) + } + for (const probe of probes) { + const matchingKeys = rows + .filter((row) => row.equalityClass === probe.equalityClass) + .map((row) => row.id) + expect(index.equalityLookup(probe.when)).toEqual( + new Set(matchingKeys), + ) + } + const ordered = index.takeFromStart(rows.length) + expect(ordered).toHaveLength(rows.length) + expect(new Set(ordered).size).toBe(ordered.length) + expect(new Set(ordered)).toEqual(expectedKeys) + } + }, + ) + + it.each(admissionFixtures)( + `publishes each valid $name row with its exact index membership`, + async ({ rows, orderErrorType, orderErrorMessage }) => { + let commitRow!: (row: AdmissionRow) => void + const collection = createCollection({ + id: `temporal-admission-${_name}-${rows[0]!.equalityClass}`, + getKey: (row) => row.id, + autoIndex: `off`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + commitRow = (row) => { + begin() + write({ type: `insert`, value: row }) + commit() + } + markReady() + }, + }, + }) + + await withOracleCleanup(async () => { + await collection.stateWhenReady() + const index = collection.createIndex((row) => row.when, { + indexType: IndexType, + }) + const observed: Array<{ + key: string + stored: boolean + exact: Array + }> = [] + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + observed.push({ + key: change.key, + stored: collection.get(change.key)?.id === change.key, + exact: [...index.equalityLookup(change.value.when)], + }) + } + }, + { includeInitialState: false }, + ) + try { + // Each synchronous commit is a public checkpoint. The second valid + // value must neither crash the Collection nor leave a stored row + // invisible to the index and subscriber. + commitRow(rows[0]!) + commitRow(rows[1]!) + expect(collection.status).toBe(`ready`) + expect([...collection.keys()]).toEqual([`a`, `b`]) + expect(observed).toEqual([ + { key: `a`, stored: true, exact: [`a`] }, + { key: `b`, stored: true, exact: [`b`] }, + ]) + // An equality index must not turn its private storage order into a + // public orderBy result. Changelog #1519 delegates to Temporal's + // static comparison, which rejects these pairs without relativeTo + // or a common ordered kind. Limit 1 catches an index that returns + // before comparing the second row. + let orderFailure: unknown + try { + collection.currentStateAsChanges({ + orderBy: [ + { + expression: new PropRef([`when`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + limit: 1, + }) + } catch (error) { + orderFailure = error + } + expect(orderFailure).toBeInstanceOf(orderErrorType) + expect((orderFailure as Error).message).toContain(orderErrorMessage) + } finally { + subscription.unsubscribe() + } + }, [() => collection.cleanup()]) + }, + ) +}) diff --git a/packages/db/tests/utils-oracle.property.test.ts b/packages/db/tests/utils-oracle.property.test.ts index 5febc6c4a5..c578ad7d87 100644 --- a/packages/db/tests/utils-oracle.property.test.ts +++ b/packages/db/tests/utils-oracle.property.test.ts @@ -9,6 +9,10 @@ import { deepEquals, equalPersistedSnapshotValues } from '../src/utils' * established examples in `utils.test.ts`, promises change-event equality for * its supported value classes. This oracle checks bounded equality and * inequality laws for those classes, not general graph isomorphism. + * The native Date timestamp rule in `docs/guides/live-queries.md` applies + * across realms. The Collection change-message mirror contract in + * `change-event-history-oracle.test.ts` requires settled events to reconstruct + * public rows. The Date law below checks the resulting equality boundary. * * Independent value constructors exercise reflexivity, symmetry, copied * structure, and changed-leaf inequality for primitives, arrays, records, @@ -19,6 +23,15 @@ import { deepEquals, equalPersistedSnapshotValues } from '../src/utils' * A native typed-array length shadow cannot change how many intrinsic * elements participate in change-event equality. A supplied changed byte * must remain unequal even when both views report an own `length` of zero. + * Ordinary native Dates compare by time across JavaScript realms. One fixed + * pair with the same own string label tests a changed-time fallback. It does + * not define equality for Dates with different own labels or replaced methods. + * Ordinary Temporal.Duration values compare by their ten supplied fields, + * including opposite signs and carried units that represent equal lengths. + * `packages/db/CHANGELOG.md` says equivalent Duration forms remain unequal + * under `eq`. This is the exact-field equality already used by query + * normalization; the direct boundary here does not establish every Collection + * publication path. * Keyless DataViews have hidden state but no typed-array elements. Distinct * views from one foreign realm remain unequal by the keyless-instance law; * pairing one with a typed array must return false without reading a typed- @@ -101,6 +114,68 @@ const arbitraryFiniteDate = fc .integer({ min: -1_000_000_000_000, max: 2_000_000_000_000 }) .map((time) => new Date(time)) +type DateRealm = `local` | `foreign` +type NativeDatePair = { + leftTime: number | null + rightTime: number | null + leftRealm: DateRealm + rightRealm: DateRealm + ownLabel?: boolean +} + +// `null` is a model-only invalid-time marker. The supplied times, rather than +// a production Date classifier, determine equality. Both foreign operands are +// constructed in one realm so a labeled unequal pair can reach the historical +// same-prototype fallback that hid its different native timestamps. +function nativeDatePair(spec: NativeDatePair): [Date, Date] { + const leftTime = spec.leftTime ?? Number.NaN + const rightTime = spec.rightTime ?? Number.NaN + const [foreignLeft, foreignRight] = runInNewContext( + `[new Date(leftTime), new Date(rightTime)]`, + { leftTime, rightTime }, + ) as [Date, Date] + const left = spec.leftRealm === `foreign` ? foreignLeft : new Date(leftTime) + const right = + spec.rightRealm === `foreign` ? foreignRight : new Date(rightTime) + if (spec.ownLabel) { + for (const date of [left, right]) { + Object.defineProperty(date, `label`, { + value: `same`, + enumerable: true, + }) + } + } + return [left, right] +} + +function assertNativeDateEquality( + spec: NativeDatePair, + equal: (left: unknown, right: unknown) => boolean = deepEquals, +): void { + const [left, right] = nativeDatePair(spec) + const expected = Object.is(spec.leftTime, spec.rightTime) + expect(equal(left, right), `forward ${JSON.stringify(spec)}`).toBe(expected) + expect(equal(right, left), `reverse ${JSON.stringify(spec)}`).toBe(expected) +} + +const arbitraryNativeDatePair = fc + .record({ + leftTime: fc.oneof( + fc.integer({ min: -1_000_000_000_000, max: 1_000_000_000_000 }), + fc.constant(null), + ), + relation: fc.constantFrom(`equal` as const, `changed` as const), + leftRealm: fc.constantFrom(`local` as const, `foreign` as const), + rightRealm: fc.constantFrom(`local` as const, `foreign` as const), + }) + .map(({ leftTime, relation, leftRealm, rightRealm }) => ({ + leftTime, + rightTime: + relation === `equal` ? leftTime : leftTime === null ? 1 : leftTime + 1, + leftRealm, + rightRealm, + })) + const arbitraryRegExp = fc .tuple(fc.string(), fc.constantFrom(``, `g`, `i`, `gi`, `m`, `gim`)) .map(([source, flags]) => { @@ -132,6 +207,92 @@ const arbitraryTemporalDuration = fc Temporal.Duration.from({ hours, minutes, seconds }), ) +// The supplied vector is the independent model: equality means every named +// Duration field agrees. It does not consult display text or the production +// normalizer. One shared sign keeps every generated Duration valid. +const exactDurationFields = [ + `years`, + `months`, + `weeks`, + `days`, + `hours`, + `minutes`, + `seconds`, + `milliseconds`, + `microseconds`, + `nanoseconds`, +] as const +type ExactDurationField = (typeof exactDurationFields)[number] +type ExactDurationSpec = Record + +// These are the adjacent unit conversions with a fixed factor. Month-to-week +// conversion has no fixed factor, so it is deliberately absent. +const durationCarryPairs = [ + [`years`, `months`, 12], + [`weeks`, `days`, 7], + [`days`, `hours`, 24], + [`hours`, `minutes`, 60], + [`minutes`, `seconds`, 60], + [`seconds`, `milliseconds`, 1000], + [`milliseconds`, `microseconds`, 1000], + [`microseconds`, `nanoseconds`, 1000], +] as const + +const arbitraryExactDurationMagnitudes: fc.Arbitrary = + fc.record({ + years: fc.integer({ min: 0, max: 3 }), + months: fc.integer({ min: 0, max: 3 }), + weeks: fc.integer({ min: 0, max: 3 }), + days: fc.integer({ min: 0, max: 3 }), + hours: fc.integer({ min: 0, max: 3 }), + minutes: fc.integer({ min: 0, max: 3 }), + seconds: fc.integer({ min: 0, max: 3 }), + milliseconds: fc.integer({ min: 0, max: 3 }), + microseconds: fc.integer({ min: 0, max: 3 }), + nanoseconds: fc.integer({ min: 0, max: 3 }), + }) + +function signedDurationSpec( + magnitudes: ExactDurationSpec, + sign: -1 | 1, +): ExactDurationSpec { + return Object.fromEntries( + exactDurationFields.map((field) => [ + field, + magnitudes[field] === 0 ? 0 : magnitudes[field] * sign, + ]), + ) as ExactDurationSpec +} + +function assertExactDurationEquality( + leftSpec: ExactDurationSpec, + rightSpec: ExactDurationSpec, + equal: (left: unknown, right: unknown) => boolean = deepEquals, +): void { + const left = Temporal.Duration.from(leftSpec) + const right = Temporal.Duration.from(rightSpec) + const expected = exactDurationFields.every( + (field) => leftSpec[field] === rightSpec[field], + ) + expect( + equal(left, right), + `forward ${JSON.stringify([leftSpec, rightSpec])}`, + ).toBe(expected) + expect( + equal(right, left), + `reverse ${JSON.stringify([leftSpec, rightSpec])}`, + ).toBe(expected) +} + +function singleDurationField( + field: ExactDurationField, + value: number, +): ExactDurationSpec { + return Object.fromEntries( + exactDurationFields.map((name) => [name, name === field ? value : 0]), + ) as ExactDurationSpec +} + // Same-type value arbitraries for testing equivalence properties // Mixed Date/Temporal inequality has its own generated and fixed witnesses below. const arbitrarySameTypePrimitive = fc.oneof( @@ -901,6 +1062,71 @@ describe(`deepEquals property-based tests`, () => { ) }) + // Legal pairs cross both realm placements and equal or changed valid and + // invalid times. The direct deepEquals result is the observation cut. A + // local-only classifier fails an equal mixed-realm pair. Own properties, + // non-Date objects, and changed prototypes are outside this grammar. + it(`reconstructs native Date realm and time margins`, () => { + for (const leftRealm of [`local`, `foreign`] as const) { + for (const rightRealm of [`local`, `foreign`] as const) { + for (const leftTime of [1, null]) { + for (const rightTime of [leftTime, leftTime === null ? 1 : 2]) { + assertNativeDateEquality({ + leftTime, + rightTime, + leftRealm, + rightRealm, + }) + } + } + } + } + }) + + it(`rejects a local-only Date classifier and a changed-time fallback`, () => { + const mixedEqual: NativeDatePair = { + leftTime: 1, + rightTime: 1, + leftRealm: `local`, + rightRealm: `foreign`, + ownLabel: false, + } + const labeledChanged: NativeDatePair = { + leftTime: 1, + rightTime: 2, + leftRealm: `foreign`, + rightRealm: `foreign`, + ownLabel: true, + } + expect(() => + assertNativeDateEquality( + mixedEqual, + (left, right) => + left instanceof Date && + right instanceof Date && + Object.is(left.getTime(), right.getTime()), + ), + ).toThrow(`forward`) + expect(() => + assertNativeDateEquality(labeledChanged, () => true), + ).toThrow(`forward`) + // This fixed diagnostic preserves the missed-event counterexample + // without extending the generated grammar to modified Date objects. + assertNativeDateEquality(labeledChanged) + }) + + it(`keeps a foreign Date distinct from an equal-keyed plain record`, () => { + const date = runInNewContext(`new Date(1)`) as Date + Object.defineProperty(date, `label`, { + value: `same`, + enumerable: true, + }) + const record = { label: `same` } + expect(Date.prototype.getTime.call(date)).toBe(1) + expect(deepEquals(date, record)).toBe(false) + expect(deepEquals(record, date)).toBe(false) + }) + utilsProperty([arbitraryTemporalPlainDate])( `Temporal.PlainDate with same values are equal`, (date) => { @@ -1280,6 +1506,139 @@ describe(`deepEquals property-based tests`, () => { (regex) => assertDistinctTypes(regex, regex.source, deepEquals), ) }) + + // Keep this generated property after every existing ordinal registration so + // saved UTILS_ORACLE_PROPERTY seed/path replays keep their original targets. + describe(`ordinary native Date realm boundary`, () => { + utilsProperty([arbitraryNativeDatePair])( + `ordinary native Date equality follows time across realms`, + assertNativeDateEquality, + ) + }) + + // Keep both registrations after every existing ordinal replay selector. + describe(`exact Temporal.Duration field equality`, () => { + // Fresh equal vectors and each single-field change reconstruct the whole + // ten-field grammar. Generated magnitudes range from 0 to 3 under one + // shared positive or negative sign; a selected field changes away from + // zero. Equal and changed relations, and both signs, contribute distinct + // outcomes. Mixed-sign vectors are invalid and excluded. The direct + // deepEquals result is the observation cut. + it(`accepts fresh copies and distinguishes each of ten fields`, () => { + for (const field of exactDurationFields) { + const zero = singleDurationField(field, 0) + assertExactDurationEquality(zero, { ...zero }) + for (const sign of [-1, 1]) { + assertExactDurationEquality(zero, singleDurationField(field, sign)) + } + } + }) + + it(`distinguishes opposite signs and every fixed-factor unit carry`, () => { + const positive = singleDurationField(`seconds`, 1) + const negative = singleDurationField(`seconds`, -1) + assertExactDurationEquality(positive, negative) + expect(() => + assertExactDurationEquality(positive, negative, (a, b) => + exactDurationFields.every( + (field) => + Math.abs((a as Temporal.Duration)[field]) === + Math.abs((b as Temporal.Duration)[field]), + ), + ), + ).toThrow(`forward`) + + for (const [larger, smaller, factor] of durationCarryPairs) { + for (const sign of [-1, 1]) { + const leftSpec = singleDurationField(larger, sign) + const rightSpec = singleDurationField(smaller, sign * factor) + assertExactDurationEquality(leftSpec, rightSpec) + // A comparator that balances just this adjacent pair is wrong even + // when the two Durations represent the same elapsed length. + expect(() => + assertExactDurationEquality(leftSpec, rightSpec, (a, b) => { + const left = a as Temporal.Duration + const right = b as Temporal.Duration + return ( + exactDurationFields.every( + (field) => + field === larger || + field === smaller || + left[field] === right[field], + ) && + left[larger] * factor + left[smaller] === + right[larger] * factor + right[smaller] + ) + }), + ).toThrow(`forward`) + } + } + }) + + it(`rejects display equality for all subsecond representations`, () => { + for (const [larger, smaller] of [ + [`seconds`, `milliseconds`], + [`milliseconds`, `microseconds`], + [`microseconds`, `nanoseconds`], + ] as const) { + for (const sign of [-1, 1]) { + const leftSpec = singleDurationField(larger, sign) + const rightSpec = singleDurationField(smaller, sign * 1000) + const left = Temporal.Duration.from(leftSpec) + const right = Temporal.Duration.from(rightSpec) + expect(left.toString()).toBe(right.toString()) + assertExactDurationEquality(leftSpec, rightSpec) + // A display-only comparator is a plausible wrong design. The model + // rejects it at the same direct-result checkpoint as production. + expect(() => + assertExactDurationEquality( + leftSpec, + rightSpec, + (a, b) => String(a) === String(b), + ), + ).toThrow(`forward`) + } + } + }) + + utilsProperty([ + arbitraryExactDurationMagnitudes, + fc.constantFrom(-1 as const, 1 as const), + fc.constantFrom(...exactDurationFields), + fc.boolean(), + ])( + `fresh Duration copies agree and any changed field differs`, + (magnitudes, sign, field, changed) => { + const base = signedDurationSpec(magnitudes, sign) + const next = { ...base, [field]: base[field] + (changed ? sign : 0) } + assertExactDurationEquality(base, next) + }, + ) + + // These three unit-pair axes are the display-collision margin. The + // magnitude ranges 1–100 with either shared sign; omitting either sign or + // an axis leaves a valid subsecond representation outside this claim. + utilsProperty([ + fc.integer({ min: 1, max: 100 }), + fc.constantFrom(-1 as const, 1 as const), + fc.constantFrom( + [`seconds`, `milliseconds`] as const, + [`milliseconds`, `microseconds`] as const, + [`microseconds`, `nanoseconds`] as const, + ), + ])( + `equal-display Duration field vectors remain unequal`, + (n, sign, pair) => { + const [larger, smaller] = pair + const leftSpec = singleDurationField(larger, sign * n) + const rightSpec = singleDurationField(smaller, sign * 1000 * n) + expect(Temporal.Duration.from(leftSpec).toString()).toBe( + Temporal.Duration.from(rightSpec).toString(), + ) + assertExactDurationEquality(leftSpec, rightSpec) + }, + ) + }) }) // Persisted-snapshot equality decides whether a reload can leave the public diff --git a/packages/db/tests/utils.ts b/packages/db/tests/utils.ts index 2927bc9bd9..ae9070b2e7 100644 --- a/packages/db/tests/utils.ts +++ b/packages/db/tests/utils.ts @@ -598,7 +598,7 @@ export function withExpectedRejection( }).finally(() => restoreListeners()) } -type IndexInternals = { indexedKeys: Set } & ( +type IndexInternals = { indexedKeys: Map } & ( | { sortedValues: Array valueMap: Map> @@ -635,7 +635,7 @@ function indexInternals(index: object): [IndexInternals, boolean] { /** Test inspection of an index's tracked keys. */ export function indexedKeysSet(index: object): Set { - return indexInternals(index)[0].indexedKeys + return new Set(indexInternals(index)[0].indexedKeys.keys()) } /** Test inspection of an index's value buckets keyed by indexed value. */ From f3e67cb56a35c85ddaae71e293150b724436e7fe Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 08:57:45 -0600 Subject: [PATCH 31/46] docs: record exact-head oracle review and guarantee costs --- docs/contributing/oracle-coverage.md | 4 ++ .../pr-2116-correctness-and-guarantees.md | 72 +++++++++++++++++++ 2 files changed, 76 insertions(+) create mode 100644 docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 8f89487234..f99bdc74a6 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -657,6 +657,10 @@ identity, live-query publication, and provider paths remain unproved. Direct D2 `groupBy` has its separately approved kind-and-string identity policy. The earlier spoofed-prototype finding is outside the valid Temporal input domain. It needs a separate product decision before it becomes a guarantee. +The [PR #2116 exact-commit review](oracle-reviews/pr-2116-correctness-and-guarantees.md) +records R16/R17/R25/R29 laws, hostile controls, remaining cells, and the +production-weight cost of proposed guarantees at code commit +`4ed5242dbef0e7dbf87b31ca97c2fd2bc7901cad`. The [custom index order selection oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/custom-index-order-selection-oracle.test.ts) and its [review record](oracle-reviews/custom-index-order-selection.md) check diff --git a/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md b/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md new file mode 100644 index 0000000000..666c8c8680 --- /dev/null +++ b/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md @@ -0,0 +1,72 @@ +# PR #2116: correctness review and guarantee cost + +Reviewed executable commit: `4ed5242dbef0e7dbf87b31ca97c2fd2bc7901cad`. +Merge base: `b112d888f0c1f1ef4245ae48e8b071e8fad916b0`. This documentation +follows the reviewed code in a separate commit. The external review has 28 +source-order items; R29 is a sibling found while challenging the repaired +Temporal law. The task-local loss ledger retains each raw item and its evidence. + +## Laws, experiments, and limits + +| Item and authority | Independent judgment and reached checkpoint | Decisive failure and repaired evidence | Boundary | +| --- | --- | --- | --- | +| R16: a valid row admitted by a Collection cannot remain stored while its index and subscribers omit it. Existing Collection publication and changelog #1519 also require public `orderBy` to reject unsupported Temporal order. | Supplied genuine Duration-with-months, PlainMonthDay, and mixed-kind rows determine membership. BasicIndex/BTreeIndex add, reverse add, and build compare complete exact and ordered key sets. Two sequential sync commits compare the row, exact index membership, and callback batch; public limit-one `orderBy` must throw. | On the earlier code, the second commit threw and left two stored rows with one indexed/published row in six index/value combinations. A selected pre-repair oracle run had eight failed cases, including unexpected index-order throws. Restoring the public comparator at this head recreated partial publication for both index types; advertising private order as public returned a row instead of throwing. The final 12 focused admission checks pass. | One polyfill; three unsupported-order fixture classes; finite histories. No malformed Temporal promise, arbitrary callback, provider cursor, or long live-query history. | +| R17: after a returned direct index action, exact and ordered reads describe the same current keys. The index API and prior behavior authorize equal-time Date removal; a different callback position for a substituted old item has weaker authority. | A source Map retains each key's supplied timestamp or callback position. After add, remove, update, build, and re-add, both indexes are checked through key count, exact lookup, ordered take, and bounded range reads. | The old BTree cache left a detached ordered bucket after equal-distinct removal and object reuse. A nearby wrong-position removal left a ghost ordered key in both index types while exact lookup/count looked correct. The new history rejects both at take/range checkpoints. All 444 owner tests pass; 100-seed mixed-action probes per index found no mismatch. | Valid Dates, stable callbacks, unique-key add, direct index path. Arbitrary objects/comparators and Collection publication are outside this claim. | +| R25: native Dates with the same time are deep-equal across ordinary realms; a changed time must publish a Collection update so a subscriber mirrors the stored row. The live-query guide and Collection mirror law supply authority. | Supplied valid or invalid times and local or Node-vm constructors determine expected equality. The direct Date owner crosses realm and time margins; a ready unindexed sync Collection receiver compares exact batches, callback row, mirror, and settled row. | The old local `instanceof Date` branch missed changed foreign Dates and rejected equal mixed-realm Dates. Local-only and changed-time-ignored wrong answers fail direct checks; original public event traces failed before the repair. The final direct/receiving owners are included in the 552-test combined run below. | Ordinary native Dates in Node realms; seven fixed public sync cases. User-modified Date display tags have only fixed diagnostics, not a broad grammar. Indexed/optimistic publication and other hosts remain outside this witness. | +| R29: Duration query equality preserves its ten fields, even when two representations have the same display. Changelog #1519 states this for `PT60M` and `PT1H`; changed fields must publish enough for the Collection mirror. | Supplied ten-field vectors, both signs, opposite signs, eight fixed-factor unit carries, and three subsecond display collisions predict direct equality without production normalization. Fixed and generated direct properties compare `deepEquals`. Nine ready unindexed sync replacement histories compare subscriber batches, callback row, mirror, and settled row. | Display-only equality missed both directions of seconds-to-milliseconds changes. Before repair, five targeted cases failed. After adding negative controls, a display-only mutant failed seven targeted cases. An absolute-field mutant failed the opposite-sign check; a balancing mutant failed direct hour/minute and public change-message checks. Both owners pass 552 tests; saved replays #41 and #42 pass. | Direct magnitudes 0–3, display-collision magnitudes 1–100, one polyfill; representative public replacement histories. Arbitrary ten-field Collection histories, indexed/optimistic publication, and mixed Temporal implementations remain open. | + +The R25/R29 combined owner run has 552 tests. Six related owners pass 783 tests, +with no test type errors. A separate exact-head sweep compared 12,720 valid +Temporal pairs across all eight kinds with native equality or supplied Duration +fields and found no mismatch. This finite sweep does not extend the histories +or public paths above. The full DB suite passes 13,365 tests in 284 files; +DB-IVM passes 702 in 46 files; built-package tests pass 291 in five files. +Source and test TypeScript checks, ESLint with no errors, Prettier, and both +package builds pass at this code commit. + +## ORC-001 through ORC-014 + +| Requirement | Outcome at the reviewed code commit | +| --- | --- | +| ORC-001 contract | The opening law comments identify Collection publication, index refinement, native Date time, and #1519 Duration/Temporal behavior. R17's different-position substituted old item lacks explicit API authority and is cataloged below rather than credited to the general law. | +| ORC-002 independent judgment | Supplied rows, epochs, callback positions, times, and Duration field vectors determine expected results. No oracle imports the production comparator, normalizer, index cache, or `deepEquals` for its expected answer. | +| ORC-003 literate responsibilities | Each changed owner states law and limits before mechanics, then places the model, finite or generated grammar, real driver, public observation, and comparison beside their code. | +| ORC-004 grammar controls | R16 and the new R17 Date lanes are fixed finite histories. R25's generated pair crosses local/foreign placement and valid/invalid time; R29 crosses ten fields, two signs, equal/changed vectors, and three display-collision unit pairs. Each axis has a fixed reconstruction or mutant control. Mixed-sign Duration input is invalid and excluded. | +| ORC-005 public observation | Real indexes and Collections are driven. R16 checks callback-time membership and public rejection; R17 checks exact plus ordered methods after every direct action; R25/R29 check direct equality and exact Collection batches, callback row, mirror, and settled row. Complete keys expose duplicates and omissions. | +| ORC-006 calibration | The original R16, R17, R25, and R29 paths failed at their named checks. R16's unexpected legal-operation throw was itself the public violation; the ordered-read and equality faults reached assertion mismatches. Public-comparator, stale-cache, local-only Date, display-only Duration, absolute-field, and balancing wrong designs are rejected at their intended checkpoints. None was a timeout or setup error. | +| ORC-007 replay | Fixed lanes do not trigger generated replay. Existing R17 numeric properties keep their campaigns. The new R25/R29 utility properties run fixed and random campaigns with the existing direct seed/path replay interface; selectors #40–#42 preserve older #1–#39 registrations. Replays #41 and #42 passed at seed 20260911 and empty path. | +| ORC-008 model state | R17's source Map retains each key's epoch or callback position because a later legal removal or ordered read distinguishes owners with the same key count. The other new judgments recompute from supplied immutable facts. | +| ORC-009 vocabulary | Exact equality key, stored order operand, source row, change batch, and public ordered read retain their glossary/architecture meanings. The private Temporal index order is explicitly separate from public `orderBy`. | +| ORC-010 failure fidelity | The fixed direct checks have no shrinking or external cleanup. Collection receivers use the existing cleanup harness; reported RED cases reached their intended assertions before cleanup. Saved replay selectors preserve the generated target. | +| ORC-011 second formulation | R16 crosses direct index membership with public Collection publication/rejection. R25/R29 cross direct value equality with public subscriber mirrors. R17 crosses exact lookup with ordered reads that the old code separated. No real-provider formulation is claimed. | +| ORC-012 review evidence | This record binds the outcome, controls, and limits to commit `4ed5242dbef0e7dbf87b31ca97c2fd2bc7901cad`. Four hostile reviewers rechecked the committed R16, R17, R25/narrowed Temporal, and R29 owners; none found a blocker in the stated cells. | +| ORC-013 neighboring witness | R16 separates private index order from public rejection; R17 separates same-position from different-position old items and exact from ordered keys; R25 separates equal mixed-realm from changed-time Dates; R29 separates same display, opposite sign, and adjacent-unit carry from exact field equality. | +| ORC-014 provider handoff | These claims stop at direct indexes or controlled ready in-memory sync Collections. No external-provider or browser-host handoff is inferred. R21's broader IANA alias claim needs a real supported host witness before it can become a contract finding. | + +## Production weight and guarantees to decide + +Against the merge base, production TypeScript is **+876 net lines**. The +architecture document is **+56 net lines**, counted separately. A minified +build of every ESM module is **+4,647 gzip bytes** in aggregate +(205,407 to 210,054). A minified browser bundle of the public DB entry is +**+2,856 gzip bytes** (111,875 to 114,731) under the same esbuild command +at both revisions. These are two different footprint views, not additive +costs or a prediction for every consumer. The external review's +974-line +figure mixed production code and architecture prose at an earlier head. + +| Guarantee or work cost | Authority and value | Decision surface | +| --- | --- | --- | +| A captured mutable `inArray` reads current members and refreshes the whole retained query on the next source event. | The maintainer explicitly chose both rules. The receiving oracles cover WHERE, joins, selections, groups, aggregates, HAVING, order, nested queries, and selected on-demand/include paths. | Largest cost: the review estimated about 427 source lines. Full-source acquisition disables index/subset pushdown, keyed lazy demand, and pooled fast paths for this query shape. Changing this requires a contract decision; freezing a list already restores the optimizations. | +| A custom index callback orders **each stored row**, even when query equality groups two values together. | The maintainer explicitly chose this. | Both indexes retain per-row order operands and may sort/refresh more rows. Callback arguments now expose raw Date/Uint8Array values instead of the former timestamp/encoded-byte representation. That compatibility change follows the per-row rule and should be called out in release notes. | +| Default native-byte order is unsigned lexicographic order. | This PR proposes the policy; no separate maintainer decision was recorded. | It requires scan/index agreement and comparator machinery. Keep or remove as a product decision, rather than treating the old encoded-string order as authoritative. | +| A direct index removal accepts an equal-time but different old Date whose custom callback position differs from the stored Date. | The general refinement law supports normal remove/re-add, but the index API does not document this substitution. | Added per-key stored-order state supports it. This is a candidate to narrow if the direct API may require the actual prior value. | +| A fresh, equal-field Duration replacement emits no update message. | Consistent with existing change-event suppression; the mirror law requires changed-field publication, not this silence. | The public R29 receiver asserts silence. It can be relaxed independently of exact field equality and changed-field publication. | +| Native Date display-tag modification and own byte length/byteLength shadows retain intrinsic value semantics. | These are valid JavaScript objects but user-modified built-ins, with weaker product authority than ordinary Dates and byte views. | Their checks and helpers add complexity. Treat them as separate compatibility decisions. Deliberately malformed, borrowed, forged, and shadowed Temporal values were removed from the claimed domain. | +| Repeated Temporal key/display work, per-row BasicIndex build sorting, and equal-row index refresh. | Deterministic work probes observed these costs; no work budget was promised. | Optimize only if a bounded workload justifies it. The old claim that *every* Temporal comparison calls `toString` is stale after the valid-value narrowing. | + +R21 is not closed as a Firefox defect. A mocked time-zone formatter preserving +aliases exposed sensitivity, but current ECMA-402 canonicalizes zone names +and no conforming real-host failure was found. The speculative alias scan +repair was removed. Reopen on a supported engine/version where two native-equal +zones produce different resolved names and a public scan or indexed equality +misses. From bd7881331b8b149d2131b315c5c19e8b910c01fb Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 09:08:28 -0600 Subject: [PATCH 32/46] docs(db): record merged oracle evidence and guarantee costs --- .../pr-2116-correctness-and-guarantees.md | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md b/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md index 666c8c8680..e97ebb05db 100644 --- a/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md +++ b/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md @@ -70,3 +70,38 @@ and no conforming real-host failure was found. The speculative alias scan repair was removed. Reopen on a supported engine/version where two native-equal zones produce different resolved names and a public scan or indexed equality misses. + +## Integration after merging main + +Merge commit `f42e816d9c0442448624ab418074fd5008ee14c7` combines the reviewed +repair with `origin/main` at `bbbf017ef5b3884515295a4225faacdfd814499e`. +Main added replay retirement and bounded ordered-window behavior. The only +textual conflict was this architecture document; its resolution keeps both the +captured-list full-source rule and main's finite-window rule. No R16, R17, R25, +or R29 oracle expectation was changed during the merge. + +The combined tree passes 13,381 DB source tests, 705 DB-IVM tests, and 291 DB +dist tests. The packed IndexedDB package check passes 19 tests. Five focused +Temporal/index/equality/publication/replay owners pass 874 tests, with no test +type errors. DB source and test TypeScript checks and both package builds pass. +The focused integration review found no new mismatch in ordinary ready sync +publication. Date or Duration replacement during truncate replay is not a +separately generated cross-product; replay histories use numeric rows and the +replay path calls the repaired equality function. This is a receiving limit, +not evidence of a known failing legal history. + +Against this newer `origin/main`, production TypeScript is **+878 net lines** +(1,388 added, 510 removed); architecture prose is **+56 net lines** separately. +Minified ESM modules total 206,777 to 211,426 gzip bytes for DB (**+4,649**) +and 24,350 to 24,369 for DB-IVM (**+19**). A minified browser bundle of the +public DB source entry is 112,595 to 115,464 gzip bytes (**+2,869**) under +the same esbuild command at both revisions. The main bundle was reproduced +byte for byte before comparing. These views overlap and must not be added. + +The guarantee choices in the table above still apply. The two largest behavior +costs, live captured arrays with whole-result refresh and per-row custom +callback order, are explicit maintainer decisions. Unsigned default byte order, +different-position old-Date removal, silence for equal-field Duration copies, +and intrinsic semantics for user-modified Date or byte built-ins remain separate +product choices. No size reduction is credited for dropping them without a +same-path oracle and a measured implementation change. From 57017ed2b12e45080866e9ae350399d81a43d24a Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 09:11:47 -0600 Subject: [PATCH 33/46] docs(db): distinguish optional guarantee scope --- .../pr-2116-correctness-and-guarantees.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md b/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md index e97ebb05db..cc7bc01ab9 100644 --- a/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md +++ b/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md @@ -59,9 +59,11 @@ figure mixed production code and architecture prose at an earlier head. | A captured mutable `inArray` reads current members and refreshes the whole retained query on the next source event. | The maintainer explicitly chose both rules. The receiving oracles cover WHERE, joins, selections, groups, aggregates, HAVING, order, nested queries, and selected on-demand/include paths. | Largest cost: the review estimated about 427 source lines. Full-source acquisition disables index/subset pushdown, keyed lazy demand, and pooled fast paths for this query shape. Changing this requires a contract decision; freezing a list already restores the optimizations. | | A custom index callback orders **each stored row**, even when query equality groups two values together. | The maintainer explicitly chose this. | Both indexes retain per-row order operands and may sort/refresh more rows. Callback arguments now expose raw Date/Uint8Array values instead of the former timestamp/encoded-byte representation. That compatibility change follows the per-row rule and should be called out in release notes. | | Default native-byte order is unsigned lexicographic order. | This PR proposes the policy; no separate maintainer decision was recorded. | It requires scan/index agreement and comparator machinery. Keep or remove as a product decision, rather than treating the old encoded-string order as authoritative. | +| Cross-realm native Dates and byte views, plus Date subclasses with overridden coercion, use intrinsic times or bytes. | The branch adds public and direct witnesses; ordinary local Dates and bytes already have value semantics. The wider host/subclass scope has no separate maintainer decision. | Removing this scope would revise several oracle laws and could let valid foreign values change a stored row without publishing it. Keep the R25 mirror repair as long as this scope is promised. | +| Mixed-kind default index order stays total, while public Temporal `orderBy` still rejects kinds without an order. | Valid mixed rows must be admitted without partially updating a Collection; the specific rank between unlike ordinary kinds is a policy chosen by this branch. | The kind separation and private Temporal order add comparator code. A narrower accepted input domain or another internal order would need an explicit contract change and the R16 atomic-publication witness. | | A direct index removal accepts an equal-time but different old Date whose custom callback position differs from the stored Date. | The general refinement law supports normal remove/re-add, but the index API does not document this substitution. | Added per-key stored-order state supports it. This is a candidate to narrow if the direct API may require the actual prior value. | | A fresh, equal-field Duration replacement emits no update message. | Consistent with existing change-event suppression; the mirror law requires changed-field publication, not this silence. | The public R29 receiver asserts silence. It can be relaxed independently of exact field equality and changed-field publication. | -| Native Date display-tag modification and own byte length/byteLength shadows retain intrinsic value semantics. | These are valid JavaScript objects but user-modified built-ins, with weaker product authority than ordinary Dates and byte views. | Their checks and helpers add complexity. Treat them as separate compatibility decisions. Deliberately malformed, borrowed, forged, and shadowed Temporal values were removed from the claimed domain. | +| User-modified Date display tags, byte-view length properties, and an array's own `some` method cannot redefine intrinsic value or `inArray` membership. | These are valid JavaScript objects but user-modified built-ins, with weaker product authority than ordinary Dates, byte views, and arrays. | Their checks and helpers add complexity. Treat them as separate compatibility decisions. Deliberately malformed, borrowed, forged, and shadowed Temporal values were removed from the claimed domain. | | Repeated Temporal key/display work, per-row BasicIndex build sorting, and equal-row index refresh. | Deterministic work probes observed these costs; no work budget was promised. | Optimize only if a bounded workload justifies it. The old claim that *every* Temporal comparison calls `toString` is stale after the valid-value narrowing. | R21 is not closed as a Firefox defect. A mocked time-zone formatter preserving @@ -101,7 +103,8 @@ byte for byte before comparing. These views overlap and must not be added. The guarantee choices in the table above still apply. The two largest behavior costs, live captured arrays with whole-result refresh and per-row custom callback order, are explicit maintainer decisions. Unsigned default byte order, -different-position old-Date removal, silence for equal-field Duration copies, -and intrinsic semantics for user-modified Date or byte built-ins remain separate -product choices. No size reduction is credited for dropping them without a -same-path oracle and a measured implementation change. +cross-realm/subclass scope, mixed-kind default order, different-position +old-Date removal, silence for equal-field Duration copies, and intrinsic +semantics for user-modified built-ins remain separate product choices. No size +reduction is credited for dropping them without a same-path oracle and a +measured implementation change. From 28f2092162d049765c90d8f109755ad51cb01904 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 10:09:23 -0600 Subject: [PATCH 34/46] refactor(db): narrow oracle guarantees and list entry tracking --- docs/contributing/oracle-coverage.md | 163 +++---- .../borrowed-date-indexed-range.md | 6 + .../captured-inarray-publication.md | 20 +- .../oracle-reviews/cross-realm-date-order.md | 6 + .../cross-realm-date-where-publication.md | 5 + .../custom-index-order-selection.md | 6 + .../pr-2116-correctness-and-guarantees.md | 94 +++- .../oracle-reviews/temporal-equality-alias.md | 6 + docs/guides/live-queries.md | 6 +- packages/db/src/query/compiler/evaluators.ts | 2 +- packages/db/src/query/compiler/index.ts | 29 +- packages/db/src/query/live/ARCHITECTURE.md | 21 +- packages/db/src/utils.ts | 66 +-- packages/db/src/utils/comparison.ts | 53 +- .../tests/change-event-history-oracle.test.ts | 190 ++++---- .../tests/comparison-oracle.property.test.ts | 452 +----------------- ...ustom-index-order-selection-oracle.test.ts | 440 ++--------------- .../compiler/binary-equality-work.test.ts | 13 +- .../in-evaluator-oracle.property.test.ts | 254 +++------- .../query/index-path-collision-oracle.test.ts | 436 +++-------------- packages/db/tests/query/subset-dedupe.test.ts | 22 +- ...dicate-publication-oracle.property.test.ts | 266 ++--------- .../db/tests/utils-oracle.property.test.ts | 74 +-- 23 files changed, 553 insertions(+), 2077 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index dca470eaec..f0d779a361 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -467,40 +467,21 @@ sorted row IDs. Its [transfer ledger](oracle-reviews/comparison-unit-retirement- records the five retired same-realm examples. A separate [cross-realm Date witness](oracle-reviews/cross-realm-date-order.md) compares native `node:vm` Dates by supplied timestamps and invalidity, including equal -times, numeric NaN with an invalid Date, and Dates with an own display tag. -A fixed local/foreign subclass witness checks native timestamps inherited -through a Date prototype, checks valid and invalid subclass order in both -directions against finite Dates, and rejects local and foreign -borrowed-prototype impostors. The owner -also checks an unindexed public ordered scan with a tagged foreign invalid -Date. A fixed 129-byte witness changes interior byte 64, and a 200-byte +times and numeric NaN with an invalid Date. The owner also checks an unindexed +public ordered scan with an ordinary foreign invalid Date. A fixed 129-byte +witness changes interior byte 64, and a 200-byte witness changes the last byte. Both compare direct `areValuesEqual` returns in each direction and distinct `normalizeValue` keys. The generated changed-byte grammar spans lengths 1–200, but its fixed 100-run campaign did not reach those long positions; the fixed witnesses reject a first-64-byte -comparator. -The proposed default native-byte order is unsigned lexicographic content, -with shorter prefixes first and equal contents at one comparator position. -Direct signs cover fresh equal views, the 127/128 boundary, prefixes, -an equal first byte followed by different second bytes, Buffer/Uint8Array -pairs, a foreign native view, and own `length` and `byteLength` shadows. The -later-byte case and its ready public scan/index counterpart reject a -first-byte-only order -mutant at their direct comparison and exact ordered-array checkpoints. -Fixed and random direct samples vary a common prefix of 2–24 bytes before -the first differing byte; a third-byte ready public case checks both index -types and rejects a comparator that reads only the first two bytes. -A byte/object/byte triple checks transitivity without prescribing which kind -comes first. These are immediate comparator observations, not relational -`WHERE` comparisons or live-query publication. The deep-equality owner also -checks that an own `length` shadow cannot hide a changed intrinsic byte in -direct equality, including a nested row field, over fixed and random 1–20 -byte inputs. Other typed-array class transitions remain under its existing -class-agnostic equality law; whether such transitions must publish a source -change needs a separate contract witness. +comparator. Default byte-content ordering is not promised. The custom-index +order owner checks that direct indexed and public scan reads conserve the same +keys through one byte-view update, deletion, and reinsertion; exact lookup +still groups ordinary equal byte contents. Other typed-array class transitions +remain under its existing class-agnostic equality law; whether such transitions +must publish a source change needs a separate contract witness. The same deep-equality owner checks two distinct keyless DataViews from one -foreign realm against the keyless-instance rule, including a forged display -tag, plus a mixed typed-array/DataView +foreign realm against the keyless-instance rule, plus a mixed typed-array/DataView pair. The Collection change-event owner receives that boundary through a typed-array-to-foreign-DataView update and a same-realm foreign-DataView replacement. Both commits must settle with public state, BasicIndex or @@ -515,20 +496,16 @@ time across local and `node:vm` realms. A fixed matrix crosses both operand realms and equal or changed valid and invalid times. Fixed-seed and random campaigns sample finite times from -1e12 to 1e12, invalid time, equal or adjacent changed time, and both realm placements. Direct results in both -operand orders reject a local-only Date classifier. One fixed equal-label -foreign Date pair rejects an enumerable-key answer that ignores changed time; -another keeps a labeled foreign Date distinct from an equal-keyed plain record. -These fixed controls do not specify a general policy for user-added Date -properties. The Collection change-event owner receives the ordinary Date rule -through six one-row sync updates, including invalid-to-valid and -valid-to-invalid transitions. A seventh fixed equal-label update preserves the -original missed-event counterexample. Each checks exact subscriber batches, +operand orders reject a local-only Date classifier. One fixed changed-time +foreign Date pair rejects an answer that ignores changed time. The Collection +change-event owner receives the ordinary Date rule through seven one-row sync +updates, including changed foreign times, invalid-to-valid, and +valid-to-invalid transitions. Each checks exact subscriber batches, callback-time public time, and the change-message mirror against public time after commit. These witnesses cover one `node:vm` host, unindexed sync publication, and native Dates with unchanged prototypes and methods. -Optimistic publication, indexed Date updates, other hosts, proxies, user-added -properties beyond the fixed controls, and replaced Date prototypes or methods -remain open. +Optimistic publication, indexed Date updates, other hosts, proxies, and +caller-modified Dates remain outside this witness. The same deep-equality owner models ordinary Temporal.Duration equality by an independent ten-field vector. Fixed copies and one-field changes reconstruct each field under both signs. Fixed opposite-sign seconds reject absolute-field @@ -611,33 +588,29 @@ scan and BasicIndex/BTreeIndex equality lookup with an explicit total comparator; default ordering of calendar-unit Durations needs `relativeTo` and is outside this equality claim. Equal and unequal values within Instant, PlainDate, PlainDateTime, and -PlainTime reject a kind-only Temporal equality key at those same paths. A -genuine Uint8Array with a shadowed display tag and a spoof-tagged Uint16Array -check the intrinsic byte-class boundary. Native `node:vm` Uint8Arrays with -equal and unequal content, including an own tag shadow, also reach compiled +PlainTime reject a kind-only Temporal equality key at those same paths. Native +`node:vm` Uint8Arrays with equal and unequal content also reach compiled `eq`, constant `in`, and row-supplied `in` in both operand directions; the original classifier failed equal foreign/local byte pairs. The `in` property has a guarded direct replay selector that runs only its random/replay campaign. -For caller-owned arrays, the compiled `in` owner now walks current indexed -members independently of an own `some` method. Constant and row-supplied -lists check false rejection and false acceptance after compilation and a list -slot edit. The production evaluator calls native Array `some` on the current -list; sparse and nullish controls retain native membership behavior. This +For caller-owned arrays, the compiled `in` owner now reads current members. +Constant and row-supplied lists check false rejection and false acceptance +after compilation and a list slot edit. Sparse and nullish controls retain +native membership behavior. This direct evaluation does not by itself establish live-query publication. Native local and `node:vm` Dates, including invalid Dates and timestamp numbers, reach those three compiled equality paths in both operand directions. An equal-timestamp Date from a second VM context is a positive compiled equality control, so the reference cannot accept only the first foreign prototype. -Borrowed Date prototypes remain ordinary reference values. The native-Date -reference model uses a captured Date intrinsic rather than local `instanceof`. -Own `length` and `byteLength` shadows, including a foreign byte view, and a -shadowed iterator cannot redefine native byte content. The compiled owner -checks equal and unequal pairs in both operand directions through `eq` and -both `in` forms; the comparison owner checks direct normalized-key relations. -The old byte-count read failed both boolean equality and direct key -normalization. Public indexed publication, detached buffers, and proxies -need separate witnesses. +One frozen primitive constant-list matrix checks the evaluator's immutable +list path against the same independent membership model, including NaN, +signed zero, Date normalization, nullish values, bigint, and a +normalization-prefix string. +The native-Date reference model uses a captured Date intrinsic rather than +local `instanceof`. Caller-modified Dates, byte views, and arrays are outside +these owners. Public indexed publication, detached buffers, and proxies need +separate witnesses. The alias owner also checks a public unindexed `eq` scan with two native-equal PlainMonthDay rows and a distinct reference year. Direct BasicIndex/BTreeIndex add, build, removal, re-add, and public indexed `eq` cover the orderable @@ -693,15 +666,11 @@ A custom-collation cursor includes an unsent string from the cursor's comparator-equal group. Escaped strings reach direct custom callbacks and Collection custom collation as raw values across ordered snapshots, direct range/take operations, and one subscription cursor; normalized exact lookup -still distinguishes their string keys. A mixed string/number/two-byte-view -snapshot and bounded native-byte fixtures compare unsigned content order and -source-key ties with the unindexed scan and both index types. They include the -127/128 boundary, prefixes, Buffer, a foreign view, and an own `length` -shadow; exact lookup still groups equal byte content. An escaped literal -remains distinct from the BTree undefined traversal marker. One source -history silently replaces an equal-content byte view and checks scan/index -order and key count. Another adopts a peer view, deletes, and reinserts its -key, checking ordered arrays and exact sets after every commit. A custom +still distinguishes their string keys. A bounded ordinary byte-view history +compares indexed and unindexed order without specifying a default byte-content +rank. It adopts a peer view, deletes, and reinserts its key, checking key +conservation and exact content sets after every commit. An escaped literal +remains distinct from the BTree undefined traversal marker. A custom callback history moves an equal-content view to a different callback rank and checks direct take, range, and cursor positions. The working-tree version of these histories rejected stale index positions and duplicate keys on @@ -728,24 +697,17 @@ ordinary-index use and an eager source path need distinct witnesses. The [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts) also has a [cross-realm Date equality matrix](oracle-reviews/cross-realm-date-where-publication.md). For local and `node:vm` Dates with timestamps 1 and 2 or invalid timestamps, -including tagged foreign Dates and fixed local/foreign subclasses with an own -display tag, plus number 1, NaN, and a foreign borrowed-prototype impostor, it -compares public live-query keys, filtered subscriber keys, and direct snapshot -keys after preload and a four-row sync transaction. +plus number 1 and NaN, it compares public live-query keys, filtered subscriber +keys, and direct snapshot keys after preload and a four-row sync transaction. The generated WHERE histories check direct snapshots at attachment; the fixed Date matrix also checks them after its commit. Subscriber batches reconstruct subscriber state after each generated commit. Auto-indexing is off in both modes; one mode adds BasicIndex explicitly and also checks its equality lookup and an optimized-only public snapshot. This proves a public indexed snapshot result but does not prove that the live-query -planner chose the index. Other realm hosts, proxies, deeper subclass chains, -Date prototype swaps or method replacement, public range predicates, and -generated subclass histories need separate witnesses. -A focused unindexed public query also keeps one local -`Object.create(DateSubclass.prototype)` impostor outside Date equality while -retaining its same-object match. It also excludes the impostor from the -invalid-Date/NaN and valid zero Date/number equality classes at public -checkpoints. Indexed publication for that local impostor remains open. +planner chose the index. Other realm hosts, proxies, modified Dates, subclasses, +public range predicates, and generated Date +histories need separate witnesses. The [captured `inArray` publication review](oracle-reviews/captured-inarray-publication.md) records the caller-owned list law: changing its members alone does not @@ -759,8 +721,9 @@ after that repair; the plan now pays full-source transfer to observe unseen source events. A pooled query with an equality group and residual mutable list uses the live-query Collection path; a source event in another group must still refresh its result. Controlled on-demand acquisition, no-delta events on both joined -source sides, an outer QueryRef filter, a correlated include, and mutable Date -and byte-list items have fixed working-tree receivers. One 40-row direct-query +source sides, an outer QueryRef filter, a correlated include, and replaced Date +and byte-array list members have fixed working-tree receivers. In-place changes +to those members are outside the captured-list law. One 40-row direct-query witness rejects an all-candidate membership rescan on an ordinary unchanged-list event while requiring a revisit after a list change. These are bounded in-memory receiving cuts. A controlled on-demand ordered receiver checks that @@ -828,27 +791,15 @@ behavior, or exact committed-head review. The [review record](oracle-reviews/captured-inarray-publication.md) tracks the ORC-012 applicability audit and unresolved cells. -The [borrowed-Date range review](oracle-reviews/borrowed-date-indexed-range.md) -records a separate public range owner in `index-path-collision-oracle.test.ts`. -Fixed `gt` and `lt` histories compare public snapshots before and after -BasicIndex or BTreeIndex is installed. An ordinary object borrowing a local -Date subclass prototype must take the scan path whether it is a source value -or the predicate bound. A genuine foreign Date subclass is a positive indexed -range control. A later fixed history gives local and foreign native Date -subclasses misleading `valueOf` results. Public scan and indexed `gt`, `gte`, -`lt`, and `lte` snapshots use their intrinsic timestamps, with an -optimized-only index-path check. The model freezes each source timestamp -before Collection admission, so later changes to a source Date cannot alter -the expected answer. A native subclass also serves as a bound for all four -effective relations in both operand positions while its `valueOf` reports a -misleading number. -Plain, local-subclass, and foreign-subclass rows also share the bound's exact -timestamp. Inclusive and exclusive ranges compare complete key sets across -the scan and both indexed paths; an omitted tied subclass fails the exact-key -assertion. -Date/number scan predicates run in both -operand orders; mixed-domain index eligibility is not claimed. Other coercion -methods, invalid Dates, later publication, and proxies remain open there. +The [historical borrowed-Date range review](oracle-reviews/borrowed-date-indexed-range.md) +records a wider law that this PR has narrowed. The public range owner in +`index-path-collision-oracle.test.ts` now supplies ordinary local and `node:vm` +Dates before, at, and after a bound from either realm. Public `gt` and `gte` +snapshots agree with supplied timestamps before and after installing BasicIndex or BTreeIndex; +an optimized-only snapshot confirms the indexed path. Equal local and foreign +Dates distinguish inclusive from exclusive ranges. Modified Dates, subclasses, +borrowed prototypes, invalid Dates, other relational operators, and later +publication remain outside this owner's current Date range claim. The [mixed-value index order review](oracle-reviews/mixed-value-index-order.md) links the direct index and public WHERE owners. The direct owner checks @@ -1567,12 +1518,10 @@ new returned-mutation check. The direct index refinement owner does not prove this Collection mutation path. Callback-time index agreement, the reported insert-seeded string-key setup, adapter cancellation, and non-local-only persistence remain outside this lane. -Its native-byte sync witness adds a one-key `[1]` to `[2]` replacement in which -both genuine views shadow own `length` as zero. After commit, one update with -the previous and current intrinsic bytes must reach the subscriber while +Its native-byte sync witness adds a one-key `[1]` to `[2]` replacement. After +commit, one update with the previous and current bytes must reach the subscriber while public state and BasicIndex or BTreeIndex exact lookup agree with `[2]`. A -fresh equal-content view then remains silent. The pre-fix implementation -updated state and indexes but emitted no message for the changed bytes. This +fresh equal-content view then remains silent. This bounded sync cut records whole callback batches, so even an empty callback after the equal copy fails the silence check. It does not cover optimistic draft admission, other typed-array classes, or live-query notification. diff --git a/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md b/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md index 3c5ddcf2fb..758f63a1ea 100644 --- a/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md +++ b/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md @@ -1,5 +1,11 @@ # Borrowed Date prototypes in indexed ranges +Scope update for PR #2116: the tests and contract claims described below are +historical. The current indexed range owner keeps ordinary local and +cross-realm Date source values and bounds, while borrowed prototypes, +subclasses, and overridden methods are outside its contract. See the +[coverage map](../oracle-coverage.md) for the active witness. + A Collection index may change the work used to answer a range predicate, but it must not change the public key set. The live-query architecture requires a range to fall back when indexed values do not share the bound's relational diff --git a/docs/contributing/oracle-reviews/captured-inarray-publication.md b/docs/contributing/oracle-reviews/captured-inarray-publication.md index 723546a7a7..4de402a7a3 100644 --- a/docs/contributing/oracle-reviews/captured-inarray-publication.md +++ b/docs/contributing/oracle-reviews/captured-inarray-publication.md @@ -40,9 +40,10 @@ must publish one coherent reverse insert/delete batch for the already joined rows. A QueryRef receiver puts mutable membership in the outer query while the inner filter suppresses the triggering source change. A correlated include receiver changes an unrelated child row and checks the parent row's -child Collection against current membership. Fixed Date and byte-array -items mutate in place inside their captured lists and change the result at -the next unrelated source event. +child Collection against current membership. Earlier fixed Date and byte-array +witnesses mutated items in place inside their captured lists and observed a +changed result at the next unrelated source event. The current scope is +clarified below. The pooled-query receiver is `packages/db/tests/query/pooled-live-query-oracle.property.test.ts`. A query @@ -292,3 +293,16 @@ before subset settlement. Returning equality for every local comparison in the unbounded captured-list path failed the opposed key/value order check. The reviewer found no further defect in those two witnesses. Their stated composition and provider limits still apply. + +## Scope refinement for native list items + +The maintainer declined a guarantee after callers modify standard data values. +The older Date and byte-array witnesses above edited an item already held in +the captured list. The current evaluator and WHERE publication witnesses +instead replace the list entry with a new Date or byte array, then compare +current membership at evaluation or at the next source event. The prior +in-place observations remain historical evidence only. No current oracle +claims what a retained query does after an existing native item is mutated. +The compiler now snapshots entry identity rather than re-encoding each item +or checking its property descriptors at every source event. Same-length +replacement witnesses reject an array-reference or length-only detector. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-order.md b/docs/contributing/oracle-reviews/cross-realm-date-order.md index 18575753c6..845f46c113 100644 --- a/docs/contributing/oracle-reviews/cross-realm-date-order.md +++ b/docs/contributing/oracle-reviews/cross-realm-date-order.md @@ -1,5 +1,11 @@ # Date order across JavaScript realms +Scope update for PR #2116: this record preserves evidence from an earlier +commit. The current comparison oracle keeps ordinary native Date timestamps +across realms; its tagged, forged, borrowed, and subclass cases were removed +after the maintainer declined guarantees for caller-modified built-ins. See +the current [coverage map](../oracle-coverage.md) for the active witness. + The comparison loss audit led to a same-realm NaN/invalid-Date transfer. A fresh hostile review then challenged the whole published Date ordering law, including native Dates created in another JavaScript realm. The live-query diff --git a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md index 9f15ddacf5..641e7c2252 100644 --- a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md +++ b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md @@ -1,5 +1,10 @@ # Cross-realm Date equality at WHERE publication +Scope update for PR #2116: this is the review record for an earlier, wider +Date oracle. The current WHERE owner checks ordinary native Dates across realms; +tagged, subclass, borrowed-prototype, and forged cases no longer state its +contract. The [coverage map](../oracle-coverage.md) describes the active cut. + The [comparison transfer ledger](comparison-unit-retirement-round-5.md) retired five same-realm NaN and invalid-Date examples. A hostile review of their receiving law found a different defect: a native Date created in another diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md index 64c8437f38..9ce724a348 100644 --- a/docs/contributing/oracle-reviews/custom-index-order-selection.md +++ b/docs/contributing/oracle-reviews/custom-index-order-selection.md @@ -1,5 +1,11 @@ # Custom comparator index selection +Scope update for PR #2116: later sections record an unsigned default-byte +order and caller-modified byte controls from an earlier commit. Those proposed +guarantees were declined. The current oracle keeps byte-content equality, +scan/index order agreement, and the maintainer-approved per-row custom +comparator rule. The [coverage map](../oracle-coverage.md) states the active cut. + An index with a user-supplied `compareFn` cannot advertise the ordinary Collection `orderBy` comparator merely through `compareOptions`. The live-query architecture's Identity contract requires advertised comparison diff --git a/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md b/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md index cc7bc01ab9..d2e8514df0 100644 --- a/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md +++ b/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md @@ -57,13 +57,14 @@ figure mixed production code and architecture prose at an earlier head. | Guarantee or work cost | Authority and value | Decision surface | | --- | --- | --- | | A captured mutable `inArray` reads current members and refreshes the whole retained query on the next source event. | The maintainer explicitly chose both rules. The receiving oracles cover WHERE, joins, selections, groups, aggregates, HAVING, order, nested queries, and selected on-demand/include paths. | Largest cost: the review estimated about 427 source lines. Full-source acquisition disables index/subset pushdown, keyed lazy demand, and pooled fast paths for this query shape. Changing this requires a contract decision; freezing a list already restores the optimizations. | +| A captured list's Date or byte-array item changes in place. | The maintainer declined guarantees after modifying standard values. Replacing the list entry remains inside the accepted live-list rule. | The evaluator and WHERE publication witnesses replace each list entry with a new native value. No oracle promises an observation after mutating the old item. | | A custom index callback orders **each stored row**, even when query equality groups two values together. | The maintainer explicitly chose this. | Both indexes retain per-row order operands and may sort/refresh more rows. Callback arguments now expose raw Date/Uint8Array values instead of the former timestamp/encoded-byte representation. That compatibility change follows the per-row rule and should be called out in release notes. | -| Default native-byte order is unsigned lexicographic order. | This PR proposes the policy; no separate maintainer decision was recorded. | It requires scan/index agreement and comparator machinery. Keep or remove as a product decision, rather than treating the old encoded-string order as authoritative. | -| Cross-realm native Dates and byte views, plus Date subclasses with overridden coercion, use intrinsic times or bytes. | The branch adds public and direct witnesses; ordinary local Dates and bytes already have value semantics. The wider host/subclass scope has no separate maintainer decision. | Removing this scope would revise several oracle laws and could let valid foreign values change a stored row without publishing it. Keep the R25 mirror repair as long as this scope is promised. | +| Default native-byte order is unsigned lexicographic order. | This PR proposed the policy; the maintainer declined it. | The signed/unsigned and prefix order checks were removed. Byte-content equality and indexed/scan coherence remain. | +| Ordinary cross-realm native Dates and Uint8Arrays use their times or bytes. | The maintainer questioned realm-sensitive inequality after its history showed only a transport test, not a public promise. | Keep ordinary cross-realm equality with a narrow native-value detector. Date subclasses with overridden coercion are outside this choice. | | Mixed-kind default index order stays total, while public Temporal `orderBy` still rejects kinds without an order. | Valid mixed rows must be admitted without partially updating a Collection; the specific rank between unlike ordinary kinds is a policy chosen by this branch. | The kind separation and private Temporal order add comparator code. A narrower accepted input domain or another internal order would need an explicit contract change and the R16 atomic-publication witness. | | A direct index removal accepts an equal-time but different old Date whose custom callback position differs from the stored Date. | The general refinement law supports normal remove/re-add, but the index API does not document this substitution. | Added per-key stored-order state supports it. This is a candidate to narrow if the direct API may require the actual prior value. | | A fresh, equal-field Duration replacement emits no update message. | Consistent with existing change-event suppression; the mirror law requires changed-field publication, not this silence. | The public R29 receiver asserts silence. It can be relaxed independently of exact field equality and changed-field publication. | -| User-modified Date display tags, byte-view length properties, and an array's own `some` method cannot redefine intrinsic value or `inArray` membership. | These are valid JavaScript objects but user-modified built-ins, with weaker product authority than ordinary Dates, byte views, and arrays. | Their checks and helpers add complexity. Treat them as separate compatibility decisions. Deliberately malformed, borrowed, forged, and shadowed Temporal values were removed from the claimed domain. | +| User-modified Date display tags, byte-view length properties, and an array's own `some` method cannot redefine intrinsic value or `inArray` membership. | The maintainer expressly declined guarantees after callers modify standard data types. | These tests and their protective helpers were removed. Deliberately malformed, borrowed, forged, and shadowed Temporal values are also outside the claimed domain. | | Repeated Temporal key/display work, per-row BasicIndex build sorting, and equal-row index refresh. | Deterministic work probes observed these costs; no work budget was promised. | Optimize only if a bounded workload justifies it. The old claim that *every* Temporal comparison calls `toString` is stale after the valid-value narrowing. | R21 is not closed as a Firefox defect. A mocked time-zone formatter preserving @@ -100,11 +101,82 @@ public DB source entry is 112,595 to 115,464 gzip bytes (**+2,869**) under the same esbuild command at both revisions. The main bundle was reproduced byte for byte before comparing. These views overlap and must not be added. -The guarantee choices in the table above still apply. The two largest behavior -costs, live captured arrays with whole-result refresh and per-row custom -callback order, are explicit maintainer decisions. Unsigned default byte order, -cross-realm/subclass scope, mixed-kind default order, different-position -old-Date removal, silence for equal-field Duration copies, and intrinsic -semantics for user-modified built-ins remain separate product choices. No size -reduction is credited for dropping them without a same-path oracle and a -measured implementation change. +The two largest behavior costs, live captured arrays with whole-result refresh +and per-row custom callback order, are explicit maintainer decisions. Ordinary +cross-realm native equality is retained after examining the old transport +test's history. Unsigned default byte order and intrinsic semantics for +user-modified built-ins are declined. Mixed-kind default rank, +different-position old-Date removal, and silence for equal-field Duration +copies remain separate product choices. The paired measurement below records +the current narrowed tree. + +On `origin/main`, `query/where.test.ts` and `query/indexes.test.ts` exercise +fixed `inArray` lists, including Date-by-time matching and an indexed lookup. +The `in-evaluator` oracle supplies constant or row-field lists but does not +edit a captured constant list after compilation. One direct evaluator work +test mutates a retained byte-array item and expects `in` to observe the new +bytes; it does not exercise a live-query Collection. Constant `Value` lists +were compiled into a `Set` once. No earlier test requires a captured list's +entries to be added, removed, or replaced after compilation, or a whole +retained result to refresh on the next source event. The current live-list +rule is an explicit new contract choice, not a unit-test transfer. + +## Narrowing decision: bytes and modified built-ins + +The retired unit-test blocks did not assert default byte ordering or resilience +to caller-modified standard values. The maintainer declined both guarantees. +Ordinary byte-content equality remains covered by earlier unit tests and the +binary-ID equality fix (#779). The maintainer also kept the explicit choice +that a custom index comparator orders each stored row. + +Before changing oracle expectations, the distinguishing predictions are: + +| Input and checkpoint | Earlier PR law | Narrowed law | +| --- | --- | --- | +| Two ordinary byte views `[127]` and `[128]`, default `orderBy` or direct comparator | The first sorts before the second by unsigned content. | No specific byte-content order is promised. Indexed and unindexed public reads of the same rows must agree, and exact lookup still groups equal byte contents. | +| Two fresh ordinary byte views with equal contents, exact lookup | Both match one content key. | Both still match one content key. A custom comparator may place their rows separately according to the callback. | +| A Date with an own display tag, a byte view with an own count or iterator shadow, or an array with an own `some` method | Native slots or intrinsic methods override the caller's changes. | These caller-modified built-ins are outside the oracle's declared domain. The oracle makes no prediction for them. Ordinary Dates, byte views, and arrays keep their retained laws. | +| An existing Date or byte-array item in a captured `inArray` list changes in place | The earlier working tree required the next source event to observe its changed content. | The accepted live-list law covers adding, removing, and replacing array entries only. Replacing an entry with a fresh native value refreshes the whole result; changing an existing item in place is outside scope. | +| An ordinary local Date or Uint8Array and an equal-content value made in a second JavaScript environment, compiled `eq` and `in` | The PR revision predicts both values match. | Retain content/time equality for ordinary native values. The surviving `subset-dedupe.test.ts` had expected only the literal's environment to match, so this remains an explicit behavior revision. | + +Removing a law from this table does not change the expected results of any +retired unit-test block. The cross-environment case is a separate revision of +a **surviving** `subset-dedupe.test.ts` expectation. That test entered in +commit `cfb01cee34` (PR #1797) under “request transport preserves predicate +matches.” It checked that the request kept its literal object and that the +evaluator returned its then-current result. The earlier code used local +`instanceof` checks; no public guide or changelog claimed realm-sensitive +equality. The maintainer questioned why such a distinction should be retained, +so this PR keeps equality for ordinary native values across JavaScript +environments while separately checking request identity. + +The R25 table above describes the earlier reviewed commit, which included a +foreign Date with a user-added display tag. After removing that case, the +ordinary changed-time foreign-Date replacement is a guard against an +always-equal model, not a reproduction of the original missed update. Source +inspection of `origin/main` shows that distinct unmodified foreign Dates were +always treated as unequal: a changed timestamp published, while equal +timestamps could publish an extra update. The current equal-time receiver +checks the narrowed publication rule. Cross-realm equality is a deliberate +contract choice of this PR; none of the retired unit-test blocks required it. + +## Narrowed working-tree weight + +After removing the declined byte-order, modified-value, and item-in-place +cases, production TypeScript is **+804 net lines** against `origin/main` +(1,305 added, 501 removed). Architecture prose is **+61 net lines** separately. +A paired browser source-entry measurement uses esbuild 0.28.1 +with `--bundle --minify --platform=browser --format=esm --target=es2020`, then +`gzip -n -9`, while resolving the same installed dependencies for each source +snapshot. The baseline is 114,670 bytes, the published PR head is 117,566 +(**+2,896**), and the narrowed working tree is 115,901 (**+1,231**). This +isolates the DB source change and is not the all-ESM-module sum. The absolute +byte counts differ from the earlier build method above; their paired deltas +are close. The entry-only list snapshot removes per-item normalization and +descriptor checks at each source event. Holding every other current source +file fixed, restoring the previous `snapshotList` produces 117,419 bytes, +so this simplification saves **1,512 gzip bytes**. A separate comparator loop +cleanup saves another six bytes. The current tree saves 1,665 bytes against +the published head. Most remaining code weight implements the two explicit maintainer +choices: live captured lists with whole-result refresh and per-row custom +comparator ordering. diff --git a/docs/contributing/oracle-reviews/temporal-equality-alias.md b/docs/contributing/oracle-reviews/temporal-equality-alias.md index 022c61ecbc..c1028163d7 100644 --- a/docs/contributing/oracle-reviews/temporal-equality-alias.md +++ b/docs/contributing/oracle-reviews/temporal-equality-alias.md @@ -1,5 +1,11 @@ # Temporal alias equality +Scope update for PR #2116: this record retains earlier forged, borrowed, and +shadowed-value experiments. The current oracles claim valid Temporal values +from one polyfill copy and ordinary native byte arrays; caller-modified and +malformed values are outside their active laws. See the +[coverage map](../oracle-coverage.md) for current owners and limits. + Changelog #1519 promises `ZonedDateTime` query equality consistent with Temporal's `.equals()`. At one instant, `[UTC]` and `[Etc/UTC]` are equal, while `[+00:00]` is a distinct zone. Exact nanoseconds and calendar identity diff --git a/docs/guides/live-queries.md b/docs/guides/live-queries.md index 8c3097460f..06564b8a93 100644 --- a/docs/guides/live-queries.md +++ b/docs/guides/live-queries.md @@ -778,7 +778,7 @@ For a complete reference of all available functions, see the [Expression Functio Comparisons follow SQL/PostgreSQL conventions rather than raw JavaScript: - **`null` / `undefined` use three-valued logic.** Any comparison involving `null` or `undefined` evaluates to `UNKNOWN`, so the row is not matched. For example `eq(user.score, null)` matches nothing — use a dedicated null check (e.g. `isUndefined`) to match missing values. -- **Native `Date` values with a Date prototype that retains `getTime` compare by timestamp.** Dates with the same millisecond timestamp match equality and occupy the same order position. This includes Dates from another JavaScript realm and Dates with their own `Symbol.toStringTag`. Replacing a Date's prototype is outside this rule. +- **Ordinary native `Date` values compare by timestamp.** Dates with the same millisecond timestamp match equality and occupy the same order position, including Dates created in another JavaScript environment. - **`NaN` follows PostgreSQL float semantics.** `NaN` is treated as equal to itself and greater than every other (non-null) value. So `eq(row.value, NaN)` matches `NaN` rows, `gt(row.value, x)` includes `NaN`, and ordering by such a field places `NaN` last. (Invalid `Date` values, whose timestamp is `NaN`, behave the same way.) This differs from JavaScript, where `NaN === NaN` is `false`, and matches how PostgreSQL orders and indexes floating-point values. ## Select @@ -2654,7 +2654,9 @@ applies when `inArray` filters rows, computes selected values, forms a join or group, supplies an aggregate even inside another selected expression, filters with `having`, or determines order. The resulting values and order are published together. Editing the array alone -does not publish a new result. +does not publish a new result. Current members means entries added, removed, +or replaced; changing an existing Date or byte-array item in place is outside +this guarantee. #### `like(value, pattern)`, `ilike(value, pattern)` String pattern matching: diff --git a/packages/db/src/query/compiler/evaluators.ts b/packages/db/src/query/compiler/evaluators.ts index b61d7ac32f..5ae4e1ec39 100644 --- a/packages/db/src/query/compiler/evaluators.ts +++ b/packages/db/src/query/compiler/evaluators.ts @@ -509,7 +509,7 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { if (!Array.isArray(array)) { return false } - return Array.prototype.some.call(array, (item: unknown) => + return array.some((item: unknown) => valuesEqual(normalizeEqualityOperand(item), value), ) } diff --git a/packages/db/src/query/compiler/index.ts b/packages/db/src/query/compiler/index.ts index 7a2a6a5324..0b11ab96cd 100644 --- a/packages/db/src/query/compiler/index.ts +++ b/packages/db/src/query/compiler/index.ts @@ -476,36 +476,17 @@ function selectHasMutableInList(select: QueryIR[`select`]): boolean { return lists.size > 0 } -const missingListSlot = Symbol(`missing inArray list slot`) - -function snapshotList(list: ReadonlyArray): { - values: Array - volatile: boolean -} { - const values: Array = [] - let volatile = false - for (let index = 0; index < list.length; index++) { - let slot: PropertyDescriptor | undefined - try { - slot = Object.getOwnPropertyDescriptor(list, index) - } catch { - volatile = true - } - if (!slot || !(`value` in slot)) volatile = true - const item = index in list ? list[index] : missingListSlot - values.push(normalizeValue(item)) - } - return { values, volatile } +function snapshotList(list: ReadonlyArray): Array { + return Array.from(list) } function sameListSnapshot( left: ReturnType, right: ReturnType, ): boolean { - if (left.volatile || right.volatile) return false - if (left.values.length !== right.values.length) return false - return left.values.every((value, index) => - Object.is(value, right.values[index]), + return ( + left.length === right.length && + left.every((value, index) => Object.is(value, right[index])) ) } diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 3685cd95bb..fbd013c3c8 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -753,9 +753,13 @@ read in `where`, a join operand, `select`, a group key or aggregate argument, `having`, or an order expression. One source transaction publishes the new result coherently; changing a projected value, group, or order cannot leave the old stage output behind or expose an intermediate public result. +This rule covers entries added, removed, or replaced in the captured array; +it does not specify behavior after an existing Date or byte-array member is +mutated in place. -The compiler derives one list revision from raw source events and shares it -across the compiled query graph. A list change makes each affected expression +The compiler compares captured array entries by identity at raw source events +and shares one list revision across the compiled query graph. +A list change makes each affected expression stage revisit its retained inputs; an ordinary delta with an unchanged list visits only the inputs it changes. Each stage retains its prior output for retraction before it evaluates the new membership. JOIN, WHERE, SELECT, @@ -1614,12 +1618,13 @@ create recursive Collection machinery. counts as a preload. A deferring live query keeps its sources from garbage collection but is not counted in their `subscriberCount` or `subscribers:change`, which adapters read as a request for live data. -16. **Captured membership:** editing a caller-owned `inArray` list does not - publish by itself. After the next source event, the complete existing - live-query result reflects the list's current members at every expression - stage, including rows untouched by that event. Value, group, join, and - order changes publish coherently. Submitted source request data remains - immutable. +16. **Captured membership:** adding, removing, or replacing entries in a + caller-owned `inArray` list does not publish by itself. After the next + source event, the complete existing live-query result reflects those + current entries at every expression stage, including rows untouched by + that event. Value, group, join, and order changes publish coherently. + Mutating an existing Date or byte-array member in place is outside this + law. Submitted source request data remains immutable. ## Glossary diff --git a/packages/db/src/utils.ts b/packages/db/src/utils.ts index 721bbc06bb..55a9327896 100644 --- a/packages/db/src/utils.ts +++ b/packages/db/src/utils.ts @@ -9,26 +9,6 @@ interface TypedArray { [index: number]: number } -const typedArrayLength = Object.getOwnPropertyDescriptor( - Object.getPrototypeOf(Uint8Array.prototype), - `length`, -)!.get! -const typedArrayTag = Object.getOwnPropertyDescriptor( - Object.getPrototypeOf(Uint8Array.prototype), - Symbol.toStringTag, -)!.get! - -/** Read a native typed-array class across realms without trusting display tags. */ -export function nativeTypedArrayTag(value: unknown): string | undefined { - if (!ArrayBuffer.isView(value)) return undefined - return Reflect.apply(typedArrayTag, value, []) as string | undefined -} - -/** Read the native element count even when a view shadows its length property. */ -export function nativeTypedArrayLength(value: ArrayBufferView): number { - return Reflect.apply(typedArrayLength, value, []) as number -} - // Duration equality retains the exact units, including subsecond units that // share a default display. Query normalization uses the same field order. export const durationFields = [ @@ -44,32 +24,15 @@ export const durationFields = [ `nanoseconds`, ] as const -// The Date intrinsic recognizes native Dates across realms. A display tag -// cannot hide a Date's timestamp or make an ordinary object into a Date. +// Ordinary native Dates compare by timestamp, including across realms. export function dateTimestamp(value: unknown): number | undefined { if (value === null || typeof value !== `object`) return undefined + if (value instanceof Date) return value.getTime() + if (Object.prototype.toString.call(value) !== `[object Date]`) + return undefined try { - if (value instanceof Date) return Date.prototype.getTime.call(value) - if (Symbol.toStringTag in value) { - // A Date subclass can inherit getTime from its Date prototype. Walk - // descriptors without reading a user display-tag or getTime getter; - // the intrinsic below still checks for the native Date slot. - let prototype = Object.getPrototypeOf(value) - while (prototype !== null) { - const descriptor = Object.getOwnPropertyDescriptor(prototype, `getTime`) - if (descriptor) { - if (typeof descriptor.value !== `function`) return undefined - break - } - prototype = Object.getPrototypeOf(prototype) - } - if (prototype === null) return undefined - } else if (Object.prototype.toString.call(value) !== `[object Date]`) { - return undefined - } return Date.prototype.getTime.call(value) } catch { - // An ordinary object or proxy has no native Date timestamp. return undefined } } @@ -148,8 +111,6 @@ export function deepEqualsInternal( return false } - // Native Dates compare by time even when their constructors come from - // different realms. An invalid Date has a NaN time, distinct from no Date. const aDateTime = dateTimestamp(a) const bDateTime = dateTimestamp(b) if (aDateTime !== undefined || bDateTime !== undefined) { @@ -286,13 +247,16 @@ export function deepEqualsInternal( if (b instanceof Set) return false // Handle TypedArrays - const typedArrayTagA = nativeTypedArrayTag(a) - const typedArrayTagB = nativeTypedArrayTag(b) - if (typedArrayTagA !== undefined && typedArrayTagB !== undefined) { + if ( + ArrayBuffer.isView(a) && + ArrayBuffer.isView(b) && + Object.prototype.toString.call(a) !== `[object DataView]` && + Object.prototype.toString.call(b) !== `[object DataView]` + ) { const typedA = a as unknown as TypedArray const typedB = b as unknown as TypedArray - const lengthA = nativeTypedArrayLength(a) - const lengthB = nativeTypedArrayLength(b) + const lengthA = typedA.length + const lengthB = typedB.length // Elements compare like numbers: -0 equals 0, NaN equals NaN. Only a // draft treats a change of typed-array class as a change. if ( @@ -310,7 +274,11 @@ export function deepEqualsInternal( return true } // Symmetric check: if b is TypedArray but a is not, they're not equal - if (typedArrayTagB !== undefined && !ArrayBuffer.isView(a)) { + if ( + ArrayBuffer.isView(b) && + Object.prototype.toString.call(b) !== `[object DataView]` && + !ArrayBuffer.isView(a) + ) { return false } diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index fbe20320a0..af1e36cc04 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -1,10 +1,4 @@ -import { - dateTimestamp, - durationFields, - isTemporal, - nativeTypedArrayLength, - nativeTypedArrayTag, -} from '../utils' +import { dateTimestamp, durationFields, isTemporal } from '../utils' import { getRuntimeReferenceIdentity } from '../query/runtime-reference-identity' import { codedMessage, devBuild } from '../error-message' import type { CompareOptions } from '../query/builder/types' @@ -112,21 +106,6 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { return compareTemporalValues(a, b) } - // Native byte views use their unsigned contents for default order. This - // agrees with their content-based equality and gives fresh equal views the - // same position; source keys break ties between rows. - const aIsBytes = isUint8Array(a) - const bIsBytes = isUint8Array(b) - if (aIsBytes && bIsBytes) { - const aLength = nativeTypedArrayLength(a) - const bLength = nativeTypedArrayLength(b) - for (let index = 0; index < Math.min(aLength, bLength); index++) { - const difference = a[index]! - b[index]! - if (difference !== 0) return difference - } - return aLength - bLength - } - // Symbols have identity but no built-in order: relational comparison throws. // A stable runtime ID gives tree indexes a total order while preserving // equality only for the same symbol. @@ -156,18 +135,14 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { ? 1 : aTemporal ? 2 - : aIsBytes - ? 3 - : 4 + : 3 const bKind = Array.isArray(b) ? 0 : bDateTime !== undefined ? 1 : bTemporal ? 2 - : bIsBytes - ? 3 - : 4 + : 3 if (aKind !== bKind) return aKind - bKind const aId = getObjectId(a) const bId = getObjectId(b) @@ -277,15 +252,18 @@ export const defaultComparator = makeComparator({ stringSort: `locale`, }) -/** Recognize native Uint8Array values, including Buffers and foreign realms. */ +/** Recognize ordinary Uint8Array values, including Buffers and foreign realms. */ export function isUint8Array(value: unknown): value is Uint8Array { - return nativeTypedArrayTag(value) === `Uint8Array` + return ( + ArrayBuffer.isView(value) && + Object.prototype.toString.call(value) === `[object Uint8Array]` + ) } /** Compare two Uint8Arrays for content equality. */ function areUint8ArraysEqual(a: Uint8Array, b: Uint8Array): boolean { - const length = nativeTypedArrayLength(a) - if (length !== nativeTypedArrayLength(b)) { + const length = a.length + if (length !== b.length) { return false } for (let i = 0; i < length; i++) { @@ -322,9 +300,8 @@ function canonicalTemporalZone(zoneId: string): string { function normalizeBinary(value: Uint8Array): string { let bytes = `` - const length = nativeTypedArrayLength(value) - for (let index = 0; index < length; index++) { - bytes += String.fromCharCode(value[index]!) + for (const byte of value) { + bytes += String.fromCharCode(byte) } return normalizedKey(`binary`, bytes) } @@ -491,9 +468,9 @@ export function compareTemporalValues(a: unknown, b: unknown): number { * * Temporal types intentionally throw from `valueOf` to prevent silent * miscomparison via the native relational operators — delegate to - * `compareTemporalValues` for them. Native Dates compare by their intrinsic - * timestamps, including against numbers and when a subclass overrides - * `valueOf`. Other values retain the native relational comparison. + * `compareTemporalValues` for them. Ordinary native Dates compare by their + * timestamps, including against numbers and across realms. Other values + * retain the native relational comparison. * * Callers must handle null/undefined themselves — this helper assumes both * arguments are non-null. diff --git a/packages/db/tests/change-event-history-oracle.test.ts b/packages/db/tests/change-event-history-oracle.test.ts index 38943d8cdd..b23ed3b340 100644 --- a/packages/db/tests/change-event-history-oracle.test.ts +++ b/packages/db/tests/change-event-history-oracle.test.ts @@ -879,10 +879,9 @@ describe(`change-event history oracle`, () => { describe(`native-byte change publication`, () => { /** - * A sync replacement with different intrinsic bytes changes the visible - * row, so one update must reach a subscriber after commit. An own `length` - * shadow is display state, not a way to erase the byte difference. The - * source's one-key before/after sequence supplies the expected payload; + * A sync replacement with different bytes changes the visible row, so one + * update must reach a subscriber after commit. The source's one-key sequence + * supplies the expected payload; * exact index lookup is a separate receiving path. An equal-byte copy then * remains silent. This bounded history does not claim optimistic writes or * other typed-array classes. @@ -890,94 +889,88 @@ describe(`native-byte change publication`, () => { it.each([ [`BasicIndex`, BasicIndex], [`BTreeIndex`, BTreeIndex], - ] as const)( - `publishes changed intrinsic bytes with %s`, - async (name, IndexType) => { - type ByteRow = { id: string; value: Uint8Array } - type ByteSync = Parameters[`sync`]>[0] - const shadow = (value: number) => - Object.defineProperty(Uint8Array.of(value), `length`, { value: 0 }) - const bytes = (view: Uint8Array) => - Array.from(Uint8Array.prototype.values.call(view)) - const before = shadow(1) - const after = shadow(2) - const equalCopy = shadow(2) - let sync!: ByteSync - let unsubscribe: (() => unknown) | undefined - let cleanup: (() => unknown) | undefined + ] as const)(`publishes changed bytes with %s`, async (name, IndexType) => { + type ByteRow = { id: string; value: Uint8Array } + type ByteSync = Parameters[`sync`]>[0] + const bytes = (view: Uint8Array) => Array.from(view) + const before = Uint8Array.of(1) + const after = Uint8Array.of(2) + const equalCopy = Uint8Array.of(2) + let sync!: ByteSync + let unsubscribe: (() => unknown) | undefined + let cleanup: (() => unknown) | undefined - await withOracleCleanup(async () => { - const collection = createCollection({ - id: `native-byte-publication-${name}`, - getKey: ({ id }) => id, - autoIndex: `off`, - startSync: true, - sync: { - sync: (actions) => { - sync = actions - actions.begin() - actions.write({ - type: `insert`, - value: { id: `a`, value: before }, - }) - actions.commit() - actions.markReady() - }, - }, - }) - cleanup = () => collection.cleanup() - await collection.stateWhenReady() - const index = collection.createIndex((row) => row.value, { - indexType: IndexType, - }) - const batches: Array< - Array<{ - type: string - key: string - previous: Array | undefined - value: Array - }> - > = [] - const subscription = collection.subscribeChanges( - (changes) => { - batches.push( - changes.map((change) => ({ - type: change.type, - key: change.key, - previous: - change.type === `update` - ? bytes(change.previousValue!.value) - : undefined, - value: bytes(change.value.value), - })), - ) + await withOracleCleanup(async () => { + const collection = createCollection({ + id: `native-byte-publication-${name}`, + getKey: ({ id }) => id, + autoIndex: `off`, + startSync: true, + sync: { + sync: (actions) => { + sync = actions + actions.begin() + actions.write({ + type: `insert`, + value: { id: `a`, value: before }, + }) + actions.commit() + actions.markReady() }, - { includeInitialState: false }, - ) - unsubscribe = () => subscription.unsubscribe() - expect(batches).toEqual([]) + }, + }) + cleanup = () => collection.cleanup() + await collection.stateWhenReady() + const index = collection.createIndex((row) => row.value, { + indexType: IndexType, + }) + const batches: Array< + Array<{ + type: string + key: string + previous: Array | undefined + value: Array + }> + > = [] + const subscription = collection.subscribeChanges( + (changes) => { + batches.push( + changes.map((change) => ({ + type: change.type, + key: change.key, + previous: + change.type === `update` + ? bytes(change.previousValue!.value) + : undefined, + value: bytes(change.value.value), + })), + ) + }, + { includeInitialState: false }, + ) + unsubscribe = () => subscription.unsubscribe() + expect(batches).toEqual([]) - sync.begin() - sync.write({ type: `update`, value: { id: `a`, value: after } }) - await sync.commit() - expect(bytes(collection.get(`a`)!.value)).toEqual([2]) - expect(index.equalityLookup(Uint8Array.of(1))).toEqual(new Set()) - expect(index.equalityLookup(Uint8Array.of(2))).toEqual(new Set([`a`])) - expect(batches).toEqual([ - [{ type: `update`, key: `a`, previous: [1], value: [2] }], - ]) + sync.begin() + sync.write({ type: `update`, value: { id: `a`, value: after } }) + await sync.commit() + expect(bytes(collection.get(`a`)!.value)).toEqual([2]) + expect(index.equalityLookup(Uint8Array.of(1))).toEqual(new Set()) + expect(index.equalityLookup(Uint8Array.of(2))).toEqual(new Set([`a`])) + expect(batches).toEqual([ + [{ type: `update`, key: `a`, previous: [1], value: [2] }], + ]) - sync.begin() - sync.write({ type: `update`, value: { id: `a`, value: equalCopy } }) - await sync.commit() - expect(bytes(collection.get(`a`)!.value)).toEqual([2]) - expect(index.equalityLookup(Uint8Array.of(2))).toEqual(new Set([`a`])) - expect(batches).toEqual([ - [{ type: `update`, key: `a`, previous: [1], value: [2] }], - ]) - }, [() => unsubscribe?.(), () => cleanup?.()]) - }, - ) + sync.begin() + sync.write({ type: `update`, value: { id: `a`, value: equalCopy } }) + await sync.commit() + expect(bytes(collection.get(`a`)!.value)).toEqual([2]) + expect(index.equalityLookup(Uint8Array.of(2))).toEqual(new Set([`a`])) + expect(batches).toEqual([ + [{ type: `update`, key: `a`, previous: [1], value: [2] }], + ]) + }, [() => unsubscribe?.(), () => cleanup?.()]) + }) }) describe(`opaque DataView change publication`, () => { @@ -1122,20 +1115,17 @@ describe(`native Date change publication across realms`, () => { * states. The native Date timestamp rule in `docs/guides/live-queries.md` * includes Dates from another JavaScript realm and invalid Dates. The model * knows the two supplied times and predicts one update exactly when they - * differ. One fixed pair has the same own string label on both Dates; it - * preserves a missed-update counterexample without defining behavior for - * differently labeled Dates. The driver uses real sync writes and a public + * differ. The driver uses real sync writes and a public * subscriber; it checks the callback and the settled commit. These fixed * histories do not claim optimistic writes, indexes, or other hosts. */ it.each([ { - name: `changed equally labeled Dates in one foreign realm`, + name: `changed Dates in one foreign realm`, beforeTime: 1, afterTime: 2, beforeRealm: `foreign`, afterRealm: `foreign`, - ownLabel: true, }, { name: `equal Dates in one foreign realm`, @@ -1143,7 +1133,6 @@ describe(`native Date change publication across realms`, () => { afterTime: 1, beforeRealm: `foreign`, afterRealm: `foreign`, - ownLabel: false, }, { name: `equal local-to-foreign Dates`, @@ -1151,7 +1140,6 @@ describe(`native Date change publication across realms`, () => { afterTime: 1, beforeRealm: `local`, afterRealm: `foreign`, - ownLabel: false, }, { name: `equal foreign-to-local Dates`, @@ -1159,7 +1147,6 @@ describe(`native Date change publication across realms`, () => { afterTime: 1, beforeRealm: `foreign`, afterRealm: `local`, - ownLabel: false, }, { name: `equal invalid local-to-foreign Dates`, @@ -1167,7 +1154,6 @@ describe(`native Date change publication across realms`, () => { afterTime: null, beforeRealm: `local`, afterRealm: `foreign`, - ownLabel: false, }, { name: `foreign invalid-to-valid Dates`, @@ -1175,7 +1161,6 @@ describe(`native Date change publication across realms`, () => { afterTime: 1, beforeRealm: `foreign`, afterRealm: `foreign`, - ownLabel: false, }, { name: `foreign valid-to-invalid Dates`, @@ -1183,7 +1168,6 @@ describe(`native Date change publication across realms`, () => { afterTime: null, beforeRealm: `foreign`, afterRealm: `foreign`, - ownLabel: false, }, ] as const)(`keeps the public mirror for $name`, async (sample) => { type DateRow = { id: string; value: Date } @@ -1198,14 +1182,6 @@ describe(`native Date change publication across realms`, () => { sample.beforeRealm === `foreign` ? foreignBefore : new Date(beforeTime) const after = sample.afterRealm === `foreign` ? foreignAfter : new Date(afterTime) - if (sample.ownLabel) { - for (const date of [before, after]) { - Object.defineProperty(date, `label`, { - value: `same`, - enumerable: true, - }) - } - } // The native intrinsic verifies the host supplied real Dates. The model // itself uses the constructor inputs, never the production classifier. expect(Date.prototype.getTime.call(before)).toEqual(beforeTime) diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index fdbce68577..8437082b62 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -1,4 +1,3 @@ -import { Buffer } from 'node:buffer' import { runInNewContext } from 'node:vm' import { describe, expect, it } from 'vitest' import { fc, test as fcTest } from '@fast-check/vitest' @@ -28,7 +27,7 @@ import type { CompareOptions } from '../src/query/builder/types' * position and follow finite values; null remains first. The published NaN * ordering change in `packages/db/CHANGELOG.md` (#1582) and the live-query * guide establish that law for Dates with native prototypes regardless of - * their JavaScript realm. + * their JavaScript realm in the currently broader Date-order lane. * * Authority: `OrderByOptions` and `CompareOptions` in the query builder's * option types declare these modes. The index comparator contract accepts @@ -40,9 +39,7 @@ import type { CompareOptions } from '../src/query/builder/types' * input tags and finite epoch-day values, rather than the production * unorderable classifier. It covers number-only and Date-only sorts with * optional null. A separate realm-tagged Date model below checks native Date - * timestamps and invalidity across realms, plus one public ordered scan. A - * fixed Date-subclass witness covers an inherited native `getTime` beyond the - * generated plain-Date grammar. + * timestamps and invalidity across realms, plus one public ordered scan. * Neither lane claims mixed-type order or every query/index path. Equality laws separately * cover the normalized value classes and changed-byte controls. Comparator * signs reject NaN and non-number results before reduction. Signed infinity @@ -54,20 +51,11 @@ import type { CompareOptions } from '../src/query/builder/types' * recorded in `packages/db/CHANGELOG.md` (#779). The bounded key-size and * indexed-byte checks retain the regression contract introduced with #1797; * they constrain this internal utility, not a public serialized format or a - * universal performance bound. Own `length` and `byteLength` properties do - * not change native byte content. The 32-character margin is the existing + * universal performance bound. The 32-character margin is the existing * constant-overhead budget for the sampled 0..200-byte keys. Changed-byte * equality and key histories cover lengths 1..200. Fixed changes at byte 64 * of a 129-byte view and the last byte of a 200-byte view pin positions that * a short generated campaign need not sample. - * - * The proposed native-byte ORDER BY law compares genuine Uint8Arrays and - * Buffers by unsigned byte content, with a shorter equal prefix first. Equal - * contents compare at one position. Genuine foreign-realm views and own - * length shadows retain their native bytes. The mixed-object witness checks - * total-order transitivity without choosing which kind comes first. This - * bounded direct-return witness does not claim relational WHERE comparison, - * other typed-array classes, arbitrary object order, or mutation. */ const defaultOpts: CompareOptions = { @@ -353,23 +341,13 @@ type RealmDateSample = { rightTime: number | null leftForeign: boolean rightForeign: boolean - leftOwnTag?: boolean - rightOwnTag?: boolean } -function dateInRealm( - time: number | null, - foreign: boolean, - ownTag = false, -): Date { +function dateInRealm(time: number | null, foreign: boolean): Date { const timestamp = time ?? Number.NaN - const date = foreign + return foreign ? (runInNewContext(`new Date(timestamp)`, { timestamp }) as Date) : new Date(timestamp) - if (ownTag) { - Object.defineProperty(date, Symbol.toStringTag, { value: `Other` }) - } - return date } // `null` is a model-only invalid-Date tag; production receives a real Date @@ -383,16 +361,8 @@ function modelDateSign(left: number | null, right: number | null): -1 | 0 | 1 { } function assertRealmDateOrder(sample: RealmDateSample): void { - const left = dateInRealm( - sample.leftTime, - sample.leftForeign, - sample.leftOwnTag, - ) - const right = dateInRealm( - sample.rightTime, - sample.rightForeign, - sample.rightOwnTag, - ) + const left = dateInRealm(sample.leftTime, sample.leftForeign) + const right = dateInRealm(sample.rightTime, sample.rightForeign) const expected = modelDateSign(sample.leftTime, sample.rightTime) const reverseExpected = expected === 0 ? 0 : -expected expect(isUnorderable(left)).toBe(sample.leftTime === null) @@ -412,10 +382,8 @@ function assertRealmDateOrder(sample: RealmDateSample): void { // Legal pairs cross local and foreign native Dates, valid epoch milliseconds // in -1e12..1e12 and invalid Dates. The fixed witnesses below force the // foreign-valid and foreign-invalid paths; generated pairs add equal times, -// both operand orders, both invalid values, and Dates with an own display tag. -// A display tag cannot change native Date identity. Proxies, non-Date objects, -// and user-defined Date subclasses are outside this generated grammar. The -// fixed witness below covers local and foreign subclasses with an own tag. +// both operand orders and both invalid values. Caller-modified built-ins, +// proxies, and user-defined Date subclasses are outside this grammar. const arbitraryDateTime = fc.oneof( fc.integer({ min: -1_000_000_000_000, max: 1_000_000_000_000 }), fc.constant(null), @@ -426,24 +394,18 @@ const arbitraryRealmDateSample: fc.Arbitrary = fc.oneof( rightTime: arbitraryDateTime, leftForeign: fc.boolean(), rightForeign: fc.boolean(), - leftOwnTag: fc.boolean(), - rightOwnTag: fc.boolean(), }), fc .record({ time: arbitraryDateTime, leftForeign: fc.boolean(), rightForeign: fc.boolean(), - leftOwnTag: fc.boolean(), - rightOwnTag: fc.boolean(), }) - .map(({ time, leftForeign, rightForeign, leftOwnTag, rightOwnTag }) => ({ + .map(({ time, leftForeign, rightForeign }) => ({ leftTime: time, rightTime: time, leftForeign, rightForeign, - leftOwnTag, - rightOwnTag, })), ) @@ -1011,217 +973,6 @@ describe(`NaN and invalid Date default-order refinement`, () => { }) describe(`native Date order across JavaScript realms`, () => { - it(`uses the native Date brand without reading an unrelated display tag`, () => { - const forged = { [Symbol.toStringTag]: `Date` } - let tagReads = 0 - const unrelated = Object.defineProperty({}, Symbol.toStringTag, { - get() { - tagReads++ - return `Other` - }, - }) - expect(isUnorderable(forged)).toBe(false) - expect(normalizeValue(forged)).toBe(forged) - expect( - sign(ascComparator(forged, new Date(0), DEFAULT_COMPARE_OPTIONS)), - ).not.toBe(0) - expect(isUnorderable(unrelated)).toBe(false) - expect(tagReads).toBe(0) - }) - - it(`keeps native Date semantics when a foreign Date has an own display tag`, () => { - const timestamp = 123 - const foreignValid = dateInRealm(timestamp, true, true) - const foreignInvalid = dateInRealm(null, true, true) - const foreignTemporalTagged = dateInRealm(timestamp, true) - Object.defineProperty(foreignTemporalTagged, Symbol.toStringTag, { - value: `Temporal.PlainDate`, - }) - const foreignPrototypeTagged = dateInRealm(timestamp, true) - Object.defineProperty( - Object.getPrototypeOf(foreignPrototypeTagged), - Symbol.toStringTag, - { value: `Other` }, - ) - const foreignGetter = dateInRealm(timestamp, true) - let tagReads = 0 - Object.defineProperty(foreignGetter, Symbol.toStringTag, { - get() { - tagReads++ - return `Other` - }, - }) - const localValid = new Date(timestamp) - // The intrinsic establishes the native Date premise without consulting - // the production detector or the object's display tag. - expect(Date.prototype.getTime.call(foreignValid)).toBe(timestamp) - expect(Date.prototype.getTime.call(foreignInvalid)).toBeNaN() - expect(normalizeValue(foreignValid)).toBe(timestamp) - expect(normalizeValue(foreignTemporalTagged)).toBe(timestamp) - expect(normalizeValue(foreignPrototypeTagged)).toBe(timestamp) - expect(normalizeValue(foreignGetter)).toBe(timestamp) - expect(tagReads).toBe(0) - expect(normalizeValue(foreignInvalid)).toBeNaN() - expect(isUnorderable(foreignInvalid)).toBe(true) - expect( - ascComparator(foreignValid, localValid, DEFAULT_COMPARE_OPTIONS), - ).toBe(0) - expect( - ascComparator(localValid, foreignValid, DEFAULT_COMPARE_OPTIONS), - ).toBe(0) - expect( - ascComparator(foreignTemporalTagged, localValid, DEFAULT_COMPARE_OPTIONS), - ).toBe(0) - expect( - ascComparator( - foreignPrototypeTagged, - localValid, - DEFAULT_COMPARE_OPTIONS, - ), - ).toBe(0) - expect( - sign(ascComparator(foreignInvalid, localValid, DEFAULT_COMPARE_OPTIONS)), - ).toBe(1) - }) - - it(`uses the native timestamp of a tagged foreign Date subclass`, () => { - class LocalDate extends Date {} - const timestamp = 123 - const localSubclass = new LocalDate(timestamp) - const foreignSubclass = runInNewContext( - `class D extends Date {}; new D(timestamp)`, - { timestamp }, - ) as Date - const localInvalid = new LocalDate(Number.NaN) - const foreignInvalid = runInNewContext( - `class D extends Date {}; new D(Number.NaN)`, - ) as Date - for (const value of [ - localSubclass, - foreignSubclass, - localInvalid, - foreignInvalid, - ]) { - Object.defineProperty(value, Symbol.toStringTag, { value: `Other` }) - } - // The supplied time and native slot establish the Date premise. The - // foreign subclass's immediate prototype inherits native getTime. - expect(Date.prototype.getTime.call(foreignSubclass)).toBe(timestamp) - expect(foreignSubclass instanceof Date).toBe(false) - expect( - Object.getOwnPropertyDescriptor( - Object.getPrototypeOf(foreignSubclass), - `getTime`, - ), - ).toBeUndefined() - - const assertEarlierDateOrder = ( - value: Date, - compare: typeof ascComparator = ascComparator, - ) => { - expect( - sign(compare(value, new Date(timestamp - 1), DEFAULT_COMPARE_OPTIONS)), - ).toBe(1) - expect( - sign(compare(new Date(timestamp - 1), value, DEFAULT_COMPARE_OPTIONS)), - ).toBe(-1) - } - - for (const value of [localSubclass, foreignSubclass]) { - expect(normalizeValue(value)).toBe(timestamp) - expect( - ascComparator(value, new Date(timestamp), DEFAULT_COMPARE_OPTIONS), - ).toBe(0) - expect( - ascComparator(new Date(timestamp), value, DEFAULT_COMPARE_OPTIONS), - ).toBe(0) - // A subclass-first comparator would put this value before both - // unequal Dates. The earlier Date must instead precede it. - assertEarlierDateOrder(value) - expect( - sign( - ascComparator( - value, - new Date(timestamp + 1), - DEFAULT_COMPARE_OPTIONS, - ), - ), - ).toBe(-1) - expect( - sign( - ascComparator( - new Date(timestamp + 1), - value, - DEFAULT_COMPARE_OPTIONS, - ), - ), - ).toBe(1) - } - // Feed a subclass-first answer through the same earlier-Date comparison. - // It reaches the comparison and fails at the expected sign assertion. - let reachedWrongOrder = false - expect(() => - assertEarlierDateOrder(foreignSubclass, (left, right, options) => { - if (left === foreignSubclass && right instanceof Date) { - reachedWrongOrder = true - return -1 - } - return ascComparator(left, right, options) - }), - ).toThrowError() - expect(reachedWrongOrder).toBe(true) - - for (const invalid of [localInvalid, foreignInvalid]) { - expect(normalizeValue(invalid)).toBeNaN() - expect(isUnorderable(invalid)).toBe(true) - expect(ascComparator(invalid, Number.NaN, DEFAULT_COMPARE_OPTIONS)).toBe( - 0, - ) - expect( - sign( - ascComparator(invalid, new Date(timestamp), DEFAULT_COMPARE_OPTIONS), - ), - ).toBe(1) - expect( - sign( - ascComparator(new Date(timestamp), invalid, DEFAULT_COMPARE_OPTIONS), - ), - ).toBe(-1) - } - - // Borrowing that prototype and display tag cannot forge native Date - // slots. The intrinsic rejects this adjacent non-Date control. - const forged = Object.create(Object.getPrototypeOf(foreignSubclass)) as Date - Object.defineProperty(forged, Symbol.toStringTag, { value: `Other` }) - expect(() => Date.prototype.getTime.call(forged)).toThrow(TypeError) - expect(normalizeValue(forged)).toBe(forged) - expect(isUnorderable(forged)).toBe(false) - const forward = sign( - ascComparator(forged, foreignSubclass, DEFAULT_COMPARE_OPTIONS), - ) - const reverse = sign( - ascComparator(foreignSubclass, forged, DEFAULT_COMPARE_OPTIONS), - ) - expect(forward).not.toBe(0) - expect(reverse).toBe(-forward) - }) - - it(`rejects a local Date subclass prototype without a native Date slot`, () => { - class LocalDate extends Date {} - const forged = Object.create(LocalDate.prototype) as object - const real = new Date(1) - // instanceof sees the borrowed local prototype, but the Date intrinsic - // rejects the missing native slot. The object keeps reference identity. - expect(forged instanceof Date).toBe(true) - expect(() => Date.prototype.getTime.call(forged)).toThrow(TypeError) - expect(normalizeValue(forged)).toBe(forged) - expect(isUnorderable(forged)).toBe(false) - const forward = sign(ascComparator(forged, real, DEFAULT_COMPARE_OPTIONS)) - const reverse = sign(ascComparator(real, forged, DEFAULT_COMPARE_OPTIONS)) - expect(forward).not.toBe(0) - expect(reverse).toBe(-forward) - }) - it(`treats numeric NaN and invalid Dates as one non-finite order position`, () => { for (const foreign of [false, true]) { const invalid = dateInRealm(null, foreign) @@ -1269,15 +1020,11 @@ describe(`native Date order across JavaScript realms`, () => { ) it(`orders a foreign invalid Date after a valid Date in a public scan`, async () => { - for (const [foreign, ownTag] of [ - [false, false], - [true, false], - [true, true], - ] as const) { - const invalid = dateInRealm(null, foreign, ownTag) + for (const foreign of [false, true]) { + const invalid = dateInRealm(null, foreign) const valid = new Date(Date.UTC(2023, 0, 1)) const source = createCollection({ - id: `realm-date-order-${foreign}-${ownTag}`, + id: `realm-date-order-${foreign}`, getKey: (row: { id: number; when: Date }) => row.id, autoIndex: `off`, sync: { @@ -1403,28 +1150,6 @@ describe(`makeComparator property-based tests`, () => { }) describe(`normalizeValue property-based tests`, () => { - it(`binary keys ignore own byte-count shadows`, () => { - // Supplied native bytes, rather than public count properties, establish - // the key relation. The function return is the comparison checkpoint. - const equalKey = normalizeValue(new Uint8Array([1, 2])) - const unequalKey = normalizeValue(new Uint8Array([9, 9])) - expect(equalKey).not.toBe(unequalKey) - for (const field of [`length`, `byteLength`] as const) { - const equal = Object.defineProperty(new Uint8Array([1, 2]), field, { - value: 0, - }) - const unequal = Object.defineProperty(new Uint8Array([9, 9]), field, { - value: 0, - }) - expect(equal[field]).toBe(0) - expect(unequal[field]).toBe(0) - expect(Array.from(equal)).toEqual([1, 2]) - expect(Array.from(unequal)).toEqual([9, 9]) - expect(normalizeValue(equal)).toBe(equalKey) - expect(normalizeValue(unequal)).toBe(unequalKey) - } - }) - it(`binary key size and content laws reach the bounded length margins`, () => { for (const length of [0, 1, 128, 129, 200]) { const bytes = Uint8Array.from({ length }, (_, index) => index % 256) @@ -1484,17 +1209,6 @@ describe(`normalizeValue property-based tests`, () => { expect(normalizeValue(normalized)).not.toBe(normalized) }, ) - - fcTest(`reads binary keys from indexed bytes, not custom iteration`, () => { - const bytes = new Uint8Array([2]) - Object.defineProperty(bytes, Symbol.iterator, { - value: function* () { - yield 1 - }, - }) - - expect(normalizeValue(bytes)).toBe(normalizeValue(new Uint8Array([2]))) - }) }) describe(`areValuesEqual property-based tests`, () => { @@ -1849,146 +1563,6 @@ describe(`native binary key content`, () => { ) }) -describe(`native byte content order`, () => { - // The independent rule reads only the supplied unsigned elements. It does - // not use the production object-identity allocator, comparator, or exact-key - // normalization. Each fresh pair exercises immediate comparator returns. - const model = (left: Uint8Array, right: Uint8Array): number => { - // Native iteration reads the byte slots even when an own `length` - // property shadows the public getter. - const leftBytes = Array.from(Uint8Array.prototype.values.call(left)) - const rightBytes = Array.from(Uint8Array.prototype.values.call(right)) - for ( - let index = 0; - index < Math.min(leftBytes.length, rightBytes.length); - index++ - ) { - const difference = leftBytes[index]! - rightBytes[index]! - if (difference !== 0) return difference - } - return leftBytes.length - rightBytes.length - } - - it.each([ - { - caseId: `equal-content distinct views`, - makePair: () => [Uint8Array.of(1), Uint8Array.of(1)] as const, - expectedSign: 0, - }, - { - caseId: `first differing unsigned byte`, - makePair: () => [Uint8Array.of(1, 255), Uint8Array.of(2, 0)] as const, - expectedSign: -1, - }, - { - caseId: `later differing byte with equal first byte`, - makePair: () => [Uint8Array.of(1, 2), Uint8Array.of(1, 3)] as const, - expectedSign: -1, - }, - { - caseId: `unsigned 127/128 boundary`, - makePair: () => [Uint8Array.of(127), Uint8Array.of(128)] as const, - expectedSign: -1, - }, - { - caseId: `foreign native byte view`, - makePair: () => - [ - runInNewContext(`new Uint8Array([1])`) as Uint8Array, - Uint8Array.of(2), - ] as const, - expectedSign: -1, - }, - { - caseId: `own length shadow`, - makePair: () => - [ - Uint8Array.of(1), - Object.defineProperty(Uint8Array.of(2), `length`, { value: 0 }), - ] as const, - expectedSign: -1, - }, - { - caseId: `own byteLength shadow`, - makePair: () => - [ - Uint8Array.of(1), - Object.defineProperty(Uint8Array.of(2), `byteLength`, { value: 0 }), - ] as const, - expectedSign: -1, - }, - { - caseId: `shorter equal prefix`, - makePair: () => [Uint8Array.of(1), Uint8Array.of(1, 0)] as const, - expectedSign: -1, - }, - { - caseId: `Buffer and Uint8Array with equal content`, - makePair: () => [Uint8Array.of(1), Buffer.from([1])] as const, - expectedSign: 0, - }, - { - caseId: `Buffer and Uint8Array with differing content`, - makePair: () => [Uint8Array.of(1), Buffer.from([2])] as const, - expectedSign: -1, - }, - ])(`compares $caseId`, ({ makePair, expectedSign }) => { - const [lessOrEqual, greaterOrEqual] = makePair() - expect(sign(model(lessOrEqual, greaterOrEqual))).toBe(expectedSign) - const reverseSign = expectedSign === 0 ? 0 : -expectedSign - - // Call the reverse direction first. An identity-order implementation - // assigns its first runtime ID to the greater operand here; content order - // must still return the opposite sign. Both directions and descending - // have the same immediate checkpoint. - expect - .soft(sign(ascComparator(greaterOrEqual, lessOrEqual, defaultOpts))) - .toBe(reverseSign) - expect - .soft(sign(ascComparator(lessOrEqual, greaterOrEqual, defaultOpts))) - .toBe(expectedSign) - expect - .soft(sign(descComparator(lessOrEqual, greaterOrEqual, defaultOpts))) - .toBe(reverseSign) - }) - - comparisonProperty([ - fc.array(fc.integer({ min: 0, max: 255 }), { - minLength: 2, - maxLength: 24, - }), - fc.integer({ min: 0, max: 254 }), - fc.array(fc.integer({ min: 0, max: 255 }), { maxLength: 8 }), - ])( - `compares the first differing byte after a common prefix`, - (prefix, lowerByte, suffix) => { - // The prefix varies the first-difference position. A comparator that - // inspects only one or two leading bytes agrees on the pinned examples - // but disagrees with this independent content rule. - const lower = Uint8Array.from([...prefix, lowerByte, ...suffix]) - const higher = Uint8Array.from([...prefix, lowerByte + 1, ...suffix]) - expect(sign(model(lower, higher))).toBe(-1) - expect(sign(ascComparator(higher, lower, defaultOpts))).toBe(1) - expect(sign(ascComparator(lower, higher, defaultOpts))).toBe(-1) - expect(sign(descComparator(lower, higher, defaultOpts))).toBe(1) - }, - ) - - it(`keeps byte content order transitive through ordinary objects`, () => { - const high = Uint8Array.of(2) - const ordinary = {} - const low = Uint8Array.of(1) - // These two mixed comparisons assign identity IDs in an implementation - // that wrongly groups bytes with ordinary objects. The content edge then - // closes a strict cycle: high < ordinary < low < high. - const highToOrdinary = ascComparator(high, ordinary, defaultOpts) - const ordinaryToLow = ascComparator(ordinary, low, defaultOpts) - const highToLow = ascComparator(high, low, defaultOpts) - expect(highToLow).toBeGreaterThan(0) - assertTransitive(highToOrdinary, ordinaryToLow, highToLow) - }) -}) - if (requestedProperty !== undefined) { it(`selects a registered comparison oracle property`, () => { expect(requestedProperty).toMatch(/^\d+$/) diff --git a/packages/db/tests/custom-index-order-selection-oracle.test.ts b/packages/db/tests/custom-index-order-selection-oracle.test.ts index 7e2711f955..a67973c1d6 100644 --- a/packages/db/tests/custom-index-order-selection-oracle.test.ts +++ b/packages/db/tests/custom-index-order-selection-oracle.test.ts @@ -1,5 +1,4 @@ import { Buffer } from 'node:buffer' -import { runInNewContext } from 'node:vm' import { describe, expect, it, vi } from 'vitest' import { createCollection } from '../src/collection/index.js' import { BasicIndex } from '../src/indexes/basic-index.js' @@ -36,15 +35,13 @@ import type { SyncConfig } from '../src/types.js' * checks initial and cursor snapshots of an equal-rank group. Raw escaped * strings must reach custom-collation callbacks without their exact-key * encoding; direct range and cursor operations use those same positions. - * This PR's proposed default native-byte order is unsigned lexicographic by - * content, with shorter prefixes first and source keys ascending at equal - * content. A mixed string, number, and byte fixture checks that order beside - * content-based exact equality. - * Bounded source histories replace one byte view with a fresh equal-content - * view, or another row's existing view, then delete and reinsert its key. - * They compare indexed and unindexed order with a reference based on source - * facts; exact membership comes only from byte content. Equal-content - * replacement keeps ordinary public order and publishes no source change. + * Native byte views retain content-based exact equality, and indexed public + * order agrees with the unindexed scan without promising a particular default + * byte-content order. A source history replaces one byte view with another + * row's value, then deletes and reinserts its key. Exact membership follows + * byte content, while public scan and indexed reads have the same complete + * order and conserve current keys. A `coalesce` wrapper around the scan's + * non-null byte field prevents the single-ref index shortcut. * The maintainer's custom-comparator decision also requires each current row * to follow the callback when equal-content views have distinct callback ranks. * They do not claim arbitrary mutation histories, provider ordering, arbitrary @@ -168,27 +165,6 @@ const indexTypes = [ [`BTreeIndex`, BTreeIndex], ] as const -// For genuine native byte views in this bounded domain, the first differing -// unsigned byte determines order; an exhausted prefix comes first. This model -// reads the supplied bytes and never calls the production comparator or index. -function compareByteContent(left: Uint8Array, right: Uint8Array): number { - const leftBytes = Array.from(Uint8Array.prototype.values.call(left)) - const rightBytes = Array.from(Uint8Array.prototype.values.call(right)) - for ( - let index = 0; - index < Math.min(leftBytes.length, rightBytes.length); - index++ - ) { - const difference = leftBytes[index]! - rightBytes[index]! - if (difference !== 0) return difference - } - return leftBytes.length - rightBytes.length -} - -function compareStringKeys(left: string, right: string): number { - return left < right ? -1 : left > right ? 1 : 0 -} - describe.each(indexTypes)( `%s custom comparator order selection`, (name, IndexType) => { @@ -1050,359 +1026,6 @@ describe.each(indexTypes)(`%s public key ties`, (name, IndexType) => { expect(index.equalityLookup(literal)).toEqual(new Set([`literal`])) }) - it.each([ - { - caseId: `equal-content-key-tie`, - makeRows: () => { - const z = new Uint8Array([1]) - const a = new Uint8Array([1]) - return [ - { id: `a`, value: a }, - { id: `z`, value: z }, - ] - }, - expected: [`a`, `z`], - }, - { - caseId: `first-differing-byte`, - makeRows: () => { - const high = new Uint8Array([2, 0]) - const low = new Uint8Array([1, 255]) - return [ - { id: `high`, value: high }, - { id: `low`, value: low }, - ] - }, - expected: [`low`, `high`], - }, - { - caseId: `later-differing-byte`, - makeRows: () => [ - { id: `a`, value: Uint8Array.of(1, 3) }, - { id: `z`, value: Uint8Array.of(1, 2) }, - ], - expected: [`z`, `a`], - }, - { - caseId: `third-differing-byte`, - makeRows: () => [ - { id: `a`, value: Uint8Array.of(1, 1, 3) }, - { id: `z`, value: Uint8Array.of(1, 1, 2) }, - ], - expected: [`z`, `a`], - }, - { - caseId: `unsigned-127-128-boundary`, - makeRows: () => { - const low = Uint8Array.of(127) - const high = Uint8Array.of(128) - return [ - { id: `low`, value: low }, - { id: `high`, value: high }, - ] - }, - expected: [`low`, `high`], - }, - { - caseId: `foreign-native-byte-view`, - makeRows: () => { - const low = runInNewContext(`new Uint8Array([1])`) as Uint8Array - const high = Uint8Array.of(2) - return [ - { id: `foreign-low`, value: low }, - { id: `local-high`, value: high }, - ] - }, - expected: [`foreign-low`, `local-high`], - }, - { - caseId: `own-length-shadow`, - makeRows: () => { - const low = Uint8Array.of(1) - const high = Object.defineProperty(Uint8Array.of(2), `length`, { - value: 0, - }) - return [ - { id: `low`, value: low }, - { id: `high`, value: high }, - ] - }, - expected: [`low`, `high`], - }, - { - caseId: `own-byte-length-shadow`, - makeRows: () => [ - { id: `low`, value: Uint8Array.of(1) }, - { - id: `high`, - value: Object.defineProperty(Uint8Array.of(2), `byteLength`, { - value: 0, - }), - }, - ], - expected: [`low`, `high`], - }, - { - caseId: `shorter-prefix`, - makeRows: () => { - const long = new Uint8Array([1, 0]) - const short = new Uint8Array([1]) - return [ - { id: `long`, value: long }, - { id: `short`, value: short }, - ] - }, - expected: [`short`, `long`], - }, - { - caseId: `buffer-and-uint8array`, - makeRows: () => { - const buffer = Buffer.from([1]) - const view = new Uint8Array([1]) - return [ - { id: `a-view`, value: view }, - { id: `z-buffer`, value: buffer }, - ] - }, - expected: [`a-view`, `z-buffer`], - }, - ])(`orders native bytes by content / $caseId`, async (example) => { - type ByteRow = { id: string; value: Uint8Array } - const source: ReadonlyArray = example.makeRows() - // The independent model sorts supplied unsigned content and uses the - // source key only for content-equal values. The fixtures include orders - // that disagree with reference identity, signed bytes, and host collation. - const model = [...source] - .sort( - (left, right) => - compareByteContent(left.value, right.value) || - compareStringKeys(left.id, right.id), - ) - .map(({ id }) => id) - expect(model).toEqual(example.expected) - let cleanup: (() => unknown) | undefined - await withOracleCleanup(async () => { - const collection = createCollection({ - id: `byte-content-${name}-${example.caseId}`, - getKey: ({ id }) => id, - autoIndex: `off`, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of source) write({ type: `insert`, value: row }) - commit() - markReady() - }, - }, - }) - cleanup = () => collection.cleanup() - await collection.stateWhenReady() - const orderBy = [ - { - expression: new PropRef([`value`]), - compareOptions: DEFAULT_COMPARE_OPTIONS, - }, - ] - const read = (optimizedOnly?: boolean) => - collection - .currentStateAsChanges({ orderBy, optimizedOnly }) - ?.map(({ key }) => key) - - // Read before index installation, then force the installed index path. - // Exact arrays preserve both order and multiplicity at readiness. - expect.soft(read(), `unindexed ${example.caseId}`).toEqual(model) - const index = collection.createIndex((row) => row.value, { - indexType: IndexType, - }) - const take = vi.spyOn(index, `takeFromStart`) - expect.soft(read(true), `indexed ${example.caseId}`).toEqual(model) - expect(take).toHaveBeenCalled() - expect.soft(index.takeFromStart(source.length)).toEqual(model) - expect(index.keyCount).toBe(source.length) - for (const row of source) { - expect( - index.equalityLookup( - Uint8Array.from(Uint8Array.prototype.values.call(row.value)), - ), - ).toEqual( - new Set( - source - .filter( - (other) => compareByteContent(other.value, row.value) === 0, - ) - .map(({ id }) => id), - ), - ) - } - }, [() => cleanup?.()]) - }) - - it(`keeps mixed-value byte order and content equality aligned`, async () => { - type MixedRow = { id: string; value: string | number | Uint8Array } - const firstByte = new Uint8Array([1]) - const secondByte = new Uint8Array([1]) - const source: ReadonlyArray = [ - { id: `s`, value: `z` }, - { id: `n`, value: 1 }, - { id: `z-byte`, value: firstByte }, - { id: `a-byte`, value: secondByte }, - ] - let cleanup: (() => unknown) | undefined - await withOracleCleanup(async () => { - const collection = createCollection({ - id: `mixed-byte-order-${name}`, - getKey: ({ id }) => id, - autoIndex: `off`, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of source) write({ type: `insert`, value: row }) - commit() - markReady() - }, - }, - }) - cleanup = () => collection.cleanup() - await collection.stateWhenReady() - const orderBy = [ - { - expression: new PropRef([`value`]), - compareOptions: DEFAULT_COMPARE_OPTIONS, - }, - ] - const read = (optimizedOnly?: boolean) => - collection - .currentStateAsChanges({ orderBy, optimizedOnly }) - ?.map(({ key }) => key) - - // The value-kind law puts string and number before native bytes. The - // two byte views have equal content, so public source keys break their - // tie even though source admission put z-byte first. - const scanned = read() - expect(scanned).toEqual([`s`, `n`, `a-byte`, `z-byte`]) - const index = collection.createIndex((row) => row.value, { - indexType: IndexType, - }) - const take = vi.spyOn(index, `takeFromStart`) - expect(read(true)).toEqual(scanned) - expect(take).toHaveBeenCalled() - expect(index.equalityLookup(new Uint8Array([1]))).toEqual( - new Set([`z-byte`, `a-byte`]), - ) - expect(index.equalityLookup(`\u0000tanstack-db:binary:\u0001`)).toEqual( - new Set(), - ) - }, [() => cleanup?.()]) - }) - - it(`keeps a content-equal replacement in key order without a source event`, async () => { - type ByteRow = { id: string; value: Uint8Array } - type Sync = Parameters[`sync`]>[0] - const firstKey = `A` - const peerKey = `a` - const first = new Uint8Array([1]) - const peer = new Uint8Array([1]) - const replacement = new Uint8Array([1]) - const source = new Map([ - [peerKey, peer], - [firstKey, first], - ]) - let sync!: Sync - let cleanup: (() => unknown) | undefined - let unsubscribe: (() => unknown) | undefined - await withOracleCleanup(async () => { - const collection = createCollection({ - id: `byte-replacement-order-${name}`, - getKey: ({ id }) => id, - autoIndex: `off`, - startSync: true, - sync: { - sync: (actions) => { - sync = actions - actions.begin() - for (const [id, value] of source) - actions.write({ type: `insert`, value: { id, value } }) - actions.commit() - actions.markReady() - }, - }, - }) - cleanup = () => collection.cleanup() - await collection.stateWhenReady() - const orderBy = [ - { - expression: new PropRef([`value`]), - compareOptions: DEFAULT_COMPARE_OPTIONS, - }, - ] - const read = (optimizedOnly?: boolean) => - collection - .currentStateAsChanges({ orderBy, optimizedOnly }) - ?.map(({ key }) => key) - - // The plain source Map is the model. Every live value has byte content - // [1], so the public order is the source-key tie regardless of which - // native view supplies that content. The change-event equality law - // suppresses a source event for the equal-content replacement. - const model = () => - [...source] - .sort( - ([leftKey, leftValue], [rightKey, rightValue]) => - compareByteContent(leftValue, rightValue) || - compareStringKeys(leftKey, rightKey), - ) - .map(([key]) => key) - expect(model()).toEqual([firstKey, peerKey]) - expect.soft(read()).toEqual(model()) - const index = collection.createIndex((row) => row.value, { - indexType: IndexType, - }) - const take = vi.spyOn(index, `takeFromStart`) - expect.soft(read(true)).toEqual(model()) - expect(take).toHaveBeenCalled() - expect.soft(index.takeFromStart(2)).toEqual(model()) - expect(index.keyCount).toBe(source.size) - expect(index.equalityLookup(new Uint8Array([1]))).toEqual( - new Set(source.keys()), - ) - const events: Array<{ type: string; key: string }> = [] - const subscription = collection.subscribeChanges( - (changes) => { - events.push(...changes.map(({ type, key }) => ({ type, key }))) - }, - { includeInitialState: false }, - ) - unsubscribe = () => subscription.unsubscribe() - expect(events).toEqual([]) - - // Commit changes the retained source instance while preserving its - // content. The indexed read and scan must both retain source-key order; - // neither a duplicate key nor a value event is allowed at this cut. - sync.begin() - sync.write({ - type: `update`, - value: { id: firstKey, value: replacement }, - }) - source.set(firstKey, replacement) - await sync.commit() - expect(collection.state.get(firstKey)?.value).toBe(replacement) - expect(events).toEqual([]) - const indexed = read(true) - const direct = index.takeFromStart(2) - expect(index.keyCount).toBe(source.size) - expect(index.equalityLookup(new Uint8Array([1]))).toEqual( - new Set(source.keys()), - ) - expect(collection.removeIndex(index)).toBe(true) - const scanned = read() - expect.soft(scanned).toEqual(model()) - expect.soft(indexed).toEqual(model()) - expect.soft(direct).toEqual(model()) - }, [() => unsubscribe?.(), () => cleanup?.()]) - }) - it(`conserves keys when a byte view adopts a peer value and is reinserted`, async () => { type ByteRow = { id: string; value: Uint8Array } type Sync = Parameters[`sync`]>[0] @@ -1439,26 +1062,24 @@ describe.each(indexTypes)(`%s public key ties`, (name, IndexType) => { compareOptions: DEFAULT_COMPARE_OPTIONS, }, ] - const read = (optimizedOnly?: boolean) => + const scanOrderBy = [ + { + expression: new Func(`coalesce`, [new PropRef([`value`])]), + compareOptions: DEFAULT_COMPARE_OPTIONS, + }, + ] + const readScan = () => + collection + .currentStateAsChanges({ orderBy: scanOrderBy }) + ?.map(({ key }) => key) + const readIndexed = (optimizedOnly?: boolean) => collection .currentStateAsChanges({ orderBy, optimizedOnly }) ?.map(({ key }) => key) - // A plain Map models source row ownership. Unsigned content puts every - // [1] view before [2], regardless of its native class or identity; - // source keys break the [1] tie. Adopting the peer value, deleting, and - // reinserting key 0 change only current membership. This reference never - // reads an index bucket or the production comparator. - const expectedOrder = () => - [...source] - .sort( - ([leftKey, leftValue], [rightKey, rightValue]) => - compareByteContent(leftValue, rightValue) || - compareStringKeys(leftKey, rightKey), - ) - .map(([key]) => key) - expect(expectedOrder()).toEqual([`0`, `1`, `3`, `2`]) - expect.soft(read(), `initial unindexed order`).toEqual(expectedOrder()) + // The source Map independently determines membership. No law chooses + // the relative default order of two distinct byte views, so compare + // index output with the public scan at each observation checkpoint. const expectedExact = (byte: number) => new Set( [...source] @@ -1468,11 +1089,22 @@ describe.each(indexTypes)(`%s public key ties`, (name, IndexType) => { const index = collection.createIndex((row) => row.value, { indexType: IndexType, }) + const indexedTake = vi.spyOn(index, `takeFromStart`) const check = (phase: string) => { - const expected = expectedOrder() + indexedTake.mockClear() + const expected = readScan()! + expect( + indexedTake, + `${phase} scan bypasses index`, + ).not.toHaveBeenCalled() + expect(new Set(expected), `${phase} scan keys`).toEqual( + new Set(source.keys()), + ) + expect(expected.length, `${phase} scan multiplicity`).toBe(source.size) expect - .soft(read(true), `${phase} public indexed order`) + .soft(readIndexed(true), `${phase} public indexed order`) .toEqual(expected) + expect(indexedTake, `${phase} indexed traversal ran`).toHaveBeenCalled() expect .soft( index.takeFromStart(source.size + 1), @@ -1512,9 +1144,9 @@ describe.each(indexTypes)(`%s public key ties`, (name, IndexType) => { check(`after reinsert`) // Removing the index makes the same public ordered read use its scan. - // Its final keys are an independent receiving control for the history. + // Its exact final order remains an independent receiving control. expect(collection.removeIndex(index)).toBe(true) - expect(read()).toEqual(expectedOrder()) + expect(readIndexed()).toEqual(readScan()) }, [() => cleanup?.()]) }) diff --git a/packages/db/tests/query/compiler/binary-equality-work.test.ts b/packages/db/tests/query/compiler/binary-equality-work.test.ts index ddbbeed670..10e9d62a95 100644 --- a/packages/db/tests/query/compiler/binary-equality-work.test.ts +++ b/packages/db/tests/query/compiler/binary-equality-work.test.ts @@ -66,20 +66,25 @@ describe(`binary equality work`, () => { ) it.each([`eq`, `in`])( - `compares a MiB using %s without caching mutable bytes`, + `compares distinct MiB values using %s without string encoding`, (operator) => { const left = new Uint8Array(1024 * 1024).fill(65) const right = left.slice() + const list = [right] const evaluate = compileSingleRowExpression( new Func(operator, [ new PropRef([`blob`]), - new Value(operator === `in` ? [right] : right), + new Value(operator === `in` ? list : right), ]), ) const equal = measureEncoding(() => evaluate({ blob: left })) expect(equal).toEqual({ result: true, bytes: 0 }) - right[right.length - 1] = 66 - const different = measureEncoding(() => evaluate({ blob: left })) + const other = left.slice() + other[other.length - 1] = 66 + if (operator === `in`) list[0] = other + const different = measureEncoding(() => + evaluate({ blob: operator === `in` ? left : other }), + ) expect(different).toEqual({ result: false, bytes: 0 }) }, ) diff --git a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts index 914e04a94c..b3b6310281 100644 --- a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts +++ b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts @@ -11,9 +11,9 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * * Transfer law: The public `inArray` API describes array membership. A * compiled expression holding a caller-owned array reads that array's current - * members whenever it evaluates, even if the array or one of its items has - * changed since compilation. An own `some` method is not a list member and - * cannot decide membership. The current evaluator returns UNKNOWN (`null`) + * members whenever it evaluates, including entries replaced since compilation. + * Mutating a Date or byte-array item in place is outside this law. The current + * evaluator returns UNKNOWN (`null`) * for a null or undefined value and FALSE for a list that is not an array. * Otherwise the result is TRUE when some list item equals the value under the * evaluator's equality, and FALSE when none does. A null or undefined item @@ -24,10 +24,9 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * * - `NaN` and an invalid Date equal each other and nothing else. * - A native Date from either realm equals its millisecond timestamp, and - * another Date with the same timestamp. A borrowed Date prototype does not - * give an ordinary object native Date identity. - * - Two Uint8Arrays (Buffers included) are equal when their native bytes are - * equal. Own `length` or `byteLength` properties cannot change that result. + * another Date with the same timestamp. + * - Two ordinary Uint8Arrays (Buffers included) are equal when their bytes are + * equal. * - Temporal kinds with an intrinsic `.equals()` use that relation. This * includes equivalent time-zone aliases for ZonedDateTime, but excludes a * distinct fixed-offset zone. PlainMonthDay and PlainYearMonth also retain @@ -40,27 +39,24 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * * Model: `referenceIn` walks present indices through the length captured at * the start of evaluation and applies `referenceEqual`, which restates the - * rules above. It does not call the array's overridable methods and imports - * no production comparison or normalization helper. + * rules above. It imports no production comparison or normalization helper. * * Domain: values and items come from a pool that contains the boundaries * of each rule: equal and unequal strings, a string that begins with the * internal normalization prefix, booleans, `0`, `-0`, `1`, `NaN`, `1n`, a * local and foreign valid Dates and their timestamps, local and foreign - * invalid Dates, borrowed Date prototypes, equal and unequal byte - * arrays, a Buffer, native bytes from another realm with equal and unequal - * content, genuine local and foreign bytes with own display-tag shadows, bytes - * with own `length` or `byteLength` shadows, and a wide typed array with a - * spoofed byte tag. A fixed control also shadows byte iteration. Temporal + * invalid Dates, equal and unequal byte arrays, a Buffer, and native bytes + * from another realm with equal and unequal content. Temporal * values include ZonedDateTime zone aliases, unequal values within each * modeled Temporal kind, reference ISO year/day * differences hidden by default display, and Duration carry forms with equal * display but distinct fields, one shared object and a structurally equal * copy, `null` and `undefined`. Lists have zero to six * items and may repeat. The list is a constant `Value`, the shape that join - * demand builds, or a row field. Fixed histories also compile a caller-owned - * list, then replace a slot or mutate a Date or byte item before reevaluating. - * A fixed history also compiles, shadows `some`, and edits a list slot. + * demand builds, or a row field. A fixed immutable primitive list checks the + * compiled constant-list path against the same independent membership model. + * Fixed histories also compile a caller-owned list, then replace a scalar, + * Date, or byte-array slot before reevaluating. * * Production driver: compile `in` with the public IR and evaluate it for * each row, as a WHERE clause does. @@ -73,8 +69,8 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * * Limits: this owner checks the boolean result of `in` for ordinary values * from one Temporal polyfill copy. It does not claim mixed Temporal - * implementations, malformed Temporal lookalikes, deliberately shadowed - * Temporal members, or replaced genuine prototypes. WHERE + * implementations, caller-modified built-ins, malformed Temporal lookalikes, + * deliberately shadowed Temporal members, or replaced genuine prototypes. WHERE * publication, indexes, and subset loading have other owners. */ @@ -97,44 +93,10 @@ const foreignDate = runInNewContext(`new Date(1000)`) as Date const laterForeignDate = runInNewContext(`new Date(1001)`) as Date const localInvalidDate = new Date(Number.NaN) const foreignInvalidDate = runInNewContext(`new Date(NaN)`) as Date -const borrowedDate = Object.create(Date.prototype) as object -const otherBorrowedDate = Object.create(Date.prototype) as object -const shadowedTagBytes = new Uint8Array([1, 2]) -Object.defineProperty(shadowedTagBytes, Symbol.toStringTag, { - value: `ordinary`, -}) const foreignBytes = runInNewContext(`new Uint8Array([1, 2])`) as Uint8Array const unequalForeignBytes = runInNewContext( `new Uint8Array([1, 3])`, ) as Uint8Array -const shadowedForeignBytes = runInNewContext( - `Object.defineProperty(new Uint8Array([1, 2]), Symbol.toStringTag, { value: 'ordinary' })`, -) as Uint8Array -const byteLengthShadowedForeignBytes = Object.defineProperty( - runInNewContext(`new Uint8Array([1, 2])`) as Uint8Array, - `byteLength`, - { value: 0 }, -) -function shadowByteCount( - bytes: ReadonlyArray, - field: `length` | `byteLength`, -): Uint8Array { - return Object.defineProperty(Uint8Array.from(bytes), field, { value: 0 }) -} -const byteLengthShadowed = shadowByteCount([1, 2], `byteLength`) -const equalByteLengthShadowed = shadowByteCount([1, 2], `byteLength`) -const unequalByteLengthShadowed = shadowByteCount([9, 9], `byteLength`) -const lengthShadowed = shadowByteCount([1, 2], `length`) -const equalLengthShadowed = shadowByteCount([1, 2], `length`) -const unequalLengthShadowed = shadowByteCount([9, 9], `length`) -const spoofedWideBytes = new Uint16Array([1, 2]) -Object.defineProperty(spoofedWideBytes, Symbol.toStringTag, { - value: `Uint8Array`, -}) -const otherSpoofedWideBytes = new Uint16Array([1, 2]) -Object.defineProperty(otherSpoofedWideBytes, Symbol.toStringTag, { - value: `Uint8Array`, -}) const samePlainDate = Temporal.PlainDate.from(`2024-01-01`) const leapMonthDay = new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972) const sameLeapMonthDay = new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972) @@ -191,25 +153,12 @@ const pool: ReadonlyArray = [ 1000, localInvalidDate, foreignInvalidDate, - borrowedDate, - otherBorrowedDate, new Uint8Array([1, 2]), new Uint8Array([1, 2]), new Uint8Array([1, 3]), Buffer.from([1, 2]), - shadowedTagBytes, foreignBytes, unequalForeignBytes, - shadowedForeignBytes, - byteLengthShadowedForeignBytes, - byteLengthShadowed, - equalByteLengthShadowed, - unequalByteLengthShadowed, - lengthShadowed, - equalLengthShadowed, - unequalLengthShadowed, - spoofedWideBytes, - otherSpoofedWideBytes, plainDate, Temporal.PlainDate.from(`2024-01-01`), otherPlainDate, @@ -246,17 +195,11 @@ const pool: ReadonlyArray = [ undefined, ] -// The intrinsic typed-array tag reads the internal class across realms. An -// instance's own display tag cannot hide genuine bytes or forge wide bytes. +// The intrinsic typed-array tag reads ordinary byte-view classes across realms. const typedArrayTag = Object.getOwnPropertyDescriptor( Object.getPrototypeOf(Uint8Array.prototype), Symbol.toStringTag, )!.get! -const typedArrayLength = Object.getOwnPropertyDescriptor( - Object.getPrototypeOf(Uint8Array.prototype), - `length`, -)!.get! -const typedArrayValues = Uint8Array.prototype.values // Date's native method reads the internal timestamp in either realm and // rejects an object that only borrows the prototype. The reference captures @@ -353,19 +296,12 @@ function referenceEqual(left: unknown, right: unknown): boolean { } if (isBytes(left) || isBytes(right)) { if (!isBytes(left) || !isBytes(right)) return false - // Captured native length and value iteration ignore own properties. The - // model materializes the two byte sequences before comparing them; - // production walks indexes directly. - const leftLength = Reflect.apply(typedArrayLength, left, []) as number - const rightLength = Reflect.apply(typedArrayLength, right, []) as number - const leftBytes = Array.from( - Reflect.apply(typedArrayValues, left, []) as Iterable, - ) - const rightBytes = Array.from( - Reflect.apply(typedArrayValues, right, []) as Iterable, - ) + // Materialize the ordinary byte sequences for an independent comparison; + // production walks indexed elements directly. + const leftBytes = Array.from(left) + const rightBytes = Array.from(right) return ( - leftLength === rightLength && + leftBytes.length === rightBytes.length && leftBytes.every((byte, index) => byte === rightBytes[index]) ) } @@ -492,8 +428,6 @@ function evaluateWithKindOnlyTemporalKey(testCase: InCase): boolean | null { function describeCase(testCase: InCase): string { const show = (value: unknown): string => { - if (value === borrowedDate) return `borrowed Date prototype` - if (value === otherBorrowedDate) return `other borrowed Date prototype` const dateTime = nativeDateTimestamp(value) return typeof value === `bigint` ? `${value}n` @@ -554,42 +488,6 @@ describe(`in evaluator`, () => { checkCurrent(2) }) - it.each([ - { listFromRow: false, source: `constant` }, - { listFromRow: true, source: `row-supplied` }, - ])( - `ignores an own some method after compiling a $source list and editing its member`, - ({ listFromRow }) => { - const list: Array = [1] - const listExpression = listFromRow - ? new PropRef([`row`, `list`]) - : new Value(list) - const evaluate = compileExpression( - new Func(`in`, [new PropRef([`row`, `value`]), listExpression]), - ) - const checkCurrent = (value: unknown) => - expect - .soft(evaluate({ row: { value, list } })) - .toBe(referenceIn(value, list)) - - // A hostile own method can lie, but indexed members remain the input. - checkCurrent(1) - Object.defineProperty(list, `some`, { - configurable: true, - value: () => false, - }) - checkCurrent(1) - list[0] = 2 - Object.defineProperty(list, `some`, { - configurable: true, - value: () => true, - }) - checkCurrent(1) - checkCurrent(2) - checkCurrent(null) - }, - ) - it(`skips sparse holes and nullish items while retaining present members`, () => { const list = Array(4) list[1] = null @@ -604,9 +502,8 @@ describe(`in evaluator`, () => { } }) - it(`reads a Date item's current timestamp after compilation`, () => { - const dateItem = new Date(1) - const list = [dateItem] + it(`reads a replaced Date member after compilation`, () => { + const list = [new Date(1)] const evaluate = compileExpression( new Func(`in`, [new PropRef([`row`, `value`]), new Value(list)]), ) @@ -614,14 +511,13 @@ describe(`in evaluator`, () => { expect(evaluate({ row: { value } })).toBe(referenceIn(value, list)) checkCurrent(1) - dateItem.setTime(2) + list[0] = new Date(2) checkCurrent(1) checkCurrent(2) }) - it(`reads a byte item's current content after compilation`, () => { - const byteItem = new Uint8Array([1, 2]) - const list = [byteItem] + it(`reads a replaced byte-array member after compilation`, () => { + const list = [new Uint8Array([1, 2])] const evaluate = compileExpression( new Func(`in`, [new PropRef([`row`, `value`]), new Value(list)]), ) @@ -629,7 +525,7 @@ describe(`in evaluator`, () => { expect(evaluate({ row: { value } })).toBe(referenceIn(value, list)) checkCurrent(new Uint8Array([1, 2])) - byteItem[1] = 3 + list[0] = new Uint8Array([1, 3]) checkCurrent(new Uint8Array([1, 2])) checkCurrent(new Uint8Array([1, 3])) }) @@ -649,43 +545,52 @@ describe(`in evaluator`, () => { check(evaluateWithProduction, testCase) }) - it(`uses intrinsic byte class when a typed array shadows its display tag`, () => { - expect(Object.prototype.toString.call(shadowedTagBytes)).toBe( - `[object ordinary]`, - ) - expect(Object.prototype.toString.call(spoofedWideBytes)).toBe( - `[object Uint8Array]`, - ) - expect(isBytes(shadowedTagBytes)).toBe(true) - expect(isBytes(spoofedWideBytes)).toBe(false) - checkEqualityPaths(shadowedTagBytes, new Uint8Array([1, 2]), true) - checkEqualityPaths(spoofedWideBytes, otherSpoofedWideBytes, false) + it(`keeps frozen primitive constants in the same membership classes`, () => { + const lists: ReadonlyArray> = [ + Object.freeze([]), + Object.freeze([0, Number.NaN, 1n, `a`, PREFIX, null, undefined]), + ] + const values = [ + 0, + -0, + Number.NaN, + new Date(0), + runInNewContext(`new Date(0)`) as Date, + new Date(Number.NaN), + 1n, + 1, + `a`, + PREFIX, + `b`, + null, + undefined, + new Uint8Array([0]), + ] + for (const list of lists) { + const evaluate = compileExpression( + new Func(`in`, [new PropRef([`row`, `value`]), new Value(list)]), + ) + for (const value of values) { + expect(evaluate({ row: { value } })).toBe(referenceIn(value, list)) + } + } }) it(`compares native bytes from another realm by content in every compiled equality path`, () => { - // The independent model's captured typed-array intrinsic accepts a native - // view across realms. An own display tag does not change its byte class. + // The independent model's typed-array intrinsic accepts an ordinary + // native view across realms. expect(foreignBytes instanceof Uint8Array).toBe(false) expect(ArrayBuffer.isView(foreignBytes)).toBe(true) - expect(Object.prototype.toString.call(shadowedForeignBytes)).toBe( - `[object ordinary]`, - ) expect(isBytes(foreignBytes)).toBe(true) - expect(isBytes(shadowedForeignBytes)).toBe(true) - expect(isBytes(spoofedWideBytes)).toBe(false) checkEqualityPaths(foreignBytes, new Uint8Array([1, 2]), true) - checkEqualityPaths(shadowedForeignBytes, shadowedTagBytes, true) checkEqualityPaths(unequalForeignBytes, new Uint8Array([1, 2]), false) - checkEqualityPaths(foreignBytes, spoofedWideBytes, false) }) - it(`compares foreign native Dates by timestamp and excludes borrowed prototypes in every compiled equality path`, () => { + it(`compares foreign native Dates by timestamp in every compiled equality path`, () => { expect(foreignDate instanceof Date).toBe(false) expect(foreignInvalidDate instanceof Date).toBe(false) - expect(borrowedDate instanceof Date).toBe(true) expect(nativeDateTimestamp(foreignDate)).toBe(1000) expect(Number.isNaN(nativeDateTimestamp(foreignInvalidDate))).toBe(true) - expect(nativeDateTimestamp(borrowedDate)).toBeUndefined() checkEqualityPaths(foreignDate, localDate, true) checkEqualityPaths(foreignDate, 1000, true) checkEqualityPaths(laterForeignDate, localDate, false) @@ -696,47 +601,6 @@ describe(`in evaluator`, () => { checkEqualityPaths(foreignInvalidDate, localInvalidDate, true) checkEqualityPaths(foreignInvalidDate, Number.NaN, true) checkEqualityPaths(foreignInvalidDate, localDate, false) - checkEqualityPaths(borrowedDate, localDate, false) - checkEqualityPaths(borrowedDate, 1000, false) - checkEqualityPaths(borrowedDate, borrowedDate, true) - checkEqualityPaths(borrowedDate, otherBorrowedDate, false) - }) - - it(`ignores own byte-count shadows in every compiled equality path`, () => { - // Both shadowed arrays still hold two native bytes. Pairing two zero - // shadows catches a comparison that stops before reading their content; - // pairing one with an unshadowed array catches a false inequality. - expect(byteLengthShadowed.byteLength).toBe(0) - expect(lengthShadowed.length).toBe(0) - expect(Reflect.apply(typedArrayLength, byteLengthShadowed, [])).toBe(2) - expect(Reflect.apply(typedArrayLength, lengthShadowed, [])).toBe(2) - expect(byteLengthShadowedForeignBytes instanceof Uint8Array).toBe(false) - expect(byteLengthShadowedForeignBytes.byteLength).toBe(0) - checkEqualityPaths(byteLengthShadowed, new Uint8Array([1, 2]), true) - checkEqualityPaths(byteLengthShadowed, equalByteLengthShadowed, true) - checkEqualityPaths(byteLengthShadowed, unequalByteLengthShadowed, false) - checkEqualityPaths(byteLengthShadowedForeignBytes, byteLengthShadowed, true) - checkEqualityPaths( - byteLengthShadowedForeignBytes, - unequalByteLengthShadowed, - false, - ) - checkEqualityPaths(lengthShadowed, new Uint8Array([1, 2]), true) - checkEqualityPaths(lengthShadowed, equalLengthShadowed, true) - checkEqualityPaths(lengthShadowed, unequalLengthShadowed, false) - }) - - it(`reads native byte values when an array shadows its iterator`, () => { - const bytes = new Uint8Array([1, 2]) - Object.defineProperty(bytes, Symbol.iterator, { - value: function* () { - yield 9 - yield 9 - }, - }) - expect(Array.from(bytes)).toEqual([9, 9]) - checkEqualityPaths(bytes, new Uint8Array([1, 2]), true) - checkEqualityPaths(bytes, new Uint8Array([9, 9]), false) }) it(`distinguishes unequal intrinsic values within each Temporal kind`, () => { diff --git a/packages/db/tests/query/index-path-collision-oracle.test.ts b/packages/db/tests/query/index-path-collision-oracle.test.ts index c33ddc27fb..c3f585382a 100644 --- a/packages/db/tests/query/index-path-collision-oracle.test.ts +++ b/packages/db/tests/query/index-path-collision-oracle.test.ts @@ -18,25 +18,14 @@ * argument order, bound inclusivity, reversed operands, and bound field. * A selected-field ordering reads both paths through the public `$selected` * callback proxy. - * A separate range history stores genuine Dates, a native Date subclass, and - * an ordinary object with a borrowed Date subclass prototype. Its public - * `gt` result must be the same before and after installing either index type. - * Another range history uses the borrowed object as the predicate bound. - * Both mixed domains require scan fallback after installation. - * A foreign Date subclass is the adjacent valid-domain control: its native - * timestamp keeps the range eligible for index lookup across realms. - * A further valid-Date history overrides `valueOf` in local and foreign - * subclasses while retaining native `getTime`. An ordinary Date bound and a - * subclass bound each reach all four effective relations; the subclass bound - * occupies both operand positions. Plain, local-subclass, and foreign-subclass - * rows share the bound's exact timestamp, so inclusive and exclusive ranges - * must respectively keep or exclude every tied key. The intrinsic-timestamp - * model compares public scan and indexed snapshots, and both index types must - * remain eligible for optimized-only reads. The reference captures each source - * timestamp before the rows enter the Collection, so later source-Date changes - * cannot change the expected answer. Date/number scan comparisons in both - * operand orders use the same model; mixed-domain range indexes make no - * eligibility claim here. + * A separate range history stores ordinary native Dates from this and another + * JavaScript realm. Supplied timestamps predict inclusive and exclusive range + * keys before and after installing either index type. Local and foreign Dates + * at the exact bound distinguish a missing equality tie; a foreign Date after + * the bound distinguishes a scan or index that loses that row. The bound also + * comes from both realms, so index eligibility cannot recognize only local + * bounds. The timestamp model does not read production comparison or index + * classification. * * Production driver: `currentStateAsChanges`, Collection subscriptions, and * live-query Collections run the real predicates and callback compilers. @@ -53,8 +42,9 @@ * Known omissions: This bounded owner does not claim arbitrary path segments, * nullish values, custom collation, or incremental publication histories. * Lazy demand target deduplication has a separate compiler-boundary witness. - * The Date-domain histories cover a `valueOf` override but not other coercion - * methods, invalid Dates, or publication after later writes. + * The Date range history covers ordinary native Dates only. Modified Date + * objects, subclasses, borrowed prototypes, invalid Dates, and publication + * after later writes are outside its claimed domain. */ import { runInNewContext } from 'node:vm' import { expect, it, vi } from 'vitest' @@ -197,300 +187,66 @@ it.each([ { name: `BasicIndex`, indexType: BasicIndex }, { name: `BTreeIndex`, indexType: BTreeIndex }, ])( - `keeps borrowed Date prototypes outside indexed range results: $name`, + `keeps ordinary foreign Dates in indexed timestamp ranges: $name`, async ({ name, indexType }) => { await checkWithCleanup(async (register) => { - class NativeDate extends Date {} - const borrowed = Object.create(NativeDate.prototype) as object + // Supplied times, rather than production values, define the reference. + // Both realms have a Date at the exact lower bound. + const inputs = [ + { id: `local-before`, time: 0, foreign: false }, + { id: `local-equal`, time: 1, foreign: false }, + { id: `foreign-equal`, time: 1, foreign: true }, + { id: `foreign-after`, time: 2, foreign: true }, + ] as const + const dateRows = inputs.map(({ id, time, foreign }) => ({ + id, + v: foreign + ? (runInNewContext(`new Date(time)`, { time }) as Date) + : new Date(time), + })) + expect(dateRows[2]!.v instanceof Date).toBe(false) + expect(Date.prototype.getTime.call(dateRows[2]!.v)).toBe(1) const bound = new Date(1) - const dateRows: Array<{ id: string; v: object }> = [ - { id: `before`, v: new Date(0) }, - { id: `borrowed`, v: borrowed }, - { id: `after`, v: new Date(2) }, - { id: `subclass`, v: new NativeDate(3) }, - ] - const collection = createCollection( - mockSyncCollectionOptions({ - id: `index-borrowed-date-${name}`, - getKey: (row: (typeof dateRows)[number]) => row.id, - initialData: dateRows, - autoIndex: `off`, - }), - ) - register(() => collection.cleanup()) - await collection.stateWhenReady() - - // The reference uses JavaScript numeric coercion for this Date bound. - // Borrowing a Date prototype cannot supply its native timestamp, so the - // coercion fails and that source row cannot satisfy `v > bound`. - const expected = dateRows - .filter(({ v }) => { - try { - return Number(v) > bound.getTime() - } catch { - return false - } - }) - .map(({ id }) => id) - .sort() - expect(expected).toEqual([`after`, `subclass`]) - - const where = gt(new PropRef([`v`]), bound) - const snapshotKeys = () => - collection - .currentStateAsChanges({ where })! - .map((change) => String(change.key)) - .sort() - - // An unindexed public snapshot establishes the first checkpoint. The - // indexed snapshot must refine the same result; optimizedOnly may decline - // this mixed-domain index rather than claiming a complete answer. - expect(snapshotKeys(), `before ${name}`).toEqual(expected) - const index = collection.createIndex((row) => row.v, { indexType }) - const lookup = vi.spyOn(index, `lookup`) - expect(snapshotKeys(), `after ${name}`).toEqual(expected) - expect(lookup, `unsafe range index not used`).not.toHaveBeenCalled() - expect( - collection.currentStateAsChanges({ where, optimizedOnly: true }), - ).toBeUndefined() - }) - }, -) - -it.each([ - { name: `BasicIndex`, indexType: BasicIndex }, - { name: `BTreeIndex`, indexType: BTreeIndex }, -])( - `does not treat a borrowed Date prototype as a range bound: $name`, - async ({ name, indexType }) => { - await checkWithCleanup(async (register) => { - class NativeDate extends Date {} - const borrowed = Object.create(NativeDate.prototype) as Date - const dateRows = [ - { id: `before`, v: new Date(0) }, - { id: `after`, v: new NativeDate(2) }, - ] - const collection = createCollection( - mockSyncCollectionOptions({ - id: `index-borrowed-date-bound-${name}`, - getKey: (row: (typeof dateRows)[number]) => row.id, - initialData: dateRows, - autoIndex: `off`, - }), - ) - register(() => collection.cleanup()) - await collection.stateWhenReady() - - // Numeric coercion of the borrowed bound throws. No source row can - // satisfy the relational predicate, regardless of index availability. - const expected = dateRows - .filter(({ v }) => { - try { - return Number(v) < Number(borrowed) - } catch { - return false - } - }) - .map(({ id }) => id) - expect(expected).toEqual([]) - - const where = lt(new PropRef([`v`]), borrowed) - const snapshotKeys = () => - collection - .currentStateAsChanges({ where })! - .map((change) => String(change.key)) - .sort() - expect(snapshotKeys(), `before ${name}`).toEqual(expected) - const index = collection.createIndex((row) => row.v, { indexType }) - const lookup = vi.spyOn(index, `lookup`) - expect(snapshotKeys(), `after ${name}`).toEqual(expected) - expect(lookup, `unsafe range index not used`).not.toHaveBeenCalled() - expect( - collection.currentStateAsChanges({ where, optimizedOnly: true }), - ).toBeUndefined() - }) - }, -) - -it.each([ - { name: `BasicIndex`, indexType: BasicIndex }, - { name: `BTreeIndex`, indexType: BTreeIndex }, -])( - `keeps foreign Date subclasses in the indexed Date domain: $name`, - async ({ name, indexType }) => { - await checkWithCleanup(async (register) => { - const bound = new Date(1) - const dateRows = [ - { id: `before`, v: new Date(0) }, - { - id: `foreign-subclass`, - v: runInNewContext(`new (class extends Date {})(2)`) as Date, - }, - ] - const collection = createCollection( - mockSyncCollectionOptions({ - id: `index-foreign-date-${name}`, - getKey: (row: (typeof dateRows)[number]) => row.id, - initialData: dateRows, - autoIndex: `off`, - }), - ) - register(() => collection.cleanup()) - await collection.stateWhenReady() - - // A foreign subclass has a native Date timestamp, so the direct numeric - // relation predicts the same key on scans and both indexed paths. - const expected = dateRows - .filter(({ v }) => Number(v) > bound.getTime()) - .map(({ id }) => id) - expect(expected).toEqual([`foreign-subclass`]) - const where = gt(new PropRef([`v`]), bound) - const snapshotKeys = () => - collection.currentStateAsChanges({ where })!.map((change) => change.key) - expect(snapshotKeys(), `before ${name}`).toEqual(expected) - const index = collection.createIndex((row) => row.v, { indexType }) - const lookup = vi.spyOn(index, `lookup`) - expect(snapshotKeys(), `after ${name}`).toEqual(expected) - expect(lookup, `native Date range index reached`).toHaveBeenCalled() - expect( - collection - .currentStateAsChanges({ where, optimizedOnly: true }) - ?.map((change) => change.key), - ).toEqual(expected) - }) - }, -) - -it.each([ - { name: `BasicIndex`, indexType: BasicIndex }, - { name: `BTreeIndex`, indexType: BTreeIndex }, -])( - `compares native Date subclasses by timestamp across scan and indexed ranges: $name`, - async ({ name, indexType }) => { - await checkWithCleanup(async (register) => { - class LaterDate extends Date { - override valueOf() { - return -100 - } - } - const earlierForeign = runInNewContext( - `new (class extends Date { valueOf() { return 100 } })(0)`, - ) as Date - const equalForeign = runInNewContext( - `new (class extends Date { valueOf() { return 100 } })(1)`, - ) as Date - const equalLocal = new LaterDate(1) - const localLater = new LaterDate(2) - const dateRows = [ - { id: `plain-before`, v: new Date(-1) }, - { id: `foreign-earlier`, v: earlierForeign }, - { id: `equal`, v: new Date(1) }, - { id: `local-equal`, v: equalLocal }, - { id: `foreign-equal`, v: equalForeign }, - { id: `local-later`, v: localLater }, - { id: `plain-after`, v: new Date(3) }, - ] - const bound = new Date(1) - const subclassBound = new LaterDate(1) - const nativeGetTime = Date.prototype.getTime - const boundTime = nativeGetTime.call(bound) - const subclassBoundTime = nativeGetTime.call(subclassBound) - // Capture the model before Collection admission. Neither the scan nor - // the indexed checkpoint may rederive its answer from mutable source - // Date objects handed to production. - const referenceRows = Object.freeze( - dateRows.map(({ id, v }) => - Object.freeze({ id, time: nativeGetTime.call(v) }), - ), - ) + const foreignBound = runInNewContext(`new Date(1)`) as Date + expect(foreignBound instanceof Date).toBe(false) const field = new PropRef([`v`]) - const predicates = [ + const rangeCases = [ { - name: `gt`, + name: `exclusive`, where: gt(field, bound), - accepts: (time: number) => time > boundTime, + accepts: (time: number) => time > 1, }, { - name: `gte`, + name: `inclusive`, where: gte(field, bound), - accepts: (time: number) => time >= boundTime, - }, - { - name: `lt`, - where: lt(field, bound), - accepts: (time: number) => time < boundTime, + accepts: (time: number) => time >= 1, }, { - name: `lte`, - where: lte(field, bound), - accepts: (time: number) => time <= boundTime, - }, - { - name: `gt with subclass bound`, - where: gt(field, subclassBound), - accepts: (time: number) => time > subclassBoundTime, - }, - { - name: `gte with subclass bound`, - where: gte(field, subclassBound), - accepts: (time: number) => time >= subclassBoundTime, - }, - { - name: `lt with subclass bound`, - where: lt(field, subclassBound), - accepts: (time: number) => time < subclassBoundTime, - }, - { - name: `lte with subclass bound`, - where: lte(field, subclassBound), - accepts: (time: number) => time <= subclassBoundTime, - }, - { - name: `subclass bound less than field`, - where: lt(subclassBound, field), - accepts: (time: number) => subclassBoundTime < time, - }, - { - name: `subclass bound less than or equal to field`, - where: lte(subclassBound, field), - accepts: (time: number) => subclassBoundTime <= time, - }, - { - name: `subclass bound greater than field`, - where: gt(subclassBound, field), - accepts: (time: number) => subclassBoundTime > time, - }, - { - name: `subclass bound greater than or equal to field`, - where: gte(subclassBound, field), - accepts: (time: number) => subclassBoundTime >= time, - }, - ] - const mixedPredicates = [ - { - name: `Date greater than number`, - where: gt(field, 1), + name: `foreign exclusive bound`, + where: gt(field, foreignBound), accepts: (time: number) => time > 1, }, { - name: `number less than Date`, - where: lt(1, field), - accepts: (time: number) => 1 < time, - }, - { - name: `Date less than number`, - where: lt(field, 1), - accepts: (time: number) => time < 1, - }, - { - name: `number greater than Date`, - where: gt(1, field), - accepts: (time: number) => 1 > time, + name: `foreign inclusive bound`, + where: gte(field, foreignBound), + accepts: (time: number) => time >= 1, }, ] + const matchingKeys = (accepts: (time: number) => boolean) => + inputs + .filter(({ time }) => accepts(time)) + .map(({ id }) => id) + .sort() + expect(matchingKeys(rangeCases[0]!.accepts)).toEqual([`foreign-after`]) + expect(matchingKeys(rangeCases[1]!.accepts)).toEqual([ + `foreign-after`, + `foreign-equal`, + `local-equal`, + ]) + const collection = createCollection( mockSyncCollectionOptions({ - id: `index-valueof-date-${name}`, + id: `index-foreign-date-${name}`, getKey: (row: (typeof dateRows)[number]) => row.id, initialData: dateRows, autoIndex: `off`, @@ -498,20 +254,6 @@ it.each([ ) register(() => collection.cleanup()) await collection.stateWhenReady() - - // The model captured native timestamps, which ignore subclass valueOf. - // The overrides deliberately move their native-operator positions, - // including at the exact bound, while ordinary Dates control both sides. - expect(Number(earlierForeign)).toBe(100) - expect(Number(equalForeign)).toBe(100) - expect(Number(equalLocal)).toBe(-100) - expect(Number(localLater)).toBe(-100) - expect(Number(subclassBound)).toBe(-100) - const expectedDateKeys = (accepts: (time: number) => boolean) => - referenceRows - .filter(({ time }) => accepts(time)) - .map(({ id }) => id) - .sort() const snapshotKeys = ( where: BasicExpression, optimizedOnly = false, @@ -520,72 +262,42 @@ it.each([ .currentStateAsChanges({ where, optimizedOnly }) ?.map((change) => String(change.key)) .sort() - expect( - referenceRows - .filter(({ time }) => time === boundTime) - .map(({ id }) => id) - .sort(), - `plain and native subclass rows at the range bound`, - ).toEqual([`equal`, `foreign-equal`, `local-equal`]) - // A ready Collection's public snapshots are the comparison cuts. Scan - // first, then install one index and require both the ordinary and - // optimized-only path to refine the same timestamp relation. - for (const predicate of predicates) { - expect(snapshotKeys(predicate.where), `scan ${predicate.name}`).toEqual( - expectedDateKeys(predicate.accepts), + // Exact public keys at the ready Collection checkpoint expose missing + // foreign rows and missing ties, before and after index installation. + for (const range of rangeCases) { + expect(snapshotKeys(range.where), `scan ${range.name}`).toEqual( + matchingKeys(range.accepts), ) - expect( - snapshotKeys(predicate.where, true), - `unindexed optimized-only ${predicate.name}`, - ).toBeUndefined() + expect(snapshotKeys(range.where, true)).toBeUndefined() } - // Calibrate the exact-key refinement check with a plausible wrong - // result: dropping one tied subclass from an inclusive public scan. - const inclusive = predicates[1]! + const inclusive = rangeCases[1]! const fullScan = snapshotKeys(inclusive.where)! - expect(fullScan).toContain(`foreign-equal`) expect(() => - expect( - fullScan.filter((key) => key !== `foreign-equal`), - `wrong-result control: omitted foreign Date tie`, - ).toEqual(expectedDateKeys(inclusive.accepts)), + expect(fullScan.filter((key) => key !== `foreign-equal`)).toEqual( + matchingKeys(inclusive.accepts), + ), ).toThrow() - // Equality already normalizes native Dates to numbers. Relational scan - // results must use the same timestamp when either operand is a number. - for (const predicate of mixedPredicates) { - expect(snapshotKeys(predicate.where), `scan ${predicate.name}`).toEqual( - expectedDateKeys(predicate.accepts), - ) - } + const index = collection.createIndex((row) => row.v, { indexType }) const lookup = vi.spyOn(index, `lookup`) - for (const predicate of predicates) { - expect( - snapshotKeys(predicate.where), - `indexed ${predicate.name}`, - ).toEqual(expectedDateKeys(predicate.accepts)) - expect(lookup, `indexed ${predicate.name} reached`).toHaveBeenCalled() + for (const range of rangeCases) { + const expected = matchingKeys(range.accepts) + expect(snapshotKeys(range.where), `indexed ${range.name}`).toEqual( + expected, + ) + expect(lookup, `indexed ${range.name} reached`).toHaveBeenCalled() lookup.mockClear() expect( - snapshotKeys(predicate.where, true), - `optimized-only ${predicate.name}`, - ).toEqual(expectedDateKeys(predicate.accepts)) + snapshotKeys(range.where, true), + `optimized-only ${range.name}`, + ).toEqual(expected) expect( lookup, - `optimized-only ${predicate.name} reached`, + `optimized-only ${range.name} reached`, ).toHaveBeenCalled() lookup.mockClear() } - - // This hostile mutation crosses the bound after every public check. - // The expected keys stay fixed because the model recorded the original - // timestamps before production received the source rows. - const greaterKeys = expectedDateKeys((time) => time > boundTime) - expect(greaterKeys).toContain(`local-later`) - localLater.setTime(-2) - expect(nativeGetTime.call(localLater)).toBe(-2) - expect(expectedDateKeys((time) => time > boundTime)).toEqual(greaterKeys) }) }, ) diff --git a/packages/db/tests/query/subset-dedupe.test.ts b/packages/db/tests/query/subset-dedupe.test.ts index 10168b35e4..18600b45f0 100644 --- a/packages/db/tests/query/subset-dedupe.test.ts +++ b/packages/db/tests/query/subset-dedupe.test.ts @@ -383,26 +383,6 @@ describe(`DeduplicatedLoadSubset`, () => { expect(loadSubset).toHaveBeenCalledTimes(2) }) - it(`does not substitute comparison payloads with custom instance methods`, () => { - const date = new Date(2) - const bytes = new Uint8Array([1, 2, 3]) - Object.defineProperty(date, `getTime`, { value: () => 1 }) - Object.defineProperty(bytes, `slice`, { value: () => bytes }) - const where = new Func(`and`, [ - eq(ref(`date`), val(date)), - eq(ref(`bytes`), val(bytes)), - ]) - const request = captureRequest({ where }) - const rows = [ - { date, bytes }, - { date: new Date(2), bytes: new Uint8Array([1, 2, 3]) }, - ] - expect(request.where).toBe(where) - expect(rows.map(compileSingleRowExpression(request.where!))).toEqual( - rows.map(compileSingleRowExpression(where)), - ) - }) - describe.each([`Date`, `Uint8Array`] as const)( `request transport preserves %s predicate matches`, (type) => { @@ -429,6 +409,8 @@ describe(`DeduplicatedLoadSubset`, () => { expect(request).toBe(options) expect(request.where).toBe(where) const matches = compileSingleRowExpression(request.where!) + // Transport keeps the literal object; ordinary native value + // equality uses its time or bytes across JavaScript environments. expect( [foreign, local].map((item) => matches({ value: item })), ).toEqual([true, true]) diff --git a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts index 138c6c9f2f..265f0bafa6 100644 --- a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts +++ b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts @@ -66,15 +66,10 @@ * source restarted after cleanup retracts a vanished row even when its first * batch cannot satisfy the predicate. A third witness checks that a same-key * update which stays TRUE publishes its new row and exact subscriber payload. - * A focused tagged-Date matrix checks native Dates from this and another + * A focused Date matrix checks ordinary native Dates from this and another * JavaScript realm. It compares direct snapshot, filtered subscriber, and live * query keys after preload and one sync transaction; an explicit BasicIndex - * lookup checks the indexed equality key when an index is present. Fixed local - * and foreign Date subclasses with inherited `getTime` join the same - * timestamp classes. A forged object with a borrowed subclass prototype does - * not join a Date class. A separate unindexed public query checks the local - * borrowed-prototype case against Date inequality and object identity, - * including the zero timestamp class. + * lookup checks the indexed equality key when an index is present. * * Captured-list law: a caller-owned array passed to `inArray` remains live * input. Editing that array alone does not publish. At the next delivered @@ -95,8 +90,9 @@ * At the next source transaction, one subscriber callback contains the * complete insert/delete batch and reads the new whole result. It checks no * duplicate event when the list did not change. Separate fixed - * Date and byte-item cases change contents inside the captured list, then - * check the next source event. An ordered limit witness deletes a source row + * Date and byte-array cases replace members of the captured list, then check + * the next source event. Mutating an item in place is outside this law. + * An ordered limit witness deletes a source row * outside the old loaded prefix after a list edit; it requires the newly * matching row to appear, so ordered acquisition must retain all source rows * while this live list is captured. Joined and included on-demand sources use @@ -128,12 +124,9 @@ * open; the lifecycle publication owner needs a predicate dimension to reach * them. The mock sync source is a controlled provider; this oracle claims only * how the Collection filters and routes the sync transactions it supplies. - * The tagged-Date matrix covers native `node:vm` Dates with ordinary Date - * prototypes, including own display tags, and fixed local and foreign Date - * subclasses with inherited `getTime`. It excludes proxies, prototype swaps, - * replaced `getTime` methods, other hosts, and generated subclass histories. - * The local borrowed-prototype query uses one row and observes keys after - * preload. It does not claim indexed publication for that forged object. + * The Date matrix covers ordinary native `node:vm` Dates. Modified Date + * objects, subclasses, borrowed prototypes, proxies, and other hosts are + * outside its claimed domain. * A pinned mixed-scalar history checks an indexed source after deleting and * replacing rows. Its WHERE result still follows `eq` truth; the index's * private order must not lose an equality member or interrupt publication. @@ -320,21 +313,17 @@ function syncedModelRow(id: string, v: FieldValue): ModelRow { // This focused model starts from tags, not runtime Date classification. A // native Date and a number with the same finite timestamp are equal. Invalid // Dates and NaN occupy one equality class under the documented float rule. -// An own display tag or a subclass does not change a native Date's equality -// class. A borrowed prototype without the native Date slot matches no Date. type DateEqualityTag = | { kind: `date` realm: `local` | `foreign` timestamp: number | `invalid` - ownTag?: boolean - subclass?: boolean } | { kind: `number`; value: number | `NaN` } type TaggedDateRow = { id: string - tag: DateEqualityTag | { kind: `forged` } + tag: DateEqualityTag } function dateEqualityClass(tag: DateEqualityTag): number | `unordered` { @@ -350,11 +339,7 @@ function expectedTaggedDateKeys( ): Array { const expectedClass = dateEqualityClass(literal) return rows - .filter( - (row) => - row.tag.kind !== `forged` && - dateEqualityClass(row.tag) === expectedClass, - ) + .filter((row) => dateEqualityClass(row.tag) === expectedClass) .map((row) => row.id) .sort() } @@ -640,19 +625,13 @@ const pinnedChangeHistories: ReadonlyArray = [ }, ] -// The fixed realm grammar crosses the Date constructor's realm with equal, -// unequal, and invalid timestamps, plus foreign Dates with an own display tag. -// Fixed subclass rows and literals exercise inherited getTime on both sides of -// eq; the forged row controls against trusting only the borrowed prototype. -// Number literals test the Date/number and invalid-Date/NaN equality classes. -// A later sync transaction moves four rows between those classes. +// The fixed grammar crosses the Date constructor's realm with equal, unequal, +// and invalid timestamps. Number literals test the Date/number and +// invalid-Date/NaN equality classes. A later sync transaction moves four rows +// between those classes. const taggedDateRows: ReadonlyArray = [ { id: `local-one`, tag: { kind: `date`, realm: `local`, timestamp: 1 } }, { id: `foreign-one`, tag: { kind: `date`, realm: `foreign`, timestamp: 1 } }, - { - id: `foreign-tagged-one`, - tag: { kind: `date`, realm: `foreign`, timestamp: 1, ownTag: true }, - }, { id: `foreign-two`, tag: { kind: `date`, realm: `foreign`, timestamp: 2 } }, { id: `local-invalid`, @@ -662,86 +641,18 @@ const taggedDateRows: ReadonlyArray = [ id: `foreign-invalid`, tag: { kind: `date`, realm: `foreign`, timestamp: `invalid` }, }, - { - id: `foreign-tagged-invalid`, - tag: { kind: `date`, realm: `foreign`, timestamp: `invalid`, ownTag: true }, - }, { id: `number-one`, tag: { kind: `number`, value: 1 } }, { id: `number-nan`, tag: { kind: `number`, value: `NaN` } }, - { - id: `local-subclass-one`, - tag: { - kind: `date`, - realm: `local`, - timestamp: 1, - subclass: true, - ownTag: true, - }, - }, - { - id: `foreign-subclass-one`, - tag: { - kind: `date`, - realm: `foreign`, - timestamp: 1, - subclass: true, - ownTag: true, - }, - }, - { - id: `foreign-subclass-two`, - tag: { - kind: `date`, - realm: `foreign`, - timestamp: 2, - subclass: true, - ownTag: true, - }, - }, - { - id: `foreign-subclass-invalid`, - tag: { - kind: `date`, - realm: `foreign`, - timestamp: `invalid`, - subclass: true, - ownTag: true, - }, - }, - { id: `borrowed-prototype-object`, tag: { kind: `forged` } }, ] const taggedDateLiterals: ReadonlyArray = [ { kind: `date`, realm: `local`, timestamp: 1 }, { kind: `date`, realm: `foreign`, timestamp: 1 }, - { kind: `date`, realm: `foreign`, timestamp: 1, ownTag: true }, { kind: `date`, realm: `foreign`, timestamp: 2 }, { kind: `date`, realm: `local`, timestamp: `invalid` }, { kind: `date`, realm: `foreign`, timestamp: `invalid` }, - { kind: `date`, realm: `foreign`, timestamp: `invalid`, ownTag: true }, { kind: `number`, value: 1 }, { kind: `number`, value: `NaN` }, - { - kind: `date`, - realm: `local`, - timestamp: 1, - subclass: true, - ownTag: true, - }, - { - kind: `date`, - realm: `foreign`, - timestamp: 1, - subclass: true, - ownTag: true, - }, - { - kind: `date`, - realm: `foreign`, - timestamp: `invalid`, - subclass: true, - ownTag: true, - }, ] // --------------------------------------------------------------------------- @@ -816,42 +727,17 @@ function sourceRow(id: string, v: FieldValue): Row { return v === MISSING ? { id } : { id, v } } -function runtimeTaggedDateValue( - tag: TaggedDateRow[`tag`], -): Date | number | object { - if (tag.kind === `forged`) { - const foreignSubclass = runInNewContext( - `class D extends Date {}; new D(1)`, - ) as Date - const forged = Object.create( - Object.getPrototypeOf(foreignSubclass), - ) as object - Object.defineProperty(forged, Symbol.toStringTag, { value: `Other` }) - return forged - } +function runtimeTaggedDateValue(tag: DateEqualityTag): Date | number { if (tag.kind === `number`) { return tag.value === `NaN` ? Number.NaN : tag.value } const timestamp = tag.timestamp === `invalid` ? Number.NaN : tag.timestamp - class LocalDate extends Date {} - const date = - tag.realm === `foreign` - ? (runInNewContext( - tag.subclass - ? `class D extends Date {}; new D(timestamp)` - : `new Date(timestamp)`, - { timestamp }, - ) as Date) - : tag.subclass - ? new LocalDate(timestamp) - : new Date(timestamp) - if (tag.ownTag) { - Object.defineProperty(date, Symbol.toStringTag, { value: `Other` }) - } - return date + return tag.realm === `foreign` + ? (runInNewContext(`new Date(timestamp)`, { timestamp }) as Date) + : new Date(timestamp) } -type TaggedValueRow = { id: string; v: Date | number | object } +type TaggedValueRow = { id: string; v: Date | number } // The tag model predicts keys before the runtime values are constructed. The // driver observes the public live query, filtered subscription, and direct @@ -939,21 +825,9 @@ async function observeTaggedDateEquality( index: 0, tag: { kind: `date`, realm: `foreign`, timestamp: `invalid` }, }, - { - index: 3, - tag: { kind: `date`, realm: `foreign`, timestamp: 1, ownTag: true }, - }, - { index: 4, tag: { kind: `date`, realm: `foreign`, timestamp: 2 } }, - { - index: 10, - tag: { - kind: `date`, - realm: `foreign`, - timestamp: 2, - subclass: true, - ownTag: true, - }, - }, + { index: 2, tag: { kind: `date`, realm: `foreign`, timestamp: 1 } }, + { index: 3, tag: { kind: `date`, realm: `foreign`, timestamp: 2 } }, + { index: 5, tag: { kind: `date`, realm: `local`, timestamp: 2 } }, ] collection.utils.begin() for (const transition of transitions) { @@ -1548,7 +1422,7 @@ describe(`WHERE predicate publication oracle`, () => { }, [() => unsubscribe?.(), () => live.cleanup(), () => source.cleanup()]) }) - it(`a source event reads current Date and byte list-item contents`, async () => { + it(`a source event reads replaced Date and byte-array list members`, async () => { type MembershipRow = { id: string date: Date @@ -1557,7 +1431,7 @@ describe(`WHERE predicate publication oracle`, () => { } const source = createCollection( mockSyncCollectionOptions({ - id: `where-publication-mutable-items-${collectionSerial++}`, + id: `where-publication-replaced-items-${collectionSerial++}`, getKey: (row) => row.id, initialData: [1, 2, 3].map((value) => ({ id: String(value), @@ -1584,8 +1458,8 @@ describe(`WHERE predicate publication oracle`, () => { expect([...dates.keys()]).toEqual([`1`]) expect([...bytes.keys()]).toEqual([`1`]) - dateMembers[0]!.setTime(2) - byteMembers[0]![0] = 2 + dateMembers[0] = new Date(2) + byteMembers[0] = new Uint8Array([2]) source.utils.begin() source.utils.write({ type: `update`, @@ -2556,7 +2430,7 @@ describe(`WHERE predicate publication oracle`, () => { }, [() => subscription.unsubscribe(), () => collection.cleanup()]) }) - it(`native Dates and subclasses publish by timestamp and invalid-Date equality`, async () => { + it(`ordinary native Dates publish by timestamp across realms and invalid-Date equality`, async () => { const foreign = runtimeTaggedDateValue({ kind: `date`, realm: `foreign`, @@ -2565,16 +2439,6 @@ describe(`WHERE predicate publication oracle`, () => { // This checks the host premise: a native Date can fail local instanceof. expect(foreign instanceof Date).toBe(false) expect(Date.prototype.getTime.call(foreign)).toBe(1) - const foreignSubclass = runtimeTaggedDateValue({ - kind: `date`, - realm: `foreign`, - timestamp: 1, - subclass: true, - ownTag: true, - }) - expect(Date.prototype.getTime.call(foreignSubclass)).toBe(1) - expect(foreignSubclass instanceof Date).toBe(false) - for (const indexed of [false, true]) { for (const literal of taggedDateLiterals) { await observeTaggedDateEquality(literal, indexed) @@ -2582,82 +2446,6 @@ describe(`WHERE predicate publication oracle`, () => { } }) - it(`keeps a forged local Date subclass prototype outside Date equality`, async () => { - class LocalDate extends Date {} - const forged = Object.create(LocalDate.prototype) as object - // The native-slot check establishes that this is an ordinary object, - // despite local instanceof reporting Date. - expect(forged instanceof Date).toBe(true) - expect(() => Date.prototype.getTime.call(forged)).toThrow(TypeError) - let collectionCleanup: (() => unknown) | undefined - let notDateCleanup: (() => unknown) | undefined - let sameObjectCleanup: (() => unknown) | undefined - let invalidDateCleanup: (() => unknown) | undefined - let zeroDateCleanup: (() => unknown) | undefined - let zeroNumberCleanup: (() => unknown) | undefined - await withOracleCleanup(async () => { - const collection = createCollection( - mockSyncCollectionOptions<{ id: string; v: object }>({ - id: `where-publication-forged-date-${collectionSerial++}`, - getKey: (row) => row.id, - autoIndex: `off`, - initialData: [{ id: `forged`, v: forged }], - }), - ) - collectionCleanup = () => collection.cleanup() - const notDate = createLiveQueryCollection((q) => - q - .from({ row: collection }) - .where(({ row }) => not(eq(row.v, new Date(1)))), - ) - notDateCleanup = () => notDate.cleanup() - const sameObject = createLiveQueryCollection((q) => - q.from({ row: collection }).where(({ row }) => eq(row.v, forged)), - ) - sameObjectCleanup = () => sameObject.cleanup() - const invalidDate = createLiveQueryCollection((q) => - q - .from({ row: collection }) - .where(({ row }) => eq(row.v, new Date(Number.NaN))), - ) - invalidDateCleanup = () => invalidDate.cleanup() - const zeroDate = createLiveQueryCollection((q) => - q - .from({ row: collection }) - .where(({ row }) => eq(row.v, new Date(0))), - ) - zeroDateCleanup = () => zeroDate.cleanup() - const zeroNumber = createLiveQueryCollection((q) => - q - .from({ row: collection }) - .where( - () => new Func(`eq`, [new PropRef([`row`, `v`]), new Value(0)]), - ), - ) - zeroNumberCleanup = () => zeroNumber.cleanup() - await notDate.preload() - // A distinct Date is unequal to this object. The same object remains - // equal by identity at the public query publication checkpoint. It - // also cannot join the invalid-Date/NaN equality class. - expect([...notDate.keys()]).toEqual([`forged`]) - await sameObject.preload() - expect([...sameObject.keys()]).toEqual([`forged`]) - await invalidDate.preload() - expect([...invalidDate.keys()]).toEqual([]) - await zeroDate.preload() - expect([...zeroDate.keys()]).toEqual([]) - await zeroNumber.preload() - expect([...zeroNumber.keys()]).toEqual([]) - }, [ - () => notDateCleanup?.(), - () => sameObjectCleanup?.(), - () => invalidDateCleanup?.(), - () => zeroDateCleanup?.(), - () => zeroNumberCleanup?.(), - () => collectionCleanup?.(), - ]) - }) - for (const history of [ ...pinnedHistories, pinnedPrefilteredSnapshot, diff --git a/packages/db/tests/utils-oracle.property.test.ts b/packages/db/tests/utils-oracle.property.test.ts index c578ad7d87..3de217b24c 100644 --- a/packages/db/tests/utils-oracle.property.test.ts +++ b/packages/db/tests/utils-oracle.property.test.ts @@ -20,12 +20,9 @@ import { deepEquals, equalPersistedSnapshotValues } from '../src/utils' * pairs remain unequal, even when a primitive displays or encodes a Date or * RegExp value. Separate bounded graph laws construct corresponding rings * and acyclic copies without reusing the production walk. - * A native typed-array length shadow cannot change how many intrinsic - * elements participate in change-event equality. A supplied changed byte - * must remain unequal even when both views report an own `length` of zero. * Ordinary native Dates compare by time across JavaScript realms. One fixed - * pair with the same own string label tests a changed-time fallback. It does - * not define equality for Dates with different own labels or replaced methods. + * changed-time pair tests a false-equality fallback. Caller-modified built-ins + * are outside this oracle's declared domain. * Ordinary Temporal.Duration values compare by their ten supplied fields, * including opposite signs and carried units that represent equal lengths. * `packages/db/CHANGELOG.md` says equivalent Duration forms remain unequal @@ -34,8 +31,7 @@ import { deepEquals, equalPersistedSnapshotValues } from '../src/utils' * publication path. * Keyless DataViews have hidden state but no typed-array elements. Distinct * views from one foreign realm remain unequal by the keyless-instance law; - * pairing one with a typed array must return false without reading a typed- - * array length from the DataView, even if it spoofs a display tag. + * pairing one with a typed array must return false. * * Arbitrary different cycle topologies are outside this model. Naming that * exclusion matters: more random examples cannot establish semantics the @@ -120,13 +116,11 @@ type NativeDatePair = { rightTime: number | null leftRealm: DateRealm rightRealm: DateRealm - ownLabel?: boolean } // `null` is a model-only invalid-time marker. The supplied times, rather than // a production Date classifier, determine equality. Both foreign operands are -// constructed in one realm so a labeled unequal pair can reach the historical -// same-prototype fallback that hid its different native timestamps. +// constructed in one realm to exercise changed timestamps on one prototype. function nativeDatePair(spec: NativeDatePair): [Date, Date] { const leftTime = spec.leftTime ?? Number.NaN const rightTime = spec.rightTime ?? Number.NaN @@ -137,14 +131,6 @@ function nativeDatePair(spec: NativeDatePair): [Date, Date] { const left = spec.leftRealm === `foreign` ? foreignLeft : new Date(leftTime) const right = spec.rightRealm === `foreign` ? foreignRight : new Date(rightTime) - if (spec.ownLabel) { - for (const date of [left, right]) { - Object.defineProperty(date, `label`, { - value: `same`, - enumerable: true, - }) - } - } return [left, right] } @@ -1089,14 +1075,12 @@ describe(`deepEquals property-based tests`, () => { rightTime: 1, leftRealm: `local`, rightRealm: `foreign`, - ownLabel: false, } - const labeledChanged: NativeDatePair = { + const changed: NativeDatePair = { leftTime: 1, rightTime: 2, leftRealm: `foreign`, rightRealm: `foreign`, - ownLabel: true, } expect(() => assertNativeDateEquality( @@ -1107,24 +1091,10 @@ describe(`deepEquals property-based tests`, () => { Object.is(left.getTime(), right.getTime()), ), ).toThrow(`forward`) - expect(() => - assertNativeDateEquality(labeledChanged, () => true), - ).toThrow(`forward`) - // This fixed diagnostic preserves the missed-event counterexample - // without extending the generated grammar to modified Date objects. - assertNativeDateEquality(labeledChanged) - }) - - it(`keeps a foreign Date distinct from an equal-keyed plain record`, () => { - const date = runInNewContext(`new Date(1)`) as Date - Object.defineProperty(date, `label`, { - value: `same`, - enumerable: true, - }) - const record = { label: `same` } - expect(Date.prototype.getTime.call(date)).toBe(1) - expect(deepEquals(date, record)).toBe(false) - expect(deepEquals(record, date)).toBe(false) + expect(() => assertNativeDateEquality(changed, () => true)).toThrow( + `forward`, + ) + assertNativeDateEquality(changed) }) utilsProperty([arbitraryTemporalPlainDate])( @@ -1365,26 +1335,6 @@ describe(`deepEquals property-based tests`, () => { }, ) - utilsProperty([fc.uint8Array({ minLength: 1, maxLength: 20 }), fc.nat(20)])( - `native bytes ignore own length shadows`, - (bytes, indexHint) => { - const changedIndex = indexHint % bytes.length - const changed = Uint8Array.from(bytes) - changed[changedIndex] = (changed[changedIndex]! + 1) % 256 - const shadow = (view: Uint8Array) => - Object.defineProperty(view, `length`, { value: 0 }) - const original = shadow(Uint8Array.from(bytes)) - const copy = shadow(Uint8Array.from(bytes)) - const different = shadow(changed) - - // The supplied byte sequence, rather than either public `length`, - // determines both expected outcomes at the direct equality return. - expectEqualityPair(original, copy, true) - expectEqualityPair(original, different, false) - expectEqualityPair({ value: original }, { value: different }, false) - }, - ) - it(`keeps foreign DataViews opaque and distinct from typed arrays`, () => { const [first, second] = runInNewContext( `[new DataView(new Uint8Array([1]).buffer), @@ -1392,11 +1342,7 @@ describe(`deepEquals property-based tests`, () => { ) as [DataView, DataView] const local = new DataView(Uint8Array.of(1).buffer) const bytes = Uint8Array.of(1) - Object.defineProperty(first, Symbol.toStringTag, { - value: `Uint8Array`, - }) - - // Neither equal bytes nor a display tag supplies typed-array elements. + // A DataView has no typed-array elements. expectEqualityPair(first, first, true) expectEqualityPair(first, second, false) expectEqualityPair(local, first, false) From 24a61a8d5b6cef06d488c490e39f351b4d55fb68 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 11:00:06 -0600 Subject: [PATCH 35/46] refactor(db): narrow oracle transfers and reduce production code --- .changeset/fix-basic-index-nullish-build.md | 5 + .../fix-custom-index-order-selection.md | 5 - .../fix-index-build-cross-realm-dates.md | 5 - .../fix-mixed-index-and-foreign-bytes.md | 5 - .changeset/fix-temporal-index-order.md | 5 - .changeset/refresh-captured-inarray-lists.md | 6 - docs/contributing/oracle-coverage.md | 462 +----- .../basic-index-nullish-build.md | 11 + .../borrowed-date-indexed-range.md | 159 -- .../captured-inarray-publication.md | 308 ---- .../comparison-unit-retirement-round-5.md | 195 +-- .../comparison-unit-retirement-round-6.md | 197 +-- .../oracle-reviews/cross-realm-date-order.md | 220 --- .../cross-realm-date-where-publication.md | 195 --- .../custom-comparator-signed-zero.md | 68 - .../custom-index-order-selection.md | 483 ------ .../oracle-reviews/mixed-value-index-order.md | 89 -- .../oracle-reviews/perf-in-constant-set.md | 65 +- .../pr-2116-correctness-and-guarantees.md | 182 --- .../pr-2116-deferred-findings.md | 31 + .../oracle-reviews/temporal-equality-alias.md | 352 ----- .../oracle-reviews/temporal-index-order.md | 171 --- docs/errors.md | 10 - docs/guides/live-queries.md | 12 - .../functions/compareTemporalValues.md | 10 +- packages/db-ivm/src/operators/groupBy.ts | 67 +- packages/db/package.json | 2 +- packages/db/src/collection/change-events.ts | 82 +- packages/db/src/collection/state.ts | 21 +- packages/db/src/collection/subscription.ts | 65 +- packages/db/src/indexes/auto-index.ts | 69 +- packages/db/src/indexes/base-index.ts | 49 +- packages/db/src/indexes/basic-index.ts | 213 +-- packages/db/src/indexes/btree-index.ts | 166 +- packages/db/src/query/builder/functions.ts | 2 +- packages/db/src/query/compiler/evaluators.ts | 65 +- packages/db/src/query/compiler/group-by.ts | 217 ++- packages/db/src/query/compiler/index.ts | 225 +-- packages/db/src/query/compiler/joins.ts | 67 +- .../query/compiler/mutable-list-refresh.ts | 47 - packages/db/src/query/compiler/order-by.ts | 46 +- packages/db/src/query/compiler/select.ts | 24 +- packages/db/src/query/effect.ts | 4 - packages/db/src/query/live/ARCHITECTURE.md | 151 +- .../query/live/collection-config-builder.ts | 3 - .../src/query/live/collection-subscriber.ts | 1 - .../query/live/subset-demand-controller.ts | 7 +- packages/db/src/query/live/utils.ts | 5 +- packages/db/src/query/pooled-live-query.ts | 10 +- packages/db/src/utils.ts | 64 +- packages/db/src/utils/btree.ts | 10 +- packages/db/src/utils/comparison.ts | 220 +-- packages/db/src/utils/index-optimization.ts | 28 +- .../tests/change-event-history-oracle.test.ts | 559 +------ .../tests/comparison-oracle.property.test.ts | 296 +--- ...ustom-index-order-selection-oracle.test.ts | 1234 --------------- .../tests/fixtures/error-site-messages.json | 5 - .../index-update-oracle.property.test.ts | 595 +------- packages/db/tests/oracle-replay-manifest.ts | 7 - packages/db/tests/oracle-replay.test.ts | 8 +- .../captured-inarray-stages-oracle.test.ts | 510 ------- .../compiler/binary-equality-work.test.ts | 14 +- .../in-evaluator-oracle.property.test.ts | 724 +-------- ...stom-index-window-selection-oracle.test.ts | 198 --- .../query/index-path-collision-oracle.test.ts | 141 +- packages/db/tests/query/indexes.test.ts | 4 +- .../tests/query/load-subset-subquery.test.ts | 6 +- .../pooled-live-query-oracle.property.test.ts | 55 +- packages/db/tests/query/subset-dedupe.test.ts | 37 - ...dicate-publication-oracle.property.test.ts | 1359 +---------------- .../temporal-equality-alias-oracle.test.ts | 671 -------- .../tests/temporal-index-order-oracle.test.ts | 798 ---------- .../db/tests/utils-oracle.property.test.ts | 352 ----- packages/db/tests/utils.ts | 29 +- .../query-db-collection/tests/query.test.ts | 32 +- 75 files changed, 851 insertions(+), 11964 deletions(-) create mode 100644 .changeset/fix-basic-index-nullish-build.md delete mode 100644 .changeset/fix-custom-index-order-selection.md delete mode 100644 .changeset/fix-index-build-cross-realm-dates.md delete mode 100644 .changeset/fix-mixed-index-and-foreign-bytes.md delete mode 100644 .changeset/fix-temporal-index-order.md delete mode 100644 .changeset/refresh-captured-inarray-lists.md delete mode 100644 docs/contributing/oracle-reviews/borrowed-date-indexed-range.md delete mode 100644 docs/contributing/oracle-reviews/captured-inarray-publication.md delete mode 100644 docs/contributing/oracle-reviews/cross-realm-date-order.md delete mode 100644 docs/contributing/oracle-reviews/cross-realm-date-where-publication.md delete mode 100644 docs/contributing/oracle-reviews/custom-comparator-signed-zero.md delete mode 100644 docs/contributing/oracle-reviews/custom-index-order-selection.md delete mode 100644 docs/contributing/oracle-reviews/mixed-value-index-order.md delete mode 100644 docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md create mode 100644 docs/contributing/oracle-reviews/pr-2116-deferred-findings.md delete mode 100644 docs/contributing/oracle-reviews/temporal-equality-alias.md delete mode 100644 docs/contributing/oracle-reviews/temporal-index-order.md delete mode 100644 packages/db/src/query/compiler/mutable-list-refresh.ts delete mode 100644 packages/db/tests/custom-index-order-selection-oracle.test.ts delete mode 100644 packages/db/tests/query/captured-inarray-stages-oracle.test.ts delete mode 100644 packages/db/tests/query/custom-index-window-selection-oracle.test.ts delete mode 100644 packages/db/tests/temporal-equality-alias-oracle.test.ts delete mode 100644 packages/db/tests/temporal-index-order-oracle.test.ts diff --git a/.changeset/fix-basic-index-nullish-build.md b/.changeset/fix-basic-index-nullish-build.md new file mode 100644 index 0000000000..d8663b8549 --- /dev/null +++ b/.changeset/fix-basic-index-nullish-build.md @@ -0,0 +1,5 @@ +--- +'@tanstack/db': patch +--- + +Keep `undefined` in the configured nulls-first position when BasicIndex builds from existing rows. diff --git a/.changeset/fix-custom-index-order-selection.md b/.changeset/fix-custom-index-order-selection.md deleted file mode 100644 index c4f381a442..0000000000 --- a/.changeset/fix-custom-index-order-selection.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -'@tanstack/db': patch ---- - -Use a custom-comparator index for exact lookup without selecting its order for an ordinary Collection `orderBy` read. Select a compatible ordinary index for ordered reads, including live-query window moves with cached rows. Eager indexing creates a compatible index when needed, including for equivalent value-left range predicates. Ordered indexes compare raw row values while exact lookup keeps normalized keys. This preserves custom collation of escaped strings. Default ordering compares native byte views by unsigned content and source-key ties. Equal-content replacements refresh custom index positions without publishing a value change. Changed byte content still publishes when a view shadows its own `length` property. A sync update to a foreign-realm DataView now commits and publishes without leaving its index on the prior value. diff --git a/.changeset/fix-index-build-cross-realm-dates.md b/.changeset/fix-index-build-cross-realm-dates.md deleted file mode 100644 index 2903ec38da..0000000000 --- a/.changeset/fix-index-build-cross-realm-dates.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -'@tanstack/db': patch ---- - -Keep `undefined` values in nulls-first order when BasicIndex builds from existing rows. Compare native Dates from other JavaScript realms by timestamp in ordering, equality, and relational queries, including invalid Dates and subclasses that override `valueOf`. Treat an ordinary object with a borrowed Date prototype as an ordinary object, including when deciding whether a range index is safe. diff --git a/.changeset/fix-mixed-index-and-foreign-bytes.md b/.changeset/fix-mixed-index-and-foreign-bytes.md deleted file mode 100644 index de45b53ad6..0000000000 --- a/.changeset/fix-mixed-index-and-foreign-bytes.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -'@tanstack/db': patch ---- - -Keep mixed values in default ordered indexes through deletes and updates, and preserve public Date/number order after installing an index. Compare native Uint8Arrays from different JavaScript realms by byte content in compiled equality and membership predicates, even when an instance shadows its byte count. diff --git a/.changeset/fix-temporal-index-order.md b/.changeset/fix-temporal-index-order.md deleted file mode 100644 index 2ccd9408af..0000000000 --- a/.changeset/fix-temporal-index-order.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -'@tanstack/db': patch ---- - -Order Temporal values by their calendar or instant semantics in indexed reads. Match ZonedDateTime equality across equivalent time-zone aliases while keeping distinct zones separate, including genuine subclasses. Preserve PlainMonthDay reference years in unindexed equality, and PlainYearMonth reference days and Duration fields in indexed equality, when default display strings collide. Keep Temporal equality when genuine values shadow display members, and reject ordinary objects with forged tags. Preserve custom index comparator operands and per-row order even when values share one query-equality class, including signed zero through updates and BTree ranges. Prevent one Temporal value from changing the comparator used for later values. diff --git a/.changeset/refresh-captured-inarray-lists.md b/.changeset/refresh-captured-inarray-lists.md deleted file mode 100644 index 55c4e14958..0000000000 --- a/.changeset/refresh-captured-inarray-lists.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -'@tanstack/db': patch -'@tanstack/db-ivm': patch ---- - -Read the current members of a caller-owned array captured by `inArray` when a compiled expression evaluates. After the array or one of its items changes, refresh the existing live-query result on the next source event, including rows untouched by that event. Let grouped queries retain their prior key and aggregate input values while they replace affected results. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index f0d779a361..7d63feb0b8 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -442,9 +442,8 @@ generic visitor behavior outside the named collectors. | Backend Collection E2E | [shared provider suites](https://github.com/TanStack/db/tree/main/packages/db-collection-e2e/src/suites); [Electric registration](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/e2e/electric.e2e.test.ts); [Query registration](https://github.com/TanStack/db/blob/main/packages/query-db-collection/e2e/query.e2e.test.ts); [TrailBase registration](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/e2e/trailbase.e2e.test.ts) | Electric, Query, and TrailBase register reusable predicate, pagination, join, and live-update checks at their provider boundaries; Electric also registers the move suite. These are receiving paths for core query and Collection laws, not framework render cuts. | | Window-controller overlap | [shared infinite-query conformance](https://github.com/TanStack/db/blob/main/packages/db/tests/conformance/infinite-suite-oracle.ts), [React driver](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-conformance.test.tsx), [Vue driver](https://github.com/TanStack/db/blob/main/packages/vue-db/tests/infinite-query-conformance.test.ts), [Svelte driver](https://github.com/TanStack/db/blob/main/packages/svelte-db/tests/infinite-query-conformance.svelte.test.ts), [review record](oracle-reviews/dec-03-window-overlap.md) | Four or five ordered source rows, two window-settlement orders, and public snapshots after each settlement, a subscribed observer update, a detached controller read, and explicit recovery. Insertion and removal of the fifth row distinguish both continuation directions while an earlier error remains visible. The detached read follows a source change with no controller subscriber; its preloaded Collection retains the five-row window. The exact overlap uses the exported DB controller in each package realm. Framework hooks expose no controller preload, so this cell does not establish a hook scheduling path; direct hook paging remains in the adjacent shared scenarios. An unsubscribed controller has no notification claim. | | Infinite-query publication | [shared publication laws](https://github.com/TanStack/db/blob/main/packages/db/tests/conformance/infinite-suite-oracle.ts), [React render cuts](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-render-cuts-oracle.test.tsx), [Svelte hook tests](https://github.com/TanStack/db/blob/main/packages/svelte-db/tests/useLiveInfiniteQuery.svelte.test.ts), [review record](oracle-reviews/2026-10-05-infinite-first-paint.md) | Each driver records every value its framework published: React layout commits, Vue pre-flushes, and the Svelte construction value plus pre-effect runs. The shared suite checks that a synchronous source shows its ready first page on the first value, for a query callback and for a supplied collection that has not started, that every ready value in a fixed-source, fixed-query scenario equals the source prefix for its own page count, including an equal dependency replacement that keeps page depth, and that a mount requests an on-demand first window once, for a query callback and for a supplied collection whose finite window is wider than or equal to the one the hook needs, judged by the rows the on-demand source holds after mount. Under deferred acquisition, an on-demand first value is either the ready first page or a value with no rows, by framework: React and Svelte publish before the hook subscribes, Vue after. `preloaded-collection-first-paint` checks the documented route-loader pattern: a limited live query preloaded before mount shows its ready first page on the first value in every framework and asks the source for nothing more; a preload that does not count as demand never settles there. Pre-commit work has no shared form, because Vue setup is its commit. The React driver owns renders that never commit, StrictMode double renders (one first-window request on React 19), retries after GC, a mounted replacement whose startup error must reach the error boundary, and a collection kept from an abandoned render that enters terminal error before its query is selected again, each as under `useLiveQuery`. The React source ID reuse oracle drives the infinite hook too. Svelte owns construction and superseded recomputes before the subscribing effect. In both, the hook acquires an on-demand source as often as `useLiveQuery` does under the same history, for either input form, direct or behind a live-query Collection. On-demand requests from renders that never commit are now owned by the live-query deferred acquisition oracle; the React render-cut oracle checks that an abandoned render makes no acquisition. Open cells, owned here: React 18 StrictMode, where neither hook keeps refs across the double render; a React commit held after the 50 ms unsubscribed GC floor, where a controlled-premise controller witness shows no non-ready value but no React scheduling witness exists; ready values in scenarios that write the source or replace the query, which need a per-step source model; a join of eager and on-demand sources, which needs a zero-acquisition check before commit; a supplied collection with no `.limit`, which sends one unbounded request on `main` too, because the controller cannot narrow it before the source request; and Svelte `state_unsafe_mutation` when a start inside `$derived.by` synchronously writes rows a peer is waiting for, shared with Svelte's `useLiveQuery`, which a two-component probe reproduces and the deferred-acquisition follow-up removes. | -| Structural values and ordered primitives | [hash values](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-oracle.property.test.ts), [hash identity](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-identity-oracle.property.test.ts), [MultiSet consolidation](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/multiset-consolidate-oracle.property.test.ts), [hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-graph-oracle.property.test.ts), [mixed hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-mixed-graph-oracle.property.test.ts), [hash retry](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-failure-retry-oracle.property.test.ts), [murmur stream](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/murmur-stream-oracle.property.test.ts), [comparison](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts), [deep equality](https://github.com/TanStack/db/blob/main/packages/db/tests/utils-oracle.property.test.ts), [cursor](https://github.com/TanStack/db/blob/main/packages/db/tests/cursor-oracle.property.test.ts), [indexes](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update-oracle.property.test.ts), [BTree Map](https://github.com/TanStack/db/blob/main/packages/db/tests/btree-map-oracle.test.ts), [query identity](https://github.com/TanStack/db/blob/main/packages/db/tests/query/identity-output-shape-oracle.test.ts), [LIKE semantics](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/evaluators-oracle.test.ts) | Independent flat values, graph topology, algebraic laws, structural object keys in code-unit order (the flat-value, identity and consolidation grammars include keys that tie under locale collation), murmur byte framing against a byte-at-a-time reference (block mixing, the finalizer and symbol framing have no owner there), Map/group/sort recomputation, expression denotation, custom comparator dispatch/reference identity/index compatibility, LIKE wildcard refinement, compiled output bags, declared value-pair agreement between D2 equality keys and IR equality operands, and Collection collation with exact public scan/index order and one-index reuse across repeated reads. The auto-index cells cross BasicIndex and BTreeIndex; scan cells omit indexes. Both paths cover lexical and numeric `en-US` locale defaults plus an explicit ordinary locale override of a numeric default. An unindexed six-row work cell bounds Collection collation resolution to at most two reads per ordered scan: one index-path check and one scan clause. Other locales and ordered histories are outside this owner. The identity pair grammar covers primitives, signed zero, NaN/invalid Date, valid Date/timestamp, binary/Buffer, Temporal, nested references, symbols, and functions; fixed and sampled pairs do not prove every JavaScript value or hash collision freedom. Ref proxies are expressions rather than literal values in this lane. The deep-equality owner also pins that `deepEquals` ignores Map and Set insertion order, RegExp `lastIndex`, array holes, and typed-array class, the rules draft equality keeps. It also pins that objects of another class differ, that plain and null-prototype objects are one class, that a class instance without enumerable keys equals only itself, that URLs compare by `href`, and that typed-array elements compare like numbers. Its [Date/RegExp unit transfer](oracle-reviews/utils-unit-retirement-round-7.md) adds fixed and generated direct-return witnesses that Date differs from its ISO text and epoch number, and RegExp differs from its source text, while preserving the original copy, changed-time, changed-flag, and changed-source operands. Other cross-type pairs and public change-event traces remain outside that transfer. The hash-values owner checks equal-hash consequences for independently constructed flat values. Distinct type identities are checked by the hash-identity owner through `equalHashValues`, including forced digest collisions; distinct 32-bit digests are not promised. The hash-identity owner checks `hash` and `equalHashValues` against one spec-level identity model: primitives with signed zero, NaN, and bigint; reference leaves (functions, registered handles, `File`, binary over 128 bytes); Date, binary, Temporal, and RegExp headers; array length and holes; Map and Set insertion order; and object keys in any order, for acyclic values and for cycles through arrays, Map values, Sets, and plain objects. Map/Set order sensitivity is pinned current behavior, not a promise. Arrays from another realm, RegExps with a `NaN` `lastIndex`, getters, proxies, other cycle carriers, and pairs that differ only in a back-edge target are outside its grammar; pinned cases keep equality's cross-realm length check and strict RegExp field comparison. `hash-work.test.ts` pins how visited values and array/RegExp headers count toward the structural work cap, and that equality copies a shared Map's entries once per distinct pair ([code-weight hash dispatch review](oracle-reviews/code-weight-hash-dispatch.md)). Custom string indexes do not optimize ordinary ranges, and custom cursors remain unsupported. The LIKE owner covers boolean string matching and bounded work, not nullish three-valued logic or a general Unicode collation contract. Unsupported composite cursors reject. The MultiSet consolidation owner checks keyed, single-type, and structural identity, signed zero and NaN in keyed and single-number data, the retained first record, zero-sum removal, and input record contents. It does not assert output order. Its keyed grammar includes cross-type key and value collisions, non-finite numeric keys, the `\|` delimiter, symbol and function identity, and a direct-object/object-tuple delimiter control from #1948. Eight named collision witnesses and both generated campaigns are RED on `c879ba6d` and GREEN with the repair ([#1948 review record](oracle-reviews/issue-1948-keyed-consolidation.md)); the [code-weight consolidation review](oracle-reviews/code-weight-multiset-consolidation.md) records the earlier exclusion. The keyed reference uses direct value/reference equality, including SameValueZero for numeric keys. The unkeyed fallback keeps its original key and value domains. The direct `MultiSet` owner does not prove which `@tanstack/db` query shapes produce primitive keyed values or mixed-type source keys; that needs a live-query production witness. The index owner checks lower-bound presence on bounded numeric histories after each add, update, delete, and build: an upper-only range ignores an unused `fromInclusive: false`, while an explicit exclusive `from` excludes its boundary. A source Map supplies expected sets for BasicIndex and BTreeIndex; wrong-output controls reject both boundary mistakes. It also checks exact `equalityLookup` and `lookup('eq')` after singleton case-insensitive `A → a → A` replacements and populated comparator-group updates, plus fresh-object same-group replacements in fixed and generated histories. These direct-index checks do not cover every JavaScript value, arbitrary comparators, or long Collection histories. The index owner enforces one invalid-comparator law across both BasicIndex and BTreeIndex: for each comparator, accepted numeric prefix (including the empty prefix), and probe operation (add, update, eq/range lookup, take, build), the operation throws exactly when it receives a `NaN` or non-number result and never when every comparison is valid. A `signed infinity` comparator is the valid control, and a custom collation `compare` supplied through `compareOptions` is checked the same way. Two BTree Map split histories check inserted-key reads, exact whole-range payloads/order, size, and extrema, proving split placement follows the insertion index without a post-mutation comparison. A rejected index add or remove leaves the index refining its accepted rows; update and build are not atomic. At the Collection boundary, both index types and both write paths (optimistic insert, sync commit) crash the collection: status becomes `error` and the next mutation throws, so a usable collection never holds rows its subscribers were not told about. Open: a sync source can still commit writes on a collection in `error` state, which stores unpublished rows; this is existing `markError` behavior and needs a lifecycle-owner witness. Comparator transitivity is outside this evidence. | -| WHERE predicate publication | [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts), [`in` evaluator](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts) ([review](oracle-reviews/perf-in-constant-set.md)) | An independent Kleene evaluator judges `eq`/`not`/`and`/`or` predicates over strings, a normalization-prefixed string, booleans, numbers, `NaN`, a valid Date, `null`, a missing field, and virtual fields. Snapshot histories with pending optimistic inserts compare a live query, direct subscribers with and without initial state, and `currentStateAsChanges`; change histories apply multi-key sync transactions, including reinsertion of a key deleted earlier, and compare every consumer's key set after each commit, on scan and `BasicIndex` paths. Peer subscribers share one field with different literals, plus one on a virtual field, so every published change must reach exactly the subscribers whose predicate it can satisfy; routing mutants that ignored the previous value, ignored the field, delivered a change twice, or routed while stale published rows awaited reconciliation fail here. Fixed witnesses cover a layout-only publication reaching a filtered subscriber as one empty batch, a filtered subscriber's empty Collection-readiness batch, retraction of a vanished row after eager cleanup and restart, and the new row plus exact subscriber update payload when a same-key row stays TRUE. A stale-live-value mutant survives the generated membership checks but fails the payload witness. A focused [property-visibility test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-prefilter-property-visibility.test.ts) compares the public unindexed snapshot with the enriched-row contract for inherited, non-enumerable, enumerable-own, and nested getter paths; the pre-fix stored-row shortcut failed three of its four controls. Other property-descriptor and stateful-getter histories remain outside those fixed controls. Hostile mutants for FALSE-for-UNKNOWN `eq`, `or` or number-literal subscription prefilters, a prefilter that ignores the previous value, a skipped readiness batch, a skip while stale published rows await reconciliation, and an unindexed snapshot scan that reads only synced rows while an optimistic insert or delete changes visibility passed the prior `@tanstack/db` suite and fail here. Change routing withholds a dropped row from an `eq` subscription before its sent key could be recorded, so a mutant that records dropped rows as sent is equivalent for routed predicates in this oracle. A focused `collection-subscription.test.ts` witness checks that a dropped insert or update, alone or beside a matching change, cannot advance a limited subscription's page offset under a routed `eq` and an unrouted `or` predicate; the unrouted cases beside a matching change kill that mutant, and the unrouted `alone` case does not. A loose-equality prefilter is an equivalent mutant: it only skips less. Skipping during truncate replay also survives; the replay's baseline diff re-derives the same retraction, so the guard keeps the prior dataflow without its own witness. Generated cleanup and restart histories for filtered subscribers remain open for the lifecycle publication owner. No-op updates, other comparison operators, Temporal and binary operands, joins, ordering, optimistic updates, and truncate are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. Change routing was later removed in favor of pooled live queries; the routing mutants above are historical, and the same peer laws now check plain subscription dispatch. The unindexed snapshot prefilter was restored for compiled queries as one stored-row `eq` test on a string or boolean literal. Its property-visibility test also covers the reverse direction: a stored row with an inherited or non-enumerable field must still match `isUndefined` on that field, which a scan that evaluated the full predicate on stored rows fails. The `in` evaluator owner checks `in` against an independent linear reference over Dates, `NaN`, `-0`, BigInt, bytes, Temporal values, objects and nullish items, for constant and row-supplied lists. Its one-polyfill Temporal model uses captured intrinsic methods for native equality and models Duration by ten fields, including PT1H versus PT60M. Malformed Temporal lookalikes and deliberately shadowed Temporal members are outside that owner, along with mixed Temporal implementations, WHERE publication, and indexed mixed ordinary-object/Temporal range order. | -| Captured `inArray` query stages | [stage oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/captured-inarray-stages-oracle.test.ts) ([review](oracle-reviews/captured-inarray-publication.md)) | Twelve fixed public cuts include nine numeric stage and acquisition witnesses after one unrelated source transaction, a limited ORDER uniqueness counterexample, grouped SUM/min/max arguments in an untouched group, and an aggregate argument wrapped by a selected addition. The controlled on-demand cut checks an unfiltered, unordered, unlimited first request. The nine initial stage cuts, ORDER duplicate, and wrapped aggregate were RED→GREEN in the working tree; grouped SUM/min/max passes in workspace Vitest. The final DB suite passes 12,406 tests across 282 files; DB-IVM passes 693 across 45. Arbitrary values and histories, real providers, and exact-head review remain open. | +| Structural values and ordered primitives | [hash values](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-oracle.property.test.ts), [hash identity](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-identity-oracle.property.test.ts), [MultiSet consolidation](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/multiset-consolidate-oracle.property.test.ts), [hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-graph-oracle.property.test.ts), [mixed hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-mixed-graph-oracle.property.test.ts), [hash retry](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-failure-retry-oracle.property.test.ts), [murmur stream](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/murmur-stream-oracle.property.test.ts), [comparison](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts), [deep equality](https://github.com/TanStack/db/blob/main/packages/db/tests/utils-oracle.property.test.ts), [cursor](https://github.com/TanStack/db/blob/main/packages/db/tests/cursor-oracle.property.test.ts), [indexes](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update-oracle.property.test.ts), [BTree Map](https://github.com/TanStack/db/blob/main/packages/db/tests/btree-map-oracle.test.ts), [query identity](https://github.com/TanStack/db/blob/main/packages/db/tests/query/identity-output-shape-oracle.test.ts), [LIKE semantics](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/evaluators-oracle.test.ts) | Independent flat values, graph topology, algebraic laws, structural object keys in code-unit order (the flat-value, identity and consolidation grammars include keys that tie under locale collation), murmur byte framing against a byte-at-a-time reference (block mixing, the finalizer and symbol framing have no owner there), Map/group/sort recomputation, expression denotation, custom comparator dispatch/reference identity/index compatibility, LIKE wildcard refinement, compiled output bags, declared value-pair agreement between D2 equality keys and IR equality operands, and Collection collation with exact public scan/index order and one-index reuse across repeated reads. The auto-index cells cross BasicIndex and BTreeIndex; scan cells omit indexes. Both paths cover lexical and numeric `en-US` locale defaults plus an explicit ordinary locale override of a numeric default. An unindexed six-row work cell bounds Collection collation resolution to at most two reads per ordered scan: one index-path check and one scan clause. Other locales and ordered histories are outside this owner. The identity pair grammar covers primitives, signed zero, NaN/invalid Date, valid Date/timestamp, binary/Buffer, Temporal, nested references, symbols, and functions; fixed and sampled pairs do not prove every JavaScript value or hash collision freedom. Ref proxies are expressions rather than literal values in this lane. The deep-equality owner also pins that `deepEquals` ignores Map and Set insertion order, RegExp `lastIndex`, array holes, and typed-array class, the rules draft equality keeps. It also pins that objects of another class differ, that plain and null-prototype objects are one class, that a class instance without enumerable keys equals only itself, that URLs compare by `href`, and that typed-array elements compare like numbers. The hash-values owner checks equal-hash consequences for independently constructed flat values. Distinct type identities are checked by the hash-identity owner through `equalHashValues`, including forced digest collisions; distinct 32-bit digests are not promised. The hash-identity owner checks `hash` and `equalHashValues` against one spec-level identity model: primitives with signed zero, NaN, and bigint; reference leaves (functions, registered handles, `File`, binary over 128 bytes); Date, binary, Temporal, and RegExp headers; array length and holes; Map and Set insertion order; and object keys in any order, for acyclic values and for cycles through arrays, Map values, Sets, and plain objects. Map/Set order sensitivity is pinned current behavior, not a promise. Arrays from another realm, RegExps with a `NaN` `lastIndex`, getters, proxies, other cycle carriers, and pairs that differ only in a back-edge target are outside its grammar; pinned cases keep equality's cross-realm length check and strict RegExp field comparison. `hash-work.test.ts` pins how visited values and array/RegExp headers count toward the structural work cap, and that equality copies a shared Map's entries once per distinct pair ([code-weight hash dispatch review](oracle-reviews/code-weight-hash-dispatch.md)). Custom string indexes do not optimize ordinary ranges, and custom cursors remain unsupported. The LIKE owner covers boolean string matching and bounded work, not nullish three-valued logic or a general Unicode collation contract. Unsupported composite cursors reject. The MultiSet consolidation owner checks keyed, single-type, and structural identity, signed zero and NaN in keyed and single-number data, the retained first record, zero-sum removal, and input record contents. It does not assert output order. Its keyed grammar includes cross-type key and value collisions, non-finite numeric keys, the `\|` delimiter, symbol and function identity, and a direct-object/object-tuple delimiter control from #1948. Eight named collision witnesses and both generated campaigns are RED on `c879ba6d` and GREEN with the repair ([#1948 review record](oracle-reviews/issue-1948-keyed-consolidation.md)); the [code-weight consolidation review](oracle-reviews/code-weight-multiset-consolidation.md) records the earlier exclusion. The keyed reference uses direct value/reference equality, including SameValueZero for numeric keys. The unkeyed fallback keeps its original key and value domains. The direct `MultiSet` owner does not prove which `@tanstack/db` query shapes produce primitive keyed values or mixed-type source keys; that needs a live-query production witness. The index owner enforces one invalid-comparator law across both BasicIndex and BTreeIndex: for each comparator, accepted numeric prefix (including the empty prefix), and probe operation (add, update, eq/range lookup, take, build), the operation throws exactly when it receives a `NaN` or non-number result and never when every comparison is valid. A `signed infinity` comparator is the valid control, and a custom collation `compare` supplied through `compareOptions` is checked the same way. Two BTree Map split histories check inserted-key reads, exact whole-range payloads/order, size, and extrema, proving split placement follows the insertion index without a post-mutation comparison. A rejected index add or remove leaves the index refining its accepted rows; update and build are not atomic. At the Collection boundary, both index types and both write paths (optimistic insert, sync commit) crash the collection: status becomes `error` and the next mutation throws, so a usable collection never holds rows its subscribers were not told about. Open: a sync source can still commit writes on a collection in `error` state, which stores unpublished rows; this is existing `markError` behavior and needs a lifecycle-owner witness. Comparator transitivity is outside this evidence. | +| WHERE predicate publication | [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts), [`in` evaluator](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts) ([review](oracle-reviews/perf-in-constant-set.md)) | An independent Kleene evaluator judges `eq`/`not`/`and`/`or` predicates over strings, a normalization-prefixed string, booleans, numbers, `NaN`, a valid Date, `null`, a missing field, and virtual fields. Snapshot histories with pending optimistic inserts compare a live query, direct subscribers with and without initial state, and `currentStateAsChanges`; change histories apply multi-key sync transactions, including reinsertion of a key deleted earlier, and compare every consumer's key set after each commit, on scan and `BasicIndex` paths. Peer subscribers share one field with different literals, plus one on a virtual field, so every published change must reach exactly the subscribers whose predicate it can satisfy; routing mutants that ignored the previous value, ignored the field, delivered a change twice, or routed while stale published rows awaited reconciliation fail here. Fixed witnesses cover a layout-only publication reaching a filtered subscriber as one empty batch, a filtered subscriber's empty Collection-readiness batch, retraction of a vanished row after eager cleanup and restart, and the new row plus exact subscriber update payload when a same-key row stays TRUE. A stale-live-value mutant survives the generated membership checks but fails the payload witness. A focused [property-visibility test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-prefilter-property-visibility.test.ts) compares the public unindexed snapshot with the enriched-row contract for inherited, non-enumerable, enumerable-own, and nested getter paths; the pre-fix stored-row shortcut failed three of its four controls. Other property-descriptor and stateful-getter histories remain outside those fixed controls. Hostile mutants for FALSE-for-UNKNOWN `eq`, `or` or number-literal subscription prefilters, a prefilter that ignores the previous value, a skipped readiness batch, a skip while stale published rows await reconciliation, and an unindexed snapshot scan that reads only synced rows while an optimistic insert or delete changes visibility passed the prior `@tanstack/db` suite and fail here. Change routing withholds a dropped row from an `eq` subscription before its sent key could be recorded, so a mutant that records dropped rows as sent is equivalent for routed predicates in this oracle. A focused `collection-subscription.test.ts` witness checks that a dropped insert or update, alone or beside a matching change, cannot advance a limited subscription's page offset under a routed `eq` and an unrouted `or` predicate; the unrouted cases beside a matching change kill that mutant, and the unrouted `alone` case does not. A loose-equality prefilter is an equivalent mutant: it only skips less. Skipping during truncate replay also survives; the replay's baseline diff re-derives the same retraction, so the guard keeps the prior dataflow without its own witness. Generated cleanup and restart histories for filtered subscribers remain open for the lifecycle publication owner. No-op updates, other comparison operators, Temporal and binary operands, joins, ordering, optimistic updates, and truncate are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. Change routing was later removed in favor of pooled live queries; the routing mutants above are historical, and the same peer laws now check plain subscription dispatch. The unindexed snapshot prefilter was restored for compiled queries as one stored-row `eq` test on a string or boolean literal. Its property-visibility test also covers the reverse direction: a stored row with an inherited or non-enumerable field must still match `isUndefined` on that field, which a scan that evaluated the full predicate on stored rows fails. The `in` evaluator owner checks `in` against an independent linear reference over Dates, `NaN`, `-0`, BigInt, bytes, Temporal values, objects and nullish items, for constant and row-supplied lists. | | Joined result keys | [Joined result key oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/join-result-key-oracle.property.test.ts) | A nested-loop model recomputes the pair set of an inner, left, or full join over source keys drawn from plain, comma-bearing, bracket-bearing, and quoted strings, numbers beside the strings that print the same, both infinities, and `NaN`. After preload and each synced group change, the published rows must equal the model's pairs and the result key count must equal the row count. The comma-joined key encoding fails its two pinned histories and both campaigns; plain `JSON.stringify`, which prints infinities as the missing-side `null`, fails the pinned infinity history and both campaigns. A pinned history in which a `NaN`-keyed pair leaves and re-forms fails when the join index compares source-key prefixes with `===`. Joins over subqueries, more than two sources, custom `getKey`, and optimistic mutations are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. | | D2 Index storage | [Index refinement oracle](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/index-refinement-oracle.property.test.ts) | A plain-`Map` model sums multiplicities under a type-and-value identity in which `NaN` equals `NaN` and `-0` equals `0`. Histories of up to twelve additions to two keys mix prefixed arrays with `NaN`, `0`, `-0`, `1`, `'1'`, `1n`, and `'a'` prefixes and unprefixed values, with cancellation and reappearance. After each addition, `get` and `has` must equal the model. Each step also checks that `get` returns what `getIterator` yields, in its order, since the two walk the storage separately. Comparing prefixes with `===`, which disagrees with the `Map` that holds them, fails three pinned histories and both campaigns; the `-0` control passes under both. Every history also runs with `prefixIdentity`, where values under one key with the same prefix are one value; mutants that merge unprefixed values or ignore the option in a prefix map fail. The [structural identity oracle](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/structural-identity-oracle.property.test.ts) checks that `structuralIdentity` agrees with D2 value identity for equal representations (array subclasses, class instances, null prototypes, key order, non-enumerable symbols) and independent values, including separator-bearing strings and keys, enumerable symbol keys, Dates, RegExps, Maps, Sets, binary values, holes, deep nesting, and a cyclic value; an independent normal-form model decides equality and is checked against `equalHashValues`. Mutants that drop a string's or key's length, an array's length, or the Date and RegExp headers fail it; lowering the depth cap is equivalent, because the cap only bounds work. groupBy's use of it is owned by the incrementalization law: its string-extrema lane fails a string identity without a length prefix, and a focused groupBy witness fails when the group key is left out of the identity. Compaction, presence tracking, and structural payloads are outside this owner; a focused test checks that an index built without presence (as reduce builds its indexes) throws when presence is read; the [reduce fast-path oracle](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/operators/reduce-fast-path-oracle.property.test.ts) compares the fast path's emitted entries one by one with the general path's, over keys that move between the paths, hash-merged outputs, -0, and zero and negative multiplicities, while an unchanged `NaN` output is owned by a focused reduce test; the join operator tests and incrementalization law own operator behavior. | | Pooled live queries | [pooled live query oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/pooled-live-query-oracle.property.test.ts) | An independent `eq` model (nullish UNKNOWN, Date as timestamp, `NaN` equal to `NaN`, `-0` equal to `0`) gives membership; a live-query Collection compiled for the same query gives order, values, and status. Histories mix sync transactions, optimistic writes confirmed or rolled back, peer mounts and unmounts, and source cleanup and restart, including cleanup while a write is pending and a mount between cleanup and restart. Wholesale observers must match after every step, granular changes must match a reference observer by type, key, value, and previous value and replay to the model's rows, the layout revision must advance when keys reorder, and observing must never build the live-query Collection. Peers may add a residual `not(eq(...))` conjunct, which the model evaluates itself; a view that ignores it, reports an unfiltered `entries()`, or delivers a row entering or leaving it as an update fails it. Other residual operators use the same compiler evaluator but are not generated. Peers may also order their rows by `id` or by a `g` that can be null, with explicit `nulls`; order comes from the reference. A comparator that ignores direction or `nulls`, an order left out of the partition key, and key order instead of the query's order each fail the pinned reorder history and both campaigns. A partition that ignores a row's previous group, orders by arrival, skips normalization, skips the initial snapshot, follows the source's status after cleanup, or never terminates fails it. Two shared conformance scenarios (`eq-filter-rows`, `eq-filter-peers`) run the pooled path under React, Vue, Solid, Svelte, and Angular; a partition that ignores a row's previous group fails both under every adapter, and `eq-filter-peers` checks through `subscriberCount` that each adapter shares one source subscription per partition. A focused [release-timing test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/pooled-live-query-gc-oracle.test.ts) compares when a pooled query and a live-query Collection release their sources across `gcTime` 1, 100, 0, and `Infinity`, with and without a subscriber, and with no `gcTime`, where both use the live-query Collection default, checks that a partition waits for its views' longest `gcTime`, checks that a view resubscribed after its partition released follows later source writes as its live-query Collection does, including a view with a residual conjunct, and checks that a partition that subscribes again still serves new mounts from one source subscription. Further focused witnesses check that a partition keeps groups only for values that have rows or watchers, that a detached reader never sees a dropped group's revision reused, keep a pooled query's public Collection live while the view is observed, accept it as a query source, and check that a pooled query turns terminal when source cleanup starts while the adapter's cleanup is still pending, as a live-query Collection does. A pinned oracle witness checks that an `eq` path whose getter throws excludes the row on both paths; a pooled path that rethrows fails it. The pooled oracle's grammar never releases a partition, so release and resubscribe histories rely on these fixed witnesses; a fixed delay, a missing unsubscribed floor, a last-`gcTime`-wins rule, and releasing at `gcTime` 0 each fail it. The [2026-10-01 review](oracle-reviews/2026-10-01-pooled-live-queries.md) records each ORC outcome. | @@ -460,418 +459,46 @@ generic visitor behavior outside the named collectors. | Boundary refinements | [cleanup/restart](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-cleanup-restart-oracle.test.ts), [idle cleanup](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-idle-cleanup.test.ts), [issue #1891 async-cleanup review](oracle-reviews/issue-1891-async-cleanup.md), [issue #1891 cleanup-start follow-up](oracle-reviews/issue-1891-cleanup-start-followup.md), [metadata publication](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-metadata-publication-oracle.property.test.ts), [row metadata composition](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-row-metadata-composition-oracle.test.ts), [nested-begin metadata rebuild review](oracle-reviews/2026-10-06-nested-begin-metadata-rebuild.md), [state mutation round 4](oracle-reviews/2026-10-06-state-mutation-round-4.md), [state retention](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-state-retention-oracle.property.test.ts), [acquisition cells](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-subscription-lifecycle-oracle.test.ts), [D2 source reconciliation](https://github.com/TanStack/db/blob/main/packages/db/tests/d2-source-reconciliation-oracle.property.test.ts), [top-K support windows](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/operators/topk-support-window-oracle.test.ts), [nested Query work](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/includes-work-counter-oracle.test.ts) | Explicit lifecycle products, independent source maps and weighted relations, exact publication cuts, support/multiplicity, and value-plus-work observations. These refine the larger subsystem models; they do not replace them. | | Small structures and test mechanics | [SortedMap](https://github.com/TanStack/db/blob/main/packages/db/tests/SortedMap-oracle.test.ts), [cleanup queue](https://github.com/TanStack/db/blob/main/packages/db/tests/cleanup-queue-oracle.property.test.ts), [live-query GC clock](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-gc-clock.test.ts), [guarded replay](https://github.com/TanStack/db/blob/main/packages/db/tests/oracle-replay.test.ts) | Map/full-sort and elapsed appointment-list models with executed target/seed/path checks. SortedMap also checks deferred bulk writes, array-movement work, scan reuse, live iterators, runtime `undefined` keys, and public single-transaction and queued-transaction Collection sync paths. It does not settle the contract for comparator-observed values mutated after publication or every legal iterator/mutation interleaving. Wall-clock steps do not move due times, and a focused public witness checks source subscription release. Callback-reentrant scheduling, OS suspend/resume, environments without `performance.now()`, and fake/real Performance clock replacement while GC is pending remain outside this coverage. Tests must drain or reset pending GC before switching timer providers. | -The [comparison oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts) -now checks default finite/non-finite order for number-only and Date-only -histories with optional null, including exact comparator zeros and stable -sorted row IDs. Its [transfer ledger](oracle-reviews/comparison-unit-retirement-round-5.md) -records the five retired same-realm examples. A separate -[cross-realm Date witness](oracle-reviews/cross-realm-date-order.md) compares -native `node:vm` Dates by supplied timestamps and invalidity, including equal -times and numeric NaN with an invalid Date. The owner also checks an unindexed -public ordered scan with an ordinary foreign invalid Date. A fixed 129-byte -witness changes interior byte 64, and a 200-byte -witness changes the last byte. Both compare direct `areValuesEqual` returns -in each direction and distinct `normalizeValue` keys. The generated -changed-byte grammar spans lengths 1–200, but its fixed 100-run campaign did -not reach those long positions; the fixed witnesses reject a first-64-byte -comparator. Default byte-content ordering is not promised. The custom-index -order owner checks that direct indexed and public scan reads conserve the same -keys through one byte-view update, deletion, and reinsertion; exact lookup -still groups ordinary equal byte contents. Other typed-array class transitions -remain under its existing class-agnostic equality law; whether such transitions -must publish a source change needs a separate contract witness. -The same deep-equality owner checks two distinct keyless DataViews from one -foreign realm against the keyless-instance rule, plus a mixed typed-array/DataView -pair. The Collection change-event owner receives that boundary through a -typed-array-to-foreign-DataView update and a same-realm foreign-DataView -replacement. Both commits must settle with public state, BasicIndex or -BTreeIndex exact lookup, and whole subscriber batches aligned. The callback -also records the current public row, new exact index keys, and absence of the -previous exact membership. A late index refresh or transient dual membership -fails at the publication cut. This rejects -the getter failure that changed the row while leaving its index and subscriber -behind. It does not assert content equality for DataViews. -The deep-equality owner also models ordinary native Date equality by supplied -time across local and `node:vm` realms. A fixed matrix crosses both operand -realms and equal or changed valid and invalid times. Fixed-seed and random -campaigns sample finite times from -1e12 to 1e12, invalid time, equal or -adjacent changed time, and both realm placements. Direct results in both -operand orders reject a local-only Date classifier. One fixed changed-time -foreign Date pair rejects an answer that ignores changed time. The Collection -change-event owner receives the ordinary Date rule through seven one-row sync -updates, including changed foreign times, invalid-to-valid, and -valid-to-invalid transitions. Each checks exact subscriber batches, -callback-time public time, and the change-message mirror against public time -after commit. These witnesses cover one `node:vm` host, unindexed sync -publication, and native Dates with unchanged prototypes and methods. -Optimistic publication, indexed Date updates, other hosts, proxies, and -caller-modified Dates remain outside this witness. -The same deep-equality owner models ordinary Temporal.Duration equality by an -independent ten-field vector. Fixed copies and one-field changes reconstruct -each field under both signs. Fixed opposite-sign seconds reject absolute-field -comparison. Eight fixed-factor unit carries from years/months through -microseconds/nanoseconds under both signs reject pairwise balancing, including -the `PT1H` versus `PT60M` distinction promised by -`packages/db/CHANGELOG.md`. Fixed-seed and seedless campaigns vary field -magnitudes from 0 to 3 with one shared positive or negative sign, one selected -field change, and three equal-display subsecond representations with signed -multipliers of magnitude 1 to 100. A display-only comparator fails the fixed -wrong-design control, and the original implementation fails -the equal-display property. The Collection change-event owner checks ready, -unindexed sync replacements in both directions between one second and one -thousand milliseconds under both signs, plus fresh equal-field replacements -for both signs. A one-hour to 60-minutes replacement reaches the same public -checkpoint, with different default displays but equivalent elapsed length. -Exact subscriber batches, callback-time public rows, and the mirror must agree -at commit. -Mixed-sign Duration input, optimistic writes, indexed publication, other -Temporal kinds, and arbitrary ten-field Collection histories remain open. -Exact `normalizeValue` results are direct utility observations. Indexed -order, public Date range predicates, cursor continuation, proxies, deeper -subclass chains, replaced Date prototypes or `getTime` methods, and other -Date-consuming paths remain outside that witness. - -The comparison oracle also owns a bounded [Temporal order transfer](oracle-reviews/comparison-unit-retirement-round-6.md). Supplied ISO calendar fields and epoch-nanosecond offsets predict direct `ascComparator` signs and exact zeros for ordinary `PlainDate` and `ZonedDateTime` values from one polyfill copy. Fixed cases preserve the two retired unit blocks; generated cases vary calendar fields, offset zones, equal instants, and adjacent nanoseconds. A normal ZonedDateTime subclass with inherited Temporal methods compares equal in instant position to a +05:30 peer through both direct helpers. This direct comparison lane does not establish public query or index behavior. Other Temporal subclasses, malformed Temporal lookalikes, deliberately shadowed members, mixed native/polyfill implementations, other Temporal kinds, and public ordering/filter publication need separate witnesses if those behaviors become product guarantees. - -The [Temporal index order oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/temporal-index-order-oracle.test.ts) -and its [review record](oracle-reviews/temporal-index-order.md) own direct -BasicIndex and BTreeIndex ordered takes after add/build and one mutation -continuation, plus an optimized-only public Collection snapshot with an -explicit index. Supplied epoch nanoseconds and source keys establish the -order of equal-instant peers in direct indexes and public snapshots under -changelog #958; reverse admission makes a wrong public limit-one tie visible. -Distinct zones retain exact lookup identities. A focused -callback witness preserves the former string operands -for a custom comparator over two distinct pre-epoch values. A second callback -witness splits query-equal UTC aliases into per-row comparator positions after -add/build and checks direct range, cursors, removal, and same-class update. -An opposite callback puts the UTC peer first even though display-text and -source-key order put the alias first. Same-equality-class updates test both -transitions between a callback split and a source-key tie, including ranges -and cursors after each transition. -The valid-value admission lane checks default BasicIndex/BTreeIndex add and -build for Duration with calendar units, PlainMonthDay with distinct reference -years, and mixed genuine Temporal kinds. It compares exact membership and -complete key sets across forward, reverse, and build histories without -assigning those values a public sort order. A source Collection commits two -rows separately and checks its stored row and exact index membership inside -each change callback, then checks ready status. This rejects a second valid -row that leaves the Collection stored but unpublished after an index ordering -error. A public limit-one `orderBy` then rejects each pair through the strict -Temporal comparison, so the private index order cannot replace the public -one. Changelog #1519 requires public `orderBy` to reject PlainMonthDay and -mixed Temporal kinds. Arbitrary kind mixtures, public ordered reads beyond -these rejection witnesses, provider paths, and longer publication histories -remain outside this lane. -Direct removal supplies the exact prior row. Mixed Temporal/ordinary value -domains, other Temporal kinds beyond this bounded admission lane, public WHERE ranges, -provider cursors, longer histories, and live-query publication require -separate witnesses. - -The [Temporal alias equality oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/temporal-equality-alias-oracle.test.ts) -and its [review record](oracle-reviews/temporal-equality-alias.md) own the -bounded `ZonedDateTime.equals()` partition for UTC/Etc/UTC aliases, a distinct -fixed-offset zone, adjacent nanoseconds, and a different calendar. One normal -ZonedDateTime subclass with inherited Temporal methods belongs to the UTC -class. -Direct BasicIndex/BTreeIndex add, build, and removal checks compare -independent fixture-assigned classes. Unindexed and optimized-only indexed -Collection snapshots check public `eq`; the `in` evaluator oracle checks -constant and row-supplied alias lists. The `in` evaluator owner distinguishes -PlainMonthDay reference years, PlainYearMonth reference days, and Duration -field vectors that share default display text. Fixed controls reach compiled -`eq`, constant-list `in`, and row-supplied `in` in both operand directions. -Five further Duration variants isolate `years`, `months`, `weeks`, `days`, and -`nanoseconds`. Each omitted-field mutant fails the compiled membership -comparison. The alias owner checks these exact classes in an unindexed public -scan and BasicIndex/BTreeIndex equality lookup with an explicit total -comparator; default ordering of calendar-unit Durations needs `relativeTo` -and is outside this equality claim. -Equal and unequal values within Instant, PlainDate, PlainDateTime, and -PlainTime reject a kind-only Temporal equality key at those same paths. Native -`node:vm` Uint8Arrays with equal and unequal content also reach compiled -`eq`, constant `in`, and row-supplied `in` in both operand directions; the -original classifier failed equal foreign/local byte pairs. The `in` property has a guarded -direct replay selector that runs only its random/replay campaign. -For caller-owned arrays, the compiled `in` owner now reads current members. -Constant and row-supplied lists check false rejection and false acceptance -after compilation and a list slot edit. Sparse and nullish controls retain -native membership behavior. This -direct evaluation does not by itself establish live-query publication. -Native local and `node:vm` Dates, including invalid Dates and timestamp -numbers, reach those three compiled equality paths in both operand directions. -An equal-timestamp Date from a second VM context is a positive compiled -equality control, so the reference cannot accept only the first foreign -prototype. -One frozen primitive constant-list matrix checks the evaluator's immutable -list path against the same independent membership model, including NaN, -signed zero, Date normalization, nullish values, bigint, and a -normalization-prefix string. -The native-Date reference model uses a captured Date intrinsic rather than -local `instanceof`. Caller-modified Dates, byte views, and arrays are outside -these owners. Public indexed publication, detached buffers, and proxies need -separate witnesses. -The alias owner also checks a public unindexed `eq` scan with two native-equal -PlainMonthDay rows and a distinct reference year. Direct BasicIndex/BTreeIndex -add, build, removal, re-add, and public indexed `eq` cover the orderable -PlainYearMonth and Duration cases. Indexed PlainMonthDay equality with multiple -classes remains open: the ordered indexes cannot compare those classes. A -future exact-only path must preserve equality while making public `orderBy` -throw for PlainMonthDay, as changelog #1519 requires. A one-class index bypass -failed that order law and was not retained. The `in` model treats a non-null -value tested against a null-only list as FALSE, while the live-query guide's -broad SQL/PostgreSQL comparison statement suggests UNKNOWN. The compiled `in` -owner tracks this contract-authority question; a product decision is needed -before changing its expected result or claiming SQL-style null-list behavior. -Other zone aliases, malformed Temporal lookalikes, deliberately shadowed -Temporal members, mixed Temporal implementations, -mixed ordinary-object/Temporal ordered indexes, joins, grouping, query -identity, live-query publication, and provider paths remain unproved. Direct -D2 `groupBy` has its separately approved kind-and-string identity policy. -The earlier spoofed-prototype finding is outside the valid Temporal input -domain. It needs a separate product decision before it becomes a guarantee. -The [PR #2116 exact-commit review](oracle-reviews/pr-2116-correctness-and-guarantees.md) -records R16/R17/R25/R29 laws, hostile controls, remaining cells, and the -production-weight cost of proposed guarantees at code commit -`4ed5242dbef0e7dbf87b31ca97c2fd2bc7901cad`. - -The [custom index order selection oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/custom-index-order-selection-oracle.test.ts) -and its [review record](oracle-reviews/custom-index-order-selection.md) check -that a public ordered Collection snapshot does not select a reverse-number -custom comparator by default ascending or descending `compareOptions`. -Unindexed and ordinary-index snapshots supply controls; optimized-only reads -reject the custom order while optimized exact equality still uses that index. -With a custom index created before a compatible ordinary index, ordered reads -select the latter in both directions. Eager indexing adds one compatible -index when the custom index alone is present and does not add a duplicate on -repeated reads. Single and compound numeric range predicates also select -the compatible index; an eager range subscriber creates one when needed for -field-left and value-left single or paired relational predicates. A -custom-only index cannot serve an optimized range read. The fixed ready-state -fixture covers BasicIndex and BTreeIndex, one numeric field, and limit one. -Positive spies observe the later ordinary index's ordered take and numeric -lookup/range calls, including a separate reverse take for descending order. -The paired inclusive range selects only rank 20 from ranks 10, 20, and 30, -so omitting either bound changes the complete public key set. Inclusive -value-left boundaries are fixed controls. -Another fixed ready-state fixture compares unindexed and optimized indexed -Date/number order, both full and limit one, with an index-take reach check. -Separate public tie witnesses compare unindexed and optimized indexed -snapshots for ascending and descending numeric ties and custom-collation and -nullish ties. Source-key order selects finite windows, including a window cut -through a comparator group, and unbounded results; both index types receive -the same public rule. A direct on-demand ordered subscription also checks an -initial tied prefix and continuation from a unique rank into a tied rank. -A custom-collation cursor includes an unsent string from the cursor's -comparator-equal group. Escaped strings reach direct custom callbacks and -Collection custom collation as raw values across ordered snapshots, direct -range/take operations, and one subscription cursor; normalized exact lookup -still distinguishes their string keys. A bounded ordinary byte-view history -compares indexed and unindexed order without specifying a default byte-content -rank. It adopts a peer view, deletes, and reinserts its key, checking key -conservation and exact content sets after every commit. An escaped literal -remains distinct from the BTree undefined traversal marker. A custom -callback history moves an equal-content view to a different callback rank and -checks direct take, range, and cursor positions. The working-tree version of -these histories rejected stale index positions and duplicate keys on -`d93ae1dd6`. Arbitrary callbacks and mutation histories, provider ordering, -other binary values, and full live-query publication remain outside this -owner. The separate -[custom-index window selection oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/custom-index-window-selection-oracle.test.ts) -checks the live-query compiler's receiving path. A controlled on-demand source -starts with ranks 20 and 40 cached, then serves exact ordered requests from -ranks 10, 20, 30, and 40 in physical order distinct from rank order. The -controlled provider sorts only when the request carries `orderBy`, and each -finite request must ask for ascending rank. Moving a public window from offset -0/limit 1 to -offset 1/limit 2 must publish ranks 20 and 30 once, even though the provider -does not rewrite cached rank 20. Custom-first, ordinary-first, no-index, and -eager-custom-only layouts cross BasicIndex and BTreeIndex. Provider reply and -write observations prove that the cached row reaches the local ordered path. -The owner does not establish ties, nullish or nonnumeric orders, mutations, -replays, multiple order terms, arbitrary callbacks, real providers, or other -window schedules. The eager-custom-only layout is eager auto-indexing on an -on-demand source. The public result rejects wrong custom-index admission; -ordinary-index use and an eager source path need distinct witnesses. - -The [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts) -also has a [cross-realm Date equality matrix](oracle-reviews/cross-realm-date-where-publication.md). -For local and `node:vm` Dates with timestamps 1 and 2 or invalid timestamps, -plus number 1 and NaN, it compares public live-query keys, filtered subscriber -keys, and direct snapshot keys after preload and a four-row sync transaction. -The generated WHERE histories check direct snapshots at attachment; the fixed -Date matrix also checks them after its commit. Subscriber batches reconstruct -subscriber state after each generated commit. -Auto-indexing is off in both modes; one mode adds BasicIndex explicitly and -also checks its equality lookup and an optimized-only public snapshot. This -proves a public indexed snapshot result but does not prove that the live-query -planner chose the index. Other realm hosts, proxies, modified Dates, subclasses, -public range predicates, and generated Date -histories need separate witnesses. - -The [captured `inArray` publication review](oracle-reviews/captured-inarray-publication.md) -records the caller-owned list law: changing its members alone does not -publish, but the next source event must reconcile the complete existing -live-query result. Fixed direct-snapshot, retained live-query, and ordered -limit-one cuts compare current list membership with public keys and, where -applicable, exact insert/delete events for untouched rows. A second ordered -cut deletes a row outside the formerly acquired prefix after the list changes. -It failed before the ordered plan required full-source acquisition and passes -after that repair; the plan now pays full-source transfer to observe unseen -source events. A pooled query with an equality group and residual mutable -list uses the live-query Collection path; a source event in another group -must still refresh its result. Controlled on-demand acquisition, no-delta events on both joined -source sides, an outer QueryRef filter, a correlated include, and replaced Date -and byte-array list members have fixed working-tree receivers. In-place changes -to those members are outside the captured-list law. One 40-row direct-query -witness rejects an all-candidate membership rescan on an ordinary -unchanged-list event while requiring a revisit after a list change. These are bounded -in-memory receiving cuts. A controlled on-demand ordered receiver checks that -its first request omits `where`, `orderBy`, and `limit`. An unbounded ordered -receiver with a lagging on-demand source checks the same no-hints request rule, -stays pending through an unrelated event, and becomes ready at subset settlement -while its source stays loading. The adapter rejects hinted requests and delivers -rows out of order; the public result must apply local order to current members. -A companion Effect receiver -checks no-hints acquisition and its initial enter. An on-demand LEFT-join -receiver was RED when a delete outside keyed lazy demand failed to refresh -membership, then GREEN after dynamic-list compilation disabled that demand. -An on-demand correlated include receiver now checks that an unacquired spare -child deletion refreshes membership and that its first child request omits -`where`; it passes after the same graph-wide lazy-demand gate. The earlier -WHERE-focused working-tree and neighboring run passed 988 tests across 11 -suites with no type errors; exact committed-head review remains pending. The -graph-wide gate can increase full-source transfer for on-demand joins and includes. -The retained direct receiver also checks one microtask turn after a list edit -and before a source transaction: public keys stay unchanged and no subscriber -callback runs. At the next transaction, one callback contains the complete -delete/insert batch and reads the new whole public key set inside that -callback. A split-batch recorder mutant fails at this checkpoint. -Source inspection shows list snapshots cost work proportional to list length -per delivered graph event, with candidate reevaluation after list changes. -The existing index-optimization and subquery-forwarding positive tests now -use frozen `inArray` lists: immutable membership can retain index pushdown -and finite predicate forwarding, while mutable captured membership requires -full-source reach at the covered boundaries. Their 34 focused tests pass. -`ExpressionLike` accepts these `ReadonlyArray` inputs, and a separate missing -error-catalog entry, code 243, is recorded. The full DB suite passes 12,406 -tests across 282 files; DB-IVM passes 693 tests across 45 files. Both builds -and explicit source and test typechecks pass. -Real providers, broader schedules and values, arbitrary list histories, and -general work bounds remain open for these WHERE paths. An unbounded ordered -include child with mutable membership and a second eager source remains an -untested composition of no-hints acquisition, readiness, and whole-result -refresh. Effect `skipInitial` with lagging on-demand acquisition also needs a -separate witness. - -The [captured `inArray` stage oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/captured-inarray-stages-oracle.test.ts) -extends the same law to SELECT, ORDER, GROUP keys, aggregate arguments, -HAVING, either JOIN operand, and an inner QueryRef SELECT. Twelve fixed public -witnesses use plain arrays and current membership to predict results. Nine -initial stage and acquisition cuts change `[1]` to `[2]` before one unrelated -source transaction. SELECT also checks one coherent subscriber batch, and an -on-demand projected ordered limit checks that the first source request has no -`where`, `orderBy`, or `limit`. These nine were RED against the earlier stage -implementation and GREEN after shared raw-source list revision and retained -stage-output repair. A hostile review then found a limited ORDER duplicate: -a separate four-row, limit-three witness was RED before deferred Collection -order maintenance and GREEN afterward. A grouped SUM/min/max receiver checks -an untouched group after an unrelated group update and passes in workspace -Vitest; a contrary standalone probe imported stale DB-IVM output and did not -reproduce there. A twelfth witness wraps a mutable aggregate argument in -selected `add(sum(caseWhen(inArray(...))), 10)`. The prior implementation -kept total 11 after the list changed, while the plain-array model required -12; recursive aggregate-input detection makes this public comparison pass. -Hostile nullish grouped/public and projected-sort probes were bounded GREEN -controls, not confirmed defects. Before the wrapped witness, 11 stage and -33 WHERE/publication cases passed together; the final DB suite includes all -twelve stage witnesses and the WHERE/publication receivers. This owner does -not yet establish arbitrary values or source histories, real-provider -behavior, or exact committed-head review. -The [review record](oracle-reviews/captured-inarray-publication.md) tracks the -ORC-012 applicability audit and unresolved cells. - -The [historical borrowed-Date range review](oracle-reviews/borrowed-date-indexed-range.md) -records a wider law that this PR has narrowed. The public range owner in -`index-path-collision-oracle.test.ts` now supplies ordinary local and `node:vm` -Dates before, at, and after a bound from either realm. Public `gt` and `gte` -snapshots agree with supplied timestamps before and after installing BasicIndex or BTreeIndex; -an optimized-only snapshot confirms the indexed path. Equal local and foreign -Dates distinguish inclusive from exclusive ranges. Modified Dates, subclasses, -borrowed prototypes, invalid Dates, other relational operators, and later -publication remain outside this owner's current Date range claim. - -The [mixed-value index order review](oracle-reviews/mixed-value-index-order.md) -links the direct index and public WHERE owners. The direct owner checks -BasicIndex and BTreeIndex key conservation, exact membership, and within-kind -order through build, incremental admission, delete, and update for fixed -string/number and callable/number histories. Array/object, Date/object, and -PlainDate/object histories check incremental admission and updates; they do -not check build or delete for those families. The WHERE owner pins a generated -string/number delete/update counterexample and compares public key sets after -each commit. -The original default comparator failed these histories because cross-family -ties and runtime-ID order conflicted with within-family order. Cross-family -relative positions in these direct witnesses remain unspecified. A separate -public ordered-snapshot witness in the custom index selection owner checks -that adding either index preserves the scan's full and first-row order for -one Date/number fixture. It does not define a universal cross-family order. -Longer mixed histories, proxies, mixed Temporal kinds, and non-transitive user -comparators need separate witnesses before a universal sorted-index claim. - -The direct [index refinement oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update-oracle.property.test.ts) -also owns exact nullish membership and direct `IN` set selection for BasicIndex -and BTreeIndex. Fresh single-key updates cross `undefined→null`, -`null→undefined`, `1→undefined`, `undefined→1`, and -`undefined→undefined`; a coexisting-add history keeps null and undefined keys -apart. Four more fixed histories preserve two undefined owners beside string -values, an all-defined nonempty index, a mixed `[undefined, 1]` list with two -defined distractors, and a singleton `[undefined]` list with two owners. A -plain source Map supplies expected `equalityLookup`, `lookup('eq')`, -`inArrayLookup`, and `lookup('in')` sets after returned actions. Fixed and random -three-key put/delete/build histories exercise nearby states with empty, repeated, -both orders of a mixed list, and nullish-only `IN` lists. The same source Map -now predicts unfiltered ordered arrays and selected forward range sets over `undefined`, `null`, -and positive numbers. Fixed histories preserve endpoint versus explicit -undefined cursors, three undefined owners, removal of one of two owners, -explicit versus omitted lower bounds, and two owners between double undefined -bounds. The generated histories compare after each returned direct action. -Fixed build histories use both `undefined`/`1` insertion orders and a larger -`undefined`/`null`/defined source; the generated build action replaces source -ownership. The [BasicIndex build bug record](oracle-reviews/basic-index-nullish-build.md) -contains the original `takeFromStart(1)` failure and repair evidence. -Skipped-update, wrong equality, lost-owner, dropped-operand, wrong-dispatch, -wrong-cursor, and wrong-range controls fail the check. A genuine-Date history -also removes a row with an equal but distinct Date object, re-adds its original -object, retains a same-timestamp peer, and moves a row by update. After every -direct action a source Map of supplied epochs predicts exact membership, -ordered takes, and lower-bound sets. A separate callback assigns two order -positions to Dates with the same timestamp; source labels predict their order -and bounded ranges through removal, re-addition, and update. A fresh old item -can share the timestamp while its callback position differs from the stored -row's position; both index types left a stale ordered row in that history. -The stale BTree operand cache also failed the same-object re-add take before -repair. These fixed histories do not cover cross-realm Dates, arbitrary object -equivalence, or Collection publication. This direct-index evidence does not -establish mixed-type ordering, arbitrary custom comparator behavior, all -forward bound combinations, reverse nullish ranges, filtered takes, -private ordered-entry representation, live-query -behavior, bucket identity, or instrumentation. The [nullish transfer ledger](oracle-reviews/index-unit-retirement-round-2.md) -records the first five retired legacy blocks; the [IN transfer ledger](oracle-reviews/index-unit-retirement-round-3.md) -records the next four; the [ordered/range ledger](oracle-reviews/index-unit-retirement-round-4.md) -records the current transfer boundary and review. - -The [signed-zero custom comparator witness](oracle-reviews/custom-comparator-signed-zero.md) -in the index refinement oracle separates `-0` and `+0` into two ordered -positions while exact lookup retains one SameValueZero class. Fixed add/build -and removal histories check BasicIndex and BTreeIndex takes, cursors, ranges, -and exact membership. Sign-changing updates and removal/re-addition must move -or release the ordered position; mixed-sign inclusive/exclusive ranges with -reversed bounds are checked at a BTree leaf and across a 33-row split. The -signed-zero checker does not directly call `rangeQueryReversed`. It does not -establish arbitrary custom callbacks or public planner selection; the custom -index order selection owner above checks that separate boundary. +## Unit-test retirement transfers in PR #2116 + +The [direct index refinement oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update-oracle.property.test.ts) +owns the seven behaviors moved from index unit tests: an absent lower bound +with an exclusive flag; exact membership through comparator-equal updates; +`null` and `undefined` exact and `IN` membership, including repeated owners; +undefined endpoint and cursor traversal; and explicit, omitted, and +double-undefined range bounds. A source Map predicts exact key sets and +ordered reads after bounded direct actions on BasicIndex and BTreeIndex. A +full-build witness catches native `Array.sort` moving raw `undefined` last. +The [round 1](oracle-reviews/index-unit-retirement-round-1.md), +[round 2](oracle-reviews/index-unit-retirement-round-2.md), +[round 3](oracle-reviews/index-unit-retirement-round-3.md), and +[round 4](oracle-reviews/index-unit-retirement-round-4.md) ledgers retain each +retired assertion. The [build review](oracle-reviews/basic-index-nullish-build.md) +records the direct RED/GREEN repair. This owner does not establish public +Collection publication, arbitrary custom callbacks, or all JavaScript value +kinds. +The [comparison oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts) +owns the transferred same-realm NaN and invalid-Date default-order cases, +plus ordinary PlainDate calendar and ZonedDateTime instant order. Its +independent supplied values predict direct comparator signs and ties. The +[round 5](oracle-reviews/comparison-unit-retirement-round-5.md) and +[round 6](oracle-reviews/comparison-unit-retirement-round-6.md) ledgers name +the retired examples. Mixed-kind order, cross-realm values, modified built-ins, +and public indexed reads are outside this transfer. + +The [deep-equality oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/utils-oracle.property.test.ts) +owns the transferred same-realm Date and RegExp equality boundaries: equal +copies, changed values, and inequality with timestamp, ISO, or source text. +The [round 7 ledger](oracle-reviews/utils-unit-retirement-round-7.md) retains +the retired examples. Caller-modified built-ins and Collection publication +are outside this direct-return owner. + +The [deferred findings record](oracle-reviews/pr-2116-deferred-findings.md) +assigns unclosed Duration equality and Collection publication, Temporal index +admission, mixed-kind order, and custom-comparator questions to future owners. +Earlier PR-branch counterexamples do not by themselves establish a current +`origin/main` failure; each future repair needs its own baseline witness. The Query DB ownership oracle owns the direct-write no-automatic-refetch law ([review](oracle-reviews/issue-1935-direct-write-refetch.md)). @@ -1518,13 +1145,6 @@ new returned-mutation check. The direct index refinement owner does not prove this Collection mutation path. Callback-time index agreement, the reported insert-seeded string-key setup, adapter cancellation, and non-local-only persistence remain outside this lane. -Its native-byte sync witness adds a one-key `[1]` to `[2]` replacement. After -commit, one update with the previous and current bytes must reach the subscriber while -public state and BasicIndex or BTreeIndex exact lookup agree with `[2]`. A -fresh equal-content view then remains silent. This -bounded sync cut records whole callback batches, so even an empty callback -after the equal copy fails the silence check. It does not cover optimistic -draft admission, other typed-array classes, or live-query notification. The [issue #1912 review record](oracle-reviews/issue-1912-eager-index-history.md) records the grammar controls, exact reviewed commit, and ORC audit. diff --git a/docs/contributing/oracle-reviews/basic-index-nullish-build.md b/docs/contributing/oracle-reviews/basic-index-nullish-build.md index 23e400c030..d1a3e719d4 100644 --- a/docs/contributing/oracle-reviews/basic-index-nullish-build.md +++ b/docs/contributing/oracle-reviews/basic-index-nullish-build.md @@ -130,3 +130,14 @@ type errors. `git diff --exit-code -- packages/db/tests/index-update-oracle.property.test.ts` confirmed that the oracle file again matched the reviewed commit. No calibration fault remains. The preceding ORC-007 and ORC-012 gaps are closed by this executed evidence. + +## Transfer-only narrowing + +The later production refactor groups each normalized build value during the +existing entry pass. It removes the intermediate entries array and keeps the +wrapper sort that fixes the nullish position. A fresh hostile recheck read +the source Map model, original unit ledgers, and current build code. Its +focused index, work, and Collection-index run passed 535 tests with no type +errors. The recheck found no lost direct-index assertion after the grouping +change. It did not rerun the original wrong-sort mutant or extend the law to +raw duplicate source keys or custom callbacks within one normalized value. diff --git a/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md b/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md deleted file mode 100644 index 758f63a1ea..0000000000 --- a/docs/contributing/oracle-reviews/borrowed-date-indexed-range.md +++ /dev/null @@ -1,159 +0,0 @@ -# Borrowed Date prototypes in indexed ranges - -Scope update for PR #2116: the tests and contract claims described below are -historical. The current indexed range owner keeps ordinary local and -cross-realm Date source values and bounds, while borrowed prototypes, -subclasses, and overridden methods are outside its contract. See the -[coverage map](../oracle-coverage.md) for the active witness. - -A Collection index may change the work used to answer a range predicate, but -it must not change the public key set. The live-query architecture requires a -range to fall back when indexed values do not share the bound's relational -domain. A native Date subclass has a Date timestamp even across JavaScript -realms; an ordinary object made with `Object.create(DateSubclass.prototype)` -does not. The latter cannot enter the index's Date range domain merely because -`instanceof Date` returns true. - -The receiving owner is `packages/db/tests/query/index-path-collision-oracle.test.ts`. -Its finite model applies JavaScript numeric coercion to supplied Date values -and treats a throwing borrowed-prototype coercion as a nonmatch. The driver -compares public `currentStateAsChanges` keys before and after installing -BasicIndex or BTreeIndex. Its lookup spy and `optimizedOnly` read establish -whether the index is eligible. The source is a controlled, ready in-memory -Collection, so no provider or scheduling claim follows. - -At `3668d1b88`, a hostile WHERE review found a public counterexample: -`gt(v, Date(1))` omitted a borrowed-prototype row in the unindexed snapshot -and included it after installing BasicIndex. The new oracle reproduces the -same result mismatch for both index types. A second history supplies the -borrowed object as a predicate bound; its scan result is empty, and an index -lookup would be unsafe. The positive control uses a genuine foreign Date -subclass and requires an optimized indexed result, so the repair cannot -disable all Date range optimization. - -Production now uses the comparator's guarded native Date timestamp check when -it classifies an indexed value's domain and a range bound's eligibility. The -ordinary borrowed object therefore takes the scan path; native local and -foreign Dates remain eligible. The original code failed the after-index key -set, and the repaired focused oracle passes all 16 cases. The Date comparison owner -separately checks direct timestamp ordering and borrowed-object identity. - -This witness covers the named `gt` and `lt` snapshots, two index types, one -local borrowed prototype, and one foreign native subclass. It does not prove -arbitrary coercion methods, invalid Dates, later publication histories, -other range operators, proxies, or provider reads. A fresh hostile review of -the committed repair is pending. - -## Native timestamp law for subclass ranges - -The frozen-head review at `210651d207eefe9bbc025c780f6259e470225c9e` -found a separate production defect. A genuine Date subclass can override -`valueOf` so native relational operators place it away from its actual -timestamp. The working-tree oracle uses a local and a foreign native subclass -whose overrides reverse their apparent positions. An independent model calls -the captured native `Date.prototype.getTime` and checks public scan results -for `gt`, `gte`, `lt`, and `lte` against a Date bound. The old relational scan -failed at this public key-set comparison. After the production repair, both -BasicIndex and BTreeIndex preserve those sets in ordinary and -`optimizedOnly` snapshots; lookup spies prove the indexed path is reached. -Four mixed Date/number scan predicates check both operand positions without -claiming mixed-domain index eligibility. The focused receiving run passed -132 tests. - -This extends the range witness beyond the earlier borrowed-prototype `gt` -and `lt` histories. It covers one ready in-memory Collection, fixed rows, -four Date-bound operators, and both index types. Invalid Dates, -arbitrary coercion methods, later publication histories, provider cursors, -and generated subclass histories remain outside the evidence. A fresh -committed-head hostile review is pending. - -Review of committed head `d974d4a4f` found that the misleading `valueOf` -subclasses were source values only. The fixed matrix now also uses a native -subclass as the predicate bound on both operand sides, where its override -returns a number opposite the native timestamp. Public scan and indexed -key sets still come from captured native `Date.prototype.getTime` values. -The focused owner passed 18 tests. This reaches the Date-bound compiler path; -it does not establish arbitrary coercion methods, generated subclass -histories, or other hosts. Fresh hostile review of the final committed -follow-up is pending. - -## Read-only review of `0c236f448` - -The hostile reviewer ran 18 focused cases and found a false-green reference -model. The earlier expected-key function called the native Date intrinsic on -source objects after those objects had entered the Collection. A source Date -changed after admission could therefore change the model's expected answer -along with production. A read-only probe demonstrated that risk; it did not -report a production range failure. - -The model now freezes each source row's native timestamp before Collection -admission and derives every expected public key set from those numbers. A -post-check Date mutation confirms the recorded expectation stays fixed. The -subclass bound now reaches all four effective relations in both operand -positions; source values include local and foreign subclasses whose -`valueOf` disagrees with native `getTime`. Public scan and optimized indexed -snapshots still compare against the same frozen model. The focused owner -passes 18 tests after the change. This is a fixed in-memory matrix, not -evidence for later publication, other realms, arbitrary coercion, or provider -cursors. The new witnesses await hostile review at a committed head. - -## Read-only review of `d917ba4c9` - -The exact-head hostile review found an enforcement gap in the fixed Date-range -matrix. Its only source row at the bound's intrinsic timestamp was a plain -`Date`. A wrong comparator that sent a native subclass tied with the bound to -the strict side of a range could pass every subclass assertion: the local and -foreign subclass rows were previously below or above the bound. This was an -oracle-history gap, not a demonstrated production defect. The repair belongs -in the source-row grammar and exact-key comparison; changing the accepted -timestamp law or production comparator was not justified by this finding. - -The working-tree owner now admits a local and a foreign native Date subclass -at the bound's exact millisecond timestamp. Their `valueOf` overrides report -`-100` and `100`, respectively, while the bound's native timestamp is `1`. -The model freezes all source timestamps through `Date.prototype.getTime` -before Collection admission. Thus `gt` and `lt` exclude all three tied rows, -while `gte` and `lte` include all three, including the plain-Date control. -The same exact expected key sets judge the unindexed scan, ordinary indexed -snapshot, and `optimizedOnly` indexed snapshot for BasicIndex and BTreeIndex; -the lookup spy confirms both indexed paths. A wrong-result control removes -`foreign-equal` from a real inclusive scan result and reaches the same -exact-key `toEqual` comparison, which rejects it as an assertion failure. -The focused Vitest run passed 18 tests with no type errors. These results are -working-tree evidence after the `d917ba4c9` review, not a fresh committed-head -hostile review. - -### Oracle-guide applicability for this repair - -| Requirement | Outcome at this working tree | -| --- | --- | -| ORC-001 | The live-query guide's comparison semantics promise native Date timestamp ordering, including foreign Dates; the owner limits its claim to fixed ready-Collection snapshots and the named Date domain. | -| ORC-002 | Expected keys come from native `getTime` numbers frozen before Collection admission and simple numeric relations; no production comparator or index classifier computes the reference. | -| ORC-003 | Opening prose states the law and limits. Adjacent comments identify the model, finite row and predicate grammar, Collection driver, public key observation, and scan/index checkpoints. | -| ORC-004 | Not triggered: this is a fixed bounded matrix, not a generated-history property. | -| ORC-005 | Public `currentStateAsChanges` keys are compared exactly before indexing and after BasicIndex or BTreeIndex installation; `optimizedOnly` and lookup spies witness the indexed path. | -| ORC-006 | Removing `foreign-equal` from a real `gte` scan result fails the exact-key assertion at its intended checkpoint. The control is an assertion failure, not a setup or runtime failure. | -| ORC-007 | Not triggered: the fixed matrix makes no important generated-property claim. | -| ORC-008 | Not triggered: the reference recomputes from frozen rows and introduces no stateful transition model. | -| ORC-009 | The reference maps plain, local-subclass, and foreign-subclass Date values to one intrinsic millisecond timestamp domain. The Collection retains the source values; the model intentionally does not represent their overridden coercion methods as ordering state. | -| ORC-010 | The owner's cleanup wrapper preserves a primary comparison failure as the `AggregateError` cause if release also fails; its separate cleanup witness checks both errors and release order. | -| ORC-011 | Not triggered by this review: the named fault was a missing tied-subclass history, not a plausible semantic error shared by production and the native-timestamp reference for which a second formulation was identified. | -| ORC-012 | This table records the other applicable requirements and explains non-applicable triggers. It does not substitute for a new exact-head hostile review. | -| ORC-013 | `local-equal` and `foreign-equal` reach the boundary premise. Inclusive and exclusive relations on both sides distinguish correct timestamp ties from a wrong strict-side rule; the wrong-result control rejects an omitted tie. | -| ORC-014 | Not triggered: the claim is limited to the controlled ready in-memory Collection and makes no real-provider or host handoff claim. | - -The bounded claim remains: a ready in-memory Collection, fixed rows, these -four range operators and operand arrangements, two index types, and public -snapshot keys. Later publication, generated subclass histories, invalid Dates, -arbitrary coercion methods, provider cursors, and other hosts still need their -own witnesses before broader closure. - -## Exact-commit review of `5c97f18dc` - -A fresh read-only reviewer found no blocker in the tied-subclass Date-range -matrix. The frozen native timestamps predict strict exclusion and inclusive -inclusion of all three tied keys. Exact public key arrays, index lookup spies, -and the omitted-key wrong-result control reach the intended scan and indexed -checkpoints for both index types. The focused owner passed 18 tests with no -type errors. This verdict remains limited to ready in-memory snapshots and -the fixed rows, operators, and bounds described above. diff --git a/docs/contributing/oracle-reviews/captured-inarray-publication.md b/docs/contributing/oracle-reviews/captured-inarray-publication.md deleted file mode 100644 index 4de402a7a3..0000000000 --- a/docs/contributing/oracle-reviews/captured-inarray-publication.md +++ /dev/null @@ -1,308 +0,0 @@ -# Captured `inArray` membership at source publication - -The accepted contract for a live query that captures a caller-owned `inArray` -list is that its current members govern the whole existing result when the -next source event is processed. Changing the list alone publishes nothing. -An untouched row can enter or leave after an unrelated source update. This -query input is distinct from an already submitted `LoadSubsetOptions` request: -the latter's membership data remains immutable for its lifetime under -`packages/db/src/query/live/ARCHITECTURE.md`. - -The primary receiving owner is -`packages/db/tests/query/where-predicate-publication-oracle.property.test.ts`. -Its fixed model filters authored source facts with the list's current -members; it does not use compiled membership or source-index decisions to -predict expected keys. A direct Collection snapshot checks current members -after a source transaction. A retained live-query Collection starts with -members `[1]`, changes them to `[2]` and then `[2, 1]`, and delivers an -unrelated source update each time. It compares the complete public key set -and exact insert/delete events for untouched rows, then checks that an -unchanged list does not republish those rows. A separate ordered limit-one -history admits a previously excluded row ahead of the old top and then -removes all prefix matches, requiring refill to the next current member. -A design review then found that this green history missed an unseen source -event. A second ordered history changes `[1]` to `[0]` and deletes rank 4, -outside the acquired finite prefix. Before repair, its public result stayed -`['1']` instead of `['0']`. The repaired dynamic-list ordered plan requires a -full-source acquisition, so that delete reaches the query and the receiver -passes. This costs a full-source acquisition for such ordered plans; the -in-memory receiver does not measure a real provider's bandwidth or latency. -A controlled on-demand ordered receiver checks that its first `loadSubset` -request has no `where`, `orderBy`, or `limit`, then checks the public result -after an unseen delete and list change. - -The controlled on-demand receiver deliberately returns only its first row -when a source request contains a `where` predicate. The query must acquire -unfiltered rows so a later list member is available after an unrelated -update. An inner join receiver changes the list, then updates an unmatched -row on either source side. Neither write yields a joined-row delta, but each -must publish one coherent reverse insert/delete batch for the already joined -rows. A QueryRef receiver puts mutable membership in the outer query while -the inner filter suppresses the triggering source change. A correlated -include receiver changes an unrelated child row and checks the parent row's -child Collection against current membership. Earlier fixed Date and byte-array -witnesses mutated items in place inside their captured lists and observed a -changed result at the next unrelated source event. The current scope is -clarified below. - -The pooled-query receiver is -`packages/db/tests/query/pooled-live-query-oracle.property.test.ts`. A query -with `eq(group, 'g')` and a residual `inArray` list changes membership, then -updates a source row in another equality group. A pooled group would not -notify that view, so the fixed receiver requires the live-query Collection -path and checks the complete result after the event. The unguarded pooled -path failed this public snapshot before the guard; the receiving cut passes -after fallback. This does not assert that the fallback has the same work cost -as a pooled view. - -A 40-row work witness observes indexed reads of one captured list slot. An -ordinary source update with unchanged membership reads that slot fewer than -ten times; changing the list before the next update reads it at least 40 -times and refreshes the result. This rejects an all-candidate membership -rescan for every ordinary event in that fixed direct-query path. It does not -establish a general time bound, provider work bound, or cost for the other -query shapes above. Source inspection shows that each delivered graph event -snapshots the captured list in work proportional to its length; a list change -also revisits retained candidates. The fixed counter observes one list slot, -not an asymptotic cost bound for all list values and query shapes. - -At committed head `0c236f448`, the evaluator's compiled equality owner passed -249 focused tests for its declared value grammar. That result did not test -the retained live-query result after a caller changed a captured list. The -new fixed receiving histories address that separate publication boundary. -The revised source and receiver still await an exact committed-head hostile -review. A further design review found an on-demand LEFT-join gap: -main rows `one(v=1,g)` and `two(v=2,g)` join group `g`, while group `h` is -outside the lazy joined-side demand. After the list changes from `[1]` to -`[2]`, deleting `h` left the public result at `one` instead of changing it -to `two`. The joined source event was real, but keyed lazy demand did not -deliver this unacquired row to the query. The new public receiver was RED -before the compiler disabled keyed lazy joined-side demand for dynamic-list -compilation and GREEN afterward. This controlled source does not establish -real-provider behavior or the transfer cost of full acquisition. A parallel -on-demand include receiver has parent `g`, children `one(v=1,g)` and -`two(v=2,g)`, and spare child `(v=3,h)` outside the former lazy child demand. -After `[1]` changes to `[2]`, deleting the spare must change the materialized -children from `[one]` to `[two]`. The working-tree receiver passes and checks -that the first child `loadSubset` request has no `where` predicate. A separate -public probe saw `[one]` before this gate and `[two]` afterward; the oracle -suite's combined run establishes the receiving check after the gate. - -An earlier working-tree focused and neighboring run passed 988 tests across -11 suites, with no type errors. It includes the direct, pooled, ordered, -on-demand, joined, QueryRef, include, Date/byte, and 40-row work receivers. -The later full DB run exposed two older positive fixtures that assumed an -ordinary mutable `inArray` literal could still use index pushdown or forward -a finite subset predicate. Those assumptions conflict with the accepted -captured-list law: a changed member can admit an existing row outside the -former subset. The index-optimization and subquery-forwarding tests now use -`Object.freeze` lists, so they continue to check those optimizations within -the immutable-list domain. Their 34 focused tests pass. This fixture change -does not retire either test. `ExpressionLike` now accepts `ReadonlyArray` so -the frozen expressions typecheck; a separate error-catalog failure was fixed -by recording code 243. The final DB suite passes 12,383 tests across 282 files, -and DB-IVM passes 693 tests across 45 files. Both suites have no test type -errors. The DB and DB-IVM builds and explicit source and test typechecks pass. -Real providers, broader source schedules, arbitrary list values and histories, -and general work bounds remain outside these fixed receivers. Exact -committed-head hostile review remains pending. - -## Query-stage transfer - -The new primary stage owner is -`packages/db/tests/query/captured-inarray-stages-oracle.test.ts`. Its plain -array model checks numeric list membership after `[1]` changes to `[2]` and -one unrelated source transaction updates value 3. Values 1 and 2 remain -untouched. The initial nine fixed public witnesses cover selected membership, -including one coherent update batch and a callback that reads the complete -new result. The other cuts cover a retained ordered limit, an on-demand -selected value under an ordered limit, group-key moves, an aggregate argument, -HAVING, membership on each JOIN operand side, and an inner QueryRef SELECT -consumed by an outer filter. The -controlled on-demand witness also requires its first `loadSubset` request to -omit `where`, `orderBy`, and `limit`. Three later fixed cuts check a distinct -four-row limited ORDER result, grouped SUM/min/max arguments while a -different group receives the source update, and a mutable aggregate argument -wrapped by a selected addition. These are stage and acquisition -cuts of the same accepted source-publication law, not a generated grammar. - -The nine initial witnesses were RED against the earlier stage implementation -and GREEN after the working-tree repair; each also compares public rows. -Source inspection shows one list revision driven by raw source events across -the compiled graph. A revision -revisits retained stage inputs; each stage retains its prior output for -retraction before evaluating current membership. The repair applies this at -JOIN, WHERE, SELECT, GROUP and aggregate input, HAVING, and ORDER, with an -optional mapped-input hook in the D2 groupBy operator. The fixed witnesses -establish the named observations, not all combinations of these stages. - -Hostile design review found one further public counterexample: after -membership changed, a limited ordered result duplicated a source row. A -four-row, limit-three receiver sorts each source identity once by current -membership and key. It was RED before Collection deferred order maintenance -for a layout change and GREEN afterward. A separate grouped receiver checks -SUM, min, and max arguments for an untouched group after a different group -updates; it passes in workspace Vitest. The standalone probe that suggested a -min/max failure imported stale DB-IVM output, so it did not establish a -production defect. The wrapped-aggregate receiver selects -`add(sum(caseWhen(inArray(row.v, members), row.v, 0)), 10)`. After `[1]` changes -to `[2]` and an unrelated row updates, the plain-array model requires total -12; the earlier implementation retained 11. Recursive aggregate-input -detection makes the twelfth public witness GREEN. Hostile probes of nullish -grouped/public output and projected sorting were bounded GREEN controls, not -confirmed production defects. The earlier working-tree run passed 11 stage -tests and 33 WHERE/publication cases; the final DB suite includes all twelve -stage witnesses and the WHERE/publication receivers. The exact committed-head -hostile verdict remains pending. Real-provider event delivery, arbitrary list -values or histories, mixed stage combinations, and general work bounds remain -outside this fixed stage owner. - -### Provisional oracle-guide applicability - -- **ORC-001–003:** The accepted next-source-event law and fixed numeric limits - are stated in the owner; plain-array `includes` computes expected results - without compiler or D2 outputs; opening prose distinguishes model, source - history, public observations, and comparison cuts. Exact-head review of - that prose remains pending. -- **ORC-004, ORC-007, ORC-008, ORC-009:** No generated history, seed/path - campaign, stateful reference model, or model-only subsystem vocabulary is - introduced by these fixed recomputations. -- **ORC-005–006:** The real live-query Collection is compared immediately - after preload and one source commit. SELECT additionally checks one public - subscriber batch; the on-demand case checks request shape before comparing - public rows. The nine initial pre-repair runs failed their receiving checks - and passed after repair in the working tree. The separate limited ORDER - duplicate was also RED at its public comparison and GREEN after its repair. - The wrapped aggregate was RED at total 11 against expected 12 and GREEN - after recursive detection. The grouped SUM/min/max receiver passes in - workspace Vitest. This - calibrates these fixed cuts without claiming all legal histories. -- **ORC-010:** `withOracleCleanup` releases each source, live-query Collection, - and subscription where present while preserving the primary assertion - failure. No shrink or replay capture occurs. -- **ORC-011:** The limited ORDER duplicate and top-level-only aggregate - detection name additional wrong designs; plain-array ordering and wrapped - sum predictions reject them. The apparent min/max - failure came from a stale standalone module import; the workspace receiver - passes. No general stage-law closure is claimed. -- **ORC-013:** This owner claims twelve fixed cuts and checkpoints, not all - legal values, histories, or stage combinations. New reachable cases remain - within this stage owner's coverage responsibility. -- **ORC-014:** The on-demand source is controlled. Its unfiltered request and - public result establish the Collection path; real-provider adherence and - bandwidth remain open. -- **ORC-012:** This applicability record is provisional. Verdict-critical - evidence needs a fresh hostile review tied to the exact committed head. The - full DB and DB-IVM suites, builds, and source/test typechecks pass at the - working-tree checkpoint above. - -## Exact-commit publication review of `5c97f18dc` - -A fresh read-only reviewer found no blocker in the retained direct WHERE -receiver's callback and pre-event checks. The accepted law requires a whole -current result after the next source transaction. The fixture changes -`[1]` to `[2]` and then `[2,1]`, while an excluded source row supplies each -event. The independent source-row model predicts all public keys. The -recorder preserves callback boundaries, checks one complete delete/insert -batch, and reads the whole public result during delivery. The focused -checkpoint passed with no type errors. The earlier pre-event assertion -covered only the synchronous call stack. The next repair waits one microtask -turn after each edit, then checks the old keys and zero callbacks again. The -33-test focused WHERE owner passes. Later timers, arbitrary asynchronous -schedules, and real providers remain outside this fixed history. A fresh -review of the committed microtask addition remains pending. - -The captured-list stage owner received a separate read-only review at -`d917ba4c9`. All twelve fixed checks passed. Four temporary public probes -also passed for repeated list edits, combined stages, an unmatched LEFT JOIN, -and a UNION refreshed by another source. Those probes were removed and do -not extend the committed owner's claim. Broader callback coherence and real -provider behavior remain open for that owner. - -## Exact-commit microtask review of `d93ae1dd6` - -A fresh read-only reviewer found no behavior defect in the retained direct -WHERE history. The selected test passed with no type errors. The independent -source-row model predicts the current whole key set. The direct receiver checks -old keys and zero callbacks through one microtask after each list edit. After -the next source transaction, it checks one complete callback batch and reads -the current whole result during that callback. The compiler schedules list -refreshes from source input events; the reviewer found no list-only scheduling -path. The review found one contract-wording gap: the oracle said a list edit -"need not publish," while the architecture says it does not publish. The -follow-up changes that prose to the accepted law. The existing split-batch -control calibrates event coherence, but no separate delayed-schedule mutant -calibrates the new microtask checkpoint. Nested microtasks, timers, arbitrary -list/event histories, and real providers remain outside this witness. It does -not retire an additional legacy test block. - -## Unbounded ordered acquisition after merging `origin/main` - -A fresh hostile merge review found that the finite ordered receiver did not -cover an unbounded `orderBy` over a captured mutable list. The unbounded path -used ordinary subscription hints and sent `orderBy` with its unfiltered -full-source request. A controlled on-demand adapter that accepts full-source -requests but rejects sorted requests made preload reject and put the -live-query Collection in error. This contradicted the architecture's -full-source-without-hints rule. The main branch's new readiness law was not -the cause: a separate held-load probe reached the correct row and became -ready at subset settlement while the source remained loading. - -The WHERE publication owner now has an unbounded ordered receiver with a -lagging on-demand source. Its independent model filters all three authored -source rows against the list's current members and orders the matches by -`v`. The adapter delivers `a-two` before `z-one`, records the complete first -request, and rejects any `where`, `orderBy`, or `limit` hint. While the subset -is pending, the list changes from `[1]` to `[2, 1]` and an unrelated source -update clocks the graph. The live-query Collection and its preload stay -pending through that event. After subset settlement, the live query becomes -ready and publishes `[z-one, a-two]` in local order, even though the source -Collection remains loading. The request assertion failed before the repair -with `{hasOrder:true}` and passed after it. A companion Effect receiver checks -the same unbounded acquisition shape and its initial `enter` event. - -A hostile comparator mutant returned equality for every local comparison in -this unbounded captured-list path. The first fixture let it pass because key -order matched value order. With opposing key and value orders, the mutant -publishes `[a-two, z-one]` and fails at the public key-order assertion. The -unchanged implementation passes. The Effect receiver checks initial -acquisition and initial entry only; later membership refresh is outside it. - -Compilation now carries the graph-wide captured-list finding to live-query -and Effect subscriptions, which omit order and limit hints for that graph. -This witness covers one first acquisition and one pending source event. It -does not establish repeated acquisitions, cleanup/restart, or every graph -shape. A separate hostile source review found no further production defect -in the acquisition flag path. It identified a remaining composition: an -unbounded ordered include child with mutable membership, a second eager -source, and child subset settlement while the child Collection stays loading. -That composition needs a receiving witness for request hints, readiness, and -whole-result refresh. An Effect with `skipInitial` and a lagging on-demand -source is another untested intersection. These are coverage limits, not -observed production failures. The execution-based hostile review found no -further defect in the two new witnesses. - -## Exact pushed-head review at `b51f0c79d` - -A read-only hostile reviewer reran the unbounded Collection and Effect -witnesses on the clean pushed head. Both passed with no type errors. Three -isolated mutants failed at their intended observations. Sending `orderBy` in -either first request failed the recorded no-hints request shape. Treating the -second graph-input revision as readiness failed the pending-status assertion -before subset settlement. Returning equality for every local comparison in -the unbounded captured-list path failed the opposed key/value order check. -The reviewer found no further defect in those two witnesses. Their stated -composition and provider limits still apply. - -## Scope refinement for native list items - -The maintainer declined a guarantee after callers modify standard data values. -The older Date and byte-array witnesses above edited an item already held in -the captured list. The current evaluator and WHERE publication witnesses -instead replace the list entry with a new Date or byte array, then compare -current membership at evaluation or at the next source event. The prior -in-place observations remain historical evidence only. No current oracle -claims what a retained query does after an existing native item is mutated. -The compiler now snapshots entry identity rather than re-encoding each item -or checking its property descriptors at every source event. Same-length -replacement witnesses reject an array-reference or length-only detector. diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md index 67ffc7efde..55ddc816f4 100644 --- a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md @@ -1,143 +1,62 @@ # Comparison unit retirement: non-finite default order -The source-isolated Field Lab loss audit compared `comparison.test.ts` with +A source-isolated Field Lab loss audit compared `comparison.test.ts` with `comparison-oracle.property.test.ts` at frozen head -`f6857fee9de59dc82cbe596ca7df4a18152004ed`. The receiving grammar used -finite numbers and valid Dates; it had no NaN input, invalid Date input, or -`defaultComparator` sort observation. This ledger freezes the five relevant -legacy blocks before retirement. The other five tests in that file have -separate Temporal, symbol, and `compareValues` obligations and remain. +`f6857fee9de59dc82cbe596ca7df4a18152004ed`. The receiver initially +excluded NaN and invalid Dates and did not observe `defaultComparator` sorting. +The audit froze the five blocks below before retirement. The remaining three +unit tests cover symbol order and `compareValues` behavior at another entry +point. -| Legacy block | Source assertion at the function-return or sort checkpoint | Loss in frozen receiver | Receiving check | -| --- | --- | --- | --- | -| `orders NaN greater than every number` | `ascComparator(NaN, 5)` is positive and the reverse negative under `DEFAULT_COMPARE_OPTIONS`. | NaN input excluded. | Tagged numeric finite/non-finite pairs compare signs in both directions. | -| `treats NaN as equal to NaN` | Direct comparator returns zero. | Reflexivity grammar excludes NaN. | Duplicate tagged NaNs compare as one order position. | -| `produces a stable total order with NaN sorting last` | Sorting `[3, NaN, 1, 5, NaN]` with `defaultComparator` yields finite prefix `[1,3,5]` and two NaNs at the end. | No default-sort production path. | The fixed tagged history reproduces these values; a stable source-rank sort predicts every output row ID. | -| `keeps null before non-null values regardless of NaN` | Sorting `[5, NaN, null, 1]` yields null, 1, 5, NaN. | Null and finite signs were separate from NaN/default-sort behavior. | The fixed tagged history combines null and NaN, and the generated grammar varies null presence. | -| `orders an invalid Date greater than valid Dates` | Direct comparator orders invalid Date after `2023-01-01` in both directions. | Date grammar excludes invalid Dates. | A fixed date history uses that day; generated valid epoch-day Dates and invalid Dates compare both ways and in sorted arrays. | - -The changelog entry for #1582 promises PostgreSQL-style NaN ordering and -applies it to invalid Dates, with null placement independent of it. The -`DEFAULT_COMPARE_OPTIONS` constant and `defaultComparator` document the -default ascending, nulls-first path. The model receives explicit tags for -null, finite, and non-finite input values and compares rank and finite epoch -day; it does not call the production unorderable classifier to predict a sign. -Direct `ascComparator` returns and `defaultComparator` sort row IDs are the -observations. The comparison occurs at each function return and after sorting. - -The bounded grammar uses 1–5 distinct finite numbers or valid Dates in -epoch-day range -20,000..20,000, one or two NaNs/invalid Dates, and optional -null. Fixed histories preserve the old examples. Mixed number/Date order, -Temporal values, symbols, other option sets, and public query/index paths are -outside this transfer. A separate direct `compareValues` behavior remains in -the old file because it has a different production entry point and fallback -law. - -## Review and execution - -The five listed blocks were removed. The receiving oracle and remaining legacy -file passed with 102 tests (including one Vitest-selected integration file), -and Vitest reported no type errors. Both the fixed seed and seedless campaigns -run the same 100-case tagged property. Its direct replay at property 23, -seed 20260911, path 0 passed. The direct comparator control ties a -non-finite value to a finite value and fails the sign comparison in both -number and Date families. A separate wrong-sort control ties NaN during -`defaultComparator` sort and fails the exact output-row-ID comparison. - -The fresh hostile reviewer `/root/review_nonfinite_round5` read the complete -source file, ledger, owner, production comparator, public authority, and -coverage map. It found one exact-assertion loss: Vitest `toBe(0)` on NaN/NaN -rejects negative zero, while the initial sign-only checker did not. The -revised oracle checks the raw result with `toBe(0)` for every non-finite pair; -the reviewer rechecked that change and the separate sort control and found no -remaining loss from the five retired blocks. Its challenge of the broader -Date law found a separate public cross-realm bug. That has its own RED witness -and review record in `cross-realm-date-order.md`; it does not make the retired -same-realm examples unique. - -## Oracle guide audit - -| Requirement | Evidence and limit | +| Retired assertion | Receiving check | | --- | --- | -| ORC-001 | Changelog #1582 and the live-query guide promise NaN/invalid-Date order; `DEFAULT_COMPARE_OPTIONS` and `defaultComparator` establish the tested default path. This transfer claims number-only and Date-only same-realm sorts with optional null. | -| ORC-002 | Input tags and supplied finite values determine expected signs and sort ranks. The model never imports production classification or comparator output. | -| ORC-003 | Opening and adjacent prose identify the law, tagged model, bounded inputs, comparator and sort drivers, result/sign and ordered-ID observations, and immediate checkpoints. | -| ORC-004 | Fixed inputs reconstruct all five source blocks. Generated families, finite values, duplicate non-finite values, and optional null each distinguish a missing behavior. The stated range is -20,000..20,000 finite epoch days, 1–5 distinct finite inputs, and 1–2 non-finite values. Mixed types and unrelated value classes are excluded. | -| ORC-005 | The production `ascComparator` receives `DEFAULT_COMPARE_OPTIONS`; `defaultComparator` drives an actual stable JavaScript sort. Pair signs, exact zero, and every sorted row ID are compared at return. | -| ORC-006 | Separate wrong-result controls reach and fail the direct-sign and sorted-ID comparisons. | -| ORC-007 | Fixed-seed and seedless lanes use the same property and 100-run budget through the existing direct seed/path replay harness. Property 23 replayed at seed 20260911, path 0. | -| ORC-008 | Not applicable: there is no stateful model. | -| ORC-009 | `category` and `order` are model-only tags, not production storage states. | -| ORC-010 | Existing comparison harness preserves primary checkpoint through shrinking and direct replay; this stateless check owns no cleanup. | -| ORC-011 | No shared production/model semantic fault was named within this bounded same-realm transfer. The cross-realm hostile finding has a separate public-query formulation. | -| ORC-012 | This versioned record contains the source ledger, execution, hostile verdict, limits, and the exact-head recheck below. | -| ORC-013 | Finite versus non-finite and null versus non-finite comparisons, plus the wrong-sort control, distinguish a tie-all or misplaced-nulls rule within the bounded default path. | -| ORC-014 | Not applicable: the direct comparator and JavaScript sort run in the package test host; no controlled external provider is extrapolated. | - -## Exact committed-head recheck - -At published commit `9a145fba130f27c914a4a1fd4f44c5871a917483`, the fresh -hostile reviewer `/root/exact_review_comparison_round5` found no loss from the -five retired same-realm blocks. It checked the complete source file against the -tagged model, exact comparator zeros, and stable sorted row IDs. The focused -comparison suite passed 102 tests with no type errors. - -The reviewer also found a separate bug in the broader cross-realm Date repair. -A native foreign Date with an own `Symbol.toStringTag` escaped the production -Date detector. The new direct and public oracle controls failed against that -published implementation. The repair and its fresh committed-head verdict are -recorded in [the Date order review](cross-realm-date-order.md). This finding -does not restore a unique assertion to any of the five retired blocks. - -## Frozen-head hostile recheck and adjacent byte-key sensitivity - -At `210651d207eefe9bbc025c780f6259e470225c9e`, the hostile review again -found no lost assertion among the five retired same-realm number/Date blocks. -It reconstructed the ledger operands and the direct comparator and sort -checkpoints. This verdict applies to the frozen commit, not to the later -uncommitted repairs. - -Source inspection found a separate sensitivity gap in the comparison owner's -native byte normalization law. Its changed-byte property reached the final -byte, but could miss a wrong implementation that ignored an interior byte. -The working-tree oracle now requires different normalized keys for -`[1, 0, 2]` and `[1, 9, 2]` and runs an independently supplied changed-byte -property at the direct `normalizeValue` return. No hostile mutant was run for -this addition, so the fixed pair and property are sensitivity evidence, not -an exact-commit kill. The [Temporal transfer](comparison-unit-retirement-round-6.md) -and [alias owner](temporal-equality-alias.md) record their separate domains. - -Review of committed head `d974d4a4f` found that the changed-byte generator -reached only lengths 1–50 while the owner's bounded key-size claim reached -200. The follow-up expands independently supplied changed-byte pairs to -lengths 1–200 for both direct `areValuesEqual` and `normalizeValue` checks. -A fixed 129-byte pair differs only at byte 64. The former fixed assertion -checked `normalizeValue` through `assertChangedBytes`, but did not call -production `areValuesEqual`; the 100-run generated campaign declared lengths -through 200 without guaranteeing a long changed-byte sample. Replaying that -fixed-seed generator and budget reached at most length 12 and changed index -10; an executed first-64-byte comparator fault survived the generated lane. -Exact-head hostile review of `d917ba4c9` found this direct-equality gap. - -The next fixed witness supplies a 129-byte pair changed at interior byte 64 -and a 200-byte pair changed at its last byte, 199. For each pair, it calls -production `areValuesEqual` in both directions at the direct-return -checkpoint, checks an unchanged copy, and compares the normalized keys. An -executed first-64-byte comparator fault fails the same changed-byte checker -used by the generated equality property for both pairs. The generated grammar -still spans lengths 1–200; these pins guarantee upper-position reach within -the normal 100-run budget. This is bounded sensitivity evidence, not a claim -that every byte sequence or all binary consumers were exercised. The focused -comparison owner passed 117 tests with no Vitest type errors; explicit test -TypeScript, ESLint, Prettier, and whitespace checks passed. Final exact-commit -hostile review of this repair is pending. - -## Exact-commit review of `5c97f18dc` - -A fresh read-only reviewer found no blocker for the five retired same-realm -comparison blocks or the two fixed byte positions. It checked the old unit -assertions, the independent tagged model, production comparison code, and the -new direct equality and normalization returns. The first-64-byte fault fails -at the intended changed-byte assertion. The focused comparison owner passed -117 tests with no type errors. This verdict does not establish every byte -position, offset view, or public binary query path. +| `ascComparator(NaN, 5)` is positive and the reverse is negative. | Tagged finite and non-finite number pairs check both signs. | +| `ascComparator(NaN, NaN)` returns exact zero. | Duplicate non-finite pairs check `toBe(0)`, including the sign of zero. | +| Sorting `[3, NaN, 1, 5, NaN]` puts the finite values in order and both NaNs last. | A fixed tagged history predicts every stable sorted row ID. | +| Sorting `[5, NaN, null, 1]` puts null first and NaN last. | Fixed and generated histories combine optional null with non-finite values. | +| An invalid Date sorts after a valid 2023-01-01 Date in both directions. | A fixed history preserves that day; generated valid epoch days and invalid Dates check both signs and sorted IDs. | + +Changelog #1582 promises PostgreSQL-style NaN and invalid-Date ordering. +`DEFAULT_COMPARE_OPTIONS` and `defaultComparator` supply the default ascending, +nulls-first path. The independent model receives tags for null, finite, and +non-finite values; it compares tag rank and supplied finite values. It never +uses production's unorderable classifier to predict a result. The driver calls +`ascComparator` and sorts with `defaultComparator`. It checks direct signs and +exact zeros at return, then compares every ordered row ID after sorting. + +The generated grammar uses 1–5 distinct finite numbers or valid Dates in +epoch-day range -20,000..20,000, one or two NaNs or invalid Dates, and optional +null. Number and Date histories remain separate. Mixed number/Date order, +Temporal values, symbols, other options, `compareValues`, and public query or +index paths are outside this transfer. + +## Review and replay + +The direct wrong-result control ties a non-finite value to a finite value and +fails the sign check. A separate wrong-sort control ties NaN during sorting +and fails the ordered-ID check. Fixed-seed and seedless campaigns share the +same 100-case property. Direct property 23 replay at seed `20260911`, path +`0`, passed. + +The latest hostile reviewer read the retired unit blocks, ledger, model, +production comparator, and coverage map. It found no lost assertion at the +declared direct and sort checkpoints. The focused comparison suite passed +100 tests with no type errors. Earlier PR revisions explored cross-realm and +long-byte behavior; those experiments are outside this transfer and are not +claimed by this record. + +## Oracle guide boundary + +The opening law and ledger state authority and limits (ORC-001). Tagged +finite values and source ranks give an independent computation (ORC-002); +the oracle places law, model, grammar, driver, observation, and checkpoint +beside their code (ORC-003). Fixed operands reconstruct the five old blocks, +and generated finite, duplicate non-finite, and null cases exercise nearby +histories (ORC-004). Direct comparator returns and sorted row IDs are the +declared observations (ORC-005). The two wrong-result controls +fail at those comparisons (ORC-006 and ORC-013). The replay above checks the +shared fixed-seed and seedless harness (ORC-007 and ORC-010). Model tags are +input labels, with no state transition or host provider (ORC-008, ORC-009, +and ORC-014). This ledger records the loss audit, hostile challenge, and +bounded result (ORC-011 and ORC-012); it makes no claim about other value +domains. diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md index 932662f5dd..e59994c673 100644 --- a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md @@ -1,157 +1,62 @@ # Comparison unit retirement: Temporal order -A source-isolated Field Lab loss audit compared the two Temporal blocks in +A source-isolated Field Lab loss audit compared two Temporal blocks in `packages/db/tests/comparison.test.ts` with the comparison oracle at frozen -head `c038b2b673a601d3bfbc70261f2cbb61f6f934ea`. The receiver expressly -excluded Temporal. The audit recovered these four assertions before the two -blocks were removed. The symbol block and both `compareValues` NaN blocks stay -in the unit file because they have different obligations. +head `c038b2b673a601d3bfbc70261f2cbb61f6f934ea`. The receiver had no +Temporal input. These four observations were frozen before the blocks were +removed. The symbol block and two `compareValues` NaN blocks stay in the unit +file because they have different obligations. | Retired block | Original observation | Receiving oracle | | --- | --- | --- | -| `orders PlainDate values by calendar date` | January 2024 precedes June 2024; the reverse comparison is positive; a fresh January value compares at exact zero. | Supplied ISO year/month/day tuples predict both signs and exact equal-date zero. The fixed case reconstructs January and June; generated tuples vary all three fields. | -| `treats ZonedDateTime values at the same instant as equal regardless of zone` | The 06:30 UTC and 12:00 +05:30 displays of one instant compare at exact zero. | Supplied epoch-nanosecond offsets predict both comparison directions. The fixed case reconstructs the two displays; generated cases vary offset zones, equal instants, and adjacent instants. | +| PlainDate calendar order | January 2024 precedes June 2024; reversing the operands is positive; a fresh January value compares at exact zero. | Supplied ISO year, month, and day fields predict both signs and exact equal-date zero. Fixed operands reconstruct January and June; generated fields vary each part. | +| ZonedDateTime instant order | 06:30 UTC and 12:00 +05:30 representations of one instant compare at exact zero. | Supplied epoch-nanosecond offsets predict both signs and zeros across four offset zones, including one-nanosecond neighbors. | Changelog #1519 promises Temporal static comparison for `orderBy` and range -operators. It says `ZonedDateTime` comparison uses the instant while `eq` -keeps the zone as part of identity. The receiving model uses calendar tuples -and supplied nanosecond offsets, not production's Temporal dispatch or -normalization. The driver constructs ordinary values from one polyfill copy. -Each `ascComparator` return under default options is the comparison cut. -This direct lane also probes the shared `compareValues` helper in hostile -dispatch histories. It does not claim public query, index, or provider behavior. +operators. It specifies instant order for ZonedDateTime, while `eq` retains +zone as part of identity. The receiver models PlainDate calendar fields and +ZonedDateTime instants independently of production dispatch and Temporal's +static comparator. Its driver builds ordinary values from one polyfill copy, +calls `ascComparator` with default options, and compares each direct return. +It does not claim `compareValues`, index, public query, or provider behavior. The grammar uses ISO PlainDates in years 2000–2030 with days 1–28. It uses ZonedDateTimes within two seconds of a fixed 2024 instant in UTC, +05:30, --05:00, and +14:00 zones. Equal-value branches construct distinct instances; -an adjacent-instant branch differs by one nanosecond. Fixed text-order and -millisecond-truncation controls fail the model comparison. Mixed Temporal -kinds, unsupported `PlainMonthDay` ordering, custom subclasses or prototype -replacement, and mixed native/polyfill implementations are outside this -transfer. The separate index owner below tests another production path. - -## RED witness and hostile review - -At unchanged production head `c038b2b673a601d3bfbc70261f2cbb61f6f934ea`, -the tag-wide comparator cache could learn a bogus static `compare` from the -first operand. A genuine PlainDate with an own `constructor.compare` returning -zero made January and June compare equal. A plain object with a forged -`Temporal.PlainDate` display tag could poison later genuine comparisons too. -Fresh-module histories prevent test order from seeding the cache first. The -focused run had eight passing Temporal checks and three intended failures at -the comparator-result assertions, with no type errors. A right-operand shadow -case also rejects a plausible repair that reads the second operand's own -constructor. A `compareValues`-first history reaches the other direct entry -point. The original four assertions pass in the fixed reconstruction. - -The fresh hostile reviewer `/root/hostile_temporal_direct` compared each -retired assertion with the new model and original source. It found the forged -tag, right-operand, and nanosecond precision gaps; the revised oracle added -those controls before production changed. The reviewer then rechecked the RED -run and found no remaining loss in the two retired blocks. It found a separate -published API-reference description of the old cache; this change updates -that page. Replaced prototypes, overridden subclass static comparators, and -mixed native/polyfill values remain excluded. These are recorded limits, not -claims that the comparator handles every altered Temporal object. - -The production repair removes the tag-wide cache and reads the static compare -function from the value's prototype on each call. The repaired direct and -evaluator suite passed 248 tests with no type errors. The exact committed-head -verdict follows after commit. - -The later equality review exposed a separate order defect: a genuine -ZonedDateTime with an own `Symbol.toStringTag` shadow was treated as an -ordinary object. A direct witness failed at the equal-instant -`ascComparator` return on the unrepaired code. The shared branded dispatch -now reads the Temporal prototype and checks native slots; direct -`ascComparator` and `compareValues` preserve the supplied instant order. -The index-order owner has a corresponding value-cursor witness. - -## Oracle guide audit - -| Requirement | Evidence and limit | -| --- | --- | -| ORC-001 | Changelog #1519 supplies calendar/instant order and zone-sensitive equality. The opening and this record bound value kinds, implementation, paths, and checkpoints. | -| ORC-002 | Supplied date fields and nanosecond offsets predict signs without calling production comparison, normalization, or Temporal static compare. | -| ORC-003 | Prose beside the models explains the law, grammar, direct production paths, return observations, and refinement checks. | -| ORC-004 | Fixed cases reconstruct all four source assertions. Additional year-, month-, and day-separated pairs distinguish calendar-field omissions; distinct instances, zones, equal instants, and one-nanosecond neighbors distinguish the other axes. The bounded ranges and excluded value classes are stated above. | -| ORC-005 | Both direct exported helpers run on real Temporal values. Their exact returns, signs, and equal-value zeros are compared immediately. This is not public query evidence. | -| ORC-006 | A text-order control and a millisecond-truncation control fail at the sign check. The original cache fails three hostile histories at the same comparison cut. | -| ORC-007 | The shared comparison harness runs fixed-seed and seedless 100-case campaigns with the same property and supports direct seed/path replay. Replay evidence is recorded below. | -| ORC-008 | Not applicable: tuple and instant models have no mutable state. Fresh modules control production cache history, not reference-model state. | -| ORC-009 | Date tuples and nanosecond offsets are model input labels. Comparator zero describes order position, while `eq` identity retains the zone. | -| ORC-010 | The shared harness keeps the original failure checkpoint through shrinking. The direct tests own no cleanup resource. | -| ORC-011 | Not triggered: the reviewer named no plausible fault shared by the calendar/instant model and production within this bounded lane. Text-order and millisecond injections are ORC-006 wrong-result controls. The index/public path has its own oracle. | -| ORC-012 | This record contains the source ledger, RED/GREEN evidence, hostile verdict, limits, and committed-head recheck. | -| ORC-013 | Equal instants in different zones and adjacent nanoseconds distinguish text order and millisecond truncation from the promised instant law. | -| ORC-014 | Not applicable: this lane has no controlled provider or claim across a host boundary. | - -## Direct replay and exact committed-head review - -The shared harness directly replayed the PlainDate and ZonedDateTime generated -properties as selectors 39 and 40 with seed `20260911` and path `0`. Each -requested replay passed with no type errors; 39 other properties were skipped -in each run. After the final index oracle edit, the full DB suite passed -12,154 tests in 278 files with no type errors. The DB package build passed. -At committed head `8b04592724fb943b04a56c6682757448d26a8ff6`, the fresh -hostile reviewer `/root/exact_temporal_comparison_review` found no lost -assertion in the two retired blocks. It reconstructed the original PlainDate -and ZonedDateTime operands and checked all four assertions at the direct -`ascComparator` return. Its focused run passed 112 comparison tests and 149 -evaluator/index tests with no type errors. It replayed properties 39 and 40 -with seed `20260911` and path `0`. The reviewer corrected the ORC-011 -classification above. Cross-realm Temporal values remain outside this verdict. - -## Review of the next committed head - -At `c4c07da42`, a fresh reviewer again found all four retired assertions -preserved. It found a helper-specific false green: the right-hand -constructor-shadow history exercised `ascComparator` with the shadowed -PlainDate, but passed ordinary values to `compareValues`. The direct helper -now receives that same shadowed right operand and must return the supplied -negative calendar sign. The focused comparison, alias, and utils run passed -258 tests with no type errors after this addition. - -The reviewer also reproduced asymmetric comparisons for ordinary objects -given a spoofed Temporal prototype. This is outside the declared -custom-prototype domain and is not evidence of closure for that domain. The -comparison owner needs a separate brand and mixed-value witness before -claiming it. Fresh review of the final committed head remains pending. - -## Frozen-head hostile recheck - -At `210651d207eefe9bbc025c780f6259e470225c9e`, a hostile ledger/source -review found no lost assertion in the two retired PlainDate and ZonedDateTime -comparison blocks. Their direct helper checkpoints and supplied operand pairs -remain covered. The review does not extend that verdict to the subsequent -working-tree changes. - -A distinct counterexample involved a genuine `ZonedDateTime` subclass whose -inherited native `.equals()` accepts an ordinary UTC peer. The old classifier -treated it as an ordinary object. The [Temporal alias owner](temporal-equality-alias.md) -now supplies the subclass equality and borrowed-prototype controls and records -the RED/GREEN production repair. This issue is outside the two retired direct -comparison assertions. The custom-prototype question above remains open. - -At committed head `d974d4a4f`, review exposed a second subclass boundary: -a genuine `ZonedDateTime` subclass with a shadowed prototype display tag was -classified as an ordinary object. The direct comparison owner now supplies -the instant and one-nanosecond successor independently, verifies the native -`Temporal.ZonedDateTime.compare` premise, and checks exact equal-instant zeros -and both signs through `ascComparator` and `compareValues`. The adjacent -[alias review](temporal-equality-alias.md#tagged-subclass-follow-up-after-d974d4a4f) -records the direct-index and public-`eq` RED/GREEN repair. This is an adjacent -product defect, not a recovered assertion from the two retired ordinary-value -blocks. Other subclasses, replaced methods, mixed implementations, and -arbitrary prototype chains remain open. A fresh hostile review of the final -committed follow-up is pending. - -Read-only hostile review of `0c236f448` ran 122 comparison tests and found -no lost assertion from the two retired ordinary Temporal blocks. It corrected -the coverage-map attribution for the tagged subclass: direct -`ascComparator` and `compareValues` compare the subclass with a +05:30 peer -at the same instant, while the alias owner separately checks native -`.equals()` against a UTC peer and the borrowed-prototype control. Those -observations support different boundaries; the direct comparison owner does -not itself prove the public equality result. The final committed follow-up -still needs a fresh hostile review. +-05:00, and +14:00. Equal-value branches create separate objects; adjacent +instants differ by one nanosecond. Mixed Temporal kinds, unsupported +PlainMonthDay order, subclasses, altered constructors or prototypes, and +mixed native/polyfill implementations are outside this transfer. + +## Review and replay + +The fixed cases reconstruct all four old assertions. A wrong comparator +that uses zoned display text fails the equal-instant sign check; a +millisecond-truncating comparator fails the adjacent-instant check. ISO +PlainDate display text has the same order as the tested calendar fields, so +the text-order control only distinguishes the zoned law. The generated +properties run fixed-seed and seedless 100-case campaigns. Direct replays +38 and 39 at seed `20260911`, path `0`, passed. + +The latest hostile reviewer read the retired assertions, model, production +comparator, and coverage map. It found no lost assertion within the direct +`ascComparator` scope. The focused comparison suite passed 100 tests with +no type errors. The production change removes a tag-wide dispatch cache. +For ordinary Temporal objects this preserves the required static comparison +while reducing code. Earlier PR revisions tested caller-modified constructors, +spoofed tags, and subclasses; the maintainer declined guarantees for modified +standard values, and those witnesses are not part of this transfer. + +## Oracle guide boundary + +The opening contract identifies #1519 and the direct-return limit (ORC-001). +Supplied fields and offsets determine the expected result without importing +production classification or comparison (ORC-002). The oracle places model, +grammar, driver, observation, and comparison beside explanatory prose +(ORC-003). Fixed operands and generated year, month, day, zone, equal-instant, +and adjacent-nanosecond dimensions cover the bounded grammar (ORC-004). +`ascComparator` returns are checked immediately (ORC-005). Wrong text and +truncated-instant results fail those checks (ORC-006 and ORC-013). Direct +seed/path replay verifies the shared campaign harness (ORC-007 and ORC-010). +The tuple and instant model has no mutable state or host provider (ORC-008, +ORC-009, and ORC-014). This record preserves the loss audit, hostile review, +and explicit limits (ORC-011 and ORC-012); it makes no cross-boundary claim. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-order.md b/docs/contributing/oracle-reviews/cross-realm-date-order.md deleted file mode 100644 index 845f46c113..0000000000 --- a/docs/contributing/oracle-reviews/cross-realm-date-order.md +++ /dev/null @@ -1,220 +0,0 @@ -# Date order across JavaScript realms - -Scope update for PR #2116: this record preserves evidence from an earlier -commit. The current comparison oracle keeps ordinary native Date timestamps -across realms; its tagged, forged, borrowed, and subclass cases were removed -after the maintainer declined guarantees for caller-modified built-ins. See -the current [coverage map](../oracle-coverage.md) for the active witness. - -The comparison loss audit led to a same-realm NaN/invalid-Date transfer. A -fresh hostile review then challenged the whole published Date ordering law, -including native Dates created in another JavaScript realm. The live-query -guide's comparison section and changelog #1582 promise that invalid Dates act -like NaN in ordering and equality, without a realm exception. A valid Date -orders by timestamp. This is a product-law question beyond the five retired -same-realm examples, so it has its own witness and review. - -At unchanged production head `175adce0cb32e29e5838853c821ee5c56fe0089a`, -`ascComparator(foreign Jan 2, local Jan 1)` returned a negative sign rather -than positive. The new tagged Date oracle constructed local and `node:vm` -Dates from model timestamps and compared both operand orders, then checked -`normalizeValue` against the supplied timestamp. The fixed valid-Date case -failed at the direct comparator sign. The fixed-seed generated campaign failed -normalization at seed `20260911`, path `0:0:1` and directly reproduced that -checkpoint; the seedless campaign likewise failed and replayed. These are -product mismatches, not setup or cleanup failures. - -A separate public ordered-scan witness inserted an invalid Date before a valid -Date in a Collection with `autoIndex: 'off'`. The local invalid-Date control -returned IDs `[2,1]`; the foreign invalid-Date case returned `[1,2]`, failing -the same exact public-row comparison. The source Collection and live query -cleaned up in `finally`. - -The model uses `null` only as an invalid-Date *tag*: production receives a -native Date with a NaN timestamp. Valid tags range from -1e12 to +1e12 epoch -milliseconds. Both local and foreign constructors, both argument positions, -equal times, invalid values, and an own display tag are legal. One fixed -control changes the foreign Date prototype's display tag. Another checks that -numeric NaN and either local or foreign -invalid Date compare at exactly one order position. A forged -`[Symbol.toStringTag] = 'Date'` ordinary object is a negative control: neither -the classifier nor the normalizer accepts it as a Date, and the comparator -does not tie it to a Date. Proxies, subclasses, other value classes, cursor -pagination, and other realm mechanisms remain outside the bounded grammar. - -## Repair, GREEN evidence, and hostile review - -`dateTimestamp` now recognizes a native Date created in another realm and -reads its timestamp with `Date.prototype.getTime.call`. A fast shape gate -avoids a caught exception for ordinary objects and Temporal values. For an -object with a display tag, it checks the immediate prototype's own `getTime` -descriptor without reading the tag. The intrinsic remains the final brand -check, so a forged tag cannot establish Date identity, while an own or -prototype display tag cannot hide a normal Date. Local Dates use the same -intrinsic, so an own `getTime` method cannot silently replace the timestamp. -`ascComparator` classifies each operand once, -uses its timestamp for valid Date order, and treats an invalid Date like NaN. -`isUnorderable` and `normalizeValue` use the same timestamp extraction. - -After the repair, the comparison oracle, remaining comparison unit tests, -direct index oracle, and selected integration test passed 514 tests with no -type errors. The direct Date property ran with the fixed seed and the ordinary -seedless campaign. Its equal-timestamp branch forces ties independently of a -random collision. The new classifier assertions, both comparator directions, -and exact normalization diagnostics compare against supplied model tags. The -public scan checks complete ordered IDs after `preload()`; the local control -ensures this query path actually sorts before the foreign case is judged. - -The fresh hostile reviewer `/root/review_realm_date_oracle` followed the law -to the live-query guide and changelog #1582, inspected the unchanged-head RED -results, challenged model independence and a forged Date tag, and found no -blocker for this bounded repair. It identified the mixed NaN/invalid-Date -gap; the exact cross-family control now covers that comparison. It also found -that repeated Date classification could reread an unrelated object's tag; -the comparator now extracts each timestamp once per operand. A final -committed-head review follows below. - -## Oracle guide audit and limits - -The guide's law, authority, and scope requirements (ORC-001, -003, -011) -are stated in the oracle opening and beside this tagged Date model. Supplied -timestamps and invalidity tags predict order without production classification -(ORC-002, -009). Fixed foreign-valid, foreign-invalid, equal-time, mixed-NaN, -tagged-Date, and forged-tag controls complement generated realm/time/tag pairs -(ORC-004, -013). -Direct classifier, comparator, and normalization checks compare at function -return; a public ordered scan compares after preload (ORC-005). The original -production fails both a direct and public-row observation, supplying an actual -wrong implementation rather than only a hypothetical mutant (ORC-006). -The shared property harness runs fixed-seed and seedless campaigns and supports -seed/path replay (ORC-007). The scan uses `withOracleCleanup` to attempt both -Collection cleanups and preserve the primary mismatch (ORC-010). There is no stateful model or external -provider in this lane (ORC-008, -014). This record preserves the exact RED -head, reviewer verdict, and bounded GREEN result (ORC-012). - -`normalizeValue` is an internal index-key diagnostic, not public equality -evidence. The separate -[WHERE publication witness](cross-realm-date-where-publication.md) checks -public Date equality and explicit BasicIndex equality lookup. This record's -public scan uses `autoIndex: 'off'`; indexed order, range WHERE paths, -cursor/provider continuation, and other Date-consuming paths need separate -witnesses before they enter this closure claim. The subsystem has -realm-sensitive `instanceof Date` checks outside the three repaired utilities. -This record does not infer their correctness from the unindexed scan. - -## Exact committed-head recheck - -At published commit `9a145fba130f27c914a4a1fd4f44c5871a917483`, the fresh -hostile reviewer `/root/exact_review_comparison_round5` found a reachable -counterexample to the native-Date claim. A foreign native Date with an own -`Symbol.toStringTag = 'Other'` still passes `Date.prototype.getTime.call`, but -the production detector used `Object.prototype.toString` and treated it as an -ordinary object. The reviewer executed a public ordered query with a foreign -invalid Date carrying that tag. It returned IDs `[1,2]` instead of `[2,1]`. -The reviewer also observed that `isUnorderable` read an unrelated object's -display-tag getter. The existing oracle had plain foreign Dates and a forged -ordinary object, but no genuine Date whose tag differed. - -The expanded fixed controls and generated tag axis failed against that -published implementation. The focused RED run reported five failures: the -unrelated getter-count control, tagged-Date normalization, both generated -campaigns, and the public ordered scan. The fixed campaign captured seed -`20260911`, shrink path `0:0:0`, at the function-return checkpoint. The guarded -intrinsic detector passed the focused Date suite: seven tests and no type -errors. The public scan now includes the tagged foreign invalid Date. Fixed -controls also cover a genuine foreign Date with a misleading Temporal tag, a -tag getter that must not run, and an altered foreign Date prototype tag. - -The reviewer also found a harness risk by source inspection. A rejected -`query.cleanup()` in the original `finally` block could replace the ordered-ID -mismatch and skip `source.cleanup()`. The scan now uses the repository's -failure-preserving `withOracleCleanup` helper. A fresh review of the next -committed head remains pending. - -An intermediate intrinsic-only repair threw and caught for every ordinary -object and Temporal comparison. Hostile microbenchmarks found a substantial -Temporal cost. The final shape gate avoids that hot path without trusting tag -text as a Date brand. Dates with a replaced or stripped Date prototype, Date -subclasses, and proxies remain outside this claim. - -At clean commit `636e71e3d6084da6badeb0055fef5aae19ca4f58`, the fresh -hostile reviewer `/root/exact_review_comparison_round5` found no blocker for -the declared Date and retired-test scope. The focused suite passed 103 tests -with no type errors. The reviewer checked that genuine tagged foreign Dates -retain timestamp semantics, forged ordinary objects fail the native brand -check, unrelated display-tag getters remain unread, and the comparator avoids -the rejected exception-heavy path for Temporal values. The public scan cleanup -preserves a primary mismatch if cleanup also fails. This verdict does not -extend to replaced Date prototypes or `getTime` methods, proxies, subclasses, -indexed order, range queries, or cursor continuation. - -## Tagged Date subclass follow-up - -At `ae0b67fd2`, a new hostile review found a native `node:vm` Date subclass -with an own display tag that the immediate-prototype `getTime` descriptor gate -rejected. The native intrinsic returned timestamp 123, yet normalization left -the object unchanged and a public `eq` query missed its row. This is a separate -product bug under the guide's native-Date timestamp law, not a lost assertion -from the five retired same-realm blocks. The comparison owner now fixes local -and foreign subclasses with an inherited Date method, valid and invalid -timestamps, direct normalization and order, and an ordinary object borrowing -the foreign subclass prototype. On the original code the direct witness failed -at `normalizeValue(foreignSubclass)`, returning the object instead of 123. The -repair walks prototype descriptors without reading user getters, then calls -the native `Date.prototype.getTime` as the final brand check. Both receiving -oracles passed 127 focused tests after the repair. The separate WHERE owner -records the public RED and GREEN observations. Proxies, deeper subclass chains, -replaced methods or prototypes, indexed order, ranges, and cursor continuation -remain unproved. Fresh exact-commit review of this follow-up remains pending. - -At committed head `be5b7a3c4`, a fresh hostile reviewer found no loss from the -five retired comparison blocks and confirmed the tagged-subclass repair. It -found a narrower order-check gap: valid subclasses were compared with the -subclass only on the left, and an invalid subclass was compared with NaN but -not a finite Date. The receiving oracle now checks both operand directions -against equal and later finite Dates and places an invalid subclass after a -finite Date in both directions. These are direct order observations; they do -not extend the indexed or public range claim. - -The same review wave found a local ordinary object that borrowed a Date -subclass prototype. `instanceof Date` accepted it, but the native Date -intrinsic rejected its missing internal slot. At `be5b7a3c4`, direct -`normalizeValue` threw instead of retaining the object's reference identity. -The repair places the fast local branch inside the existing guarded intrinsic -call, so this object falls through as an ordinary object. The direct witness -failed before the repair and passed after it. The separate WHERE record owns -the public key-set witness. Proxies, arbitrary prototype changes, and indexed -behavior for the impostor remain outside this addition. - -At `3668d1b88`, the next hostile comparison review found two direct-order -sensitivity gaps. Valid local and foreign Date subclasses were compared with -an equal and a later Date, but not an earlier Date. The owner now requires -opposite finite signs in both directions against that earlier Date; an -injected subclass-first comparator fails the new assertion. A foreign -borrowed-prototype control now also requires `isUnorderable` to be false and -opposite nonzero signs, rather than accepting any result other than zero. -The focused comparison oracle passed 111 tests with no type, lint, formatting, -or diff-check errors. These additions do not establish indexed range behavior; -the separate [range-domain review](borrowed-date-indexed-range.md) owns that -public boundary. - -## Working-tree subclass and relational-order follow-up - -The hostile comparison review of frozen head -`210651d207eefe9bbc025c780f6259e470225c9e` found no loss from the five -retired same-realm blocks. It found that the direct invalid-Date control only -used a foreign subclass. The working-tree control now also creates a local -invalid subclass with an own display tag and requires native invalid-Date -normalization, unorderability, and both finite comparison directions. This -closes a fixture sensitivity gap; the review did not identify a new -production failure for that added local value. - -The separate [public range owner](borrowed-date-indexed-range.md) found a -production defect in relational scans: a genuine Date subclass can override -`valueOf` while its native timestamp remains unchanged. A `gt` or `lt` scan -could follow the override instead of the timestamp. The working-tree repair -uses the native Date timestamp for relational comparisons, including mixed -Date/number operands. The range owner records the original RED public -snapshots and the bounded scan/index GREEN evidence. Direct Date order and -public range order are separate observations; this record does not claim -arbitrary coercion methods, providers, or cursor continuation. diff --git a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md b/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md deleted file mode 100644 index 641e7c2252..0000000000 --- a/docs/contributing/oracle-reviews/cross-realm-date-where-publication.md +++ /dev/null @@ -1,195 +0,0 @@ -# Cross-realm Date equality at WHERE publication - -Scope update for PR #2116: this is the review record for an earlier, wider -Date oracle. The current WHERE owner checks ordinary native Dates across realms; -tagged, subclass, borrowed-prototype, and forged cases no longer state its -contract. The [coverage map](../oracle-coverage.md) describes the active cut. - -The [comparison transfer ledger](comparison-unit-retirement-round-5.md) -retired five same-realm NaN and invalid-Date examples. A hostile review of -their receiving law found a different defect: a native Date created in another -JavaScript realm was not normalized to its timestamp. The -[ordering record](cross-realm-date-order.md) preserves direct and public scan -RED witnesses. This record tests the separate equality and publication path. -It does not retire another unit test. - -The live-query guide now states explicitly that native valid Dates compare by -timestamp across realms and display tags. Changelog #1582 covers the separate -invalid-Date/NaN equality class. The WHERE publication owner already models -these classes for local values. Its focused matrix tags each source value and -literal as a local or `node:vm` Date with timestamp 1, timestamp 2, or an -invalid timestamp, or as number 1 or NaN. Some foreign Dates have an own -display tag. The model reduces tags to equality classes before production -values exist. It predicts complete, sorted public key sets, independent of -`normalizeValue`, the evaluator, or either index implementation. - -The driver starts a source Collection with auto-indexing off, then exercises -both no-index and explicit BasicIndex modes. At each mode and literal it -compares public live-query Collection keys, filtered source subscriber keys, -and direct Collection snapshot keys after preload. The indexed mode also -compares `BasicIndex.lookup('eq')` and a public snapshot with -`optimizedOnly: true`; that snapshot cannot silently use the scan fallback. -One sync transaction moves four source rows across equality classes, and the -same checks run after commit. The test does not prove that the public live -query planner selected the index. The source and live-query Collections and -subscription are released through the owner's cleanup helper. - -## RED and GREEN evidence - -An isolated archive of unchanged head -`175adce0cb32e29e5838853c821ee5c56fe0089a` ran the new test without -swapping production files in the shared worktree. It failed at the intended -unindexed public live-query comparison: `eq(v, local Date(1))` expected keys -`[foreign-one, local-one, number-one]` and returned -`[local-one, number-one]`. The cross-realm premise itself passed: the foreign -value failed local `instanceof Date` but was accepted by the native Date -intrinsic. This is a product result mismatch at a public checkpoint. - -With the comparison utility repair, the complete WHERE publication oracle -passed 18 tests with no type errors. The DB oracle suite passed 5,006 tests -across 73 files with no type errors. At that first repair, the scope was native -local and `node:vm` Dates, finite timestamps 1 and 2, invalid Dates and NaN, -one three-row sync transaction, and the named observations. Other realm hosts, -proxies, subclasses, spoofed tags, arbitrary timestamp histories, and -planner proof of live-query index selection remain outside this witness. - -The full DB suite initially reported two failures in the existing -`subset-dedupe.test.ts` Date transport test. It expected local `Date(2)` and -foreign `Date(2)` to match different `eq` and `in` requests. That assertion -encoded the old realm-local implementation, contrary to timestamp equality. -The test now expects both Dates to match for either literal realm and directly -checks that the request object is passed through unchanged. Its separate -Uint8Array realm observations remain untouched. The focused subset-dedupe and -WHERE files then passed 53 tests with no type errors; the full -`@tanstack/db` test command also passed. - -## Oracle guide audit and hostile review - -The opening prose and adjacent comments state the equality law, model, -bounded grammar, production driver, public observations, and two checkpoints -(ORC-001 through -005, -009). The original implementation fails a public -comparison at the intended cut; `optimizedOnly: true` is a positive indexed -snapshot reach witness, although its frozen-head RED was not separately run -(ORC-006). This is a fixed matrix rather than a generated campaign; all -declared cells run on every execution, so seed/path replay is inapplicable -(ORC-004, -007). The model's source rows change only after a committed -transaction, and all consumers compare with that same authored state -(ORC-008). The cleanup helper preserves the primary failure while releasing -consumers (ORC-010). The comparison transfer and ordering records identify -the original test gap and rival identity-versus-timestamp rule (ORC-011 -through -013). The sync source is controlled; the witness makes no provider -claim (ORC-014). - -The fresh hostile reviewer `/root/review_realm_date_where` inspected the -model, the original unit ledger, current production, public authority, and -both lanes. It found one reach gap: direct index lookup did not prove a public -read used the index. The added optimized-only public snapshot closes that -gap. Its final verdict found no blocker for the bounded matrix. The reviewer -also traced the full-suite failure above to the old test's realm-specific -assumption, not to a request-identity contract. Final key sets do not prove -subscriber event multiplicity, payloads, or intermediate publication. - -## Exact committed-head recheck - -At published commit `9a145fba130f27c914a4a1fd4f44c5871a917483`, the fresh -hostile reviewer `/root/exact_review_where_realm` found no blocker in the -bounded matrix for unmodified native Dates. The focused WHERE and subset suites -passed 53 tests with no type errors. Source inspection confirmed that the -`optimizedOnly: true` read cannot fall back to a scan. - -The same reviewer independently found the foreign native Date with an own -`Symbol.toStringTag`. The published comparison utility classified it by object -identity, so the broader native-Date claim was open. The matrix now includes -tagged foreign source rows, tagged literals, and a tagged post-commit update. -The reviewer ran this expanded matrix in an isolated archive of `9a145fba1`. -At its first unindexed public live-query checkpoint, a local `Date(1)` literal -expected `foreign-tagged-one` in the key set, but production omitted it. The -complete expected set was `[foreign-one, foreign-tagged-one, local-one, -number-one]`; production returned `[foreign-one, local-one, number-one]`. -Vitest reported an assertion failure with no type errors. The unmodified -matrix passed against the same archived code. After the guarded intrinsic -repair, the expanded matrix passed in both indexed and unindexed modes. A -fresh committed-head review of the repaired matrix follows. - -At clean commit `636e71e3d6084da6badeb0055fef5aae19ca4f58`, the fresh -hostile reviewer `/root/exact_review_where_realm` found no blocker for this -bounded WHERE Date equality matrix. The focused WHERE and subset suites passed -53 tests with no type errors. The reviewer reconstructed the model classes, -checked tagged source rows and literals at initial and post-commit cuts, and -traced the live keys, subscriber reconstruction, and public snapshots in both -modes. The explicit BasicIndex lookup and `optimizedOnly: true` snapshot prove -an indexed public read, without claiming the live-query planner selected it. -The archived `9a145fba1` run failed at the intended unindexed public equality -comparison by omitting `foreign-tagged-one`; its unmodified controls passed. -The verdict does not cover other hosts, public ranges, indexed order, -arbitrary histories, or Date objects with replaced prototypes or methods. - -## Tagged Date subclass follow-up - -The hostile review at `ae0b67fd2` found that a native foreign Date subclass -with an own display tag missed `eq(Date(1))` in a public live query. The matrix -now assigns fixed local and foreign subclass rows and literals to timestamp -classes independently of production classification. It also includes an -invalid foreign subclass and an ordinary object borrowing a foreign subclass -prototype as a negative control. Initial and post-commit observations compare -the complete public live-query rows, filtered subscriber keys, direct snapshot -keys, explicit BasicIndex lookup, and an optimized-only snapshot. The original -implementation failed first at the unindexed live-query key set by omitting -`foreign-subclass-one`. The descriptor-chain repair passed both receiving -oracles, 127 tests, with no type errors. Live-query planner index use, generated -subclass histories, deeper subclass chains, other hosts, public ranges, and -prototype/method replacement remain outside this witness. Fresh exact-commit -review is pending. - -## Local borrowed-prototype counterexample at `be5b7a3c4` - -A fresh hostile reviewer found that the matrix's foreign borrowed-prototype -object never reached the local `instanceof Date` fast path. An ordinary object -made with `Object.create(LocalDateSubclass.prototype)` has no native Date -slot, although local `instanceof Date` returns true. In an unindexed public -query, `not(eq(v, Date(1)))` omitted its only row instead of publishing that -row. The new focused witness establishes the missing slot with the native -intrinsic, then checks both Date inequality and same-object identity after -preload. It failed at the public key set on the reviewed commit and passed -after the guarded Date-classifier repair. The matrix still preserves the -foreign borrowed-prototype control. This addition covers one unindexed row; -indexed publication for the local impostor and other host realms remain open. - -## Invalid-Date exclusion at the public WHERE cut - -The hostile review of frozen head -`210651d207eefe9bbc025c780f6259e470225c9e` found one remaining -source-inspection gap in the local borrowed-prototype control. The existing -unindexed `not(eq(v, Date(1)))` and same-object `eq` checks could pass if -the borrowed object also joined the invalid-Date/NaN equality class. The -working-tree witness now preloads a public `eq(row.v, new Date(NaN))` query -and requires an empty key set for that same borrowed row. This added cut -did not demonstrate a new production failure. The coverage map now states -the bounded matrix's four post-commit updated rows accurately. - -The witness establishes this single local borrowed object at an unindexed -public checkpoint. It does not cover indexed publication for that object, -arbitrary prototype chains, or other host realms. A committed-head hostile -review of the working-tree addition remains pending. - -The follow-up review of `d974d4a4f` also found an overstatement in the -WHERE publication oracle's opening prose. Its generated histories compare -direct snapshots when subscribers attach, then reconstruct subscriber and -live-query key sets after each sync commit. The fixed Date matrix compares -direct snapshots both before and after its commit. The prose now names those -separate checkpoints. No behavior or product expectation changed in this -clarification. Final exact-commit hostile review remains pending. - -## Zero timestamp false-green control at `0c236f448` - -A read-only hostile review found that the local borrowed-prototype object's -existing checks could still pass if it erroneously joined the valid zero -Date/number equality class. The owner now asks two additional public live -queries for `eq(row.v, new Date(0))` and `eq(row.v, 0)` after preload. Both -must return no rows for that ordinary object. These controls use public -publication rather than a direct comparison helper and distinguish object -identity from timestamp zero and numeric zero. The focused 81-test run -including them passed with no type errors. The later working-tree focused and -neighboring run passed 985 tests across 11 suites. This does not establish -indexed publication for the forged object, other borrowed prototypes, or -other host realms. Exact committed-head review of these controls is pending. diff --git a/docs/contributing/oracle-reviews/custom-comparator-signed-zero.md b/docs/contributing/oracle-reviews/custom-comparator-signed-zero.md deleted file mode 100644 index 752666a915..0000000000 --- a/docs/contributing/oracle-reviews/custom-comparator-signed-zero.md +++ /dev/null @@ -1,68 +0,0 @@ -# Signed zero under a custom index comparator - -The direct IndexInterface contract permits a numeric custom comparator. -JavaScript `Map` equality treats `-0` and `+0` as one exact lookup class, while -a callback can distinguish their order. For this PR, the maintainer chose -per-row callback order inside a query-equality class. The index refinement -oracle supplies independent rank labels for `-0`, `+0`, and `1`, then checks -exact lookup, full ordered take, a bounded range, and a cursor after `add` or -`build`. It removes the negative row and checks the surviving class and order. - -The unmodified BasicIndex ordered-bucket `Map` used SameValueZero keys for the -callback positions. Both new add/build histories returned `positive` twice -and omitted `negative` at `takeFromStart`; BTreeIndex passed both controls. -BasicIndex now encodes negative zero separately for its private ordered-bucket -key, while exact membership keeps the ordinary `Map` key. All four new -histories and the full 416-test index refinement owner pass with no type errors. -The independent rank model is supplied by the fixture, not copied from either -index. This fixed witness covers only the stated signed-zero callback and -direct operations; arbitrary callbacks, public order selection, and provider -paths remain separate. - -## Exact-commit hostile review and follow-up - -At `c4c07da42`, a fresh reviewer found three legal histories the first -witness missed. Both index types skipped ordered maintenance for `-0 → +0` -because SameValueZero treats the positions as equal, even though the callback -does not. BTreeIndex could then keep a stale ordered bucket after update, -remove, and re-add. BTree range traversal also used JavaScript `===` to -collapse distinct signed-zero bounds at leaf and internal nodes; a 33-row -fixture reached the split-tree case. - -The extended direct oracle uses an independent source Map and a supplied -numeric rank to predict membership and order after sign-changing updates, -removal, and re-addition. It checks exact SameValueZero lookup, ordered takes, -cursors, and inclusive, exclusive, reversed-bound, and point ranges on both index -types; the split case crosses BTree children. Four focused cases failed on -the old production at the intended ordered or range comparison. Changing -only the BTree point-range shortcuts left the two update cases red. Ordered -maintenance now uses `Object.is` in both indexes; the BTree shortcuts use it -to recognize only identical bounds. The index/BTree owner passed 436 tests -with no type errors after repair. The five lint warnings were preexisting. - -## Oracle guide audit - -| Requirement | Evidence and limit | -| --- | --- | -| ORC-001 | The direct index contract accepts a numeric callback. The maintainer selected per-row callback order within a SameValueZero exact class; the architecture's comparator-group law requires every row to survive ordered traversal. | -| ORC-002 | A source Map records owners and raw values. Fixture ranks predict order independently of index buckets and traversal code. The callback itself is the input contract, not an imported production comparator. | -| ORC-003 | Prose beside the source model, actions, driver, and assertions states the law, legal history, direct path, observations, and after-action comparison. | -| ORC-004 | Not triggered: the sign-changing and split-tree histories are fixed witnesses, not a generated grammar claim. | -| ORC-005 | Both real index types return exact sets, ordered arrays, cursor arrays, and range sets after each action. Those recordings expose missing, duplicate, and misplaced rows. | -| ORC-006 | Four new cases failed on the pre-repair code at intended take or range comparisons; the BTree-only fix left the update cases red. | -| ORC-007 | Not triggered by these fixed witnesses. Other generated index properties keep their own campaigns. | -| ORC-008 | The Map retains each owner's raw sign because a later update or removal can distinguish `-0` from `+0`; merging those states would lose the next ordered observation. | -| ORC-009 | Model ownership maps to indexed row keys. Model rank is a fixture abstraction of callback position, separate from exact SameValueZero membership. | -| ORC-010 | Direct indexes own no external resources or cleanup action; an assertion is the failure checkpoint. | -| ORC-011 | The source Map and fresh rank sort differ from the incremental indexes. No plausible shared semantic fault within this bounded callback remains identified; arbitrary callback laws remain open. | -| ORC-012 | This record names the counterexamples, RED/GREEN sequence, scoped claim, and pending final review. | -| ORC-013 | `-0` and `+0` share exact lookup but have distinct callback positions; inclusive versus exclusive and reversed mixed-sign bounds reject a SameValueZero range rule. The signed-zero checker does not directly call `rangeQueryReversed`. | -| ORC-014 | No provider or host handoff is claimed. | - -At `ae0b67fd2`, a fresh hostile reviewer found no blocker in the declared -direct-index scope. The 420-test focused oracle and targeted generated replays -passed. Independent bounded histories matched BasicIndex and BTreeIndex for -signed-zero exact lookup, order, cursors, and forward range queries. The -reviewer also noted that part of the broader numeric property inspects private -layout, while separate public method checks protect the transferred behavior. -Arbitrary callbacks and public planner selection remain outside this verdict. diff --git a/docs/contributing/oracle-reviews/custom-index-order-selection.md b/docs/contributing/oracle-reviews/custom-index-order-selection.md deleted file mode 100644 index 9ce724a348..0000000000 --- a/docs/contributing/oracle-reviews/custom-index-order-selection.md +++ /dev/null @@ -1,483 +0,0 @@ -# Custom comparator index selection - -Scope update for PR #2116: later sections record an unsigned default-byte -order and caller-modified byte controls from an earlier commit. Those proposed -guarantees were declined. The current oracle keeps byte-content equality, -scan/index order agreement, and the maintainer-approved per-row custom -comparator rule. The [coverage map](../oracle-coverage.md) states the active cut. - -An index with a user-supplied `compareFn` cannot advertise the ordinary -Collection `orderBy` comparator merely through `compareOptions`. The -live-query architecture's Identity contract requires advertised comparison -options to describe the comparator the index executes. Exact equality lookup -remains a separate index operation. - -The new oracle supplies three numeric rows in source order 20, 30, 10. Its -independent model sorts their supplied ranks numerically, so an ascending -`limit: 1` read must return rank 10. It compares unindexed and indexed public -Collection snapshots after readiness for both BasicIndex and BTreeIndex. A -custom callback orders ranks descending, and an ordinary index is a control. -The driver also checks descending reads, `optimizedOnly` refusal of a custom -index for ordinary order, and optimized exact `eq` on that same custom index. - -At frozen head `1265abeb5`, both custom index types claimed default ascending -eligibility and returned rank 30. The new oracle failed twice at the public -first-key comparison, with no type errors. The repair makes public ordered -selection respect the existing `supportsRangeOptimization` capability, which -is false for custom comparators. This leaves exact lookup available and does -not change the custom callback's direct index behavior. The focused 460-test -index/query campaign passed with no type errors after repair. - -The oracle covers a ready in-memory Collection, one numeric field, one order -term, limit one, and two built-in index types. It does not establish arbitrary -custom comparator laws, mutation histories, live-query publication, or -provider ordering. - -## Oracle guide audit - -| Requirement | Evidence and limit | -| --- | --- | -| ORC-001 | The architecture's index metadata law and public ordered-read contract authorize the numeric first-key observation. | -| ORC-002 | Supplied ranks determine expected order without reading the index comparator or its metadata. | -| ORC-003 | The oracle explains law, fixture, production paths, snapshot, and comparison beside its code. | -| ORC-004 | Fixed three-row source order distinguishes insertion order from ascending and descending rank; no generated-history claim is made. | -| ORC-005 | Both public Collection snapshot paths are observed after readiness; optimized-only reads check planner admission. | -| ORC-006 | The original wrong planner fails at both public first-key assertions; an ordinary index and exact lookup are adjacent controls. | -| ORC-007 | Not triggered: this owner is a bounded fixed witness. | -| ORC-008 | Not triggered: the model has no mutable state. | -| ORC-009 | Ranks are model facts; `supportsRangeOptimization` is production capability observed separately from expected row order. | -| ORC-010 | `withOracleCleanup` releases each Collection while preserving the primary assertion. | -| ORC-011 | The unindexed scan and numeric model check the indexed result through different formulations; both production paths still share the Collection API. | -| ORC-012 | This record names RED/GREEN evidence, owner, limits, and pending exact-commit review. | -| ORC-013 | A reverse-number callback makes the wrong index choice visibly return rank 30 instead of rank 10. | -| ORC-014 | Controlled in-memory sync ends at the ready snapshot; no provider handoff is claimed. | - -## Exact-commit hostile review and follow-up - -At `c4c07da42`, a fresh reviewer reproduced a planner failure with two -indexes on the same field: when the custom index was created first and a -compatible ordinary index second, `optimizedOnly` ordered reads still -returned no result. The original oracle put the indexes in separate -Collections, so first-match selection was never challenged. An eager -Collection also treated the custom index as sufficient and created no -compatible ordinary index. Numeric range predicates had the same first-match -problem. The reviewer reproduced both index types and both read directions. - -The extended fixture now puts both indexes in one Collection and checks -ascending and descending public first rows. It also starts with a custom -index under eager mode, requires one compatible index to be created, and -checks a repeated read does not create another. Optimized exact `eq` still -uses the first custom index, verified through its lookup spy. The fixture -also checks optimized single and compound numeric ranges and an eager range -subscriber. Four ordered-selection cases failed before that repair at the -missing first row. Four more cases failed before the range repair: both index -types returned no optimized `gt` result, and both eager subscriptions failed -to create a compatible index. The compound assertion followed the single -range assertion in one test, so no separate pre-fix compound failure was -observed. Selection and eager admission now skip a custom index only when -ordered traversal is required. The new oracle passed 8 cases, and the -six-file related campaign passed 145 tests with no type errors. Lint and -diff checks passed. - -Fresh review of the final committed head remains pending. - -At `5df06de2e`, another reviewer found that eager admission recognized -`gt(rank, 10)` but not its equivalent value-left form `lt(10, rank)` after a -custom index. Initial subscription rows were correct, but the value-left -history failed its optimized-only snapshot because no compatible index was -created. The author added single and paired value-left RED witnesses for -BasicIndex and BTreeIndex before extending eager expression extraction to -identify the field on either side. The repaired tests compare initial rows, -optimized rows, and index count against independent numeric row filters. A -custom-only optimized range refusal is an adjacent control. The focused -oracle passed eight tests and the neighboring index campaign passed 547. -This establishes neither arbitrary value domains nor mutation histories. -The live-query compiler has a separate index-admission path; simple public -probes were correct, but the comparable coexisting-index schedule remains -unproved there. - -## Live-query receiving path after `ae0b67fd2` - -A fresh hostile review found a public counterexample to the compiler's ordered -index admission. An on-demand source had rank-20 and rank-40 rows cached. -It held four authoritative rows at ranks 10, 20, 30, and 40. With a reverse -custom index declared before an ordinary index, an ascending live query -preloaded its first row correctly. Moving to offset one and limit two then -published ranks 30 and 40, instead of 20 and 30. The provider fulfilled its -exact ordered request for ranks 10, 20, and 30. It did not rewrite cached -rank 20, which the wrong local index traversal had failed to send into the -graph. Ordinary-first, ordinary-only, and no-index controls returned the -correct rows. The reviewer reproduced both index types and found 16 public -mismatches in a separate 64-scenario cache/window sweep. - -The new [window oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/custom-index-window-selection-oracle.test.ts) -assigns numeric ranks in its source model and sorts and slices those four -facts independently of index selection. Its controlled adapter evaluates -each request's predicate and cursor against the source facts, records the -selected provider rows, and writes only uncached rows. It checks the live -query's rows after preload and after `setWindow()` settles, plus the sole -publication batch. A reply `[1,2,3]` with writes `[1,3]` proves that row 2 -was already cached and still belonged to the requested result. The fixed -history crosses custom-first, ordinary-first, no-index, and eager-custom-only -layouts with BasicIndex and BTreeIndex. On the original compiler, four cases -failed at the moved public window: `[3,4]` instead of `[2,3]`. Controls -passed. All eight cases passed after the repair; 677 related tests and the -explicit test TypeScript check passed. - -The compiler now requires an ordinary ordered-traversal index when it creates -or selects the first order term's source index. Eager creation lets BaseIndex -derive the comparator from `compareOptions`; passing an explicit `compareFn` -would mark that index custom and make it ineligible. This changes only source -index admission. Direct custom callbacks still order each row according to -the maintainer's decision. The production edit removes two net lines. - -The oracle states the ordered window law and its source limits beside its -model, request grammar, driver, and two public cuts (ORC-001–005). The -original compiler fails at the promised post-window comparison, while -ordinary-first and no-index controls reach the same public boundary -(ORC-006, -011, -013). This is a fixed finite matrix, so generated replay -does not apply (ORC-004, -007). The provider is controlled and awaits applied -receipts; the claim does not extend to remote hosts (ORC-014). Ties, nullish -or nonnumeric orders, mutations, replay, multiple order terms, arbitrary -callbacks, and other window schedules remain open. Fresh hostile review of -the final committed oracle is pending. - -At committed head `be5b7a3c4`, a fresh hostile reviewer found no additional -numeric window mismatch in 128 controlled cache/layout scenarios. A read-only -wrong-index admission fault changed the settled public rows, confirming the -comparison rejects that error. The reviewer found that readiness and live -query construction happened before cleanup protection; the driver now enters -`withOracleCleanup` before those steps and releases any handles created before -a setup failure. The `eager-custom-only` layout means eager *auto-indexing on -an on-demand source*. The eight-case oracle proves the public result and the -provider's cached-row interaction. It does not measure ordinary-index use or -an eager source path. The architecture's index work law needs a separate -index-use witness before this owner claims that property. - -## Frozen-head review and mixed Date/number order - -Review of `210651d207eefe9bbc025c780f6259e470225c9e` found three -source-inspection gaps in the ordered-selection oracle. Correct public rows -alone did not prove the later ordinary index supplied the ordered and range -reads. Inclusive value-left `gte`/`lte` boundaries were absent. Some -Collection construction still preceded cleanup protection. The working-tree -oracle now spies on the ordinary index's take, lookup, and range methods; -checks lower, upper, and paired inclusive value-left predicates against -independent row filters; and registers each Collection inside protected -construction. These gaps did not demonstrate an additional production bug. - -An adjacent public witness did expose a production order mismatch for one -Date/number fixture. Before the repair, the unindexed ordered snapshot was -`[equal-number, number, date]`, while BasicIndex and BTreeIndex optimized -snapshots returned `[date, equal-number, number]`. The two new index-type -cases failed at that public full-order comparison. They now compare both -full order and the first-row window before and after index installation, -check the Date/timestamp exact-equality class, and spy on the optimized -index take. The repair retains raw native Dates as index order values while -preserving their normalized exact lookup key. The focused index campaign -passed 497 tests, and the full `@tanstack/db` run passed 12,345 tests in 281 -files, with no reported type errors. These are working-tree runs, not a -committed-head review. The witness requires scan/index agreement for this -fixture; it does not define a universal order between value families. - -## Follow-up to committed-head review - -Review of `d974d4a4f` found that the earlier paired inclusive range -(`20 <= rank <= 30` over ranks 10, 20, and 30) did not make either bound -exclude a row. The paired fixture now uses an upper bound of 25, so its -independent model expects only rank 20. The ordinary index has separate -positive spies for ascending take, descending reverse take, numeric lookup, -and range traversal; the full mixed Date/number optimized read also proves -its take was called. These additions make a wrong selected index or a skipped -bound observable at the public snapshot and traversal checkpoints. The -focused owner passed. They do not prove arbitrary numeric histories, -providers, or the live-query compiler's index-use premise. A fresh hostile -review of the final committed follow-up is pending. - -The read-only hostile review of `0c236f448` ran all 10 owner tests and found -no blocker in the fixed selection domain. It corrected one claim about the -paired inclusive bound: with ranks 10, 20, and 30 and predicate -`20 <= rank <= 25`, omitting either bound changes the result. Moving either -bound slightly does not necessarily change it, so the coverage map now says -"omitting" rather than "misplacing." This is a documentation correction, -not a new production bug or an added witness. Final review of the next -committed head remains pending. - -## Finite provider order handoff - -Review of `d917ba4c9` found that the window oracle's controlled provider read -its source rows from the rank-sorted expected result. Removing `orderBy` from -every request therefore left all eight cases green: the fixture still served -the right prefix without an ordering hint. The source facts now have physical -rank order 30, 10, 40, 20. The independent public model still sorts by numeric -rank. The adapter sorts its filtered source rows only when the request carries -`orderBy`, and it checks that every finite request asks for one ascending -`rank` term. This keeps provider work at the request boundary; the public -preload and moved-window comparisons keep their original law. - -All eight fixed cases passed after this oracle repair. A temporary test-only -mutant stripped `orderBy` as the adapter received it. All eight cases failed -at the finite-request assertion. With that assertion temporarily disabled, -the BasicIndex/custom-first case reached the initial public snapshot and -failed with `[2]` instead of `[1]`. Both mutations were removed. These are -assertion failures at the intended request and public checkpoints, not -setup failures. The controlled provider still does not establish how a remote -provider interprets an ordered request. - -## Public comparator ties at `d917ba4c9` - -The public `currentStateAsChanges` path returned `[b]` for descending -`orderBy rank, limit: 1` over tied source keys, while an unindexed scan -returned `[a]`. BasicIndex and BTreeIndex both exposed the -failure through their reverse views. The low-level reverse index may reverse -tie order; the public Collection rule from changelog #958 requires source keys -ascending within comparator-equal positions. New numeric, mixed-key, -custom-collation, and nullish fixtures compare independently sorted source -facts with unindexed and optimized indexed snapshots. Finite prefixes cut -through tied groups, and index-take spies establish the optimized path. - -The same defect reached direct on-demand `requestLimitedSnapshot`: a source -with rank-1 keys `c`, `b`, `a` published `[c]` for a descending limit-one -request. That was a separate RED public receiving witness. A neighboring -history first publishes a unique rank-2 row, then continues at a cursor into -the rank-1 tie. A custom string-length comparator checks that a cursor at -`xx` still includes the unsent `yy` peer in the same comparator position. -The repair shares indexed prefix selection across ready snapshots and direct -subscriptions, completes only a finite prefix's boundary group, and sorts -source keys within each comparator group. It does not change the low-level -ReverseIndex contract. The repaired focused ordering, lifecycle, pagination, -and work suites passed 748 tests, plus TypeScript, lint, formatting, and diff -checks. The evidence covers these two public snapshot paths and fixed -histories; provider ordering, source mutations, and full live-query -publication require separate receiving witnesses. Fresh hostile review of -the committed repair remains pending. - -## Exact-commit hostile finding at `5c97f18dc` - -The fresh reviewer confirmed that ordered indexes could compare encoded -equality keys instead of source values. A custom string-length callback saw -an escaped representation of `"\u0000tanstack-db:x"`, so an indexed ascending -limit-one read selected a 17-character string before that 14-character -string. The unindexed public read selected the 14-character row. Both -BasicIndex and BTreeIndex failed. Direct take, range, and cursor reads also -misordered or omitted it, and a direct subscription continued to the wrong -row. The previous tie fixtures used ordinary strings, so they never crossed -the encoding boundary. - -The same encoded-order error moved native byte views into the string order -class. A bounded 40-case hostile sweep crossed five value groups, both index -types, both null placements, and both order directions. String/byte, -number/byte, and equal-content distinct byte-view groups disagreed with the -public scan in their corresponding eight-cell lanes. String/Date and -nullish/byte controls agreed. Exact byte lookups stayed correct. This was an -order-operand bug, not an equality-key bug. - -The repaired oracle supplies raw string lengths for the custom callback, -checks direct ordered take/range/cursor and one subscription continuation, -then compares ready public scan and indexed snapshots. A mixed-value fixture -checks primitive-before-byte order and preserves the scan order of two -distinct equal-content byte views after indexing. Exact lookup still groups -those byte views. A literal string that resembles the old BTree undefined -marker stays distinct from undefined in both order and exact lookup. The -production repair keeps raw order operands and normalized exact keys, with a -unique undefined traversal marker for BTree. Six new public assertions and -four direct comparisons per index type were RED on `5c97f18dc`; the focused -649-test campaign passed afterward. A normalized-byte mutant fails both -public indexed comparisons, and an old string marker fails the BTree ordered -read. Both faults were restored. - -The earlier prefix helper's take spies establish that an index supplied the -keys. They do not prove a strict work bound: ReverseIndex can enumerate its -whole nullish group inside one finite take. The helper comment now limits -its claim to the requested prefix and boundary group at its own boundary. -The fixed rows and controlled subscription do not establish arbitrary value -mixtures, source mutations, remote provider order, or all live-query -publication schedules. A fresh hostile review of the next committed code -remains necessary. - -## Exact-commit hostile finding at `d93ae1dd6` - -Two independent read-only reviewers found a legal source-update history that -the ready-state byte witness missed. The public comparator orders distinct -native byte views by runtime reference ID, while change-event equality groups -views with equal contents. Replacing one row's byte view with a fresh -equal-content view therefore changed the public scan order but published no -update to its index. Both built-in index types kept the old order. A later -delete could remove the exact equality owner while leaving its old ordered -entry; reinsertion then produced a duplicate public key. The two-row -replacement and a four-row update/delete/reinsert history failed at exact -public indexed arrays and direct index traversals. The latter retained a -deleted key even while `keyCount` and exact lookup appeared correct. - -A third test-owned callback ranks three equal-content views at distinct -positions. After Collection replaces a row with another view, the direct -custom index must use the new callback position while exact lookup retains -content equality. This witness failed for BasicIndex and BTreeIndex at take, -range, and cursor comparisons. Direct index update histories had passed -because their callers supplied both raw values; Collection suppressed the -structurally equal source update before it reached the index. The repair must -refresh indexed raw positions from the visible before/after rows even when -change-event equality suppresses an ordinary change message. - -These witnesses establish indexed-versus-scan parity under the current default -comparator. They do not, by themselves, make runtime-reference ordering of -native bytes the permanent product contract. The default-byte order and its -live-query publication consequence need an explicit contract choice before -claiming this bug class closed. - -## Proposed native-byte order and hostile review - -This PR proposes unsigned lexicographic content order for genuine native -Uint8Arrays and Buffers in default ordered reads. The old comparator put fresh -equal-content views at distinct runtime-reference positions: replacing one -could move the public scan without a source event or index refresh. The -proposed comparator puts equal contents at one position and uses the public -source key to break row ties, so an equal-content replacement preserves the -ordinary ordered result and remains silent. A changed first byte orders by -unsigned value; a shorter equal prefix comes first. Relational `WHERE` -comparison remains a separate contract and was not changed to follow this -order. A user custom index comparator still reads each current raw view, so -Collection must refresh its index even when ordinary change-event equality -suppresses a source message. These rival predictions distinguish the design -choice from the original production behavior. - -The first content-order tests failed on `d93ae1dd6` at direct comparator, -unindexed scan, and indexed snapshot checkpoints: 11 tests failed and four -passed in the selected run after separating the `WHERE` assertion. With the -new default comparator and silent index refresh, the comparison and custom -index owners passed 162 tests. A fresh read-only hostile reviewer then found -that the proposed unsigned law could pass with signed-byte ordering; the -revised pair crosses 127/128. It also found missing cross-realm and mixed -ordinary-object distinctions. The revised direct owner uses a foreign native -view, an own `length` shadow, and a byte/object/byte transitivity triple. The -public fixture adds the 127/128, foreign-view, and length-shadow cases for -both index types. These owners now pass 172 tests. The mixed-kind check -protects total order without fixing whether byte values precede ordinary -objects. Full live-query `ORDER BY` materialization and publication after byte -updates remain outside these ready-snapshot checks. - -A second fresh hostile reviewer found two reference-model errors. Byte ties -used host locale collation rather than the public source key's code-unit order; -the equal-content replacement now uses `A` and `a` so that error would fail. -The byte model also called an instance's overrideable `.values()` method; it -now invokes the native typed-array iterator on each view. It confirmed the -index refresh reaches both built-in index types after a sync replacement and -found no further mismatch within those bounded histories. It did not find an -optimistic same-content replacement reaching Collection state: draft tracking -discards that assignment before index maintenance. Arbitrary optimistic -histories remain unclaimed. - -The same review found a separate public publication defect. A native `[1]` -view and a native `[2]` view that each shadow own `length` as zero were treated -as equal by `deepEquals`, although Collection state and exact index lookup -changed. Both index types emitted no source update; an unshadowed control did. -The direct deep-equality property and Collection change-event witness were RED -on the uncorrected implementation: two generated equality lanes and both -index-type publication cases failed at their intended comparisons. Reading -the native typed-array element count repaired this boundary; the four focused -owners then passed 693 tests with no type errors. A class transition between -Uint8Array and Uint16Array with equal elements presents a neighboring -deep-equality/query-equality tension. Existing general equality deliberately -ignores typed-array class, so that contract needs separate review before -changing publication for that transition. - -These results preceded the exact-commit review recorded below. - -## Exact-commit window review of `5c97f18dc` - -A separate fresh reviewer found no blocker in the finite provider order-hint -repair. Physical provider order differs from rank order. Missing `orderBy` -fails the finite-request guard; without that guard, it fails the initial -public snapshot. All eight fixed window cases passed with no type errors. -This controlled provider does not prove real-provider collation, index-use -efficiency, ties, mutation, or replay. - -## Exact-commit hostile review of `ea569cc2a` - -Three read-only agents reviewed the committed byte/order and Collection -publication changes against the source and retirement ledgers. One found a -foreign-realm DataView regression: `ArrayBuffer.isView` accepted it, a local -`instanceof DataView` failed to exclude it, and the new native typed-array -length getter threw. A public sync commit had already changed the row, but -left BasicIndex and BTreeIndex on the old value and emitted no update. Direct -foreign-DataView equality and public typed-array-to-DataView then -DataView-to-DataView histories failed before the repair at the getter. The -native typed-array brand check now excludes DataViews across realms. Both -commits settle, with exact index lookup and subscriber batches matching the -public row. Distinct DataViews from one foreign realm have equal prototypes -but remain unequal by the established opaque-instance law. - -The publication reviewer also found an enforcement gap: flattening messages -would miss an extra empty callback after an equal-byte replacement. The -native-byte sync witness now records whole callback batches and rejects that -callback. Current production already suppresses it. This is a test repair, -not an observed product failure. - -The order reviewer put a first-byte/length comparator mutant in an isolated -test copy. All 172 prior comparison and custom-index tests passed, although -the mutant ignored every later byte. New `[1, 2]` versus `[1, 3]` direct -signs and a ready public `a:[1,3], z:[1,2]` order fail that mutant at both -unindexed and indexed comparisons for BasicIndex and BTreeIndex. The same -witnesses pass against production. A second mutant read an own `byteLength` -shadow instead of the native element count and passed those revised tests. -Direct and public byte-length-shadow fixtures now distinguish it. These -fixtures close the named finite sensitivity gaps; they do not prove arbitrary -byte lengths or live-query publication. - -Content equality and source-key tie behavior predate this PR; unsigned native -byte order is a proposed policy. It makes equal-content replacement stable -under default ordering, but byte relational `WHERE` comparisons remain a -separate rule. The current direct and ready-snapshot checks do not establish -how all byte range predicates, providers, or live queries should interact with -that ordering. - -The index reviewer found no additional mismatch in bounded 250-action sweeps -per built-in index for native-byte order, key conservation, and custom -callback positions. These sweeps are diagnostic evidence, not a replacement -for the committed oracle. - -## Exact-commit hostile review of `23a478e78` - -Two fresh read-only reviewers challenged the repaired byte and DataView -oracles. Neither found a further production defect in the paths exercised. -The byte reviewer showed that a comparator reading only the first two bytes -passed the prior direct and ready-Collection owners. A third-byte fixture -failed that mutant at the direct comparator, unindexed scan, and both built-in -index paths; it passed unchanged production. The direct owner now generates a -common prefix of 2–24 bytes before a differing byte, and the ready public -owner includes a third-byte witness with source keys opposed to content order. -The fixed and random direct runs vary the first-difference position; the -public witness checks one later position. - -The DataView reviewer moved index maintenance after subscriber notification -in an isolated copy. The prior two-step Collection oracle stayed green even -though the callback could read a new public row with an old index. Recording -the row and exact index lookup inside each callback failed that mutant and -passed unchanged production for both index types. The existing after-commit -checks remain. This strengthens the atomic-publication observation for one -key and two sync commits; it does not establish every callback schedule. - -The byte reviewer confirmed that relational `WHERE` still uses JavaScript -coercion for byte comparisons, while the proposed default ordered-read rule -uses unsigned byte content. The planner declined byte range optimization in -the reviewed public path, so indexed and unindexed results agreed there. The -two comparison rules remain a product-design question beyond this PR's byte -equality repair. - -## Pushed-head byte and DataView recheck at `b51f0c79d` - -A fresh hostile reviewer checked the byte comparison, direct index, and -Collection publication paths against the retirement ledgers at the clean -pushed head. Ten receiving and retained files passed 1,246 tests. An isolated -comparator that read only the first two bytes failed the generated direct -comparison and both ready indexed order checks. No production defect appeared -in those paths. - -The reviewer found another callback-time enforcement gap. An isolated index -mutant kept both the old and new exact memberships during notification, then -removed the old membership before `commit()` settled. Both existing DataView -cases passed because they checked the new membership during notification and -the old membership only afterward. The revised callback snapshot also checks -that the old membership is absent. It fails the mutant in both index types and -passes unchanged production. This checks one-key, two-commit publication; it -does not prove every callback schedule or broader byte live-query behavior. diff --git a/docs/contributing/oracle-reviews/mixed-value-index-order.md b/docs/contributing/oracle-reviews/mixed-value-index-order.md deleted file mode 100644 index 321c24c8ad..0000000000 --- a/docs/contributing/oracle-reviews/mixed-value-index-order.md +++ /dev/null @@ -1,89 +0,0 @@ -# Mixed values in ordered indexes - -An indexed field can contain values from different JavaScript families. The -index API accepts these values, while mixed-domain range optimization is -disabled. Exact lookup, public WHERE membership, and unfiltered traversal -must still preserve each live key through legal deletes and updates. The -contract does not assign a public order between unlike families. Within one -orderable family, the configured comparator still applies. - -## Discovery and RED evidence - -An existing generated WHERE publication history failed with seed `907500611` -and path `78:8:6:6:8:7:6:7:8:10:10:9`. Its legal source held `b`, `1`, and -`a`, then deleted and updated rows. `BasicIndex.addToOrder` threw while it -looked up an ordered bucket. The default comparator had tied both strings -with the number through native relational coercion, although it ordered `a` -before `b`. Those ties do not form a partition that an ordered index can -maintain. - -The WHERE oracle now pins that history and compares public key sets after -each commit. The direct index owner also checks build and incremental -admission, reversed source order, exact lookup, and key conservation after -deletion and replacement for both BasicIndex and BTreeIndex. A source Map -predicts live keys and exact members. The test checks string collation only -within the string family; it does not prescribe where the number appears. -The public WHERE check establishes that writes and publication complete with -the right keys. It does not prove that a public indexed lookup ran after each -update; the direct index owner supplies exact lookup observations there. - -The same comparison fault reached other legal values. A fixed -PlainDate/ordinary-object history updated one row from an ordinary object to -a later PlainDate, then an earlier PlainDate. A later update lost or duplicated -keys, and the original BasicIndex implementation could throw at the same -bucket lookup. A callable/number history also lost keys because two distinct -functions each tied the number after native coercion. Direct witnesses now -check arrays, Dates, PlainDates, and callables beside ordinary objects or -numbers. They compare a full ordered take with the source Map as a key -permutation, check exact membership after each action, and retain order only -within comparable values of the same family. The original comparator failed -the PlainDate and callable witnesses before repair. - -## Repair and scope - -The default comparator now separates strings from other primitives. It -separates arrays, native Dates, branded Temporal values, and ordinary -references before the runtime object-ID fallback. Functions use that same -reference path. These family positions prevent a cross-family comparison -from tying or reversing an order established within another family. Same-kind -string, array, Date, and Temporal comparisons remain in their existing paths. -Mixed Temporal kinds that lack a common defined order still reject ordering. -The relational WHERE helper keeps its existing comparison semantics. - -After the repair, six affected test files passed 600 tests with no Vitest type -errors. The explicit test TypeScript check, formatting, and diff checks -passed. ESLint reported five existing `require-await` warnings in the older -part of the direct index oracle and no warning in the new code. - -This repair covers the named finite direct histories and the public WHERE -history. It does not claim a specific cross-family sort order, arbitrary -mixed-value histories, proxy behavior, mixed Temporal-kind ordering, or -validity of an arbitrary user comparator. A user comparator that violates -transitivity can still corrupt an ordered index. The direct index and -comparison owners need further generated family-mix histories before a -universal sorted-index claim. A fresh exact-commit hostile review and final -package verification follow after commit. - -## Frozen-head retirement audit and public-order neighbor - -The hostile reviewer inspected the ledgers, both index implementations, and -the receiving direct and public oracles at -`210651d207eefe9bbc025c780f6259e470225c9e`. It found no lost assertion -among the 18 retired BasicIndex/BTreeIndex blocks. The direct grammar's -build and delete observations include string/number and callable/number -histories. Array/object, Date/object, and PlainDate/object have incremental -add/update observations here, without a build/delete claim. The -[coverage map](../oracle-coverage.md) now states those boundaries explicitly. -This was source and ledger inspection, not an exhaustive generated proof. - -An adjacent public Date/number history exposed a separate order defect. -The unindexed ordered snapshot returned -`[equal-number, number, date]`; after either index type was installed, the -optimized snapshot returned `[date, equal-number, number]`. Two intended -index-type assertions failed before the repair. The -[custom-index selection owner](custom-index-order-selection.md) now compares -the full public order and limit-one result before and after index installation -and proves an index take ran. Retaining native Dates as order values while -keeping their numeric exact keys made the 497-test focused campaign and -12,345-test full DB campaign pass. This establishes scan/index agreement for -that fixture, not a universal mixed-family order or arbitrary histories. diff --git a/docs/contributing/oracle-reviews/perf-in-constant-set.md b/docs/contributing/oracle-reviews/perf-in-constant-set.md index b9c4b8da97..f8d3fcc6a8 100644 --- a/docs/contributing/oracle-reviews/perf-in-constant-set.md +++ b/docs/contributing/oracle-reviews/perf-in-constant-set.md @@ -5,17 +5,8 @@ production change is `18ea072b5`. This record adds the audit only. ## Contract and evidence -The original review called `inArray(value, list)` SQL `IN` under three-valued -logic. That wording is superseded for null-containing lists by the -[compiled `in` contract review](temporal-equality-alias.md#review-of-committed-head-5df06de2e) -and its coverage-map entry. The current evaluator treats a non-null value -against a null-only list as FALSE; SQL would yield UNKNOWN. This is an open -product-law decision, not a claim that the current rule is SQL-equivalent. -The normalization details below describe the reviewed `perf-in-set` head; -the linked review records later Temporal equality changes. - -Under the reviewed evaluator rule, a null or undefined value is UNKNOWN -(`null`). A list that is not an array is FALSE. The +`inArray(value, list)` is SQL `IN` under three-valued logic. A null or +undefined value is UNKNOWN (`null`). A list that is not an array is FALSE. The result is TRUE when some non-null list item equals the value under the evaluator's equality, and FALSE otherwise. @@ -106,55 +97,3 @@ times with no source index: Median write times did not change beyond noise. Two of the three comparisons were 1.00× and 0.98×, and a same-code comparison was 0.96×. - -## Frozen-head reference-model correction - -The hostile review of `210651d207eefe9bbc025c780f6259e470225c9e` -found a false-green risk in `referenceEqual`: realm-local `instanceof Date` -could classify a native foreign Date as an ordinary reference even when the -compiled evaluator correctly compared its timestamp. The working-tree model -now reads a captured native `Date.prototype.getTime`, uses valid and invalid -foreign Dates, and rejects a local object made with -`Object.create(Date.prototype)`. Fixed witnesses check compiled `eq`, constant-list `in`, and -row-supplied `in` in both operand directions. This was a reference-model gap, -not an observed production defect. The focused receiving run passed 72 tests. - -The transport controls for equal native Uint8Arrays from different realms -now expect `[true, true]` for the compiled equality paths; request-identity -assertions remain in place. This corrects old test expectations rather than -altering the native byte-content law. The open null-containing-list authority -question recorded by the [Temporal alias review](temporal-equality-alias.md) -is unchanged. The working-tree changes still need a fresh committed-head -hostile review. - -The review of committed head `d974d4a4f` found that the positive foreign-Date -controls came from one VM context, while a second context supplied only a -different timestamp. The receiving oracle now requires that second context's -Date to equal its own timestamp through compiled `eq`, constant-list `in`, and -row-supplied `in`. The focused `in` owner passed after this addition. This -checks the compiled equality return for two VM contexts; browser iframe -behavior and a fresh exact-commit hostile review of the follow-up remain open. - -## Own-method membership gap at `d917ba4c9` - -The compiled `in` evaluator and its reference both called the caller array's -overridable `some` method. A compiled `[1]` with an own `some` returning false -rejected present member `1`; after the list changed to `[2]`, an own `some` -returning true accepted absent member `1`. Constant and row-supplied lists -were both false green. The reference now walks present integer indices through -the length captured at evaluation, independently of the array's methods. -Production calls the native Array `some` with the current list as receiver. -Both false rejection and false acceptance failed at the compiled return before -the repair; the sparse-hole and nullish controls pass afterward. The focused -`in` owner passed 44 tests. This proves the named array-method boundary at -evaluation, not every possible proxy or mutation during one evaluation. - -## Exact-commit review of `5c97f18dc` - -A fresh read-only reviewer found no blocker in the own-method repair. The -model walks present indices independently of production's native `some` -call. Fixed constant and row-supplied cases reject both lying-false and -lying-true methods after a slot edit; sparse and nullish controls preserve the -stated behavior. The compiled `in` and byte-work owners passed 120 tests with -no type errors. Generated lists remain dense and static. Arbitrary getters, -proxies, and mutation during one evaluation remain outside this verdict. diff --git a/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md b/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md deleted file mode 100644 index d2e8514df0..0000000000 --- a/docs/contributing/oracle-reviews/pr-2116-correctness-and-guarantees.md +++ /dev/null @@ -1,182 +0,0 @@ -# PR #2116: correctness review and guarantee cost - -Reviewed executable commit: `4ed5242dbef0e7dbf87b31ca97c2fd2bc7901cad`. -Merge base: `b112d888f0c1f1ef4245ae48e8b071e8fad916b0`. This documentation -follows the reviewed code in a separate commit. The external review has 28 -source-order items; R29 is a sibling found while challenging the repaired -Temporal law. The task-local loss ledger retains each raw item and its evidence. - -## Laws, experiments, and limits - -| Item and authority | Independent judgment and reached checkpoint | Decisive failure and repaired evidence | Boundary | -| --- | --- | --- | --- | -| R16: a valid row admitted by a Collection cannot remain stored while its index and subscribers omit it. Existing Collection publication and changelog #1519 also require public `orderBy` to reject unsupported Temporal order. | Supplied genuine Duration-with-months, PlainMonthDay, and mixed-kind rows determine membership. BasicIndex/BTreeIndex add, reverse add, and build compare complete exact and ordered key sets. Two sequential sync commits compare the row, exact index membership, and callback batch; public limit-one `orderBy` must throw. | On the earlier code, the second commit threw and left two stored rows with one indexed/published row in six index/value combinations. A selected pre-repair oracle run had eight failed cases, including unexpected index-order throws. Restoring the public comparator at this head recreated partial publication for both index types; advertising private order as public returned a row instead of throwing. The final 12 focused admission checks pass. | One polyfill; three unsupported-order fixture classes; finite histories. No malformed Temporal promise, arbitrary callback, provider cursor, or long live-query history. | -| R17: after a returned direct index action, exact and ordered reads describe the same current keys. The index API and prior behavior authorize equal-time Date removal; a different callback position for a substituted old item has weaker authority. | A source Map retains each key's supplied timestamp or callback position. After add, remove, update, build, and re-add, both indexes are checked through key count, exact lookup, ordered take, and bounded range reads. | The old BTree cache left a detached ordered bucket after equal-distinct removal and object reuse. A nearby wrong-position removal left a ghost ordered key in both index types while exact lookup/count looked correct. The new history rejects both at take/range checkpoints. All 444 owner tests pass; 100-seed mixed-action probes per index found no mismatch. | Valid Dates, stable callbacks, unique-key add, direct index path. Arbitrary objects/comparators and Collection publication are outside this claim. | -| R25: native Dates with the same time are deep-equal across ordinary realms; a changed time must publish a Collection update so a subscriber mirrors the stored row. The live-query guide and Collection mirror law supply authority. | Supplied valid or invalid times and local or Node-vm constructors determine expected equality. The direct Date owner crosses realm and time margins; a ready unindexed sync Collection receiver compares exact batches, callback row, mirror, and settled row. | The old local `instanceof Date` branch missed changed foreign Dates and rejected equal mixed-realm Dates. Local-only and changed-time-ignored wrong answers fail direct checks; original public event traces failed before the repair. The final direct/receiving owners are included in the 552-test combined run below. | Ordinary native Dates in Node realms; seven fixed public sync cases. User-modified Date display tags have only fixed diagnostics, not a broad grammar. Indexed/optimistic publication and other hosts remain outside this witness. | -| R29: Duration query equality preserves its ten fields, even when two representations have the same display. Changelog #1519 states this for `PT60M` and `PT1H`; changed fields must publish enough for the Collection mirror. | Supplied ten-field vectors, both signs, opposite signs, eight fixed-factor unit carries, and three subsecond display collisions predict direct equality without production normalization. Fixed and generated direct properties compare `deepEquals`. Nine ready unindexed sync replacement histories compare subscriber batches, callback row, mirror, and settled row. | Display-only equality missed both directions of seconds-to-milliseconds changes. Before repair, five targeted cases failed. After adding negative controls, a display-only mutant failed seven targeted cases. An absolute-field mutant failed the opposite-sign check; a balancing mutant failed direct hour/minute and public change-message checks. Both owners pass 552 tests; saved replays #41 and #42 pass. | Direct magnitudes 0–3, display-collision magnitudes 1–100, one polyfill; representative public replacement histories. Arbitrary ten-field Collection histories, indexed/optimistic publication, and mixed Temporal implementations remain open. | - -The R25/R29 combined owner run has 552 tests. Six related owners pass 783 tests, -with no test type errors. A separate exact-head sweep compared 12,720 valid -Temporal pairs across all eight kinds with native equality or supplied Duration -fields and found no mismatch. This finite sweep does not extend the histories -or public paths above. The full DB suite passes 13,365 tests in 284 files; -DB-IVM passes 702 in 46 files; built-package tests pass 291 in five files. -Source and test TypeScript checks, ESLint with no errors, Prettier, and both -package builds pass at this code commit. - -## ORC-001 through ORC-014 - -| Requirement | Outcome at the reviewed code commit | -| --- | --- | -| ORC-001 contract | The opening law comments identify Collection publication, index refinement, native Date time, and #1519 Duration/Temporal behavior. R17's different-position substituted old item lacks explicit API authority and is cataloged below rather than credited to the general law. | -| ORC-002 independent judgment | Supplied rows, epochs, callback positions, times, and Duration field vectors determine expected results. No oracle imports the production comparator, normalizer, index cache, or `deepEquals` for its expected answer. | -| ORC-003 literate responsibilities | Each changed owner states law and limits before mechanics, then places the model, finite or generated grammar, real driver, public observation, and comparison beside their code. | -| ORC-004 grammar controls | R16 and the new R17 Date lanes are fixed finite histories. R25's generated pair crosses local/foreign placement and valid/invalid time; R29 crosses ten fields, two signs, equal/changed vectors, and three display-collision unit pairs. Each axis has a fixed reconstruction or mutant control. Mixed-sign Duration input is invalid and excluded. | -| ORC-005 public observation | Real indexes and Collections are driven. R16 checks callback-time membership and public rejection; R17 checks exact plus ordered methods after every direct action; R25/R29 check direct equality and exact Collection batches, callback row, mirror, and settled row. Complete keys expose duplicates and omissions. | -| ORC-006 calibration | The original R16, R17, R25, and R29 paths failed at their named checks. R16's unexpected legal-operation throw was itself the public violation; the ordered-read and equality faults reached assertion mismatches. Public-comparator, stale-cache, local-only Date, display-only Duration, absolute-field, and balancing wrong designs are rejected at their intended checkpoints. None was a timeout or setup error. | -| ORC-007 replay | Fixed lanes do not trigger generated replay. Existing R17 numeric properties keep their campaigns. The new R25/R29 utility properties run fixed and random campaigns with the existing direct seed/path replay interface; selectors #40–#42 preserve older #1–#39 registrations. Replays #41 and #42 passed at seed 20260911 and empty path. | -| ORC-008 model state | R17's source Map retains each key's epoch or callback position because a later legal removal or ordered read distinguishes owners with the same key count. The other new judgments recompute from supplied immutable facts. | -| ORC-009 vocabulary | Exact equality key, stored order operand, source row, change batch, and public ordered read retain their glossary/architecture meanings. The private Temporal index order is explicitly separate from public `orderBy`. | -| ORC-010 failure fidelity | The fixed direct checks have no shrinking or external cleanup. Collection receivers use the existing cleanup harness; reported RED cases reached their intended assertions before cleanup. Saved replay selectors preserve the generated target. | -| ORC-011 second formulation | R16 crosses direct index membership with public Collection publication/rejection. R25/R29 cross direct value equality with public subscriber mirrors. R17 crosses exact lookup with ordered reads that the old code separated. No real-provider formulation is claimed. | -| ORC-012 review evidence | This record binds the outcome, controls, and limits to commit `4ed5242dbef0e7dbf87b31ca97c2fd2bc7901cad`. Four hostile reviewers rechecked the committed R16, R17, R25/narrowed Temporal, and R29 owners; none found a blocker in the stated cells. | -| ORC-013 neighboring witness | R16 separates private index order from public rejection; R17 separates same-position from different-position old items and exact from ordered keys; R25 separates equal mixed-realm from changed-time Dates; R29 separates same display, opposite sign, and adjacent-unit carry from exact field equality. | -| ORC-014 provider handoff | These claims stop at direct indexes or controlled ready in-memory sync Collections. No external-provider or browser-host handoff is inferred. R21's broader IANA alias claim needs a real supported host witness before it can become a contract finding. | - -## Production weight and guarantees to decide - -Against the merge base, production TypeScript is **+876 net lines**. The -architecture document is **+56 net lines**, counted separately. A minified -build of every ESM module is **+4,647 gzip bytes** in aggregate -(205,407 to 210,054). A minified browser bundle of the public DB entry is -**+2,856 gzip bytes** (111,875 to 114,731) under the same esbuild command -at both revisions. These are two different footprint views, not additive -costs or a prediction for every consumer. The external review's +974-line -figure mixed production code and architecture prose at an earlier head. - -| Guarantee or work cost | Authority and value | Decision surface | -| --- | --- | --- | -| A captured mutable `inArray` reads current members and refreshes the whole retained query on the next source event. | The maintainer explicitly chose both rules. The receiving oracles cover WHERE, joins, selections, groups, aggregates, HAVING, order, nested queries, and selected on-demand/include paths. | Largest cost: the review estimated about 427 source lines. Full-source acquisition disables index/subset pushdown, keyed lazy demand, and pooled fast paths for this query shape. Changing this requires a contract decision; freezing a list already restores the optimizations. | -| A captured list's Date or byte-array item changes in place. | The maintainer declined guarantees after modifying standard values. Replacing the list entry remains inside the accepted live-list rule. | The evaluator and WHERE publication witnesses replace each list entry with a new native value. No oracle promises an observation after mutating the old item. | -| A custom index callback orders **each stored row**, even when query equality groups two values together. | The maintainer explicitly chose this. | Both indexes retain per-row order operands and may sort/refresh more rows. Callback arguments now expose raw Date/Uint8Array values instead of the former timestamp/encoded-byte representation. That compatibility change follows the per-row rule and should be called out in release notes. | -| Default native-byte order is unsigned lexicographic order. | This PR proposed the policy; the maintainer declined it. | The signed/unsigned and prefix order checks were removed. Byte-content equality and indexed/scan coherence remain. | -| Ordinary cross-realm native Dates and Uint8Arrays use their times or bytes. | The maintainer questioned realm-sensitive inequality after its history showed only a transport test, not a public promise. | Keep ordinary cross-realm equality with a narrow native-value detector. Date subclasses with overridden coercion are outside this choice. | -| Mixed-kind default index order stays total, while public Temporal `orderBy` still rejects kinds without an order. | Valid mixed rows must be admitted without partially updating a Collection; the specific rank between unlike ordinary kinds is a policy chosen by this branch. | The kind separation and private Temporal order add comparator code. A narrower accepted input domain or another internal order would need an explicit contract change and the R16 atomic-publication witness. | -| A direct index removal accepts an equal-time but different old Date whose custom callback position differs from the stored Date. | The general refinement law supports normal remove/re-add, but the index API does not document this substitution. | Added per-key stored-order state supports it. This is a candidate to narrow if the direct API may require the actual prior value. | -| A fresh, equal-field Duration replacement emits no update message. | Consistent with existing change-event suppression; the mirror law requires changed-field publication, not this silence. | The public R29 receiver asserts silence. It can be relaxed independently of exact field equality and changed-field publication. | -| User-modified Date display tags, byte-view length properties, and an array's own `some` method cannot redefine intrinsic value or `inArray` membership. | The maintainer expressly declined guarantees after callers modify standard data types. | These tests and their protective helpers were removed. Deliberately malformed, borrowed, forged, and shadowed Temporal values are also outside the claimed domain. | -| Repeated Temporal key/display work, per-row BasicIndex build sorting, and equal-row index refresh. | Deterministic work probes observed these costs; no work budget was promised. | Optimize only if a bounded workload justifies it. The old claim that *every* Temporal comparison calls `toString` is stale after the valid-value narrowing. | - -R21 is not closed as a Firefox defect. A mocked time-zone formatter preserving -aliases exposed sensitivity, but current ECMA-402 canonicalizes zone names -and no conforming real-host failure was found. The speculative alias scan -repair was removed. Reopen on a supported engine/version where two native-equal -zones produce different resolved names and a public scan or indexed equality -misses. - -## Integration after merging main - -Merge commit `f42e816d9c0442448624ab418074fd5008ee14c7` combines the reviewed -repair with `origin/main` at `bbbf017ef5b3884515295a4225faacdfd814499e`. -Main added replay retirement and bounded ordered-window behavior. The only -textual conflict was this architecture document; its resolution keeps both the -captured-list full-source rule and main's finite-window rule. No R16, R17, R25, -or R29 oracle expectation was changed during the merge. - -The combined tree passes 13,381 DB source tests, 705 DB-IVM tests, and 291 DB -dist tests. The packed IndexedDB package check passes 19 tests. Five focused -Temporal/index/equality/publication/replay owners pass 874 tests, with no test -type errors. DB source and test TypeScript checks and both package builds pass. -The focused integration review found no new mismatch in ordinary ready sync -publication. Date or Duration replacement during truncate replay is not a -separately generated cross-product; replay histories use numeric rows and the -replay path calls the repaired equality function. This is a receiving limit, -not evidence of a known failing legal history. - -Against this newer `origin/main`, production TypeScript is **+878 net lines** -(1,388 added, 510 removed); architecture prose is **+56 net lines** separately. -Minified ESM modules total 206,777 to 211,426 gzip bytes for DB (**+4,649**) -and 24,350 to 24,369 for DB-IVM (**+19**). A minified browser bundle of the -public DB source entry is 112,595 to 115,464 gzip bytes (**+2,869**) under -the same esbuild command at both revisions. The main bundle was reproduced -byte for byte before comparing. These views overlap and must not be added. - -The two largest behavior costs, live captured arrays with whole-result refresh -and per-row custom callback order, are explicit maintainer decisions. Ordinary -cross-realm native equality is retained after examining the old transport -test's history. Unsigned default byte order and intrinsic semantics for -user-modified built-ins are declined. Mixed-kind default rank, -different-position old-Date removal, and silence for equal-field Duration -copies remain separate product choices. The paired measurement below records -the current narrowed tree. - -On `origin/main`, `query/where.test.ts` and `query/indexes.test.ts` exercise -fixed `inArray` lists, including Date-by-time matching and an indexed lookup. -The `in-evaluator` oracle supplies constant or row-field lists but does not -edit a captured constant list after compilation. One direct evaluator work -test mutates a retained byte-array item and expects `in` to observe the new -bytes; it does not exercise a live-query Collection. Constant `Value` lists -were compiled into a `Set` once. No earlier test requires a captured list's -entries to be added, removed, or replaced after compilation, or a whole -retained result to refresh on the next source event. The current live-list -rule is an explicit new contract choice, not a unit-test transfer. - -## Narrowing decision: bytes and modified built-ins - -The retired unit-test blocks did not assert default byte ordering or resilience -to caller-modified standard values. The maintainer declined both guarantees. -Ordinary byte-content equality remains covered by earlier unit tests and the -binary-ID equality fix (#779). The maintainer also kept the explicit choice -that a custom index comparator orders each stored row. - -Before changing oracle expectations, the distinguishing predictions are: - -| Input and checkpoint | Earlier PR law | Narrowed law | -| --- | --- | --- | -| Two ordinary byte views `[127]` and `[128]`, default `orderBy` or direct comparator | The first sorts before the second by unsigned content. | No specific byte-content order is promised. Indexed and unindexed public reads of the same rows must agree, and exact lookup still groups equal byte contents. | -| Two fresh ordinary byte views with equal contents, exact lookup | Both match one content key. | Both still match one content key. A custom comparator may place their rows separately according to the callback. | -| A Date with an own display tag, a byte view with an own count or iterator shadow, or an array with an own `some` method | Native slots or intrinsic methods override the caller's changes. | These caller-modified built-ins are outside the oracle's declared domain. The oracle makes no prediction for them. Ordinary Dates, byte views, and arrays keep their retained laws. | -| An existing Date or byte-array item in a captured `inArray` list changes in place | The earlier working tree required the next source event to observe its changed content. | The accepted live-list law covers adding, removing, and replacing array entries only. Replacing an entry with a fresh native value refreshes the whole result; changing an existing item in place is outside scope. | -| An ordinary local Date or Uint8Array and an equal-content value made in a second JavaScript environment, compiled `eq` and `in` | The PR revision predicts both values match. | Retain content/time equality for ordinary native values. The surviving `subset-dedupe.test.ts` had expected only the literal's environment to match, so this remains an explicit behavior revision. | - -Removing a law from this table does not change the expected results of any -retired unit-test block. The cross-environment case is a separate revision of -a **surviving** `subset-dedupe.test.ts` expectation. That test entered in -commit `cfb01cee34` (PR #1797) under “request transport preserves predicate -matches.” It checked that the request kept its literal object and that the -evaluator returned its then-current result. The earlier code used local -`instanceof` checks; no public guide or changelog claimed realm-sensitive -equality. The maintainer questioned why such a distinction should be retained, -so this PR keeps equality for ordinary native values across JavaScript -environments while separately checking request identity. - -The R25 table above describes the earlier reviewed commit, which included a -foreign Date with a user-added display tag. After removing that case, the -ordinary changed-time foreign-Date replacement is a guard against an -always-equal model, not a reproduction of the original missed update. Source -inspection of `origin/main` shows that distinct unmodified foreign Dates were -always treated as unequal: a changed timestamp published, while equal -timestamps could publish an extra update. The current equal-time receiver -checks the narrowed publication rule. Cross-realm equality is a deliberate -contract choice of this PR; none of the retired unit-test blocks required it. - -## Narrowed working-tree weight - -After removing the declined byte-order, modified-value, and item-in-place -cases, production TypeScript is **+804 net lines** against `origin/main` -(1,305 added, 501 removed). Architecture prose is **+61 net lines** separately. -A paired browser source-entry measurement uses esbuild 0.28.1 -with `--bundle --minify --platform=browser --format=esm --target=es2020`, then -`gzip -n -9`, while resolving the same installed dependencies for each source -snapshot. The baseline is 114,670 bytes, the published PR head is 117,566 -(**+2,896**), and the narrowed working tree is 115,901 (**+1,231**). This -isolates the DB source change and is not the all-ESM-module sum. The absolute -byte counts differ from the earlier build method above; their paired deltas -are close. The entry-only list snapshot removes per-item normalization and -descriptor checks at each source event. Holding every other current source -file fixed, restoring the previous `snapshotList` produces 117,419 bytes, -so this simplification saves **1,512 gzip bytes**. A separate comparator loop -cleanup saves another six bytes. The current tree saves 1,665 bytes against -the published head. Most remaining code weight implements the two explicit maintainer -choices: live captured lists with whole-result refresh and per-row custom -comparator ordering. diff --git a/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md b/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md new file mode 100644 index 0000000000..032662dff4 --- /dev/null +++ b/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md @@ -0,0 +1,31 @@ +# PR #2116: deferred findings and guarantee decisions + +This record separates findings discovered during the unit-test retirement +workflow from the transfers retained in PR #2116. The final PR keeps only +the seven direct-index families, five non-finite comparison blocks, two +ordinary Temporal comparison blocks, and six Date/RegExp equality assertions. +It repairs the BasicIndex nullish full-build mismatch. A finding below is not +closed by passing those transfer oracles. + +| Finding | Authority and evidence | Next owner and witness | +| --- | --- | --- | +| R29: two valid `Temporal.Duration` values can have the same display text but different fields. A changed field can be suppressed as an equal Collection value. | Changelog #1519 says Duration equality preserves its fields. An earlier PR revision recorded a RED direct `deepEquals` result and a missing change event for a one-second versus 1000-millisecond replacement. The current `origin/main` implementation still normalizes by display text; this narrowed PR has not rerun that RED trace on the current base. | A separate Duration equality owner should model all ten fields independently, then a Collection change-event owner should compare the stored row, callback batch, and subscriber mirror after a valid replacement. | +| R16: an indexed Collection can admit a valid Temporal row whose index comparator has no public order, then throw after partial state changes. | Earlier PR revisions recorded RED admission and publication traces for Duration, PlainMonthDay, and mixed-kind rows. Source inspection suggests the risk remains on `origin/main`, but this narrowed PR has not executed a current-base RED witness. The public order law and internal index storage order must be distinguished. | The direct-index and Collection publication owners should pair a valid-row admission history with exact membership, ordered storage, stored rows, and subscriber observations. Choose the smallest internal ordering or atomic-admission repair that satisfies the accepted contract. | +| Mixed-kind indexed order and custom-index query selection may disagree with public query results. | Earlier branch experiments exposed risks, but the exact policy for unlike value kinds and comparator-equal rows is not settled by the retired unit blocks. The user withdrew the added guarantee that each row within one exact-equality class follows its custom callback position. No current-base RED claim is made here. | First state a supported-domain law and run indexed versus scan query witnesses on `origin/main`. Keep a custom comparator witness only if it follows that law without reintroducing the withdrawn per-row guarantee. | +| R17: an equal-time Date removal left an ordered ghost under added index machinery. | The recorded RED trace depended on code introduced during this PR. It does not establish an `origin/main` defect. The stronger rule for a substituted old row with a different callback position was withdrawn. | If ordinary direct remove/update still has a current-base counterexample, add a source-Map history to the direct-index owner before changing production. | +| Cross-realm native equality was asserted by a surviving request-transport unit test. | PR #1797 captured the then-current `instanceof` result. No public realm-distinctness promise was found. The maintainer questioned that distinction. This PR removes that incidental test expectation and makes no cross-realm equality claim. | A future public contract decision needs a host-boundary witness before either equality policy becomes an oracle law. | + +Captured `inArray` list-entry refresh and whole-live-query reconstruction were +explored as new features. The pre-PR evaluator and live graph did not already +deliver the whole-result behavior. The maintainer withdrew that expansion for +this PR because of its production-code and query-planning cost. A separate +proposal must define its dependency and invalidation boundary before adding +an oracle or production graph machinery. In-place edits to a Date or byte +item, unsigned default byte ordering, and guarantees after callers alter +standard objects are also outside the accepted contract. + +Earlier experiments and RED/GREEN records remain in published commits on +`codex/oracle-test-retirement-2-blocks`; this narrowed source tree does not +retain their production fixes. Statuses in this record distinguish an executed +earlier-branch failure, current-base source inspection, a proposed policy, +and a current-base observation that still needs to be run. diff --git a/docs/contributing/oracle-reviews/temporal-equality-alias.md b/docs/contributing/oracle-reviews/temporal-equality-alias.md deleted file mode 100644 index c1028163d7..0000000000 --- a/docs/contributing/oracle-reviews/temporal-equality-alias.md +++ /dev/null @@ -1,352 +0,0 @@ -# Temporal alias equality - -Scope update for PR #2116: this record retains earlier forged, borrowed, and -shadowed-value experiments. The current oracles claim valid Temporal values -from one polyfill copy and ordinary native byte arrays; caller-modified and -malformed values are outside their active laws. See the -[coverage map](../oracle-coverage.md) for current owners and limits. - -Changelog #1519 promises `ZonedDateTime` query equality consistent with -Temporal's `.equals()`. At one instant, `[UTC]` and `[Etc/UTC]` are equal, -while `[+00:00]` is a distinct zone. Exact nanoseconds and calendar identity -also matter. The new oracle assigns those classes in its fixture, then checks -direct BasicIndex and BTreeIndex lookups after add, build, and removal, plus -unindexed and optimized-only indexed public `eq` snapshots. The existing `in` -evaluator oracle now uses captured native Temporal methods as its independent -reference for kinds with `.equals()` and ten intrinsic fields for Duration. It -checks constant and row-supplied lists, including forged, borrowed, and -shadowed controls. - -## Failure and repair - -At frozen commit `8b04592724fb943b04a56c6682757448d26a8ff6`, exact -keys used display strings. Eight new direct and public alias checks failed at -their equality comparisons; the distinct-zone fixture control passed. -Four `in` checks also failed against the corrected reference, with no type -errors. The shared production cause was `normalizeValue`, so a repair limited -to an index would leave unindexed `eq` and `in` wrong. - -The repair keys ordinary ZonedDateTimes by kind, epoch nanoseconds, -canonical zone ID, and calendar ID. It reads those three fields through -prototype getters so an own property cannot change a Temporal instance's -equality class. A plain object with a forged Temporal tag remains an ordinary -object. Before that guard, the forged object and a genuine fixed-offset value -whose `timeZoneId` property shadowed its internal zone both matched a UTC -probe; the new direct-index and public-scan control rejected them. - -An independent read-only sweep compared the proposed zone key with native -`.equals()` on 177,906 same-instant pairs across 597 host zone IDs, finding -no partition mismatch. This bounds evidence for that host, not a proof for -every Temporal implementation or time-zone database. The focused index, -evaluator, and Temporal runs passed 453 tests with no type errors after the -repair. The full DB suite passed 12,171 tests in 279 files with no type -errors. The exact-commit review then found the gaps below, so that count -belongs to an intermediate commit. - -## Exact-commit counterexamples and follow-up - -At commit `1265abeb5`, hostile reviewers found that a genuine PlainDate with -an own `Symbol.toStringTag` or `toString` shadow could miss a native-equal -value through compiled `in`. A plain object with a forged PlainDate tag could -match a genuine value. A genuine ZonedDateTime with a shadowed tag could miss -through direct indexes and public `eq`. The `in` reference had trusted an -instance's display tag and therefore agreed with one wrong production answer. -Its Duration wording also claimed `.equals()` even though Duration has no such -method. Eleven new brand, shadow, and Duration checks failed at the intended -comparisons on the original code, with no type errors. - -The follow-up reference calls captured polyfill intrinsics on the candidate to -establish its brand and models Duration with ten field getters. Production -normalization reads slot-checking methods from the value's prototype rather -than its own display members. The alias owner also checks the later-instant -and alternate-calendar classes and a PlainDate forged/borrowed/shadowed lane -through direct singleton lookups and an unindexed public snapshot. The focused -39-test alias and `in` oracle run passed with no type errors after repair. -The mixed ordinary-object/Temporal ordered index is excluded because its -range order is unspecified. Arbitrary custom prototypes that imitate genuine -Temporal methods remain outside the one-polyfill brand guarantee. - -## Oracle guide audit - -| Requirement | Evidence and limit | -| --- | --- | -| ORC-001 | Changelog #1519 and Temporal `.equals()` authorize the equality classes. The Duration field-vector rule follows #1519's explicit `eq` distinction. The opening oracle comment limits implementations, paths, and observations. | -| ORC-002 | Fixture-assigned classes predict key sets without calling production normalization. Native `.equals()` calibrates supported kinds; captured intrinsic Duration fields calibrate Duration. | -| ORC-003 | Prose beside the model, add/build history, Collection driver, and assertions explains their roles. | -| ORC-004 | Not triggered by this fixed oracle. The separate generated `in` owner controls its bounded list grammar. | -| ORC-005 | Direct lookups observe both index types for orderable kinds; an indexed public snapshot spies on `lookup`, and unindexed snapshots check the evaluator path after readiness. PlainMonthDay has no indexed claim. | -| ORC-006 | The frozen display-string design failed alias checks. The first tuple design failed forged-tag and shadowed-zone controls. Commit `1265abeb5` failed eleven brand/shadow/Duration checks; `c4c07da42` failed the intrinsic-field equality controls. A lexical `in` mutant fails its comparison. | -| ORC-007 | Not triggered by this fixed oracle. The adjacent `in` owner retains fixed and random generated campaigns. | -| ORC-008 | The fixture model is immutable and has no reference-model state. | -| ORC-009 | The model's class labels denote query equality, distinct from the order oracle's instant labels. | -| ORC-010 | Collection cleanup preserves the primary mismatch. Direct index checks own no external resource. | -| ORC-011 | Direct lookup and public scan cross different production paths, but share normalization. Native `.equals()` and the Duration field-vector rule provide independent semantic calibration; this record does not claim a second full reference model. | -| ORC-012 | This record names the owner, RED controls, limits, the rejected PlainMonthDay index design, and required exact-commit review. | -| ORC-013 | Alias equality versus fixed-offset inequality separates display-only and instant-only keys; same-display reference-field and Duration pairs reject string-only keys. Native-brand controls reject forged/borrowed impostors while genuine display shadows preserve equality. | -| ORC-014 | The Collection uses controlled in-memory sync and ends its claim at the snapshot; no provider handoff is claimed. | - -## Limits and exact-commit review - -The ZonedDateTime lane covers one polyfill copy and one instant; the -PlainDate lane checks brand shadows at one calendar date. Other IANA aliases, -arbitrary spoofed prototypes, mixed Temporal implementations, joins, grouping, -query identity, live-query publication, provider paths, and longer histories -need their own witnesses. The `in` owner adds generated list membership but -not those other boundaries. `Intl` canonicalization has a raw-ID fallback when -the host rejects a valid Temporal zone; alias equivalence in that case is -unproved. The index-order owner now checks a custom comparator that -separates query-equal aliases. Direct D2 `groupBy` has a separately approved -kind-and-string policy. - -At committed head `c4c07da42`, a fresh reviewer found no mismatch in the -exercised alias cases but found that the previously reported genuine -ZonedDateTime own-tag failure lacked a direct witness here. The fixture now -adds genuine UTC rows with an own shadowed display tag and an own shadowed -`toString`. Native `ZonedDateTime.prototype.equals` confirms their equality -with the alias probe. Add, build, removal, unindexed public `eq`, and -optimized-only indexed public `eq` must all retain those rows. The focused -run passed 11 alias tests with no type errors. Final committed-head review -remains pending. - -## Intrinsic-key counterexamples and index boundary - -The same `c4c07da42` review found three Temporal equality classes whose -default display omits identity fields. Native `.equals()` distinguishes -PlainMonthDay reference years and PlainYearMonth reference days. Changelog -#1519 keeps Duration field representations distinct under `eq`, even when -both display as `PT1S`. Compiled `eq`, constant `in`, and row-supplied `in` -returned false matches before the production repair. New fixed pairs check -equal copies and unequal same-display values in both operand directions. -Normalization now reads the intrinsic reference fields and ten Duration -fields. The receiving `in` model reads captured polyfill methods and getters. - -This alias oracle carries the same equality classes to a public unindexed -PlainMonthDay scan. Its PlainYearMonth and Duration lanes check direct index -add, build, removal, and re-add, plus public scan and indexed `eq`. Each -fixture assigns classes from constructed fields without using production -normalization. The six focused suites passed 619 tests after the repair. - -An attempted one-class PlainMonthDay index bypass was rejected during law -review. With two native-equal rows, a BasicIndex made public `orderBy` return -keys, while the unindexed query threw as changelog #1519 specifies. Adding a -later distinct reference-year row then threw during a valid sync commit. The -bypass was removed. Direct and public indexed PlainMonthDay equality across -multiple classes remains open. An exact-only index path needs a separate -oracle for add, build, remove, update, and the orderBy rejection. - -## Review of committed head `5df06de2e` - -Fresh review found no mismatch in the exercised alias cases. It identified -one remaining sensitivity gap in the own-property controls: genuine -ZonedDateTimes with own `epochNanoseconds` or `calendarId` shadows were not -present. The fixture now gives both values an independently assigned UTC -identity class. Native `ZonedDateTime.prototype.equals` confirms each equals -the alias probe, and direct add/build/removal plus unindexed and optimized-only -public `eq` must return them. The focused alias owner passed 28 tests. - -The separate compiled `in` review found a false-green kind-only possibility: -its pool contained only one intrinsic value of several Temporal kinds. -Equal and unequal Instant, PlainDate, PlainDateTime, and PlainTime values now -reach compiled `eq`, constant `in`, and row-supplied `in` in both directions. -A kind-only membership mutant reaches the shared comparison and is rejected -as an expected-false/actual-true mismatch. The review also found that the -byte reference classifier trusted an instance's display tag. Captured -typed-array intrinsic classification now admits a genuine Uint8Array with an -own tag shadow and rejects spoof-tagged Uint16Arrays; fixed controls reach all -three compiled paths. The focused `in` owner passed 33 tests. - -`evaluators.in` previously had no guarded replay owner. Its replay manifest -now names this oracle and selects only the random/replay campaign. The -property uses an async fast-check runner so the guarded witness reporter -records the completed run; a targeted replay test passed with one selected -property and the fixed campaign skipped. -For failure calibration, a temporary kind-only driver substitution failed the -random campaign after 11 tests at seed `42`, path `10:2:3`. Guarded replay of -that coordinate ran only this property, reproduced the same unequal PlainDate -membership mismatch in one test, recorded `failed: true`, and exited nonzero. -The substitution was removed; the production driver again evaluates compiled -`in`. - -One contract-authority question stays open. For a non-null value and a -null-only list, the current model and evaluator return FALSE, so a compiled -`not(inArray(...))` returns TRUE. PostgreSQL would return UNKNOWN. The -[live-query guide](../../guides/live-queries.md#comparison-semantics) broadly -says comparisons follow SQL/PostgreSQL null conventions, while its `inArray` -description states array membership without defining null-list behavior. The -compiled `in` owner records the discrepancy; changing the law requires a -product decision and separate public WHERE/negation witnesses. No SQL-style -null-list claim is made for this transfer. - -## Review of committed head `ae0b67fd2` - -Fresh hostile review found no lost retired comparison assertion in the alias -owner. It found one sensitivity gap: forged and borrowed ordinary objects were -checked only for inequality with genuine Temporal values, so an index could -drop their own entries and still pass. Direct BasicIndex/BTreeIndex lookups now -require each impostor to find itself, while a distinct copy must not match. -Unindexed public `eq` makes the same self-identity and distinct-copy checks. -The focused alias owner passed 28 tests after the extension. This guards the -ordinary-object side of brand rejection without treating a spoofed display -tag as Temporal identity. Fresh exact-commit review of these new controls is -pending. Indexed PlainMonthDay equality across multiple classes remains open. - -The same committed-head review found no blocker in the compiled `in` owner's -stated grammar: 33 tests and guarded replay passed. It confirmed the -null-only-list authority question above and identified the older -`perf-in-constant-set.md` review's SQL-equivalence sentence as too broad. That -record now explicitly supersedes the sentence for null-containing lists. -Two lint-only edits to the compiled `in` owner follow this review and require -a fresh exact-commit check. - -## Hostile review of `be5b7a3c4` - -The alias reviewer found no loss from the retired comparison assertions. It -identified a positive native-equality gap: a genuine `[+00:00]` -ZonedDateTime with an own `timeZoneId = 'UTC'` shadow had been checked against -a UTC value and itself, but not against a fresh native-equal `[+00:00]` -value. The oracle now calibrates native `.equals()` on that fresh probe and -requires direct BasicIndex/BTreeIndex lookup and unindexed public `eq` to find -the shadowed row. Current production passed this probe. It rejects a wrong -classifier that treats every shadowed value as reference-only. - -The reviewer also demonstrated a custom-prototype ordinary object that -supplies a `Temporal.PlainDate` tag and matching display text. Current direct -indexes and public `eq` match it with a genuine PlainDate and a distinct fake -copy, while the native PlainDate intrinsic rejects it as a receiver. This is -outside this owner's declared custom-prototype grammar, so it does not undo -the bounded retirement verdict. It remains an open brand/ordinary-object -contract question under this owner: decide whether all non-branded objects -retain reference identity, then add genuine/fake/copy direct-index and public -`eq` witnesses. A shape or candidate-method guard can be spoofed; a robust -repair needs a trusted Temporal intrinsic boundary that supports the imported -polyfill without relying on `globalThis.Temporal`. - -The separate compiled `in` reviewer found native `Uint8Array` values from a -second JavaScript realm missing from its otherwise independent byte model. -Equal foreign/local bytes failed compiled `eq`, constant `in`, and row-supplied -`in` in both operand directions at `be5b7a3c4`. The receiving owner now -generates equal and unequal foreign bytes, including an own display-tag -shadow, and keeps a spoof-tagged wide-array negative control. The production -classifier uses a captured native typed-array getter after `ArrayBuffer.isView` -so realm and user display tags cannot change the byte class. The fixed -witness failed in twelve intended assertions before that repair and passed -after it; 359 related tests and static checks passed. This proves compiled -boolean equality in the `node:vm` realm, not browser iframe behavior or public -WHERE publication. The null-only-list law question above remains open. - -## Native byte-content follow-up at `3668d1b88` - -The next compiled `in` reviewer found that two unequal native Uint8Arrays -could each define an own `byteLength = 0` property and compare equal. A -one-sided shadow could also make equal arrays compare unequal. The compiled -`eq`, constant-list `in`, and row-supplied `in` paths all reached the wrong -result. Direct `normalizeValue` also generated the empty-byte key for a -nonempty shadowed array. These are production defects under the native -byte-content law, beyond the retired Temporal and comparison examples. - -The receiving `in` reference now reads captured native length and iteration -intrinsics, while production compares indexed bytes. Fixed controls cross -equal and unequal native bytes with own `byteLength` or `length` shadows, a -foreign byte array with a count shadow, and an iterator that yields values -different from the native bytes. Every compiled equality path checks both -operand directions. The comparison owner separately checks normalized key -equality and inequality under count shadows. Before the source repair, the -three compiled paths failed at their boolean comparisons and direct -normalization collided with the empty-byte key. The repaired focused set -passed 362 tests with no type, lint, formatting, or diff-check errors. - -These witnesses cover native Uint8Array and Buffer content at the compiled -equality and direct-key boundaries. They do not establish public indexed -publication, detached buffers, proxies, or browser iframe behavior. A fresh -hostile review of the committed repair is pending. - -## Genuine subclass counterexample after frozen-head review - -At frozen head `210651d207eefe9bbc025c780f6259e470225c9e`, a genuine -`Temporal.ZonedDateTime` subclass inherited the native slot-checking -`.equals()` relation but the production classifier treated it as an ordinary -reference. The alias owner now constructs that subclass from independently -supplied instant and zone facts, verifies native equality against a UTC peer, -and requires its class in direct BasicIndex/BTreeIndex lookups and unindexed -public `eq` reads. It also removes the subclass representative and checks the -remaining class. An object that merely borrows the subclass prototype and a -distinct borrowed copy must retain reference identity. Eight intended -comparisons failed before the classifier repair; the focused receiving run -passed 28 tests afterward. The repair recognizes an inherited Temporal -prototype whose method checks native slots. - -This is a production defect adjacent to the retired comparison assertions, -not a lost assertion from those blocks. The bounded claim covers this -polyfill's genuine subclass with inherited methods. The custom-prototype -fake described above, replaced subclass methods, mixed implementations, and -provider paths remain open. The working-tree repair still needs a fresh -committed-head hostile review. - -## Tagged subclass follow-up after `d974d4a4f` - -The committed-head review found another native-brand hole. A genuine -`ZonedDateTime` subclass whose own prototype shadows `Symbol.toStringTag` -inherits Temporal's slot-checking methods. Native `.equals()` groups it with -a UTC alias, and native `compare` gives it the same instant position. The -classifier stopped at the immediate prototype, treated that genuine value as -an ordinary object, and missed it in direct BasicIndex/BTreeIndex lookups and -public `eq` snapshots. A distinct ordinary object borrowing that subclass -prototype must still retain reference identity. - -The alias owner now adds the tagged subclass to its independent UTC class, -checks direct add, build, lookup, removal, and unindexed and optimized-only -public `eq`, and retains the borrowed-prototype negative controls. The -comparison owner checks exact zeros and both signs through `ascComparator` -and `compareValues`. Ten intended comparisons were RED before the one-line -classifier repair. The repaired focused run passed 144 tests: 28 alias and -116 comparison, with explicit test TypeScript, ESLint, Prettier, and diff -checks passing. The repair removes the immediate-prototype tag gate and -continues to require a slot-checking inherited Temporal method. This evidence -covers one imported polyfill's subclass with inherited methods and one -shadowed prototype tag. Replaced intrinsic methods, other implementations, -arbitrary prototype chains, and provider paths remain open. A fresh hostile -review of the final committed change is pending. - -## Read-only review of `0c236f448` - -The reviewer ran 28 alias tests and found no blocker in the then-declared -subclass domain. It identified a neighboring sensitivity gap: the tagged -subclass used an unrecognized tag, so a classifier that trusted a recognized -but misleading `Temporal.PlainDate` tag could still put a genuine -`ZonedDateTime` in the wrong equality kind. The fixture now gives another -genuine subclass that tag while leaving its inherited native methods and -slots intact. Native `ZonedDateTime.prototype.equals` confirms its UTC alias -class. Both direct index types check add, build, exact lookup, and removal; -unindexed and optimized-only public `eq` check the same class. An ordinary -object borrowing that subclass prototype and a distinct copy remain -reference-identity controls. The revised owner passes 28 tests. This adds -one recognized wrong-kind tag to the fixed polyfill grammar; arbitrary -prototype impersonation and mixed implementations remain open. Fresh hostile -review of the next committed head is pending. - -## Duration field sensitivity after `d917ba4c9` - -Read-only review found that the alias and compiled `in` owners claimed all -ten intrinsic Duration fields while their fixtures did not vary `years`, -`months`, `weeks`, `days`, or `nanoseconds`. Five new variants each change one -of those fields from an equal baseline. The compiled owner checks `eq` and -both `in` forms in both operand directions; its omit-one-field mutant fails at -the membership comparison for every omitted field. The alias owner checks -the same independent equality classes in an unindexed public scan and exact -BasicIndex/BTreeIndex lookups. Those indexes use an explicit total comparator: -default ordering of calendar-unit Durations requires `relativeTo`, so this -evidence does not claim their default ordered-index behavior. The focused -compiled and alias owners passed 75 tests. A fresh committed-head hostile -review remains pending. - -## Exact-commit Duration review of `5c97f18dc` - -A fresh read-only reviewer found no blocker for the five new intrinsic-field -classes. Each fixture variant differs from its baseline in one captured -Duration field. Compiled `eq` and both `in` forms check both operand -directions, and an omit-one-field wrong answer fails the independent -membership comparison. The alias owner checks a public unindexed scan and -both built index types' exact lookups. The explicit text comparator supplies -only the index's required order. The two focused owners passed 75 tests with -no type errors. The five variants do not themselves exercise indexed public -reads, index removal, or default ordering of calendar-unit Durations. diff --git a/docs/contributing/oracle-reviews/temporal-index-order.md b/docs/contributing/oracle-reviews/temporal-index-order.md deleted file mode 100644 index 68afccfeb7..0000000000 --- a/docs/contributing/oracle-reviews/temporal-index-order.md +++ /dev/null @@ -1,171 +0,0 @@ -# Temporal indexed order - -The direct comparison transfer in [round 6](comparison-unit-retirement-round-6.md) -does not exercise an index. A separate oracle checks the public index contract: -an ordered take must follow `ZonedDateTime` instant order, while exact equality -keeps zones distinct. The law comes from Temporal's `ZonedDateTime.compare` and -`equals` semantics, changelog #1519, and the Collection's ordered-read and -IndexInterface contracts. The live-query architecture also requires an -ordered comparator group to keep a representative from a live exact-value -bucket. Changelog #958 promises deterministic key-based tie breaking for -Collection and index iteration, including live queries with `orderBy` and -`limit`. Both direct indexes and the public ordered snapshot therefore use -the source-key tie rule for two rows at the same instant. - -## Original failure and repair - -At frozen head `c038b2b673a601d3bfbc70261f2cbb61f6f934ea`, both BasicIndex -and BTreeIndex normalized Temporal values to display strings before ordering. -For the supplied rows, `d-minus-five` has a later instant than the UTC and -+14:00 rows but an earlier display string. Both direct index types returned it -first after `add` and `build`. An optimized-only public Collection snapshot -with `limit: 1` exposed the same wrong first row; the unindexed control returned -an earliest-instant row. These failures occurred at the intended ordered-key -and public-snapshot comparisons. The exact same-instant zones retained separate -equality lookups. - -The repair retains normalized exact keys for equality and stores raw Temporal -values for ordered positions. A removal control caught the first repair -retaining a BTree representative whose exact-value bucket had departed. -The corrected BTree replaces that representative with one from a remaining -bucket. An initial hostile objection treated this as an unwarranted private -assertion; rechecking the normative architecture established its authority. -The existing index refinement oracle also rejects the version that removed -the replacement. The original repaired commit passed 13 focused tests and -the index refinement oracle's 412 tests, then the full DB suite passed -12,154 tests in 278 files. After the alias and comparator controls, the full -DB suite passed 12,171 tests in 279 files with no type errors. A later exact -commit review found the custom alias gap below, so those counts describe an -intermediate commit rather than final verification. - -The independent fixture supplies epoch nanoseconds and zone-identity classes. -It never calls the index's normalization or comparator to compute expected -results. A prefix checker requires all earlier instants, no later instants, -unique keys, and instant order. It permits either choice at an equal-instant -limit boundary. Exact lookup compares zone-sensitive key sets. The grammar has -two opposing offset zones at one instant, a successor one nanosecond later, -a later instant with earlier display text, and three pre-epoch values that -distinguish signed numeric order from decimal-string order. It exercises direct -`add`, `build`, a value cursor past both equal-instant peers, one -remove/insert/update continuation, and an explicit index in an optimized-only -Collection read after sync readiness. The collection driver spies on the -chosen index's `takeFromStart` to prove the intended path ran. - -## Hostile review and limits - -A separate hostile reviewer traced the exact-key and ordered-value paths in -both indexes, examined the original failure, and checked the repaired public -index selection. It found no blocker within the bounded default-comparator -domain. It identified a callback compatibility regression: a custom -`compareFn` received raw Temporal objects where it previously received -normalized strings. A first follow-up kept the string type but supplied a -new epoch tuple; a pre-epoch callback witness rejected its wrong direct -ordered result in both index types. The repair now passes the former display -key to custom callbacks while storing raw values for index order. This is a -bounded compatibility decision; published types do not promise the operand -representation. The exact-commit reviewer then found that a callback could -distinguish UTC and Etc/UTC values in one query-equality class. Both indexes -merged the exact key before comparing those rows, so ordered takes, ranges, -and cursors disagreed with the callback. Six new add/build/update witnesses -failed at `takeFromStart` on `1265abeb5`, with no type errors. The repair keeps -exact lookup and comparator positions as separate partitions. The witnesses -now check the callback's exact former display-string operands, equality lookup, -range boundaries, both cursor directions, representative removal, and a -same-class update. The focused 470-test index campaign passed with no type -errors after repair. Direct removal supplies the actual prior source value; -no equivalence-substitution guarantee is claimed for a different old item. -For this PR, the maintainer selected per-row callback order when a comparator -distinguishes values in one query-equality class. This is the contract the -separate partitions implement. -Mixed domains, other Temporal kinds, public WHERE ranges, provider cursors, -longer histories, and live-query publication remain outside this oracle. -Existing range optimization does not claim Temporal WHERE correctness. - -A subsequent direct comparison witness found that a genuine ZonedDateTime -with an own shadowed `Symbol.toStringTag` lost its native instant order. -The matching index witness failed in all four BasicIndex/BTreeIndex add/build -histories at the value cursor: it returned the equal-instant peer instead of -the one-nanosecond successor. Classification now reads a slot-checking -prototype method, so the own display tag cannot choose an ordinary-object -position. The direct and index witnesses pass after that repair. The fixed -index witness does not establish custom comparator behavior with a shadowed -tag or mixed Temporal implementations. - -## Oracle guide audit - -| Requirement | Evidence and limit | -| --- | --- | -| ORC-001 | Temporal comparison and changelog #1519 authorize instant order and zone-sensitive equality. Changelog #958 authorizes source-key ties in direct indexes and public ordered snapshots. The fixed domain remains bounded. | -| ORC-002 | Supplied BigInt instants and zone classes determine expected order and exact matches independently of normalization and comparison code. | -| ORC-003 | The oracle describes its law, fixture model, legal actions, direct and public drivers, observations, and checkpoints beside executable code. | -| ORC-004 | Not triggered: the bounded fixture matrix and one continuation are fixed histories, not a generated-history claim. The nearby invalid mixed-kind domain is excluded. | -| ORC-005 | Direct takes, ranges, cursors, and equality lookups observe both index implementations. An optimized-only Collection snapshot invokes the explicit index's `takeFromStart` after readiness. | -| ORC-006 | The original normalized-string design failed at direct ordered takes and the public `limit: 1` snapshot. A wrong-result control rejects both a later instant and the wrong source-key peer at the equal-instant limit. The first BTree repair failed the live-bucket representative check; a tuple-fed custom comparator failed the pre-epoch take check; the merged-exact-key design failed six alias/callback takes. Own-tag shadows failed all four value-cursor checks before the branded dispatch repair. | -| ORC-007 | Not triggered: this is a bounded fixed oracle, not an important generated property. The existing index refinement property keeps its own campaigns. | -| ORC-008 | Not triggered: the fixture's independent model has no mutable reference state. | -| ORC-009 | The model's instant label represents default order only; its zone class represents exact equality. One comparator position can contain two exact values, and a custom callback can split one exact value into two positions. The callback-argument witness checks exact former input text. | -| ORC-010 | `withOracleCleanup` preserves a primary public mismatch and releases the Collection. Direct indexes own no external resources. | -| ORC-011 | The unindexed public scan is a second production formulation for the ordered result. The numeric BigInt model also distinguishes a shared display-string mistake. | -| ORC-012 | This record contains RED/GREEN evidence, a hostile review, and scoped unresolved cells. An exact-commit verdict follows below. | -| ORC-013 | Equal-instant zones admitted in reverse key order distinguish source-key ties from admission order. A one-nanosecond neighbor distinguishes collapse and truncation; a later instant with earlier display text distinguishes lexical order. | -| ORC-014 | Not triggered: the Collection has controlled sync and the claim ends at its in-memory ordered snapshot, with no real-provider claim. | - -## Exact committed-head review - -At `ae0b67fd2`, a fresh hostile reviewer found no blocker in the stated -default and custom ordered-index domain. All 25 focused tests passed. Its -independent bounded histories exercised both index types under both order -rules, and a wrong-result probe confirmed that the optimized public snapshot -uses the index result. Provider cursors, arbitrary callbacks, and live-query -publication remain outside this verdict. - -## Frozen-head direct tie-order recheck - -The hostile review of `210651d207eefe9bbc025c780f6259e470225c9e` -found a source-inspection sensitivity gap, not a demonstrated production -defect. The direct oracle had allowed either source-key order for two -equal-instant peers. It now admits those peers in reverse key order through -both `add` and `build` and requires the direct `takeFromStart(2)` result to -follow the established source-key tie rule in BasicIndex and BTreeIndex. The -earlier public limit-one allowance was too broad under changelog #958. This -addition initially covered only the named direct-index checkpoint; the public -receiving cut is described below. - -The maintainer confirmed the custom comparator contract: when a callback -distinguishes two `ZonedDateTime` rows that native `.equals()` groups -together, ordered index operations follow the callback for each row. The -existing oracle checks that decision through add/build, exact lookup, -range and cursor reads, removal, and update for UTC/Etc-UTC peers. Review of -committed head `d974d4a4f` found that those checks could accept display-text -order in place of callback order. A new opposite callback puts UTC first, -while both display-text and source-key order put the alias first; add/build -takes, range, and cursor results must follow that callback. Its focused run -passed 33 tests. This round does not collapse the peers' positions or reject -that comparator/value combination. The earlier 497-test focused campaign -and 12,345-test DB campaign describe the committed baseline, before this -opposite-callback witness. A fresh exact-commit hostile review of the -follow-up remains pending. - -## Read-only review of `0c236f448` - -The reviewer ran 54 focused and neighboring tests and found two enforcement -gaps, with no reported production mismatch. First, the public limit-one -checker allowed either equal-instant peer even though changelog #958 promises -source-key tie breaking for Collection and index iteration. The model now -sorts supplied instant labels and then source keys. A wrong-tie control fails -at the exact prefix comparison. The public witness admits the larger-key UTC -row first, then compares unindexed and optimized indexed limit-one snapshots -with the smaller-key +14:00 row; the index take spy confirms the optimized -path ran. - -Second, the opposite custom callback had distinguished display order on -add/build, but a later same-equality-class update could still leave a stale -comparator position. The new continuation changes Etc/UTC to UTC and checks -the resulting source-key tie, inclusive ranges, and empty cursors. A second -history begins with a callback tie, updates one row to UTC, then requires -the callback to put it first despite both display text and source keys -preferring the other row. Both BasicIndex and BTreeIndex run these histories. -The revised owner passes 35 tests. The law remains confined to this polyfill, -the named UTC aliases, fixed histories, and direct/indexed Collection cuts; -provider cursors and longer histories remain open. Fresh hostile review of -the next committed head is pending. diff --git a/docs/errors.md b/docs/errors.md index 678dae164d..f33e22aa84 100644 --- a/docs/errors.md +++ b/docs/errors.md @@ -2449,13 +2449,3 @@ Development builds show this message, with `${...}` replaced by values: ```text LocalStorage sync belongs to its options Collection. ``` - - - -## Error 243: `query/compiler/mutable-list-refresh.ts` - -Development builds show this message, with `${...}` replaced by values: - -```text -An inner mutable-list revision lost its candidate -``` diff --git a/docs/guides/live-queries.md b/docs/guides/live-queries.md index 06564b8a93..8b1ea145c8 100644 --- a/docs/guides/live-queries.md +++ b/docs/guides/live-queries.md @@ -778,7 +778,6 @@ For a complete reference of all available functions, see the [Expression Functio Comparisons follow SQL/PostgreSQL conventions rather than raw JavaScript: - **`null` / `undefined` use three-valued logic.** Any comparison involving `null` or `undefined` evaluates to `UNKNOWN`, so the row is not matched. For example `eq(user.score, null)` matches nothing — use a dedicated null check (e.g. `isUndefined`) to match missing values. -- **Ordinary native `Date` values compare by timestamp.** Dates with the same millisecond timestamp match equality and occupy the same order position, including Dates created in another JavaScript environment. - **`NaN` follows PostgreSQL float semantics.** `NaN` is treated as equal to itself and greater than every other (non-null) value. So `eq(row.value, NaN)` matches `NaN` rows, `gt(row.value, x)` includes `NaN`, and ordering by such a field places `NaN` last. (Invalid `Date` values, whose timestamp is `NaN`, behave the same way.) This differs from JavaScript, where `NaN === NaN` is `false`, and matches how PostgreSQL orders and indexes floating-point values. ## Select @@ -2647,17 +2646,6 @@ inArray(user.id, [1, 2, 3]) inArray(user.role, ['admin', 'moderator']) ``` -If a query captures a caller-owned array, it reads that array's current -members when the next source change is processed. That source change updates -the whole existing query result, including rows that did not change. This -applies when `inArray` filters rows, computes selected values, forms a join or -group, supplies an aggregate even inside another selected expression, filters -with `having`, or determines order. The -resulting values and order are published together. Editing the array alone -does not publish a new result. Current members means entries added, removed, -or replaced; changing an existing Date or byte-array item in place is outside -this guarantee. - #### `like(value, pattern)`, `ilike(value, pattern)` String pattern matching: ```ts diff --git a/docs/reference/functions/compareTemporalValues.md b/docs/reference/functions/compareTemporalValues.md index c3c7ccf1e1..a96de8a7ef 100644 --- a/docs/reference/functions/compareTemporalValues.md +++ b/docs/reference/functions/compareTemporalValues.md @@ -7,18 +7,14 @@ title: compareTemporalValues function compareTemporalValues(a, b): number; ``` -Defined in: [packages/db/src/utils/comparison.ts:345](https://github.com/TanStack/db/blob/main/packages/db/src/utils/comparison.ts#L345) +Defined in: [packages/db/src/utils/comparison.ts:300](https://github.com/TanStack/db/blob/main/packages/db/src/utils/comparison.ts#L300) Compare two Temporal values of the same type, returning -1, 0, or 1. -The tag checks type agreement. The first value's prototype supplies the static -compare function, so an own `constructor` property cannot change dispatch or -poison later comparisons. Types without a static `.compare` (e.g. +The tags check type agreement. Types without a static `.compare` (e.g. `PlainMonthDay`) throw instead of using text order. -Callers must ensure both arguments are ordinary Temporal objects from one -compatible implementation. Mixed types throw. Replaced prototypes and custom -subclass comparators are outside this contract. +Callers must ensure both arguments are Temporal objects; mixed types throw. ## Parameters diff --git a/packages/db-ivm/src/operators/groupBy.ts b/packages/db-ivm/src/operators/groupBy.ts index 71818dc955..69c293a82b 100644 --- a/packages/db-ivm/src/operators/groupBy.ts +++ b/packages/db-ivm/src/operators/groupBy.ts @@ -5,10 +5,6 @@ import { reduce } from './reduce.js' import type { IStreamBuilder, KeyValue } from '../types.js' type GroupKey = Record -type ProjectedGroupInput = KeyValue< - string, - [string | null, Record] -> type BasicAggregateFunction = { preMap: (data: T) => V @@ -45,14 +41,7 @@ export function groupBy< T, K extends GroupKey, A extends Record>, ->( - keyExtractor: (data: T) => K, - aggregates: A = {} as A, - mapInput?: ( - stream: IStreamBuilder, - project: (data: T) => ProjectedGroupInput, - ) => IStreamBuilder, -) { +>(keyExtractor: (data: T) => K, aggregates: A = {} as A) { type ResultType = K & AggregatesReturnType const basicAggregates = Object.fromEntries( @@ -82,33 +71,35 @@ export function groupBy< // values when every one of them has one. Within one group the reduce // index then merges equal contributions by that identity instead of // hashing them. A null identity falls back to the structural hash. - const project = (data: T): ProjectedGroupInput => { - const key = keyExtractor(data) - const keyString = serializeValue(key) - - // Create values object with pre-aggregated values - const values: Record = {} - - // Store the original key object. Contributions whose keys serialize - // alike but hash apart must not merge, so the key is part of the - // identity. - values[KEY_SENTINEL] = key - - // Add pre-aggregated values - let identity: string | null = structuralIdentity(key) ?? null - for (const [name, aggregate] of aggregateEntries) { - const value = aggregate.preMap(data) - values[name] = value - if (identity === null) continue - const part = structuralIdentity(value) - identity = part === undefined ? null : `${identity}|${part}` - } + const withKeysAndValues = stream.pipe( + map((data) => { + const key = keyExtractor(data) + const keyString = serializeValue(key) + + // Create values object with pre-aggregated values + const values: Record = {} + + // Store the original key object. Contributions whose keys serialize + // alike but hash apart must not merge, so the key is part of the + // identity. + values[KEY_SENTINEL] = key - return [keyString, [identity, values]] - } - const withKeysAndValues = mapInput - ? mapInput(stream, project) - : stream.pipe(map(project)) + // Add pre-aggregated values + let identity: string | null = structuralIdentity(key) ?? null + for (const [name, aggregate] of aggregateEntries) { + const value = aggregate.preMap(data) + values[name] = value + if (identity === null) continue + const part = structuralIdentity(value) + identity = part === undefined ? null : `${identity}|${part}` + } + + return [keyString, [identity, values]] as KeyValue< + string, + [string | null, Record] + > + }), + ) // Then reduce to compute aggregates const reduced = withKeysAndValues.pipe( diff --git a/packages/db/package.json b/packages/db/package.json index e05430595f..81dc7f5217 100644 --- a/packages/db/package.json +++ b/packages/db/package.json @@ -23,7 +23,7 @@ "test": "vitest --run", "test:dist": "vitest run --config vitest.dist.config.ts", "test:facade-retention": "node --expose-gc --import tsx tests/facade-retention.probe.ts", - "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/temporal-index-order-oracle.test.ts tests/temporal-equality-alias-oracle.test.ts tests/custom-index-order-selection-oracle.test.ts tests/query/custom-index-window-selection-oracle.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/captured-inarray-stages-oracle.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", + "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", "bench:nested-includes": "vitest bench tests/query/includes-performance.bench.ts --run", "test:indexed-db:package": "pnpm --filter @tanstack/db... build && vitest run --config vitest.indexed-db.package.config.ts", "test:indexed-db:browser": "playwright test --config playwright.indexed-db.config.ts", diff --git a/packages/db/src/collection/change-events.ts b/packages/db/src/collection/change-events.ts index 365288869e..f69ab13d43 100644 --- a/packages/db/src/collection/change-events.ts +++ b/packages/db/src/collection/change-events.ts @@ -1,4 +1,3 @@ -import { compareKeys } from '@tanstack/db-ivm' import { compileSingleRowExpression, toBooleanPredicate, @@ -17,7 +16,6 @@ import { import { makeComparator } from '../utils/comparison.js' import { buildCompareOptions } from '../query/compiler/order-by' import { codedMessage, codedWarning, devBuild } from '../error-message.js' -import type { IndexReader } from '../indexes/base-index.js' import type { ChangeMessage, CollectionLike, @@ -25,7 +23,7 @@ import type { SubscribeChangesOptions, } from '../types' import type { CollectionImpl } from './index.js' -import type { BasicExpression, OrderBy, OrderByClause } from '../query/ir.js' +import type { BasicExpression, OrderBy } from '../query/ir.js' import type { WithVirtualProps } from '../virtual-props.js' /** @@ -353,17 +351,12 @@ function getOrderedKeys( fieldPath, collection as CollectionImpl, compareOpts, - undefined, - true, ) // Find the index - const index = findIndexForField(collection, fieldPath, compareOpts, true) + const index = findIndexForField(collection, fieldPath, compareOpts) - // A user-supplied index comparator is not described by compareOptions. - // Exact equality lookup may still use that index, but its ordered walk - // cannot stand in for the requested order. - if (index && index.supports(`gt`) && index.supportsRangeOptimization) { + if (index && index.supports(`gt`)) { // Use index optimization const filterFn = (key: TKey): boolean => { const value = collection.get(key) @@ -373,7 +366,10 @@ function getOrderedKeys( return whereFilter?.(value) ?? true } - return takeOrderedIndexKeys(collection, index, clause, limit, filterFn) + // Take the keys that match the filter and limit + // if no limit is provided `index.keyCount` is used, + // i.e. we will take all keys that match the filter + return index.takeFromStart(limit ?? index.keyCount, filterFn) } } } @@ -396,13 +392,18 @@ function getOrderedKeys( })) const compare = (a: { key: TKey; value: T }, b: { key: TKey; value: T }) => { for (const clause of clauses) { + // Extract values for comparison const aValue = extractValueFromItem(a.value, clause.expression) const bValue = extractValueFromItem(b.value, clause.expression) + const result = clause.compare(aValue, bValue) - if (result !== 0) return result + if (result !== 0) { + return result + } } - return compareKeys(a.key, b.key) + return 0 } + allItems.sort(compare) const sortedKeys = allItems.map((item) => item.key) @@ -415,61 +416,6 @@ function getOrderedKeys( return sortedKeys } -/** - * Takes an indexed ordered prefix, completing its comparator-equal boundary. - * Public snapshots keep source keys ascending within ties even when a reversed - * index walks them backward. A finite read asks for an indexed prefix and - * completes its boundary group; it does not bound work inside the index. - * An unbounded read asks for all indexed keys. `from` marks an exclusive - * comparator position, including an explicit nullish value. - */ -export function takeOrderedIndexKeys< - T extends object, - TKey extends string | number, ->( - collection: CollectionLike, - index: IndexReader, - clause: OrderByClause, - limit: number | undefined, - filterFn: (key: TKey) => boolean, - from?: { value: unknown }, -): Array { - if (limit !== undefined && limit <= 0) return [] - const count = limit ?? index.keyCount - const taken = from - ? index.take(count, from.value, filterFn) - : index.takeFromStart(count, filterFn) - if (taken.length === 0) return taken - - const valueFor = (key: TKey) => - extractValueFromItem(collection.get(key)!, clause.expression) - let keys = taken - if (limit !== undefined && taken.length === limit) { - const boundaryValue = valueFor(taken[taken.length - 1]!) - const boundaryKeys = [ - ...index.rangeQuery({ from: boundaryValue, to: boundaryValue }), - ].filter(filterFn) - keys = [...new Set([...taken, ...boundaryKeys])] - } - - // The index already orders distinct comparator positions. Sort only the - // source keys inside each equal position, including a completed boundary. - const values = keys.map(valueFor) - const compare = makeComparator(buildCompareOptions(clause, collection)) - const ordered: Array = [] - for (let start = 0; start < keys.length;) { - let end = start + 1 - while (end < keys.length && compare(values[start], values[end]) === 0) { - end++ - } - for (const key of keys.slice(start, end).sort(compareKeys)) { - ordered.push(key) - } - start = end - } - return limit === undefined ? ordered : ordered.slice(0, limit) -} - /** * Helper function to extract a value from an item based on an expression */ diff --git a/packages/db/src/collection/state.ts b/packages/db/src/collection/state.ts index 74721a2fe9..9f47f0261a 100644 --- a/packages/db/src/collection/state.ts +++ b/packages/db/src/collection/state.ts @@ -1129,11 +1129,7 @@ export class CollectionStateManager< // Application is now the point of no return. Event listeners run before // the receipts resolve, so a signal aborted from one of those listeners // must not cancel writes that are already becoming visible. - // A layout-only update can change the comparator's fractional index - // before these writes run. Rebuild after the batch rather than searching - // the old order with a comparator that already sees new positions. const deferOrder = - layoutChanged || committedSyncedTransactions.reduce( (count, transaction) => count + transaction.operations.length, 0, @@ -1184,7 +1180,6 @@ export class CollectionStateManager< } const events: Array> = [] - const indexOnlyChanges: Array> = [] let reappliedKeys: ReadonlySet | undefined if (hasTruncateSync) { // All queued transactions publish as one batch. Its clear prefix must @@ -1404,18 +1399,6 @@ export class CollectionStateManager< value: newVisibleValue, previousValue: withPreviousVirtualProps(previousVisibleValue), }) - } else if ( - previousVisibleValue !== newVisibleValue && - this.indexes.indexes.size > 0 - ) { - // Change-event equality can group rows whose raw index order differs. - // Refresh their index positions without publishing a value change. - indexOnlyChanges.push({ - type: `update`, - key, - value: newVisibleValue, - previousValue: withPreviousVirtualProps(previousVisibleValue), - }) } } @@ -1423,8 +1406,8 @@ export class CollectionStateManager< this.size = this.calculateSize() // Update indexes for all events before emitting - if (events.length > 0 || indexOnlyChanges.length > 0) { - this.indexes.updateIndexes([...events, ...indexOnlyChanges]) + if (events.length > 0) { + this.indexes.updateIndexes(events) } // End batching and emit all events (combines any batched events with sync events) diff --git a/packages/db/src/collection/subscription.ts b/packages/db/src/collection/subscription.ts index 9ec70b3661..76fbbad68a 100644 --- a/packages/db/src/collection/subscription.ts +++ b/packages/db/src/collection/subscription.ts @@ -1,7 +1,6 @@ -import { compareKeys } from '@tanstack/db-ivm' import { ensureIndexForExpression } from '../indexes/auto-index.js' -import { and } from '../query/builder/functions.js' -import { PropRef, getPropRefPropertyPath } from '../query/ir.js' +import { and, eq } from '../query/builder/functions.js' +import { PropRef, Value, getPropRefPropertyPath } from '../query/ir.js' import { EventEmitter } from '../event-emitter.js' import { compileExpression } from '../query/compiler/evaluators.js' import { buildCursor, buildCursorCurrent } from '../utils/cursor.js' @@ -14,7 +13,6 @@ import { codedMessage, devBuild } from '../error-message.js' import { createFilterFunctionFromExpression, createFilteredCallback, - takeOrderedIndexKeys, } from './change-events.js' import type { BasicExpression, OrderBy } from '../query/ir.js' import type { IndexReader } from '../indexes/base-index.js' @@ -1506,36 +1504,34 @@ export class CollectionSubscription // so we load all rows equal to minValue first, to be sure we don't skip any duplicate values let keys: Array = [] if (hasMinValue) { - // A cursor names a comparator position, which can contain several - // unequal values (or both null and undefined). Keep that whole group - // before walking later positions, with public keys ascending. - keys = [ - ...index.rangeQuery({ - from: minValueForIndex, - to: minValueForIndex, - }), - ] - .filter(filterFn) - .sort(compareKeys) - keys.push( - ...takeOrderedIndexKeys( - this.collection, - index, - orderBy[0]!, + // First, get all items with the same FIRST COLUMN value as minValue + // This provides wide bounds for the local index + const { expression } = orderBy[0]! + const allRowsWithMinValue = this.collection.currentStateAsChanges({ + where: eq(expression, new Value(minValueForIndex)), + }) + + if (allRowsWithMinValue) { + const keysWithMinValue = allRowsWithMinValue + .map((change) => change.key) + .filter(filterFn) + + // Add items with the minValue first + keys.push(...keysWithMinValue) + + // Then get items greater than minValue + const keysGreaterThanMin = index.take( limit - keys.length, + minValueForIndex!, filterFn, - { value: minValueForIndex }, - ), - ) + ) + keys.push(...keysGreaterThanMin) + } else { + keys = index.take(limit, minValueForIndex!, filterFn) + } } else { // No min value provided, start from the beginning - keys = takeOrderedIndexKeys( - this.collection, - index, - orderBy[0]!, - limit, - filterFn, - ) + keys = index.takeFromStart(limit, filterFn) } const valuesNeeded = () => Math.max(limit - changes.length, 0) @@ -1564,14 +1560,7 @@ export class CollectionSubscription biggestObservedValue = valueExtractor ? valueExtractor(value) : value } - keys = takeOrderedIndexKeys( - this.collection, - index, - orderBy[0]!, - valuesNeeded(), - filterFn, - { value: biggestObservedValue }, - ) + keys = index.take(valuesNeeded(), biggestObservedValue!, filterFn) } // Reserve rows before publication. A direct truncate replay keeps them in diff --git a/packages/db/src/indexes/auto-index.ts b/packages/db/src/indexes/auto-index.ts index b96b067b2f..c72c82807a 100644 --- a/packages/db/src/indexes/auto-index.ts +++ b/packages/db/src/indexes/auto-index.ts @@ -3,7 +3,7 @@ import { hasVirtualPropPath } from '../virtual-props' import { codedWarning, devBuild } from '../error-message.js' import { checkCollectionSizeForIndex, isDevModeEnabled } from './index-registry' import type { CompareOptions } from '../query/builder/types' -import type { BasicExpression, PropRef } from '../query/ir' +import type { BasicExpression } from '../query/ir' import type { CollectionImpl } from '../collection/index.js' function shouldAutoIndex(collection: CollectionImpl) { @@ -22,7 +22,6 @@ export function ensureIndexForField< collection: CollectionImpl, compareOptions?: CompareOptions, compareFn?: (a: any, b: any) => number, - requireOrderedTraversal = false, ) { if (hasVirtualPropPath(fieldPath)) { return @@ -37,13 +36,10 @@ export function ensureIndexForField< ...collection.compareOptions, } - // An ordered read needs an index whose comparator matches its options; - // an exact predicate can still use a custom index's equality buckets. + // Check if we already have an index for this field const existingIndex = Array.from(collection.indexes.values()).find( (index) => - index.matchesField(fieldPath) && - index.matchesCompareOptions(compareOpts) && - (!requireOrderedTraversal || index.supportsRangeOptimization), + index.matchesField(fieldPath) && index.matchesCompareOptions(compareOpts), ) if (existingIndex) { @@ -98,42 +94,29 @@ export function ensureIndexForExpression< const indexableExpressions = extractIndexableExpressions(expression) - for (const { - fieldName, - fieldPath, - requireOrderedTraversal, - } of indexableExpressions) { - ensureIndexForField( - fieldName, - fieldPath, - collection, - undefined, - undefined, - requireOrderedTraversal, - ) + for (const { fieldName, fieldPath } of indexableExpressions) { + ensureIndexForField(fieldName, fieldPath, collection) } } -type IndexableExpression = { - fieldName: string - fieldPath: Array - requireOrderedTraversal: boolean -} - -/** Extracts indexable predicates and whether they need relational order. */ +/** + * Extracts all indexable expressions from a where expression + */ function extractIndexableExpressions( expression: BasicExpression, -): Array { - const results: Array = [] +): Array<{ fieldName: string; fieldPath: Array }> { + const results: Array<{ fieldName: string; fieldPath: Array }> = [] function extractFromExpression(expr: BasicExpression): void { if (expr.type !== `func`) { return } + const func = expr as any + // Handle 'and' expressions by recursively processing all arguments - if (expr.name === `and`) { - for (const arg of expr.args) { + if (func.name === `and`) { + for (const arg of func.args) { extractFromExpression(arg) } return @@ -141,26 +124,16 @@ function extractIndexableExpressions( // Check if this is a supported operation const supportedOperations = [`eq`, `gt`, `gte`, `lt`, `lte`, `in`] - if (!supportedOperations.includes(expr.name)) { + if (!supportedOperations.includes(func.name)) { return } - // Preserve field-first admission for existing expressions. A scalar on - // the left of a comparison can also put the indexable field on the right. - // IN keeps its field-first shape. - let fieldRef: PropRef | undefined - if (expr.args[0]?.type === `ref`) { - fieldRef = expr.args[0] - } else if (expr.name !== `in` && expr.args.length === 2) { - const [left, right] = expr.args - if (left?.type === `val` && right?.type === `ref`) { - fieldRef = right - } - } - if (!fieldRef) { + // Check if the first argument is a property reference + if (func.args.length < 1 || func.args[0].type !== `ref`) { return } + const fieldRef = func.args[0] const fieldPath = fieldRef.path // Skip if the path is empty @@ -171,11 +144,7 @@ function extractIndexableExpressions( // For nested paths, use the full path joined with underscores as the field name // For simple paths, use the first (and only) element const fieldName = fieldPath.join(`_`) - results.push({ - fieldName, - fieldPath, - requireOrderedTraversal: expr.name !== `eq` && expr.name !== `in`, - }) + results.push({ fieldName, fieldPath }) } extractFromExpression(expression) diff --git a/packages/db/src/indexes/base-index.ts b/packages/db/src/indexes/base-index.ts index 967dae74e9..b8f3a59bb4 100644 --- a/packages/db/src/indexes/base-index.ts +++ b/packages/db/src/indexes/base-index.ts @@ -1,14 +1,7 @@ import { compileSingleRowExpression } from '../query/compiler/evaluators.js' import { comparisonFunctions } from '../query/builder/functions.js' import { DEFAULT_COMPARE_OPTIONS, deepEquals } from '../utils.js' -import { - dateTimestamp, - indexCustomComparatorValue, - indexOrderValue, - indexTemporalKind, - makeCheckedComparator, - makeIndexComparator, -} from '../utils/comparison.js' +import { makeCheckedComparator, makeComparator } from '../utils/comparison.js' import { codedMessage, codedWarning, devBuild } from '../error-message.js' import type { CompiledSingleRowExpression } from '../query/compiler/evaluators.js' import type { RangeQueryOptions } from './btree-index.js' @@ -103,10 +96,8 @@ export interface IndexInterface< * Whether range lookups (gt/gte/lt/lte) on this index can be trusted to * return every matching key. Range traversal relies on the index ordering, so * it is unsafe when the index uses a custom comparator, whose order may not - * match the WHERE evaluator's relational operators. A custom comparator is - * also not represented by compareOptions, so an ordered Collection read must - * not select this index by compareOptions alone. Callers must fall back to a - * full scan when this is `false`. + * match the WHERE evaluator's relational operators. Callers must fall back to + * a full scan when this is `false`. */ get supportsRangeOptimization(): boolean @@ -156,15 +147,8 @@ export abstract class BaseIndex< this.expression = expression this.compareOptions = options?.compareOptions ?? DEFAULT_COMPARE_OPTIONS this.hasCustomComparator = options?.compareFn != null - const compareFn = options?.compareFn this.compareFn = makeCheckedComparator( - compareFn == null - ? makeIndexComparator(this.compareOptions) - : (a, b) => - compareFn( - indexCustomComparatorValue(a), - indexCustomComparatorValue(b), - ), + options?.compareFn ?? makeComparator(this.compareOptions), ) this.name = name this.initialize(options) @@ -220,26 +204,7 @@ export abstract class BaseIndex< } get supportsRangeOptimization(): boolean { - if (this.hasCustomComparator) return false - // The default index can retain these genuine values for exact lookup, - // but its internal total order does not stand in for public `orderBy`. - let temporalKind: string | undefined - for (const domain of this.rangeValueDomains.keys()) { - if (!domain.startsWith(`Temporal.`)) continue - if ( - domain === `Temporal.Duration` || - domain === `Temporal.PlainMonthDay` || - (temporalKind !== undefined && temporalKind !== domain) - ) - return false - temporalKind = domain - } - return true - } - - /** Ordering sees raw values; exact lookup keeps its normalized key. */ - protected orderValue(value: unknown, exactKey: unknown): unknown { - return indexOrderValue(value, exactKey) + return !this.hasCustomComparator } protected addRangeValue(value: unknown): void { @@ -355,8 +320,8 @@ export abstract class BaseIndex< function rangeValueDomain(value: unknown): string | undefined { if (value == null) return undefined - if (dateTimestamp(value) !== undefined) return `date` - return indexTemporalKind(value) ?? typeof value + if (value instanceof Date) return `date` + return typeof value } function isNativeRangeDomain(domain: string): boolean { diff --git a/packages/db/src/indexes/basic-index.ts b/packages/db/src/indexes/basic-index.ts index 7182ad698e..0da1e1ffe3 100644 --- a/packages/db/src/indexes/basic-index.ts +++ b/packages/db/src/indexes/basic-index.ts @@ -55,13 +55,10 @@ export class BasicIndex< // Map for O(1) equality lookups: indexedValue -> Set of PKs private valueMap = new Map>() - // Equality and comparator positions are independent: a custom comparator - // may distinguish values that have one exact query-equality key. + // Sorted array of unique indexed values for range queries private sortedValues: Array = [] - // Each bucket stays at the same array position as its order representative. - private orderedBuckets: Array> = [] - // Each indexed key retains the order value used for its current position. - private indexedKeys = new Map() + // Set of all indexed PKs + private indexedKeys = new Set() protected initialize(_options?: BasicIndexOptions): void {} @@ -71,42 +68,28 @@ export class BasicIndex< add(key: TKey, item: any): void { const indexedValue = this.evaluateAddedValue(key, item) const normalizedValue = normalizeValue(indexedValue) - const orderValue = this.orderValue(indexedValue, normalizedValue) - this.addToBucket(key, normalizedValue, orderValue) + this.addToBucket(key, normalizedValue) this.addRangeValue(indexedValue) - this.indexedKeys.set(key, orderValue) + this.indexedKeys.add(key) } - private addToBucket( - key: TKey, - normalizedValue: unknown, - orderValue: unknown, - ): void { - this.addToOrder(key, orderValue) - const exactKeys = this.valueMap.get(normalizedValue) - if (exactKeys) exactKeys.add(key) - else this.valueMap.set(normalizedValue, new Set([key])) - } - - private addToOrder(key: TKey, orderValue: unknown): void { - // Compare before changing either partition. One comparator position can - // contain several exact values, and one exact value can occupy several - // custom comparator positions. - const insertIdx = findInsertPositionInArray( - this.sortedValues, - orderValue, - this.compareFn, - ) - if ( - insertIdx < this.sortedValues.length && - this.compareFn(this.sortedValues[insertIdx], orderValue) === 0 - ) { - this.orderedBuckets[insertIdx]!.set(key, orderValue) + private addToBucket(key: TKey, normalizedValue: unknown): void { + const keySet = this.valueMap.get(normalizedValue) + if (keySet) { + // Value already exists, just add the key to the set + keySet.add(key) } else { - this.sortedValues.splice(insertIdx, 0, orderValue) - this.orderedBuckets.splice(insertIdx, 0, new Map([[key, orderValue]])) + // New value. Find its sorted position first: a comparator failure must + // throw before the index changes. + const insertIdx = findInsertPositionInArray( + this.sortedValues, + normalizedValue, + this.compareFn, + ) + this.valueMap.set(normalizedValue, new Set([key])) + this.sortedValues.splice(insertIdx, 0, normalizedValue) } } @@ -125,53 +108,43 @@ export class BasicIndex< const normalizedValue = normalizeValue(indexedValue) - this.removeFromBucket(key, normalizedValue, this.indexedKeys.get(key)) + this.removeFromBucket(key, normalizedValue) this.removeRangeValue(indexedValue) this.indexedKeys.delete(key) } - private removeFromBucket( - key: TKey, - normalizedValue: unknown, - orderValue: unknown, - ): void { - const exactKeys = this.valueMap.get(normalizedValue) - if (!exactKeys?.has(key)) return - this.removeFromOrder(key, orderValue) - exactKeys.delete(key) - if (exactKeys.size === 0) this.valueMap.delete(normalizedValue) - } - - private removeFromOrder(key: TKey, orderValue: unknown): void { - const index = findInsertPositionInArray( - this.sortedValues, - orderValue, - this.compareFn, - ) - if (index >= this.sortedValues.length) return - const representative = this.sortedValues[index] - if (this.compareFn(representative, orderValue) !== 0) return - const members = this.orderedBuckets[index]! - if (!members.has(key)) return - if (members.size === 1) { - this.sortedValues.splice(index, 1) - this.orderedBuckets.splice(index, 1) - return - } - // Keep a live representative when its row leaves the comparator group. - if (areSameValueZeroEqual(members.get(key), representative)) { - const values = members.entries() - const first = values.next().value! - const successor = first[0] === key ? values.next().value![1] : first[1] - // A custom comparator may reject the promoted value against a neighbor. - // Check before changing either the sorted or exact partition. - if (index > 0) this.compareFn(this.sortedValues[index - 1], successor) - if (index + 1 < this.sortedValues.length) - this.compareFn(successor, this.sortedValues[index + 1]) - this.sortedValues[index] = successor + private removeFromBucket(key: TKey, normalizedValue: unknown): void { + const keySet = this.valueMap.get(normalizedValue) + if (keySet?.has(key)) { + // The last key for this value leaves the sorted array too. Locate it + // before any change, so a comparator failure leaves the index unchanged. + if (keySet.size === 1) { + let sortedIndex = findInsertPositionInArray( + this.sortedValues, + normalizedValue, + this.compareFn, + ) + // Distinct equality keys may share one comparator position. + while ( + sortedIndex < this.sortedValues.length && + this.compareFn(this.sortedValues[sortedIndex], normalizedValue) === 0 + ) { + if ( + areSameValueZeroEqual( + this.sortedValues[sortedIndex], + normalizedValue, + ) + ) { + this.sortedValues.splice(sortedIndex, 1) + break + } + sortedIndex++ + } + this.valueMap.delete(normalizedValue) + } + keySet.delete(key) } - members.delete(key) } /** @@ -191,28 +164,21 @@ export class BasicIndex< const oldValue = normalizeValue(oldIndexedValue) const newValue = normalizeValue(newIndexedValue) - const oldOrder = this.indexedKeys.get(key) - const newOrder = this.orderValue(newIndexedValue, newValue) if ( areSameValueZeroEqual(oldValue, newValue) && this.valueMap.get(newValue)?.has(key) && this.indexedKeys.has(key) ) { - if (!Object.is(oldOrder, newOrder)) { - this.removeFromOrder(key, oldOrder) - this.addToOrder(key, newOrder) - } this.removeRangeValue(oldIndexedValue) this.addRangeValue(newIndexedValue) - this.indexedKeys.set(key, newOrder) return } - this.removeFromBucket(key, oldValue, oldOrder) + this.removeFromBucket(key, oldValue) this.removeRangeValue(oldIndexedValue) - this.addToBucket(key, newValue, newOrder) + this.addToBucket(key, newValue) this.addRangeValue(newIndexedValue) - this.indexedKeys.set(key, newOrder) + this.indexedKeys.add(key) } /** @@ -221,43 +187,21 @@ export class BasicIndex< build(entries: Iterable<[TKey, any]>): void { this.clear() - // Collect all entries first - const entriesArray: Array<{ key: TKey; value: any; orderValue: any }> = [] for (const [key, item] of entries) { const indexedValue = this.evaluateAddedValue(key, item) const value = normalizeValue(indexedValue) - const orderValue = this.orderValue(indexedValue, value) - entriesArray.push({ - key, - value, - orderValue, - }) + const keys = this.valueMap.get(value) + if (keys) keys.add(key) + else this.valueMap.set(value, new Set([key])) this.addRangeValue(indexedValue) - this.indexedKeys.set(key, orderValue) + this.indexedKeys.add(key) } - // Native Array.sort moves raw undefined values last without comparing them. - // Wrap values so nulls-first comparison also runs during a full build. - const sortable = entriesArray.map((entry) => ({ entry })) - sortable.sort((left, right) => - this.compareFn(left.entry.orderValue, right.entry.orderValue), - ) - for (const { entry } of sortable) { - const exactKeys = this.valueMap.get(entry.value) - if (exactKeys) exactKeys.add(entry.key) - else this.valueMap.set(entry.value, new Set([entry.key])) - - const representative = this.sortedValues.at(-1) - if ( - this.sortedValues.length > 0 && - this.compareFn(representative, entry.orderValue) === 0 - ) { - this.orderedBuckets.at(-1)!.set(entry.key, entry.orderValue) - } else { - this.sortedValues.push(entry.orderValue) - this.orderedBuckets.push(new Map([[entry.key, entry.orderValue]])) - } - } + // Array.sort moves raw undefined values last without comparing them. + // Wrap values so nulls-first comparison runs during a full build. + const sortable = Array.from(this.valueMap.keys(), (value) => ({ value })) + sortable.sort((left, right) => this.compareFn(left.value, right.value)) + this.sortedValues = sortable.map(({ value }) => value) } /** @@ -266,7 +210,6 @@ export class BasicIndex< clear(): void { this.valueMap.clear() this.sortedValues = [] - this.orderedBuckets = [] this.indexedKeys.clear() this.clearRangeValues() } @@ -334,8 +277,6 @@ export class BasicIndex< const normalizedFrom = normalizeValue(from) const normalizedTo = normalizeValue(to) - const orderFrom = this.orderValue(from, normalizedFrom) - const orderTo = this.orderValue(to, normalizedTo) const hasFrom = `from` in options const hasTo = `to` in options @@ -344,7 +285,7 @@ export class BasicIndex< if (hasFrom) { startIdx = findInsertPositionInArray( this.sortedValues, - orderFrom, + normalizedFrom, this.compareFn, ) // Comparator-equal values form one range boundary even when they are @@ -352,7 +293,7 @@ export class BasicIndex< while ( !fromInclusive && startIdx < this.sortedValues.length && - this.compareFn(this.sortedValues[startIdx], orderFrom) === 0 + this.compareFn(this.sortedValues[startIdx], normalizedFrom) === 0 ) { startIdx++ } @@ -363,14 +304,14 @@ export class BasicIndex< if (hasTo) { endIdx = findInsertPositionInArray( this.sortedValues, - orderTo, + normalizedTo, this.compareFn, ) // Include the whole comparator group at an inclusive upper boundary. while ( toInclusive && endIdx < this.sortedValues.length && - this.compareFn(this.sortedValues[endIdx], orderTo) === 0 + this.compareFn(this.sortedValues[endIdx], normalizedTo) === 0 ) { endIdx++ } @@ -378,8 +319,9 @@ export class BasicIndex< // Collect all keys in range for (let i = startIdx; i < endIdx; i++) { - for (const key of this.orderedBuckets[i]!.keys()) { - result.add(key) + const keys = this.valueMap.get(this.sortedValues[i]) + if (keys) { + keys.forEach((key) => result.add(key)) } } @@ -391,16 +333,15 @@ export class BasicIndex< */ take(n: number, from: any, filterFn?: (key: TKey) => boolean): Array { const normalizedFrom = normalizeValue(from) - const orderFrom = this.orderValue(from, normalizedFrom) let startIdx = findInsertPositionInArray( this.sortedValues, - orderFrom, + normalizedFrom, this.compareFn, ) // Skip past the 'from' value (exclusive) while ( startIdx < this.sortedValues.length && - this.compareFn(this.sortedValues[startIdx], orderFrom) <= 0 + this.compareFn(this.sortedValues[startIdx], normalizedFrom) <= 0 ) { startIdx++ } @@ -417,14 +358,16 @@ export class BasicIndex< filterFn?: (key: TKey) => boolean, ): Array { const normalizedFrom = normalizeValue(from) - const orderFrom = this.orderValue(from, normalizedFrom) let startIdx = - findInsertPositionInArray(this.sortedValues, orderFrom, this.compareFn) - - 1 + findInsertPositionInArray( + this.sortedValues, + normalizedFrom, + this.compareFn, + ) - 1 // Skip past the 'from' value (exclusive) while ( startIdx >= 0 && - this.compareFn(this.sortedValues[startIdx], orderFrom) >= 0 + this.compareFn(this.sortedValues[startIdx], normalizedFrom) >= 0 ) { startIdx-- } @@ -465,7 +408,7 @@ export class BasicIndex< const groupValue = this.sortedValues[index] const groupKeys: Array = [] do { - for (const key of this.orderedBuckets[index]!.keys()) { + for (const key of this.valueMap.get(this.sortedValues[index]) ?? []) { groupKeys.push(key) } index += step diff --git a/packages/db/src/indexes/btree-index.ts b/packages/db/src/indexes/btree-index.ts index 33cffb5ceb..2266c18a1f 100644 --- a/packages/db/src/indexes/btree-index.ts +++ b/packages/db/src/indexes/btree-index.ts @@ -4,7 +4,6 @@ import { BTree } from '../utils/btree.js' import { areSameValueZeroEqual, denormalizeUndefined, - indexCustomComparatorValue, normalizeForBTree, } from '../utils/comparison.js' import { codedMessage, devBuild } from '../error-message.js' @@ -36,14 +35,10 @@ export interface RangeQueryOptions { type OrderedBucket = { representative: unknown - // One comparator position may contain several exact values, while one - // exact value may occupy several custom comparator positions. - values: Map + exactValues: Set + keys: Set } -// Map uses SameValueZero, but a custom callback may order -0 apart from +0. -const NEGATIVE_ZERO_ORDER_OPERAND = Symbol(`negative zero order operand`) - /** * B+Tree index for sorted data with range queries * This maintains items in sorted order and provides efficient range operations @@ -66,14 +61,11 @@ export class BTreeIndex< private orderedEntries = new BTree>((a, b) => this.compareStored(a, b), ) - private valueMap = new Map>() - // Repeated identical callback operands belong to the same comparator - // position. This avoids another tree search for every row in a hot bucket. - private orderOperands = new Map< + private valueMap = new Map< unknown, - { bucket: OrderedBucket; owners: number } + { keys: Set; ordered: OrderedBucket } >() - private indexedKeys = new Map() + private indexedKeys = new Set() // Stored keys use UNDEFINED_SENTINEL; the comparator, which may be // user-supplied, expects undefined. @@ -81,13 +73,6 @@ export class BTreeIndex< return this.compareFn(denormalizeUndefined(a), denormalizeUndefined(b)) } - private orderOperand(value: unknown): unknown { - const operand = this.hasCustomComparator - ? indexCustomComparatorValue(value) - : value - return Object.is(operand, -0) ? NEGATIVE_ZERO_ORDER_OPERAND : operand - } - protected initialize(_options?: BTreeIndexOptions): void {} /** @@ -98,45 +83,37 @@ export class BTreeIndex< // Normalize the value for Map key usage const normalizedValue = normalizeForBTree(indexedValue) - const orderValue = this.orderValue(indexedValue, normalizedValue) - this.addToBucket(key, normalizedValue, orderValue) + this.addToBucket(key, normalizedValue) this.addRangeValue(indexedValue) - this.indexedKeys.set(key, orderValue) + this.indexedKeys.add(key) } - private addToBucket( - key: TKey, - normalizedValue: unknown, - orderValue: unknown, - ): void { - this.addToOrder(key, orderValue) + private addToBucket(key: TKey, normalizedValue: unknown): void { const exact = this.valueMap.get(normalizedValue) - if (exact) exact.add(key) - else this.valueMap.set(normalizedValue, new Set([key])) - } - - private addToOrder(key: TKey, orderValue: unknown): void { - const operand = this.orderOperand(orderValue) - const cached = this.orderOperands.get(operand) - if (cached) { - cached.bucket.values.set(key, orderValue) - cached.owners++ + if (exact) { + exact.keys.add(key) + exact.ordered.keys.add(key) return } - let orderedBucket = this.orderedEntries.get(orderValue) + let orderedBucket = this.orderedEntries.get(normalizedValue) if (orderedBucket) { - orderedBucket.values.set(key, orderValue) + orderedBucket.keys.add(key) + orderedBucket.exactValues.add(normalizedValue) } else { orderedBucket = { - representative: orderValue, - values: new Map([[key, orderValue]]), + representative: normalizedValue, + exactValues: new Set([normalizedValue]), + keys: new Set([key]), } - this.orderedEntries.set(orderValue, orderedBucket) + this.orderedEntries.set(normalizedValue, orderedBucket) } - this.orderOperands.set(operand, { bucket: orderedBucket, owners: 1 }) + this.valueMap.set(normalizedValue, { + keys: new Set([key]), + ordered: orderedBucket, + }) } /** @@ -154,51 +131,39 @@ export class BTreeIndex< // Normalize the value for Map key usage const normalizedValue = normalizeForBTree(indexedValue) - this.removeFromBucket(key, normalizedValue, this.indexedKeys.get(key)) + this.removeFromBucket(key, normalizedValue) this.removeRangeValue(indexedValue) this.indexedKeys.delete(key) } - private removeFromBucket( - key: TKey, - normalizedValue: unknown, - orderValue: unknown, - ): void { + private removeFromBucket(key: TKey, normalizedValue: unknown): void { const exact = this.valueMap.get(normalizedValue) - if (!exact?.has(key)) return - this.removeFromOrder(key, orderValue) - exact.delete(key) - if (exact.size === 0) this.valueMap.delete(normalizedValue) - } - - private removeFromOrder(key: TKey, orderValue: unknown): void { - const operand = this.orderOperand(orderValue) - const cached = this.orderOperands.get(operand) - const orderedBucket = cached?.bucket ?? this.orderedEntries.get(orderValue) - if (!orderedBucket?.values.has(key)) return + if (!exact?.keys.has(key)) return + const orderedBucket = exact.ordered + const removedExactValue = exact.keys.size === 1 const replacesRepresentative = - orderedBucket.values.size > 1 && - areSameValueZeroEqual( - orderedBucket.representative, - orderedBucket.values.get(key), - ) + removedExactValue && + orderedBucket.keys.size > 1 && + areSameValueZeroEqual(orderedBucket.representative, normalizedValue) // Tree writes compare first, so a comparator failure throws before the // index changes. A comparator-equal `set` overwrites the entry's key. - if (orderedBucket.values.size === 1) { - this.orderedEntries.delete(orderedBucket.representative) + if (orderedBucket.keys.size === 1) { + this.orderedEntries.delete(normalizedValue) } else if (replacesRepresentative) { - const values = orderedBucket.values.entries() - const first = values.next().value! - const representative = - first[0] === key ? values.next().value![1] : first[1] - if (!areSameValueZeroEqual(representative, orderedBucket.representative)) - this.orderedEntries.set(representative, orderedBucket) + const representative = [...orderedBucket.exactValues].find( + (value) => !areSameValueZeroEqual(value, normalizedValue), + ) + this.orderedEntries.set(representative, orderedBucket) orderedBucket.representative = representative } - orderedBucket.values.delete(key) - if (cached && --cached.owners === 0) this.orderOperands.delete(operand) + exact.keys.delete(key) + orderedBucket.keys.delete(key) + if (removedExactValue) { + this.valueMap.delete(normalizedValue) + orderedBucket.exactValues.delete(normalizedValue) + } } /** @@ -218,27 +183,20 @@ export class BTreeIndex< const oldValue = normalizeForBTree(oldIndexedValue) const newValue = normalizeForBTree(newIndexedValue) - const oldOrder = this.indexedKeys.get(key) - const newOrder = this.orderValue(newIndexedValue, newValue) if ( areSameValueZeroEqual(oldValue, newValue) && - this.valueMap.get(newValue)?.has(key) + this.valueMap.get(newValue)?.keys.has(key) ) { - if (!Object.is(oldOrder, newOrder)) { - this.removeFromOrder(key, oldOrder) - this.addToOrder(key, newOrder) - } this.removeRangeValue(oldIndexedValue) this.addRangeValue(newIndexedValue) - this.indexedKeys.set(key, newOrder) return } - this.removeFromBucket(key, oldValue, oldOrder) + this.removeFromBucket(key, oldValue) this.removeRangeValue(oldIndexedValue) - this.addToBucket(key, newValue, newOrder) + this.addToBucket(key, newValue) this.addRangeValue(newIndexedValue) - this.indexedKeys.set(key, newOrder) + this.indexedKeys.add(key) } /** @@ -258,7 +216,6 @@ export class BTreeIndex< clear(): void { this.orderedEntries.clear() this.valueMap.clear() - this.orderOperands.clear() this.indexedKeys.clear() this.clearRangeValues() } @@ -312,7 +269,7 @@ export class BTreeIndex< */ equalityLookup(value: any): Set { const normalizedValue = normalizeForBTree(value) - return new Set(this.valueMap.get(normalizedValue) ?? []) + return new Set(this.valueMap.get(normalizedValue)?.keys ?? []) } /** @@ -329,11 +286,9 @@ export class BTreeIndex< const hasTo = `to` in options const fromKey = hasFrom - ? this.orderValue(from, normalizeForBTree(from)) + ? normalizeForBTree(from) : this.orderedEntries.minKey() - const toKey = hasTo - ? this.orderValue(to, normalizeForBTree(to)) - : this.orderedEntries.maxKey() + const toKey = hasTo ? normalizeForBTree(to) : this.orderedEntries.maxKey() this.orderedEntries.forRange( fromKey, @@ -343,7 +298,8 @@ export class BTreeIndex< // Only exclude the boundary when an exclusive lower bound was // actually provided. Without a `from` bound, `fromKey` defaults to // the minimum key and must not be dropped. Compare against the - // stored order value; exact lookup uses a separate normalized key. + // normalized key since indexed values are stored normalized + // (e.g. dates as timestamps), so the raw `from` would never match. if ( hasFrom && !fromInclusive && @@ -354,7 +310,7 @@ export class BTreeIndex< return } - for (const key of bucket.values.keys()) result.add(key) + bucket.keys.forEach((key) => result.add(key)) }, ) @@ -365,7 +321,7 @@ export class BTreeIndex< * Internal method for taking items from the index. * @param n - The number of items to return * @param nextPair - Function to get the next pair from the BTree - * @param from - Stored order value; undefined means "start from beginning/end", sentinel means "start from the key undefined" + * @param from - Already normalized! undefined means "start from beginning/end", sentinel means "start from the key undefined" * @param filterFn - Optional filter function * @param reversed - Whether to reverse the order of keys within each value */ @@ -378,13 +334,13 @@ export class BTreeIndex< ): Array { const result: Array = [] let pair: [any, OrderedBucket] | undefined - let key = from + let key = from // Use as-is - it's already normalized by the caller // Every key owns exactly one bucket, so the walk never repeats a key. while ((pair = nextPair(key)) !== undefined && result.length < n) { key = pair[0] // Sort keys for deterministic order within a comparator position. - const sorted = Array.from(pair[1].values.keys()).sort( + const sorted = Array.from(pair[1].keys).sort( reversed ? compareKeysReversed : compareKeys, ) for (const ks of sorted) { @@ -404,8 +360,9 @@ export class BTreeIndex< */ take(n: number, from: any, filterFn?: (key: TKey) => boolean): Array { const nextPair = (k?: any) => this.orderedEntries.nextHigherPair(k) - const orderFrom = this.orderValue(from, normalizeForBTree(from)) - return this.takeInternal(n, nextPair, orderFrom, filterFn) + // Normalize the from value + const normalizedFrom = normalizeForBTree(from) + return this.takeInternal(n, nextPair, normalizedFrom, filterFn) } /** @@ -432,8 +389,9 @@ export class BTreeIndex< filterFn?: (key: TKey) => boolean, ): Array { const nextPair = (k?: any) => this.orderedEntries.nextLowerPair(k) - const orderFrom = this.orderValue(from, normalizeForBTree(from)) - return this.takeInternal(n, nextPair, orderFrom, filterFn, true) + // Normalize the from value + const normalizedFrom = normalizeForBTree(from) + return this.takeInternal(n, nextPair, normalizedFrom, filterFn, true) } /** @@ -459,7 +417,7 @@ export class BTreeIndex< for (const value of values) { const normalizedValue = normalizeForBTree(value) - const keys = this.valueMap.get(normalizedValue) + const keys = this.valueMap.get(normalizedValue)?.keys if (keys) { keys.forEach((key) => result.add(key)) } diff --git a/packages/db/src/query/builder/functions.ts b/packages/db/src/query/builder/functions.ts index d5490a0837..c2095a5cb6 100644 --- a/packages/db/src/query/builder/functions.ts +++ b/packages/db/src/query/builder/functions.ts @@ -42,7 +42,7 @@ type ExpressionLike = | Date | null | undefined - | ReadonlyArray + | Array export type CaseWhenValue = | ExpressionLike diff --git a/packages/db/src/query/compiler/evaluators.ts b/packages/db/src/query/compiler/evaluators.ts index 5ae4e1ec39..cf4d5df51c 100644 --- a/packages/db/src/query/compiler/evaluators.ts +++ b/packages/db/src/query/compiler/evaluators.ts @@ -17,7 +17,7 @@ import { getParentContextProjectedValue, } from '../equality-value-identity.js' import { getNamespacedRouteMetadata } from './route-metadata.js' -import type { Aggregate, BasicExpression, Func, PropRef } from '../ir.js' +import type { BasicExpression, Func, PropRef } from '../ir.js' import type { NamespacedRow } from '../../types.js' /** @@ -32,42 +32,6 @@ function normalizeEqualityOperand(value: unknown): unknown { return isUint8Array(value) ? value : normalizeValue(value) } -export function isStablePrimitiveList(list: ReadonlyArray): boolean { - if (!Object.isFrozen(list)) return false - for (let index = 0; index < list.length; index++) { - const slot = Object.getOwnPropertyDescriptor(list, index) - // Frozen arrays can still have getters or mutable objects. A missing slot - // can also resolve through the prototype, so none may be cached. - if (!slot || !(`value` in slot)) return false - const item = slot.value - if ( - (typeof item === `object` && item !== null) || - typeof item === `function` - ) - return false - } - return true -} - -export function hasMutableInList( - expression: BasicExpression | Aggregate, - lists?: Set>, -): boolean { - if (expression.type !== `func` && expression.type !== `agg`) return false - const list = expression.args[1] - const values = list?.type === `val` ? list.value : undefined - let found = - expression.type === `func` && - expression.name === `in` && - Array.isArray(values) && - !isStablePrimitiveList(values) - if (found) lists?.add(values as Array) - for (const arg of expression.args) { - found = hasMutableInList(arg, lists) || found - } - return found -} - /** * Equality that follows PostgreSQL float semantics for `NaN`/invalid Dates: * such values are equal to one another and unequal to anything else. For all @@ -481,22 +445,23 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { const valueEvaluator = compiledArgs[0]! const arrayEvaluator = compiledArgs[1]! const list = func.args[1]! - if ( - list.type === `val` && - Array.isArray(list.value) && - isStablePrimitiveList(list.value) - ) { - // An immutable primitive list, such as internally owned join demand, - // can be indexed once. Caller-owned arrays and mutable members must - // instead be read at each evaluation. SameValueZero on normalized - // keys preserves NaN and -0 equality. Byte values cannot match a - // primitive item and are never encoded here. + if (list.type === `val` && Array.isArray(list.value)) { + // A constant list, such as join demand's key list, becomes a Set of + // normalized keys. Set membership uses SameValueZero, which matches + // `valuesEqual` on normalized operands: NaN equals NaN and -0 equals + // 0. Byte arrays stay out of the Set and compare by content, so no + // byte array is encoded as a string. const items = list.value.filter((item) => !isUnknown(item)) - const keys = new Set(items.map(normalizeValue)) + const bytes = items.filter(isUint8Array) + const keys = new Set( + items.filter((item) => !isUint8Array(item)).map(normalizeValue), + ) return (data) => { const value = valueEvaluator(data) if (isUnknown(value)) return null - return isUint8Array(value) ? false : keys.has(normalizeValue(value)) + return isUint8Array(value) + ? bytes.some((item) => areValuesEqual(item, value)) + : keys.has(normalizeValue(value)) } } return (data) => { @@ -509,7 +474,7 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { if (!Array.isArray(array)) { return false } - return array.some((item: unknown) => + return array.some((item) => valuesEqual(normalizeEqualityOperand(item), value), ) } diff --git a/packages/db/src/query/compiler/group-by.ts b/packages/db/src/query/compiler/group-by.ts index cc08471a28..528dd30d17 100644 --- a/packages/db/src/query/compiler/group-by.ts +++ b/packages/db/src/query/compiler/group-by.ts @@ -27,11 +27,9 @@ import { } from '../equality-value-identity.js' import { compileExpression, - hasMutableInList, isCaseWhenConditionTrue, toBooleanPredicate, } from './evaluators.js' -import { refreshMap } from './mutable-list-refresh.js' import { INCLUDES_PUBLIC_KEY, attachRouteMetadata, @@ -39,7 +37,6 @@ import { stripInternalCallbackMetadata, } from './route-metadata.js' import type { ValueIdentity } from '../equality-value-identity.js' -import type { MutableListRevision } from './mutable-list-refresh.js' import type { RouteMetadata } from './route-metadata.js' import type { Aggregate, @@ -308,9 +305,6 @@ export function processGroupBy( aggregateCollectionId?: string, mainSource?: string, sanitizeCallbackRows = false, - mutableRevision?: MutableListRevision, - mutableSelect = false, - mutableAggregateInput = false, ): NamespacedAndKeyedStream { const fields = createInternalGroupFields(groupByClause.length, selectClause) const virtualAggregates: Record = { @@ -468,105 +462,92 @@ export function processGroupBy( } // Apply the groupBy operator - const refreshInputs = - mutableRevision !== undefined && - (mutableAggregateInput || - groupByClause.some((expr) => hasMutableInList(expr))) - pipeline = pipeline.pipe( - groupBy( - keyExtractor, - aggregates, - refreshInputs - ? (stream, project) => refreshMap(stream, mutableRevision, project) - : undefined, - ), - ) + pipeline = pipeline.pipe(groupBy(keyExtractor, aggregates)) // Update $selected to handle GROUP BY results - const projectGroup = ([, aggregatedRow]: [unknown, Record]) => { - const finalResults: Record = {} - - if (selectClause) { - // Every field takes its place in select order. Plain aggregates are - // filled first; every other value reads them, the group keys, and - // the route's parent context. - for (const [alias, expr] of Object.entries(selectClause)) { - finalResults[alias] = - expr.type === `agg` ? aggregatedRow[alias] : undefined + pipeline = pipeline.pipe( + map(([, aggregatedRow]) => { + const finalResults: Record = {} + + if (selectClause) { + // Every field takes its place in select order. Plain aggregates are + // filled first; every other value reads them, the group keys, and + // the route's parent context. + for (const [alias, expr] of Object.entries(selectClause)) { + finalResults[alias] = + expr.type === `agg` ? aggregatedRow[alias] : undefined + } + evaluateWrappedAggregates( + finalResults, + aggregatedRow as Record, + wrappedAggExprs, + fields, + ) + } else { + // No SELECT clause - just use the group keys + for (let i = 0; i < groupByClause.length; i++) { + finalResults[`__key_${i}`] = aggregatedRow[fields.groupValues[i]!] + } } - evaluateWrappedAggregates( - finalResults, - aggregatedRow, - wrappedAggExprs, - fields, - ) - } else { - // No SELECT clause - just use the group keys + + // Generate a simple key for the live collection using group values. + // In includes mode, add the complete route so correlated groups do not + // collide. + const route = mainSource + ? getGroupRoute(aggregatedRow, fields) + : undefined + const correlationKey = route?.correlationKey + const correlationRoute = mainSource + ? getCorrelationRouteIdentity(aggregatedRow, fields) + : undefined + const keyParts: Array = [] + const publicKeyParts: Array = [] for (let i = 0; i < groupByClause.length; i++) { - finalResults[`__key_${i}`] = aggregatedRow[fields.groupValues[i]!] + keyParts.push(aggregatedRow[fields.groupKeys[i]!]) + publicKeyParts.push(aggregatedRow[fields.groupValues[i]!]) } - } - - // Generate a simple key for the live collection using group values. - // In includes mode, add the complete route so correlated groups do not - // collide. - const route = mainSource ? getGroupRoute(aggregatedRow, fields) : undefined - const correlationKey = route?.correlationKey - const correlationRoute = mainSource - ? getCorrelationRouteIdentity(aggregatedRow, fields) - : undefined - const keyParts: Array = [] - const publicKeyParts: Array = [] - for (let i = 0; i < groupByClause.length; i++) { - keyParts.push(aggregatedRow[fields.groupKeys[i]!]) - publicKeyParts.push(aggregatedRow[fields.groupValues[i]!]) - } - if (correlationRoute !== undefined) { - keyParts.push(correlationRoute) - } - const finalKey = singleGroup - ? correlationRoute !== undefined - ? `single_group_${serializeValue(correlationRoute)}` - : `single_group` - : keyParts.length === 1 - ? keyParts[0] - : serializeValue(keyParts) - const publicKey = singleGroup - ? `single_group` - : createPublicGroupKey(publicKeyParts) - - // When in includes mode, restore route metadata for output routing. - const resultRow: Record = { - ...aggregatedRow, - $selected: finalResults, - } - const virtual = aggregatedRow[fields.virtual] as - RowVirtualMetadata | undefined - resultRow.$synced = virtual?.synced ?? true - resultRow.$hasPendingWrites = !resultRow.$synced - resultRow.$origin = ( - virtual?.hasLocal ? `local` : `remote` - ) satisfies VirtualOrigin - resultRow.$key = publicKey - resultRow.$collectionId = aggregateCollectionId ?? resultRow.$collectionId - if (mainSource && correlationKey !== undefined) { - attachPublicGroupKey(resultRow, publicKey) - attachRouteMetadata( - resultRow, - correlationKey, - route?.parentContext ?? null, - ) - } - return [mainSource ? finalKey : publicKey, resultRow] as [ - unknown, - Record, - ] - } - pipeline = ( - mutableRevision && mutableSelect - ? refreshMap(pipeline, mutableRevision, projectGroup) - : pipeline.pipe(map(projectGroup)) - ) as NamespacedAndKeyedStream + if (correlationRoute !== undefined) { + keyParts.push(correlationRoute) + } + const finalKey = singleGroup + ? correlationRoute !== undefined + ? `single_group_${serializeValue(correlationRoute)}` + : `single_group` + : keyParts.length === 1 + ? keyParts[0] + : serializeValue(keyParts) + const publicKey = singleGroup + ? `single_group` + : createPublicGroupKey(publicKeyParts) + + // When in includes mode, restore route metadata for output routing. + const resultRow: Record = { + ...(aggregatedRow as Record), + $selected: finalResults, + } + const virtual = (aggregatedRow as Record)[fields.virtual] as + RowVirtualMetadata | undefined + resultRow.$synced = virtual?.synced ?? true + resultRow.$hasPendingWrites = !resultRow.$synced + resultRow.$origin = ( + virtual?.hasLocal ? `local` : `remote` + ) satisfies VirtualOrigin + resultRow.$key = publicKey + resultRow.$collectionId = aggregateCollectionId ?? resultRow.$collectionId + if (mainSource && correlationKey !== undefined) { + attachPublicGroupKey(resultRow, publicKey) + attachRouteMetadata( + resultRow, + correlationKey, + route?.parentContext ?? null, + ) + } + return [mainSource ? finalKey : publicKey, resultRow] as [ + unknown, + Record, + ] + }), + ) // Apply HAVING clauses if present if (havingClauses && havingClauses.length > 0) { @@ -582,26 +563,22 @@ export function processGroupBy( ) const compiledHaving = compileExpression(transformedHavingClause) - const passesHaving = ([, row]: [unknown, Record]) => { - let selected = row.$selected as Record - if (fields.groupKeyRefs.length > 0) { - const withKeys: Record = { ...selected } - fields.groupKeyRefs.forEach((ref, i) => { - withKeys[ref] = row[fields.groupValues[i]!] - }) - selected = withKeys - } - const namespacedRow = getGroupEvaluationRow(row, fields, selected) - const result = compiledHaving(namespacedRow) - // Preserve each path's coercion for unchecked nonboolean IR values. - return singleGroup ? toBooleanPredicate(result) : result - } - pipeline = - mutableRevision && hasMutableInList(havingExpression) - ? (refreshMap(pipeline, mutableRevision, (entry) => - passesHaving(entry) ? entry : undefined, - ) as NamespacedAndKeyedStream) - : pipeline.pipe(filter(passesHaving)) + pipeline = pipeline.pipe( + filter(([, row]) => { + let selected = row.$selected as Record + if (fields.groupKeyRefs.length > 0) { + const withKeys: Record = { ...selected } + fields.groupKeyRefs.forEach((ref, i) => { + withKeys[ref] = row[fields.groupValues[i]!] + }) + selected = withKeys + } + const namespacedRow = getGroupEvaluationRow(row, fields, selected) + const result = compiledHaving(namespacedRow) + // Preserve each path's coercion for unchecked nonboolean IR values. + return singleGroup ? toBooleanPredicate(result) : result + }), + ) } } diff --git a/packages/db/src/query/compiler/index.ts b/packages/db/src/query/compiler/index.ts index 0b11ab96cd..91e125f364 100644 --- a/packages/db/src/query/compiler/index.ts +++ b/packages/db/src/query/compiler/index.ts @@ -63,7 +63,6 @@ import { normalizeValue } from '../../utils/comparison.js' import { codedMessage, devBuild } from '../../error-message.js' import { compileExpression, - hasMutableInList, isCaseWhenConditionTrue, toBooleanPredicate, } from './evaluators.js' @@ -71,7 +70,6 @@ import { processJoins, registerLazyDemandPlan } from './joins.js' import { containsAggregate, processGroupBy } from './group-by.js' import { getLazyLoadTargets } from './lazy-targets.js' import { processOrderBy } from './order-by.js' -import { refreshMap } from './mutable-list-refresh.js' import { crossJoinParentRoutes } from './parent-routes.js' import { queriesMatchForCaching } from './query-equivalence.js' import { @@ -90,7 +88,6 @@ import { processSelect } from './select.js' import type { ValueIdentity } from '../equality-value-identity.js' import type { CollectionSubscription } from '../../collection/subscription.js' import type { OrderByOptimizationInfo } from './order-by.js' -import type { MutableListRevision } from './mutable-list-refresh.js' import type { Aggregate, BasicExpression, @@ -341,9 +338,6 @@ export interface CompilationResult { /** Map of opaque source IDs to their WHERE clauses for index optimization */ sourceWhereClauses: Map> - /** Captured membership requires unfiltered source rows without order hints. */ - hasCapturedMutableList: boolean - /** * Maps FROM and JOIN source aliases to collection IDs within this compiled * query. Includes keep their own alias maps in their child results. @@ -373,123 +367,11 @@ export interface CompilationResult { } const valueIdentitiesByCache = new WeakMap() -const mutableListRevisions = new WeakMap() function getCompilationValueIdentity(cache: QueryCache): ValueIdentity { return getOrCreate(valueIdentitiesByCache, cache, createValueIdentity) } -function collectMutableSelectLists( - value: unknown, - lists: Set>, - includeSubqueries = true, -): void { - if (value instanceof ConditionalSelect) { - for (const branch of value.branches) { - hasMutableInList(branch.condition, lists) - collectMutableSelectLists(branch.value, lists, includeSubqueries) - } - collectMutableSelectLists(value.defaultValue, lists, includeSubqueries) - } else if (value instanceof IncludesSubquery) { - if (!includeSubqueries) return - for (const where of value.parentFilters ?? []) { - hasMutableInList(getWhereExpression(where), lists) - } - collectMutableQueryLists(value.query, lists) - } else if (isBasicOrAggregateExpression(value)) { - if (value.type === `agg`) { - for (const arg of value.args) hasMutableInList(arg, lists) - } else { - hasMutableInList(value, lists) - } - } else if (value && typeof value === `object`) { - for (const child of Object.values(value)) { - collectMutableSelectLists(child, lists, includeSubqueries) - } - } -} - -function selectHasMutableAggregateInput(value: unknown): boolean { - if (value instanceof ConditionalSelect) { - return ( - value.branches.some( - (branch) => - selectHasMutableAggregateInput(branch.condition) || - selectHasMutableAggregateInput(branch.value), - ) || selectHasMutableAggregateInput(value.defaultValue) - ) - } - if (value instanceof IncludesSubquery) return false - if (isBasicOrAggregateExpression(value)) { - if (value.type !== `agg` && value.type !== `func`) return false - return ( - (value.type === `agg` && - value.args.some((arg) => hasMutableInList(arg))) || - value.args.some(selectHasMutableAggregateInput) - ) - } - return ( - value !== null && - typeof value === `object` && - Object.values(value).some(selectHasMutableAggregateInput) - ) -} - -function collectMutableQueryLists( - query: QueryIR, - lists: Set>, -): void { - for (const where of query.where ?? []) { - hasMutableInList(getWhereExpression(where), lists) - } - for (const join of query.join ?? []) { - hasMutableInList(join.on, lists) - if (join.from.type === `queryRef`) { - collectMutableQueryLists(join.from.query, lists) - } - } - for (const expression of query.groupBy ?? []) { - hasMutableInList(expression, lists) - } - for (const having of query.having ?? []) { - hasMutableInList(getHavingExpression(having), lists) - } - for (const { expression } of query.orderBy ?? []) { - hasMutableInList(expression, lists) - } - if (query.select) collectMutableSelectLists(query.select, lists) - if (query.from.type === `unionAll`) { - for (const branch of query.from.queries) { - collectMutableQueryLists(branch, lists) - } - } - for (const source of getFromSources(query.from)) { - if (source.type === `queryRef`) { - collectMutableQueryLists(source.query, lists) - } - } -} - -function selectHasMutableInList(select: QueryIR[`select`]): boolean { - const lists = new Set>() - collectMutableSelectLists(select, lists, false) - return lists.size > 0 -} - -function snapshotList(list: ReadonlyArray): Array { - return Array.from(list) -} - -function sameListSnapshot( - left: ReturnType, - right: ReturnType, -): boolean { - return ( - left.length === right.length && - left.every((value, index) => Object.is(value, right[index])) - ) -} - /** * Compiles a query IR into a D2 pipeline * @param rawQuery The query IR to compile @@ -519,45 +401,6 @@ type CompileQueryArgs = [ ] export function compileQuery(...args: CompileQueryArgs): CompilationResult { - // A mutable-list plan cannot reuse a subquery compiled earlier with source - // pushdown. Give this graph its own cache and share it only recursively. - const mutableLists = new Set>() - collectMutableQueryLists(args[0], mutableLists) - const dynamic = mutableLists.size > 0 - const cache = dynamic ? new WeakMap() : (args[8] ?? new WeakMap()) - args[8] = cache - if (dynamic) { - const lists = [...mutableLists] - let snapshots = lists.map(snapshotList) - let revision = 0 - // Raw source events clock every expression stage, including events that - // a join, filter, or nested QueryRef suppresses before that stage. - const inputs = [...new Set(Object.values(args[1]))] - let pulses = inputs[0]!.pipe(map(() => [0, 0] as [number, number])) - for (const input of inputs.slice(1)) { - pulses = pulses.pipe( - concatOperator(input.pipe(map(() => [0, 0] as [number, number]))), - ) - } - mutableListRevisions.set( - cache, - pulses.pipe( - reduce(() => { - const next = lists.map(snapshotList) - if ( - next.some( - (snapshot, index) => - !sameListSnapshot(snapshot, snapshots[index]!), - ) - ) { - revision++ - } - snapshots = next - return [[revision, 1]] - }), - ), - ) - } return compileQueryInternal(args, false) } @@ -592,23 +435,9 @@ function compileQueryInternal( if (!scopeValidated) validateQueryStructure(rawQuery) const rawSources = collectCollectionSources(rawQuery) - // A captured mutable membership list can change between source events. - // A mutable membership list needs the complete pre-WHERE relation. On-demand - // sources acquire unfiltered rows rather than only the current list's subset. - const refreshMutableWhere = - rawQuery.where?.some((where) => - hasMutableInList(getWhereExpression(where)), - ) ?? false - const mutableRevision = mutableListRevisions.get(cache) - // Optimize the query before compilation const { optimizedQuery, sourceWhereClauses: aliasWhereClauses } = - mutableListRevisions.has(cache) - ? { - optimizedQuery: rawQuery, - sourceWhereClauses: new Map>(), - } - : optimizeQuery(rawQuery) + optimizeQuery(rawQuery) const sourceWhereClauses = keyWhereClausesBySource( rawQuery, aliasWhereClauses, @@ -783,8 +612,6 @@ function compileQueryInternal( joinsParentDirectly, valueIdentity, parentKeyStream, - mutableListRevisions.has(cache), - mutableRevision, ) } @@ -812,26 +639,15 @@ function compileQueryInternal( // Process the WHERE clause if it exists if (query.where && query.where.length > 0) { - if (refreshMutableWhere && mutableRevision) { - const predicates = query.where.map((where) => - compileExpression(getWhereExpression(where)), + // Apply each WHERE condition as a filter (they are ANDed together) + for (const where of query.where) { + const whereExpression = getWhereExpression(where) + const compiledWhere = compileExpression(whereExpression) + pipeline = pipeline.pipe( + filter(([_key, namespacedRow]) => { + return toBooleanPredicate(compiledWhere(namespacedRow)) + }), ) - pipeline = refreshMap(pipeline, mutableRevision, (pair) => - predicates.every((predicate) => toBooleanPredicate(predicate(pair[1]))) - ? pair - : undefined, - ) as NamespacedAndKeyedStream - } else { - // Apply each WHERE condition as a filter (they are ANDed together). - for (const where of query.where) { - const whereExpression = getWhereExpression(where) - const compiledWhere = compileExpression(whereExpression) - pipeline = pipeline.pipe( - filter(([_key, namespacedRow]) => { - return toBooleanPredicate(compiledWhere(namespacedRow)) - }), - ) - } } } @@ -1010,9 +826,7 @@ function compileQueryInternal( aliasRemapping, ) - // Every child source event can refresh a mutable membership list, even - // when the child does not match the current parent route. - if (!mutableListRevisions.has(cache) && lazyTargets.length > 0) { + if (lazyTargets.length > 0) { // 1. Mark child source as lazy so CollectionSubscriber skips initial full load for (const target of lazyTargets) { lazySources.add(target.sourceId) @@ -1191,7 +1005,6 @@ function compileQueryInternal( valueIdentity, collectionId: mainCollectionId, sourceWhereClauses, - hasCapturedMutableList: mutableListRevisions.has(cache), aliasToCollectionId, aliasRemapping, }) @@ -1244,12 +1057,7 @@ function compileQueryInternal( // An aggregate query computes its select values from each group, so it // does not project rows first. if (!selectHasAggregates && !(query.groupBy && query.groupBy.length > 0)) - pipeline = processSelect( - pipeline, - query.select, - allInputs, - selectHasMutableInList(query.select) ? mutableRevision : undefined, - ) + pipeline = processSelect(pipeline, query.select, allInputs) } else { // If no SELECT clause, create $selected with the main table data pipeline = pipeline.pipe( @@ -1318,9 +1126,6 @@ function compileQueryInternal( mainCollectionId, groupByMainSource, sourceCarriesInternalRouteState || includesRoutingFns.length > 0, - mutableRevision, - query.select ? selectHasMutableInList(query.select) : false, - query.select ? selectHasMutableAggregateInput(query.select) : false, ) } else if (selectHasAggregates) { // SELECT contains aggregates but no GROUP BY: implicit single-group aggregation @@ -1335,9 +1140,6 @@ function compileQueryInternal( mainCollectionId, groupByMainSource, sourceCarriesInternalRouteState || includesRoutingFns.length > 0, - mutableRevision, - query.select ? selectHasMutableInList(query.select) : false, - query.select ? selectHasMutableAggregateInput(query.select) : false, ) } @@ -1470,10 +1272,6 @@ function compileQueryInternal( resultLimit, query.offset, includesGroupKeyFn, - mutableListRevisions.has(cache), - query.orderBy?.some(({ expression }) => hasMutableInList(expression)) - ? mutableRevision - : undefined, ).pipe( map(([key, [row, orderByIndex]]) => finalizeRow(key, row, orderByIndex)), ) as ResultStream @@ -1489,7 +1287,6 @@ function compileQueryInternal( pipeline: resultPipeline, valueIdentity, sourceWhereClauses, - hasCapturedMutableList: mutableListRevisions.has(cache), aliasToCollectionId, aliasRemapping, includes: includesResults.length > 0 ? includesResults : undefined, diff --git a/packages/db/src/query/compiler/joins.ts b/packages/db/src/query/compiler/joins.ts index 55b1f36460..65f1e38092 100644 --- a/packages/db/src/query/compiler/joins.ts +++ b/packages/db/src/query/compiler/joins.ts @@ -27,8 +27,7 @@ import { getPropRefSourceAlias, requireCollectionSource, } from '../ir.js' -import { compileExpression, hasMutableInList } from './evaluators.js' -import { refreshMap } from './mutable-list-refresh.js' +import { compileExpression } from './evaluators.js' import { getLazyLoadTargets } from './lazy-targets.js' import { crossJoinParentRoutes } from './parent-routes.js' import { queriesMatchForCaching } from './query-equivalence.js' @@ -44,7 +43,6 @@ import { import type { ValueIdentity } from '../equality-value-identity.js' import type { CompileQueryFn } from './index.js' import type { OrderByOptimizationInfo } from './order-by.js' -import type { MutableListRevision } from './mutable-list-refresh.js' import type { BasicExpression, CollectionSourceRef, @@ -237,8 +235,6 @@ export function processJoins( mainSourceIsParentFiltered: boolean, valueIdentity: ValueIdentity, parentKeyStream?: KeyedStream, - disableLazyDemand = false, - mutableRevision?: MutableListRevision, ): NamespacedAndKeyedStream { let resultPipeline = pipeline const localBindings = new Set( @@ -270,8 +266,6 @@ export function processJoins( valueIdentity, localBindings, parentKeyStream, - disableLazyDemand, - mutableRevision, ) localBindings.add(joinClause.from.bindingId) } @@ -307,8 +301,6 @@ function processJoin( valueIdentity: ValueIdentity, localBindings: ReadonlySet, parentKeyStream?: KeyedStream, - disableLazyDemand = false, - mutableRevision?: MutableListRevision, ): NamespacedAndKeyedStream { const isCollectionRef = joinClause.from.type === `collectionRef` @@ -414,34 +406,33 @@ function processJoin( ) // Prepare both sides with the same key and raw-demand rules. - const keyMain = ([currentKey, namespacedRow]: [unknown, NamespacedRow]) => - keyJoinInput( - currentKey, - namespacedRow, - mainSource, - `main`, - compiledMainExpressions, - routeJoinedSource, - valueIdentity, - ) - const keyJoined = ([currentKey, row]: [unknown, any]) => - keyJoinInput( - currentKey, - wrapJoinedInputRow(joinedSource, row), - joinedSource, - `joined`, - compiledJoinedExpressions, - routeJoinedSource, - valueIdentity, - ) - const refreshJoin = - mutableRevision !== undefined && hasMutableInList(joinClause.on) - let mainPipeline = refreshJoin - ? refreshMap(pipeline, mutableRevision, keyMain) - : pipeline.pipe(map(keyMain)) - let joinedPipeline = refreshJoin - ? refreshMap(joinedInput, mutableRevision, keyJoined) - : joinedInput.pipe(map(keyJoined)) + let mainPipeline = pipeline.pipe( + map(([currentKey, namespacedRow]) => + keyJoinInput( + currentKey, + namespacedRow, + mainSource, + `main`, + compiledMainExpressions, + routeJoinedSource, + valueIdentity, + ), + ), + ) + + let joinedPipeline = joinedInput.pipe( + map(([currentKey, row]) => + keyJoinInput( + currentKey, + wrapJoinedInputRow(joinedSource, row), + joinedSource, + `joined`, + compiledJoinedExpressions, + routeJoinedSource, + valueIdentity, + ), + ), + ) // Apply the join operation if (![`inner`, `left`, `right`, `full`].includes(joinClause.type)) { @@ -474,7 +465,7 @@ function processJoin( aliasRemapping, ) - if (!disableLazyDemand && !limitedSubquery && lazyTargets.length > 0) { + if (!limitedSubquery && lazyTargets.length > 0) { // This join can be optimized by having the active collection // dynamically load keys into the lazy collection // based on the value of the joinKey and by looking up diff --git a/packages/db/src/query/compiler/mutable-list-refresh.ts b/packages/db/src/query/compiler/mutable-list-refresh.ts deleted file mode 100644 index 2d00da3c0c..0000000000 --- a/packages/db/src/query/compiler/mutable-list-refresh.ts +++ /dev/null @@ -1,47 +0,0 @@ -import { join, map, reduce, unkey } from '@tanstack/db-ivm' -import { codedMessage, devBuild } from '../../error-message.js' -import type { IStreamBuilder, KeyValue } from '@tanstack/db-ivm' - -/** A singleton relation whose value changes only when a captured list changes. */ -export type MutableListRevision = IStreamBuilder> - -/** - * Retain the output of an expression-bearing stage by its input key. A list - * revision visits every retained input, while an ordinary delta visits only - * its own key. The reduction owns the exact prior output: a retraction never - * reevaluates an old input against the newly edited list. - * - * A missing projection drops an input, which also makes this a retained filter. - */ -export function refreshMap( - input: IStreamBuilder>, - revision: MutableListRevision, - project: (entry: KeyValue) => O | undefined, -): IStreamBuilder { - const candidates = input.pipe( - map((entry) => [0, entry] as KeyValue>), - ) - return candidates.pipe( - join(revision, `inner`), - map(([, [entry]]) => { - if (entry === null) { - throw new Error( - devBuild() && process.env.NODE_ENV !== `production` - ? `An inner mutable-list revision lost its candidate` - : codedMessage(243), - ) - } - return [entry[0], entry] as KeyValue> - }), - reduce((values) => - values.flatMap(([entry, multiplicity]) => { - if (multiplicity <= 0) return [] - const output = project(entry) - return output === undefined - ? [] - : [[output, multiplicity] as [O, number]] - }), - ), - unkey(), - ) -} diff --git a/packages/db/src/query/compiler/order-by.ts b/packages/db/src/query/compiler/order-by.ts index 9a4f3aef6e..88c0777f6c 100644 --- a/packages/db/src/query/compiler/order-by.ts +++ b/packages/db/src/query/compiler/order-by.ts @@ -14,13 +14,11 @@ import { ensureIndexForField } from '../../indexes/auto-index.js' import { findIndexForField } from '../../utils/index-optimization.js' import { devBuild } from '../../error-message.js' import { compileExpression } from './evaluators.js' -import { refreshMap } from './mutable-list-refresh.js' import { getSourceAliasesFromExpression } from './expressions.js' import { replaceAggregatesByRefs } from './group-by.js' import type { CompareOptions } from '../builder/types.js' import type { WindowOptions } from './types.js' import type { CompiledSingleRowExpression } from './evaluators.js' -import type { MutableListRevision } from './mutable-list-refresh.js' import type { OrderBy, OrderByClause, QueryIR, Select } from '../ir.js' import type { CollectionLike, @@ -67,8 +65,6 @@ export type OrderByOptimizationInfo = { joinedFilterSourceId?: string } -const CAPTURED_ORDER_VALUE = Symbol(`captured_order_value`) - /** * Processes the ORDER BY clause * Works with the new structure that has both namespaced row data and $selected @@ -85,8 +81,6 @@ export function processOrderBy( limit?: number, offset?: number, groupKeyFn?: (key: unknown, value: unknown) => unknown, - requiresFullSourceForMutableList = false, - mutableRevision?: MutableListRevision, ): IStreamBuilder> { // Pre-compile all order by expressions const compiledOrderBy = orderByClause.map((clause) => { @@ -125,21 +119,6 @@ export function processOrderBy( // Default case - no ordering return null } - // topK must retract the old sort key before a changed list supplies the - // new one. Retain the evaluated key beside each candidate, not in topK's - // callback over a raw row that can see only current list members. - if (mutableRevision) { - pipeline = refreshMap(pipeline, mutableRevision, ([key, row]) => [ - key, - { ...row, [CAPTURED_ORDER_VALUE]: valueExtractor(row) }, - ]) as NamespacedAndKeyedStream - } - const readOrderValue = mutableRevision - ? (row: NamespacedRow) => - (row as NamespacedRow & { [CAPTURED_ORDER_VALUE]: unknown })[ - CAPTURED_ORDER_VALUE - ] - : valueExtractor // Create a multi-property comparator that respects the order and direction of each property const compare = (a: unknown, b: unknown) => { @@ -175,9 +154,7 @@ export function processOrderBy( // When there's a limit, create orderByOptimizationInfo for top-K source // loading. Unbounded queries use subscription hints instead. A plan whose // local semantics cannot be established from a provider prefix records - // requiresFullSource and issues one full-source acquisition. A mutable - // captured list needs that acquisition without a source WHERE filter so - // later changes to previously unseen rows can trigger membership refresh. + // requiresFullSource and issues one filtered full-source acquisition. // We try to use an index on the FIRST orderBy column for lazy loading, // even for multi-column orderBy (using wider bounds on first column). // Skip this optimization when using grouped ordering (includes with limit), @@ -212,16 +189,19 @@ export function processOrderBy( const compareOpts = buildCompareOptions(firstClause, collection) if (fieldName) { - // The index derives this term's comparator from compareOptions. - // Supplying a compareFn would mark it as a custom comparator whose - // order cannot be used for ordinary ordered traversal. + // Use a single-column comparator for the index, not the + // multi-column `compare` function. The multi-column comparator + // expects array values [col1, col2, ...] but the index stores + // individual field values. Passing `compare` here causes the + // BTree to treat all single values as equal (since number[0] + // === undefined for both sides of the comparison). + const firstColumnCompareFn = makeComparator(compareOpts) ensureIndexForField( fieldName, followRefResult.path, followRefCollection, compareOpts, - undefined, - true, + firstColumnCompareFn, ) } @@ -229,7 +209,6 @@ export function processOrderBy( followRefCollection, followRefResult.path, compareOpts, - true, ) // Only use the index if it supports range queries @@ -307,9 +286,6 @@ export function processOrderBy( sourceHoldsAllRows: followRefResult.collection.config.syncMode !== `on-demand`, requiresFullSource: - // An unseen source delete must still trigger a captured mutable - // membership list, so a finite prefix cannot own this subscription. - requiresFullSourceForMutableList || sourceOrderBy.some( ({ compareOptions }) => compareOptions.stringSort === `custom`, ) || @@ -366,7 +342,7 @@ export function processOrderBy( // otherwise use the standard global ordering operator. if (groupKeyFn) { return pipeline.pipe( - groupedOrderByWithFractionalIndex(readOrderValue, { + groupedOrderByWithFractionalIndex(valueExtractor, { limit, offset, comparator: compare, @@ -391,7 +367,7 @@ export function processOrderBy( // Use fractional indexing and return the tuple [value, index] return pipeline.pipe( - orderByWithFractionalIndex(readOrderValue, { + orderByWithFractionalIndex(valueExtractor, { limit, offset, comparator: compare, diff --git a/packages/db/src/query/compiler/select.ts b/packages/db/src/query/compiler/select.ts index 88f0ef799b..f9db0515be 100644 --- a/packages/db/src/query/compiler/select.ts +++ b/packages/db/src/query/compiler/select.ts @@ -7,9 +7,7 @@ import { } from '../ir.js' import { UnsafeAliasPathError } from '../../errors.js' import { compileExpression, isCaseWhenConditionTrue } from './evaluators.js' -import { refreshMap } from './mutable-list-refresh.js' import type { BasicExpression, Select, SelectValueExpression } from '../ir.js' -import type { MutableListRevision } from './mutable-list-refresh.js' import type { KeyedStream, NamespacedAndKeyedStream, @@ -148,19 +146,14 @@ export function processSelect( pipeline: NamespacedAndKeyedStream, select: Select, _allInputs: Record, - mutableRevision?: MutableListRevision, ): NamespacedAndKeyedStream { if (!isNestedSelectObject(select)) { const compiled = compileSelectValue(select as SelectValueExpression) - const project = ([key, namespacedRow]: [unknown, NamespacedRow]) => - [key, { ...namespacedRow, $selected: compiled(namespacedRow) }] as [ - unknown, - NamespacedRow, - ] - return ( - mutableRevision - ? refreshMap(pipeline, mutableRevision, project) - : pipeline.pipe(map(project)) + return pipeline.pipe( + map(([key, namespacedRow]) => [ + key, + { ...namespacedRow, $selected: compiled(namespacedRow) }, + ]), ) as NamespacedAndKeyedStream } @@ -169,12 +162,7 @@ export function processSelect( addFromObject([], select, ops) - const project = (row: [unknown, NamespacedRow]) => processRow(row, ops) - return ( - mutableRevision - ? refreshMap(pipeline, mutableRevision, project) - : pipeline.pipe(map(project)) - ) as NamespacedAndKeyedStream + return pipeline.pipe(map((row) => processRow(row, ops))) } function compileSelectObject( diff --git a/packages/db/src/query/effect.ts b/packages/db/src/query/effect.ts index 683f08eddf..7d699b83f8 100644 --- a/packages/db/src/query/effect.ts +++ b/packages/db/src/query/effect.ts @@ -419,7 +419,6 @@ class EffectPipelineRunner { private graph: D2 | undefined private inputs: Record> | undefined private sourceWhereClauses: Map> | undefined - private hasCapturedMutableList = false // Mutable objects passed to compileQuery by reference. // The join compiler captures these references and reads them later when @@ -517,7 +516,6 @@ class EffectPipelineRunner { ) this.sourceWhereClauses = compilation.sourceWhereClauses - this.hasCapturedMutableList = compilation.hasCapturedMutableList // Attach the output operator that accumulates changes compilation.pipeline.pipe( @@ -1083,7 +1081,6 @@ class EffectPipelineRunner { this.query, alias, this.queryCompareOptions, - this.hasCapturedMutableList, ) return { @@ -1166,7 +1163,6 @@ class EffectPipelineRunner { this.graph = undefined this.inputs = undefined this.sourceWhereClauses = undefined - this.hasCapturedMutableList = false } } diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index fbd013c3c8..a83d71be47 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -232,11 +232,9 @@ domain. An index's advertised comparison options also define its executable comparator; metadata cannot claim an order that the index does not use. Explicit `undefined` range and cursor bounds denote the indexed nullish comparator group, while an absent bound denotes the start or end of the index. An ordered index -groups row values by comparator position independently of exact equality -buckets and keeps a live representative for each group. One exact bucket may -span several positions under a custom comparator, while distinct exact values -may share a position. Range traversal and ordered limits must retain every row -in either case. Compiler tokens belong to one +groups exact value buckets that compare at the same position and keeps a live +representative for each group, so range traversal and ordered limits cannot drop +rows whose distinct values are comparator-equal. Compiler tokens belong to one compiled graph. This keeps every operator in the graph on the same identity relation. Objects, functions, and local symbols are weakly keyed where the runtime supports weak symbol keys. Older runtimes retain local symbols strongly @@ -744,49 +742,6 @@ lifecycle remains live. Cancellation and release are not data mutations. The immutable-demand boundary matrix checks direct and deferred sync startup, adapter return and asynchronous settlement, cancellation, and release identity. -A caller-owned array captured by a query's `inArray` expression is a query -input, not permission to mutate submitted request data. Editing that array -alone publishes no query change. At the next source event, the query must -evaluate its current members and bring its whole existing result into line, -including rows untouched by that event. The rule applies when membership is -read in `where`, a join operand, `select`, a group key or aggregate argument, -`having`, or an order expression. One source transaction publishes the new -result coherently; changing a projected value, group, or order cannot leave -the old stage output behind or expose an intermediate public result. -This rule covers entries added, removed, or replaced in the captured array; -it does not specify behavior after an existing Date or byte-array member is -mutated in place. - -The compiler compares captured array entries by identity at raw source events -and shares one list revision across the compiled query graph. -A list change makes each affected expression -stage revisit its retained inputs; an ordinary delta with an unchanged list -visits only the inputs it changes. Each stage retains its prior output for -retraction before it evaluates the new membership. JOIN, WHERE, SELECT, -GROUP BY and aggregate input, HAVING, and ORDER use this rule. The groupBy -operator accepts an optional mapped-input hook for the same revision. These -mechanics explain the receiving paths; they do not prove a general work bound. -Selected-expression inspection must descend through wrappers to aggregate -arguments. For example, `add(sum(caseWhen(inArray(...))), 10)` still needs -the groupBy input refresh when membership changes. - -At the public Collection boundary, a layout-only order change defers order -maintenance until the sync batch has applied. Otherwise a comparator reading -new fractional positions can search an old order and duplicate a source row -in a limited result. - -Any source request derived from the query still obeys the immutable -request-data boundary above. The compiler and pooled-query paths preserve -the captured-list law when they choose where to filter or retain source -rows. For a graph with captured mutable membership, on-demand source -acquisition is unfiltered. The compiler disables predicate pushdown and -keyed lazy join and include demand for that graph. An ordered plan also -acquires the full source without order or limit hints. An event on a source -row outside the current match, join key, child route, or ordered prefix must -still clock the list's current members. This can increase source transfer -and retained rows; controlled in-memory receivers establish the path, not -a real provider's bandwidth. - A Collection subscription installs each logical subset owner before it calls the source adapter. Reentrant release during `loadSubset` therefore retires the logical owner at once, but physical release waits until the adapter returns and @@ -980,9 +935,7 @@ if it rejects, an explicit retry must also start at offset zero without a cursor. The same holds after any later ordered request fails or is canceled: the adapter may already have written only part of its response, so those rows cannot establish a continuation boundary. The next explicit retry starts from -the source as one authoritative full-source request. The request retains the -ordinary plan's source predicate, but a captured mutable-list graph requests -unfiltered source rows. Core cannot know +the source as one authoritative filtered full-source request. Core cannot know which rows a failed request wrote, and a successful limited request proves neither how many authoritative rows it applied nor source exhaustion. Recovery therefore does not infer a safe finite prefix from local row count or boundary @@ -1021,9 +974,8 @@ before adapter code runs, and reentrant truncate or disposal stops the current retirement pass. No copied rows or additional cursor history are retained. A visible delete or source-order change after an otherwise settled, -expressible window that does not require full-source acquisition uses a -narrower authoritative path. Core reissues the ordered prefix from the start -with the query predicate, order terms, and +expressible window uses a narrower authoritative path. Core reissues the +ordered prefix from the start with the query predicate, order terms, and `limit = offset + limit`; the transport `offset` is omitted because this acquisition starts at the provider's source prefix. It then reacquires the first-column tie boundary and any required refill before publishing. Only the @@ -1070,20 +1022,19 @@ remains bounded by `orderBy` and `limit`; local delivery cardinality is a separate observation. Bounded repair is available only when a provider prefix is sufficient for the -local plan. A captured mutable `inArray` list excludes bounded repair. Core -requires full-source recovery for an indirect order expression, a non-root -ordered source, any join other than a LEFT join, a +local plan. Core requires full-source recovery for an indirect order +expression, a non-root ordered source, any join other than a LEFT join, a residual or cross-alias predicate, a functional predicate, grouping, `having`, functional `having`, or `distinct`. These cases can discard or reorder an otherwise valid provider prefix even when a cursor value itself is expressible. A right, full or outer join also adds joined rows whose root match may sit past the prefix; within it they would look unmatched and could take the -window. A single direct LEFT join to a Collection is an exception for a -cross-alias filter when every order term comes from the root source. Its joined -filter can remove root rows, but cannot make a later root precede an earlier -root. Before loading another root page, core waits for the current lazy join -demand and any joined subset load to settle, then reruns the graph. Joined-side -changes also recheck ordered demand +window. A single direct +LEFT join to a Collection is an exception for a cross-alias filter when every order +term comes from the root source. Its joined filter can remove root rows, but +cannot make a later root precede an earlier root. Before loading another root +page, core waits for the current lazy join demand and any joined subset load to +settle, then reruns the graph. Joined-side changes also recheck ordered demand so a removed match can refill a short window. An empty joined replay still releases held Effect callbacks. @@ -1093,18 +1044,17 @@ after that boundary does not count toward a full window: an earlier eligible row may be unsent. The window then asks for the next page from the boundary, as an underfilled window does, until every window row is at or before it. This holds however a row leaves the window: a joined-side delete or insert, a -join-key update, or a root change, and for Collection and Effect consumers. The -adapter receives ordinary finite root requests; this contract does not imply -that it can evaluate the relation filter remotely. +join-key update, or a root change, and for Collection and Effect consumers. The adapter receives ordinary +finite root requests; this contract does not imply that it can evaluate the +relation filter remotely. Finite continuation still requires an order index; an underfilled unindexed prefix retains the established full-source fallback. A custom string comparator is a local ordering contract. If any resolved source order term uses `stringSort: 'custom'`, bounded and unbounded queries -use one authoritative full-source request for the first acquisition -and every replacement: the ordinary source predicate remains, while `orderBy`, -`limit`, `offset`, and cursor are omitted. A captured mutable-list graph uses -no source predicate, as above. Core then sorts locally and uses ascending +use one authoritative filtered full-source request for the first acquisition +and every replacement: the source predicate remains, while `orderBy`, `limit`, +`offset`, and cursor are omitted. Core then sorts locally and uses ascending public-key order for comparator-equal rows, including when the primary term is descending. It never interprets the comparator function as a stable backend collation identity or @@ -1418,9 +1368,7 @@ preload. An initial ordered load also has a synchronous observation cut. The initial load is an ordinary ordered request or, for a plan that requires the full -source, its first full-source request. An ordinary plan retains its source -predicate; a captured mutable-list graph acquires unfiltered rows. The cut -holds when the first +source, its first filtered full-source request. The cut holds when the first subscriber or preload starts the live-query Collection's sync run, or when its source Collections are eager. A sync run that started earlier resumes deferred on-demand acquisition through the subscription's restart path, which settles @@ -1618,13 +1566,6 @@ create recursive Collection machinery. counts as a preload. A deferring live query keeps its sources from garbage collection but is not counted in their `subscriberCount` or `subscribers:change`, which adapters read as a request for live data. -16. **Captured membership:** adding, removing, or replacing entries in a - caller-owned `inArray` list does not publish by itself. After the next - source event, the complete existing live-query result reflects those - current entries at every expression stage, including rows untouched by - that event. Value, group, join, and order changes publish coherently. - Mutating an existing Date or byte-array member in place is outside this - law. Submitted source request data remains immutable. ## Glossary @@ -1658,32 +1599,30 @@ keep the meanings defined there. ## Executable contracts -| Contract | Test suite | -| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | -| State equivalence, route lifecycle, transition history, and batch partition | `packages/db/tests/query/includes-oracle.property.test.ts` | -| Joined multiplicity, alias identity, and null-key normalization | `packages/db/tests/query/includes-query-shape-oracle.test.ts` | -| Alias reuse, captured bindings, public keys, and `SourceId` preservation | (`packages/db/tests/query/includes-oracle.property.test.ts`, `packages/db/tests/query/includes-alias-shadowing-oracle.test.ts`) | -| Same-scope and union alias rejection | `packages/db/tests/query/validate-aliases.test.ts` | -| Demand, cancellation, progressive timing, and source readiness | `packages/db/tests/query/includes-temporal-oracle.test.ts` | -| Optimistic confirmation, rollback, and later reactivity | `packages/db/tests/query/includes-optimistic-oracle.property.test.ts` | -| Coherent layered publication | `packages/db/tests/query/includes-publication-oracle.test.ts` | -| Collection facades, event coherence, and route activation | `packages/db/tests/query/includes-collection-oracle.property.test.ts` | -| Correlated physical work | `packages/db/tests/query/includes-work-counter-oracle.test.ts` | -| Constructed and retained facades in a nested Collection tree | `packages/db/tests/query/includes-space-oracle.test.ts` | -| Route-context discovery and transport across recursive and join boundaries | `packages/db/tests/query/includes-context-transport-oracle.test.ts` | -| Functional projection input boundaries, timing, and output preservation | `packages/db/tests/query/includes-functional-projection-oracle.test.ts` | -| Functional input rejection and inline alternatives | `packages/db/tests/query/includes-functional-input-boundary.test.ts` | -| Public-container descriptors and reference-key matches across internal query stages | `packages/db/tests/query/public-container-copy.test.ts` | -| Cross-formulation equivalence and reference-sensitive route identity | `packages/db/tests/query/includes-cross-formulation-oracle.property.test.ts` | -| Query-db ownership | `packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts` | -| Failed replay retention, peer isolation, and explicit consumer-only recovery | `packages/db/tests/query/replay-failure-boundary.test.ts` | -| Replay lease balance, reference-counted peers, and failed-start recovery | `packages/db/tests/replay-adapter-ownership.test.ts` | -| Reachable nested shape | `packages/query-db-collection/tests/includes-work-counter-oracle.test.ts` | -| Cleanup-start invalidation, settlement, and restart admission | `packages/db/tests/collection-cleanup-restart-oracle.test.ts` | -| Captured `inArray` membership at source publication, including unbounded acquisition and Effect cuts | `packages/db/tests/query/where-predicate-publication-oracle.property.test.ts` | -| Captured `inArray` membership in SELECT, ORDER, GROUP, aggregates, HAVING, JOIN, and QueryRef stages | `packages/db/tests/query/captured-inarray-stages-oracle.test.ts` | -| Pooled live queries match their live-query Collection, including mutable residual admission and cleanup | `packages/db/tests/query/pooled-live-query-oracle.property.test.ts` | -| Deferred acquisition before a subscriber or preload, readiness, and the synchronous cut | `packages/db/tests/live-query-deferred-acquisition-oracle.test.ts` | +| Contract | Test suite | +| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | +| State equivalence, route lifecycle, transition history, and batch partition | `packages/db/tests/query/includes-oracle.property.test.ts` | +| Joined multiplicity, alias identity, and null-key normalization | `packages/db/tests/query/includes-query-shape-oracle.test.ts` | +| Alias reuse, captured bindings, public keys, and `SourceId` preservation | (`packages/db/tests/query/includes-oracle.property.test.ts`, `packages/db/tests/query/includes-alias-shadowing-oracle.test.ts`) | +| Same-scope and union alias rejection | `packages/db/tests/query/validate-aliases.test.ts` | +| Demand, cancellation, progressive timing, and source readiness | `packages/db/tests/query/includes-temporal-oracle.test.ts` | +| Optimistic confirmation, rollback, and later reactivity | `packages/db/tests/query/includes-optimistic-oracle.property.test.ts` | +| Coherent layered publication | `packages/db/tests/query/includes-publication-oracle.test.ts` | +| Collection facades, event coherence, and route activation | `packages/db/tests/query/includes-collection-oracle.property.test.ts` | +| Correlated physical work | `packages/db/tests/query/includes-work-counter-oracle.test.ts` | +| Constructed and retained facades in a nested Collection tree | `packages/db/tests/query/includes-space-oracle.test.ts` | +| Route-context discovery and transport across recursive and join boundaries | `packages/db/tests/query/includes-context-transport-oracle.test.ts` | +| Functional projection input boundaries, timing, and output preservation | `packages/db/tests/query/includes-functional-projection-oracle.test.ts` | +| Functional input rejection and inline alternatives | `packages/db/tests/query/includes-functional-input-boundary.test.ts` | +| Public-container descriptors and reference-key matches across internal query stages | `packages/db/tests/query/public-container-copy.test.ts` | +| Cross-formulation equivalence and reference-sensitive route identity | `packages/db/tests/query/includes-cross-formulation-oracle.property.test.ts` | +| Query-db ownership | `packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts` | +| Failed replay retention, peer isolation, and explicit consumer-only recovery | `packages/db/tests/query/replay-failure-boundary.test.ts` | +| Replay lease balance, reference-counted peers, and failed-start recovery | `packages/db/tests/replay-adapter-ownership.test.ts` | +| Reachable nested shape | `packages/query-db-collection/tests/includes-work-counter-oracle.test.ts` | +| Cleanup-start invalidation, settlement, and restart admission | `packages/db/tests/collection-cleanup-restart-oracle.test.ts` | +| Pooled live queries match their live-query Collection, including cleanup | `packages/db/tests/query/pooled-live-query-oracle.property.test.ts` | +| Deferred acquisition before a subscriber or preload, readiness, and the synchronous cut | `packages/db/tests/live-query-deferred-acquisition-oracle.test.ts` | Each oracle identifies the first divergent checkpoint and compares either the whole result or one exact structural difference. Correlated-materialization diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index 38942a447f..d683d262ac 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -149,7 +149,6 @@ export class CollectionConfigBuilder< private pipelineCache: ResultStream | undefined public sourceWhereClausesCache: Map> | undefined - public hasCapturedMutableList = false private bucketFacadesCache: ReturnType[`facades`] | undefined @@ -818,7 +817,6 @@ export class CollectionConfigBuilder< this.inputsCache = undefined this.pipelineCache = undefined this.sourceWhereClausesCache = undefined - this.hasCapturedMutableList = false this.bucketFacadesCache = undefined // Reset lazy source alias state @@ -877,7 +875,6 @@ export class CollectionConfigBuilder< ) this.pipelineCache = materialized.pipeline this.sourceWhereClausesCache = compilation.sourceWhereClauses - this.hasCapturedMutableList = compilation.hasCapturedMutableList this.bucketFacadesCache = materialized.facades const missingSources = this.collectionSources diff --git a/packages/db/src/query/live/collection-subscriber.ts b/packages/db/src/query/live/collection-subscriber.ts index d268435f8c..b1e64861c9 100644 --- a/packages/db/src/query/live/collection-subscriber.ts +++ b/packages/db/src/query/live/collection-subscriber.ts @@ -271,7 +271,6 @@ export class CollectionSubscriber< this.collectionConfigBuilder.query, this.alias, this.collectionConfigBuilder.queryCompareOptions, - this.collectionConfigBuilder.hasCapturedMutableList, ) // Track loading via the loadSubset promise directly. diff --git a/packages/db/src/query/live/subset-demand-controller.ts b/packages/db/src/query/live/subset-demand-controller.ts index 2f09ea7ad0..9d80b9047b 100644 --- a/packages/db/src/query/live/subset-demand-controller.ts +++ b/packages/db/src/query/live/subset-demand-controller.ts @@ -237,12 +237,7 @@ function createSegment( plan: LazyDemandPlan, keys: Map, ): DemandSegment { - // This controller owns the request's array snapshot. Freezing it lets the - // evaluator index primitive demand keys once without freezing the values or - // changing the caller-owned array semantics of public inArray expressions. - const keyList = [...keys.values()] - Object.freeze(keyList) - const where = inArray(new PropRef(plan.path), keyList) + const where = inArray(new PropRef(plan.path), [...keys.values()]) return { keys, where, diff --git a/packages/db/src/query/live/utils.ts b/packages/db/src/query/live/utils.ts index b914cc12da..f2602bee48 100644 --- a/packages/db/src/query/live/utils.ts +++ b/packages/db/src/query/live/utils.ts @@ -198,16 +198,13 @@ export function reconcileChangesForD2< * Returns normalised orderBy and effective limit suitable for passing to * `subscribeChanges`, or `undefined` values when the query's orderBy cannot * be scoped to the given alias (e.g. cross-collection refs or aggregates), or - * when its resolved string collation is a local custom comparator. A captured - * mutable list requires the full source without ordering hints. + * when its resolved string collation is a local custom comparator. */ export function computeSubscriptionOrderByHints( query: { orderBy?: OrderBy; limit?: number; offset?: number }, alias: string, queryCompareOptions: StringCollationConfig, - hasCapturedMutableList: boolean, ): { orderBy: OrderBy | undefined; limit: number | undefined } { - if (hasCapturedMutableList) return { orderBy: undefined, limit: undefined } const { orderBy, limit, offset } = query const effectiveLimit = limit !== undefined && offset !== undefined ? limit + offset : limit diff --git a/packages/db/src/query/pooled-live-query.ts b/packages/db/src/query/pooled-live-query.ts index 32bfe51db6..9f87605ff3 100644 --- a/packages/db/src/query/pooled-live-query.ts +++ b/packages/db/src/query/pooled-live-query.ts @@ -6,11 +6,7 @@ import { isVirtualPropName } from '../virtual-props.js' import { getPersistedReadinessSource } from '../persisted-readiness.js' import { getWhereExpression } from './ir.js' import { equalityConjunct, equalityKey, readPath } from './equality-conjunct.js' -import { - compileExpression, - hasMutableInList, - toBooleanPredicate, -} from './compiler/evaluators.js' +import { compileExpression, toBooleanPredicate } from './compiler/evaluators.js' import { buildCompareOptions } from './compiler/order-by.js' import { createLiveQueryCollection } from './live-query-collection.js' import type { BasicExpression, OrderBy, QueryIR } from './ir.js' @@ -400,10 +396,6 @@ function poolableShape(query: QueryIR): PoolableShape | undefined { return undefined } } - // A source event in another partition must still refresh membership when a - // captured list changes. The shared group does not notify this view there. - if (residual.some((expression) => hasMutableInList(expression))) - return undefined // A partition needs at least one equality to group by. if (conjuncts.length === 0) return undefined const orderBy = query.orderBy?.length ? query.orderBy : undefined diff --git a/packages/db/src/utils.ts b/packages/db/src/utils.ts index 55a9327896..e472470ed2 100644 --- a/packages/db/src/utils.ts +++ b/packages/db/src/utils.ts @@ -9,34 +9,6 @@ interface TypedArray { [index: number]: number } -// Duration equality retains the exact units, including subsecond units that -// share a default display. Query normalization uses the same field order. -export const durationFields = [ - `years`, - `months`, - `weeks`, - `days`, - `hours`, - `minutes`, - `seconds`, - `milliseconds`, - `microseconds`, - `nanoseconds`, -] as const - -// Ordinary native Dates compare by timestamp, including across realms. -export function dateTimestamp(value: unknown): number | undefined { - if (value === null || typeof value !== `object`) return undefined - if (value instanceof Date) return value.getTime() - if (Object.prototype.toString.call(value) !== `[object Date]`) - return undefined - try { - return Date.prototype.getTime.call(value) - } catch { - return undefined - } -} - /** * Deep equality function that compares two values recursively * Handles primitives, objects, arrays, Date, RegExp, Map, Set, TypedArrays, and Temporal objects @@ -111,11 +83,13 @@ export function deepEqualsInternal( return false } - const aDateTime = dateTimestamp(a) - const bDateTime = dateTimestamp(b) - if (aDateTime !== undefined || bDateTime !== undefined) { - return Object.is(aDateTime, bDateTime) + // Handle Date objects + if (a instanceof Date) { + if (!(b instanceof Date)) return false + return Object.is(a.getTime(), b.getTime()) } + // Symmetric check: if b is Date but a is not, they're not equal + if (b instanceof Date) return false // Handle RegExp objects if (a instanceof RegExp) { @@ -250,22 +224,20 @@ export function deepEqualsInternal( if ( ArrayBuffer.isView(a) && ArrayBuffer.isView(b) && - Object.prototype.toString.call(a) !== `[object DataView]` && - Object.prototype.toString.call(b) !== `[object DataView]` + !(a instanceof DataView) && + !(b instanceof DataView) ) { const typedA = a as unknown as TypedArray const typedB = b as unknown as TypedArray - const lengthA = typedA.length - const lengthB = typedB.length // Elements compare like numbers: -0 equals 0, NaN equals NaN. Only a // draft treats a change of typed-array class as a change. if ( (draft && Object.getPrototypeOf(a) !== Object.getPrototypeOf(b)) || - lengthA !== lengthB + typedA.length !== typedB.length ) return false - for (let i = 0; i < lengthA; i++) { + for (let i = 0; i < typedA.length; i++) { const x = typedA[i]! const y = typedB[i]! if (x !== y && !(x !== x && y !== y)) return false @@ -276,7 +248,7 @@ export function deepEqualsInternal( // Symmetric check: if b is TypedArray but a is not, they're not equal if ( ArrayBuffer.isView(b) && - Object.prototype.toString.call(b) !== `[object DataView]` && + !(b instanceof DataView) && !ArrayBuffer.isView(a) ) { return false @@ -291,15 +263,13 @@ export function deepEqualsInternal( // If they're different Temporal types, they're not equal if (aTag !== bTag) return false - if (aTag === `Temporal.Duration`) { - const left = a as typeof a & - Record<(typeof durationFields)[number], number> - const right = b as typeof b & - Record<(typeof durationFields)[number], number> - return durationFields.every((field) => left[field] === right[field]) + // Use Temporal's built-in equals method if available + if (typeof a.equals === `function`) { + return a.equals(b) } - return a.equals!(b) + // Fallback to toString comparison for other types + return a.toString() === b.toString() } // Symmetric check: if b is Temporal but a is not, they're not equal if (isTemporal(b)) return false @@ -397,7 +367,7 @@ export interface TemporalLike { equals?: (other: unknown) => boolean } -/** Recognizes a supported Temporal tag; callers assume tagged values are valid. */ +/** Checks if the value is a Temporal object by checking for the Temporal brand */ export function isTemporal(a: unknown): a is TemporalLike { if (a == null || typeof a !== `object`) return false const tag = (a as Record)[Symbol.toStringTag] diff --git a/packages/db/src/utils/btree.ts b/packages/db/src/utils/btree.ts index 42d841c90f..82366f6deb 100644 --- a/packages/db/src/utils/btree.ts +++ b/packages/db/src/utils/btree.ts @@ -303,9 +303,8 @@ class BNode { ): void { const cmp = tree._compare let iLow, iHigh - if (Object.is(high, low)) { - // A point range (as used by delete) needs only one search. `===` also - // equates signed zero values that a supplied comparator can separate. + if (high === low) { + // A point range (as used by delete) needs only one search. if (!includeHigh) return iHigh = (iLow = this.indexOf(low, -1, cmp)) + 1 if (iLow < 0) return @@ -475,10 +474,9 @@ class BNodeInternal extends BNode { children = this.children let iLow = this.indexOf(low, 0, cmp), i = iLow - // Only identical bounds can use the one-child point-range shortcut. - // A supplied comparator may place -0 and +0 in different children. + // A point range (as used by delete) needs only one search. const iHigh = Math.min( - Object.is(high, low) ? iLow : this.indexOf(high, 0, cmp), + high === low ? iLow : this.indexOf(high, 0, cmp), keys.length - 1, ) for (; i <= iHigh; i++) { diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index af1e36cc04..367f4e25f1 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -1,10 +1,8 @@ -import { dateTimestamp, durationFields, isTemporal } from '../utils' +import { isTemporal } from '../utils' import { getRuntimeReferenceIdentity } from '../query/runtime-reference-identity' import { codedMessage, devBuild } from '../error-message' import type { CompareOptions } from '../query/builder/types' -export { dateTimestamp } from '../utils' - // WeakMap to store stable IDs for objects const objectIds = new WeakMap() let nextObjectId = 1 @@ -13,21 +11,13 @@ let nextObjectId = 1 * Get or create a stable ID for an object */ function getObjectId(obj: object): number { - if (objectIds.has(obj)) { - return objectIds.get(obj)! - } + const existing = objectIds.get(obj) + if (existing !== undefined) return existing const id = nextObjectId++ objectIds.set(obj, id) return id } -function hasUnorderableValue( - value: unknown, - timestamp: number | undefined, -): boolean { - return Number.isNaN(value) || Number.isNaN(timestamp) -} - /** * Whether a value has no IEEE-754 natural order: `NaN`, or an invalid Date * (whose timestamp is `NaN`). The query engine follows PostgreSQL float @@ -36,8 +26,11 @@ function hasUnorderableValue( * evaluator treat them explicitly instead of letting `NaN` compare unequal to * everything (which has no consistent order and cannot be indexed or sorted). */ -export function isUnorderable(value: unknown): boolean { - return hasUnorderableValue(value, dateTimestamp(value)) +export function isUnorderable(value: any): boolean { + return ( + (typeof value === `number` && Number.isNaN(value)) || + (value instanceof Date && Number.isNaN(value.getTime())) + ) } /** @@ -57,10 +50,8 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { // all equal and sort greater than every other non-null value. This keeps the // order total (NaN would otherwise compare equal to everything), so such // values can be sorted and stored in tree-based indexes. - const aDateTime = dateTimestamp(a) - const bDateTime = dateTimestamp(b) - const aUnordered = hasUnorderableValue(a, aDateTime) - const bUnordered = hasUnorderableValue(b, bDateTime) + const aUnordered = isUnorderable(a) + const bUnordered = isUnorderable(b) if (aUnordered && bUnordered) return 0 if (aUnordered) return 1 if (bUnordered) return -1 @@ -76,12 +67,6 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { // For lexical sort we rely on direct comparison for primitive values } - // Keep the string collation class separate from other value types. Native - // relational comparison ties every nonnumeric string with every number, - // although those same strings have a distinct order among themselves. - if (typeof a === `string` && typeof b !== `string`) return -1 - if (typeof b === `string` && typeof a !== `string`) return 1 - // if a and b are both arrays, compare them element by element if (Array.isArray(a) && Array.isArray(b)) { for (let i = 0; i < Math.min(a.length, b.length); i++) { @@ -95,14 +80,12 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { } // If both are dates, compare them - if (aDateTime !== undefined && bDateTime !== undefined) { - return aDateTime - bDateTime + if (a instanceof Date && b instanceof Date) { + return a.getTime() - b.getTime() } // If both are Temporal objects, use compareTemporalValues for correct semantic ordering - const aTemporal = isTemporal(a) - const bTemporal = isTemporal(b) - if (aTemporal && bTemporal) { + if (isTemporal(a) && isTemporal(b)) { return compareTemporalValues(a, b) } @@ -118,32 +101,13 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { if (aIsSymbol) return 1 if (bIsSymbol) return -1 - // Function values are references too; coercing them against numbers can - // create the same non-transitive ties as unlike ordered object kinds. - const aIsObject = typeof a === `object` || typeof a === `function` - const bIsObject = typeof b === `object` || typeof b === `function` + // If at least one of the values is an object, use stable IDs for comparison + const aIsObject = typeof a === `object` + const bIsObject = typeof b === `object` if (aIsObject || bIsObject) { - // Compare unlike ordered kinds before assigning stable IDs to ordinary objects. + // If both are objects, compare their stable IDs if (aIsObject && bIsObject) { - // Array, Date, and Temporal comparisons order values within their own - // kinds above. Keep those kinds apart from ordinary objects: mixing a - // calendar comparison with runtime object IDs creates order cycles. - const aKind = Array.isArray(a) - ? 0 - : aDateTime !== undefined - ? 1 - : aTemporal - ? 2 - : 3 - const bKind = Array.isArray(b) - ? 0 - : bDateTime !== undefined - ? 1 - : bTemporal - ? 2 - : 3 - if (aKind !== bKind) return aKind - bKind const aId = getObjectId(a) const bId = getObjectId(b) return aId - bId @@ -187,45 +151,6 @@ export function makeComparator( } } -/** - * Index storage needs a total order even for Temporal values that have no - * public `orderBy` relation. Keep kinds in separate groups, and consistently - * order every Duration and PlainMonthDay by its exact equality key. Public - * comparisons still use `makeComparator` and retain their strict semantics. - */ -export function makeIndexComparator( - opts: CompareOptions, -): (a: any, b: any) => number { - const compare = makeComparator(opts) - return (a, b) => { - const aTemporal = isTemporal(a) - const bTemporal = isTemporal(b) - if (!aTemporal || !bTemporal) return compare(a, b) - - const aKind = a[Symbol.toStringTag] - const bKind = b[Symbol.toStringTag] - let order: number - if (aKind !== bKind) { - order = aKind < bKind ? -1 : 1 - } else if ( - aKind === `Temporal.Duration` || - aKind === `Temporal.PlainMonthDay` - ) { - const aKey = normalizeValue(a) as string - const bKey = normalizeValue(b) as string - order = aKey < bKey ? -1 : aKey > bKey ? 1 : 0 - } else { - return compare(a, b) - } - return opts.direction === `desc` ? -order : order - } -} - -/** Identify a supported Temporal tag for index ordering. */ -export function indexTemporalKind(value: unknown): string | undefined { - return isTemporal(value) ? value[Symbol.toStringTag] : undefined -} - /** * Wraps a user comparator so an index crashes on a result that cannot order * its storage, instead of silently misplacing rows. @@ -252,21 +177,20 @@ export const defaultComparator = makeComparator({ stringSort: `locale`, }) -/** Recognize ordinary Uint8Array values, including Buffers and foreign realms. */ +/** Include host Buffers when the current realm has a different Uint8Array. */ export function isUint8Array(value: unknown): value is Uint8Array { return ( - ArrayBuffer.isView(value) && - Object.prototype.toString.call(value) === `[object Uint8Array]` + value instanceof Uint8Array || + (typeof Buffer !== `undefined` && value instanceof Buffer) ) } /** Compare two Uint8Arrays for content equality. */ function areUint8ArraysEqual(a: Uint8Array, b: Uint8Array): boolean { - const length = a.length - if (length !== b.length) { + if (a.byteLength !== b.byteLength) { return false } - for (let i = 0; i < length; i++) { + for (let i = 0; i < a.byteLength; i++) { if (a[i] !== b[i]) { return false } @@ -280,28 +204,10 @@ function normalizedKey(kind: string, value: string): string { return `${NORMALIZED_KEY_PREFIX}${kind}:${value}` } -const canonicalTemporalZones = new Map() - -function canonicalTemporalZone(zoneId: string): string { - const cached = canonicalTemporalZones.get(zoneId) - if (cached !== undefined) return cached - - let canonical = zoneId - try { - canonical = new Intl.DateTimeFormat(`en-US`, { - timeZone: zoneId, - }).resolvedOptions().timeZone - } catch { - // Keep valid Temporal zones that this host's Intl does not recognize. - } - canonicalTemporalZones.set(zoneId, canonical) - return canonical -} - function normalizeBinary(value: Uint8Array): string { let bytes = `` - for (const byte of value) { - bytes += String.fromCharCode(byte) + for (let index = 0; index < value.byteLength; index++) { + bytes += String.fromCharCode(value[index]!) } return normalizedKey(`binary`, bytes) } @@ -311,7 +217,7 @@ function normalizeBinary(value: Uint8Array): string { * This allows distinguishing between "start from beginning" (undefined parameter) * and "start from the key undefined" (actual undefined value in the tree). */ -export const UNDEFINED_SENTINEL = Symbol(`undefined index value`) +export const UNDEFINED_SENTINEL = normalizedKey(`undefined`, ``) /** * Normalize a value for comparison and Map key usage @@ -332,49 +238,15 @@ export function normalizeValue(value: any): any { return value } - const timestamp = dateTimestamp(value) - if (timestamp !== undefined) return timestamp + if (value instanceof Date) { + return value.getTime() + } if (isTemporal(value)) { - const kind = value[Symbol.toStringTag] - if (kind === `Temporal.ZonedDateTime`) { - const zoned = value as typeof value & { - epochNanoseconds: bigint - timeZoneId: string - calendarId: string - } - // Equality retains the instant, calendar, and canonical zone. A fixed - // +00:00 zone stays distinct from named UTC, while UTC aliases merge. - return normalizedKey( - `temporal`, - JSON.stringify([ - kind, - String(zoned.epochNanoseconds), - canonicalTemporalZone(zoned.timeZoneId), - zoned.calendarId, - ]), - ) - } - if ( - kind === `Temporal.PlainMonthDay` || - kind === `Temporal.PlainYearMonth` - ) { - // Default display omits the reference ISO year/day that native equals() - // compares. The calendar-inclusive display preserves those fields. - const fullText = ( - value.toString as (options: { calendarName: `always` }) => string - )({ calendarName: `always` }) - return normalizedKey(`temporal`, `${kind}:${fullText}`) - } - if (kind === `Temporal.Duration`) { - // Duration's default display balances subsecond units. Equality under - // the #1519 contract instead preserves all ten fields. - const duration = value as typeof value & - Record<(typeof durationFields)[number], number> - const fields = durationFields.map((field) => duration[field]) - return normalizedKey(`temporal`, JSON.stringify([kind, ...fields])) - } - return normalizedKey(`temporal`, `${kind}:${value.toString()}`) + return normalizedKey( + `temporal`, + `${value[Symbol.toStringTag]}:${value.toString()}`, + ) } // Normalize Uint8Arrays/Buffers to a string representation for Map key usage @@ -399,21 +271,6 @@ export function normalizeForBTree(value: any): any { return normalizeValue(value) } -/** Order raw values while keeping BTree's undefined position distinct from its traversal marker. */ -export function indexOrderValue(value: any, exactKey: any): any { - return value === undefined ? exactKey : value -} - -/** Supply custom comparators the Temporal display key used before raw ordering. */ -export function indexCustomComparatorValue(value: unknown): unknown { - return isTemporal(value) - ? normalizedKey( - `temporal`, - `${value[Symbol.toStringTag]}:${value.toString()}`, - ) - : value -} - /** * Compare values using the equality semantics used by Map keys. */ @@ -468,9 +325,8 @@ export function compareTemporalValues(a: unknown, b: unknown): number { * * Temporal types intentionally throw from `valueOf` to prevent silent * miscomparison via the native relational operators — delegate to - * `compareTemporalValues` for them. Ordinary native Dates compare by their - * timestamps, including against numbers and across realms. Other values - * retain the native relational comparison. + * `compareTemporalValues` for them. For everything else (numbers, strings, + * Dates via `valueOf`, etc.) the native operators do the right thing. * * Callers must handle null/undefined themselves — this helper assumes both * arguments are non-null. @@ -479,13 +335,7 @@ export function compareValues(a: unknown, b: unknown): number { if (isTemporal(a) && isTemporal(b)) { return compareTemporalValues(a, b) } - const comparableA = dateTimestamp(a) ?? a - const comparableB = dateTimestamp(b) ?? b - return (comparableA as any) < (comparableB as any) - ? -1 - : (comparableA as any) > (comparableB as any) - ? 1 - : 0 + return (a as any) < (b as any) ? -1 : (a as any) > (b as any) ? 1 : 0 } /** diff --git a/packages/db/src/utils/index-optimization.ts b/packages/db/src/utils/index-optimization.ts index aeed21681d..4ee63233b1 100644 --- a/packages/db/src/utils/index-optimization.ts +++ b/packages/db/src/utils/index-optimization.ts @@ -18,7 +18,7 @@ import { DEFAULT_COMPARE_OPTIONS } from '../utils.js' import { ReverseIndex } from '../indexes/reverse-index.js' import { hasVirtualPropPath } from '../virtual-props.js' -import { dateTimestamp, makeComparator } from './comparison.js' +import { makeComparator } from './comparison.js' import type { CompareOptions } from '../query/builder/types.js' import type { IndexOperation, IndexReader } from '../indexes/base-index.js' import type { BasicExpression } from '../query/ir.js' @@ -40,15 +40,12 @@ export interface OptimizationResult { } /** - * Finds an index that matches a field path. Exact predicates can use a custom - * comparator's equality buckets; ordered traversal needs an index whose order - * is represented by its comparison options. + * Finds an index that matches a given field path */ export function findIndexForField( collection: CollectionLike, fieldPath: Array, compareOptions?: CompareOptions, - requireOrderedTraversal = false, ): IndexReader | undefined { if (hasVirtualPropPath(fieldPath)) { return undefined @@ -61,8 +58,7 @@ export function findIndexForField( for (const index of collection.indexes.values()) { if ( index.matchesField(fieldPath) && - index.matchesCompareOptions(compareOpts) && - (!requireOrderedTraversal || index.supportsRangeOptimization) + index.matchesCompareOptions(compareOpts) ) { if (!index.matchesDirection(compareOpts.direction)) { // Reversing the index also reverses where it keeps nulls, so the @@ -173,7 +169,7 @@ function isRangeOrderingDivergent( if (value === null) return false // Dates order consistently with the evaluator: valid Dates by time, and // invalid Dates as the greatest value under PostgreSQL float semantics. - return dateTimestamp(value) === undefined + return !(value instanceof Date) } default: return false @@ -367,7 +363,7 @@ function optimizeCompoundRangeQuery< for (const [fieldKey, operations] of fieldOperations) { if (operations.length >= 2) { const fieldPath = JSON.parse(fieldKey) as Array - const index = findIndexForField(collection, fieldPath, undefined, true) + const index = findIndexForField(collection, fieldPath) // Only collapse this field into a range query when every bound's domain // orders the same way the index does and the index supports trustworthy @@ -530,12 +526,7 @@ function optimizeSimpleComparison< if (fieldArg && valueArg) { const fieldPath = (fieldArg as any).path - const index = findIndexForField( - collection, - fieldPath, - undefined, - operation !== `eq`, - ) + const index = findIndexForField(collection, fieldPath) if (index) { const queryValue = (valueArg as any).value @@ -611,12 +602,7 @@ function canOptimizeSimpleComparison< } if (fieldPath) { - const index = findIndexForField( - collection, - fieldPath, - undefined, - expression.name !== `eq`, - ) + const index = findIndexForField(collection, fieldPath) return index !== undefined } diff --git a/packages/db/tests/change-event-history-oracle.test.ts b/packages/db/tests/change-event-history-oracle.test.ts index b23ed3b340..005b450e22 100644 --- a/packages/db/tests/change-event-history-oracle.test.ts +++ b/packages/db/tests/change-event-history-oracle.test.ts @@ -1,6 +1,4 @@ -import { runInNewContext } from 'node:vm' import { fc, test as fcTest } from '@fast-check/vitest' -import { Temporal } from 'temporal-polyfill' import { describe, expect, it, vi } from 'vitest' import { BTreeIndex, @@ -12,7 +10,7 @@ import { import { localOnlyCollectionOptions } from '../src/local-only' import { PropRef } from '../src/query/ir' import { oraclePropertyOptions, oracleRuns } from './oracle-config' -import { captureCreatedTransactions, withOracleCleanup } from './utils' +import { captureCreatedTransactions } from './utils' import type { LocalOnlyCollectionUtils } from '../src/local-only' import type { Collection } from '../src/index' import type { BaseIndex } from '../src/indexes/base-index' @@ -876,558 +874,3 @@ describe(`change-event history oracle`, () => { verifyGeneratedHistory, ) }) - -describe(`native-byte change publication`, () => { - /** - * A sync replacement with different bytes changes the visible row, so one - * update must reach a subscriber after commit. The source's one-key sequence - * supplies the expected payload; - * exact index lookup is a separate receiving path. An equal-byte copy then - * remains silent. This bounded history does not claim optimistic writes or - * other typed-array classes. - */ - it.each([ - [`BasicIndex`, BasicIndex], - [`BTreeIndex`, BTreeIndex], - ] as const)(`publishes changed bytes with %s`, async (name, IndexType) => { - type ByteRow = { id: string; value: Uint8Array } - type ByteSync = Parameters[`sync`]>[0] - const bytes = (view: Uint8Array) => Array.from(view) - const before = Uint8Array.of(1) - const after = Uint8Array.of(2) - const equalCopy = Uint8Array.of(2) - let sync!: ByteSync - let unsubscribe: (() => unknown) | undefined - let cleanup: (() => unknown) | undefined - - await withOracleCleanup(async () => { - const collection = createCollection({ - id: `native-byte-publication-${name}`, - getKey: ({ id }) => id, - autoIndex: `off`, - startSync: true, - sync: { - sync: (actions) => { - sync = actions - actions.begin() - actions.write({ - type: `insert`, - value: { id: `a`, value: before }, - }) - actions.commit() - actions.markReady() - }, - }, - }) - cleanup = () => collection.cleanup() - await collection.stateWhenReady() - const index = collection.createIndex((row) => row.value, { - indexType: IndexType, - }) - const batches: Array< - Array<{ - type: string - key: string - previous: Array | undefined - value: Array - }> - > = [] - const subscription = collection.subscribeChanges( - (changes) => { - batches.push( - changes.map((change) => ({ - type: change.type, - key: change.key, - previous: - change.type === `update` - ? bytes(change.previousValue!.value) - : undefined, - value: bytes(change.value.value), - })), - ) - }, - { includeInitialState: false }, - ) - unsubscribe = () => subscription.unsubscribe() - expect(batches).toEqual([]) - - sync.begin() - sync.write({ type: `update`, value: { id: `a`, value: after } }) - await sync.commit() - expect(bytes(collection.get(`a`)!.value)).toEqual([2]) - expect(index.equalityLookup(Uint8Array.of(1))).toEqual(new Set()) - expect(index.equalityLookup(Uint8Array.of(2))).toEqual(new Set([`a`])) - expect(batches).toEqual([ - [{ type: `update`, key: `a`, previous: [1], value: [2] }], - ]) - - sync.begin() - sync.write({ type: `update`, value: { id: `a`, value: equalCopy } }) - await sync.commit() - expect(bytes(collection.get(`a`)!.value)).toEqual([2]) - expect(index.equalityLookup(Uint8Array.of(2))).toEqual(new Set([`a`])) - expect(batches).toEqual([ - [{ type: `update`, key: `a`, previous: [1], value: [2] }], - ]) - }, [() => unsubscribe?.(), () => cleanup?.()]) - }) -}) - -describe(`opaque DataView change publication`, () => { - /** - * A keyless DataView's bytes live outside enumerable row properties, so two - * distinct keyless DataViews remain distinct even when their bytes match. - * One sync history crosses from a typed array to a foreign DataView, then - * replaces that DataView with a fresh foreign one. The independent expected - * sequence is two updates. At each subscriber callback, and again after - * commit, public rows and exact index lookup must agree on that step. - */ - it.each([ - [`BasicIndex`, BasicIndex], - [`BTreeIndex`, BTreeIndex], - ] as const)( - `publishes foreign DataView replacements with %s`, - async (name, IndexType) => { - type ViewRow = { id: string; value: Uint8Array | DataView } - type ViewSync = Parameters[`sync`]>[0] - const before = Uint8Array.of(1) - const [first, second] = runInNewContext( - `[new DataView(new Uint8Array([1]).buffer), - new DataView(new Uint8Array([1]).buffer)]`, - ) as [DataView, DataView] - const labels = new Map([ - [before, `bytes`], - [first, `first`], - [second, `second`], - ]) - let sync!: ViewSync - let unsubscribe: (() => unknown) | undefined - let cleanup: (() => unknown) | undefined - - await withOracleCleanup(async () => { - const collection = createCollection({ - id: `foreign-dataview-publication-${name}`, - getKey: ({ id }) => id, - autoIndex: `off`, - startSync: true, - sync: { - sync: (actions) => { - sync = actions - actions.begin() - actions.write({ - type: `insert`, - value: { id: `a`, value: before }, - }) - actions.commit() - actions.markReady() - }, - }, - }) - cleanup = () => collection.cleanup() - await collection.stateWhenReady() - const index = collection.createIndex((row) => row.value, { - indexType: IndexType, - }) - const batches: Array< - Array<{ - type: string - key: string - previous: string | undefined - value: string | undefined - }> - > = [] - const callbackSnapshots: Array<{ - row: string | undefined - indexKeys: Array - previousIndexKeys: Array - }> = [] - const subscription = collection.subscribeChanges( - (changes) => { - // Publication must expose the new row and index before invoking - // the subscriber, without retaining the old index membership. - // Recording inside the callback distinguishes a torn publication - // that repairs either side only after notification. - const published = collection.get(`a`)?.value - const previous = - changes[0]?.type === `update` - ? changes[0].previousValue!.value - : undefined - callbackSnapshots.push({ - row: labels.get(published), - indexKeys: published ? [...index.equalityLookup(published)] : [], - previousIndexKeys: previous - ? [...index.equalityLookup(previous)] - : [], - }) - batches.push( - changes.map((change) => ({ - type: change.type, - key: String(change.key), - previous: - change.type === `update` - ? labels.get(change.previousValue!.value) - : undefined, - value: labels.get(change.value.value), - })), - ) - }, - { includeInitialState: false }, - ) - unsubscribe = () => subscription.unsubscribe() - expect(batches).toEqual([]) - - sync.begin() - sync.write({ type: `update`, value: { id: `a`, value: first } }) - await sync.commit() - expect(collection.get(`a`)!.value).toBe(first) - expect(index.equalityLookup(before)).toEqual(new Set()) - expect(index.equalityLookup(first)).toEqual(new Set([`a`])) - expect(batches).toEqual([ - [{ type: `update`, key: `a`, previous: `bytes`, value: `first` }], - ]) - expect(callbackSnapshots).toEqual([ - { row: `first`, indexKeys: [`a`], previousIndexKeys: [] }, - ]) - - sync.begin() - sync.write({ type: `update`, value: { id: `a`, value: second } }) - await sync.commit() - expect(collection.get(`a`)!.value).toBe(second) - expect(index.equalityLookup(first)).toEqual(new Set()) - expect(index.equalityLookup(second)).toEqual(new Set([`a`])) - expect(batches).toEqual([ - [{ type: `update`, key: `a`, previous: `bytes`, value: `first` }], - [{ type: `update`, key: `a`, previous: `first`, value: `second` }], - ]) - expect(callbackSnapshots).toEqual([ - { row: `first`, indexKeys: [`a`], previousIndexKeys: [] }, - { row: `second`, indexKeys: [`a`], previousIndexKeys: [] }, - ]) - }, [() => unsubscribe?.(), () => cleanup?.()]) - }, - ) -}) - -describe(`native Date change publication across realms`, () => { - /** - * A settled sync replacement must leave a change-message mirror equal to the - * Collection's public row, as the Collection change-message contract above - * states. The native Date timestamp rule in `docs/guides/live-queries.md` - * includes Dates from another JavaScript realm and invalid Dates. The model - * knows the two supplied times and predicts one update exactly when they - * differ. The driver uses real sync writes and a public - * subscriber; it checks the callback and the settled commit. These fixed - * histories do not claim optimistic writes, indexes, or other hosts. - */ - it.each([ - { - name: `changed Dates in one foreign realm`, - beforeTime: 1, - afterTime: 2, - beforeRealm: `foreign`, - afterRealm: `foreign`, - }, - { - name: `equal Dates in one foreign realm`, - beforeTime: 1, - afterTime: 1, - beforeRealm: `foreign`, - afterRealm: `foreign`, - }, - { - name: `equal local-to-foreign Dates`, - beforeTime: 1, - afterTime: 1, - beforeRealm: `local`, - afterRealm: `foreign`, - }, - { - name: `equal foreign-to-local Dates`, - beforeTime: 1, - afterTime: 1, - beforeRealm: `foreign`, - afterRealm: `local`, - }, - { - name: `equal invalid local-to-foreign Dates`, - beforeTime: null, - afterTime: null, - beforeRealm: `local`, - afterRealm: `foreign`, - }, - { - name: `foreign invalid-to-valid Dates`, - beforeTime: null, - afterTime: 1, - beforeRealm: `foreign`, - afterRealm: `foreign`, - }, - { - name: `foreign valid-to-invalid Dates`, - beforeTime: 1, - afterTime: null, - beforeRealm: `foreign`, - afterRealm: `foreign`, - }, - ] as const)(`keeps the public mirror for $name`, async (sample) => { - type DateRow = { id: string; value: Date } - type DateSync = Parameters[`sync`]>[0] - const beforeTime = sample.beforeTime ?? Number.NaN - const afterTime = sample.afterTime ?? Number.NaN - const [foreignBefore, foreignAfter] = runInNewContext( - `[new Date(beforeTime), new Date(afterTime)]`, - { beforeTime, afterTime }, - ) as [Date, Date] - const before = - sample.beforeRealm === `foreign` ? foreignBefore : new Date(beforeTime) - const after = - sample.afterRealm === `foreign` ? foreignAfter : new Date(afterTime) - // The native intrinsic verifies the host supplied real Dates. The model - // itself uses the constructor inputs, never the production classifier. - expect(Date.prototype.getTime.call(before)).toEqual(beforeTime) - expect(Date.prototype.getTime.call(after)).toEqual(afterTime) - - let sync!: DateSync - let unsubscribe: (() => unknown) | undefined - let cleanup: (() => unknown) | undefined - await withOracleCleanup(async () => { - const collection = createCollection({ - id: `native-date-publication-${sample.name}`, - getKey: ({ id }) => id, - autoIndex: `off`, - startSync: true, - sync: { - sync: (actions) => { - sync = actions - actions.begin() - actions.write({ type: `insert`, value: { id: `a`, value: before } }) - actions.commit() - actions.markReady() - }, - }, - }) - cleanup = () => collection.cleanup() - await collection.stateWhenReady() - - let mirrorTime = beforeTime - const batches: Array< - Array<{ - type: string - previousTime: number | undefined - time: number - }> - > = [] - const callbackTimes: Array = [] - const subscription = collection.subscribeChanges( - (changes) => { - batches.push( - changes.map((change) => ({ - type: change.type, - previousTime: - change.type === `update` - ? Date.prototype.getTime.call(change.previousValue!.value) - : undefined, - time: Date.prototype.getTime.call(change.value.value), - })), - ) - for (const change of changes) { - mirrorTime = Date.prototype.getTime.call(change.value.value) - } - callbackTimes.push( - Date.prototype.getTime.call(collection.get(`a`)!.value), - ) - }, - { includeInitialState: false }, - ) - unsubscribe = () => subscription.unsubscribe() - - sync.begin() - sync.write({ type: `update`, value: { id: `a`, value: after } }) - await sync.commit() - - const changed = !Object.is(beforeTime, afterTime) - expect(batches).toEqual( - changed - ? [[{ type: `update`, previousTime: beforeTime, time: afterTime }]] - : [], - ) - expect(callbackTimes).toEqual(changed ? [afterTime] : []) - expect(mirrorTime).toEqual(afterTime) - expect(Date.prototype.getTime.call(collection.get(`a`)!.value)).toEqual( - afterTime, - ) - }, [() => unsubscribe?.(), () => cleanup?.()]) - }) -}) - -describe(`Temporal.Duration change publication`, () => { - /** - * The Collection change-message contract above requires a mirror built from - * delivered messages to equal the public row after sync commit. The query - * equality contract already distinguishes all ten Duration fields, even - * when default display text agrees. The model supplies fresh ordinary - * Durations with either one second or one thousand milliseconds under a - * shared positive or negative sign, plus one hour versus 60 minutes. It - * predicts an update for each field change and no event for a fresh copy. - * The production driver writes through ready sync Collections; subscriber - * batches, callback-time public rows, the mirror, and settled public rows - * are the observations. These fixed histories do not cover optimistic - * writes, other Temporal kinds, or every possible ten-field transition. - */ - const fields = [ - `years`, - `months`, - `weeks`, - `days`, - `hours`, - `minutes`, - `seconds`, - `milliseconds`, - `microseconds`, - `nanoseconds`, - ] as const - const snapshot = (value: Temporal.Duration): Array => - fields.map((field) => value[field]) - - it.each([ - { - name: `seconds to milliseconds`, - before: { seconds: 1 }, - after: { milliseconds: 1000 }, - changed: true, - sameDisplay: true, - }, - { - name: `milliseconds to seconds`, - before: { milliseconds: 1000 }, - after: { seconds: 1 }, - changed: true, - sameDisplay: true, - }, - { - name: `fresh equal seconds`, - before: { seconds: 1 }, - after: { seconds: 1 }, - changed: false, - sameDisplay: true, - }, - { - name: `fresh equal milliseconds`, - before: { milliseconds: 1000 }, - after: { milliseconds: 1000 }, - changed: false, - sameDisplay: true, - }, - { - name: `negative seconds to milliseconds`, - before: { seconds: -1 }, - after: { milliseconds: -1000 }, - changed: true, - sameDisplay: true, - }, - { - name: `negative milliseconds to seconds`, - before: { milliseconds: -1000 }, - after: { seconds: -1 }, - changed: true, - sameDisplay: true, - }, - { - name: `fresh equal negative seconds`, - before: { seconds: -1 }, - after: { seconds: -1 }, - changed: false, - sameDisplay: true, - }, - { - name: `fresh equal negative milliseconds`, - before: { milliseconds: -1000 }, - after: { milliseconds: -1000 }, - changed: false, - sameDisplay: true, - }, - { - name: `hours to 60 minutes`, - before: { hours: 1 }, - after: { minutes: 60 }, - changed: true, - sameDisplay: false, - }, - ])(`keeps the public mirror for $name`, async (sample) => { - type DurationRow = { id: string; value: Temporal.Duration } - type DurationSync = Parameters[`sync`]>[0] - const before = Temporal.Duration.from(sample.before) - const after = Temporal.Duration.from(sample.after) - expect(before).not.toBe(after) - expect(before.toString() === after.toString()).toBe(sample.sameDisplay) - const beforeFields = snapshot(before) - const afterFields = snapshot(after) - expect( - beforeFields.every((field, index) => field === afterFields[index]), - ).toBe(!sample.changed) - - let sync!: DurationSync - let unsubscribe: (() => unknown) | undefined - let cleanup: (() => unknown) | undefined - await withOracleCleanup(async () => { - const collection = createCollection({ - id: `duration-publication-${sample.name}`, - getKey: ({ id }) => id, - autoIndex: `off`, - startSync: true, - sync: { - sync: (actions) => { - sync = actions - actions.begin() - actions.write({ type: `insert`, value: { id: `a`, value: before } }) - actions.commit() - actions.markReady() - }, - }, - }) - cleanup = () => collection.cleanup() - await collection.stateWhenReady() - - let mirror = beforeFields - const batches: Array< - Array<{ - type: string - previous: Array | undefined - value: Array - }> - > = [] - const callbackRows: Array> = [] - const subscription = collection.subscribeChanges( - (changes) => { - batches.push( - changes.map((change) => ({ - type: change.type, - previous: - change.type === `update` - ? snapshot(change.previousValue!.value) - : undefined, - value: snapshot(change.value.value), - })), - ) - for (const change of changes) mirror = snapshot(change.value.value) - callbackRows.push(snapshot(collection.get(`a`)!.value)) - }, - { includeInitialState: false }, - ) - unsubscribe = () => subscription.unsubscribe() - - sync.begin() - sync.write({ type: `update`, value: { id: `a`, value: after } }) - await sync.commit() - - expect(batches).toEqual( - sample.changed - ? [[{ type: `update`, previous: beforeFields, value: afterFields }]] - : [], - ) - expect(callbackRows).toEqual(sample.changed ? [afterFields] : []) - expect(mirror).toEqual(afterFields) - expect(snapshot(collection.get(`a`)!.value)).toEqual(afterFields) - }, [() => unsubscribe?.(), () => cleanup?.()]) - }) -}) diff --git a/packages/db/tests/comparison-oracle.property.test.ts b/packages/db/tests/comparison-oracle.property.test.ts index 8437082b62..1f8277456f 100644 --- a/packages/db/tests/comparison-oracle.property.test.ts +++ b/packages/db/tests/comparison-oracle.property.test.ts @@ -1,21 +1,15 @@ -import { runInNewContext } from 'node:vm' import { describe, expect, it } from 'vitest' import { fc, test as fcTest } from '@fast-check/vitest' import { Temporal } from 'temporal-polyfill' -import { createCollection } from '../src/collection/index' -import { createLiveQueryCollection } from '../src/query/live-query-collection' import { areValuesEqual, ascComparator, - compareValues, defaultComparator, descComparator, - isUnorderable, makeComparator, normalizeValue, } from '../src/utils/comparison' import { DEFAULT_COMPARE_OPTIONS } from '../src/utils' -import { withOracleCleanup } from './utils' import type { CompareOptions } from '../src/query/builder/types' /** @@ -25,9 +19,8 @@ import type { CompareOptions } from '../src/query/builder/types' * lexical/locale string modes are independent axes. In the bounded default * ascending domain below, NaN and invalid Dates share one non-finite order * position and follow finite values; null remains first. The published NaN - * ordering change in `packages/db/CHANGELOG.md` (#1582) and the live-query - * guide establish that law for Dates with native prototypes regardless of - * their JavaScript realm in the currently broader Date-order lane. + * ordering change in `packages/db/CHANGELOG.md` (#1582) establishes that law + * for ordinary numbers and Dates in the transferred unit-test domain. * * Authority: `OrderByOptions` and `CompareOptions` in the query builder's * option types declare these modes. The index comparator contract accepts @@ -38,12 +31,11 @@ import type { CompareOptions } from '../src/query/builder/types' * intentionally assigns those a creation-order ID. The non-finite model uses * input tags and finite epoch-day values, rather than the production * unorderable classifier. It covers number-only and Date-only sorts with - * optional null. A separate realm-tagged Date model below checks native Date - * timestamps and invalidity across realms, plus one public ordered scan. - * Neither lane claims mixed-type order or every query/index path. Equality laws separately - * cover the normalized value classes and changed-byte controls. Comparator - * signs reject NaN and non-number results before reduction. Signed infinity - * is a valid comparator result, including for custom string collation. + * optional null; it does not claim mixed-type order or query/index paths. + * Equality laws separately cover normalized value classes and changed-byte + * controls. Comparator signs reject NaN and non-number results before + * reduction. Signed infinity is a valid comparator result, including for + * custom string collation. * * Binary normalization has a narrower, established contract: equal byte * contents produce one Map key, different contents remain distinct, and a @@ -52,10 +44,7 @@ import type { CompareOptions } from '../src/query/builder/types' * indexed-byte checks retain the regression contract introduced with #1797; * they constrain this internal utility, not a public serialized format or a * universal performance bound. The 32-character margin is the existing - * constant-overhead budget for the sampled 0..200-byte keys. Changed-byte - * equality and key histories cover lengths 1..200. Fixed changes at byte 64 - * of a 129-byte view and the last byte of a 200-byte view pin positions that - * a short generated campaign need not sample. + * constant-overhead budget for the sampled 0..200-byte keys. */ const defaultOpts: CompareOptions = { @@ -222,9 +211,9 @@ function comparisonProperty< // locale/lexical, and custom-collation cases reconstruct those boundaries. // Array length 0/1 and binary length 0/1/128/129 are marginal cases. Plain // objects are outside this order model because their IDs depend on creation -// order. Mixed-type order is outside this bounded grammar. A separate tagged -// grammar below admits NaN and invalid Date inputs; an invalid comparator -// result is rejected by the control below. +// order. Mixed-type order is outside this bounded grammar. A tagged grammar +// below admits NaN and invalid Date inputs; an invalid comparator result is +// rejected by the control below. const arbitraryComparablePrimitive = fc.oneof( fc.string(), fc.integer(), @@ -336,79 +325,6 @@ const arbitraryNonFiniteInput: fc.Arbitrary = fc.record({ includeNull: fc.boolean(), }) -type RealmDateSample = { - leftTime: number | null - rightTime: number | null - leftForeign: boolean - rightForeign: boolean -} - -function dateInRealm(time: number | null, foreign: boolean): Date { - const timestamp = time ?? Number.NaN - return foreign - ? (runInNewContext(`new Date(timestamp)`, { timestamp }) as Date) - : new Date(timestamp) -} - -// `null` is a model-only invalid-Date tag; production receives a real Date -// whose timestamp is NaN. Two valid tags compare by their supplied timestamp, -// while an invalid tag follows every valid tag. The model never asks the -// production Date detector, normalizer, or comparator for an expected sign. -function modelDateSign(left: number | null, right: number | null): -1 | 0 | 1 { - if (left === null) return right === null ? 0 : 1 - if (right === null) return -1 - return left < right ? -1 : left > right ? 1 : 0 -} - -function assertRealmDateOrder(sample: RealmDateSample): void { - const left = dateInRealm(sample.leftTime, sample.leftForeign) - const right = dateInRealm(sample.rightTime, sample.rightForeign) - const expected = modelDateSign(sample.leftTime, sample.rightTime) - const reverseExpected = expected === 0 ? 0 : -expected - expect(isUnorderable(left)).toBe(sample.leftTime === null) - expect(isUnorderable(right)).toBe(sample.rightTime === null) - expect(sign(ascComparator(left, right, DEFAULT_COMPARE_OPTIONS))).toBe( - expected, - ) - expect(sign(ascComparator(right, left, DEFAULT_COMPARE_OPTIONS))).toBe( - reverseExpected, - ) - // Index equality keys use a Date's timestamp. This separate observation - // prevents a correct direct order from masking reference-key normalization. - expect(normalizeValue(left)).toBe(sample.leftTime ?? Number.NaN) - expect(normalizeValue(right)).toBe(sample.rightTime ?? Number.NaN) -} - -// Legal pairs cross local and foreign native Dates, valid epoch milliseconds -// in -1e12..1e12 and invalid Dates. The fixed witnesses below force the -// foreign-valid and foreign-invalid paths; generated pairs add equal times, -// both operand orders and both invalid values. Caller-modified built-ins, -// proxies, and user-defined Date subclasses are outside this grammar. -const arbitraryDateTime = fc.oneof( - fc.integer({ min: -1_000_000_000_000, max: 1_000_000_000_000 }), - fc.constant(null), -) -const arbitraryRealmDateSample: fc.Arbitrary = fc.oneof( - fc.record({ - leftTime: arbitraryDateTime, - rightTime: arbitraryDateTime, - leftForeign: fc.boolean(), - rightForeign: fc.boolean(), - }), - fc - .record({ - time: arbitraryDateTime, - leftForeign: fc.boolean(), - rightForeign: fc.boolean(), - }) - .map(({ time, leftForeign, rightForeign }) => ({ - leftTime: time, - rightTime: time, - leftForeign, - rightForeign, - })), -) - const arbitraryComparableArray = fc.array(arbitraryComparablePrimitive, { maxLength: 5, }) @@ -477,13 +393,10 @@ const arbitrarySameTypeTriple = fc.oneof( ), ) -// A legal changed-byte input retains length and all but one indexed byte. -// The 1..200 range spans both sides of the 128-byte key-size boundary; the -// empty input has no byte to change and is checked by the size-margin witness. const arbitraryChangedBytes = fc .tuple( - fc.uint8Array({ minLength: 1, maxLength: 200 }), - fc.integer({ min: 0, max: 199 }), + fc.uint8Array({ minLength: 1, maxLength: 50 }), + fc.integer({ min: 0, max: 49 }), fc.integer({ min: 1, max: 255 }), ) .map(([bytes, indexHint, delta]) => { @@ -972,89 +885,6 @@ describe(`NaN and invalid Date default-order refinement`, () => { }) }) -describe(`native Date order across JavaScript realms`, () => { - it(`treats numeric NaN and invalid Dates as one non-finite order position`, () => { - for (const foreign of [false, true]) { - const invalid = dateInRealm(null, foreign) - expect(ascComparator(Number.NaN, invalid, DEFAULT_COMPARE_OPTIONS)).toBe( - 0, - ) - expect(ascComparator(invalid, Number.NaN, DEFAULT_COMPARE_OPTIONS)).toBe( - 0, - ) - } - }) - - it(`orders foreign valid and invalid Dates by the promised timestamp law`, () => { - const jan1 = Date.UTC(2023, 0, 1) - const jan2 = Date.UTC(2023, 0, 2) - assertRealmDateOrder({ - leftTime: jan2, - rightTime: jan1, - leftForeign: true, - rightForeign: false, - }) - assertRealmDateOrder({ - leftTime: null, - rightTime: jan1, - leftForeign: true, - rightForeign: false, - }) - assertRealmDateOrder({ - leftTime: jan1, - rightTime: jan1, - leftForeign: true, - rightForeign: false, - }) - assertRealmDateOrder({ - leftTime: jan1, - rightTime: null, - leftForeign: false, - rightForeign: true, - }) - }) - - comparisonProperty([arbitraryRealmDateSample])( - `local and foreign Dates refine timestamp and invalidity order`, - (sample) => assertRealmDateOrder(sample), - ) - - it(`orders a foreign invalid Date after a valid Date in a public scan`, async () => { - for (const foreign of [false, true]) { - const invalid = dateInRealm(null, foreign) - const valid = new Date(Date.UTC(2023, 0, 1)) - const source = createCollection({ - id: `realm-date-order-${foreign}`, - getKey: (row: { id: number; when: Date }) => row.id, - autoIndex: `off`, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - write({ type: `insert`, value: { id: 1, when: invalid } }) - write({ type: `insert`, value: { id: 2, when: valid } }) - commit() - markReady() - }, - }, - }) - const query = createLiveQueryCollection((q) => - q - .from({ rows: source }) - .orderBy(({ rows }) => rows.when, `asc`) - .select(({ rows }) => ({ id: rows.id, when: rows.when })), - ) - // Preserve an ordered-row mismatch if either resource also fails to - // clean up, and attempt both cleanups in all cases. - await withOracleCleanup(async () => { - await query.preload() - // The local control proves that this public query actually orders - // rows; the foreign case must have the same promised result. - expect([...query.values()].map(({ id }) => id)).toEqual([2, 1]) - }, [() => query.cleanup(), () => source.cleanup()]) - } - }) -}) - describe(`descComparator property-based tests`, () => { comparisonProperty([arbitrarySameTypePair])( `descComparator reverses ascComparator ordering (same types)`, @@ -1250,8 +1080,8 @@ describe(`areValuesEqual property-based tests`, () => { }) /** - * Temporal ordering uses the value's calendar position or instant, not its - * display text. The published #1519 contract gives PlainDate calendar order + * Temporal ordering uses calendar position or instant. The published #1519 + * contract gives PlainDate calendar order * and ZonedDateTime instant order even when zones differ. These models use * supplied ISO date fields and epoch-nanosecond offsets; neither asks the * production comparator or Temporal's static compare for an expected sign. @@ -1259,14 +1089,9 @@ describe(`areValuesEqual property-based tests`, () => { * Legal inputs are ISO PlainDates from years 2000..2030 with days 1..28 and * ZonedDateTimes within two seconds of a fixed instant in four offset zones. * Equal-value branches construct separate objects, and the zoned branch forces - * different zones at the same instant. A normal ZonedDateTime subclass - * inherits its Temporal methods and retains the same instant order. - * The production paths are the exported - * `ascComparator` under the default options and `compareValues` in hostile - * dispatch histories. Each return is the observation checkpoint; - * this lane makes no indexed or public-query claim. Comparator - * zero for same-instant ZonedDateTimes is distinct from `eq`/`in` identity: - * the zone remains part of identity under the #1519 contract. + * different zones at the same instant. The production path is the exported + * `ascComparator` under default options; each return is the observation + * checkpoint. This lane makes no indexed or public-query claim. */ type IsoDateFields = { year: number; month: number; day: number } type PlainDateOrderInput = { left: IsoDateFields; right: IsoDateFields } @@ -1476,91 +1301,6 @@ describe(`Temporal default-order refinement`, () => { } expect(() => assertZonedDateTimeOrder(sample, truncated)).toThrow() }) - - it.each([ - [ - `ascComparator`, - (left: unknown, right: unknown) => - ascComparator(left, right, DEFAULT_COMPARE_OPTIONS), - ], - [`compareValues`, compareValues], - ] as const)( - `keeps a ZonedDateTime subclass in instant order through %s`, - (_name, compare) => { - class SubclassZonedDateTime extends Temporal.ZonedDateTime {} - const subclass = new SubclassZonedDateTime( - zonedBaseEpochNanoseconds, - `+00:00`, - ) - const peer = zonedValue(0, `+05:30`) - const later = zonedValue(1, `+00:00`) - - // The model position comes from the supplied nanosecond offset. - expect(Temporal.ZonedDateTime.compare(subclass, peer)).toBe(0) - expect(Temporal.ZonedDateTime.compare(subclass, later)).toBe(-1) - expect(compare(subclass, peer)).toBe(0) - expect(compare(peer, subclass)).toBe(0) - expect(sign(compare(subclass, later))).toBe(-1) - expect(sign(compare(later, subclass))).toBe(1) - }, - ) -}) - -describe(`native binary key content`, () => { - it(`distinguishes a changed interior byte at the normalized-key return`, () => { - // Equal lengths and endpoints cannot conceal an interior byte change. - // Native byte content, not the production encoding, defines inequality. - const left = Uint8Array.of(1, 0, 2) - const right = Uint8Array.of(1, 9, 2) - expect(normalizeValue(left)).not.toBe(normalizeValue(right)) - }) - - it(`distinguishes interior and last bytes of long native views`, () => { - // The supplied byte at each named position changes while length and all - // other bytes stay equal. At each function return, equality must reject - // the pair in both directions and normalization must produce distinct keys. - for (const { length, index } of [ - { length: 129, index: 64 }, - { length: 200, index: 199 }, - ]) { - const bytes = Uint8Array.from({ length }, (_, byteIndex) => byteIndex) - const changed = new Uint8Array(bytes) - changed[index] = (changed[index]! + 1) % 256 - const sample = { bytes, changed, index } - - // A plausible first-64-byte comparator passes every short pair but - // fails the same changed-byte checker used by the generated equality - // property. This fault injection reaches the intended return check. - expect(() => - assertChangedBytes( - sample, - (left, right) => - left.length === right.length && - left - .subarray(0, 64) - .every((byte, byteIndex) => byte === right[byteIndex]), - ), - ).toThrow(`changed byte must be unequal`) - - expect(areValuesEqual(bytes, new Uint8Array(bytes))).toBe(true) - expect(areValuesEqual(bytes, changed)).toBe(false) - expect(areValuesEqual(changed, bytes)).toBe(false) - expect(normalizeValue(bytes)).not.toBe(normalizeValue(changed)) - } - }) - - comparisonProperty([arbitraryChangedBytes])( - `a changed native byte changes the normalized key`, - (sample) => { - // Reuse the independently authored changed-byte history. This - // property has its own assertion site so a shrunk failure identifies - // key normalization rather than direct byte equality. - assertChangedBytes( - sample, - (left, right) => normalizeValue(left) === normalizeValue(right), - ) - }, - ) }) if (requestedProperty !== undefined) { diff --git a/packages/db/tests/custom-index-order-selection-oracle.test.ts b/packages/db/tests/custom-index-order-selection-oracle.test.ts deleted file mode 100644 index a67973c1d6..0000000000 --- a/packages/db/tests/custom-index-order-selection-oracle.test.ts +++ /dev/null @@ -1,1234 +0,0 @@ -import { Buffer } from 'node:buffer' -import { describe, expect, it, vi } from 'vitest' -import { createCollection } from '../src/collection/index.js' -import { BasicIndex } from '../src/indexes/basic-index.js' -import { BTreeIndex } from '../src/indexes/btree-index.js' -import { ReverseIndex } from '../src/indexes/reverse-index.js' -import { Func, PropRef, Value } from '../src/query/ir.js' -import { DEFAULT_COMPARE_OPTIONS } from '../src/utils.js' -import { withOracleCleanup } from './utils.js' -import type { SyncConfig } from '../src/types.js' - -/** - * An ordered Collection snapshot may use an index only when the index executes - * the requested comparison. The live-query ARCHITECTURE.md Identity contract - * says that advertised comparison options define the executable comparator. - * A custom index comparator has no identity in the ordinary ascending query - * options, so it cannot supply that query's order. Exact equality lookup on - * the index remains a separate, valid operation. - * - * This bounded witness compares public snapshots of three unique numeric rows - * in an unindexed Collection and a Collection with a descending custom index. - * It also checks a custom index followed by a compatible ordinary index, and - * eager index creation when the custom index is present first. Numeric range - * predicates must likewise find a later compatible index; a range subscriber - * must trigger eager creation when only a custom index exists, whichever side - * of a single or paired relational predicate holds the field. It covers - * BasicIndex and BTreeIndex at the ready checkpoint, with one order term and - * limit one. A neighboring mixed Date/number snapshot compares public scan - * order before index creation with optimized order afterward. It promises - * index installation preserves that order, without prescribing a general - * cross-family order. Changelog #958 and the temporal index order oracle - * establish key-ascending ties in public ordered snapshots. The witnesses - * below add equal numeric and custom-collation values, nullish values, finite - * windows, and unbounded reads. A direct on-demand ordered subscription - * checks initial and cursor snapshots of an equal-rank group. Raw escaped - * strings must reach custom-collation callbacks without their exact-key - * encoding; direct range and cursor operations use those same positions. - * Native byte views retain content-based exact equality, and indexed public - * order agrees with the unindexed scan without promising a particular default - * byte-content order. A source history replaces one byte view with another - * row's value, then deletes and reinserts its key. Exact membership follows - * byte content, while public scan and indexed reads have the same complete - * order and conserve current keys. A `coalesce` wrapper around the scan's - * non-null byte field prevents the single-ref index shortcut. - * The maintainer's custom-comparator decision also requires each current row - * to follow the callback when equal-content views have distinct callback ranks. - * They do not claim arbitrary mutation histories, provider ordering, arbitrary - * binary values, or live-query publication. - */ -type Row = { id: string; rank: number } - -const rows: ReadonlyArray = [ - { id: `middle`, rank: 20 }, - { id: `last`, rank: 30 }, - { id: `first`, rank: 10 }, -] - -// The reference sorts source facts numerically. It knows neither index order -// nor the index-selection code; the first public key must be `first`. -const expectedFirstKey = [...rows].sort((a, b) => a.rank - b.rank)[0]!.id -const greaterThanTen = new Func(`gt`, [new PropRef([`rank`]), new Value(10)]) -const tenLessThanRank = new Func(`lt`, [new Value(10), new PropRef([`rank`])]) -const twentyLessThanOrEqualRank = new Func(`lte`, [ - new Value(20), - new PropRef([`rank`]), -]) -const twentyGreaterThanOrEqualRank = new Func(`gte`, [ - new Value(20), - new PropRef([`rank`]), -]) -const expectedGreaterThanTen = rows - .filter(({ rank }) => rank > 10) - .map(({ id }) => id) - .sort() -const reversedBetweenTenAndThirty = new Func(`and`, [ - tenLessThanRank, - new Func(`gt`, [new Value(30), new PropRef([`rank`])]), -]) -const expectedBetweenTenAndThirty = rows - .filter(({ rank }) => 10 < rank && rank < 30) - .map(({ id }) => id) - .sort() -const expectedAtLeastTwenty = rows - .filter(({ rank }) => 20 <= rank) - .map(({ id }) => id) - .sort() -const expectedAtMostTwenty = rows - .filter(({ rank }) => 20 >= rank) - .map(({ id }) => id) - .sort() -const inclusiveReversedBetweenTwentyAndTwentyFive = new Func(`and`, [ - twentyLessThanOrEqualRank, - new Func(`gte`, [new Value(25), new PropRef([`rank`])]), -]) -const expectedBetweenTwentyAndTwentyFive = rows - .filter(({ rank }) => 20 <= rank && rank <= 25) - .map(({ id }) => id) - .sort() -const betweenTenAndThirty = new Func(`and`, [ - greaterThanTen, - new Func(`lt`, [new PropRef([`rank`]), new Value(30)]), -]) -const ascendingFirst = { - orderBy: [ - { - expression: new PropRef([`rank`]), - compareOptions: DEFAULT_COMPARE_OPTIONS, - }, - ], - limit: 1, -} -const descendingFirst = { - ...ascendingFirst, - orderBy: [ - { - expression: new PropRef([`rank`]), - compareOptions: { - ...DEFAULT_COMPARE_OPTIONS, - direction: `desc` as const, - }, - }, - ], -} - -function collectionWithRows( - id: string, - defaultIndexType?: typeof BasicIndex | typeof BTreeIndex, -) { - return createCollection({ - id, - getKey: (row) => row.id, - autoIndex: defaultIndexType ? `eager` : `off`, - defaultIndexType, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of rows) write({ type: `insert`, value: row }) - commit() - markReady() - }, - }, - }) -} - -// Construction belongs inside the protected check. Register each Collection -// immediately so a later setup failure still releases the earlier ones. -async function withCreatedCollections( - check: (create: typeof collectionWithRows) => Promise, -): Promise { - const cleanups: Array<() => unknown> = [] - await withOracleCleanup( - () => - check((id, defaultIndexType) => { - const collection = collectionWithRows(id, defaultIndexType) - cleanups.unshift(() => collection.cleanup()) - return collection - }), - cleanups, - ) -} - -const indexTypes = [ - [`BasicIndex`, BasicIndex], - [`BTreeIndex`, BTreeIndex], -] as const - -describe.each(indexTypes)( - `%s custom comparator order selection`, - (name, IndexType) => { - it(`selects a later compatible index while exact equality keeps the first custom index`, async () => { - await withCreatedCollections(async (create) => { - const collection = create(`custom-index-coexistence-${name}`) - await collection.stateWhenReady() - const custom = collection.createIndex((row) => row.rank, { - indexType: IndexType, - options: { compareFn: (a: number, b: number) => b - a }, - }) - const ordinary = collection.createIndex((row) => row.rank, { - indexType: IndexType, - }) - - // Legal declaration order matters to first-match index search. The - // custom index is first and still serves exact equality; the ordinary - // index is second and must serve the ordered read. - const lookup = vi.spyOn(custom, `lookup`) - const ordinaryTake = vi.spyOn(ordinary, `takeFromStart`) - const ordinaryReverseTake = vi.spyOn(ordinary, `takeReversedFromEnd`) - const ordinaryLookup = vi.spyOn(ordinary, `lookup`) - const ordinaryRange = vi.spyOn(ordinary, `rangeQuery`) - expect( - collection - .currentStateAsChanges({ - where: new Func(`eq`, [new PropRef([`rank`]), new Value(20)]), - optimizedOnly: true, - }) - ?.map(({ key }) => key), - ).toEqual([`middle`]) - expect(lookup).toHaveBeenCalledWith(`eq`, 20) - expect( - collection - .currentStateAsChanges({ ...ascendingFirst, optimizedOnly: true }) - ?.map(({ key }) => key), - ).toEqual([expectedFirstKey]) - expect(ordinaryTake).toHaveBeenCalled() - expect( - collection - .currentStateAsChanges({ ...descendingFirst, optimizedOnly: true }) - ?.map(({ key }) => key), - ).toEqual([`last`]) - expect(ordinaryReverseTake).toHaveBeenCalled() - - // Ordinary numeric predicates use relational order rather than the - // custom index's descending walk. Both single and paired bounds must - // search past the custom index to use the ordinary one. - expect( - collection - .currentStateAsChanges({ - where: greaterThanTen, - optimizedOnly: true, - }) - ?.map(({ key }) => key) - .sort(), - ).toEqual([`last`, `middle`]) - expect(ordinaryLookup).toHaveBeenCalledWith(`gt`, 10) - ordinaryRange.mockClear() - expect( - collection - .currentStateAsChanges({ - where: betweenTenAndThirty, - optimizedOnly: true, - }) - ?.map(({ key }) => key), - ).toEqual([`middle`]) - expect(ordinaryRange).toHaveBeenCalled() - expect(lookup).not.toHaveBeenCalledWith(`gt`, 10) - }) - }) - - it(`creates a compatible eager index after an existing custom index`, async () => { - await withCreatedCollections(async (create) => { - const collection = create(`custom-index-eager-${name}`, IndexType) - await collection.stateWhenReady() - collection.createIndex((row) => row.rank, { - indexType: IndexType, - options: { compareFn: (a: number, b: number) => b - a }, - }) - expect(collection.indexes.size).toBe(1) - - // The custom index cannot satisfy the ordinary order request. Eager - // indexing must add a compatible one before optimizedOnly reads it. - expect( - collection - .currentStateAsChanges({ ...ascendingFirst, optimizedOnly: true }) - ?.map(({ key }) => key), - ).toEqual([expectedFirstKey]) - expect(collection.indexes.size).toBe(2) - collection.currentStateAsChanges({ - ...ascendingFirst, - optimizedOnly: true, - }) - expect(collection.indexes.size).toBe(2) - }) - }) - - it(`creates a compatible eager index for reversed single and paired ranges`, async () => { - const forms = [ - { - name: `field-left`, - where: greaterThanTen, - expectedKeys: expectedGreaterThanTen, - }, - { - name: `field-right`, - where: tenLessThanRank, - expectedKeys: expectedGreaterThanTen, - }, - { - name: `paired-field-right`, - where: reversedBetweenTenAndThirty, - expectedKeys: expectedBetweenTenAndThirty, - }, - { - name: `field-right-inclusive-lower`, - where: twentyLessThanOrEqualRank, - expectedKeys: expectedAtLeastTwenty, - }, - { - name: `field-right-inclusive-upper`, - where: twentyGreaterThanOrEqualRank, - expectedKeys: expectedAtMostTwenty, - }, - { - name: `paired-field-right-inclusive`, - where: inclusiveReversedBetweenTwentyAndTwentyFive, - expectedKeys: expectedBetweenTwentyAndTwentyFive, - }, - ] as const - await withCreatedCollections(async (create) => { - const collections = forms.map(({ name: formName }) => - create(`custom-index-eager-range-${name}-${formName}`, IndexType), - ) - await Promise.all( - collections.map((collection) => collection.stateWhenReady()), - ) - const observations = [] - for (const [index, form] of forms.entries()) { - const collection = collections[index]! - collection.createIndex((row) => row.rank, { - indexType: IndexType, - options: { compareFn: (a: number, b: number) => b - a }, - }) - const initialKeys: Array = [] - const subscription = collection.subscribeChanges( - (changes) => initialKeys.push(...changes.map(({ key }) => key)), - { whereExpression: form.where, includeInitialState: true }, - ) - try { - observations.push({ - form: form.name, - initialKeys: initialKeys.sort(), - optimizedKeys: collection - .currentStateAsChanges({ - where: form.where, - optimizedOnly: true, - }) - ?.map(({ key }) => key) - .sort(), - indexCount: collection.indexes.size, - }) - } finally { - subscription.unsubscribe() - } - } - // The first two expressions denote the same numeric relation; the - // paired and inclusive forms exercise both bound directions. Initial - // publication and optimized-only snapshots must follow the independent - // row filters after each subscription's eager index construction. - expect(observations).toEqual( - forms.map(({ name: formName, expectedKeys }) => ({ - form: formName, - initialKeys: expectedKeys, - optimizedKeys: expectedKeys, - indexCount: 2, - })), - ) - }) - }) - - it(`keeps a descending custom index out of an ascending public snapshot`, async () => { - await withCreatedCollections(async (create) => { - const scanned = create(`custom-index-order-scan-${name}`) - const indexed = create(`custom-index-order-indexed-${name}`) - const ordinary = create(`custom-index-order-ordinary-${name}`) - await Promise.all([ - scanned.stateWhenReady(), - indexed.stateWhenReady(), - ordinary.stateWhenReady(), - ]) - const customIndex = indexed.createIndex((row) => row.rank, { - indexType: IndexType, - options: { compareFn: (a: number, b: number) => b - a }, - }) - const ordinaryIndex = ordinary.createIndex((row) => row.rank, { - indexType: IndexType, - }) - - // Legal history: install the same three rows, then add the custom index. - // The exact lookup control checks that excluding the index from order - // selection does not change its exact-value membership. - expect(customIndex.equalityLookup(20)).toEqual(new Set([`middle`])) - expect(customIndex.lookup(`eq`, 20)).toEqual(new Set([`middle`])) - expect(customIndex.supportsRangeOptimization).toBe(false) - expect(ordinaryIndex.supportsRangeOptimization).toBe(true) - expect( - indexed - .currentStateAsChanges({ - where: new Func(`eq`, [new PropRef([`rank`]), new Value(20)]), - optimizedOnly: true, - }) - ?.map(({ key }) => key), - ).toEqual([`middle`]) - - // Both reads are public Collection snapshots after readiness. The - // unindexed read is a second formulation of the same first-row law. - const scannedKeys = scanned - .currentStateAsChanges(ascendingFirst) - ?.map(({ key }) => key) - const indexedKeys = indexed - .currentStateAsChanges(ascendingFirst) - ?.map(({ key }) => key) - expect(scannedKeys).toEqual([expectedFirstKey]) - expect(indexedKeys).toEqual(scannedKeys) - expect( - ordinary - .currentStateAsChanges({ ...ascendingFirst, optimizedOnly: true }) - ?.map(({ key }) => key), - ).toEqual(scannedKeys) - - // A read that forbids a scan must decline the custom index. Returning - // `last` here is the hostile design this witness rejects. - expect( - indexed.currentStateAsChanges({ - ...ascendingFirst, - optimizedOnly: true, - }), - ).toBeUndefined() - // A range read also cannot use the custom comparator when it is the - // only index, even if no later ordinary index is available. - expect( - indexed.currentStateAsChanges({ - where: greaterThanTen, - optimizedOnly: true, - }), - ).toBeUndefined() - expect( - indexed.currentStateAsChanges({ - where: betweenTenAndThirty, - optimizedOnly: true, - }), - ).toBeUndefined() - - // Reversing the advertised direction would also reverse the unknown - // custom comparator. The fallback and ordinary reversed index must - // still agree with an unindexed descending read. - const scannedDescending = scanned - .currentStateAsChanges(descendingFirst) - ?.map(({ key }) => key) - expect(scannedDescending).toEqual([`last`]) - expect( - indexed.currentStateAsChanges(descendingFirst)?.map(({ key }) => key), - ).toEqual(scannedDescending) - expect( - ordinary - .currentStateAsChanges({ ...descendingFirst, optimizedOnly: true }) - ?.map(({ key }) => key), - ).toEqual(scannedDescending) - expect( - indexed.currentStateAsChanges({ - ...descendingFirst, - optimizedOnly: true, - }), - ).toBeUndefined() - }) - }) - }, -) - -describe.each(indexTypes)(`%s public mixed-value order`, (name, IndexType) => { - it(`keeps a Date and number in the same order after indexing`, async () => { - // The public cross-family order is not prescribed. Installing an index - // must preserve the scan's full order and first-row window for these - // immutable source facts. The index call proves the second read reaches - // the optimized path rather than another scan. - type MixedRow = { id: string; rank: number | Date } - const mixedRows: ReadonlyArray = [ - { id: `date`, rank: new Date(1) }, - { id: `equal-number`, rank: 1 }, - { id: `number`, rank: 2 }, - ] - let cleanup: (() => unknown) | undefined - await withOracleCleanup(async () => { - const collection = createCollection({ - id: `mixed-date-number-order-${name}`, - getKey: ({ id }) => id, - autoIndex: `off`, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of mixedRows) write({ type: `insert`, value: row }) - commit() - markReady() - }, - }, - }) - cleanup = () => collection.cleanup() - await collection.stateWhenReady() - const orderBy = [ - { - expression: new PropRef([`rank`]), - compareOptions: DEFAULT_COMPARE_OPTIONS, - }, - ] - const keys = (limit?: number, optimizedOnly?: boolean) => - collection - .currentStateAsChanges({ orderBy, limit, optimizedOnly }) - ?.map(({ key }) => key) - - const scanned = keys() - const scannedFirst = keys(1) - expect(new Set(scanned)).toEqual( - new Set([`date`, `equal-number`, `number`]), - ) - expect(scannedFirst).toEqual(scanned?.slice(0, 1)) - const index = collection.createIndex((row) => row.rank, { - indexType: IndexType, - }) - const take = vi.spyOn(index, `takeFromStart`) - // Query equality puts the Date and its numeric timestamp in one exact - // class, although their public ordered positions are different. - expect(index.equalityLookup(1)).toEqual(new Set([`date`, `equal-number`])) - expect(keys(undefined, true)).toEqual(scanned) - expect(take).toHaveBeenCalled() - take.mockClear() - expect(keys(1, true)).toEqual(scannedFirst) - expect(take).toHaveBeenCalled() - }, [() => cleanup?.()]) - }) -}) - -describe.each(indexTypes)(`%s public key ties`, (name, IndexType) => { - it(`publishes key-first ties through limited ordered subscriptions`, async () => { - // A direct ordered subscription is a second public receiving path. The - // first history begins inside a tie; the second first emits a unique rank - // and then continues into a tied rank. The provider accepts each request - // immediately, so callbacks are observed after each local snapshot. - const observe = async ( - suffix: string, - source: ReadonlyArray, - requests: ReadonlyArray<{ minValues?: Array }>, - ) => { - const batches: Array> = [] - const collection = createCollection({ - id: `subscription-public-ties-${name}-${suffix}`, - getKey: ({ id }) => id, - syncMode: `on-demand`, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of source) write({ type: `insert`, value: row }) - commit() - markReady() - return { loadSubset: () => true } - }, - }, - }) - await withOracleCleanup(async () => { - await collection.stateWhenReady() - const index = collection.createIndex((row) => row.rank, { - indexType: IndexType, - }) - const subscription = collection.subscribeChanges( - (changes) => { - if (changes.length) batches.push(changes.map(({ key }) => key)) - }, - { includeInitialState: false }, - ) - try { - subscription.setOrderByIndex(new ReverseIndex(index, true)) - for (const request of requests) { - subscription.requestLimitedSnapshot({ - orderBy: descendingFirst.orderBy, - limit: 1, - ...request, - }) - } - } finally { - subscription.unsubscribe() - } - }, [() => collection.cleanup()]) - return batches - } - - // Numeric rank descending, then source key ascending, predicts `a` in - // the initial tie. The unique rank-2 row is an adjacent non-tie control; - // a cursor at rank 2 then reaches the complete rank-1 tie group. - expect( - await observe( - `initial`, - [ - { id: `c`, rank: 1 }, - { id: `b`, rank: 1 }, - { id: `a`, rank: 1 }, - ], - [{}], - ), - ).toEqual([[`a`]]) - expect( - await observe( - `cursor`, - [ - { id: `z`, rank: 2 }, - { id: `c`, rank: 1 }, - { id: `b`, rank: 1 }, - { id: `a`, rank: 1 }, - ], - [{}, { minValues: [2] }], - ), - ).toEqual([[`z`], [`a`]]) - }) - it(`uses ascending keys within numeric ties in both order directions`, async () => { - type TiedRow = { id: string | number; rank: number } - const source: ReadonlyArray = [ - { id: 2, rank: 1 }, - { id: 1, rank: 1 }, - { id: `c`, rank: 1 }, - { id: `b`, rank: 1 }, - { id: `a`, rank: 1 }, - { id: `z`, rank: 0 }, - { id: `y`, rank: 0 }, - ] - let cleanup: (() => unknown) | undefined - await withOracleCleanup(async () => { - const collection = createCollection({ - id: `numeric-public-ties-${name}`, - getKey: ({ id }) => id, - autoIndex: `off`, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of source) write({ type: `insert`, value: row }) - commit() - markReady() - }, - }, - }) - cleanup = () => collection.cleanup() - await collection.stateWhenReady() - - // The independent reference sorts source facts by numeric rank, then - // by source key: strings before numbers, each in ascending order. - // Reverse admission distinguishes that law from a stable scan. A - // descending window cuts through the rank-1 group and the string-to- - // number boundary within it. - const keyOrder = (left: string | number, right: string | number) => - typeof left !== typeof right - ? typeof left === `string` - ? -1 - : 1 - : left < right - ? -1 - : left > right - ? 1 - : 0 - const expected = (direction: `asc` | `desc`, limit?: number) => - [...source] - .sort( - (left, right) => - (direction === `asc` - ? left.rank - right.rank - : right.rank - left.rank) || keyOrder(left.id, right.id), - ) - .slice(0, limit) - .map(({ id }) => id) - const read = ( - direction: `asc` | `desc`, - limit?: number, - optimizedOnly?: boolean, - ) => - collection - .currentStateAsChanges({ - orderBy: [ - { - expression: new PropRef([`rank`]), - compareOptions: { ...DEFAULT_COMPARE_OPTIONS, direction }, - }, - ], - limit, - optimizedOnly, - }) - ?.map(({ key }) => key) - const cases = [ - { direction: `asc`, limit: 1 }, - { direction: `asc`, limit: 3 }, - { direction: `desc`, limit: 1 }, - { direction: `desc`, limit: 2 }, - { direction: `desc`, limit: 4 }, - { direction: `desc`, limit: undefined }, - ] as const - - // Before index creation, the public scan is one receiving formulation. - for (const { direction, limit } of cases) { - expect(read(direction, limit)).toEqual(expected(direction, limit)) - } - const index = collection.createIndex((row) => row.rank, { - indexType: IndexType, - }) - const take = vi.spyOn(index, `takeFromStart`) - const reverseTake = vi.spyOn(index, `takeReversedFromEnd`) - // optimizedOnly forbids scan fallback. The take spies show that the - // index path supplies the same public observation after readiness. - for (const { direction, limit } of cases) { - expect(read(direction, limit, true)).toEqual(expected(direction, limit)) - } - expect(take).toHaveBeenCalled() - expect(reverseTake).toHaveBeenCalled() - }, [() => cleanup?.()]) - }) - - it(`uses source keys within custom-collation and nullish ties`, async () => { - type LabelRow = { id: string; label: string | null | undefined } - const source: ReadonlyArray = [ - { id: `b`, label: `yy` }, - { id: `a`, label: `xx` }, - { id: `n`, label: null }, - { id: `m`, label: undefined }, - { id: `c`, label: `z` }, - ] - const compare = (left: string, right: string) => left.length - right.length - let cleanup: (() => unknown) | undefined - await withOracleCleanup(async () => { - const collection = createCollection({ - id: `custom-public-ties-${name}`, - getKey: ({ id }) => id, - autoIndex: `off`, - defaultStringCollation: { stringSort: `custom`, compare }, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of source) write({ type: `insert`, value: row }) - commit() - markReady() - }, - }, - }) - cleanup = () => collection.cleanup() - await collection.stateWhenReady() - - // Null and undefined share one position. The callback compares only - // non-null strings by length. The independent reference applies the - // public key rule to both comparator-equal groups. - const expected = (direction: `asc` | `desc`, limit?: number) => - [...source] - .sort((left, right) => { - const order = - left.label == null - ? right.label == null - ? 0 - : -1 - : right.label == null - ? 1 - : direction === `asc` - ? left.label.length - right.label.length - : right.label.length - left.label.length - return ( - order || (left.id < right.id ? -1 : left.id > right.id ? 1 : 0) - ) - }) - .slice(0, limit) - .map(({ id }) => id) - const read = ( - direction: `asc` | `desc`, - limit?: number, - optimizedOnly?: boolean, - ) => - collection - .currentStateAsChanges({ - orderBy: [ - { - expression: new PropRef([`label`]), - compareOptions: { direction, nulls: `first` }, - }, - ], - limit, - optimizedOnly, - }) - ?.map(({ key }) => key) - const cases = [ - { direction: `asc`, limit: 1 }, - { direction: `asc`, limit: 4 }, - { direction: `desc`, limit: 3 }, - { direction: `desc`, limit: undefined }, - ] as const - - for (const { direction, limit } of cases) { - expect(read(direction, limit)).toEqual(expected(direction, limit)) - } - const index = collection.createIndex((row) => row.label, { - indexType: IndexType, - options: { - compareOptions: { - direction: `asc`, - nulls: `first`, - stringSort: `custom`, - compare, - }, - }, - }) - const take = vi.spyOn(index, `takeFromStart`) - const reverseTake = vi.spyOn(index, `takeReversedFromEnd`) - for (const { direction, limit } of cases) { - expect(read(direction, limit, true)).toEqual(expected(direction, limit)) - } - expect(take).toHaveBeenCalled() - expect(reverseTake).toHaveBeenCalled() - - // A cursor at `xx` names the callback's length-two position. Its - // comparator-equal `yy` peer remains eligible even though exact string - // equality would find only the already-sent `xx` row. - const batches: Array> = [] - const subscription = collection.subscribeChanges( - (changes) => { - if (changes.length) batches.push(changes.map(({ key }) => key)) - }, - { includeInitialState: false }, - ) - try { - subscription.setOrderByIndex(new ReverseIndex(index, true)) - const orderBy = [ - { - expression: new PropRef([`label`]), - compareOptions: { - direction: `desc` as const, - nulls: `first` as const, - }, - }, - ] - subscription.requestLimitedSnapshot({ orderBy, limit: 3 }) - subscription.requestLimitedSnapshot({ - orderBy, - limit: 1, - minValues: [`xx`], - }) - expect(batches).toEqual([[`m`, `n`, `a`], [`b`]]) - } finally { - subscription.unsubscribe() - } - }, [() => cleanup?.()]) - }) - - it(`passes raw escaped strings to a direct custom comparator`, () => { - const source = [ - { id: `a`, label: `abcdefghijklmnopq` }, - { id: `b`, label: `\u0000tanstack-db:x` }, - { id: `c`, label: `abcdefghij` }, - ] - const index = new IndexType(1, new PropRef([`label`]), undefined, { - compareFn: (left: string, right: string) => left.length - right.length, - }) - for (const row of source) index.add(row.id, row) - - // This callback's law is raw length: 10 < 14 < 17. Exact equality has - // its own normalized key, while every direct traversal boundary uses the - // callback position. Soft comparisons record all boundary failures on - // the original implementation before any production edit. - expect.soft(index.takeFromStart(3)).toEqual([`c`, `b`, `a`]) - expect.soft(index.take(3, source[2]!.label)).toEqual([`b`, `a`]) - expect.soft(index.take(3, source[0]!.label)).toEqual([]) - expect - .soft(index.rangeQuery({ from: source[2]!.label, to: source[0]!.label })) - .toEqual(new Set([`c`, `b`, `a`])) - expect(index.equalityLookup(source[1]!.label)).toEqual(new Set([`b`])) - }) - - it(`orders escaped strings by their raw custom-collation values`, async () => { - type LabelRow = { id: string; label: string } - const source: ReadonlyArray = [ - { id: `a`, label: `abcdefghijklmnopq` }, - { id: `b`, label: `\u0000tanstack-db:x` }, - { id: `c`, label: `abcdefghij` }, - ] - const compare = (left: string, right: string) => left.length - right.length - const model = [...source] - .sort( - (left, right) => - left.label.length - right.label.length || - (left.id < right.id ? -1 : left.id > right.id ? 1 : 0), - ) - .map(({ id }) => id) - let cleanup: (() => unknown) | undefined - await withOracleCleanup(async () => { - const collection = createCollection({ - id: `escaped-custom-collation-${name}`, - getKey: ({ id }) => id, - autoIndex: `off`, - defaultStringCollation: { stringSort: `custom`, compare }, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of source) write({ type: `insert`, value: row }) - commit() - markReady() - }, - }, - }) - cleanup = () => collection.cleanup() - await collection.stateWhenReady() - const orderBy = [ - { - expression: new PropRef([`label`]), - compareOptions: { - direction: `asc` as const, - nulls: `first` as const, - }, - }, - ] - const read = (limit?: number, optimizedOnly?: boolean) => - collection - .currentStateAsChanges({ orderBy, limit, optimizedOnly }) - ?.map(({ key }) => key) - - // The model reads the supplied string lengths, not an index key. The - // leading zero marks an ordinary user string; it has length 14, between - // c's 10 and a's 17. A finite prefix exposes the wrong encoded order. - expect(model).toEqual([`c`, `b`, `a`]) - expect(read()).toEqual(model) - expect(read(2)).toEqual(model.slice(0, 2)) - const index = collection.createIndex((row) => row.label, { - indexType: IndexType, - options: { - compareOptions: { - direction: `asc`, - nulls: `first`, - stringSort: `custom`, - compare, - }, - }, - }) - const take = vi.spyOn(index, `takeFromStart`) - expect(read(undefined, true)).toEqual(model) - expect(read(2, true)).toEqual(model.slice(0, 2)) - expect(take).toHaveBeenCalled() - - // Direct ordered traversal, both boundaries, and exact equality are - // separate observations of the same installed index. A cursor after a - // excludes every earlier callback position; equality still uses its - // normalized key, so another spelling cannot alias b. - expect(index.takeFromStart(3)).toEqual(model) - expect(index.take(3, source[2]!.label)).toEqual([`b`, `a`]) - expect(index.take(3, source[0]!.label)).toEqual([]) - expect( - index.rangeQuery({ from: source[2]!.label, to: source[0]!.label }), - ).toEqual(new Set(model)) - expect(index.equalityLookup(source[1]!.label)).toEqual(new Set([`b`])) - expect( - index.equalityLookup(`\u0000tanstack-db:string:\u0000tanstack-db:x`), - ).toEqual(new Set()) - }, [() => cleanup?.()]) - }) - - it(`continues a custom-collation subscription from the raw string position`, async () => { - type LabelRow = { id: string; label: string } - const source: ReadonlyArray = [ - { id: `a`, label: `abcdefghijklmnopq` }, - { id: `b`, label: `\u0000tanstack-db:x` }, - { id: `c`, label: `abcdefghij` }, - ] - const compare = (left: string, right: string) => left.length - right.length - let cleanup: (() => unknown) | undefined - await withOracleCleanup(async () => { - const collection = createCollection({ - id: `escaped-custom-cursor-${name}`, - getKey: ({ id }) => id, - autoIndex: `off`, - syncMode: `on-demand`, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of source) write({ type: `insert`, value: row }) - commit() - markReady() - return { loadSubset: () => true } - }, - }, - }) - cleanup = () => collection.cleanup() - await collection.stateWhenReady() - const index = collection.createIndex((row) => row.label, { - indexType: IndexType, - options: { - compareOptions: { - direction: `asc`, - nulls: `first`, - stringSort: `custom`, - compare, - }, - }, - }) - const batches: Array> = [] - const subscription = collection.subscribeChanges( - (changes) => { - if (changes.length) batches.push(changes.map(({ key }) => key)) - }, - { includeInitialState: false }, - ) - try { - subscription.setOrderByIndex(index) - const orderBy = [ - { - expression: new PropRef([`label`]), - compareOptions: { - direction: `asc` as const, - nulls: `first` as const, - }, - }, - ] - subscription.requestLimitedSnapshot({ orderBy, limit: 1 }) - subscription.requestLimitedSnapshot({ - orderBy, - limit: 1, - minValues: [source[2]!.label], - }) - expect(batches).toEqual([[`c`], [`b`]]) - } finally { - subscription.unsubscribe() - } - }, [() => cleanup?.()]) - }) - - it(`keeps an escaped user string distinct from the undefined order marker`, () => { - const literal = `\u0000tanstack-db:undefined:` - const index = new IndexType(1, new PropRef([`value`]), undefined, { - compareOptions: { - direction: `asc`, - nulls: `first`, - stringSort: `lexical`, - }, - }) - index.add(`z`, { value: `z` }) - index.add(`literal`, { value: literal }) - index.add(`missing`, { value: undefined }) - - // BTree uses a traversal marker for undefined, but this exact string is - // ordinary user data. The two have different order and equality classes. - expect(index.takeFromStart(3)).toEqual([`missing`, `literal`, `z`]) - expect(index.rangeQuery({ from: literal, to: literal })).toEqual( - new Set([`literal`]), - ) - expect(index.equalityLookup(undefined)).toEqual(new Set([`missing`])) - expect(index.equalityLookup(literal)).toEqual(new Set([`literal`])) - }) - - it(`conserves keys when a byte view adopts a peer value and is reinserted`, async () => { - type ByteRow = { id: string; value: Uint8Array } - type Sync = Parameters[`sync`]>[0] - const source = new Map([ - [`0`, new Uint8Array([1])], - [`1`, new Uint8Array([1])], - [`2`, new Uint8Array([2])], - [`3`, Buffer.from([1])], - ]) - let sync!: Sync - let cleanup: (() => unknown) | undefined - await withOracleCleanup(async () => { - const collection = createCollection({ - id: `byte-reinsertion-order-${name}`, - getKey: ({ id }) => id, - autoIndex: `off`, - startSync: true, - sync: { - sync: (actions) => { - sync = actions - actions.begin() - for (const [id, value] of source) - actions.write({ type: `insert`, value: { id, value } }) - actions.commit() - actions.markReady() - }, - }, - }) - cleanup = () => collection.cleanup() - await collection.stateWhenReady() - const orderBy = [ - { - expression: new PropRef([`value`]), - compareOptions: DEFAULT_COMPARE_OPTIONS, - }, - ] - const scanOrderBy = [ - { - expression: new Func(`coalesce`, [new PropRef([`value`])]), - compareOptions: DEFAULT_COMPARE_OPTIONS, - }, - ] - const readScan = () => - collection - .currentStateAsChanges({ orderBy: scanOrderBy }) - ?.map(({ key }) => key) - const readIndexed = (optimizedOnly?: boolean) => - collection - .currentStateAsChanges({ orderBy, optimizedOnly }) - ?.map(({ key }) => key) - - // The source Map independently determines membership. No law chooses - // the relative default order of two distinct byte views, so compare - // index output with the public scan at each observation checkpoint. - const expectedExact = (byte: number) => - new Set( - [...source] - .filter(([, value]) => value.length === 1 && value[0] === byte) - .map(([key]) => key), - ) - const index = collection.createIndex((row) => row.value, { - indexType: IndexType, - }) - const indexedTake = vi.spyOn(index, `takeFromStart`) - const check = (phase: string) => { - indexedTake.mockClear() - const expected = readScan()! - expect( - indexedTake, - `${phase} scan bypasses index`, - ).not.toHaveBeenCalled() - expect(new Set(expected), `${phase} scan keys`).toEqual( - new Set(source.keys()), - ) - expect(expected.length, `${phase} scan multiplicity`).toBe(source.size) - expect - .soft(readIndexed(true), `${phase} public indexed order`) - .toEqual(expected) - expect(indexedTake, `${phase} indexed traversal ran`).toHaveBeenCalled() - expect - .soft( - index.takeFromStart(source.size + 1), - `${phase} direct index order`, - ) - .toEqual(expected) - expect(index.keyCount, `${phase} key count`).toBe(source.size) - expect(index.equalityLookup(new Uint8Array([1]))).toEqual( - expectedExact(1), - ) - expect(index.equalityLookup(new Uint8Array([2]))).toEqual( - expectedExact(2), - ) - } - - check(`initial`) - const peerValue = source.get(`1`)! - sync.begin() - sync.write({ type: `update`, value: { id: `0`, value: peerValue } }) - source.set(`0`, peerValue) - await sync.commit() - expect(collection.state.get(`0`)?.value).toBe(peerValue) - check(`after update`) - - sync.begin() - sync.write({ type: `delete`, key: `0` }) - source.delete(`0`) - await sync.commit() - expect(collection.state.has(`0`)).toBe(false) - check(`after delete`) - - sync.begin() - sync.write({ type: `insert`, value: { id: `0`, value: peerValue } }) - source.set(`0`, peerValue) - await sync.commit() - expect(collection.state.get(`0`)?.value).toBe(peerValue) - check(`after reinsert`) - - // Removing the index makes the same public ordered read use its scan. - // Its exact final order remains an independent receiving control. - expect(collection.removeIndex(index)).toBe(true) - expect(readIndexed()).toEqual(readScan()) - }, [() => cleanup?.()]) - }) - - it(`orders current byte views by a custom callback after equal-content replacement`, async () => { - type ByteRow = { id: string; value: Uint8Array } - type Sync = Parameters[`sync`]>[0] - const first = new Uint8Array([1]) - const peer = new Uint8Array([1]) - const replacement = new Uint8Array([1]) - const callbackRanks = new WeakMap([ - [first, 1], - [peer, 2], - [replacement, 3], - ]) - const source = new Map([ - [`first`, { value: first, rank: 1 }], - [`peer`, { value: peer, rank: 2 }], - ]) - let sync!: Sync - let cleanup: (() => unknown) | undefined - await withOracleCleanup(async () => { - const collection = createCollection({ - id: `custom-byte-replacement-${name}`, - getKey: ({ id }) => id, - autoIndex: `off`, - startSync: true, - sync: { - sync: (actions) => { - sync = actions - actions.begin() - for (const [id, { value }] of source) - actions.write({ type: `insert`, value: { id, value } }) - actions.commit() - actions.markReady() - }, - }, - }) - cleanup = () => collection.cleanup() - await collection.stateWhenReady() - const index = collection.createIndex((row) => row.value, { - indexType: IndexType, - options: { - compareFn: (left: Uint8Array, right: Uint8Array) => - callbackRanks.get(left)! - callbackRanks.get(right)!, - }, - }) - - // Source facts carry numeric ranks independently of index storage. - // The callback receives the raw view for each current row; exact lookup - // instead groups these three one-byte views by content. A same-content - // update changes the first row's callback rank from 1 to 3. - const ordered = () => - [...source] - .sort(([, left], [, right]) => left.rank - right.rank) - .map(([id]) => id) - const atOrAbovePeer = () => - new Set( - [...source].filter(([, { rank }]) => rank >= 2).map(([id]) => id), - ) - const afterPeer = () => ordered().filter((id) => source.get(id)!.rank > 2) - const exact = () => - new Set( - [...source] - .filter(([, { value }]) => value.length === 1 && value[0] === 1) - .map(([id]) => id), - ) - - expect(index.takeFromStart(2)).toEqual(ordered()) - expect(index.rangeQuery({ from: peer })).toEqual(atOrAbovePeer()) - expect(index.take(2, peer)).toEqual(afterPeer()) - expect(index.equalityLookup(new Uint8Array([1]))).toEqual(exact()) - - sync.begin() - sync.write({ type: `update`, value: { id: `first`, value: replacement } }) - source.set(`first`, { value: replacement, rank: 3 }) - await sync.commit() - expect(collection.state.get(`first`)?.value).toBe(replacement) - expect(index.keyCount).toBe(source.size) - expect(index.equalityLookup(new Uint8Array([1]))).toEqual(exact()) - expect.soft(index.takeFromStart(2)).toEqual(ordered()) - expect.soft(index.rangeQuery({ from: peer })).toEqual(atOrAbovePeer()) - expect.soft(index.take(2, peer)).toEqual(afterPeer()) - }, [() => cleanup?.()]) - }) -}) diff --git a/packages/db/tests/fixtures/error-site-messages.json b/packages/db/tests/fixtures/error-site-messages.json index 375dc16bc6..38d4b408a1 100644 --- a/packages/db/tests/fixtures/error-site-messages.json +++ b/packages/db/tests/fixtures/error-site-messages.json @@ -923,10 +923,5 @@ "file": "local-storage.ts", "template": "LocalStorage sync belongs to its options Collection.", "literals": ["LocalStorage sync belongs to its options Collection."] - }, - "243": { - "file": "query/compiler/mutable-list-refresh.ts", - "template": "An inner mutable-list revision lost its candidate", - "literals": ["An inner mutable-list revision lost its candidate"] } } diff --git a/packages/db/tests/index-update-oracle.property.test.ts b/packages/db/tests/index-update-oracle.property.test.ts index 8d95709b2b..8270f6214d 100644 --- a/packages/db/tests/index-update-oracle.property.test.ts +++ b/packages/db/tests/index-update-oracle.property.test.ts @@ -1,6 +1,5 @@ import { describe, expect, expectTypeOf, test, vi } from 'vitest' import { fc } from '@fast-check/vitest' -import { Temporal } from 'temporal-polyfill' import { createCollection } from '../src/collection/index.js' import { CollectionInErrorStateError } from '../src/errors.js' import { BasicIndex } from '../src/indexes/basic-index.js' @@ -36,10 +35,8 @@ import type { CompareOptions } from '../src/query/builder/types.js' * the value at that bound is excluded. * Exact equality membership also stays distinct from comparator position: * replacing a value with one in a different exact-equality bucket at the - * same comparator position must move its exact lookup membership. These laws - * also allow a custom callback to put signed zero values in different order - * positions while Map equality keeps them in one exact bucket. These laws - * concern returned index results, not bucket identity or maintenance work. + * same comparator position must move its exact lookup membership. This law + * concerns returned index results, not bucket identity or maintenance work. * A build is one of those direct actions even when a source value is * `undefined`: default nulls-first order must hold after build just as it * does after incremental add. The nullish lane below exercises both paths. @@ -476,177 +473,6 @@ describe.each(indexTypes)(`%s update properties`, (_indexName, IndexType) => { }) }) -/** - * Equal native Dates identify one exact index value by timestamp, even when - * an add and its later remove receive different Date objects. Authority: the - * IndexInterface equality and ordered-read methods, together with the native - * Date timestamp comparison law in the comparison oracle. Both direct-index - * histories use valid Dates: one follows default timestamp order, while the - * other lets a custom callback separate rows with equal timestamps. Neither - * claims arbitrary object equivalence or Collection publication. - * - * The default lane's independent source Map stores supplied epoch numbers, - * not the objects or an index-normalized value. Its fresh sort predicts the - * public order; filtering its rows predicts exact and lower-bound membership. - * The callback lane's Map stores supplied order positions separately from - * exact timestamp membership. A same-object re-add after an equal-but-distinct - * removal distinguishes a live comparator position from an obsolete one. A - * second row and neighboring epochs exercise promotion and bounded traversal. - * Each add, remove, and update uses the direct index API and is compared after - * return. - */ -function expectDateIndexMatchesEpochModel( - index: BaseIndex, - rows: ReadonlyMap, -): void { - const ordered = [...rows].sort( - ([leftKey, leftEpoch], [rightKey, rightEpoch]) => - leftEpoch - rightEpoch || - (leftKey < rightKey ? -1 : leftKey > rightKey ? 1 : 0), - ) - expect(index.keyCount).toBe(rows.size) - expect(index.takeFromStart(rows.size + 1)).toEqual( - ordered.map(([key]) => key), - ) - expect(index.takeReversedFromEnd(rows.size + 1)).toEqual( - ordered.map(([key]) => key).reverse(), - ) - - for (const epoch of [0, 100, 200]) { - const exact = new Set( - [...rows].filter(([, value]) => value === epoch).map(([key]) => key), - ) - const atOrAbove = new Set( - [...rows].filter(([, value]) => value >= epoch).map(([key]) => key), - ) - expect(index.equalityLookup(new Date(epoch))).toEqual(exact) - expect(index.lookup(`eq`, new Date(epoch))).toEqual(exact) - expect(index.rangeQuery({ from: new Date(epoch) })).toEqual(atOrAbove) - expect(index.lookup(`gte`, new Date(epoch))).toEqual(atOrAbove) - } -} - -describe.each(indexTypes)(`%s equivalent Date removal`, (_name, IndexType) => { - test(`keeps re-added rows in ordered reads after removal by an equal Date`, () => { - const index = new IndexType(1, new PropRef([`value`])) - const rows = new Map() - const original = new Date(100) - const peer = new Date(100) - - const add = (key: string, value: Date, epoch: number) => { - index.add(key, { value }) - rows.set(key, epoch) - expectDateIndexMatchesEpochModel(index, rows) - } - const remove = (key: string, value: Date) => { - index.remove(key, { value }) - rows.delete(key) - expectDateIndexMatchesEpochModel(index, rows) - } - const update = ( - key: string, - oldValue: Date, - newValue: Date, - newEpoch: number, - ) => { - index.update(key, { value: oldValue }, { value: newValue }) - rows.set(key, newEpoch) - expectDateIndexMatchesEpochModel(index, rows) - } - - expectDateIndexMatchesEpochModel(index, rows) - add(`first`, original, 100) - remove(`first`, new Date(100)) - add(`second`, original, 100) - remove(`second`, new Date(100)) - add(`before`, new Date(0), 0) - add(`first`, original, 100) - add(`peer`, peer, 100) - remove(`first`, new Date(100)) - add(`after`, new Date(200), 200) - update(`peer`, new Date(100), new Date(200), 200) - add(`first`, original, 100) - }) - - test(`keeps callback-distinct Date positions through equivalent removal and update`, () => { - // The callback is a declared ordering fact. Two Dates with one exact - // timestamp can occupy different positions; the source model records each - // row's supplied position separately from its timestamp. A fresh old-item - // Date at position zero is legal during removal and update. - const first = new Date(100) - const second = new Date(100) - const replacement = new Date(100) - const positions = new WeakMap([ - [first, 0], - [second, 1], - [replacement, 1], - ]) - const index = new IndexType(1, new PropRef([`value`]), undefined, { - compareFn: (left: unknown, right: unknown) => - (left as Date).getTime() - (right as Date).getTime() || - (positions.get(left as Date) ?? 0) - - (positions.get(right as Date) ?? 0), - }) - const rows = new Map() - const check = () => { - const ordered = [...rows].sort( - ([leftKey, leftPosition], [rightKey, rightPosition]) => - leftPosition - rightPosition || - (leftKey < rightKey ? -1 : leftKey > rightKey ? 1 : 0), - ) - expect(index.keyCount).toBe(rows.size) - expect(index.equalityLookup(new Date(100))).toEqual(new Set(rows.keys())) - expect(index.takeFromStart(rows.size + 1)).toEqual( - ordered.map(([key]) => key), - ) - for (const [position, bound] of [ - [0, first], - [1, second], - ] as const) { - expect(index.rangeQuery({ from: bound, to: bound })).toEqual( - new Set( - [...rows] - .filter(([, rowPosition]) => rowPosition === position) - .map(([key]) => key), - ), - ) - } - } - - check() - index.add(`z`, { value: first }) - rows.set(`z`, 0) - check() - index.add(`a`, { value: second }) - rows.set(`a`, 1) - check() - index.remove(`z`, { value: new Date(100) }) - rows.delete(`z`) - check() - index.add(`z`, { value: first }) - rows.set(`z`, 0) - check() - index.update(`z`, { value: new Date(100) }, { value: replacement }) - rows.set(`z`, 1) - check() - index.add(`b`, { value: first }) - rows.set(`b`, 0) - check() - // The old item has the same exact timestamp as `a`, but its callback - // position is zero while `a` was stored at position one. `b` occupies - // position zero, so searching by the old item would find the wrong bucket. - index.remove(`a`, { value: new Date(100) }) - rows.delete(`a`) - check() - index.update(`z`, { value: new Date(100) }, { value: first }) - rows.set(`z`, 0) - check() - index.add(`a`, { value: second }) - rows.set(`a`, 1) - check() - }) -}) - /** * Direct index equality keeps `undefined` and `null` in distinct exact-value * buckets, even though the default ordering comparator places both at one @@ -1791,193 +1617,7 @@ describe.each([ }, ) -// The caller's callback gives each signed zero its own order position, while -// JavaScript Map gives them one exact membership class. This small source Map -// is the authority for ownership; a fresh sort by the caller's stated numeric -// rank predicts ordered reads. No index bucket or traversal helper is reused. -function signedZeroPosition(value: number): number { - return Object.is(value, -0) ? -0.5 : value -} - -const signedZeroCompare = (left: unknown, right: unknown): number => - signedZeroPosition(left as number) - signedZeroPosition(right as number) - -function expectSignedZeroRefinement( - index: BaseIndex, - rows: ReadonlyMap, -): void { - const ordered = [...rows].sort( - ([leftKey, leftValue], [rightKey, rightValue]) => - signedZeroPosition(leftValue) - signedZeroPosition(rightValue) || - (leftKey < rightKey ? -1 : leftKey > rightKey ? 1 : 0), - ) - const keys = ordered.map(([key]) => key) - const zeroOwners = new Set( - [...rows].filter(([, value]) => value === 0).map(([key]) => key), - ) - - // Observe the documented direct IndexInterface after every returned action. - // A set-only comparison would miss a skipped or duplicated ordered row. - expect(index.keyCount).toBe(rows.size) - expect(index.equalityLookup(-0)).toEqual(zeroOwners) - expect(index.equalityLookup(0)).toEqual(zeroOwners) - expect(index.takeFromStart(rows.size + 1)).toEqual(keys) - expect(index.takeReversedFromEnd(rows.size + 1)).toEqual([...keys].reverse()) - for (const cursor of [-0, 0]) { - const position = signedZeroPosition(cursor) - expect(index.take(rows.size + 1, cursor)).toEqual( - ordered - .filter(([, value]) => signedZeroPosition(value) > position) - .map(([key]) => key), - ) - expect(index.takeReversed(rows.size + 1, cursor)).toEqual( - ordered - .filter(([, value]) => signedZeroPosition(value) < position) - .reverse() - .map(([key]) => key), - ) - } - - // Distinct bounds with `===` equality are legal when a callback separates - // their positions. The reverse bound is empty; exclusive edges select one - // sign only. These probes reach both leaf and split BTree range traversal. - for (const options of [ - { from: -0, to: -0 }, - { from: 0, to: 0 }, - { from: -0, to: 0 }, - { from: -0, to: 0, fromInclusive: false }, - { from: -0, to: 0, toInclusive: false }, - { from: -0, to: 0, fromInclusive: false, toInclusive: false }, - { from: 0, to: -0 }, - ]) { - const lower = signedZeroPosition(options.from) - const upper = signedZeroPosition(options.to) - const expected = new Set( - [...rows] - .filter(([, value]) => { - const position = signedZeroPosition(value) - return ( - (options.fromInclusive === false - ? position > lower - : position >= lower) && - (options.toInclusive === false - ? position < upper - : position <= upper) - ) - }) - .map(([key]) => key), - ) - expect(index.rangeQuery(options)).toEqual(expected) - } -} - describe.each(indexTypes)(`%s comparator groups`, (_indexName, IndexType) => { - test.each([`add`, `build`] as const)( - `follows a custom callback that separates signed zero after %s`, - (construction) => { - const rows = [ - { key: `negative`, value: -0 }, - { key: `positive`, value: 0 }, - { key: `later`, value: 1 }, - ] - const rank = (value: unknown): number => - Object.is(value, -0) ? 0 : value === 0 ? 1 : 2 - const index = new IndexType(1, new PropRef([`value`]), undefined, { - compareFn: (left, right) => rank(left) - rank(right), - }) - if (construction === `add`) { - for (const row of rows) index.add(row.key, row) - } else { - index.build( - rows.map((row): [string, { value: number }] => [row.key, row]), - ) - } - - // Map's SameValueZero defines the expected exact membership; the - // independent rank labels define the callback's three order positions. - expect(index.equalityLookup(-0)).toEqual( - new Set([`negative`, `positive`]), - ) - expect(index.takeFromStart(3)).toEqual([`negative`, `positive`, `later`]) - expect(index.rangeQuery({ from: -0, to: -0 })).toEqual( - new Set([`negative`]), - ) - expect(index.take(2, -0)).toEqual([`positive`, `later`]) - - index.remove(`negative`, rows[0]) - expect(index.equalityLookup(0)).toEqual(new Set([`positive`])) - expect(index.takeFromStart(2)).toEqual([`positive`, `later`]) - }, - ) - - test(`tracks signed-zero order through updates, removal, and reuse`, () => { - const index = new IndexType(1, new PropRef([`value`]), undefined, { - compareFn: signedZeroCompare, - }) - const rows = new Map() - const put = (key: string, value: number): void => { - if (rows.has(key)) { - index.update(key, { value: rows.get(key) }, { value }) - } else { - index.add(key, { value }) - } - rows.set(key, value) - expectSignedZeroRefinement(index, rows) - } - const remove = (key: string): void => { - index.remove(key, { value: rows.get(key) }) - rows.delete(key) - expectSignedZeroRefinement(index, rows) - } - - // A sign-changing update keeps exact membership but must move the row's - // comparator position. Peers expose both stale and missing order buckets. - put(`negative-peer`, -0) - put(`moving`, -0) - put(`positive-peer`, 0) - put(`later`, 1) - put(`moving`, 0) - put(`moving`, -0) - remove(`moving`) - put(`moving`, 0) - remove(`negative-peer`) - put(`new-negative`, -0) - }) - - test(`preserves mixed-sign ranges across a BTree-sized split`, () => { - const index = new IndexType(1, new PropRef([`value`]), undefined, { - compareFn: signedZeroCompare, - }) - const rows = new Map() - // More than the BTree's default 32 keys places the two zero positions - // across an internal traversal. BasicIndex receives the same history. - const values = [ - ...Array.from({ length: 15 }, (_, offset) => offset - 15), - -0, - 0, - ...Array.from({ length: 16 }, (_, offset) => offset + 1), - ] - for (const [ordinal, value] of values.entries()) { - const key = `row-${String(ordinal).padStart(2, `0`)}` - index.add(key, { value }) - rows.set(key, value) - } - expectSignedZeroRefinement(index, rows) - - // Continue after the split so a stale old-position row cannot hide behind - // correct initial build order or leave an empty bucket after retirement. - const negative = `row-15` - index.update(negative, { value: -0 }, { value: 0 }) - rows.set(negative, 0) - expectSignedZeroRefinement(index, rows) - index.remove(negative, { value: 0 }) - rows.delete(negative) - expectSignedZeroRefinement(index, rows) - index.add(negative, { value: -0 }) - rows.set(negative, -0) - expectSignedZeroRefinement(index, rows) - }) - test(`rejects stale retired identity outputs without rejecting live reuse`, () => { const symbol = Symbol(`same group`) const retired = { key: `old`, groupId: 0, value: [symbol] } @@ -2641,234 +2281,3 @@ test(`ordered scan resolves inherited collation outside each comparison`, async expect(collection.indexes.size).toBe(0) }) }) - -/** - * Mixed string/number source values remain legal index inputs. A default - * comparator cannot put every string at the same position as a number while - * still ordering those strings: that would make comparator equality - * non-transitive and leave no stable ordered partition for a delete or update. - * A plain Map predicts exact membership and key conservation; the two strings - * still follow their configured collation. Their position relative to the - * number is left free. This lane covers two scalar types, three source keys, - * build and incremental admission, and returned direct actions. It does not - * claim a universal order over objects, Dates, or arbitrary custom callbacks. - */ -describe.each(indexTypes)( - `%s mixed scalar index refinement`, - (_name, IndexType) => { - test.each([ - [`build`, `forward`], - [`build`, `reverse`], - [`incremental`, `forward`], - [`incremental`, `reverse`], - ] as const)( - `%s %s keeps ordered positions and exact members through delete and update`, - (admission, sourceOrder) => { - const index = new IndexType(1, new PropRef([`value`])) - const rows = new Map([ - [`r0`, `b`], - [`r1`, 1], - [`r2`, `a`], - ]) - const sourceRows = - sourceOrder === `forward` ? [...rows] : [...rows].reverse() - const source: Array<[string, { value: string | number }]> = - sourceRows.map(([key, value]) => [key, { value }]) - if (admission === `build`) index.build(source) - else for (const [key, value] of source) index.add(key, value) - - const check = () => { - const ordered = index.takeFromStart(rows.size + 1) - expect(ordered).toHaveLength(rows.size) - expect(new Set(ordered)).toEqual(new Set(rows.keys())) - const stringKeys = [...rows] - .filter(([, value]) => typeof value === `string`) - .sort((left, right) => - (left[1] as string).localeCompare(right[1] as string), - ) - .map(([key]) => key) - expect(ordered.filter((key) => stringKeys.includes(key))).toEqual( - stringKeys, - ) - for (const value of [`a`, `b`, 1, 2]) { - expect(index.equalityLookup(value)).toEqual( - new Set( - [...rows] - .filter(([, candidate]) => candidate === value) - .map(([key]) => key), - ), - ) - } - } - - check() // source admission - index.remove(`r0`, { value: `b` }) - rows.delete(`r0`) - check() - index.update(`r1`, { value: 1 }, { value: `b` }) - rows.set(`r1`, `b`) - check() - index.update(`r1`, { value: `b` }, { value: 2 }) - rows.set(`r1`, 2) - check() - }, - ) - - test(`keeps a numeric string and its number as distinct live keys`, () => { - const index = new IndexType(1, new PropRef([`value`])) - index.add(`number`, { value: 1 }) - index.add(`text`, { value: `1` }) - expect(index.equalityLookup(1)).toEqual(new Set([`number`])) - expect(index.equalityLookup(`1`)).toEqual(new Set([`text`])) - expect(new Set(index.takeFromStart(2))).toEqual( - new Set([`text`, `number`]), - ) - }) - }, -) - -/** - * A default index must retain every key through legal mutations even when its - * field holds an array, Date, or Temporal value beside an ordinary object. - * Element, timestamp, and calendar order compare values within their own kind; - * this model makes no claim about their position relative to the ordinary - * object. Its source Map and supplied labels predict exact membership and keys in an unfiltered - * ordered take. This fixed history reaches representative replacement and a - * later update that exposed a stale ordered position in BasicIndex. - */ -describe.each(indexTypes)( - `%s mixed object index refinement`, - (_name, IndexType) => { - test.each([`Array`, `Date`, `PlainDate`] as const)( - `retains every key through %s and ordinary-object updates`, - (valueKind) => { - const values = - valueKind === `Array` - ? { high: [2], ordinary: {}, low: [1] } - : valueKind === `Date` - ? { high: new Date(2), ordinary: {}, low: new Date(1) } - : { - high: Temporal.PlainDate.from(`2024-01-02`), - ordinary: {}, - low: Temporal.PlainDate.from(`2024-01-01`), - } - type Label = keyof typeof values - const index = new IndexType(1, new PropRef([`value`])) - const rows = new Map([ - [`high`, `high`], - [`ordinary`, `ordinary`], - [`low`, `low`], - ]) - for (const [key, label] of rows) { - index.add(key, { value: values[label] }) - } - - const check = () => { - const orderedKeys = index.takeFromStart(rows.size + 1) - expect(orderedKeys).toHaveLength(rows.size) - expect(new Set(orderedKeys)).toEqual(new Set(rows.keys())) - for (const label of [`high`, `ordinary`, `low`] as const) { - expect(index.equalityLookup(values[label])).toEqual( - new Set( - [...rows] - .filter(([, candidate]) => candidate === label) - .map(([key]) => key), - ), - ) - } - const lowKeys = [...rows] - .filter(([, label]) => label === `low`) - .map(([key]) => key) - const highKeys = [...rows] - .filter(([, label]) => label === `high`) - .map(([key]) => key) - if (lowKeys.length > 0 && highKeys.length > 0) { - expect( - Math.max(...lowKeys.map((key) => orderedKeys.indexOf(key))), - ).toBeLessThan( - Math.min(...highKeys.map((key) => orderedKeys.indexOf(key))), - ) - } - } - - check() - index.update( - `ordinary`, - { value: values.ordinary }, - { value: values.high }, - ) - rows.set(`ordinary`, `high`) - check() - index.update(`ordinary`, { value: values.high }, { value: values.low }) - rows.set(`ordinary`, `low`) - check() - index.update(`low`, { value: values.low }, { value: values.high }) - rows.set(`low`, `high`) - check() - }, - ) - }, -) - -/** - * An index expression may return a function reference. The source Map predicts - * exact reference membership and one ordered occurrence per live key; it does - * not prescribe an order between functions and numbers. Two distinct function - * values beside a number exercise build, incremental admission, deletion, and - * replacement. Native relational comparison can otherwise tie each function - * to the number while ordering the two functions apart. - */ -describe.each(indexTypes)( - `%s callable index refinement`, - (_name, IndexType) => { - test.each([`build`, `incremental`] as const)( - `%s conserves live keys through function and number changes`, - (admission) => { - const values = { - a: function a() {}, - b: function b() {}, - one: 1, - two: 2, - } - type Label = keyof typeof values - const rows = new Map([ - [`b`, `b`], - [`number`, `one`], - [`a`, `a`], - ]) - const index = new IndexType(1, new PropRef([`value`])) - const source: Array<[string, { value: (typeof values)[Label] }]> = [ - ...rows, - ].map(([key, label]) => [key, { value: values[label] }]) - if (admission === `build`) index.build(source) - else for (const [key, value] of source) index.add(key, value) - - const check = () => { - const orderedKeys = index.takeFromStart(rows.size + 1) - expect(orderedKeys).toHaveLength(rows.size) - expect(new Set(orderedKeys)).toEqual(new Set(rows.keys())) - for (const label of [`a`, `b`, `one`, `two`] as const) { - expect(index.equalityLookup(values[label])).toEqual( - new Set( - [...rows] - .filter(([, candidate]) => candidate === label) - .map(([key]) => key), - ), - ) - } - } - - check() - index.remove(`b`, { value: values.b }) - rows.delete(`b`) - check() - index.update(`number`, { value: values.one }, { value: values.b }) - rows.set(`number`, `b`) - check() - index.update(`number`, { value: values.b }, { value: values.two }) - rows.set(`number`, `two`) - check() - }, - ) - }, -) diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index 31bac27019..bea5072439 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -34,11 +34,6 @@ const ownerGroups: ReadonlyArray = [ `index-update`, `reference-model nullish-membership exact-identity custom-comparator`, ], - [ - `db/tests/query/compiler/in-evaluator-oracle.property.test.ts`, - `evaluators`, - `in`, - ], [ `db/tests/cursor-oracle.property.test.ts`, `cursor`, @@ -286,8 +281,6 @@ const indexReplayTestNames = new Map([ ]) function directReplayTestName(property: string): string | undefined { - if (property === `evaluators.in`) - return `agrees with the reference over a random or replayed campaign` if (property === `cold-join.compound`) return `recomputes generated compound histories, seed=undefined` if (property === `oracle-replay.calibration`) diff --git a/packages/db/tests/oracle-replay.test.ts b/packages/db/tests/oracle-replay.test.ts index 144dacacd1..539419facd 100644 --- a/packages/db/tests/oracle-replay.test.ts +++ b/packages/db/tests/oracle-replay.test.ts @@ -96,12 +96,12 @@ describe(`guarded oracle replay`, () => { it.each([ [ - `evaluators.in`, - `tests/query/compiler/in-evaluator-oracle.property.test.ts`, - 1, + `index-update.reference-model`, + `tests/index-update-oracle.property.test.ts`, + 2, ], [ - `index-update.reference-model`, + `index-update.nullish-membership`, `tests/index-update-oracle.property.test.ts`, 2, ], diff --git a/packages/db/tests/query/captured-inarray-stages-oracle.test.ts b/packages/db/tests/query/captured-inarray-stages-oracle.test.ts deleted file mode 100644 index db6e1f2c26..0000000000 --- a/packages/db/tests/query/captured-inarray-stages-oracle.test.ts +++ /dev/null @@ -1,510 +0,0 @@ -/** - * # Where must captured membership be refreshed? - * - * Law: a caller-owned array passed to `inArray` remains query input after - * compilation. Editing it alone need not publish. After the next delivered - * source transaction, every existing public result uses its current members, - * including rows that the transaction did not touch. The fixed histories here - * exercise numeric membership in projection, ordering, grouping, plain and - * wrapped aggregate arguments, HAVING, and either operand of a join equality. - * They establish these placements for one unrelated source update, not all - * legal histories. - * - * Model: the small plain arrays below calculate membership with JavaScript - * `includes` at each checkpoint. Their expected values do not use compiled - * expressions or D2 output. The source grammar has three rows: values 1 and 2 - * are untouched, while an update to value 3 clocks the change. The joined - * grammar adds one TRUE flag. Public live-query rows and one subscriber batch - * are the observations, immediately after preload and the source commit. - * A compiler that only reevaluates the changed row fails at the second cut. - */ -import { describe, expect, it } from 'vitest' -import { createCollection } from '../../src/collection/index.js' -import { BasicIndex } from '../../src/indexes/basic-index.js' -import { - add, - caseWhen, - count, - eq, - inArray, - max, - min, - sum, -} from '../../src/query/builder/functions.js' -import { createLiveQueryCollection } from '../../src/query/index.js' -import { mockSyncCollectionOptions, withOracleCleanup } from '../utils.js' - -type Row = { id: string; v: number; revision: number } -const originalRows: Array = [ - { id: `one`, v: 1, revision: 0 }, - { id: `two`, v: 2, revision: 0 }, - { id: `trigger`, v: 3, revision: 0 }, -] -let collectionSerial = 0 - -function makeRows() { - return createCollection( - mockSyncCollectionOptions({ - id: `captured-inarray-stages-${collectionSerial++}`, - getKey: (row) => row.id, - initialData: originalRows, - }), - ) -} - -function clockRows(source: ReturnType): void { - source.utils.begin() - source.utils.write({ - type: `update`, - value: { id: `trigger`, v: 3, revision: 1 }, - }) - source.utils.commit() -} - -describe(`captured inArray query-stage oracle`, () => { - it(`refreshes every projected member in one public transaction`, async () => { - const source = makeRows() - const members = [1] - const live = createLiveQueryCollection((q) => - q.from({ row: source }).select(({ row }) => ({ - id: row.id, - member: inArray(row.v, members), - })), - ) - const batches: Array> = - [] - let unsubscribe: (() => void) | undefined - await withOracleCleanup(async () => { - await live.preload() - const subscription = live.subscribeChanges( - (changes) => { - // Publication is one coherent cut: the callback can read the whole - // new result while it receives the changed projected rows. - expect( - live.toArray - .map((row) => ({ id: row.id, member: row.member })) - .sort((a, b) => a.id.localeCompare(b.id)), - ).toEqual( - originalRows - .map((row) => ({ id: row.id, member: members.includes(row.v) })) - .sort((a, b) => a.id.localeCompare(b.id)), - ) - batches.push( - changes.map((change) => ({ - type: change.type, - id: change.value.id, - member: change.value.member, - })), - ) - }, - { includeInitialState: false }, - ) - unsubscribe = () => subscription.unsubscribe() - const expected = () => - originalRows.map((row) => ({ - id: row.id, - member: members.includes(row.v), - })) - const actual = () => - live.toArray - .map((row) => ({ id: row.id, member: row.member })) - .sort((a, b) => a.id.localeCompare(b.id)) - expect(actual()).toEqual( - expected().sort((a, b) => a.id.localeCompare(b.id)), - ) - - members[0] = 2 - clockRows(source) - expect(actual()).toEqual( - expected().sort((a, b) => a.id.localeCompare(b.id)), - ) - expect(batches).toHaveLength(1) - expect( - batches[0] - ?.filter((change) => change.id !== `trigger`) - .sort((a, b) => a.id.localeCompare(b.id)), - ).toEqual([ - { type: `update`, id: `one`, member: false }, - { type: `update`, id: `two`, member: true }, - ]) - }, [() => unsubscribe?.(), () => live.cleanup(), () => source.cleanup()]) - }) - - it(`reorders a retained limited result`, async () => { - const source = makeRows() - const members = [1] - const live = createLiveQueryCollection((q) => - q - .from({ row: source }) - .orderBy(({ row }) => inArray(row.v, members), `desc`) - .limit(1), - ) - await withOracleCleanup(async () => { - await live.preload() - const expected = () => - originalRows.find((row) => members.includes(row.v))?.id - expect(live.toArray.map((row) => row.id)).toEqual([expected()]) - members[0] = 2 - clockRows(source) - expect(live.toArray.map((row) => row.id)).toEqual([expected()]) - }, [() => live.cleanup(), () => source.cleanup()]) - }) - - it(`keeps a limited order unique when captured membership changes`, async () => { - const rows: Array = [ - { id: `a`, v: 1, revision: 0 }, - { id: `b`, v: 2, revision: 0 }, - { id: `c`, v: 3, revision: 0 }, - { id: `trigger`, v: 4, revision: 0 }, - ] - const source = createCollection( - mockSyncCollectionOptions({ - id: `captured-inarray-order-unique-${collectionSerial++}`, - getKey: (row) => row.id, - initialData: rows, - }), - ) - const members = [1] - const live = createLiveQueryCollection((q) => - q - .from({ row: source }) - .select(({ row }) => ({ id: row.id })) - .orderBy(({ row }) => inArray(row.v, members), `desc`) - .limit(3), - ) - await withOracleCleanup(async () => { - await live.preload() - // The plain-array model sorts each row once by current membership, - // then key. The public limit must contain distinct source identities. - const expected = () => - [...rows] - .sort( - (a, b) => - Number(members.includes(b.v)) - Number(members.includes(a.v)) || - a.id.localeCompare(b.id), - ) - .slice(0, 3) - .map((row) => row.id) - expect(live.toArray.map((row) => row.id)).toEqual(expected()) - members[0] = 2 - source.utils.begin() - source.utils.write({ - type: `update`, - value: { id: `trigger`, v: 4, revision: 1 }, - }) - source.utils.commit() - expect(live.toArray.map((row) => row.id)).toEqual(expected()) - }, [() => live.cleanup(), () => source.cleanup()]) - }) - - it(`requests an unbounded source for projected membership under an ordered limit`, async () => { - type RankedRow = { id: string; v: number; rank: number } - const rows: Array = [0, 1, 2, 3, 4].map((v) => ({ - id: String(v), - v, - rank: v, - })) - const requests: Array<{ - hasWhere: boolean - hasOrder: boolean - hasLimit: boolean - }> = [] - let begin!: () => void - let write!: (message: { type: `delete`; value: RankedRow }) => void - let commit!: () => unknown - const source = createCollection({ - id: `captured-inarray-projected-demand-${collectionSerial++}`, - getKey: (row) => row.id, - syncMode: `on-demand`, - startSync: true, - autoIndex: `eager`, - defaultIndexType: BasicIndex, - sync: { - sync: (params) => { - begin = params.begin - write = params.write - commit = params.commit - params.markReady() - return { - loadSubset: (options) => { - requests.push({ - hasWhere: options.where !== undefined, - hasOrder: options.orderBy !== undefined, - hasLimit: options.limit !== undefined, - }) - params.begin() - for (const row of rows) { - params.write({ type: `insert`, value: row }) - } - params.commit() - return true - }, - } - }, - }, - }) - const members = [1] - const live = createLiveQueryCollection((q) => - q - .from({ row: source }) - .select(({ row }) => ({ - id: row.id, - member: inArray(row.v, members), - })) - .orderBy(({ row }) => row.rank) - .limit(1), - ) - await withOracleCleanup(async () => { - await live.preload() - expect(requests[0]).toEqual({ - hasWhere: false, - hasOrder: false, - hasLimit: false, - }) - const expected = () => ({ id: `0`, member: members.includes(0) }) - expect( - live.toArray.map((row) => ({ id: row.id, member: row.member })), - ).toEqual([expected()]) - - members[0] = 0 - begin() - write({ type: `delete`, value: rows[4]! }) - commit() - expect( - live.toArray.map((row) => ({ id: row.id, member: row.member })), - ).toEqual([expected()]) - }, [() => live.cleanup(), () => source.cleanup()]) - }) - - it(`moves untouched rows between group keys`, async () => { - const source = makeRows() - const members = [1] - const live = createLiveQueryCollection((q) => - q - .from({ row: source }) - .groupBy(({ row }) => inArray(row.v, members)) - .select(({ row }) => ({ total: sum(row.v) })), - ) - await withOracleCleanup(async () => { - await live.preload() - const expected = () => { - const totals = new Map() - for (const row of originalRows) { - const key = members.includes(row.v) - totals.set(key, (totals.get(key) ?? 0) + row.v) - } - return [...totals.values()].sort((a, b) => a - b) - } - const actual = () => - live.toArray.map((row) => row.total).sort((a, b) => a - b) - expect(actual()).toEqual(expected()) - members[0] = 2 - clockRows(source) - expect(actual()).toEqual(expected()) - }, [() => live.cleanup(), () => source.cleanup()]) - }) - - it(`recomputes an aggregate argument for untouched rows`, async () => { - const source = makeRows() - const members = [1] - const live = createLiveQueryCollection((q) => - q.from({ row: source }).select(({ row }) => ({ - total: sum(caseWhen(inArray(row.v, members), row.v, 0)), - })), - ) - await withOracleCleanup(async () => { - await live.preload() - const expected = () => - originalRows.reduce( - (total, row) => total + (members.includes(row.v) ? row.v : 0), - 0, - ) - expect(live.toArray.map((row) => row.total)).toEqual([expected()]) - members[0] = 2 - clockRows(source) - expect(live.toArray.map((row) => row.total)).toEqual([expected()]) - }, [() => live.cleanup(), () => source.cleanup()]) - }) - - it(`recomputes a mutable aggregate argument inside a selected expression`, async () => { - const source = makeRows() - const members = [1] - const live = createLiveQueryCollection((q) => - q.from({ row: source }).select(({ row }) => ({ - total: add(sum(caseWhen(inArray(row.v, members), row.v, 0)), 10), - })), - ) - await withOracleCleanup(async () => { - await live.preload() - // The outer addition changes no grouping rule: its aggregate argument - // still sums every current member of the retained source relation. - const expected = () => - 10 + - originalRows.reduce( - (total, row) => total + (members.includes(row.v) ? row.v : 0), - 0, - ) - expect(live.toArray.map((row) => row.total)).toEqual([expected()]) - members[0] = 2 - clockRows(source) - expect(live.toArray.map((row) => row.total)).toEqual([expected()]) - }, [() => live.cleanup(), () => source.cleanup()]) - }) - - it(`recomputes grouped SUM and min/max arguments in an untouched group`, async () => { - type GroupRow = Row & { group: string } - const rows: Array = [ - { id: `a1`, group: `a`, v: 1, revision: 0 }, - { id: `a2`, group: `a`, v: 2, revision: 0 }, - { id: `b1`, group: `b`, v: 3, revision: 0 }, - ] - const source = createCollection( - mockSyncCollectionOptions({ - id: `captured-inarray-group-aggregate-${collectionSerial++}`, - getKey: (row) => row.id, - initialData: rows, - }), - ) - const members = [1] - const live = createLiveQueryCollection((q) => - q - .from({ row: source }) - .groupBy(({ row }) => row.group) - .select(({ row }) => ({ - group: row.group, - total: sum(caseWhen(inArray(row.v, members), row.v, 0)), - lowest: min(caseWhen(inArray(row.v, members), row.v, 100)), - highest: max(caseWhen(inArray(row.v, members), row.v, 0)), - })), - ) - await withOracleCleanup(async () => { - await live.preload() - // Each aggregate reads current membership over all source rows in its - // group, including group a when the source transaction touches group b. - const expected = () => - [`a`, `b`].map((group) => { - const groupRows = rows.filter((row) => row.group === group) - return { - group, - total: groupRows.reduce( - (total, row) => total + (members.includes(row.v) ? row.v : 0), - 0, - ), - lowest: Math.min( - ...groupRows.map((row) => - members.includes(row.v) ? row.v : 100, - ), - ), - highest: Math.max( - ...groupRows.map((row) => (members.includes(row.v) ? row.v : 0)), - ), - } - }) - const actual = () => - live.toArray - .map((row) => ({ - group: row.group, - total: row.total, - lowest: row.lowest, - highest: row.highest, - })) - .sort((a, b) => a.group.localeCompare(b.group)) - expect(actual()).toEqual(expected()) - members[0] = 2 - source.utils.begin() - source.utils.write({ - type: `update`, - value: { id: `b1`, group: `b`, v: 3, revision: 1 }, - }) - source.utils.commit() - expect(actual()).toEqual(expected()) - }, [() => live.cleanup(), () => source.cleanup()]) - }) - - it(`refreshes HAVING after an unrelated group receives an update`, async () => { - const source = makeRows() - const members = [1] - const live = createLiveQueryCollection((q) => - q - .from({ row: source }) - .groupBy(({ row }) => row.v) - .select(({ row }) => ({ v: row.v, n: count(row.id) })) - .having(({ $selected }) => inArray($selected.v, members)), - ) - await withOracleCleanup(async () => { - await live.preload() - const expected = () => - originalRows - .filter((row) => members.includes(row.v)) - .map((row) => row.v) - expect(live.toArray.map((row) => row.v)).toEqual(expected()) - members[0] = 2 - clockRows(source) - expect(live.toArray.map((row) => row.v)).toEqual(expected()) - }, [() => live.cleanup(), () => source.cleanup()]) - }) - - for (const sourceSide of [`main`, `joined`] as const) { - it(`refreshes membership in the ${sourceSide} JOIN operand`, async () => { - const source = makeRows() - const flags = createCollection( - mockSyncCollectionOptions<{ id: string; flag: boolean }>({ - id: `captured-inarray-join-flags-${collectionSerial++}`, - getKey: (row) => row.id, - initialData: [{ id: `true`, flag: true }], - }), - ) - const members = [1] - const live = createLiveQueryCollection((q) => - sourceSide === `main` - ? q - .from({ row: source }) - .innerJoin({ flag: flags }, ({ row, flag }) => - eq(inArray(row.v, members), flag.flag), - ) - : q - .from({ flag: flags }) - .innerJoin({ row: source }, ({ flag, row }) => - eq(flag.flag, inArray(row.v, members)), - ), - ) - await withOracleCleanup(async () => { - await live.preload() - const expected = () => - originalRows - .filter((row) => members.includes(row.v)) - .map((row) => row.id) - expect(live.toArray.map((row) => row.row.id)).toEqual(expected()) - members[0] = 2 - clockRows(source) - expect(live.toArray.map((row) => row.row.id)).toEqual(expected()) - }, [() => live.cleanup(), () => flags.cleanup(), () => source.cleanup()]) - }) - } - - it(`refreshes projected membership through a nested QueryRef`, async () => { - const source = makeRows() - const members = [1] - const live = createLiveQueryCollection((q) => { - const inner = q.from({ item: source }).select(({ item }) => ({ - id: item.id, - member: inArray(item.v, members), - })) - return q - .from({ row: inner }) - .where(({ row }) => eq(row.member, true)) - .select(({ row }) => ({ id: row.id })) - }) - await withOracleCleanup(async () => { - await live.preload() - const expected = () => - originalRows - .filter((row) => members.includes(row.v)) - .map((row) => row.id) - expect(live.toArray.map((row) => row.id)).toEqual(expected()) - members[0] = 2 - clockRows(source) - expect(live.toArray.map((row) => row.id)).toEqual(expected()) - }, [() => live.cleanup(), () => source.cleanup()]) - }) -}) diff --git a/packages/db/tests/query/compiler/binary-equality-work.test.ts b/packages/db/tests/query/compiler/binary-equality-work.test.ts index 10e9d62a95..ce17fdeafb 100644 --- a/packages/db/tests/query/compiler/binary-equality-work.test.ts +++ b/packages/db/tests/query/compiler/binary-equality-work.test.ts @@ -66,25 +66,21 @@ describe(`binary equality work`, () => { ) it.each([`eq`, `in`])( - `compares distinct MiB values using %s without string encoding`, + `compares a MiB using %s without encoding strings`, (operator) => { const left = new Uint8Array(1024 * 1024).fill(65) const right = left.slice() - const list = [right] const evaluate = compileSingleRowExpression( new Func(operator, [ new PropRef([`blob`]), - new Value(operator === `in` ? list : right), + new Value(operator === `in` ? [right] : right), ]), ) const equal = measureEncoding(() => evaluate({ blob: left })) expect(equal).toEqual({ result: true, bytes: 0 }) - const other = left.slice() - other[other.length - 1] = 66 - if (operator === `in`) list[0] = other - const different = measureEncoding(() => - evaluate({ blob: operator === `in` ? left : other }), - ) + const changedRow = left.slice() + changedRow[changedRow.length - 1] = 66 + const different = measureEncoding(() => evaluate({ blob: changedRow })) expect(different).toEqual({ result: false, bytes: 0 }) }, ) diff --git a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts index b3b6310281..f5ea6b05cb 100644 --- a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts +++ b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts @@ -1,4 +1,3 @@ -import { runInNewContext } from 'node:vm' import { describe, expect, it } from 'vitest' import fc from 'fast-check' import { Temporal } from 'temporal-polyfill' @@ -9,132 +8,49 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' /** * # Which values does `in` find in a list? * - * Transfer law: The public `inArray` API describes array membership. A - * compiled expression holding a caller-owned array reads that array's current - * members whenever it evaluates, including entries replaced since compilation. - * Mutating a Date or byte-array item in place is outside this law. The current - * evaluator returns UNKNOWN (`null`) - * for a null or undefined value and FALSE for a list that is not an array. + * Contract: `in(value, list)` is SQL `IN` under three-valued logic. A null or + * undefined value is UNKNOWN (`null`). A list that is not an array is FALSE. * Otherwise the result is TRUE when some list item equals the value under the * evaluator's equality, and FALSE when none does. A null or undefined item - * never equals a value under that membership interpretation. The public - * guide's broad SQL null statement leaves this last rule open for a separate - * product decision. The equality is the one `eq` documents in + * never equals a value. The equality is the one `eq` documents in * `src/query/compiler/evaluators.ts`: * * - `NaN` and an invalid Date equal each other and nothing else. - * - A native Date from either realm equals its millisecond timestamp, and - * another Date with the same timestamp. - * - Two ordinary Uint8Arrays (Buffers included) are equal when their bytes are - * equal. - * - Temporal kinds with an intrinsic `.equals()` use that relation. This - * includes equivalent time-zone aliases for ZonedDateTime, but excludes a - * distinct fixed-offset zone. PlainMonthDay and PlainYearMonth also retain - * their reference ISO year and day, even when default display omits them. - * Duration has no `.equals()`; its ten fields must match, so PT1H and PT60M - * are distinct. Changelog #1519 explicitly preserves that distinction for - * equivalent Duration forms under `eq`. + * - A valid Date equals its millisecond timestamp, and another Date with the + * same timestamp. + * - Two Uint8Arrays (Buffers included) are equal when their bytes are equal. + * - A Temporal value equals a Temporal value of the same kind and string. * - Every other pair compares with `===`, so `-0` equals `0`, `1n` does not * equal `1`, and objects compare by reference. * - * Model: `referenceIn` walks present indices through the length captured at - * the start of evaluation and applies `referenceEqual`, which restates the - * rules above. It imports no production comparison or normalization helper. + * Model: `referenceIn` walks the list and applies `referenceEqual`, which + * restates the rules above. It imports no production comparison or + * normalization helper. * * Domain: values and items come from a pool that contains the boundaries * of each rule: equal and unequal strings, a string that begins with the * internal normalization prefix, booleans, `0`, `-0`, `1`, `NaN`, `1n`, a - * local and foreign valid Dates and their timestamps, local and foreign - * invalid Dates, equal and unequal byte arrays, a Buffer, and native bytes - * from another realm with equal and unequal content. Temporal - * values include ZonedDateTime zone aliases, unequal values within each - * modeled Temporal kind, reference ISO year/day - * differences hidden by default display, and Duration carry forms with equal - * display but distinct fields, one shared object and a structurally equal - * copy, `null` and `undefined`. Lists have zero to six - * items and may repeat. The list is a constant `Value`, the shape that join - * demand builds, or a row field. A fixed immutable primitive list checks the - * compiled constant-list path against the same independent membership model. - * Fixed histories also compile a caller-owned list, then replace a scalar, - * Date, or byte-array slot before reevaluating. + * valid Date and its timestamp, an invalid Date, equal and unequal byte + * arrays, a Buffer, Temporal values, one shared object and a structurally + * equal copy, `null` and `undefined`. Lists have zero to six items and may + * repeat items. The list is a constant `Value`, the shape that join demand + * builds, or a row field, so a list that changes between rows is also checked. * * Production driver: compile `in` with the public IR and evaluate it for * each row, as a WHERE clause does. * * Refinement check: production and the model return the same value for every - * case at each evaluation checkpoint, including after a list edit. Each - * property runs a fixed campaign and a random or replayed campaign. + * case. Each property runs a fixed campaign and a random or replayed campaign. * A calibration test feeds the checker an evaluator that compares raw values * without normalization and requires the checker to reject it. * - * Limits: this owner checks the boolean result of `in` for ordinary values - * from one Temporal polyfill copy. It does not claim mixed Temporal - * implementations, caller-modified built-ins, malformed Temporal lookalikes, - * deliberately shadowed Temporal members, or replaced genuine prototypes. WHERE + * Limits: this owner checks the boolean result of `in` only. WHERE * publication, indexes, and subset loading have other owners. */ const PROPERTY = `evaluators.in` const PREFIX = `\u0000tanstack-db:binary:` const shared = { id: 1 } -const plainDate = Temporal.PlainDate.from(`2024-01-01`) -const otherPlainDate = Temporal.PlainDate.from(`2024-01-02`) -const instant = Temporal.Instant.from(`2024-01-01T00:00:00Z`) -const sameInstant = Temporal.Instant.from(`2024-01-01T00:00:00Z`) -const otherInstant = Temporal.Instant.from(`2024-01-02T00:00:00Z`) -const plainDateTime = Temporal.PlainDateTime.from(`2024-01-01T10:00`) -const samePlainDateTime = Temporal.PlainDateTime.from(`2024-01-01T10:00`) -const otherPlainDateTime = Temporal.PlainDateTime.from(`2024-01-01T11:00`) -const plainTime = Temporal.PlainTime.from(`10:00`) -const samePlainTime = Temporal.PlainTime.from(`10:00`) -const otherPlainTime = Temporal.PlainTime.from(`11:00`) -const localDate = new Date(1000) -const foreignDate = runInNewContext(`new Date(1000)`) as Date -const laterForeignDate = runInNewContext(`new Date(1001)`) as Date -const localInvalidDate = new Date(Number.NaN) -const foreignInvalidDate = runInNewContext(`new Date(NaN)`) as Date -const foreignBytes = runInNewContext(`new Uint8Array([1, 2])`) as Uint8Array -const unequalForeignBytes = runInNewContext( - `new Uint8Array([1, 3])`, -) as Uint8Array -const samePlainDate = Temporal.PlainDate.from(`2024-01-01`) -const leapMonthDay = new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972) -const sameLeapMonthDay = new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972) -const otherReferenceMonthDay = new Temporal.PlainMonthDay( - 2, - 29, - `iso8601`, - 2004, -) -const firstYearMonth = new Temporal.PlainYearMonth(2024, 2, `iso8601`, 1) -const sameFirstYearMonth = new Temporal.PlainYearMonth(2024, 2, `iso8601`, 1) -const otherReferenceYearMonth = new Temporal.PlainYearMonth( - 2024, - 2, - `iso8601`, - 20, -) -const hourDuration = Temporal.Duration.from(`PT1H`) -const sameHourDuration = Temporal.Duration.from(`PT1H`) -const sixtyMinuteDuration = Temporal.Duration.from(`PT60M`) -const secondDuration = Temporal.Duration.from({ seconds: 1 }) -const sameSecondDuration = Temporal.Duration.from({ seconds: 1 }) -const thousandMillisecondDuration = Temporal.Duration.from({ - milliseconds: 1000, -}) -const sameThousandMillisecondDuration = Temporal.Duration.from({ - milliseconds: 1000, -}) -const millionMicrosecondDuration = Temporal.Duration.from({ - microseconds: 1_000_000, -}) -const utcZoned = Temporal.ZonedDateTime.from(`2024-01-15T09:00:00+00:00[UTC]`) -const aliasZoned = Temporal.ZonedDateTime.from( - `2024-01-15T09:00:00+00:00[Etc/UTC]`, -) -const offsetZoned = Temporal.ZonedDateTime.from( - `2024-01-15T09:00:00+00:00[+00:00]`, -) const pool: ReadonlyArray = [ `a`, `b`, @@ -146,148 +62,53 @@ const pool: ReadonlyArray = [ 1, Number.NaN, 1n, - localDate, new Date(1000), - foreignDate, - laterForeignDate, + new Date(1000), 1000, - localInvalidDate, - foreignInvalidDate, + new Date(Number.NaN), new Uint8Array([1, 2]), new Uint8Array([1, 2]), new Uint8Array([1, 3]), Buffer.from([1, 2]), - foreignBytes, - unequalForeignBytes, - plainDate, Temporal.PlainDate.from(`2024-01-01`), - otherPlainDate, - instant, - sameInstant, - otherInstant, - plainDateTime, - samePlainDateTime, - otherPlainDateTime, - plainTime, - samePlainTime, - otherPlainTime, - samePlainDate, - leapMonthDay, - sameLeapMonthDay, - otherReferenceMonthDay, - firstYearMonth, - sameFirstYearMonth, - otherReferenceYearMonth, - hourDuration, - sameHourDuration, - sixtyMinuteDuration, - secondDuration, - sameSecondDuration, - thousandMillisecondDuration, - sameThousandMillisecondDuration, - millionMicrosecondDuration, - utcZoned, - aliasZoned, - offsetZoned, + Temporal.PlainDate.from(`2024-01-01`), + Temporal.PlainTime.from(`10:00`), shared, { id: 1 }, null, undefined, ] -// The intrinsic typed-array tag reads ordinary byte-view classes across realms. -const typedArrayTag = Object.getOwnPropertyDescriptor( - Object.getPrototypeOf(Uint8Array.prototype), - Symbol.toStringTag, -)!.get! - -// Date's native method reads the internal timestamp in either realm and -// rejects an object that only borrows the prototype. The reference captures -// that intrinsic directly; it does not import the production Date classifier. -const nativeDateGetTime = Date.prototype.getTime - -function nativeDateTimestamp(value: unknown): number | undefined { - if (typeof value !== `object` || value === null) return undefined - try { - return Reflect.apply(nativeDateGetTime, value, []) - } catch { - return undefined - } -} - +// A Buffer from another realm, such as Node's under jsdom, is not an +// `instanceof Uint8Array` here, so check the view's tag instead. function isBytes(value: unknown): value is Uint8Array { return ( ArrayBuffer.isView(value) && - Reflect.apply(typedArrayTag, value, []) === `Uint8Array` + Object.prototype.toString.call(value) === `[object Uint8Array]` ) } -// The polyfill methods supply the reference equality relation for the -// supported Temporal values. The model never imports production's classifier. -const temporalEqualities = [ - [`Instant`, Temporal.Instant.prototype.equals], - [`PlainDate`, Temporal.PlainDate.prototype.equals], - [`PlainDateTime`, Temporal.PlainDateTime.prototype.equals], - [`PlainMonthDay`, Temporal.PlainMonthDay.prototype.equals], - [`PlainTime`, Temporal.PlainTime.prototype.equals], - [`PlainYearMonth`, Temporal.PlainYearMonth.prototype.equals], - [`ZonedDateTime`, Temporal.ZonedDateTime.prototype.equals], -] as const - -const durationFields = [ - `years`, - `months`, - `weeks`, - `days`, - `hours`, - `minutes`, - `seconds`, - `milliseconds`, - `microseconds`, - `nanoseconds`, -] as const -const durationGetters = durationFields.map( - (field) => - Object.getOwnPropertyDescriptor(Temporal.Duration.prototype, field)!.get!, -) - -type TemporalBrand = - | { kind: string; equals: (other: unknown) => boolean } - | { kind: `Duration`; fields: ReadonlyArray } - -function temporalBrand(value: unknown): TemporalBrand | undefined { - if (typeof value !== `object` || value === null) return undefined - for (const [kind, equals] of temporalEqualities) { - try { - if (Reflect.apply(equals, value, [value]) === true) { - return { - kind, - equals: (other) => Reflect.apply(equals, value, [other]), - } - } - } catch { - // Ordinary non-Temporal objects fail these intrinsic calls. - } - } - try { - return { - kind: `Duration`, - fields: durationGetters.map((get) => Reflect.apply(get, value, [])), - } - } catch { - return undefined - } +function isTemporalValue(value: unknown): value is { toString: () => string } { + return ( + typeof value === `object` && + value !== null && + typeof (value as { [Symbol.toStringTag]?: unknown })[Symbol.toStringTag] === + `string` && + (value as { [Symbol.toStringTag]: string })[Symbol.toStringTag].startsWith( + `Temporal.`, + ) + ) } function isNotANumber(value: unknown): boolean { return ( (typeof value === `number` && Number.isNaN(value)) || - Number.isNaN(nativeDateTimestamp(value)) + (value instanceof Date && Number.isNaN(value.getTime())) ) } function asComparable(value: unknown): unknown { - return nativeDateTimestamp(value) ?? value + return value instanceof Date ? value.getTime() : value } function referenceEqual(left: unknown, right: unknown): boolean { @@ -295,32 +116,21 @@ function referenceEqual(left: unknown, right: unknown): boolean { return isNotANumber(left) && isNotANumber(right) } if (isBytes(left) || isBytes(right)) { - if (!isBytes(left) || !isBytes(right)) return false - // Materialize the ordinary byte sequences for an independent comparison; - // production walks indexed elements directly. - const leftBytes = Array.from(left) - const rightBytes = Array.from(right) return ( - leftBytes.length === rightBytes.length && - leftBytes.every((byte, index) => byte === rightBytes[index]) + isBytes(left) && + isBytes(right) && + left.length === right.length && + Array.from(left).every((byte, index) => byte === right[index]) ) } - const leftTemporal = temporalBrand(left) - const rightTemporal = temporalBrand(right) - if (leftTemporal || rightTemporal) { - if ( - !leftTemporal || - !rightTemporal || - leftTemporal.kind !== rightTemporal.kind - ) { - return false - } - if (`fields` in leftTemporal && `fields` in rightTemporal) { - return leftTemporal.fields.every( - (field, index) => field === rightTemporal.fields[index], - ) - } - return `equals` in leftTemporal && leftTemporal.equals(right) + if (isTemporalValue(left) || isTemporalValue(right)) { + return ( + isTemporalValue(left) && + isTemporalValue(right) && + (left as Record)[Symbol.toStringTag] === + (right as Record)[Symbol.toStringTag] && + left.toString() === right.toString() + ) } return asComparable(left) === asComparable(right) } @@ -328,17 +138,13 @@ function referenceEqual(left: unknown, right: unknown): boolean { function referenceIn(value: unknown, list: unknown): boolean | null { if (value === null || value === undefined) return null if (!Array.isArray(list)) return false - const length = list.length - for (let index = 0; index < length; index++) { - if (!(index in list)) continue - const item = list[index] - if (item !== null && item !== undefined && referenceEqual(item, value)) - return true - } - return false + return list.some( + (item) => + item !== null && item !== undefined && referenceEqual(item, value), + ) } -type InCase = { value: unknown; list: unknown; listFromRow: boolean } +type InCase = { value: unknown; list: Array; listFromRow: boolean } const item = fc.constantFrom(...pool) const inCase: fc.Arbitrary = fc.record({ @@ -362,45 +168,6 @@ function evaluateWithProduction(testCase: InCase): unknown { return evaluate({ row: { value: testCase.value, list: testCase.list } }) } -// The same normalized equality also feeds compiled `eq`. Check its public -// result alongside both `in` paths so a repaired list key cannot mask a -// remaining direct-predicate collision. -function evaluateEqWithProduction(left: unknown, right: unknown): unknown { - const evaluate = compileExpression( - new Func(`eq`, [ - new PropRef([`row`, `value`]), - new PropRef([`row`, `other`]), - ]), - ) - return evaluate({ row: { value: left, other: right } }) -} - -function checkEqualityPaths( - left: unknown, - right: unknown, - expected: boolean, -): void { - expect(referenceEqual(left, right)).toBe(expected) - for (const [direction, value, other] of [ - [`forward`, left, right], - [`reverse`, right, left], - ] as const) { - expect - .soft(evaluateEqWithProduction(value, other), `eq ${direction}`) - .toBe(expected) - for (const listFromRow of [false, true]) { - const testCase = { value, list: [other], listFromRow } - expect(referenceIn(value, testCase.list)).toBe(expected) - expect - .soft( - evaluateWithProduction(testCase), - `in ${listFromRow ? `row-supplied` : `constant`} ${direction}`, - ) - .toBe(expected) - } - } -} - function check(evaluate: Evaluate, testCase: InCase): void { const expected = referenceIn(testCase.value, testCase.list) const actual = evaluate(testCase) @@ -411,123 +178,37 @@ function check(evaluate: Evaluate, testCase: InCase): void { } } -// A kind-only Temporal list key is a plausible wrong normalization: it finds -// an equal copy, but also finds a different value of the same kind. Feed that -// answer through the same comparison used by the generated campaigns. -function evaluateWithKindOnlyTemporalKey(testCase: InCase): boolean | null { - const kind = temporalBrand(testCase.value)?.kind - if ( - kind !== undefined && - Array.isArray(testCase.list) && - testCase.list.some((candidate) => temporalBrand(candidate)?.kind === kind) - ) { - return true - } - return referenceIn(testCase.value, testCase.list) -} - function describeCase(testCase: InCase): string { - const show = (value: unknown): string => { - const dateTime = nativeDateTimestamp(value) - return typeof value === `bigint` + const show = (value: unknown): string => + typeof value === `bigint` ? `${value}n` : Object.is(value, -0) ? `-0` - : dateTime !== undefined - ? `Date(${dateTime})` + : value instanceof Date + ? `Date(${value.getTime()})` : isBytes(value) ? `bytes[${[...value].join(`,`)}]` : value === shared ? `shared` : typeof value === `object` && value !== null - ? (() => { - try { - return String(value) - } catch { - return `object` - } - })() + ? String(value) : typeof value === `string` ? JSON.stringify(value) : String(value) - } - const list = Array.isArray(testCase.list) - ? `[${testCase.list.map(show).join(`, `)}]` - : show(testCase.list) - return `value=${show(testCase.value)} list=${list} listFromRow=${testCase.listFromRow}` + return `value=${show(testCase.value)} list=[${testCase.list.map(show).join(`, `)}] listFromRow=${testCase.listFromRow}` } describe(`in evaluator`, () => { - const property = fc.asyncProperty(inCase, (testCase) => - Promise.resolve().then(() => check(evaluateWithProduction, testCase)), + const property = fc.property(inCase, (testCase) => + check(evaluateWithProduction, testCase), ) - it(`agrees with the reference over a fixed campaign`, async () => { - await fc.assert(property, { numRuns: oracleRuns(2000), seed: 0x2029 }) - }) - - it(`agrees with the reference over a random or replayed campaign`, async () => { - await fc.assert(property, oraclePropertyOptions(2000, PROPERTY)) - }) - - it(`reads replaced scalar list members after compilation`, () => { - const list: Array = [1] - const evaluate = compileExpression( - new Func(`in`, [new PropRef([`row`, `value`]), new Value(list)]), - ) - const checkCurrent = (value: unknown) => - expect(evaluate({ row: { value } })).toBe(referenceIn(value, list)) - - checkCurrent(1) - list[0] = 2 - checkCurrent(1) - checkCurrent(2) - list.push(3) - checkCurrent(3) - list.splice(0, 1) - checkCurrent(2) + it(`agrees with the reference over a fixed campaign`, () => { + fc.assert(property, { numRuns: oracleRuns(2000), seed: 0x2029 }) }) - it(`skips sparse holes and nullish items while retaining present members`, () => { - const list = Array(4) - list[1] = null - list[2] = undefined - list[3] = 2 - expect(0 in list).toBe(false) - const evaluate = compileExpression( - new Func(`in`, [new PropRef([`row`, `value`]), new Value(list)]), - ) - for (const value of [1, 2, null, undefined]) { - expect(evaluate({ row: { value } })).toBe(referenceIn(value, list)) - } - }) - - it(`reads a replaced Date member after compilation`, () => { - const list = [new Date(1)] - const evaluate = compileExpression( - new Func(`in`, [new PropRef([`row`, `value`]), new Value(list)]), - ) - const checkCurrent = (value: unknown) => - expect(evaluate({ row: { value } })).toBe(referenceIn(value, list)) - - checkCurrent(1) - list[0] = new Date(2) - checkCurrent(1) - checkCurrent(2) - }) - - it(`reads a replaced byte-array member after compilation`, () => { - const list = [new Uint8Array([1, 2])] - const evaluate = compileExpression( - new Func(`in`, [new PropRef([`row`, `value`]), new Value(list)]), - ) - const checkCurrent = (value: unknown) => - expect(evaluate({ row: { value } })).toBe(referenceIn(value, list)) - - checkCurrent(new Uint8Array([1, 2])) - list[0] = new Uint8Array([1, 3]) - checkCurrent(new Uint8Array([1, 2])) - checkCurrent(new Uint8Array([1, 3])) + it(`agrees with the reference over a random or replayed campaign`, () => { + fc.assert(property, oraclePropertyOptions(2000, PROPERTY)) }) it.each([ @@ -545,286 +226,13 @@ describe(`in evaluator`, () => { check(evaluateWithProduction, testCase) }) - it(`keeps frozen primitive constants in the same membership classes`, () => { - const lists: ReadonlyArray> = [ - Object.freeze([]), - Object.freeze([0, Number.NaN, 1n, `a`, PREFIX, null, undefined]), - ] - const values = [ - 0, - -0, - Number.NaN, - new Date(0), - runInNewContext(`new Date(0)`) as Date, - new Date(Number.NaN), - 1n, - 1, - `a`, - PREFIX, - `b`, - null, - undefined, - new Uint8Array([0]), - ] - for (const list of lists) { - const evaluate = compileExpression( - new Func(`in`, [new PropRef([`row`, `value`]), new Value(list)]), - ) - for (const value of values) { - expect(evaluate({ row: { value } })).toBe(referenceIn(value, list)) - } - } - }) - - it(`compares native bytes from another realm by content in every compiled equality path`, () => { - // The independent model's typed-array intrinsic accepts an ordinary - // native view across realms. - expect(foreignBytes instanceof Uint8Array).toBe(false) - expect(ArrayBuffer.isView(foreignBytes)).toBe(true) - expect(isBytes(foreignBytes)).toBe(true) - checkEqualityPaths(foreignBytes, new Uint8Array([1, 2]), true) - checkEqualityPaths(unequalForeignBytes, new Uint8Array([1, 2]), false) - }) - - it(`compares foreign native Dates by timestamp in every compiled equality path`, () => { - expect(foreignDate instanceof Date).toBe(false) - expect(foreignInvalidDate instanceof Date).toBe(false) - expect(nativeDateTimestamp(foreignDate)).toBe(1000) - expect(Number.isNaN(nativeDateTimestamp(foreignInvalidDate))).toBe(true) - checkEqualityPaths(foreignDate, localDate, true) - checkEqualityPaths(foreignDate, 1000, true) - checkEqualityPaths(laterForeignDate, localDate, false) - checkEqualityPaths(laterForeignDate, 1000, false) - // This Date comes from a second vm context. Its own timestamp must also - // match, so a classifier restricted to the first foreign prototype fails. - checkEqualityPaths(laterForeignDate, 1001, true) - checkEqualityPaths(foreignInvalidDate, localInvalidDate, true) - checkEqualityPaths(foreignInvalidDate, Number.NaN, true) - checkEqualityPaths(foreignInvalidDate, localDate, false) - }) - - it(`distinguishes unequal intrinsic values within each Temporal kind`, () => { - // Native equals supplies each kind's expected identity relation. The - // public compiler must reject a kind-only equality key in all three paths. - expect(plainDate.equals(otherPlainDate)).toBe(false) - expect(instant.equals(sameInstant)).toBe(true) - expect(instant.equals(otherInstant)).toBe(false) - expect(plainDateTime.equals(samePlainDateTime)).toBe(true) - expect(plainDateTime.equals(otherPlainDateTime)).toBe(false) - expect(plainTime.equals(samePlainTime)).toBe(true) - expect(plainTime.equals(otherPlainTime)).toBe(false) - for (const [left, equal, other] of [ - [plainDate, samePlainDate, otherPlainDate], - [instant, sameInstant, otherInstant], - [plainDateTime, samePlainDateTime, otherPlainDateTime], - [plainTime, samePlainTime, otherPlainTime], - ] as const) { - expect(pool).toContain(left) - expect(pool).toContain(equal) - expect(pool).toContain(other) - checkEqualityPaths(left, equal, true) - checkEqualityPaths(left, other, false) - } - }) - - it(`rejects a kind-only Temporal list key at the membership comparison`, () => { - for (const [value, equal, unequal] of [ - [plainDate, samePlainDate, otherPlainDate], - [instant, sameInstant, otherInstant], - [plainDateTime, samePlainDateTime, otherPlainDateTime], - [plainTime, samePlainTime, otherPlainTime], - ] as const) { - for (const listFromRow of [false, true]) { - expect( - evaluateWithKindOnlyTemporalKey({ - value, - list: [equal], - listFromRow, - }), - ).toBe(true) - expect(() => - check(evaluateWithKindOnlyTemporalKey, { - value, - list: [unequal], - listFromRow, - }), - ).toThrow(/in mismatch: expected false, got true/) - } - } - }) - - it.each([false, true])( - `returns false for a non-array %s list`, - (listFromRow) => { - check(evaluateWithProduction, { value: `a`, list: `a`, listFromRow }) - }, - ) - - it.each([false, true])( - `matches equivalent zone aliases in a %s list and rejects a distinct zone`, - (listFromRow) => { - // Temporal's own equality is the external semantic base, while the - // model below keeps the two expected outcomes explicit. - expect(utcZoned.equals(aliasZoned)).toBe(true) - expect(offsetZoned.equals(aliasZoned)).toBe(false) - check(evaluateWithProduction, { - value: aliasZoned, - list: [utcZoned], - listFromRow, - }) - check(evaluateWithProduction, { - value: offsetZoned, - list: [utcZoned], - listFromRow, - }) - }, - ) - - it.each([false, true])( - `matches Duration fields, while keeping equivalent representations distinct in a %s list`, - (listFromRow) => { - check(evaluateWithProduction, { - value: sameHourDuration, - list: [hourDuration], - listFromRow, - }) - check(evaluateWithProduction, { - value: sixtyMinuteDuration, - list: [hourDuration], - listFromRow, - }) - }, - ) - - it(`keeps reference ISO fields in PlainMonthDay and PlainYearMonth equality`, () => { - // Native equals, not default display, is the independent semantic base. - // Both pairs have equal display text but different reference ISO fields. - expect(leapMonthDay.toString()).toBe(otherReferenceMonthDay.toString()) - expect(leapMonthDay.equals(sameLeapMonthDay)).toBe(true) - expect(leapMonthDay.equals(otherReferenceMonthDay)).toBe(false) - expect(firstYearMonth.toString()).toBe(otherReferenceYearMonth.toString()) - expect(firstYearMonth.equals(sameFirstYearMonth)).toBe(true) - expect(firstYearMonth.equals(otherReferenceYearMonth)).toBe(false) - for (const value of [ - leapMonthDay, - sameLeapMonthDay, - otherReferenceMonthDay, - firstYearMonth, - sameFirstYearMonth, - otherReferenceYearMonth, - ]) { - expect(pool).toContain(value) - } - - checkEqualityPaths(leapMonthDay, sameLeapMonthDay, true) - checkEqualityPaths(leapMonthDay, otherReferenceMonthDay, false) - checkEqualityPaths(firstYearMonth, sameFirstYearMonth, true) - checkEqualityPaths(firstYearMonth, otherReferenceYearMonth, false) - }) - - it(`keeps Duration fields distinct across second and subsecond carries`, () => { - // Changelog #1519 promises that equivalent Duration representations stay - // unequal under `eq`. The captured intrinsic field getters in the model - // preserve that difference while their default display text collides. - expect(secondDuration.toString()).toBe( - thousandMillisecondDuration.toString(), - ) - expect(thousandMillisecondDuration.toString()).toBe( - millionMicrosecondDuration.toString(), - ) - for (const value of [ - secondDuration, - sameSecondDuration, - thousandMillisecondDuration, - sameThousandMillisecondDuration, - millionMicrosecondDuration, - ]) { - expect(pool).toContain(value) - } - - checkEqualityPaths(secondDuration, sameSecondDuration, true) - checkEqualityPaths( - thousandMillisecondDuration, - sameThousandMillisecondDuration, - true, - ) - checkEqualityPaths(secondDuration, thousandMillisecondDuration, false) - checkEqualityPaths( - thousandMillisecondDuration, - millionMicrosecondDuration, - false, - ) - }) - - it(`reads years, months, weeks, days, and nanoseconds in Duration equality`, () => { - const variedFields = [ - `years`, - `months`, - `weeks`, - `days`, - `nanoseconds`, - ] as const - const fields = { - years: 1, - months: 1, - weeks: 1, - days: 1, - nanoseconds: 1, - } - const baseline = Temporal.Duration.from(fields) - const equalCopy = Temporal.Duration.from(fields) - checkEqualityPaths(baseline, equalCopy, true) - - // Every variant differs in one intrinsic field. A comparator that drops - // that field predicts a false match, which the referenceIn check rejects. - for (const omitted of variedFields) { - const variant = Temporal.Duration.from({ ...fields, [omitted]: 2 }) - const equalWithoutField = ( - left: Temporal.Duration, - right: Temporal.Duration, - ) => - durationFields.every( - (field, index) => - field === omitted || - Reflect.apply(durationGetters[index]!, left, []) === - Reflect.apply(durationGetters[index]!, right, []), - ) - expect(equalWithoutField(baseline, variant)).toBe(true) - const wrong: Evaluate = ({ value, list }) => - Array.isArray(list) && - value instanceof Temporal.Duration && - list.some( - (candidate) => - candidate instanceof Temporal.Duration && - equalWithoutField(value, candidate), - ) - expect(() => - check(wrong, { value: variant, list: [baseline], listFromRow: false }), - ).toThrow(/in mismatch/) - checkEqualityPaths(baseline, variant, false) - } - }) - - it(`rejects display-string equality for equivalent zone aliases`, () => { - const lexical: Evaluate = ({ value, list }) => - Array.isArray(list) && - list.some((candidate) => String(candidate) === String(value)) - expect(() => - check(lexical, { - value: aliasZoned, - list: [utcZoned], - listFromRow: false, - }), - ).toThrow(/in mismatch/) - }) - it(`rejects an evaluator that compares raw values`, () => { // Calibration: without normalization a Date never equals its timestamp // and equal byte arrays are different objects. const raw: Evaluate = ({ value, list }) => value === null || value === undefined ? null - : Array.isArray(list) && list.some((entry) => entry === value) + : list.some((entry) => entry === value) expect(() => fc.assert( fc.property(inCase, (testCase) => check(raw, testCase)), diff --git a/packages/db/tests/query/custom-index-window-selection-oracle.test.ts b/packages/db/tests/query/custom-index-window-selection-oracle.test.ts deleted file mode 100644 index d87415a3d2..0000000000 --- a/packages/db/tests/query/custom-index-window-selection-oracle.test.ts +++ /dev/null @@ -1,198 +0,0 @@ -import { describe, expect, it } from 'vitest' -import { createCollection } from '../../src/collection/index.js' -import { BasicIndex } from '../../src/indexes/basic-index.js' -import { BTreeIndex } from '../../src/indexes/btree-index.js' -import { createLiveQueryCollection } from '../../src/query/live-query-collection.js' -import { evaluateReferenceExpression } from '../reference-expression-oracle.js' -import { withOracleCleanup } from '../utils.js' -import type { LoadSubsetOptions } from '../../src/types.js' - -/** - * A live query ordered by a numeric field publishes the requested slice of - * the provider's rows in ascending numeric order. The live-query architecture's - * ordered-request and atomic-window laws require the completed setWindow() - * result to include already cached rows as well as newly loaded rows. An index - * with a custom reverse comparator cannot stand in for that ordinary order. - * - * The bounded history caches ranks 20 and 40, installs indexes in one of four - * legal layouts, preloads a one-row window, then moves to offset one with two - * rows. A controlled provider holds four immutable source rows in physical - * order that differs from rank order. It sorts only when the request carries - * `orderBy`, and it never rewrites cached rows. A plain numeric sort and slice - * predicts both public checkpoints independently of index selection. Public - * snapshots and publication batches are compared after preload and after the - * window promise settles. This checks BasicIndex and BTreeIndex; it does not - * claim arbitrary comparators, ties, mutations, or remote provider behavior. - */ -type Row = { id: number; rank: number } -type IndexType = typeof BasicIndex | typeof BTreeIndex -type Layout = - `custom-first` | `ordinary-first` | `no-index` | `eager-custom-only` - -const rows: ReadonlyArray = [ - { id: 3, rank: 30 }, - { id: 1, rank: 10 }, - { id: 4, rank: 40 }, - { id: 2, rank: 20 }, -] -// The public order follows the query's rank expression, not source storage. -const expected = [...rows].sort((a, b) => a.rank - b.rank || a.id - b.id) -const expectedWindow = (offset: number, limit: number) => - expected.slice(offset, offset + limit).map(({ id }) => id) - -// This adapter models only the request grammar exercised here: a whole-source -// request, a finite prefix, or a cursor's current tie plus following rows. -// Its answers come from source facts and the request predicate, never indexes. -// Finite ordered acquisition needs a rank hint: without it, a provider slices -// physical order and cannot establish the requested prefix. -function requestedRows(options: LoadSubsetOptions): Array { - if (options.limit !== undefined) { - expect(options.orderBy).toBeDefined() - } - if (options.orderBy) { - expect(options.orderBy).toHaveLength(1) - expect(options.orderBy[0]!.expression).toMatchObject({ - type: `ref`, - path: [`rank`], - }) - expect(options.orderBy[0]!.compareOptions.direction).toBe(`asc`) - } - const matching = options.where - ? rows.filter( - (row) => evaluateReferenceExpression(options.where!, row) === true, - ) - : rows - const providerRows = options.orderBy - ? [...matching].sort((a, b) => a.rank - b.rank || a.id - b.id) - : matching - if (!options.cursor) { - const start = options.offset ?? 0 - return providerRows.slice( - start, - options.limit === undefined ? undefined : start + options.limit, - ) - } - const current = providerRows.filter((row) => - Boolean(evaluateReferenceExpression(options.cursor!.whereCurrent, row)), - ) - const following = providerRows - .filter((row) => - Boolean(evaluateReferenceExpression(options.cursor!.whereFrom, row)), - ) - .slice(0, options.limit) - return [ - ...new Map([...current, ...following].map((row) => [row.id, row])).values(), - ] -} - -async function observeWindowMove( - indexType: IndexType, - layout: Layout, - suffix: string, -) { - const replies: Array> = [] - const writes: Array = [] - const delivered = new Set([2, 4]) - const createSource = () => - createCollection({ - id: `custom-window-source-${suffix}-${layout}`, - getKey: ({ id }) => id, - syncMode: `on-demand`, - startSync: true, - autoIndex: layout === `eager-custom-only` ? `eager` : `off`, - defaultIndexType: indexType, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of rows.filter(({ id }) => delivered.has(id))) { - write({ type: `insert`, value: row }) - } - commit() - markReady() - return { - loadSubset: (options: LoadSubsetOptions) => { - const reply = requestedRows(options) - replies.push(reply.map(({ id }) => id)) - begin() - for (const row of reply) { - if (delivered.has(row.id)) continue - delivered.add(row.id) - writes.push(row.id) - write({ type: `insert`, value: row }) - } - const receipt = commit(options.signal) - return receipt === true ? Promise.resolve() : receipt - }, - } - }, - }, - }) - const publications: Array> = [] - let unsubscribe: (() => unknown) | undefined - let cleanupLive: (() => unknown) | undefined - let cleanupSource: (() => unknown) | undefined - return withOracleCleanup(async () => { - const source = createSource() - cleanupSource = () => source.cleanup() - await source.stateWhenReady() - const custom = () => - source.createIndex((row) => row.rank, { - indexType, - options: { compareFn: (a: number, b: number) => b - a }, - }) - const ordinary = () => source.createIndex((row) => row.rank, { indexType }) - if (layout === `custom-first`) { - custom() - ordinary() - } else if (layout === `ordinary-first`) { - ordinary() - custom() - } else if (layout === `eager-custom-only`) { - custom() - } - - const live = createLiveQueryCollection((query) => - query - .from({ row: source }) - .orderBy(({ row }) => row.rank, `asc`) - .offset(0) - .limit(1) - .select(({ row }) => ({ id: row.id, rank: row.rank })), - ) - cleanupLive = () => live.cleanup() - await live.preload() - const initial = Array.from(live.values(), ({ id }) => id) - expect(initial).toEqual(expectedWindow(0, 1)) - const subscription = live.subscribeChanges(() => { - publications.push(Array.from(live.values(), ({ id }) => id)) - }) - unsubscribe = () => subscription.unsubscribe() - publications.length = 0 - await live.utils.setWindow({ offset: 1, limit: 2 }) - const moved = Array.from(live.values(), ({ id }) => id) - - // The finite provider answer contains the cached rank-20 row, while - // its write stream does not. This reaches the cached-row merge path. - expect(replies.some((reply) => reply.join(`,`) === `1,2,3`)).toBe(true) - expect(writes).toEqual([1, 3]) - expect(moved).toEqual(expectedWindow(1, 2)) - expect(publications).toEqual([expectedWindow(1, 2)]) - if (layout === `eager-custom-only`) { - expect(source.indexes.size).toBe(2) - } - }, [() => unsubscribe?.(), () => cleanupLive?.(), () => cleanupSource?.()]) -} - -describe.each([ - [`BasicIndex`, BasicIndex], - [`BTreeIndex`, BTreeIndex], -] as const)(`%s ordered live window index selection`, (name, indexType) => { - it.each([ - `custom-first`, - `ordinary-first`, - `no-index`, - `eager-custom-only`, - ] as const)(`publishes the complete moved window with %s`, async (layout) => { - await observeWindowMove(indexType, layout, name) - }) -}) diff --git a/packages/db/tests/query/index-path-collision-oracle.test.ts b/packages/db/tests/query/index-path-collision-oracle.test.ts index c3f585382a..e1932ab1e8 100644 --- a/packages/db/tests/query/index-path-collision-oracle.test.ts +++ b/packages/db/tests/query/index-path-collision-oracle.test.ts @@ -18,14 +18,6 @@ * argument order, bound inclusivity, reversed operands, and bound field. * A selected-field ordering reads both paths through the public `$selected` * callback proxy. - * A separate range history stores ordinary native Dates from this and another - * JavaScript realm. Supplied timestamps predict inclusive and exclusive range - * keys before and after installing either index type. Local and foreign Dates - * at the exact bound distinguish a missing equality tie; a foreign Date after - * the bound distinguishes a scan or index that loses that row. The bound also - * comes from both realms, so index eligibility cannot recognize only local - * bounds. The timestamp model does not read production comparison or index - * classification. * * Production driver: `currentStateAsChanges`, Collection subscriptions, and * live-query Collections run the real predicates and callback compilers. @@ -42,23 +34,11 @@ * Known omissions: This bounded owner does not claim arbitrary path segments, * nullish values, custom collation, or incremental publication histories. * Lazy demand target deduplication has a separate compiler-boundary witness. - * The Date range history covers ordinary native Dates only. Modified Date - * objects, subclasses, borrowed prototypes, invalid Dates, and publication - * after later writes are outside its claimed domain. */ -import { runInNewContext } from 'node:vm' import { expect, it, vi } from 'vitest' import { createCollection } from '../../src/collection/index.js' -import { BasicIndex } from '../../src/indexes/basic-index.js' import { BTreeIndex } from '../../src/indexes/btree-index.js' -import { - add, - and, - gt, - gte, - lt, - lte, -} from '../../src/query/builder/functions.js' +import { add, and, gt, lt, lte } from '../../src/query/builder/functions.js' import { PropRef } from '../../src/query/ir.js' import { createLiveQueryCollection } from '../../src/query/live-query-collection.js' import { mockSyncCollectionOptions } from '../utils.js' @@ -183,125 +163,6 @@ it.each(cases)( }, ) -it.each([ - { name: `BasicIndex`, indexType: BasicIndex }, - { name: `BTreeIndex`, indexType: BTreeIndex }, -])( - `keeps ordinary foreign Dates in indexed timestamp ranges: $name`, - async ({ name, indexType }) => { - await checkWithCleanup(async (register) => { - // Supplied times, rather than production values, define the reference. - // Both realms have a Date at the exact lower bound. - const inputs = [ - { id: `local-before`, time: 0, foreign: false }, - { id: `local-equal`, time: 1, foreign: false }, - { id: `foreign-equal`, time: 1, foreign: true }, - { id: `foreign-after`, time: 2, foreign: true }, - ] as const - const dateRows = inputs.map(({ id, time, foreign }) => ({ - id, - v: foreign - ? (runInNewContext(`new Date(time)`, { time }) as Date) - : new Date(time), - })) - expect(dateRows[2]!.v instanceof Date).toBe(false) - expect(Date.prototype.getTime.call(dateRows[2]!.v)).toBe(1) - const bound = new Date(1) - const foreignBound = runInNewContext(`new Date(1)`) as Date - expect(foreignBound instanceof Date).toBe(false) - const field = new PropRef([`v`]) - const rangeCases = [ - { - name: `exclusive`, - where: gt(field, bound), - accepts: (time: number) => time > 1, - }, - { - name: `inclusive`, - where: gte(field, bound), - accepts: (time: number) => time >= 1, - }, - { - name: `foreign exclusive bound`, - where: gt(field, foreignBound), - accepts: (time: number) => time > 1, - }, - { - name: `foreign inclusive bound`, - where: gte(field, foreignBound), - accepts: (time: number) => time >= 1, - }, - ] - const matchingKeys = (accepts: (time: number) => boolean) => - inputs - .filter(({ time }) => accepts(time)) - .map(({ id }) => id) - .sort() - expect(matchingKeys(rangeCases[0]!.accepts)).toEqual([`foreign-after`]) - expect(matchingKeys(rangeCases[1]!.accepts)).toEqual([ - `foreign-after`, - `foreign-equal`, - `local-equal`, - ]) - - const collection = createCollection( - mockSyncCollectionOptions({ - id: `index-foreign-date-${name}`, - getKey: (row: (typeof dateRows)[number]) => row.id, - initialData: dateRows, - autoIndex: `off`, - }), - ) - register(() => collection.cleanup()) - await collection.stateWhenReady() - const snapshotKeys = ( - where: BasicExpression, - optimizedOnly = false, - ) => - collection - .currentStateAsChanges({ where, optimizedOnly }) - ?.map((change) => String(change.key)) - .sort() - - // Exact public keys at the ready Collection checkpoint expose missing - // foreign rows and missing ties, before and after index installation. - for (const range of rangeCases) { - expect(snapshotKeys(range.where), `scan ${range.name}`).toEqual( - matchingKeys(range.accepts), - ) - expect(snapshotKeys(range.where, true)).toBeUndefined() - } - const inclusive = rangeCases[1]! - const fullScan = snapshotKeys(inclusive.where)! - expect(() => - expect(fullScan.filter((key) => key !== `foreign-equal`)).toEqual( - matchingKeys(inclusive.accepts), - ), - ).toThrow() - - const index = collection.createIndex((row) => row.v, { indexType }) - const lookup = vi.spyOn(index, `lookup`) - for (const range of rangeCases) { - const expected = matchingKeys(range.accepts) - expect(snapshotKeys(range.where), `indexed ${range.name}`).toEqual( - expected, - ) - expect(lookup, `indexed ${range.name} reached`).toHaveBeenCalled() - lookup.mockClear() - expect( - snapshotKeys(range.where, true), - `optimized-only ${range.name}`, - ).toEqual(expected) - expect( - lookup, - `optimized-only ${range.name} reached`, - ).toHaveBeenCalled() - lookup.mockClear() - } - }) - }, -) - it.each([false, true])( `preserves distinct fields in a Collection subscription callback (indexed=%s)`, async (indexed) => { diff --git a/packages/db/tests/query/indexes.test.ts b/packages/db/tests/query/indexes.test.ts index 273872e326..eb7f4f17a5 100644 --- a/packages/db/tests/query/indexes.test.ts +++ b/packages/db/tests/query/indexes.test.ts @@ -286,14 +286,14 @@ describe(`Query Index Optimization`, () => { }) }) - it(`uses indexes for live queries with immutable inArray lists`, async () => { + it(`should use indexes for live queries with inArray conditions`, async () => { await withIndexTracking(collection, async (tracker) => { const liveQuery = createLiveQueryCollection({ query: (q: any) => q .from({ item: collection }) .where(({ item }: any) => - inArray(item.status, Object.freeze([`active`, `pending`])), + inArray(item.status, [`active`, `pending`]), ) .select(({ item }: any) => ({ id: item.id, diff --git a/packages/db/tests/query/load-subset-subquery.test.ts b/packages/db/tests/query/load-subset-subquery.test.ts index ee85a23cdd..ead6b921d3 100644 --- a/packages/db/tests/query/load-subset-subquery.test.ts +++ b/packages/db/tests/query/load-subset-subquery.test.ts @@ -117,10 +117,8 @@ const forwardingCases: ReadonlyArray = [ }, { name: `IN`, - // An immutable list can be forwarded once. A mutable captured list needs - // the full source so later edits can refresh existing query rows. - build: (order) => inArray(order.id, Object.freeze([1, 2, 3])), - expected: inArray(new PropRef([`id`]), Object.freeze([1, 2, 3])), + build: (order) => inArray(order.id, [1, 2, 3]), + expected: inArray(new PropRef([`id`]), [1, 2, 3]), }, { name: `NOT`, diff --git a/packages/db/tests/query/pooled-live-query-oracle.property.test.ts b/packages/db/tests/query/pooled-live-query-oracle.property.test.ts index f70d1a1152..694f1ab8a1 100644 --- a/packages/db/tests/query/pooled-live-query-oracle.property.test.ts +++ b/packages/db/tests/query/pooled-live-query-oracle.property.test.ts @@ -63,22 +63,17 @@ * * Known omissions: on-demand and persisted sources, `DbClient` hydration, * Suspense, `select`, and every other clause keep the live-query Collection - * and are outside this owner. A captured mutable `inArray` list in a residual - * predicate also keeps the live-query Collection: a pooled group would not - * notify the view about a source event in another group, although that event - * must refresh the whole result using the list's current members. + * and are outside this owner. */ import { fc, test as fcTest } from '@fast-check/vitest' import { describe, expect, it } from 'vitest' import { createCollection } from '../../src/collection/index.js' import { createLiveQueryObserver } from '../../src/live-query-observer.js' -import { resolveLiveQueryValue } from '../../src/live-query-options.js' import { Query } from '../../src/query/builder/index.js' import { and, createLiveQueryCollection, eq, - inArray, not, } from '../../src/query/index.js' import { createPooledLiveQuery } from '../../src/query/pooled-live-query.js' @@ -902,54 +897,6 @@ async function runHistory(history: History): Promise { describe(`pooled live query oracle`, () => { if (requestedReplayProperty === undefined) { - it(`routes a mutable residual inArray list to a live-query Collection`, async () => { - type PooledMembershipRow = { - id: string - g: string - v: number - revision: number - } - const source = createCollection( - mockSyncCollectionOptions({ - id: `pooled-mutable-in-${serial++}`, - getKey: (row) => row.id, - initialData: [ - { id: `one`, g: `g`, v: 1, revision: 0 }, - { id: `two`, g: `g`, v: 2, revision: 0 }, - { id: `trigger`, g: `other`, v: 3, revision: 0 }, - ], - }), - ) - const members = [1] - const query = new Query() - .from({ r: source }) - .where(({ r }) => and(eq(r.g, `g`), inArray(r.v, members))) - const resolved = resolveLiveQueryValue(query)! - const observer = createLiveQueryObserver(resolved, { mode: `wholesale` }) - const stop = observer.subscribe(() => {}) - await withOracleCleanup(async () => { - await resolved.preload() - const expected = () => - [ - { id: `one`, v: 1 }, - { id: `two`, v: 2 }, - ] - .filter((row) => members.includes(row.v)) - .map((row) => row.id) - const check = () => - expect([...observer.getSnapshot().state!.keys()]).toEqual(expected()) - check() - members[0] = 2 - source.utils.begin() - source.utils.write({ - type: `update`, - value: { id: `trigger`, g: `other`, v: 3, revision: 1 }, - }) - source.utils.commit() - check() - }, [stop, () => resolved.cleanup(), () => source.cleanup()]) - }) - // Generated fields are plain values; this pins a field whose getter // throws, which both paths treat as a row that does not match. it(`matches the live-query Collection when an eq path getter throws`, async () => { diff --git a/packages/db/tests/query/subset-dedupe.test.ts b/packages/db/tests/query/subset-dedupe.test.ts index 18600b45f0..53cabd4c0a 100644 --- a/packages/db/tests/query/subset-dedupe.test.ts +++ b/packages/db/tests/query/subset-dedupe.test.ts @@ -1,4 +1,3 @@ -import { runInNewContext } from 'node:vm' import { describe, expect, it, vi } from 'vitest' import { DeduplicatedLoadSubset } from '../../src/query/subset-dedupe' import { eq, gt } from '../../src/query/builder/functions' @@ -383,42 +382,6 @@ describe(`DeduplicatedLoadSubset`, () => { expect(loadSubset).toHaveBeenCalledTimes(2) }) - describe.each([`Date`, `Uint8Array`] as const)( - `request transport preserves %s predicate matches`, - (type) => { - it.each([`local`, `foreign`] as const)(`in the %s realm`, (realm) => { - const local = type === `Date` ? new Date(2) : new Uint8Array([1, 2]) - const foreign: unknown = runInNewContext( - type === `Date` ? `new Date(2)` : `new Uint8Array([1, 2])`, - ) - expect(foreign).not.toBe(local) - if (type === `Uint8Array`) { - // A genuine foreign view has the same native bytes despite having - // a different reference and failing this realm's instanceof check. - expect(foreign instanceof Uint8Array).toBe(false) - expect(ArrayBuffer.isView(foreign)).toBe(true) - expect(Array.from(foreign as Uint8Array)).toEqual([1, 2]) - } - const value = realm === `local` ? local : foreign - for (const where of [ - eq(ref(`value`), val(value)), - new Func(`in`, [ref(`value`), val([value])]), - ]) { - const options = { where } - const request = captureRequest(options) - expect(request).toBe(options) - expect(request.where).toBe(where) - const matches = compileSingleRowExpression(request.where!) - // Transport keeps the literal object; ordinary native value - // equality uses its time or bytes across JavaScript environments. - expect( - [foreign, local].map((item) => matches({ value: item })), - ).toEqual([true, true]) - } - }) - }, - ) - it.each([`coalesce`, `caseWhen`] as const)( `preserves membership results through %s`, (wrapper) => { diff --git a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts index 265f0bafa6..67f05b11e7 100644 --- a/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts +++ b/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts @@ -56,50 +56,14 @@ * must be routed to exactly the subscribers they can satisfy; and the public * `Collection.currentStateAsChanges` snapshot. * - * Refinement check: live-query and subscriber key sets equal the model after - * attachment and after each sync transaction commits. The generated history - * checks a direct snapshot at attachment. Direct subscribers are reconstructed - * from their callback batches. The fixed Date matrix checks direct snapshots - * before and after its sync transaction. Two fixed + * Refinement check: each consumer's exact key set equals the model after the + * subscribers attach and after each sync transaction commits. Direct + * subscribers are reconstructed from their callback batches. Two fixed * witnesses check boundaries outside the generated grammar: Collection * readiness reaches a filtered subscriber as one empty batch, and an eager * source restarted after cleanup retracts a vanished row even when its first * batch cannot satisfy the predicate. A third witness checks that a same-key * update which stays TRUE publishes its new row and exact subscriber payload. - * A focused Date matrix checks ordinary native Dates from this and another - * JavaScript realm. It compares direct snapshot, filtered subscriber, and live - * query keys after preload and one sync transaction; an explicit BasicIndex - * lookup checks the indexed equality key when an index is present. - * - * Captured-list law: a caller-owned array passed to `inArray` remains live - * input. Editing that array alone does not publish. At the next delivered - * source event, the entire retained result, including untouched rows, uses - * its current members. The fixed model starts with authored source rows and - * filters them against the list at that checkpoint; it does not reuse the - * compiled predicate, source index, or emitted delta. The direct Collection - * snapshot, retained live query, ordered limit, on-demand source, join, - * filtered QueryRef, and correlated include witnesses compare public results - * with that model. On-demand ordered, joined, and include receivers check that - * the first acquisition is unfiltered when the captured list is mutable. An - * unbounded ordered receiver also requires no source order hint, checks local - * order after unsorted delivery, and checks readiness when its full-source - * request settles while the source is loading. - * An Effect receiver checks that the same acquisition rule reaches its source. - * The retained direct query also checks that a list edit alone preserves the - * old public snapshot and sends no callback, including after a microtask turn. - * At the next source transaction, one subscriber callback contains the - * complete insert/delete batch and reads the new whole result. It checks no - * duplicate event when the list did not change. Separate fixed - * Date and byte-array cases replace members of the captured list, then check - * the next source event. Mutating an item in place is outside this law. - * An ordered limit witness deletes a source row - * outside the old loaded prefix after a list edit; it requires the newly - * matching row to appear, so ordered acquisition must retain all source rows - * while this live list is captured. Joined and included on-demand sources use - * an unrelated source delete to prove that keyed lazy demand cannot hide the - * event. A 40-row work witness requires an unchanged - * list to inspect fewer than ten list slots and a changed list to revisit - * retained candidates; a constant-key all-row rescan fails the first bound. * * Calibration: pinned histories separate Kleene from two-valued logic, and a * checker test requires the two-valued answer to fail. Hostile production @@ -110,42 +74,21 @@ * reconciliation each passed the pre-existing `@tanstack/db` suite and fail * here. * - * Known omissions: the generated WHERE grammar still covers `eq`, `not`, - * `and`, and `or`, not other comparison operators, `like`, or arbitrary - * `inArray` histories. The fixed captured-list witnesses cover the named - * scalar, Date, byte, ordered (including unseen deletion), on-demand, joined, - * QueryRef, and include cuts; - * they do not establish those paths for all list values, source schedules, - * optimizers, or real providers. Temporal operands, general join and ordering - * laws, optimistic updates and deletes, truncate, and failed replay remain - * with their structural comparison, cold-join, ordered, and subscription - * replay owners. + * Known omissions: comparison operators other than `eq`, `in`, `like`, + * Temporal and binary operands, joins, ordering, optimistic updates and + * deletes, truncate, and failed replay are outside this owner. The structural + * comparison, cold-join, and subscription replay oracles own those boundaries. * Generated cleanup and restart histories for filtered subscribers remain * open; the lifecycle publication owner needs a predicate dimension to reach * them. The mock sync source is a controlled provider; this oracle claims only * how the Collection filters and routes the sync transactions it supplies. - * The Date matrix covers ordinary native `node:vm` Dates. Modified Date - * objects, subclasses, borrowed prototypes, proxies, and other hosts are - * outside its claimed domain. - * A pinned mixed-scalar history checks an indexed source after deleting and - * replacing rows. Its WHERE result still follows `eq` truth; the index's - * private order must not lose an equality member or interrupt publication. */ -import { runInNewContext } from 'node:vm' import { fc, test as fcTest } from '@fast-check/vitest' import { describe, expect, it } from 'vitest' import { createCollection } from '../../src/collection/index.js' import { BasicIndex } from '../../src/indexes/basic-index.js' -import { - and, - eq, - inArray, - materialize, - not, - or, -} from '../../src/query/builder/functions.js' +import { and, eq, not, or } from '../../src/query/builder/functions.js' import { Func, PropRef, Value } from '../../src/query/ir.js' -import { createEffect } from '../../src/query/effect.js' import { createLiveQueryCollection } from '../../src/query/index.js' import { oraclePropertyOptions, @@ -310,40 +253,6 @@ function syncedModelRow(id: string, v: FieldValue): ModelRow { return { id, v, $synced: true, $origin: `remote`, $key: id } } -// This focused model starts from tags, not runtime Date classification. A -// native Date and a number with the same finite timestamp are equal. Invalid -// Dates and NaN occupy one equality class under the documented float rule. -type DateEqualityTag = - | { - kind: `date` - realm: `local` | `foreign` - timestamp: number | `invalid` - } - | { kind: `number`; value: number | `NaN` } - -type TaggedDateRow = { - id: string - tag: DateEqualityTag -} - -function dateEqualityClass(tag: DateEqualityTag): number | `unordered` { - if (tag.kind === `date`) { - return tag.timestamp === `invalid` ? `unordered` : tag.timestamp - } - return tag.value === `NaN` ? `unordered` : tag.value -} - -function expectedTaggedDateKeys( - rows: ReadonlyArray, - literal: DateEqualityTag, -): Array { - const expectedClass = dateEqualityClass(literal) - return rows - .filter((row) => dateEqualityClass(row.tag) === expectedClass) - .map((row) => row.id) - .sort() -} - // --------------------------------------------------------------------------- // History grammar // --------------------------------------------------------------------------- @@ -541,18 +450,6 @@ const pinnedPrefilteredSnapshot: History = { // Each change history moves rows across a predicate whose equality operand a // skip-if-no-match shortcut can misread. const pinnedChangeHistories: ReadonlyArray = [ - { - // The default ordered index must keep distinct string and number rows - // available to exact equality after a delete and two same-key updates. - kind: `changes`, - rows: [`b`, 1, `a`], - predicate: fieldEq(`a`), - transactions: [ - [{ type: `delete`, target: 0 }], - [{ type: `update`, target: 0, v: `b` }], - [{ type: `update`, target: 0, v: 2 }], - ], - }, { // `or` operands are alternatives, not conjuncts. kind: `changes`, @@ -625,36 +522,6 @@ const pinnedChangeHistories: ReadonlyArray = [ }, ] -// The fixed grammar crosses the Date constructor's realm with equal, unequal, -// and invalid timestamps. Number literals test the Date/number and -// invalid-Date/NaN equality classes. A later sync transaction moves four rows -// between those classes. -const taggedDateRows: ReadonlyArray = [ - { id: `local-one`, tag: { kind: `date`, realm: `local`, timestamp: 1 } }, - { id: `foreign-one`, tag: { kind: `date`, realm: `foreign`, timestamp: 1 } }, - { id: `foreign-two`, tag: { kind: `date`, realm: `foreign`, timestamp: 2 } }, - { - id: `local-invalid`, - tag: { kind: `date`, realm: `local`, timestamp: `invalid` }, - }, - { - id: `foreign-invalid`, - tag: { kind: `date`, realm: `foreign`, timestamp: `invalid` }, - }, - { id: `number-one`, tag: { kind: `number`, value: 1 } }, - { id: `number-nan`, tag: { kind: `number`, value: `NaN` } }, -] - -const taggedDateLiterals: ReadonlyArray = [ - { kind: `date`, realm: `local`, timestamp: 1 }, - { kind: `date`, realm: `foreign`, timestamp: 1 }, - { kind: `date`, realm: `foreign`, timestamp: 2 }, - { kind: `date`, realm: `local`, timestamp: `invalid` }, - { kind: `date`, realm: `foreign`, timestamp: `invalid` }, - { kind: `number`, value: 1 }, - { kind: `number`, value: `NaN` }, -] - // --------------------------------------------------------------------------- // Production driver // --------------------------------------------------------------------------- @@ -727,133 +594,6 @@ function sourceRow(id: string, v: FieldValue): Row { return v === MISSING ? { id } : { id, v } } -function runtimeTaggedDateValue(tag: DateEqualityTag): Date | number { - if (tag.kind === `number`) { - return tag.value === `NaN` ? Number.NaN : tag.value - } - const timestamp = tag.timestamp === `invalid` ? Number.NaN : tag.timestamp - return tag.realm === `foreign` - ? (runInNewContext(`new Date(timestamp)`, { timestamp }) as Date) - : new Date(timestamp) -} - -type TaggedValueRow = { id: string; v: Date | number } - -// The tag model predicts keys before the runtime values are constructed. The -// driver observes the public live query, filtered subscription, and direct -// Collection snapshot after preload and after a committed source transaction. -// The explicit index lookup checks the physical equality key at both cuts. -async function observeTaggedDateEquality( - literal: DateEqualityTag, - indexed: boolean, -): Promise { - const rows = taggedDateRows.map((row) => ({ ...row })) - const literalValue = runtimeTaggedDateValue(literal) - const collection = createCollection( - mockSyncCollectionOptions({ - id: `where-publication-date-${collectionSerial++}`, - getKey: (row) => row.id, - autoIndex: `off`, - initialData: rows.map((row) => ({ - id: row.id, - v: runtimeTaggedDateValue(row.tag), - })), - }), - ) - const live = createLiveQueryCollection((q) => - q.from({ row: collection }).where(({ row }) => eq(row.v, literalValue)), - ) - const predicate = new Func(`eq`, [ - new PropRef([`v`]), - new Value(literalValue), - ]) - const visible = new Set() - let subscription: { unsubscribe: () => void } | undefined - - await withOracleCleanup(async () => { - await collection.stateWhenReady() - const index = indexed - ? collection.createIndex((row) => row.v, { indexType: BasicIndex }) - : undefined - await live.preload() - subscription = collection.subscribeChanges( - (changes) => { - for (const change of changes) { - if (change.type === `delete`) visible.delete(change.key) - else visible.add(change.key) - } - }, - { includeInitialState: true, whereExpression: predicate }, - ) - - const check = (checkpoint: string) => { - const expected = expectedTaggedDateKeys(rows, literal) - const label = `${checkpoint}, ${indexed ? `indexed` : `unindexed`}, ${JSON.stringify(literal)}` - expect( - [...live.keys()].map(String).sort(), - `${label}: live query`, - ).toEqual(expected) - expect( - [...visible].map(String).sort(), - `${label}: filtered subscriber`, - ).toEqual(expected) - const snapshot = collection.currentStateAsChanges({ where: predicate }) - expect( - snapshot?.map((change) => String(change.key)).sort(), - `${label}: direct snapshot`, - ).toEqual(expected) - if (index) { - expect( - [...index.lookup(`eq`, literalValue)].map(String).sort(), - `${label}: BasicIndex equality lookup`, - ).toEqual(expected) - const optimized = collection.currentStateAsChanges({ - where: predicate, - optimizedOnly: true, - }) - expect( - optimized?.map((change) => String(change.key)).sort(), - `${label}: optimized public snapshot`, - ).toEqual(expected) - } - } - - check(`after subscribe`) - const transitions: ReadonlyArray<{ index: number; tag: DateEqualityTag }> = - [ - { - index: 0, - tag: { kind: `date`, realm: `foreign`, timestamp: `invalid` }, - }, - { index: 2, tag: { kind: `date`, realm: `foreign`, timestamp: 1 } }, - { index: 3, tag: { kind: `date`, realm: `foreign`, timestamp: 2 } }, - { index: 5, tag: { kind: `date`, realm: `local`, timestamp: 2 } }, - ] - collection.utils.begin() - for (const transition of transitions) { - collection.utils.write({ - type: `update`, - value: { - id: rows[transition.index]!.id, - v: runtimeTaggedDateValue(transition.tag), - }, - }) - } - collection.utils.commit() - for (const transition of transitions) { - rows[transition.index] = { - id: rows[transition.index]!.id, - tag: transition.tag, - } - } - check(`after transaction`) - }, [ - () => subscription?.unsubscribe(), - () => live.cleanup(), - () => collection.cleanup(), - ]) -} - type Consumer = | `live query` | `subscriber with initial state` @@ -1243,1073 +983,6 @@ describe(`WHERE predicate publication oracle`, () => { ]) }) - it(`direct snapshots read current inArray members after a source update`, async () => { - type MembershipRow = { id: string; v: number; revision: number } - const source = createCollection( - mockSyncCollectionOptions({ - id: `where-publication-snapshot-in-${collectionSerial++}`, - getKey: (row) => row.id, - initialData: [ - { id: `one`, v: 1, revision: 0 }, - { id: `two`, v: 2, revision: 0 }, - ], - }), - ) - const members = [1] - const where = inArray(new PropRef([`v`]), members) - await withOracleCleanup(async () => { - await source.stateWhenReady() - const expected = (rows: ReadonlyArray) => - rows.filter((row) => members.includes(row.v)).map((row) => row.id) - const initialRows = [ - { id: `one`, v: 1, revision: 0 }, - { id: `two`, v: 2, revision: 0 }, - ] - const snapshotKeys = () => - source - .currentStateAsChanges({ where }) - ?.map((change) => change.key) - .sort() - expect(snapshotKeys()).toEqual(expected(initialRows)) - - members[0] = 2 - source.utils.begin() - source.utils.write({ - type: `update`, - value: { id: `one`, v: 1, revision: 1 }, - }) - source.utils.write({ - type: `update`, - value: { id: `two`, v: 2, revision: 1 }, - }) - source.utils.commit() - expect(snapshotKeys()).toEqual( - expected(initialRows.map((row) => ({ ...row, revision: 1 }))), - ) - }, [() => source.cleanup()]) - }) - - it(`a source event refreshes untouched rows after a captured inArray list changes`, async () => { - type MembershipRow = { id: string; v: number; revision: number } - const source = createCollection( - mockSyncCollectionOptions({ - id: `where-publication-live-in-${collectionSerial++}`, - getKey: (row) => row.id, - autoIndex: `off`, - initialData: [ - { id: `one`, v: 1, revision: 0 }, - { id: `two`, v: 2, revision: 0 }, - { id: `trigger`, v: 3, revision: 0 }, - ], - }), - ) - const members = [1] - const modelRows = [ - { id: `one`, v: 1 }, - { id: `two`, v: 2 }, - { id: `trigger`, v: 3 }, - ] - const live = createLiveQueryCollection((q) => - q.from({ row: source }).where(({ row }) => inArray(row.v, members)), - ) - // Retain callback boundaries and read the Collection while each callback - // runs. Flattening changes would accept two partial publications with the - // right eventual keys, even though one source transaction owes one - // coherent public result. - const batches: Array<{ - changes: Array<{ type: string; key: string }> - visibleKeys: Array - }> = [] - let unsubscribe: (() => void) | undefined - await withOracleCleanup(async () => { - await live.preload() - const subscription = live.subscribeChanges( - (changes) => { - batches.push({ - changes: changes - .map((change) => ({ - type: change.type, - key: String(change.key), - })) - .sort((left, right) => left.key.localeCompare(right.key)), - visibleKeys: [...live.keys()].map(String).sort(), - }) - }, - { includeInitialState: false }, - ) - unsubscribe = () => subscription.unsubscribe() - const expected = () => - modelRows - .filter((row) => members.includes(row.v)) - .map((row) => row.id) - .sort() - const check = () => expect([...live.keys()].sort()).toEqual(expected()) - check() - expect(batches).toEqual([]) - - members[0] = 2 - // A list edit is only an input change. The previous public snapshot - // remains in place until a source event clocks the retained query. - expect([...live.keys()].sort()).toEqual([`one`]) - expect(batches).toEqual([]) - await Promise.resolve() - expect([...live.keys()].sort()).toEqual([`one`]) - expect(batches).toEqual([]) - source.utils.begin() - source.utils.write({ - type: `update`, - value: { id: `trigger`, v: 3, revision: 1 }, - }) - source.utils.commit() - check() - expect(batches).toEqual([ - { - changes: [ - { type: `delete`, key: `one` }, - { type: `insert`, key: `two` }, - ], - visibleKeys: [`two`], - }, - ]) - batches.length = 0 - - members.push(1) - expect([...live.keys()].sort()).toEqual([`two`]) - expect(batches).toEqual([]) - await Promise.resolve() - expect([...live.keys()].sort()).toEqual([`two`]) - expect(batches).toEqual([]) - source.utils.begin() - source.utils.write({ - type: `update`, - value: { id: `trigger`, v: 3, revision: 2 }, - }) - source.utils.commit() - check() - expect(batches).toEqual([ - { - changes: [{ type: `insert`, key: `one` }], - visibleKeys: [`one`, `two`], - }, - ]) - batches.length = 0 - - // A source update with no further list edit cannot republish the - // unchanged result merely because the reduction revisits every row. - source.utils.begin() - source.utils.write({ - type: `update`, - value: { id: `trigger`, v: 3, revision: 3 }, - }) - source.utils.commit() - check() - expect(batches).toEqual([]) - - source.utils.begin() - source.utils.write({ type: `delete`, key: `two` }) - source.utils.commit() - modelRows.splice( - modelRows.findIndex((row) => row.id === `two`), - 1, - ) - check() - expect(batches).toEqual([ - { - changes: [{ type: `delete`, key: `two` }], - visibleKeys: [`one`], - }, - ]) - }, [() => unsubscribe?.(), () => live.cleanup(), () => source.cleanup()]) - }) - - it(`a source event reads replaced Date and byte-array list members`, async () => { - type MembershipRow = { - id: string - date: Date - bytes: Uint8Array - revision: number - } - const source = createCollection( - mockSyncCollectionOptions({ - id: `where-publication-replaced-items-${collectionSerial++}`, - getKey: (row) => row.id, - initialData: [1, 2, 3].map((value) => ({ - id: String(value), - date: new Date(value), - bytes: new Uint8Array([value]), - revision: 0, - })), - }), - ) - const dateMembers = [new Date(1)] - const byteMembers = [new Uint8Array([1])] - const dates = createLiveQueryCollection((q) => - q - .from({ row: source }) - .where(({ row }) => inArray(row.date, dateMembers)), - ) - const bytes = createLiveQueryCollection((q) => - q - .from({ row: source }) - .where(({ row }) => inArray(row.bytes, byteMembers)), - ) - await withOracleCleanup(async () => { - await Promise.all([dates.preload(), bytes.preload()]) - expect([...dates.keys()]).toEqual([`1`]) - expect([...bytes.keys()]).toEqual([`1`]) - - dateMembers[0] = new Date(2) - byteMembers[0] = new Uint8Array([2]) - source.utils.begin() - source.utils.write({ - type: `update`, - value: { - id: `3`, - date: new Date(3), - bytes: new Uint8Array([3]), - revision: 1, - }, - }) - source.utils.commit() - expect([...dates.keys()]).toEqual([`2`]) - expect([...bytes.keys()]).toEqual([`2`]) - }, [() => dates.cleanup(), () => bytes.cleanup(), () => source.cleanup()]) - }) - - it(`an ordered limit can admit an untouched row from a changed inArray list`, async () => { - type MembershipRow = { - id: string - v: number - rank: number - revision: number - } - const source = createCollection( - mockSyncCollectionOptions({ - id: `where-publication-ordered-in-${collectionSerial++}`, - getKey: (row) => row.id, - autoIndex: `eager`, - initialData: [0, 1, 2, 3, 4].map((v) => ({ - id: String(v), - v, - rank: v, - revision: 0, - })), - }), - ) - const members = [1] - const live = createLiveQueryCollection((q) => - q - .from({ row: source }) - .where(({ row }) => inArray(row.v, members)) - .orderBy(({ row }) => row.rank) - .limit(1), - ) - await withOracleCleanup(async () => { - await live.preload() - const expected = () => - [0, 1, 2, 3, 4] - .filter((value) => members.includes(value)) - .sort((left, right) => left - right) - .slice(0, 1) - .map(String) - const check = () => expect([...live.keys()]).toEqual(expected()) - check() - // The previously excluded row 0 is before the old top row. An - // unfiltered ordered prefix must already hold it. - members.push(0) - source.utils.begin() - source.utils.write({ - type: `update`, - value: { id: `2`, v: 2, rank: 2, revision: 1 }, - }) - source.utils.commit() - check() - // Removing every prefix match requires the loader to continue until - // it finds the next current member. - members.splice(0, members.length, 4) - source.utils.begin() - source.utils.write({ - type: `update`, - value: { id: `2`, v: 2, rank: 2, revision: 2 }, - }) - source.utils.commit() - check() - }, [() => live.cleanup(), () => source.cleanup()]) - }) - - it(`an unseen ordered-source delete refreshes changed inArray members`, async () => { - type MembershipRow = { id: string; v: number; rank: number } - const source = createCollection( - mockSyncCollectionOptions({ - id: `where-publication-unseen-ordered-delete-${collectionSerial++}`, - getKey: (row) => row.id, - autoIndex: `eager`, - initialData: [0, 1, 2, 3, 4].map((v) => ({ - id: String(v), - v, - rank: v, - })), - }), - ) - const members = [1] - const live = createLiveQueryCollection((q) => - q - .from({ row: source }) - .where(({ row }) => inArray(row.v, members)) - .orderBy(({ row }) => row.rank) - .limit(1), - ) - await withOracleCleanup(async () => { - await live.preload() - expect([...live.keys()]).toEqual([`1`]) - members[0] = 0 - // The deleted rank-4 row is outside the acquired prefix. The event - // still clocks the complete result to the list's current membership. - source.utils.begin() - source.utils.write({ type: `delete`, key: `4` }) - source.utils.commit() - expect([...live.keys()]).toEqual([`0`]) - }, [() => live.cleanup(), () => source.cleanup()]) - }) - - it(`an ordered on-demand query requests the full source for a mutable inArray list`, async () => { - type MembershipRow = { id: string; v: number; rank: number } - const rows = [0, 1, 2, 3, 4].map((v) => ({ - id: String(v), - v, - rank: v, - })) - const requests: Array<{ - hasWhere: boolean - hasOrder: boolean - hasLimit: boolean - }> = [] - let begin!: () => void - let write!: (message: { type: `delete`; key: string }) => void - let commit!: () => unknown - const source = createCollection({ - id: `where-publication-ordered-demand-in-${collectionSerial++}`, - getKey: (row) => row.id, - syncMode: `on-demand`, - startSync: true, - autoIndex: `eager`, - defaultIndexType: BasicIndex, - sync: { - sync: (params) => { - begin = params.begin - write = params.write - commit = params.commit - params.markReady() - return { - loadSubset: (options) => { - requests.push({ - hasWhere: options.where !== undefined, - hasOrder: options.orderBy !== undefined, - hasLimit: options.limit !== undefined, - }) - params.begin() - for (const row of rows) { - params.write({ type: `insert`, value: row }) - } - params.commit() - return true - }, - } - }, - }, - }) - const members = [1] - const live = createLiveQueryCollection((q) => - q - .from({ row: source }) - .where(({ row }) => inArray(row.v, members)) - .orderBy(({ row }) => row.rank) - .limit(1), - ) - await withOracleCleanup(async () => { - await live.preload() - expect(requests[0]).toEqual({ - hasWhere: false, - hasOrder: false, - hasLimit: false, - }) - expect([...live.keys()]).toEqual([`1`]) - - members[0] = 0 - begin() - write({ type: `delete`, key: `4` }) - commit() - expect([...live.keys()]).toEqual([`0`]) - }, [() => live.cleanup(), () => source.cleanup()]) - }) - - it(`an unbounded ordered inArray query requests unsorted full source and readies at subset settlement`, async () => { - type MembershipRow = { id: string; v: number; revision: number } - const rows: Array = [ - { id: `a-two`, v: 2, revision: 0 }, - { id: `z-one`, v: 1, revision: 0 }, - { id: `trigger`, v: 3, revision: 0 }, - ] - const requests: Array<{ - hasWhere: boolean - hasOrder: boolean - hasLimit: boolean - }> = [] - let settleLoad!: () => void - let begin!: () => void - let write!: (message: { type: `update`; value: MembershipRow }) => void - let commit!: () => unknown - const source = createCollection({ - id: `where-publication-unbounded-ordered-demand-${collectionSerial++}`, - getKey: (row) => row.id, - syncMode: `on-demand`, - sync: { - sync: (params) => { - begin = params.begin - write = params.write - commit = params.commit - // This source deliberately stays loading. Its full-source subset - // receipt, rather than source readiness, releases the live query. - return { - loadSubset: (options) => { - requests.push({ - hasWhere: options.where !== undefined, - hasOrder: options.orderBy !== undefined, - hasLimit: options.limit !== undefined, - }) - if ( - options.where !== undefined || - options.orderBy !== undefined || - options.limit !== undefined - ) { - return Promise.reject(new Error(`full-source-only adapter`)) - } - params.begin() - for (const row of rows) { - params.write({ type: `insert`, value: row }) - } - params.commit() - return new Promise((resolve) => { - settleLoad = resolve - }) - }, - } - }, - }, - }) - const members = [1] - const live = createLiveQueryCollection((q) => - q - .from({ row: source }) - .where(({ row }) => inArray(row.v, members)) - .orderBy(({ row }) => row.v), - ) - await withOracleCleanup(async () => { - const preload = live.preload() - let preloadSettled = false - void preload.then( - () => { - preloadSettled = true - }, - () => { - preloadSettled = true - }, - ) - void preload.catch(() => {}) - await Promise.resolve() - expect(requests).toEqual([ - { hasWhere: false, hasOrder: false, hasLimit: false }, - ]) - expect(live.status).toBe(`loading`) - - // The independent model filters every retained source row against - // the current list when an unrelated source update clocks the graph. - members.splice(0, members.length, 2, 1) - begin() - write({ type: `update`, value: { id: `trigger`, v: 3, revision: 1 } }) - commit() - await Promise.resolve() - expect(live.status).toBe(`loading`) - expect(source.status).toBe(`loading`) - expect(preloadSettled).toBe(false) - settleLoad() - await preload - expect(preloadSettled).toBe(true) - expect(source.status).toBe(`loading`) - expect(live.status).toBe(`ready`) - expect([...live.keys()]).toEqual([`z-one`, `a-two`]) - }, [() => live.cleanup(), () => source.cleanup()]) - }) - - it(`an Effect with captured inArray membership requests unsorted full source`, async () => { - type MembershipRow = { id: string; v: number } - const rows: Array = [ - { id: `one`, v: 1 }, - { id: `two`, v: 2 }, - ] - const requests: Array<{ - hasWhere: boolean - hasOrder: boolean - hasLimit: boolean - }> = [] - const source = createCollection({ - id: `where-publication-effect-demand-${collectionSerial++}`, - getKey: (row) => row.id, - syncMode: `on-demand`, - sync: { - sync: (params) => { - params.markReady() - return { - loadSubset: (options) => { - requests.push({ - hasWhere: options.where !== undefined, - hasOrder: options.orderBy !== undefined, - hasLimit: options.limit !== undefined, - }) - params.begin() - for (const row of rows) { - params.write({ type: `insert`, value: row }) - } - params.commit() - return true - }, - } - }, - }, - }) - const members = [2] - const entered: Array = [] - let effect: ReturnType | undefined - await withOracleCleanup(async () => { - effect = createEffect({ - query: (q) => - q - .from({ row: source }) - .where(({ row }) => inArray(row.v, members)) - .orderBy(({ row }) => row.v), - onEnter: ({ value }) => { - entered.push(value.id) - }, - }) - await new Promise((resolve) => setTimeout(resolve, 0)) - expect(requests).toEqual([ - { hasWhere: false, hasOrder: false, hasLimit: false }, - ]) - expect(entered).toEqual([`two`]) - }, [() => effect?.dispose(), () => source.cleanup()]) - }) - - it(`an on-demand query acquires rows for a captured inArray list's future members`, async () => { - type MembershipRow = { id: string; v: number; revision: number } - const rows: Array = [ - { id: `one`, v: 1, revision: 0 }, - { id: `two`, v: 2, revision: 0 }, - { id: `trigger`, v: 3, revision: 0 }, - ] - let begin!: () => void - let write!: (message: { type: `update`; value: MembershipRow }) => void - let commit!: () => unknown - const source = createCollection({ - id: `where-publication-demand-in-${collectionSerial++}`, - getKey: (row) => row.id, - syncMode: `on-demand`, - sync: { - sync: (params) => { - begin = params.begin - write = params.write - commit = params.commit - params.markReady() - return { - loadSubset: ({ where }) => { - params.begin() - for (const row of where ? rows.slice(0, 1) : rows) { - params.write({ type: `insert`, value: row }) - } - params.commit() - return true - }, - } - }, - }, - }) - const members = [1] - const live = createLiveQueryCollection((q) => - q.from({ row: source }).where(({ row }) => inArray(row.v, members)), - ) - await withOracleCleanup(async () => { - await live.preload() - const expected = () => - rows.filter((row) => members.includes(row.v)).map((row) => row.id) - const check = () => expect([...live.keys()].sort()).toEqual(expected()) - check() - - members[0] = 2 - begin() - write({ - type: `update`, - value: { id: `trigger`, v: 3, revision: 1 }, - }) - commit() - check() - }, [() => live.cleanup(), () => source.cleanup()]) - }) - - it(`a source event with no joined-row delta refreshes a changed inArray list`, async () => { - type MemberRow = { - id: string - v: number - groupId: string - revision: number - } - const rows = createCollection( - mockSyncCollectionOptions({ - id: `where-publication-join-in-rows-${collectionSerial++}`, - getKey: (row) => row.id, - autoIndex: `eager`, - initialData: [ - { id: `one`, v: 1, groupId: `g`, revision: 0 }, - { id: `two`, v: 2, groupId: `g`, revision: 0 }, - { id: `trigger`, v: 3, groupId: `missing`, revision: 0 }, - ], - }), - ) - const groups = createCollection( - mockSyncCollectionOptions<{ id: string; revision: number }>({ - id: `where-publication-join-in-groups-${collectionSerial++}`, - getKey: (row) => row.id, - initialData: [ - { id: `g`, revision: 0 }, - { id: `spare`, revision: 0 }, - ], - }), - ) - const members = [1] - const live = createLiveQueryCollection((q) => - q - .from({ row: rows }) - .innerJoin({ group: groups }, ({ row, group }) => - eq(row.groupId, group.id), - ) - .where(({ row }) => inArray(row.v, members)), - ) - const batches: Array> = [] - let unsubscribe: (() => void) | undefined - await withOracleCleanup(async () => { - await live.preload() - const subscription = live.subscribeChanges( - (changes) => - batches.push( - changes.map((change) => ({ - type: change.type, - id: change.value.row.id, - })), - ), - { includeInitialState: false }, - ) - unsubscribe = () => subscription.unsubscribe() - const expected = () => - [ - { id: `one`, v: 1 }, - { id: `two`, v: 2 }, - ] - .filter((row) => members.includes(row.v)) - .map((row) => row.id) - const check = () => - expect(live.toArray.map((result) => result.row.id).sort()).toEqual( - expected(), - ) - check() - - members[0] = 2 - rows.utils.begin() - rows.utils.write({ - type: `update`, - value: { id: `trigger`, v: 3, groupId: `missing`, revision: 1 }, - }) - rows.utils.commit() - check() - expect(batches).toHaveLength(1) - expect(batches[0]!.sort((a, b) => a.id.localeCompare(b.id))).toEqual([ - { type: `delete`, id: `one` }, - { type: `insert`, id: `two` }, - ]) - batches.length = 0 - - rows.utils.begin() - rows.utils.write({ - type: `update`, - value: { id: `trigger`, v: 3, groupId: `missing`, revision: 2 }, - }) - rows.utils.commit() - check() - expect(batches).toEqual([]) - - // A pulse must also come from the joined source. The spare group has - // no joined row, so this write cannot reach the WHERE through the - // joined relation itself. - members[0] = 1 - groups.utils.begin() - groups.utils.write({ - type: `update`, - value: { id: `spare`, revision: 1 }, - }) - groups.utils.commit() - check() - expect(batches).toHaveLength(1) - expect(batches[0]!.sort((a, b) => a.id.localeCompare(b.id))).toEqual([ - { type: `insert`, id: `one` }, - { type: `delete`, id: `two` }, - ]) - }, [ - () => unsubscribe?.(), - () => live.cleanup(), - () => rows.cleanup(), - () => groups.cleanup(), - ]) - }) - - it(`a lazy joined source must deliver unrelated events to refresh inArray membership`, async () => { - type MemberRow = { id: string; groupId: string; v: number } - type GroupRow = { id: string; revision: number } - const membersSource = createCollection( - mockSyncCollectionOptions({ - id: `where-publication-lazy-join-main-${collectionSerial++}`, - getKey: (row) => row.id, - initialData: [ - { id: `one`, groupId: `g`, v: 1 }, - { id: `two`, groupId: `g`, v: 2 }, - ], - }), - ) - const requests: Array = [] - let begin!: () => void - let write!: (message: { type: `delete`; key: string }) => void - let commit!: () => unknown - const groups = createCollection({ - id: `where-publication-lazy-join-groups-${collectionSerial++}`, - getKey: (row) => row.id, - syncMode: `on-demand`, - sync: { - sync: (params) => { - begin = params.begin - write = params.write - commit = params.commit - params.begin() - params.write({ type: `insert`, value: { id: `g`, revision: 0 } }) - params.write({ type: `insert`, value: { id: `h`, revision: 0 } }) - params.commit() - params.markReady() - return { - loadSubset: ({ where }) => { - requests.push(where !== undefined) - return true - }, - } - }, - }, - }) - const members = [1] - const live = createLiveQueryCollection((q) => - q - .from({ row: membersSource }) - .leftJoin({ group: groups }, ({ row, group }) => - eq(row.groupId, group.id), - ) - .where(({ row }) => inArray(row.v, members)), - ) - await withOracleCleanup(async () => { - await live.preload() - expect(requests[0]).toBe(false) - const expected = () => - [ - { id: `one`, v: 1 }, - { id: `two`, v: 2 }, - ] - .filter((row) => members.includes(row.v)) - .map((row) => row.id) - const check = () => - expect(live.toArray.map((result) => result.row.id)).toEqual( - expected(), - ) - check() - - members[0] = 2 - begin() - write({ type: `delete`, key: `h` }) - commit() - check() - }, [ - () => live.cleanup(), - () => groups.cleanup(), - () => membersSource.cleanup(), - ]) - }) - - it(`an outer QueryRef filter refreshes after an inner filter suppresses a source event`, async () => { - type MembershipRow = { - id: string - v: number - groupId: string - revision: number - } - const source = createCollection( - mockSyncCollectionOptions({ - id: `where-publication-queryref-in-${collectionSerial++}`, - getKey: (row) => row.id, - initialData: [ - { id: `one`, v: 1, groupId: `g`, revision: 0 }, - { id: `two`, v: 2, groupId: `g`, revision: 0 }, - { id: `trigger`, v: 3, groupId: `other`, revision: 0 }, - ], - }), - ) - const members = [1] - const live = createLiveQueryCollection((q) => { - const inner = q - .from({ item: source }) - .where(({ item }) => eq(item.groupId, `g`)) - .select(({ item }) => ({ id: item.id, v: item.v })) - return q - .from({ row: inner }) - .where(({ row }) => inArray(row.v, members)) - .select(({ row }) => ({ id: row.id })) - }) - await withOracleCleanup(async () => { - await live.preload() - const expected = () => - [ - { id: `one`, v: 1 }, - { id: `two`, v: 2 }, - ] - .filter((row) => members.includes(row.v)) - .map((row) => row.id) - const check = () => - expect(live.toArray.map((row) => row.id).sort()).toEqual(expected()) - check() - members[0] = 2 - source.utils.begin() - source.utils.write({ - type: `update`, - value: { id: `trigger`, v: 3, groupId: `other`, revision: 1 }, - }) - source.utils.commit() - check() - }, [() => live.cleanup(), () => source.cleanup()]) - }) - - it(`a correlated include reads current inArray members after an unrelated child event`, async () => { - type ParentRow = { id: string; groupId: string } - type ChildRow = { - id: string - groupId: string - v: number - revision: number - } - const parents = createCollection( - mockSyncCollectionOptions({ - id: `where-publication-include-parents-${collectionSerial++}`, - getKey: (row) => row.id, - initialData: [{ id: `parent`, groupId: `g` }], - }), - ) - const children = createCollection( - mockSyncCollectionOptions({ - id: `where-publication-include-children-${collectionSerial++}`, - getKey: (row) => row.id, - initialData: [ - { id: `one`, groupId: `g`, v: 1, revision: 0 }, - { id: `two`, groupId: `g`, v: 2, revision: 0 }, - { id: `trigger`, groupId: `other`, v: 3, revision: 0 }, - ], - }), - ) - const members = [1] - const live = createLiveQueryCollection((q) => - q.from({ parent: parents }).select(({ parent }) => ({ - id: parent.id, - children: materialize( - q - .from({ child: children }) - .where(({ child }) => - and( - eq(child.groupId, parent.groupId), - inArray(child.v, members), - ), - ) - .orderBy(({ child }) => child.id) - .select(({ child }) => ({ id: child.id })), - ), - })), - ) - await withOracleCleanup(async () => { - await live.preload() - const expected = () => - [ - { id: `one`, v: 1 }, - { id: `two`, v: 2 }, - ] - .filter((row) => members.includes(row.v)) - .map((row) => row.id) - const check = () => - expect(live.toArray[0]?.children.map((child) => child.id)).toEqual( - expected(), - ) - check() - - members[0] = 2 - children.utils.begin() - children.utils.write({ - type: `update`, - value: { id: `trigger`, groupId: `other`, v: 3, revision: 1 }, - }) - children.utils.commit() - check() - }, [ - () => live.cleanup(), - () => children.cleanup(), - () => parents.cleanup(), - ]) - }) - - it(`an on-demand include acquires unfiltered children for mutable inArray membership`, async () => { - type ParentRow = { id: string; groupId: string } - type ChildRow = { id: string; groupId: string; v: number } - const parents = createCollection( - mockSyncCollectionOptions({ - id: `where-publication-demand-include-parents-${collectionSerial++}`, - getKey: (row) => row.id, - initialData: [{ id: `parent`, groupId: `g` }], - }), - ) - const requests: Array = [] - let begin!: () => void - let write!: (message: { type: `delete`; key: string }) => void - let commit!: () => unknown - const children = createCollection({ - id: `where-publication-demand-include-children-${collectionSerial++}`, - getKey: (row) => row.id, - syncMode: `on-demand`, - sync: { - sync: (params) => { - begin = params.begin - write = params.write - commit = params.commit - params.begin() - params.write({ - type: `insert`, - value: { id: `one`, groupId: `g`, v: 1 }, - }) - params.write({ - type: `insert`, - value: { id: `two`, groupId: `g`, v: 2 }, - }) - params.write({ - type: `insert`, - value: { id: `spare`, groupId: `h`, v: 3 }, - }) - params.commit() - params.markReady() - return { - loadSubset: ({ where }) => { - requests.push(where !== undefined) - return true - }, - } - }, - }, - }) - const members = [1] - const live = createLiveQueryCollection((q) => - q.from({ parent: parents }).select(({ parent }) => ({ - id: parent.id, - children: materialize( - q - .from({ child: children }) - .where(({ child }) => - and( - eq(child.groupId, parent.groupId), - inArray(child.v, members), - ), - ) - .orderBy(({ child }) => child.id) - .select(({ child }) => ({ id: child.id })), - ), - })), - ) - await withOracleCleanup(async () => { - await live.preload() - expect(requests[0]).toBe(false) - const expected = () => - [ - { id: `one`, v: 1 }, - { id: `two`, v: 2 }, - ] - .filter((row) => members.includes(row.v)) - .map((row) => row.id) - const check = () => - expect(live.toArray[0]?.children.map((child) => child.id)).toEqual( - expected(), - ) - check() - - members[0] = 2 - begin() - write({ type: `delete`, key: `spare` }) - commit() - check() - }, [ - () => live.cleanup(), - () => children.cleanup(), - () => parents.cleanup(), - ]) - }) - - it(`checks only changed rows until captured inArray members change`, async () => { - type MembershipRow = { id: string; v: number; revision: number } - let listReads = 0 - const row = (v: number, revision: number): MembershipRow => ({ - id: String(v), - v, - revision, - }) - const source = createCollection( - mockSyncCollectionOptions({ - id: `where-publication-in-work-${collectionSerial++}`, - getKey: (item) => item.id, - autoIndex: `off`, - initialData: Array.from({ length: 40 }, (_, v) => row(v, 0)), - }), - ) - const members = new Proxy([1], { - get(target, slotKey, receiver) { - if (slotKey === `0`) listReads++ - return Reflect.get(target, slotKey, receiver) - }, - }) - const live = createLiveQueryCollection((q) => - q.from({ item: source }).where(({ item }) => inArray(item.v, members)), - ) - await withOracleCleanup(async () => { - await live.preload() - expect([...live.keys()]).toEqual([`1`]) - - listReads = 0 - source.utils.begin() - source.utils.write({ type: `update`, value: row(39, 1) }) - source.utils.commit() - expect([...live.keys()]).toEqual([`1`]) - expect(listReads).toBeLessThan(10) - - members[0] = 2 - listReads = 0 - source.utils.begin() - source.utils.write({ type: `update`, value: row(39, 2) }) - source.utils.commit() - expect([...live.keys()]).toEqual([`2`]) - expect(listReads).toBeGreaterThanOrEqual(40) - }, [() => live.cleanup(), () => source.cleanup()]) - }) - it(`a pending optimistic delete hides a row from a prefiltered unindexed scan`, async () => { // The scan may read synced rows directly only when no optimistic state // changes visibility. A pending delete removes a synced row. @@ -2430,22 +1103,6 @@ describe(`WHERE predicate publication oracle`, () => { }, [() => subscription.unsubscribe(), () => collection.cleanup()]) }) - it(`ordinary native Dates publish by timestamp across realms and invalid-Date equality`, async () => { - const foreign = runtimeTaggedDateValue({ - kind: `date`, - realm: `foreign`, - timestamp: 1, - }) - // This checks the host premise: a native Date can fail local instanceof. - expect(foreign instanceof Date).toBe(false) - expect(Date.prototype.getTime.call(foreign)).toBe(1) - for (const indexed of [false, true]) { - for (const literal of taggedDateLiterals) { - await observeTaggedDateEquality(literal, indexed) - } - } - }) - for (const history of [ ...pinnedHistories, pinnedPrefilteredSnapshot, diff --git a/packages/db/tests/temporal-equality-alias-oracle.test.ts b/packages/db/tests/temporal-equality-alias-oracle.test.ts deleted file mode 100644 index da82b9244e..0000000000 --- a/packages/db/tests/temporal-equality-alias-oracle.test.ts +++ /dev/null @@ -1,671 +0,0 @@ -import { describe, expect, it, vi } from 'vitest' -import { Temporal } from 'temporal-polyfill' -import { createCollection } from '../src/collection/index.js' -import { BasicIndex } from '../src/indexes/basic-index.js' -import { BTreeIndex } from '../src/indexes/btree-index.js' -import { Func, PropRef, Value } from '../src/query/ir.js' -import { withOracleCleanup } from './utils.js' - -/** - * A ZonedDateTime's exact query equality follows Temporal's `.equals()`. - * Changelog #1519 promises that relation for `eq`, distinct from instant - * order. At one instant, UTC and Etc/UTC are equivalent zone aliases, while - * the fixed-offset +00:00 zone remains a different identity. - * - * The ZonedDateTime lane covers values from one polyfill copy and one instant, - * including a normal subclass with inherited Temporal methods. - * Fixture-assigned classes, - * independent of production normalization, predict exact key sets. Direct - * BasicIndex/BTreeIndex add, build, and removal histories compare those sets - * after each returned action. Public Collection - * reads compare an unindexed scan and an optimized-only indexed snapshot after - * readiness. - * A second lane uses one polyfill copy to check `PlainMonthDay` and - * `PlainYearMonth` reference ISO fields, and Duration's ten fields, under - * the same exact query equality. Changelog #1519 explicitly keeps equivalent - * Duration representations unequal for `eq`. Its fixture assigns equality - * classes from native constructors and intrinsic relations, never from - * production key encoding. Compiled `eq` and `in` for these classes are - * checked by the in-evaluator oracle. PlainMonthDay has no natural order, so - * this file checks its public Collection equality through an unindexed scan. - * Indexed PlainMonthDay equality remains open. YearMonth and Duration store - * several classes with identical default display and check indexed add, - * build, removal, re-add, and public reads. Duration also varies years, - * months, weeks, days, and nanoseconds one field at a time; a key that omits - * any one of those fields would merge its distinct fixture classes. - * This oracle does not claim all IANA aliases, subclasses with replaced - * Temporal methods, malformed Temporal lookalikes or deliberately shadowed - * Temporal members, - * mixed Temporal implementations, joins, live-query publication, or provider - * behavior. - */ -type Row = { id: string; when: Temporal.ZonedDateTime } -type Sample = { - id: string - text: string - identity: `utc` | `offset` | `later` | `calendar` - subclass?: boolean -} - -class ZonedDateTimeSubclass extends Temporal.ZonedDateTime {} - -const samples: ReadonlyArray = [ - { - id: `utc`, - text: `2024-01-15T09:00:00+00:00[UTC]`, - identity: `utc`, - }, - { - id: `utc-alias`, - text: `2024-01-15T09:00:00+00:00[Etc/UTC]`, - identity: `utc`, - }, - { - id: `fixed-offset`, - text: `2024-01-15T09:00:00+00:00[+00:00]`, - identity: `offset`, - }, - { - id: `one-nanosecond-later`, - text: `2024-01-15T09:00:00.000000001+00:00[UTC]`, - identity: `later`, - }, - { - id: `gregory-calendar`, - text: `2024-01-15T09:00:00+00:00[UTC][u-ca=gregory]`, - identity: `calendar`, - }, - { - id: `utc-subclass`, - text: `2024-01-15T09:00:00+00:00[UTC]`, - identity: `utc`, - subclass: true, - }, -] - -function rows(): Array { - return samples.map(({ id, text, subclass }) => { - const base = Temporal.ZonedDateTime.from(text) - const when = subclass - ? new ZonedDateTimeSubclass( - base.epochNanoseconds, - base.timeZoneId, - base.calendarId, - ) - : base - return { id, when } - }) -} - -function expectedKeys(identity: Sample[`identity`]): Set { - return new Set( - samples - .filter((sample) => sample.identity === identity) - .map(({ id }) => id), - ) -} - -const aliasProbe = Temporal.ZonedDateTime.from( - `2024-01-15T09:00:00+00:00[Etc/UTC]`, -) -const subclassProbe = new ZonedDateTimeSubclass( - aliasProbe.epochNanoseconds, - `UTC`, -) -const offsetProbe = Temporal.ZonedDateTime.from( - `2024-01-15T09:00:00+00:00[+00:00]`, -) - -const indexTypes = [ - [`BasicIndex`, BasicIndex], - [`BTreeIndex`, BTreeIndex], -] as const - -type IntrinsicKind = `PlainMonthDay` | `PlainYearMonth` | `Duration` -type IntrinsicRow = { id: string; when: unknown; identity: string } -type IntrinsicProbe = { when: unknown; identity: string } -type IntrinsicFixture = { - rows: Array - probes: Array - checkNativeRelation: () => void -} - -const indexedIntrinsicKinds: ReadonlyArray< - Exclude -> = [`PlainYearMonth`, `Duration`] - -// The fixture identity is a model-only equality class assigned from the -// constructed Temporal fields. It combines no production state. Fresh probes -// distinguish equal copies from different fields with the same display text. -function intrinsicFixture(kind: IntrinsicKind): IntrinsicFixture { - if (kind === `PlainMonthDay`) { - const stored = new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972) - const storedCopy = new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972) - const equal = new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972) - const other = new Temporal.PlainMonthDay(2, 29, `iso8601`, 2004) - return { - rows: [ - { id: `reference-1972`, when: stored, identity: `1972` }, - { id: `reference-1972-copy`, when: storedCopy, identity: `1972` }, - { id: `reference-2004`, when: other, identity: `2004` }, - ], - probes: [ - { when: equal, identity: `1972` }, - { when: other, identity: `2004` }, - ], - checkNativeRelation: () => { - expect(stored.toString()).toBe(other.toString()) - expect(stored.equals(storedCopy)).toBe(true) - expect(stored.equals(equal)).toBe(true) - expect(stored.equals(other)).toBe(false) - }, - } - } - - if (kind === `PlainYearMonth`) { - const stored = new Temporal.PlainYearMonth(2024, 2, `iso8601`, 1) - const equal = new Temporal.PlainYearMonth(2024, 2, `iso8601`, 1) - const other = new Temporal.PlainYearMonth(2024, 2, `iso8601`, 20) - return { - rows: [ - { id: `reference-day-1`, when: stored, identity: `day-1` }, - { id: `reference-day-1-copy`, when: equal, identity: `day-1` }, - { id: `reference-day-20`, when: other, identity: `day-20` }, - ], - probes: [ - { - when: new Temporal.PlainYearMonth(2024, 2, `iso8601`, 1), - identity: `day-1`, - }, - { - when: new Temporal.PlainYearMonth(2024, 2, `iso8601`, 20), - identity: `day-20`, - }, - ], - checkNativeRelation: () => { - expect(stored.toString()).toBe(other.toString()) - expect(stored.equals(equal)).toBe(true) - expect(stored.equals(other)).toBe(false) - }, - } - } - - const seconds = Temporal.Duration.from({ seconds: 1 }) - const secondsCopy = Temporal.Duration.from({ seconds: 1 }) - const milliseconds = Temporal.Duration.from({ milliseconds: 1000 }) - const microseconds = Temporal.Duration.from({ microseconds: 1_000_000 }) - return { - rows: [ - { id: `seconds`, when: seconds, identity: `seconds` }, - { id: `seconds-copy`, when: secondsCopy, identity: `seconds` }, - { id: `milliseconds`, when: milliseconds, identity: `milliseconds` }, - { id: `microseconds`, when: microseconds, identity: `microseconds` }, - ], - probes: [ - { when: Temporal.Duration.from({ seconds: 1 }), identity: `seconds` }, - { - when: Temporal.Duration.from({ milliseconds: 1000 }), - identity: `milliseconds`, - }, - { - when: Temporal.Duration.from({ microseconds: 1_000_000 }), - identity: `microseconds`, - }, - ], - checkNativeRelation: () => { - expect(seconds.toString()).toBe(milliseconds.toString()) - expect(milliseconds.toString()).toBe(microseconds.toString()) - expect(seconds.seconds).toBe(1) - expect(milliseconds.milliseconds).toBe(1000) - expect(microseconds.microseconds).toBe(1_000_000) - }, - } -} - -function variedDurationFixture(): IntrinsicFixture { - const fields = { - years: 1, - months: 1, - weeks: 1, - days: 1, - nanoseconds: 1, - } - const baseline = Temporal.Duration.from(fields) - const variedFields = [ - `years`, - `months`, - `weeks`, - `days`, - `nanoseconds`, - ] as const - const variants = variedFields.map((field) => ({ - field, - when: Temporal.Duration.from({ ...fields, [field]: 2 }), - })) - const allFields = [ - `years`, - `months`, - `weeks`, - `days`, - `hours`, - `minutes`, - `seconds`, - `milliseconds`, - `microseconds`, - `nanoseconds`, - ] as const - const getters = allFields.map( - (field) => - Object.getOwnPropertyDescriptor(Temporal.Duration.prototype, field)!.get!, - ) - return { - rows: [ - { id: `mixed`, when: baseline, identity: `mixed` }, - { - id: `mixed-copy`, - when: Temporal.Duration.from(fields), - identity: `mixed`, - }, - ...variants.map(({ field, when }) => ({ - id: `mixed-${field}`, - when, - identity: `mixed-${field}`, - })), - ], - probes: [ - { when: Temporal.Duration.from(fields), identity: `mixed` }, - ...variants.map(({ field }) => ({ - when: Temporal.Duration.from({ ...fields, [field]: 2 }), - identity: `mixed-${field}`, - })), - ], - checkNativeRelation: () => { - // Each class differs from mixed in exactly one intrinsic field. Dropping - // that field from an exact key would merge the two classes. - for (const { field, when } of variants) { - expect( - allFields.filter( - (_, index) => - Reflect.apply(getters[index]!, baseline, []) !== - Reflect.apply(getters[index]!, when, []), - ), - ).toEqual([field]) - } - }, - } -} - -function intrinsicExpectedKeys( - source: ReadonlyArray, - identity: string, -): Set { - return new Set( - source.filter((row) => row.identity === identity).map((row) => row.id), - ) -} - -it(`distinguishes equivalent aliases from a fixed-offset zone`, () => { - const [utc, alias, fixed, later, gregory, subclass] = rows() - // This checks the fixture against the external Temporal relation. Expected - // index and query sets still come from the assigned identity classes. - expect(utc!.when.equals(alias!.when)).toBe(true) - expect(utc!.when.equals(fixed!.when)).toBe(false) - expect(utc!.when.equals(later!.when)).toBe(false) - expect(utc!.when.equals(gregory!.when)).toBe(false) - // A subclass inherits Temporal's slot-checking method. The fixture assigns - // it the UTC class independently of the production classifier. - expect(Object.getPrototypeOf(subclass!.when)).toBe( - ZonedDateTimeSubclass.prototype, - ) - expect( - Temporal.ZonedDateTime.prototype.equals.call(subclass!.when, alias!.when), - ).toBe(true) - expect( - Temporal.ZonedDateTime.prototype.equals.call(subclass!.when, fixed!.when), - ).toBe(false) - expect( - Temporal.ZonedDateTime.prototype.equals.call(subclassProbe, alias!.when), - ).toBe(true) - expect(utc!.when.toString()).not.toBe(alias!.when.toString()) - expect(expectedKeys(`utc`)).toEqual( - new Set([`utc`, `utc-alias`, `utc-subclass`]), - ) - expect(expectedKeys(`offset`)).toEqual(new Set([`fixed-offset`])) -}) - -describe.each(indexTypes)(`%s Temporal alias equality`, (_name, IndexType) => { - it.each([`add`, `build`] as const)( - `finds the exact equality class after %s and representative removals`, - (construction) => { - const source = rows() - const index = new IndexType(1, new PropRef([`when`])) - if (construction === `add`) { - const seenAliasKeys = new Set() - for (const [position, row] of source.entries()) { - index.add(row.id, row) - if (samples[position]!.identity === `utc`) seenAliasKeys.add(row.id) - expect(index.equalityLookup(aliasProbe)).toEqual(seenAliasKeys) - } - } else { - index.build(source.map((row): [string, Row] => [row.id, row])) - } - - // The query value is a fresh instance. A display-string key would omit - // UTC; an instant-only key would also include fixed-offset. - for (const probe of [aliasProbe, subclassProbe]) { - expect(index.equalityLookup(probe)).toEqual(expectedKeys(`utc`)) - expect(index.lookup(`eq`, probe)).toEqual(expectedKeys(`utc`)) - } - expect(index.inArrayLookup([aliasProbe])).toEqual(expectedKeys(`utc`)) - expect(index.equalityLookup(offsetProbe)).toEqual(expectedKeys(`offset`)) - - // The neighboring instant and calendar classes ensure that a repair - // does not collapse every Temporal value sharing a named zone. - expect(index.equalityLookup(source[3]!.when)).toEqual( - expectedKeys(`later`), - ) - expect(index.equalityLookup(source[4]!.when)).toEqual( - expectedKeys(`calendar`), - ) - - // Removing the subclass itself must retract one member of the class. - // Each native-equal probe still finds every remaining UTC alias. - const remainingUtc = expectedKeys(`utc`) - for (const [id, probe] of [ - [`utc`, aliasProbe], - [`utc-subclass`, subclassProbe], - ] as const) { - const row = source.find((candidate) => candidate.id === id)! - index.remove(row.id, row) - remainingUtc.delete(id) - expect(index.equalityLookup(probe)).toEqual(remainingUtc) - } - expect(index.equalityLookup(offsetProbe)).toEqual(expectedKeys(`offset`)) - }, - ) - - it.each([false, true])( - `returns alias-equal rows in a public snapshot with optimizedOnly=%s`, - async (optimizedOnly) => { - const source = rows() - const collection = createCollection({ - id: `temporal-alias-equality-${_name}`, - getKey: (row) => row.id, - autoIndex: `off`, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of source) write({ type: `insert`, value: row }) - commit() - markReady() - }, - }, - }) - const readKeys = (probe: Temporal.ZonedDateTime) => { - const predicate = new Func(`eq`, [ - new PropRef([`when`]), - new Value(probe), - ]) - return new Set( - collection - .currentStateAsChanges({ where: predicate, optimizedOnly }) - ?.map(({ value }) => value.id), - ) - } - - await withOracleCleanup(async () => { - await collection.stateWhenReady() - expect(collection.indexes.size).toBe(0) - const lookup = optimizedOnly - ? vi.spyOn( - collection.createIndex((row) => row.when, { - indexType: IndexType, - }), - `lookup`, - ) - : undefined - for (const probe of [aliasProbe, subclassProbe]) { - lookup?.mockClear() - const result = readKeys(probe) - if (lookup) expect(lookup).toHaveBeenCalled() - expect(result).toEqual(expectedKeys(`utc`)) - } - }, [() => collection.cleanup()]) - }, - ) -}) - -it(`keeps PlainMonthDay reference years distinct in an unindexed public equality scan`, async () => { - const fixture = intrinsicFixture(`PlainMonthDay`) - fixture.checkNativeRelation() - const collection = createCollection({ - id: `temporal-monthday-unindexed-equality`, - getKey: (row) => row.id, - autoIndex: `off`, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of fixture.rows) write({ type: `insert`, value: row }) - commit() - markReady() - }, - }, - }) - - // The source contains two native-equal copies and a distinct reference - // year with the same default display. The public scan has no index order - // obligation; each query must return exactly its modeled equality class. - await withOracleCleanup(async () => { - await collection.stateWhenReady() - expect(collection.indexes.size).toBe(0) - for (const probe of fixture.probes) { - const predicate = new Func(`eq`, [ - new PropRef([`when`]), - new Value(probe.when), - ]) - const keys = new Set( - collection - .currentStateAsChanges({ where: predicate, optimizedOnly: false }) - ?.map(({ value }) => value.id), - ) - expect(collection.indexes.size).toBe(0) - expect(keys).toEqual(intrinsicExpectedKeys(fixture.rows, probe.identity)) - } - }, [() => collection.cleanup()]) -}) - -it(`keeps five additional Duration fields distinct in an unindexed public equality scan`, async () => { - const fixture = variedDurationFixture() - fixture.checkNativeRelation() - const collection = createCollection({ - id: `temporal-duration-varied-unindexed-equality`, - getKey: (row) => row.id, - autoIndex: `off`, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of fixture.rows) write({ type: `insert`, value: row }) - commit() - markReady() - }, - }, - }) - - // The scan reads exact classes without asking Temporal to order calendar - // units, which would require relativeTo. - await withOracleCleanup(async () => { - await collection.stateWhenReady() - expect(collection.indexes.size).toBe(0) - for (const probe of fixture.probes) { - const predicate = new Func(`eq`, [ - new PropRef([`when`]), - new Value(probe.when), - ]) - const keys = new Set( - collection - .currentStateAsChanges({ where: predicate, optimizedOnly: false }) - ?.map(({ value }) => value.id), - ) - expect(keys).toEqual(intrinsicExpectedKeys(fixture.rows, probe.identity)) - } - }, [() => collection.cleanup()]) -}) - -it.each(indexTypes)( - `%s keeps five additional Duration fields in exact index equality`, - (_name, IndexType) => { - const fixture = variedDurationFixture() - fixture.checkNativeRelation() - // A lexical display comparator supplies only the index's required total - // order. The independently assigned classes judge exact equality. - const compareText = (left: unknown, right: unknown) => { - const a = String(left) - const b = String(right) - return a < b ? -1 : a > b ? 1 : 0 - } - const index = new IndexType(1, new PropRef([`when`]), undefined, { - compareFn: compareText, - }) - index.build( - fixture.rows.map((row): [string, IntrinsicRow] => [row.id, row]), - ) - for (const probe of fixture.probes) { - const expected = intrinsicExpectedKeys(fixture.rows, probe.identity) - expect(index.equalityLookup(probe.when)).toEqual(expected) - expect(index.lookup(`eq`, probe.when)).toEqual(expected) - expect(index.inArrayLookup([probe.when])).toEqual(expected) - } - }, -) - -describe.each(indexTypes)( - `%s Temporal intrinsic field equality`, - (indexName, IndexType) => { - it.each(indexedIntrinsicKinds)( - `keeps %s exact classes after each add, removal, and re-add`, - (kind) => { - const fixture = intrinsicFixture(kind) - fixture.checkNativeRelation() - const index = new IndexType(1, new PropRef([`when`])) - const added: Array = [] - - // Each returned add is a checkpoint. The model recomputes membership - // from fixture classes, independent of the index's equality key. - for (const row of fixture.rows) { - index.add(row.id, row) - added.push(row) - for (const probe of fixture.probes) { - expect(index.equalityLookup(probe.when)).toEqual( - intrinsicExpectedKeys(added, probe.identity), - ) - } - } - - for (const probe of fixture.probes) { - const expected = intrinsicExpectedKeys(added, probe.identity) - expect(index.lookup(`eq`, probe.when)).toEqual(expected) - expect(index.inArrayLookup([probe.when])).toEqual(expected) - } - - const removed = added.shift()! - index.remove(removed.id, removed) - for (const probe of fixture.probes) { - expect(index.equalityLookup(probe.when)).toEqual( - intrinsicExpectedKeys(added, probe.identity), - ) - } - - const freshEqual = fixture.probes.find( - (probe) => probe.identity === removed.identity, - )! - const readded = { ...removed, when: freshEqual.when } - index.add(readded.id, readded) - added.push(readded) - for (const probe of fixture.probes) { - expect(index.equalityLookup(probe.when)).toEqual( - intrinsicExpectedKeys(added, probe.identity), - ) - } - }, - ) - - it.each(indexedIntrinsicKinds)( - `keeps %s exact classes after build`, - (kind) => { - const fixture = intrinsicFixture(kind) - fixture.checkNativeRelation() - const index = new IndexType(1, new PropRef([`when`])) - index.build( - fixture.rows.map((row): [string, IntrinsicRow] => [row.id, row]), - ) - for (const probe of fixture.probes) { - expect(index.equalityLookup(probe.when)).toEqual( - intrinsicExpectedKeys(fixture.rows, probe.identity), - ) - } - }, - ) - - it.each( - indexedIntrinsicKinds.flatMap((kind) => [ - { kind, optimizedOnly: false }, - { kind, optimizedOnly: true }, - ]), - )( - `returns $kind exact classes in a public snapshot with optimizedOnly=$optimizedOnly`, - async ({ kind, optimizedOnly }) => { - const fixture = intrinsicFixture(kind) - fixture.checkNativeRelation() - const collection = createCollection({ - id: `temporal-intrinsic-${kind}-${indexName}-${optimizedOnly}`, - getKey: (row) => row.id, - autoIndex: `off`, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of fixture.rows) - write({ type: `insert`, value: row }) - commit() - markReady() - }, - }, - }) - - // A scan supplies the public control. With optimizedOnly, a spy proves - // that the same `eq` predicate reaches the explicit index lookup. - await withOracleCleanup(async () => { - await collection.stateWhenReady() - expect(collection.indexes.size).toBe(0) - const lookup = optimizedOnly - ? vi.spyOn( - collection.createIndex((row) => row.when, { - indexType: IndexType, - }), - `lookup`, - ) - : undefined - for (const probe of fixture.probes) { - lookup?.mockClear() - const predicate = new Func(`eq`, [ - new PropRef([`when`]), - new Value(probe.when), - ]) - const keys = new Set( - collection - .currentStateAsChanges({ where: predicate, optimizedOnly }) - ?.map(({ value }) => value.id), - ) - if (lookup) expect(lookup).toHaveBeenCalled() - expect(keys).toEqual( - intrinsicExpectedKeys(fixture.rows, probe.identity), - ) - } - }, [() => collection.cleanup()]) - }, - ) - }, -) diff --git a/packages/db/tests/temporal-index-order-oracle.test.ts b/packages/db/tests/temporal-index-order-oracle.test.ts deleted file mode 100644 index 77094500f4..0000000000 --- a/packages/db/tests/temporal-index-order-oracle.test.ts +++ /dev/null @@ -1,798 +0,0 @@ -import { describe, expect, it, vi } from 'vitest' -import { Temporal } from 'temporal-polyfill' -import { createCollection } from '../src/collection/index.js' -import { BasicIndex } from '../src/indexes/basic-index.js' -import { BTreeIndex } from '../src/indexes/btree-index.js' -import { PropRef } from '../src/query/ir.js' -import { orderedEntriesArray, withOracleCleanup } from './utils.js' - -/** - * A ZonedDateTime's ordered position is its instant. Two zones at the same - * instant share an ordered position but remain distinct equality values. - * Authority: Temporal.ZonedDateTime.compare/equals, the established WHERE - * evaluator contract in query/compiler/evaluators-oracle.test.ts, the - * ordered Collection contract, and the established direct key-tie rule in - * deterministic-ordering.test.ts. IndexInterface names the direct methods. - * - * This bounded oracle covers one polyfill copy, two opposing offset zones, - * one equal-instant pair, a one-nanosecond successor, and negative instants - * around the epoch. Direct indexes and public Collection snapshots order - * equal-instant peers by source key, including at a limit boundary. It - * checks direct BasicIndex/BTreeIndex adds, builds, and value cursors, one - * removal/insertion/update continuation, then the public ordered Collection - * snapshot with limit 1. Direct custom-comparator witnesses preserve the - * former display-string operands and let the callback order two rows whose - * values share one query-equality class. An opposite callback distinguishes - * callback-result order from plain display-text order. They check add/build, - * exact lookup, range and cursor boundaries, representative removal, and - * updates that change callback order without changing query equality. These - * checks do not claim arbitrary - * Temporal kinds, time-zone transitions, longer histories, - * public/provider cursor continuation, or live-query publication. - * The separate admission lane below protects default equality indexes for - * genuine Temporal values that lack a public order. It does not assign those - * values a public `orderBy` result: changelog #1519 requires `orderBy` to - * reject PlainMonthDay and mixed Temporal kinds. Its admission and exact - * membership authority is the public createIndex contract and #1519's - * unchanged Temporal `eq` relation; #1370 established Temporal index keys. - */ - -type Row = { id: string; when: Temporal.ZonedDateTime } -type Fixture = Readonly<{ - id: string - text: string - epochNanoseconds: bigint - equalityClass: string -}> - -// The supplied epoch nanoseconds are the model's order labels. Neither the -// index's normalization nor its comparator computes an expected position. -// The first and second rows have the same instant in different zones; the -// third is only one nanosecond later. The fourth has later instant but earlier -// local clock text than the first, so textual index order is observably wrong. -const fixtures: ReadonlyArray = [ - { - id: `b-plus-fourteen`, - text: `2024-01-15T23:00:00+14:00[+14:00]`, - epochNanoseconds: 1_705_309_200_000_000_000n, - equalityClass: `plus-fourteen`, - }, - { - id: `c-utc`, - text: `2024-01-15T09:00:00+00:00[+00:00]`, - epochNanoseconds: 1_705_309_200_000_000_000n, - equalityClass: `utc`, - }, - { - id: `a-one-nanosecond-later`, - text: `2024-01-15T09:00:00.000000001+00:00[+00:00]`, - epochNanoseconds: 1_705_309_200_000_000_001n, - equalityClass: `one-nanosecond-later`, - }, - { - id: `d-minus-five`, - text: `2024-01-15T05:00:00-05:00[-05:00]`, - epochNanoseconds: 1_705_312_800_000_000_000n, - equalityClass: `minus-five`, - }, -] - -// Textual sorting of signed decimal epoch values puts -10 before -100. These -// pre-epoch neighbors reject that tempting numeric-string repair, even though -// their ZonedDateTime display strings happen to be in chronological order. -const negativeFixtures: ReadonlyArray = [ - { - id: `a-minus-ten`, - text: `1969-12-31T23:59:59.999999990+00:00[+00:00]`, - epochNanoseconds: -10n, - equalityClass: `minus-ten`, - }, - { - id: `b-minus-hundred`, - text: `1969-12-31T23:59:59.999999900+00:00[+00:00]`, - epochNanoseconds: -100n, - equalityClass: `minus-hundred`, - }, - { - id: `c-zero`, - text: `1970-01-01T00:00:00+00:00[+00:00]`, - epochNanoseconds: 0n, - equalityClass: `zero`, - }, -] - -function rowsFromFixtures( - samples: ReadonlyArray = fixtures, -): Array { - return samples.map(({ id, text, epochNanoseconds }) => { - const when = Temporal.ZonedDateTime.from(text) - // This checks fixture construction, not the production comparison rule. - expect(when.epochNanoseconds).toBe(epochNanoseconds) - return { id, when } - }) -} - -// The model sorts supplied instant labels, then source keys at a tied instant. -// It has no access to index normalization or production comparison. An exact -// prefix comparison detects omissions, duplicates, order, and a wrong limit tie. -function expectOrderedPrefix( - actual: Array | undefined, - count: number, - samples: ReadonlyArray = fixtures, -): void { - const expected = [...samples] - .sort((left, right) => - left.epochNanoseconds < right.epochNanoseconds - ? -1 - : left.epochNanoseconds > right.epochNanoseconds - ? 1 - : left.id < right.id - ? -1 - : left.id > right.id - ? 1 - : 0, - ) - .slice(0, count) - .map(({ id }) => id) - expect(actual).toEqual(expected) -} - -const indexTypes = [ - [`BasicIndex`, BasicIndex], - [`BTreeIndex`, BTreeIndex], -] as const - -it(`rejects a later instant and the wrong equal-instant boundary key`, () => { - // A lexical index chose d-minus-five at the first-row checkpoint. A tie - // resolved by admission order can choose c-utc. Both violate the model. - expect(() => expectOrderedPrefix([`b-plus-fourteen`], 1)).not.toThrow() - expect(() => expectOrderedPrefix([`c-utc`], 1)).toThrow() - expect(() => expectOrderedPrefix([`d-minus-five`], 1)).toThrow() -}) - -describe.each(indexTypes)( - `%s ZonedDateTime indexed order`, - (_name, IndexType) => { - it.each([`add`, `build`] as const)( - `refines instant order and zone-sensitive equality after %s`, - (construction) => { - const rows = rowsFromFixtures() - const index = new IndexType(1, new PropRef([`when`])) - - // The legal direct history either adds each unique source key or builds - // once from that same source snapshot. No source key changes afterward. - if (construction === `add`) { - for (const row of rows) index.add(row.id, row) - } else { - index.build(rows.map((row): [string, Row] => [row.id, row])) - } - - // Exact lookup and ordered position are distinct observations. An - // epoch-only key would incorrectly merge the two equal-instant zones. - for (const row of rows) { - const fixture = fixtures.find(({ id }) => id === row.id)! - const expected = new Set( - fixtures - .filter( - ({ equalityClass }) => equalityClass === fixture.equalityClass, - ) - .map(({ id }) => id), - ) - const equalButDistinct = Temporal.ZonedDateTime.from( - row.when.toString(), - ) - expect(equalButDistinct).not.toBe(row.when) - expect(index.equalityLookup(equalButDistinct)).toEqual(expected) - } - expect(rows[0]!.when.epochNanoseconds).toBe( - rows[1]!.when.epochNanoseconds, - ) - expect(index.equalityLookup(rows[1]!.when).has(rows[0]!.id)).toBe(false) - - // Returned key arrays preserve instant order and multiplicity at the - // direct IndexInterface checkpoint. Limit 1 detects a later instant - // or the wrong source-key tie inside the equal-instant first group. - expectOrderedPrefix(index.takeFromStart(rows.length), rows.length) - expectOrderedPrefix(index.takeFromStart(1), 1) - // A cursor at the shared instant must skip both zones, while retaining - // the one-nanosecond successor. An ordered key with a zone suffix would - // incorrectly leave one equal-instant row beyond this cursor. - expect(index.take(2, rows[0]!.when)).toEqual([ - `a-one-nanosecond-later`, - `d-minus-five`, - ]) - }, - ) - - it.each([`add`, `build`] as const)( - `orders equal-instant source keys after reverse %s admission`, - (construction) => { - // The instant model ties these two rows; the direct IndexInterface - // contract orders keys inside a comparator group. Reverse admission - // distinguishes that law from insertion-order traversal. - const [plusFourteen, utc] = rowsFromFixtures() - const reversed = [utc!, plusFourteen!] - const index = new IndexType(1, new PropRef([`when`])) - if (construction === `add`) { - for (const row of reversed) index.add(row.id, row) - } else { - index.build(reversed.map((row): [string, Row] => [row.id, row])) - } - expect(index.takeFromStart(2)).toEqual([`b-plus-fourteen`, `c-utc`]) - }, - ) - - it(`retains an equal-instant peer through removal, insertion, and zone update`, () => { - const [plusFourteen, utc, nanosecondLater] = rowsFromFixtures() - const index = new IndexType(1, new PropRef([`when`])) - for (const row of [plusFourteen!, utc!, nanosecondLater!]) { - index.add(row.id, row) - } - - // A source removal changes membership but leaves the peer at the same - // instant. The peer becomes the ordered representative of that position. - index.remove(plusFourteen!.id, plusFourteen) - expect(index.takeFromStart(3)).toEqual([ - `c-utc`, - `a-one-nanosecond-later`, - ]) - expect(index.equalityLookup(plusFourteen!.when)).toEqual(new Set()) - expect(index.equalityLookup(utc!.when)).toEqual(new Set([`c-utc`])) - - // The ordered group must keep a live representative after removal. - // The live-query architecture requires this for stable traversal. - expect(orderedEntriesArray(index)[0]?.[0]).toBe(utc!.when) - - const inserted: Fixture = { - id: `one-nanosecond-earlier`, - text: `2024-01-15T08:59:59.999999999+00:00[+00:00]`, - epochNanoseconds: fixtures[1]!.epochNanoseconds - 1n, - equalityClass: `one-nanosecond-earlier`, - } - const [earlier] = rowsFromFixtures([inserted]) - index.add(earlier!.id, earlier) - expectOrderedPrefix(index.takeFromStart(3), 3, [ - inserted, - fixtures[1]!, - fixtures[2]!, - ]) - expect(index.take(2, earlier!.when)).toEqual([ - `c-utc`, - `a-one-nanosecond-later`, - ]) - - // Updating that peer to another zone at the same supplied instant moves - // its exact lookup bucket without changing its ordered position. - const moved = { - id: utc!.id, - when: Temporal.ZonedDateTime.from(`2024-01-15T04:00:00-05:00[-05:00]`), - } - expect(moved.when.epochNanoseconds).toBe(fixtures[1]!.epochNanoseconds) - index.update(utc!.id, utc, moved) - expect(index.equalityLookup(utc!.when)).toEqual(new Set()) - expect(index.equalityLookup(moved.when)).toEqual(new Set([`c-utc`])) - expectOrderedPrefix(index.takeFromStart(3), 3, [ - inserted, - fixtures[1]!, - fixtures[2]!, - ]) - expect(index.take(2, utc!.when)).toEqual([`a-one-nanosecond-later`]) - }) - - it.each([`add`, `build`] as const)( - `orders negative nanosecond instants after %s`, - (construction) => { - const rows = rowsFromFixtures(negativeFixtures) - const index = new IndexType(1, new PropRef([`when`])) - if (construction === `add`) { - for (const row of rows) index.add(row.id, row) - } else { - index.build(rows.map((row): [string, Row] => [row.id, row])) - } - // The model sorts supplied signed BigInts, not their decimal strings. - expectOrderedPrefix( - index.takeFromStart(rows.length), - rows.length, - negativeFixtures, - ) - }, - ) - - it(`retains a string-compatible custom comparator across negative instants`, () => { - const [minusTen, minusHundred] = rowsFromFixtures(negativeFixtures) - const index = new IndexType(1, new PropRef([`when`]), undefined, { - compareFn: (left: string, right: string) => left.localeCompare(right), - }) - - // This legal callback ordered normalized strings before the Temporal - // index repair. Negative decimal strings distinguish the former display - // operands from the new exact-equality tuple. - index.add(minusTen!.id, minusTen) - index.add(minusHundred!.id, minusHundred) - expect(index.takeFromStart(2)).toEqual([`b-minus-hundred`, `a-minus-ten`]) - expect(index.equalityLookup(minusHundred!.when)).toEqual( - new Set([`b-minus-hundred`]), - ) - }) - - it.each([`add`, `build`] as const)( - `orders equal UTC aliases by a custom callback after %s`, - (construction) => { - const source = [ - { - id: `a-utc`, - when: Temporal.ZonedDateTime.from(`2024-01-15T09:00:00+00:00[UTC]`), - }, - { - id: `z-alias`, - when: Temporal.ZonedDateTime.from( - `2024-01-15T09:00:00+00:00[Etc/UTC]`, - ), - }, - ] - const [utc, alias] = source - expect(utc!.when.equals(alias!.when)).toBe(true) - const callbackArguments: Array<[string, string]> = [] - const index = new IndexType(1, new PropRef([`when`]), undefined, { - compareFn: (left: string, right: string) => { - callbackArguments.push([left, right]) - return left.localeCompare(right) - }, - }) - - // The exact-equality class has both keys. The callback's legacy - // display keys place Etc/UTC before UTC, despite the opposite key order. - if (construction === `add`) { - for (const row of source) index.add(row.id, row) - } else { - index.build(source.map((row): [string, Row] => [row.id, row])) - } - expect(index.equalityLookup(utc!.when)).toEqual( - new Set([`a-utc`, `z-alias`]), - ) - expect(index.takeFromStart(2)).toEqual([`z-alias`, `a-utc`]) - expect(index.rangeQuery({ to: alias!.when })).toEqual( - new Set([`z-alias`]), - ) - expect( - index.rangeQuery({ from: alias!.when, fromInclusive: false }), - ).toEqual(new Set([`a-utc`])) - expect(index.take(2, alias!.when)).toEqual([`a-utc`]) - expect(index.takeReversed(2, utc!.when)).toEqual([`z-alias`]) - expect(callbackArguments.length).toBeGreaterThan(0) - const expectedOperands = new Set( - source.map( - ({ when }) => - `\u0000tanstack-db:temporal:Temporal.ZonedDateTime:${when.toString()}`, - ), - ) - for (const [left, right] of callbackArguments) { - expect(expectedOperands.has(left)).toBe(true) - expect(expectedOperands.has(right)).toBe(true) - } - - // Removal supplies the exact prior source row. The first stored - // representative must be replaced by a surviving row's value. - index.remove(utc!.id, utc) - expect(index.takeFromStart(2)).toEqual([`z-alias`]) - expect(orderedEntriesArray(index)[0]?.[0]).toBe(alias!.when) - expect(index.equalityLookup(alias!.when)).toEqual(new Set([`z-alias`])) - }, - ) - - it.each([`add`, `build`] as const)( - `follows an opposite callback for equal UTC aliases after %s and update`, - (construction) => { - const source = [ - { - id: `z-utc`, - when: Temporal.ZonedDateTime.from(`2024-01-15T09:00:00+00:00[UTC]`), - }, - { - id: `a-alias`, - when: Temporal.ZonedDateTime.from( - `2024-01-15T09:00:00+00:00[Etc/UTC]`, - ), - }, - ] - const [utc, alias] = source - expect(utc!.when.equals(alias!.when)).toBe(true) - expect( - alias!.when.toString().localeCompare(utc!.when.toString()), - ).toBeLessThan(0) - const index = new IndexType(1, new PropRef([`when`]), undefined, { - compareFn: (left: string, right: string) => right.localeCompare(left), - }) - // The callback puts UTC first. Both display-text order and source-key - // order put the alias first, so either shortcut fails this observation. - if (construction === `add`) { - for (const row of source) index.add(row.id, row) - } else { - index.build(source.map((row): [string, Row] => [row.id, row])) - } - expect(index.equalityLookup(utc!.when)).toEqual( - new Set([`z-utc`, `a-alias`]), - ) - expect(index.takeFromStart(2)).toEqual([`z-utc`, `a-alias`]) - expect(index.rangeQuery({ to: utc!.when })).toEqual(new Set([`z-utc`])) - expect(index.take(1, utc!.when)).toEqual([`a-alias`]) - - // Updating Etc/UTC to UTC preserves query equality but makes the - // callback operands identical. Both rows must now occupy the same - // callback position and use the source-key tie rule. An index that - // retains the old per-row position would leak a stale row through a - // bound or cursor at UTC. - const moved = { - id: alias!.id, - when: Temporal.ZonedDateTime.from(utc!.when.toString()), - } - expect(moved.when.equals(alias!.when)).toBe(true) - index.update(alias!.id, alias, moved) - expect(index.equalityLookup(utc!.when)).toEqual( - new Set([`z-utc`, `a-alias`]), - ) - expect(index.takeFromStart(2)).toEqual([`a-alias`, `z-utc`]) - expect(index.rangeQuery({ to: utc!.when, toInclusive: true })).toEqual( - new Set([`a-alias`, `z-utc`]), - ) - expect( - index.rangeQuery({ from: utc!.when, fromInclusive: true }), - ).toEqual(new Set([`a-alias`, `z-utc`])) - expect(index.take(2, utc!.when)).toEqual([]) - expect(index.takeReversed(2, utc!.when)).toEqual([]) - }, - ) - - it(`reorders equal UTC aliases by the callback after a same-class update`, () => { - const alias = Temporal.ZonedDateTime.from( - `2024-01-15T09:00:00+00:00[Etc/UTC]`, - ) - const utc = Temporal.ZonedDateTime.from(`2024-01-15T09:00:00+00:00[UTC]`) - expect(alias.equals(utc)).toBe(true) - expect(alias.toString().localeCompare(utc.toString())).toBeLessThan(0) - const first = { id: `z-utc`, when: alias } - const second = { - id: `a-alias`, - when: Temporal.ZonedDateTime.from(alias.toString()), - } - const index = new IndexType(1, new PropRef([`when`]), undefined, { - compareFn: (left: string, right: string) => right.localeCompare(left), - }) - - // The initial callback operands are identical, so source keys order - // the two rows. The update changes one callback operand to UTC while - // keeping its query-equality class. Reverse locale order then puts - // z-utc first even though display text and source keys put a-alias first. - index.add(first.id, first) - index.add(second.id, second) - expect(index.takeFromStart(2)).toEqual([`a-alias`, `z-utc`]) - const moved = { id: first.id, when: utc } - index.update(first.id, first, moved) - expect(index.equalityLookup(utc)).toEqual(new Set([`z-utc`, `a-alias`])) - expect(index.takeFromStart(2)).toEqual([`z-utc`, `a-alias`]) - expect(index.rangeQuery({ to: utc, toInclusive: true })).toEqual( - new Set([`z-utc`]), - ) - expect(index.take(2, utc)).toEqual([`a-alias`]) - }) - - it(`moves one row within its exact class when its custom order changes`, () => { - const utc = { - id: `a-utc`, - when: Temporal.ZonedDateTime.from(`2024-01-15T09:00:00+00:00[UTC]`), - } - const alias = { - id: `z-alias`, - when: Temporal.ZonedDateTime.from(`2024-01-15T09:00:00+00:00[Etc/UTC]`), - } - const index = new IndexType(1, new PropRef([`when`]), undefined, { - compareFn: (left: string, right: string) => left.localeCompare(right), - }) - index.add(utc.id, utc) - index.add(alias.id, alias) - expect(index.takeFromStart(2)).toEqual([`z-alias`, `a-utc`]) - - // The query equality class stays fixed, while the callback position - // changes from the later UTC display string to the earlier alias text. - const moved = { id: utc.id, when: alias.when } - index.update(utc.id, utc, moved) - expect(index.equalityLookup(alias.when)).toEqual( - new Set([`a-utc`, `z-alias`]), - ) - expect(index.takeFromStart(2)).toEqual([`a-utc`, `z-alias`]) - expect(index.take(2, alias.when)).toEqual([]) - }) - - it(`publishes the source-key first equal-instant row from an ordered Collection snapshot`, async () => { - const rows = rowsFromFixtures() - // Admit the UTC peer before the smaller-key +14:00 peer. Changelog #958 - // establishes key order for comparator ties; this reversal distinguishes - // that public limit-one result from admission order on both read paths. - const admission = [rows[1]!, rows[0]!, ...rows.slice(2)] - const collection = createCollection({ - id: `temporal-index-order-${_name}`, - getKey: (row) => row.id, - autoIndex: `off`, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - begin() - for (const row of admission) write({ type: `insert`, value: row }) - commit() - markReady() - }, - }, - }) - - const orderBy = [ - { - expression: new PropRef([`when`]), - compareOptions: { - direction: `asc` as const, - nulls: `first` as const, - }, - }, - ] - const readFirst = (optimizedOnly: boolean) => - collection - .currentStateAsChanges({ orderBy, limit: 1, optimizedOnly }) - ?.map(({ value }) => value.id) - - await withOracleCleanup(async () => { - await collection.stateWhenReady() - expect(collection.indexes.size).toBe(0) - // The unindexed public scan checks the same ordering law through a - // separate production path before index creation. - expect(readFirst(false)).toEqual([`b-plus-fourteen`]) - - const index = collection.createIndex((row) => row.when, { - indexType: IndexType, - }) - const take = vi.spyOn(index, `takeFromStart`) - // optimizedOnly forbids a scan fallback; the spy shows that the intended - // index path supplied this public snapshot after the build. - const indexed = readFirst(true) - expect(take).toHaveBeenCalled() - expect(indexed).toEqual([`b-plus-fourteen`]) - }, [() => collection.cleanup()]) - }) - }, -) - -type AdmissionRow = { id: string; when: unknown; equalityClass: string } -type AdmissionProbe = { when: unknown; equalityClass: string } - -// These are valid values from one Temporal implementation. The model's class -// labels come from constructor fields and the native equality distinction, -// independent of index normalization or the index's internal sort order. -const admissionFixtures = [ - { - name: `Duration with calendar units`, - orderErrorType: RangeError, - orderErrorMessage: `Missing relativeTo`, - rows: [ - { - id: `a`, - when: Temporal.Duration.from({ months: 1 }), - equalityClass: `one-month`, - }, - { - id: `b`, - when: Temporal.Duration.from({ months: 2 }), - equalityClass: `two-months`, - }, - { - id: `c`, - when: Temporal.Duration.from({ hours: 1 }), - equalityClass: `one-hour`, - }, - ], - probes: [ - { - when: Temporal.Duration.from({ months: 1 }), - equalityClass: `one-month`, - }, - { - when: Temporal.Duration.from({ months: 2 }), - equalityClass: `two-months`, - }, - { when: Temporal.Duration.from({ hours: 1 }), equalityClass: `one-hour` }, - ], - }, - { - name: `PlainMonthDay with distinct reference years`, - orderErrorType: TypeError, - orderErrorMessage: `Temporal.PlainMonthDay has no defined ordering`, - rows: [ - { - id: `a`, - when: new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972), - equalityClass: `1972`, - }, - { - id: `b`, - when: new Temporal.PlainMonthDay(2, 29, `iso8601`, 2004), - equalityClass: `2004`, - }, - { - id: `c`, - when: new Temporal.PlainMonthDay(3, 1, `iso8601`, 1972), - equalityClass: `march`, - }, - ], - probes: [ - { - when: new Temporal.PlainMonthDay(2, 29, `iso8601`, 1972), - equalityClass: `1972`, - }, - { - when: new Temporal.PlainMonthDay(2, 29, `iso8601`, 2004), - equalityClass: `2004`, - }, - { - when: new Temporal.PlainMonthDay(3, 1, `iso8601`, 1972), - equalityClass: `march`, - }, - ], - }, - { - name: `mixed Temporal kinds`, - orderErrorType: TypeError, - orderErrorMessage: `Cannot order Temporal values of different types`, - rows: [ - { - id: `a`, - when: Temporal.Instant.from(`2024-01-01T00:00:00Z`), - equalityClass: `instant-one`, - }, - { - id: `b`, - when: Temporal.PlainDate.from(`2024-01-02`), - equalityClass: `date`, - }, - { - id: `c`, - when: Temporal.Instant.from(`2024-01-03T00:00:00Z`), - equalityClass: `instant-two`, - }, - ], - probes: [ - { - when: Temporal.Instant.from(`2024-01-01T00:00:00Z`), - equalityClass: `instant-one`, - }, - { when: Temporal.PlainDate.from(`2024-01-02`), equalityClass: `date` }, - { - when: Temporal.Instant.from(`2024-01-03T00:00:00Z`), - equalityClass: `instant-two`, - }, - ], - }, -] satisfies Array<{ - name: string - orderErrorType: typeof Error - orderErrorMessage: string - rows: Array - probes: Array -}> - -describe.each(indexTypes)(`%s valid Temporal admission`, (_name, IndexType) => { - it.each(admissionFixtures)( - `retains exact membership across add/build order for $name`, - ({ rows, probes }) => { - const expectedKeys = new Set(rows.map(({ id }) => id)) - // Admission order is not a public order contract for these values. - // Every construction must retain every key in its internal traversal. - for (const construction of [`forward`, `reverse`, `build`] as const) { - const index = new IndexType(1, new PropRef([`when`])) - const admission = - construction === `reverse` ? [...rows].reverse() : rows - if (construction === `build`) { - index.build( - admission.map((row): [string, AdmissionRow] => [row.id, row]), - ) - } else { - for (const row of admission) index.add(row.id, row) - } - for (const row of rows) { - expect(index.equalityLookup(row.when)).toEqual(new Set([row.id])) - } - for (const probe of probes) { - const matchingKeys = rows - .filter((row) => row.equalityClass === probe.equalityClass) - .map((row) => row.id) - expect(index.equalityLookup(probe.when)).toEqual( - new Set(matchingKeys), - ) - } - const ordered = index.takeFromStart(rows.length) - expect(ordered).toHaveLength(rows.length) - expect(new Set(ordered).size).toBe(ordered.length) - expect(new Set(ordered)).toEqual(expectedKeys) - } - }, - ) - - it.each(admissionFixtures)( - `publishes each valid $name row with its exact index membership`, - async ({ rows, orderErrorType, orderErrorMessage }) => { - let commitRow!: (row: AdmissionRow) => void - const collection = createCollection({ - id: `temporal-admission-${_name}-${rows[0]!.equalityClass}`, - getKey: (row) => row.id, - autoIndex: `off`, - startSync: true, - sync: { - sync: ({ begin, write, commit, markReady }) => { - commitRow = (row) => { - begin() - write({ type: `insert`, value: row }) - commit() - } - markReady() - }, - }, - }) - - await withOracleCleanup(async () => { - await collection.stateWhenReady() - const index = collection.createIndex((row) => row.when, { - indexType: IndexType, - }) - const observed: Array<{ - key: string - stored: boolean - exact: Array - }> = [] - const subscription = collection.subscribeChanges( - (changes) => { - for (const change of changes) { - observed.push({ - key: change.key, - stored: collection.get(change.key)?.id === change.key, - exact: [...index.equalityLookup(change.value.when)], - }) - } - }, - { includeInitialState: false }, - ) - try { - // Each synchronous commit is a public checkpoint. The second valid - // value must neither crash the Collection nor leave a stored row - // invisible to the index and subscriber. - commitRow(rows[0]!) - commitRow(rows[1]!) - expect(collection.status).toBe(`ready`) - expect([...collection.keys()]).toEqual([`a`, `b`]) - expect(observed).toEqual([ - { key: `a`, stored: true, exact: [`a`] }, - { key: `b`, stored: true, exact: [`b`] }, - ]) - // An equality index must not turn its private storage order into a - // public orderBy result. Changelog #1519 delegates to Temporal's - // static comparison, which rejects these pairs without relativeTo - // or a common ordered kind. Limit 1 catches an index that returns - // before comparing the second row. - let orderFailure: unknown - try { - collection.currentStateAsChanges({ - orderBy: [ - { - expression: new PropRef([`when`]), - compareOptions: { direction: `asc`, nulls: `first` }, - }, - ], - limit: 1, - }) - } catch (error) { - orderFailure = error - } - expect(orderFailure).toBeInstanceOf(orderErrorType) - expect((orderFailure as Error).message).toContain(orderErrorMessage) - } finally { - subscription.unsubscribe() - } - }, [() => collection.cleanup()]) - }, - ) -}) diff --git a/packages/db/tests/utils-oracle.property.test.ts b/packages/db/tests/utils-oracle.property.test.ts index 3de217b24c..2bfaaa4f08 100644 --- a/packages/db/tests/utils-oracle.property.test.ts +++ b/packages/db/tests/utils-oracle.property.test.ts @@ -1,4 +1,3 @@ -import { runInNewContext } from 'node:vm' import { describe, expect, it } from 'vitest' import { fc, test as fcTest } from '@fast-check/vitest' import { Temporal } from 'temporal-polyfill' @@ -9,29 +8,12 @@ import { deepEquals, equalPersistedSnapshotValues } from '../src/utils' * established examples in `utils.test.ts`, promises change-event equality for * its supported value classes. This oracle checks bounded equality and * inequality laws for those classes, not general graph isomorphism. - * The native Date timestamp rule in `docs/guides/live-queries.md` applies - * across realms. The Collection change-message mirror contract in - * `change-event-history-oracle.test.ts` requires settled events to reconstruct - * public rows. The Date law below checks the resulting equality boundary. - * * Independent value constructors exercise reflexivity, symmetry, copied * structure, and changed-leaf inequality for primitives, arrays, records, * dates, regexes, typed bytes, and selected Temporal types. Named cross-type * pairs remain unequal, even when a primitive displays or encodes a Date or * RegExp value. Separate bounded graph laws construct corresponding rings * and acyclic copies without reusing the production walk. - * Ordinary native Dates compare by time across JavaScript realms. One fixed - * changed-time pair tests a false-equality fallback. Caller-modified built-ins - * are outside this oracle's declared domain. - * Ordinary Temporal.Duration values compare by their ten supplied fields, - * including opposite signs and carried units that represent equal lengths. - * `packages/db/CHANGELOG.md` says equivalent Duration forms remain unequal - * under `eq`. This is the exact-field equality already used by query - * normalization; the direct boundary here does not establish every Collection - * publication path. - * Keyless DataViews have hidden state but no typed-array elements. Distinct - * views from one foreign realm remain unequal by the keyless-instance law; - * pairing one with a typed array must return false. * * Arbitrary different cycle topologies are outside this model. Naming that * exclusion matters: more random examples cannot establish semantics the @@ -110,58 +92,6 @@ const arbitraryFiniteDate = fc .integer({ min: -1_000_000_000_000, max: 2_000_000_000_000 }) .map((time) => new Date(time)) -type DateRealm = `local` | `foreign` -type NativeDatePair = { - leftTime: number | null - rightTime: number | null - leftRealm: DateRealm - rightRealm: DateRealm -} - -// `null` is a model-only invalid-time marker. The supplied times, rather than -// a production Date classifier, determine equality. Both foreign operands are -// constructed in one realm to exercise changed timestamps on one prototype. -function nativeDatePair(spec: NativeDatePair): [Date, Date] { - const leftTime = spec.leftTime ?? Number.NaN - const rightTime = spec.rightTime ?? Number.NaN - const [foreignLeft, foreignRight] = runInNewContext( - `[new Date(leftTime), new Date(rightTime)]`, - { leftTime, rightTime }, - ) as [Date, Date] - const left = spec.leftRealm === `foreign` ? foreignLeft : new Date(leftTime) - const right = - spec.rightRealm === `foreign` ? foreignRight : new Date(rightTime) - return [left, right] -} - -function assertNativeDateEquality( - spec: NativeDatePair, - equal: (left: unknown, right: unknown) => boolean = deepEquals, -): void { - const [left, right] = nativeDatePair(spec) - const expected = Object.is(spec.leftTime, spec.rightTime) - expect(equal(left, right), `forward ${JSON.stringify(spec)}`).toBe(expected) - expect(equal(right, left), `reverse ${JSON.stringify(spec)}`).toBe(expected) -} - -const arbitraryNativeDatePair = fc - .record({ - leftTime: fc.oneof( - fc.integer({ min: -1_000_000_000_000, max: 1_000_000_000_000 }), - fc.constant(null), - ), - relation: fc.constantFrom(`equal` as const, `changed` as const), - leftRealm: fc.constantFrom(`local` as const, `foreign` as const), - rightRealm: fc.constantFrom(`local` as const, `foreign` as const), - }) - .map(({ leftTime, relation, leftRealm, rightRealm }) => ({ - leftTime, - rightTime: - relation === `equal` ? leftTime : leftTime === null ? 1 : leftTime + 1, - leftRealm, - rightRealm, - })) - const arbitraryRegExp = fc .tuple(fc.string(), fc.constantFrom(``, `g`, `i`, `gi`, `m`, `gim`)) .map(([source, flags]) => { @@ -193,92 +123,6 @@ const arbitraryTemporalDuration = fc Temporal.Duration.from({ hours, minutes, seconds }), ) -// The supplied vector is the independent model: equality means every named -// Duration field agrees. It does not consult display text or the production -// normalizer. One shared sign keeps every generated Duration valid. -const exactDurationFields = [ - `years`, - `months`, - `weeks`, - `days`, - `hours`, - `minutes`, - `seconds`, - `milliseconds`, - `microseconds`, - `nanoseconds`, -] as const -type ExactDurationField = (typeof exactDurationFields)[number] -type ExactDurationSpec = Record - -// These are the adjacent unit conversions with a fixed factor. Month-to-week -// conversion has no fixed factor, so it is deliberately absent. -const durationCarryPairs = [ - [`years`, `months`, 12], - [`weeks`, `days`, 7], - [`days`, `hours`, 24], - [`hours`, `minutes`, 60], - [`minutes`, `seconds`, 60], - [`seconds`, `milliseconds`, 1000], - [`milliseconds`, `microseconds`, 1000], - [`microseconds`, `nanoseconds`, 1000], -] as const - -const arbitraryExactDurationMagnitudes: fc.Arbitrary = - fc.record({ - years: fc.integer({ min: 0, max: 3 }), - months: fc.integer({ min: 0, max: 3 }), - weeks: fc.integer({ min: 0, max: 3 }), - days: fc.integer({ min: 0, max: 3 }), - hours: fc.integer({ min: 0, max: 3 }), - minutes: fc.integer({ min: 0, max: 3 }), - seconds: fc.integer({ min: 0, max: 3 }), - milliseconds: fc.integer({ min: 0, max: 3 }), - microseconds: fc.integer({ min: 0, max: 3 }), - nanoseconds: fc.integer({ min: 0, max: 3 }), - }) - -function signedDurationSpec( - magnitudes: ExactDurationSpec, - sign: -1 | 1, -): ExactDurationSpec { - return Object.fromEntries( - exactDurationFields.map((field) => [ - field, - magnitudes[field] === 0 ? 0 : magnitudes[field] * sign, - ]), - ) as ExactDurationSpec -} - -function assertExactDurationEquality( - leftSpec: ExactDurationSpec, - rightSpec: ExactDurationSpec, - equal: (left: unknown, right: unknown) => boolean = deepEquals, -): void { - const left = Temporal.Duration.from(leftSpec) - const right = Temporal.Duration.from(rightSpec) - const expected = exactDurationFields.every( - (field) => leftSpec[field] === rightSpec[field], - ) - expect( - equal(left, right), - `forward ${JSON.stringify([leftSpec, rightSpec])}`, - ).toBe(expected) - expect( - equal(right, left), - `reverse ${JSON.stringify([leftSpec, rightSpec])}`, - ).toBe(expected) -} - -function singleDurationField( - field: ExactDurationField, - value: number, -): ExactDurationSpec { - return Object.fromEntries( - exactDurationFields.map((name) => [name, name === field ? value : 0]), - ) as ExactDurationSpec -} - // Same-type value arbitraries for testing equivalence properties // Mixed Date/Temporal inequality has its own generated and fixed witnesses below. const arbitrarySameTypePrimitive = fc.oneof( @@ -1048,55 +892,6 @@ describe(`deepEquals property-based tests`, () => { ) }) - // Legal pairs cross both realm placements and equal or changed valid and - // invalid times. The direct deepEquals result is the observation cut. A - // local-only classifier fails an equal mixed-realm pair. Own properties, - // non-Date objects, and changed prototypes are outside this grammar. - it(`reconstructs native Date realm and time margins`, () => { - for (const leftRealm of [`local`, `foreign`] as const) { - for (const rightRealm of [`local`, `foreign`] as const) { - for (const leftTime of [1, null]) { - for (const rightTime of [leftTime, leftTime === null ? 1 : 2]) { - assertNativeDateEquality({ - leftTime, - rightTime, - leftRealm, - rightRealm, - }) - } - } - } - } - }) - - it(`rejects a local-only Date classifier and a changed-time fallback`, () => { - const mixedEqual: NativeDatePair = { - leftTime: 1, - rightTime: 1, - leftRealm: `local`, - rightRealm: `foreign`, - } - const changed: NativeDatePair = { - leftTime: 1, - rightTime: 2, - leftRealm: `foreign`, - rightRealm: `foreign`, - } - expect(() => - assertNativeDateEquality( - mixedEqual, - (left, right) => - left instanceof Date && - right instanceof Date && - Object.is(left.getTime(), right.getTime()), - ), - ).toThrow(`forward`) - expect(() => assertNativeDateEquality(changed, () => true)).toThrow( - `forward`, - ) - assertNativeDateEquality(changed) - }) - utilsProperty([arbitraryTemporalPlainDate])( `Temporal.PlainDate with same values are equal`, (date) => { @@ -1335,20 +1130,6 @@ describe(`deepEquals property-based tests`, () => { }, ) - it(`keeps foreign DataViews opaque and distinct from typed arrays`, () => { - const [first, second] = runInNewContext( - `[new DataView(new Uint8Array([1]).buffer), - new DataView(new Uint8Array([1]).buffer)]`, - ) as [DataView, DataView] - const local = new DataView(Uint8Array.of(1).buffer) - const bytes = Uint8Array.of(1) - // A DataView has no typed-array elements. - expectEqualityPair(first, first, true) - expectEqualityPair(first, second, false) - expectEqualityPair(local, first, false) - expectEqualityPair(bytes, first, false) - }) - utilsProperty([ fc.array(fc.oneof(fc.integer(), fc.constant(`hole`)), { minLength: 1, @@ -1452,139 +1233,6 @@ describe(`deepEquals property-based tests`, () => { (regex) => assertDistinctTypes(regex, regex.source, deepEquals), ) }) - - // Keep this generated property after every existing ordinal registration so - // saved UTILS_ORACLE_PROPERTY seed/path replays keep their original targets. - describe(`ordinary native Date realm boundary`, () => { - utilsProperty([arbitraryNativeDatePair])( - `ordinary native Date equality follows time across realms`, - assertNativeDateEquality, - ) - }) - - // Keep both registrations after every existing ordinal replay selector. - describe(`exact Temporal.Duration field equality`, () => { - // Fresh equal vectors and each single-field change reconstruct the whole - // ten-field grammar. Generated magnitudes range from 0 to 3 under one - // shared positive or negative sign; a selected field changes away from - // zero. Equal and changed relations, and both signs, contribute distinct - // outcomes. Mixed-sign vectors are invalid and excluded. The direct - // deepEquals result is the observation cut. - it(`accepts fresh copies and distinguishes each of ten fields`, () => { - for (const field of exactDurationFields) { - const zero = singleDurationField(field, 0) - assertExactDurationEquality(zero, { ...zero }) - for (const sign of [-1, 1]) { - assertExactDurationEquality(zero, singleDurationField(field, sign)) - } - } - }) - - it(`distinguishes opposite signs and every fixed-factor unit carry`, () => { - const positive = singleDurationField(`seconds`, 1) - const negative = singleDurationField(`seconds`, -1) - assertExactDurationEquality(positive, negative) - expect(() => - assertExactDurationEquality(positive, negative, (a, b) => - exactDurationFields.every( - (field) => - Math.abs((a as Temporal.Duration)[field]) === - Math.abs((b as Temporal.Duration)[field]), - ), - ), - ).toThrow(`forward`) - - for (const [larger, smaller, factor] of durationCarryPairs) { - for (const sign of [-1, 1]) { - const leftSpec = singleDurationField(larger, sign) - const rightSpec = singleDurationField(smaller, sign * factor) - assertExactDurationEquality(leftSpec, rightSpec) - // A comparator that balances just this adjacent pair is wrong even - // when the two Durations represent the same elapsed length. - expect(() => - assertExactDurationEquality(leftSpec, rightSpec, (a, b) => { - const left = a as Temporal.Duration - const right = b as Temporal.Duration - return ( - exactDurationFields.every( - (field) => - field === larger || - field === smaller || - left[field] === right[field], - ) && - left[larger] * factor + left[smaller] === - right[larger] * factor + right[smaller] - ) - }), - ).toThrow(`forward`) - } - } - }) - - it(`rejects display equality for all subsecond representations`, () => { - for (const [larger, smaller] of [ - [`seconds`, `milliseconds`], - [`milliseconds`, `microseconds`], - [`microseconds`, `nanoseconds`], - ] as const) { - for (const sign of [-1, 1]) { - const leftSpec = singleDurationField(larger, sign) - const rightSpec = singleDurationField(smaller, sign * 1000) - const left = Temporal.Duration.from(leftSpec) - const right = Temporal.Duration.from(rightSpec) - expect(left.toString()).toBe(right.toString()) - assertExactDurationEquality(leftSpec, rightSpec) - // A display-only comparator is a plausible wrong design. The model - // rejects it at the same direct-result checkpoint as production. - expect(() => - assertExactDurationEquality( - leftSpec, - rightSpec, - (a, b) => String(a) === String(b), - ), - ).toThrow(`forward`) - } - } - }) - - utilsProperty([ - arbitraryExactDurationMagnitudes, - fc.constantFrom(-1 as const, 1 as const), - fc.constantFrom(...exactDurationFields), - fc.boolean(), - ])( - `fresh Duration copies agree and any changed field differs`, - (magnitudes, sign, field, changed) => { - const base = signedDurationSpec(magnitudes, sign) - const next = { ...base, [field]: base[field] + (changed ? sign : 0) } - assertExactDurationEquality(base, next) - }, - ) - - // These three unit-pair axes are the display-collision margin. The - // magnitude ranges 1–100 with either shared sign; omitting either sign or - // an axis leaves a valid subsecond representation outside this claim. - utilsProperty([ - fc.integer({ min: 1, max: 100 }), - fc.constantFrom(-1 as const, 1 as const), - fc.constantFrom( - [`seconds`, `milliseconds`] as const, - [`milliseconds`, `microseconds`] as const, - [`microseconds`, `nanoseconds`] as const, - ), - ])( - `equal-display Duration field vectors remain unequal`, - (n, sign, pair) => { - const [larger, smaller] = pair - const leftSpec = singleDurationField(larger, sign * n) - const rightSpec = singleDurationField(smaller, sign * 1000 * n) - expect(Temporal.Duration.from(leftSpec).toString()).toBe( - Temporal.Duration.from(rightSpec).toString(), - ) - assertExactDurationEquality(leftSpec, rightSpec) - }, - ) - }) }) // Persisted-snapshot equality decides whether a reload can leave the public diff --git a/packages/db/tests/utils.ts b/packages/db/tests/utils.ts index ae9070b2e7..c952b7674e 100644 --- a/packages/db/tests/utils.ts +++ b/packages/db/tests/utils.ts @@ -598,14 +598,10 @@ export function withExpectedRejection( }).finally(() => restoreListeners()) } -type IndexInternals = { indexedKeys: Map } & ( +type IndexInternals = { indexedKeys: Set } & ( + | { sortedValues: Array; valueMap: Map> } | { - sortedValues: Array - valueMap: Map> - orderedBuckets: Array> - } - | { - valueMap: Map> + valueMap: Map }> orderedEntries: { size: number minKey: () => unknown @@ -614,10 +610,7 @@ type IndexInternals = { indexedKeys: Map } & ( low: unknown, high: unknown, includeHigh: boolean, - onFound: ( - key: unknown, - bucket: { values: Map }, - ) => void, + onFound: (key: unknown, bucket: { keys: Set }) => void, ) => void } } @@ -635,7 +628,7 @@ function indexInternals(index: object): [IndexInternals, boolean] { /** Test inspection of an index's tracked keys. */ export function indexedKeysSet(index: object): Set { - return new Set(indexInternals(index)[0].indexedKeys.keys()) + return indexInternals(index)[0].indexedKeys } /** Test inspection of an index's value buckets keyed by indexed value. */ @@ -643,8 +636,8 @@ export function valueMapData(index: object): Map> { const [internals] = indexInternals(index) if (`sortedValues` in internals) return internals.valueMap const result = new Map>() - for (const [key, keys] of internals.valueMap) { - result.set(denormalizeUndefined(key), keys) + for (const [key, bucket] of internals.valueMap) { + result.set(denormalizeUndefined(key), bucket.keys) } return result } @@ -656,18 +649,16 @@ export function orderedEntriesArray( const [internals, reversed] = indexInternals(index) let entries: Array<[unknown, Set]> if (`sortedValues` in internals) { - if (internals.sortedValues.length !== internals.orderedBuckets.length) - throw new Error(`BasicIndex order buckets are misaligned`) - entries = internals.sortedValues.map((value, position) => [ + entries = internals.sortedValues.map((value) => [ value, - new Set(internals.orderedBuckets[position]!.keys()), + internals.valueMap.get(value) ?? new Set(), ]) } else { const tree = internals.orderedEntries entries = [] if (tree.size > 0) { tree.forRange(tree.minKey(), tree.maxKey(), true, (key, bucket) => { - entries.push([denormalizeUndefined(key), new Set(bucket.values.keys())]) + entries.push([denormalizeUndefined(key), bucket.keys]) }) } } diff --git a/packages/query-db-collection/tests/query.test.ts b/packages/query-db-collection/tests/query.test.ts index 0f97550a27..1dbd38b2df 100644 --- a/packages/query-db-collection/tests/query.test.ts +++ b/packages/query-db-collection/tests/query.test.ts @@ -3141,21 +3141,18 @@ describe(`QueryCollection`, () => { syncMode: `on-demand`, }), ) - // These are fixed provider demands; mutable lists acquire the full source. - const firstIds = Object.freeze([`1`, `2`]) - const secondIds = Object.freeze([`2`, `3`]) const firstLiveQuery = createLiveQueryCollection({ query: (q) => q .from({ item: collection }) - .where(({ item }) => inArray(item.id, firstIds)) + .where(({ item }) => inArray(item.id, [`1`, `2`])) .select(({ item }) => ({ id: item.id, name: item.name })), }) const secondLiveQuery = createLiveQueryCollection({ query: (q) => q .from({ item: collection }) - .where(({ item }) => inArray(item.id, secondIds)) + .where(({ item }) => inArray(item.id, [`2`, `3`])) .select(({ item }) => ({ id: item.id, name: item.name })), }) @@ -7168,20 +7165,17 @@ describe(`QueryCollection`, () => { syncMode: `on-demand`, }) const collection = createCollection(options) - // Separate fixed demands make shared row ownership observable. - const firstIds = Object.freeze([`1`, `2`]) - const secondIds = Object.freeze([`2`, `3`]) const firstSubset = createLiveQueryCollection({ query: (q) => q .from({ item: collection }) - .where(({ item }) => inArray(item.id, firstIds)), + .where(({ item }) => inArray(item.id, [`1`, `2`])), }) const secondSubset = createLiveQueryCollection({ query: (q) => q .from({ item: collection }) - .where(({ item }) => inArray(item.id, secondIds)), + .where(({ item }) => inArray(item.id, [`2`, `3`])), }) await firstSubset.preload() @@ -11208,12 +11202,6 @@ describe(`QueryCollection`, () => { }), ) - // These fixed lists retain distinct subset demands while writes are held. - const taskIds = Object.freeze([10]) - const allResources = Object.freeze([1, 2, 3, 4]) - const postInsertResources = Object.freeze([1, 2, 4]) - const firstTwoResources = Object.freeze([1, 2]) - // Complete server population is deliberately returned for every demand: // this fixture tests cache writeback/ownership, not backend filtering. const world = new Map([ @@ -11258,7 +11246,7 @@ describe(`QueryCollection`, () => { query: (q) => q .from({ assignment: collection }) - .where(({ assignment }) => inArray(assignment.task_id, taskIds)), + .where(({ assignment }) => inArray(assignment.task_id, [10])), }) cleanups.push(() => taskQuery.cleanup()) try { @@ -11283,7 +11271,7 @@ describe(`QueryCollection`, () => { q .from({ assignment: collection }) .where(({ assignment }) => - inArray(assignment.resource_id, allResources), + inArray(assignment.resource_id, [1, 2, 3, 4]), ), }) cleanups.push(() => workloadQuery.cleanup()) @@ -11347,7 +11335,7 @@ describe(`QueryCollection`, () => { q .from({ assignment: collection }) .where(({ assignment }) => - inArray(assignment.resource_id, postInsertResources), + inArray(assignment.resource_id, [1, 2, 4]), ), }) const firstLoad = startHeldWorkloadLoad() @@ -11388,7 +11376,7 @@ describe(`QueryCollection`, () => { q .from({ assignment: collection }) .where(({ assignment }) => - inArray(assignment.resource_id, firstTwoResources), + inArray(assignment.resource_id, [1, 2]), ), }) const deleteLoad = startHeldWorkloadLoad() @@ -11412,7 +11400,7 @@ describe(`QueryCollection`, () => { q .from({ assignment: collection }) .where(({ assignment }) => - inArray(assignment.resource_id, allResources), + inArray(assignment.resource_id, [1, 2, 3, 4]), ), }) await workloadQuery.preload() @@ -11443,7 +11431,7 @@ describe(`QueryCollection`, () => { q .from({ assignment: collection }) .where(({ assignment }) => - inArray(assignment.resource_id, postInsertResources), + inArray(assignment.resource_id, [1, 2, 4]), ), }) const secondLoad = startHeldWorkloadLoad() From 2b3947fb2c55ab00a1278cce93c15546913c2883 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 11:01:57 -0600 Subject: [PATCH 36/46] docs(db): record exact-commit hostile oracle reviews --- .../oracle-reviews/basic-index-nullish-build.md | 9 +++++++++ .../comparison-unit-retirement-round-5.md | 10 ++++++++++ .../comparison-unit-retirement-round-6.md | 10 ++++++++++ .../oracle-reviews/utils-unit-retirement-round-7.md | 10 ++++++++++ 4 files changed, 39 insertions(+) diff --git a/docs/contributing/oracle-reviews/basic-index-nullish-build.md b/docs/contributing/oracle-reviews/basic-index-nullish-build.md index d1a3e719d4..5ab6dc279e 100644 --- a/docs/contributing/oracle-reviews/basic-index-nullish-build.md +++ b/docs/contributing/oracle-reviews/basic-index-nullish-build.md @@ -141,3 +141,12 @@ focused index, work, and Collection-index run passed 535 tests with no type errors. The recheck found no lost direct-index assertion after the grouping change. It did not rerun the original wrong-sort mutant or extend the law to raw duplicate source keys or custom callbacks within one normalized value. + +At exact code commit `24a61a8d5b6cef06d488c490e39f351b4d55fb68`, a +fresh hostile reviewer rechecked the one-pass build against the seven retired +index law families and the nullish fixed history. The source Map, wrapper +sort, and direct observations remained intact after the commit hook. A +focused index, work, and Collection-index run passed 535 tests with no type +errors. The reviewer found no in-scope loss. This recheck did not rerun the +original wrong-sort mutant; the earlier RED run remains the sensitivity +evidence. diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md index 55ddc816f4..337c0c7738 100644 --- a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-5.md @@ -60,3 +60,13 @@ input labels, with no state transition or host provider (ORC-008, ORC-009, and ORC-014). This ledger records the loss audit, hostile challenge, and bounded result (ORC-011 and ORC-012); it makes no claim about other value domains. + +## Exact code-commit review + +At `24a61a8d5b6cef06d488c490e39f351b4d55fb68`, a fresh hostile reviewer +compared all five retired blocks with the committed tagged model, direct +comparator checks, and sorted row IDs. It found no lost assertion. The +committed production and oracle blobs match the reviewed working tree; the +commit hook did not change them. A focused comparison and evaluator run +passed 236 tests with no type errors. This verdict does not extend to +`compareValues` or public query and index paths. diff --git a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md index e59994c673..c503924e11 100644 --- a/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md +++ b/docs/contributing/oracle-reviews/comparison-unit-retirement-round-6.md @@ -60,3 +60,13 @@ seed/path replay verifies the shared campaign harness (ORC-007 and ORC-010). The tuple and instant model has no mutable state or host provider (ORC-008, ORC-009, and ORC-014). This record preserves the loss audit, hostile review, and explicit limits (ORC-011 and ORC-012); it makes no cross-boundary claim. + +## Exact code-commit review + +At `24a61a8d5b6cef06d488c490e39f351b4d55fb68`, a fresh hostile reviewer +reconstructed the three PlainDate assertions and equal-instant zoned +assertion from the committed oracle. The comparator and oracle blobs match +the versions reviewed before commit. Its focused comparison and evaluator +run passed 236 tests with no type errors. It found no lost ordinary +same-realm assertion. Modified Temporal inputs, `compareValues`, and public +query or index paths remain outside this verdict. diff --git a/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md b/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md index a4e40399b0..5bdfa6d9b0 100644 --- a/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md +++ b/docs/contributing/oracle-reviews/utils-unit-retirement-round-7.md @@ -104,3 +104,13 @@ read the six retired Date/RegExp assertions, their source, the direct found no missing operand or expected result in the bounded transfer. This is a frozen-head no-loss verdict by inspection; it is not a new mutant kill or a fresh committed-head execution of subsequent working-tree changes. + +## Exact code-commit review of the narrowed transfer + +At `24a61a8d5b6cef06d488c490e39f351b4d55fb68`, a fresh hostile reviewer +checked the committed six retired Date/RegExp assertions against their +fixed witnesses, generated laws, and production `deepEquals` path. It found +no loss at the direct-return checkpoint. The committed oracle and remaining +unit file passed 208 tests with no type errors. The scoped files and commit +check were clean. This verdict does not establish public change events or +cross-realm behavior. From ee895c8c69adae16f71e773efb92ba6842b85abd Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 12:13:11 -0600 Subject: [PATCH 37/46] fix(db): retain confirmed oracle-discovered behavior --- .changeset/fix-basic-index-nullish-build.md | 2 +- docs/contributing/oracle-coverage.md | 49 +- .../oracle-reviews/pr-2116-confirmed-fixes.md | 49 ++ .../pr-2116-deferred-findings.md | 23 +- packages/db/package.json | 2 +- packages/db/src/collection/change-events.ts | 2 +- packages/db/src/indexes/auto-index.ts | 3 +- packages/db/src/indexes/base-index.ts | 49 +- packages/db/src/indexes/basic-index.ts | 83 +--- packages/db/src/indexes/btree-index.ts | 72 +-- packages/db/src/query/compiler/order-by.ts | 8 - packages/db/src/utils.ts | 24 + packages/db/src/utils/comparison.ts | 111 ++++- .../tests/change-event-history-oracle.test.ts | 84 ++++ .../duration-query-equality-oracle.test.ts | 173 +++++++ .../index-update-oracle.property.test.ts | 266 +++++++++- .../in-evaluator-oracle.property.test.ts | 52 +- ...ustom-index-order-selection-oracle.test.ts | 84 ++++ .../tests/temporal-index-order-oracle.test.ts | 458 ++++++++++++++++++ .../db/tests/utils-oracle.property.test.ts | 82 +++- 20 files changed, 1479 insertions(+), 197 deletions(-) create mode 100644 docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md create mode 100644 packages/db/tests/duration-query-equality-oracle.test.ts create mode 100644 packages/db/tests/query/custom-index-order-selection-oracle.test.ts create mode 100644 packages/db/tests/temporal-index-order-oracle.test.ts diff --git a/.changeset/fix-basic-index-nullish-build.md b/.changeset/fix-basic-index-nullish-build.md index d8663b8549..c0610889b1 100644 --- a/.changeset/fix-basic-index-nullish-build.md +++ b/.changeset/fix-basic-index-nullish-build.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Keep `undefined` in the configured nulls-first position when BasicIndex builds from existing rows. +Keep `undefined` in the configured nulls-first position when BasicIndex builds from existing rows. Preserve indexed key ownership and public string order across mixed and prefixed values. Make Duration equality follow its fields, order indexed ZonedDateTimes by instant and time-only Durations by elapsed time, and ignore a manual index whose order conflicts with a limited public read. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 7d63feb0b8..107d146986 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -474,9 +474,19 @@ The [round 1](oracle-reviews/index-unit-retirement-round-1.md), [round 3](oracle-reviews/index-unit-retirement-round-3.md), and [round 4](oracle-reviews/index-unit-retirement-round-4.md) ledgers retain each retired assertion. The [build review](oracle-reviews/basic-index-nullish-build.md) -records the direct RED/GREEN repair. This owner does not establish public -Collection publication, arbitrary custom callbacks, or all JavaScript value -kinds. +records the direct RED/GREEN repair. A separate mixed number/string lane +checks that full direct traversals and one indexed, bounded public Collection +snapshot return each live source key once after deletion and after a direct +delete/update/re-add history. Two more direct histories keep an ordinary string +beside ordered ZonedDateTime or Duration keys and verify key conservation after +one Temporal deletion. An en-US locale and a custom string-collation cell compare +the first bounded public row through a scan and each index when an ordinary +string begins the internal encoding prefix. An `origin/main` probe failed the +default-locale public prefix case. The pre-classifier PR comparator failed all +four public prefix cases and both BTree Temporal-deletion histories. This owner +does not prescribe mixed-kind order or cover arbitrary custom index callbacks, +all JavaScript value kinds, or +general Collection publication histories. The [comparison oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts) owns the transferred same-realm NaN and invalid-Date default-order cases, @@ -494,11 +504,34 @@ The [round 7 ledger](oracle-reviews/utils-unit-retirement-round-7.md) retains the retired examples. Caller-modified built-ins and Collection publication are outside this direct-return owner. -The [deferred findings record](oracle-reviews/pr-2116-deferred-findings.md) -assigns unclosed Duration equality and Collection publication, Temporal index -admission, mixed-kind order, and custom-comparator questions to future owners. -Earlier PR-branch counterexamples do not by themselves establish a current -`origin/main` failure; each future repair needs its own baseline witness. +The [Duration query equality oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/duration-query-equality-oracle.test.ts) +uses ten public fields to predict direct BasicIndex/BTreeIndex exact lookups and +public `eq` snapshots before and after indexing. The `in` evaluator owner also +checks same-display Duration forms through constant and row-supplied lists. +The deep-equality and Collection change-event history owners check a Duration +field replacement and its subscriber mirror. These fixed histories do not +establish arbitrary multirow Duration updates or live-query publication. + +The [Temporal index order oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/temporal-index-order-oracle.test.ts) +owns ZonedDateTime instant ordering and time-only Duration ordering for direct +BasicIndex/BTreeIndex reads and bounded public first-row reads. It separates +exact Duration fields from order ties, including exclusive cursor behavior. +Calendar-unit Duration order, mixed Temporal kinds, arbitrary updates, and +custom index callbacks remain outside this owner. + +The [custom-index selection oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/custom-index-order-selection-oracle.test.ts) +checks that a reverse manual index cannot change an ascending limited public +snapshot or eager live-query result. Its two distinct numeric ranks avoid any +claim about per-row custom callback order inside a comparator tie. On-demand +acquisition and later updates remain outside this witness. + +The [PR #2116 confirmed-fixes review](oracle-reviews/pr-2116-confirmed-fixes.md) +records baseline failures, hostile controls, and the bounded scope of these +repairs. The [deferred findings record](oracle-reviews/pr-2116-deferred-findings.md) +tracks remaining design and coverage questions, including valid Temporal kinds +without public order, extended-year PlainDate under lexical string collation, +and mixed-kind query order. Earlier PR-branch +counterexamples do not by themselves establish an `origin/main` failure. The Query DB ownership oracle owns the direct-write no-automatic-refetch law ([review](oracle-reviews/issue-1935-direct-write-refetch.md)). diff --git a/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md b/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md new file mode 100644 index 0000000000..e7ec02e82e --- /dev/null +++ b/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md @@ -0,0 +1,49 @@ +# PR #2116: confirmed fixes found during unit-test retirement + +The seven index, five non-finite comparison, two ordinary Temporal comparison, +and six Date/RegExp assertions retired by this PR have their own loss ledgers in +the [coverage map](../oracle-coverage.md#unit-test-retirement-transfers-in-pr-2116). +Those transfers did not authorize new product guarantees. Each additional fix +below has an accepted law, a failing observation before repair, an independent +receiving witness, and a bounded hostile challenge. `origin/main` was +`bbbf017ef5b3884515295a4225faacdfd814499e` for the baseline checks. + +| Law and authority | Baseline failure | Receiving observation and hostile result | Limit | +| --- | --- | --- | --- | +| A full BasicIndex build keeps `undefined` at the configured nulls-first position. The default comparator and surviving ordered-index tests supply the rule. | Native `Array.sort` moved a raw `undefined` value last without calling the comparator. | The source-Map direct-index owner compares complete ordered reads after full build. The [build review](basic-index-nullish-build.md) records RED/GREEN and hostile evidence. | Default nullish order and bounded build histories. | +| Duration identity preserves all ten public fields. Changelog #1519 explicitly keeps equivalent displayed forms unequal for query equality; the Collection subscriber-mirror contract requires a changed row to reach its subscriber. | One second and 1000 milliseconds display alike. The former `deepEquals` and normalized index key treated them as equal, suppressing a Collection change message or returning a wrong exact match. The original implementation failed direct BasicIndex/BTreeIndex lookup, public scan and indexed `eq`, and constant and row-supplied `in` probes. | The deep-equality and change-event oracles compare a direct result, stored row, callback batch, and subscriber mirror. The Duration query owner checks ten fields through BasicIndex/BTreeIndex exact lookup and public `eq`, while the `in` evaluator owner uses constant and row-supplied lists. A seconds-only production mutant fails the adjacent-unit deep-equality property. Executed wrong-key controls reject display-string, sign-loss, and first-nonzero-field query keys; they were not production mutants. | Fixed replacement and single-row query histories; no arbitrary multirow update or live-query publication claim. | +| An indexed ZonedDateTime `orderBy` follows instant order, as promised by changelog #1519. A time-only Duration uses its elapsed-time order while keeping exact field identity. | Indexed first-row reads placed a later ZonedDateTime first when display text opposed epoch order, including a 9ns/10ns digit-width pair. PT120M also preceded PT1H under string-key order. | The Temporal index-order oracle compares exact direct BasicIndex/BTreeIndex reads and bounded public first rows, including an on-demand Collection. Adjacent nanoseconds at a large epoch reject `Number` precision loss. PT1H/PT60M exclusive cursors reject an exact-suffix tie-break; negative durations reject absolute-value order. The hostile reviewer found no sign mismatch in 2,000 seeded valid day/time pairs. | ZonedDateTime instant and time-only Duration order in the named fixed histories. Calendar units needing `relativeTo`, mixed Temporal kinds, custom index callbacks, and arbitrary update histories remain open. | +| A manual index cannot change the declared ascending `orderBy(rank).limit(1)` answer. Query order is public; index selection is physical planning. | A reverse manual BTree index made `currentStateAsChanges` return rank 30 instead of rank 10. | The custom-index selection oracle independently sorts two distinct numeric ranks and checks the direct public snapshot and an eager live-query result. The hostile reviewer found the direct path that the first witness missed; both paths now pass. An added compiler guard had no demonstrated effect on this witness and was removed. | Eager initial reads with distinct ranks; no callback-tie, on-demand, later-update, or work-bound claim. | +| Index updates conserve each source key even when number and string values occupy different comparison domains. | The old comparator could make `2` tie `"2"` while putting them on opposite sides of `"10"`; a direct deletion then duplicated or lost a key. | A source Map predicts full direct traversals after bounded delete/update/re-add actions, and a bounded indexed Collection read checks the public key set. The hostile reviewer confirmed the original comparator fails the source-Map comparison. | Key conservation, not a mixed-kind public order policy or all Collection histories. | +| An ordinary string beginning `\0tanstack-db:` obeys its declared string collation in indexed and scan reads. Internal Temporal order keys must not cause a tree to lose a neighboring ordinary string. | On `origin/main`, an en-US locale scan selected `\0tanstack-db:a` first while BasicIndex/BTreeIndex selected `tanstack-db:r`. Before the final kind classifier, deleting a neighboring ZonedDateTime or Duration also made BTreeIndex omit the surviving string. | The direct-index owner compares first public rows through scan and indexed paths under locale and custom string collations, with row IDs independent of label order. Two three-key BTree histories compare all surviving keys. The hostile reviewer ran 24 focused tests, 108 add/delete permutations, and a 27-value comparator transitivity sweep across five option variants. After the final classifier simplification, it also ran 3,456 four-key BasicIndex/BTreeIndex insertion/deletion histories (27,648 checkpoints) and a 16-value comparator sweep; no in-domain counterexample remained. | Named scalar, binary, Temporal, and string-kind histories. No arbitrary nested values, mixed-kind relative order, or custom index `compareFn` guarantee. | + +## Design decisions and open work + +The captured-`inArray` list rebuild and per-row custom index callback ordering +were proposed extensions, then withdrawn by the maintainer. This PR does not +promise behavior after callers modify standard data objects in place, unsigned +byte ordering, or special handling for malformed Temporal lookalikes. The +[deferred record](pr-2116-deferred-findings.md) keeps valid Temporal admission, +mixed-kind query order, and earlier-branch observations separate from the +confirmed fixes above. + +The repeatable workflow is: freeze source assertions in a loss ledger; name +the receiving law and its limits; obtain a baseline failure before production +changes; extend an independent model at the public checkpoint; ask a fresh +hostile reviewer to attack authority, reach, and plausible mutants; then +recheck the exact committed head. A production-code budget is evaluated +separately from oracle and documentation growth. A green generated campaign +supports only its legal histories, paths, and observations. + +## Production-code weight + +Against the recorded `origin/main` baseline, the final source diff under +`packages/db/src` adds 200 lines and removes 211, for **11 fewer production +lines**. The built DB ESM entry bundled with esbuild (`bundle`, `minify`, +`format: esm`, `platform: neutral`, `NODE_ENV=production`; DB IVM, pacer-lite, +and standard-schema external) is 334,795 bytes / 100,227 gzip at baseline and +335,886 bytes / 100,644 gzip for this change. That is **1,091 more minified +bytes / 417 more gzip bytes**. The source-line budget is negative; the bundle +budget is still positive. The remaining runtime addition is concentrated in +distinct index-storage comparison and exact Temporal keys, which the bounded +fixes above require. Do not treat source-line reduction as a bundle-size win. diff --git a/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md b/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md index 032662dff4..8bf29842c7 100644 --- a/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md +++ b/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md @@ -1,18 +1,19 @@ # PR #2116: deferred findings and guarantee decisions This record separates findings discovered during the unit-test retirement -workflow from the transfers retained in PR #2116. The final PR keeps only -the seven direct-index families, five non-finite comparison blocks, two -ordinary Temporal comparison blocks, and six Date/RegExp equality assertions. -It repairs the BasicIndex nullish full-build mismatch. A finding below is not -closed by passing those transfer oracles. +workflow from the bounded fixes retained in PR #2116. The transfer ledgers +remain the authority for retired assertions. The +[confirmed-fixes review](pr-2116-confirmed-fixes.md) records the baseline and +hostile evidence for fixes that this PR now keeps. A deferred finding below is +not closed by passing a neighboring transfer oracle. | Finding | Authority and evidence | Next owner and witness | | --- | --- | --- | -| R29: two valid `Temporal.Duration` values can have the same display text but different fields. A changed field can be suppressed as an equal Collection value. | Changelog #1519 says Duration equality preserves its fields. An earlier PR revision recorded a RED direct `deepEquals` result and a missing change event for a one-second versus 1000-millisecond replacement. The current `origin/main` implementation still normalizes by display text; this narrowed PR has not rerun that RED trace on the current base. | A separate Duration equality owner should model all ten fields independently, then a Collection change-event owner should compare the stored row, callback batch, and subscriber mirror after a valid replacement. | +| R29: two valid `Temporal.Duration` values can have the same display text but different fields. A changed field can be suppressed as an equal Collection value. | Closed in this PR for the tested direct and Collection paths. The Duration field model checks all ten fields; a same-display replacement and adjacent-unit histories make the former `deepEquals` and change-event behavior fail before the repair. The query equality owner and `in` evaluator owner separately check Duration field identity. | General multirow and live-query publication histories remain outside these owners. Add them to the relevant owner if a new failure reaches those paths. | | R16: an indexed Collection can admit a valid Temporal row whose index comparator has no public order, then throw after partial state changes. | Earlier PR revisions recorded RED admission and publication traces for Duration, PlainMonthDay, and mixed-kind rows. Source inspection suggests the risk remains on `origin/main`, but this narrowed PR has not executed a current-base RED witness. The public order law and internal index storage order must be distinguished. | The direct-index and Collection publication owners should pair a valid-row admission history with exact membership, ordered storage, stored rows, and subscriber observations. Choose the smallest internal ordering or atomic-admission repair that satisfies the accepted contract. | -| Mixed-kind indexed order and custom-index query selection may disagree with public query results. | Earlier branch experiments exposed risks, but the exact policy for unlike value kinds and comparator-equal rows is not settled by the retired unit blocks. The user withdrew the added guarantee that each row within one exact-equality class follows its custom callback position. No current-base RED claim is made here. | First state a supported-domain law and run indexed versus scan query witnesses on `origin/main`. Keep a custom comparator witness only if it follows that law without reintroducing the withdrawn per-row guarantee. | +| Mixed-kind indexed order and custom-index query selection may disagree with public query results. | The mixed number/string direct-index history confirms key duplication after deletion on `origin/main`; the new source-Map lane closes that bounded key-conservation bug. A reverse manual index also returned the wrong first row through `currentStateAsChanges`; the new selection oracle closes that bounded public bug. The user withdrew a per-row custom callback position guarantee for comparator-equal exact values. | A general public order policy for unlike kinds remains open. The current owners prove key conservation and the two-rank manual-index selection case, not arbitrary mixed-kind order, custom callback ties, or later live updates. | | R17: an equal-time Date removal left an ordered ghost under added index machinery. | The recorded RED trace depended on code introduced during this PR. It does not establish an `origin/main` defect. The stronger rule for a substituted old row with a different callback position was withdrawn. | If ordinary direct remove/update still has a current-base counterexample, add a source-Map history to the direct-index owner before changing production. | +| PlainDate extended-year order under lexical string collation may diverge between scans and indexes. | On both the recorded `origin/main` baseline and this branch, a direct scan comparator orders `-100000-01-01` before `+010000-01-01`, while direct BasicIndex and BTreeIndex traversal returns the positive date first with `stringSort: 'lexical'`. This was found during hostile review of the final kind classifier, not in the retired unit-test ledger. A public indexed query witness has not run. | Extend the Temporal index-order owner with a scan/indexed public read under lexical collation, confirm the contract for this other Temporal kind, and reproduce RED before a separate repair. The present owner is bounded to ZonedDateTime and time-only Duration. | | Cross-realm native equality was asserted by a surviving request-transport unit test. | PR #1797 captured the then-current `instanceof` result. No public realm-distinctness promise was found. The maintainer questioned that distinction. This PR removes that incidental test expectation and makes no cross-realm equality claim. | A future public contract decision needs a host-boundary witness before either equality policy becomes an oracle law. | Captured `inArray` list-entry refresh and whole-live-query reconstruction were @@ -25,7 +26,7 @@ item, unsigned default byte ordering, and guarantees after callers alter standard objects are also outside the accepted contract. Earlier experiments and RED/GREEN records remain in published commits on -`codex/oracle-test-retirement-2-blocks`; this narrowed source tree does not -retain their production fixes. Statuses in this record distinguish an executed -earlier-branch failure, current-base source inspection, a proposed policy, -and a current-base observation that still needs to be run. +`codex/oracle-test-retirement-2-blocks`. This PR restores the confirmed, +bounded fixes described above. The R16 and R17 rows still distinguish +earlier-branch observations from a current-base failure; neither justifies +new production machinery without a baseline witness under the accepted law. diff --git a/packages/db/package.json b/packages/db/package.json index 81dc7f5217..80d4c5eb59 100644 --- a/packages/db/package.json +++ b/packages/db/package.json @@ -23,7 +23,7 @@ "test": "vitest --run", "test:dist": "vitest run --config vitest.dist.config.ts", "test:facade-retention": "node --expose-gc --import tsx tests/facade-retention.probe.ts", - "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", + "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/duration-query-equality-oracle.test.ts tests/temporal-index-order-oracle.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/custom-index-order-selection-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", "bench:nested-includes": "vitest bench tests/query/includes-performance.bench.ts --run", "test:indexed-db:package": "pnpm --filter @tanstack/db... build && vitest run --config vitest.indexed-db.package.config.ts", "test:indexed-db:browser": "playwright test --config playwright.indexed-db.config.ts", diff --git a/packages/db/src/collection/change-events.ts b/packages/db/src/collection/change-events.ts index f69ab13d43..ec9a2cc359 100644 --- a/packages/db/src/collection/change-events.ts +++ b/packages/db/src/collection/change-events.ts @@ -356,7 +356,7 @@ function getOrderedKeys( // Find the index const index = findIndexForField(collection, fieldPath, compareOpts) - if (index && index.supports(`gt`)) { + if (index && index.supports(`gt`) && index.supportsRangeOptimization) { // Use index optimization const filterFn = (key: TKey): boolean => { const value = collection.get(key) diff --git a/packages/db/src/indexes/auto-index.ts b/packages/db/src/indexes/auto-index.ts index c72c82807a..2098e4f3cc 100644 --- a/packages/db/src/indexes/auto-index.ts +++ b/packages/db/src/indexes/auto-index.ts @@ -21,7 +21,6 @@ export function ensureIndexForField< fieldPath: Array, collection: CollectionImpl, compareOptions?: CompareOptions, - compareFn?: (a: any, b: any) => number, ) { if (hasVirtualPropPath(fieldPath)) { return @@ -69,7 +68,7 @@ export function ensureIndexForField< }, { name: `auto:${fieldPath.join(`.`)}`, - options: { compareFn, compareOptions: compareOpts }, + options: { compareOptions: compareOpts }, }, ) } catch (error) { diff --git a/packages/db/src/indexes/base-index.ts b/packages/db/src/indexes/base-index.ts index b8f3a59bb4..393618dcfd 100644 --- a/packages/db/src/indexes/base-index.ts +++ b/packages/db/src/indexes/base-index.ts @@ -42,6 +42,11 @@ export const builtInIndexResolverNames = new WeakMap() */ export type IndexOperation = (typeof comparisonFunctions)[number] +export interface IndexComparisonOptions { + compareFn?: (a: any, b: any) => number + compareOptions?: CompareOptions +} + /** The read-side surface consumers use on a resolved (possibly reversed) index. */ export type IndexReader = Pick< IndexInterface, @@ -122,7 +127,7 @@ export abstract class BaseIndex< public readonly id: number public readonly name?: string public readonly expression: BasicExpression - public abstract readonly supportedOperations: Set + public readonly supportedOperations: Set protected readonly compareOptions: CompareOptions /** * Orders indexed values. Every result is checked, because a custom @@ -148,9 +153,17 @@ export abstract class BaseIndex< this.compareOptions = options?.compareOptions ?? DEFAULT_COMPARE_OPTIONS this.hasCustomComparator = options?.compareFn != null this.compareFn = makeCheckedComparator( - options?.compareFn ?? makeComparator(this.compareOptions), + options?.compareFn ?? makeComparator(this.compareOptions, `index`), ) this.name = name + this.supportedOperations = new Set([ + `eq`, + `gt`, + `gte`, + `lt`, + `lte`, + `in`, + ]) this.initialize(options) } @@ -181,10 +194,38 @@ export abstract class BaseIndex< ): Array abstract get keyCount(): number abstract equalityLookup(value: any): Set - abstract inArrayLookup(values: Array): Set + protected exactBucketKeys(value: any): Set | undefined { + return this.equalityLookup(value) + } abstract rangeQuery(options: RangeQueryOptions): Set // Common methods + protected lookupSupportedOperation( + operation: IndexOperation, + value: any, + ): Set | undefined { + if (operation === `eq`) return this.equalityLookup(value) + if (operation === `in`) return this.inArrayLookup(value) + if (operation === `gt` || operation === `gte`) + return this.rangeQuery({ + from: value, + fromInclusive: operation === `gte`, + }) + if (operation === `lt` || operation === `lte`) + return this.rangeQuery({ to: value, toInclusive: operation === `lte` }) + return undefined + } + + /** Combine exact buckets without copying each bucket first. */ + inArrayLookup(values: Array): Set { + const result = new Set() + for (const value of values) { + const keys = this.exactBucketKeys(value) + if (keys) keys.forEach((key) => result.add(key)) + } + return result + } + rangeQueryReversed(options: RangeQueryOptions = {}): Set { const { from, to, fromInclusive = true, toInclusive = true } = options const reversed: RangeQueryOptions = {} @@ -295,7 +336,7 @@ export abstract class BaseIndex< return this.compareOptions.direction === direction } - protected abstract initialize(options?: any): void + protected initialize(_options?: any): void {} protected evaluateIndexExpression(item: any): any { const evaluator = (this.compiledIndexEvaluator ??= diff --git a/packages/db/src/indexes/basic-index.ts b/packages/db/src/indexes/basic-index.ts index 0da1e1ffe3..b4ce5ba31f 100644 --- a/packages/db/src/indexes/basic-index.ts +++ b/packages/db/src/indexes/basic-index.ts @@ -10,26 +10,15 @@ import { builtInIndexResolverNames, warnRemovalFailure, } from './base-index.js' -import type { CompareOptions } from '../query/builder/types.js' -import type { IndexOperation } from './base-index.js' +import type { IndexComparisonOptions, IndexOperation } from './base-index.js' +import type { RangeQueryOptions } from './btree-index.js' -/** - * Options for range queries - */ -export interface RangeQueryOptions { - from?: any - to?: any - fromInclusive?: boolean - toInclusive?: boolean -} +export type { RangeQueryOptions } from './btree-index.js' /** * Options for Basic index */ -export interface BasicIndexOptions { - compareFn?: (a: any, b: any) => number - compareOptions?: CompareOptions -} +export type BasicIndexOptions = IndexComparisonOptions /** * Basic index using Map + sorted Array. @@ -44,15 +33,6 @@ export interface BasicIndexOptions { export class BasicIndex< TKey extends string | number = string | number, > extends BaseIndex { - public readonly supportedOperations = new Set([ - `eq`, - `gt`, - `gte`, - `lt`, - `lte`, - `in`, - ]) - // Map for O(1) equality lookups: indexedValue -> Set of PKs private valueMap = new Map>() // Sorted array of unique indexed values for range queries @@ -60,8 +40,6 @@ export class BasicIndex< // Set of all indexed PKs private indexedKeys = new Set() - protected initialize(_options?: BasicIndexOptions): void {} - /** * Adds a value to the index */ @@ -218,35 +196,13 @@ export class BasicIndex< * Performs a lookup operation */ lookup(operation: IndexOperation, value: any): Set { - let result: Set - - switch (operation) { - case `eq`: - result = this.equalityLookup(value) - break - case `gt`: - result = this.rangeQuery({ from: value, fromInclusive: false }) - break - case `gte`: - result = this.rangeQuery({ from: value, fromInclusive: true }) - break - case `lt`: - result = this.rangeQuery({ to: value, toInclusive: false }) - break - case `lte`: - result = this.rangeQuery({ to: value, toInclusive: true }) - break - case `in`: - result = this.inArrayLookup(value) - break - default: - throw new Error( - devBuild() && process.env.NODE_ENV !== `production` - ? `Operation ${operation} not supported by BasicIndex` - : codedMessage(100, { operation }), - ) - } - return result + const result = this.lookupSupportedOperation(operation, value) + if (result !== undefined) return result + throw new Error( + devBuild() && process.env.NODE_ENV !== `production` + ? `Operation ${operation} not supported by BasicIndex` + : codedMessage(100, { operation }), + ) } /** @@ -426,21 +382,8 @@ export class BasicIndex< return result } - /** - * Performs an IN array lookup - O(k) where k is values.length - */ - inArrayLookup(values: Array): Set { - const result = new Set() - - for (const value of values) { - const normalizedValue = normalizeValue(value) - const keys = this.valueMap.get(normalizedValue) - if (keys) { - keys.forEach((key) => result.add(key)) - } - } - - return result + protected exactBucketKeys(value: any): Set | undefined { + return this.valueMap.get(normalizeValue(value)) } } diff --git a/packages/db/src/indexes/btree-index.ts b/packages/db/src/indexes/btree-index.ts index 2266c18a1f..c94f8a0610 100644 --- a/packages/db/src/indexes/btree-index.ts +++ b/packages/db/src/indexes/btree-index.ts @@ -12,16 +12,12 @@ import { builtInIndexResolverNames, warnRemovalFailure, } from './base-index.js' -import type { CompareOptions } from '../query/builder/types.js' -import type { IndexOperation } from './base-index.js' +import type { IndexComparisonOptions, IndexOperation } from './base-index.js' /** * Options for Ordered index */ -export interface BTreeIndexOptions { - compareFn?: (a: any, b: any) => number - compareOptions?: CompareOptions -} +export type BTreeIndexOptions = IndexComparisonOptions /** * Options for range queries @@ -46,15 +42,6 @@ type OrderedBucket = { export class BTreeIndex< TKey extends string | number = string | number, > extends BaseIndex { - public readonly supportedOperations = new Set([ - `eq`, - `gt`, - `gte`, - `lt`, - `lte`, - `in`, - ]) - // Internal data structures - private to hide implementation details // The `orderedEntries` B+ tree groups values that occupy the same comparator // position. The `valueMap` keeps exact values separate for equality lookups. @@ -73,8 +60,6 @@ export class BTreeIndex< return this.compareFn(denormalizeUndefined(a), denormalizeUndefined(b)) } - protected initialize(_options?: BTreeIndexOptions): void {} - /** * Adds a value to the index */ @@ -224,35 +209,13 @@ export class BTreeIndex< * Performs a lookup operation */ lookup(operation: IndexOperation, value: any): Set { - let result: Set - - switch (operation) { - case `eq`: - result = this.equalityLookup(value) - break - case `gt`: - result = this.rangeQuery({ from: value, fromInclusive: false }) - break - case `gte`: - result = this.rangeQuery({ from: value, fromInclusive: true }) - break - case `lt`: - result = this.rangeQuery({ to: value, toInclusive: false }) - break - case `lte`: - result = this.rangeQuery({ to: value, toInclusive: true }) - break - case `in`: - result = this.inArrayLookup(value) - break - default: - throw new Error( - devBuild() && process.env.NODE_ENV !== `production` - ? `Operation ${operation} not supported by BTreeIndex` - : codedMessage(101, { operation }), - ) - } - return result + const result = this.lookupSupportedOperation(operation, value) + if (result !== undefined) return result + throw new Error( + devBuild() && process.env.NODE_ENV !== `production` + ? `Operation ${operation} not supported by BTreeIndex` + : codedMessage(101, { operation }), + ) } /** @@ -409,21 +372,8 @@ export class BTreeIndex< return this.takeInternal(n, nextPair, undefined, filterFn, true) } - /** - * Performs an IN array lookup - */ - inArrayLookup(values: Array): Set { - const result = new Set() - - for (const value of values) { - const normalizedValue = normalizeForBTree(value) - const keys = this.valueMap.get(normalizedValue)?.keys - if (keys) { - keys.forEach((key) => result.add(key)) - } - } - - return result + protected exactBucketKeys(value: any): Set | undefined { + return this.valueMap.get(normalizeForBTree(value))?.keys } } diff --git a/packages/db/src/query/compiler/order-by.ts b/packages/db/src/query/compiler/order-by.ts index 88c0777f6c..47004990e4 100644 --- a/packages/db/src/query/compiler/order-by.ts +++ b/packages/db/src/query/compiler/order-by.ts @@ -189,19 +189,11 @@ export function processOrderBy( const compareOpts = buildCompareOptions(firstClause, collection) if (fieldName) { - // Use a single-column comparator for the index, not the - // multi-column `compare` function. The multi-column comparator - // expects array values [col1, col2, ...] but the index stores - // individual field values. Passing `compare` here causes the - // BTree to treat all single values as equal (since number[0] - // === undefined for both sides of the comparison). - const firstColumnCompareFn = makeComparator(compareOpts) ensureIndexForField( fieldName, followRefResult.path, followRefCollection, compareOpts, - firstColumnCompareFn, ) } diff --git a/packages/db/src/utils.ts b/packages/db/src/utils.ts index e472470ed2..0c6cd53d01 100644 --- a/packages/db/src/utils.ts +++ b/packages/db/src/utils.ts @@ -9,6 +9,20 @@ interface TypedArray { [index: number]: number } +export const durationFields = [ + `years`, + `months`, + `weeks`, + `days`, + `hours`, + `minutes`, + `seconds`, + `milliseconds`, + `microseconds`, + `nanoseconds`, +] as const +type DurationFields = Record<(typeof durationFields)[number], number> + /** * Deep equality function that compares two values recursively * Handles primitives, objects, arrays, Date, RegExp, Map, Set, TypedArrays, and Temporal objects @@ -263,6 +277,16 @@ export function deepEqualsInternal( // If they're different Temporal types, they're not equal if (aTag !== bTag) return false + // Duration identity preserves its fields, even when toString normalizes + // different units to the same elapsed time. + if (aTag === `Temporal.Duration`) { + const durationA = a as unknown as DurationFields + const durationB = b as unknown as DurationFields + return durationFields.every( + (field) => durationA[field] === durationB[field], + ) + } + // Use Temporal's built-in equals method if available if (typeof a.equals === `function`) { return a.equals(b) diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index 367f4e25f1..ec0c4ed344 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -1,23 +1,8 @@ -import { isTemporal } from '../utils' +import { durationFields, isTemporal } from '../utils' import { getRuntimeReferenceIdentity } from '../query/runtime-reference-identity' import { codedMessage, devBuild } from '../error-message' import type { CompareOptions } from '../query/builder/types' -// WeakMap to store stable IDs for objects -const objectIds = new WeakMap() -let nextObjectId = 1 - -/** - * Get or create a stable ID for an object - */ -function getObjectId(obj: object): number { - const existing = objectIds.get(obj) - if (existing !== undefined) return existing - const id = nextObjectId++ - objectIds.set(obj, id) - return id -} - /** * Whether a value has no IEEE-754 natural order: `NaN`, or an invalid Date * (whose timestamp is `NaN`). The query engine follows PostgreSQL float @@ -106,10 +91,10 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { const bIsObject = typeof b === `object` if (aIsObject || bIsObject) { - // If both are objects, compare their stable IDs + // Symbols and objects share the runtime's reference-identity sequence. if (aIsObject && bIsObject) { - const aId = getObjectId(a) - const bId = getObjectId(b) + const aId = getRuntimeReferenceIdentity(a)[2] + const bId = getRuntimeReferenceIdentity(b)[2] return aId - bId } @@ -141,8 +126,44 @@ export const descComparator = ( export function makeComparator( opts: CompareOptions, + scope: `values` | `index` = `values`, ): (a: any, b: any) => number { + const direction = opts.direction === `asc` ? 1 : -1 return (a, b) => { + // Index storage needs a transitive order across primitive kinds. Native + // coercion can make 2 tie "2", yet put them on opposite sides of "10". + if (scope === `index` && a != null && b != null && typeof a !== typeof b) { + const order = typeof a < typeof b ? -1 : 1 + return order * direction + } + if (scope === `index` && typeof a === `string` && typeof b === `string`) { + const aKind = indexStringKind(a) + const bKind = indexStringKind(b) + if (aKind !== bKind) { + const order = aKind < bKind ? -1 : 1 + return order * direction + } + if (aKind === `string`) { + a = a.startsWith(ESCAPED_STRING_PREFIX) + ? a.slice(ESCAPED_STRING_PREFIX.length) + : a + b = b.startsWith(ESCAPED_STRING_PREFIX) + ? b.slice(ESCAPED_STRING_PREFIX.length) + : b + } else { + if (aKind === ZONED_EPOCH_PREFIX || aKind === DURATION_TIME_PREFIX) { + // Exact index keys retain identity; these ordinals define order ties. + const aOrdinal = BigInt( + a.slice(aKind.length, a.indexOf(`:`, aKind.length)), + ) + const bOrdinal = BigInt( + b.slice(aKind.length, b.indexOf(`:`, aKind.length)), + ) + const order = aOrdinal < bOrdinal ? -1 : aOrdinal > bOrdinal ? 1 : 0 + return order * direction + } + } + } if (opts.direction === `asc`) { return ascComparator(a, b, opts) } else { @@ -199,6 +220,24 @@ function areUint8ArraysEqual(a: Uint8Array, b: Uint8Array): boolean { } const NORMALIZED_KEY_PREFIX = `\u0000tanstack-db:` +const ESCAPED_STRING_PREFIX = `${NORMALIZED_KEY_PREFIX}string:` +const TEMPORAL_PREFIX = `${NORMALIZED_KEY_PREFIX}temporal:` +const ZONED_EPOCH_PREFIX = `${NORMALIZED_KEY_PREFIX}temporal:Temporal.ZonedDateTime:` +const DURATION_TIME_PREFIX = `${NORMALIZED_KEY_PREFIX}temporal:Temporal.Duration:time:` + +function indexStringKind(value: string): string { + if ( + !value.startsWith(NORMALIZED_KEY_PREFIX) || + value.startsWith(ESCAPED_STRING_PREFIX) + ) { + return `string` + } + if (value.startsWith(DURATION_TIME_PREFIX)) return DURATION_TIME_PREFIX + const afterKind = value.startsWith(TEMPORAL_PREFIX) + ? TEMPORAL_PREFIX.length + : NORMALIZED_KEY_PREFIX.length + return value.slice(0, value.indexOf(`:`, afterKind) + 1) +} function normalizedKey(kind: string, value: string): string { return `${NORMALIZED_KEY_PREFIX}${kind}:${value}` @@ -212,6 +251,20 @@ function normalizeBinary(value: Uint8Array): string { return normalizedKey(`binary`, bytes) } +type DurationFields = Record<(typeof durationFields)[number], number> + +function durationTimeNanoseconds(value: DurationFields): bigint { + return ( + BigInt(value.days) * 86_400_000_000_000n + + BigInt(value.hours) * 3_600_000_000_000n + + BigInt(value.minutes) * 60_000_000_000n + + BigInt(value.seconds) * 1_000_000_000n + + BigInt(value.milliseconds) * 1_000_000n + + BigInt(value.microseconds) * 1_000n + + BigInt(value.nanoseconds) + ) +} + /** * Sentinel value representing undefined in normalized form. * This allows distinguishing between "start from beginning" (undefined parameter) @@ -243,10 +296,22 @@ export function normalizeValue(value: any): any { } if (isTemporal(value)) { - return normalizedKey( - `temporal`, - `${value[Symbol.toStringTag]}:${value.toString()}`, - ) + const tag = value[Symbol.toStringTag] + if (tag === `Temporal.Duration`) { + const fields = value as unknown as DurationFields + const exact = durationFields.map((field) => fields[field]).join(`,`) + const order = + fields.years || fields.months || fields.weeks + ? `` + : `time:${durationTimeNanoseconds(fields)}:` + return normalizedKey(`temporal`, `${tag}:${order}${exact}`) + } + if (tag === `Temporal.ZonedDateTime`) { + const epoch = (value as unknown as { epochNanoseconds: bigint }) + .epochNanoseconds + return normalizedKey(`temporal`, `${tag}:${epoch}:${value.toString()}`) + } + return normalizedKey(`temporal`, `${tag}:${value.toString()}`) } // Normalize Uint8Arrays/Buffers to a string representation for Map key usage diff --git a/packages/db/tests/change-event-history-oracle.test.ts b/packages/db/tests/change-event-history-oracle.test.ts index 005b450e22..719cdb404b 100644 --- a/packages/db/tests/change-event-history-oracle.test.ts +++ b/packages/db/tests/change-event-history-oracle.test.ts @@ -1,4 +1,5 @@ import { fc, test as fcTest } from '@fast-check/vitest' +import { Temporal } from 'temporal-polyfill' import { describe, expect, it, vi } from 'vitest' import { BTreeIndex, @@ -572,6 +573,89 @@ async function verifyGeneratedHistory({ } describe(`change-event history oracle`, () => { + // Published #1519 keeps equivalent Duration forms distinct for query `eq`. + // Under #1901's change-message reconstruction law, a source replacement + // that changes the Duration's intrinsic fields must reach the consumer's + // mirror as well as the public row. This two-step history inserts one second, + // then replaces it with 1000 milliseconds. The reference Map applies that + // valid write despite the shared display text; at the update commit, both + // public observations must reflect the second value. + // This witness covers ordinary same-realm Durations, not altered objects or + // a requirement to suppress events for a separately allocated equal copy. + it(`publishes a Duration field change to the Collection and change-message consumer`, async () => { + interface DurationRow extends Record { + id: number + duration: Temporal.Duration + } + let sync!: Parameters[`sync`]>[0] + const collection = createCollection({ + id: `duration-field-change-history`, + getKey: (row) => row.id, + startSync: true, + sync: { + sync: (actions) => { + sync = actions + actions.markReady() + }, + }, + }) + + await runWithCleanup( + async () => { + await collection.stateWhenReady() + const first = { + id: 1, + duration: Temporal.Duration.from({ seconds: 1 }), + } + const second = { + id: 1, + duration: Temporal.Duration.from({ milliseconds: 1000 }), + } + expect(first.duration.toString()).toBe(second.duration.toString()) + sync.begin() + sync.write({ type: `insert`, value: first }) + expect(sync.commit()).toBe(true) + + // The model applies each valid source write to a plain keyed Map. + // A subscriber maintains an independent mirror from public messages. + const expected = new Map([[1, first]]) + const mirror = new Map(expected) + const received: Array> = [] + const subscription = collection.subscribeChanges( + (changes) => { + received.push(...changes) + for (const change of changes) { + if (change.type === `delete`) mirror.delete(change.key) + else mirror.set(change.key, change.value) + } + }, + { includeInitialState: false }, + ) + try { + expected.set(1, second) + sync.begin() + sync.write({ type: `update`, value: second }) + expect(sync.commit()).toBe(true) + + const expectedFields = [0, 1000] + const fields = (row: DurationRow | undefined) => [ + row?.duration.seconds, + row?.duration.milliseconds, + ] + expect(fields(expected.get(1))).toEqual(expectedFields) + expect(fields(collection.state.get(1))).toEqual(expectedFields) + expect(received.map((message) => message.type)).toEqual([`update`]) + expect(fields(received[0]?.previousValue)).toEqual([1, 0]) + expect(fields(received[0]?.value)).toEqual(expectedFields) + expect(fields(mirror.get(1))).toEqual(expectedFields) + } finally { + subscription.unsubscribe() + } + }, + () => collection.cleanup(), + ) + }) + it(`preserves each key's causal messages within a deferred callback`, async () => { const rounds: ReadonlyArray> = [ [ diff --git a/packages/db/tests/duration-query-equality-oracle.test.ts b/packages/db/tests/duration-query-equality-oracle.test.ts new file mode 100644 index 0000000000..cc9c89becf --- /dev/null +++ b/packages/db/tests/duration-query-equality-oracle.test.ts @@ -0,0 +1,173 @@ +/** + * Query equality preserves each of a Temporal.Duration's ten fields. Changelog + * #1519 explicitly keeps equivalent Duration forms unequal under `eq`: one + * second and 1000 milliseconds display alike but have different fields. + * + * The independent model reads the ten public fields rather than formatting a + * Duration or importing query normalization. The fixed input grammar inserts + * one ordinary Duration row, then probes an equal copy and a changed value + * for every field. Three further probes distinguish a same-display unit + * change, opposite signs, and a changed second nonzero field. + * Each direct BasicIndex/BTreeIndex exact lookup and each public Collection + * `eq` read before and after indexing must return the model's exact key set at + * that read. An index spy confirms the indexed public route ran. These checks + * make no claim about Duration ordering, other Temporal kinds, mutations, or + * live subscriber publication. + */ +import { expect, it, vi } from 'vitest' +import { Temporal } from 'temporal-polyfill' +import { createCollection } from '../src/collection/index.js' +import { BasicIndex } from '../src/indexes/basic-index.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { eq } from '../src/query/builder/functions.js' +import { PropRef } from '../src/query/ir.js' +import { mockSyncCollectionOptions } from './utils.js' + +const fields = [ + `years`, + `months`, + `weeks`, + `days`, + `hours`, + `minutes`, + `seconds`, + `milliseconds`, + `microseconds`, + `nanoseconds`, +] as const + +function modelEqual( + left: Temporal.Duration, + right: Temporal.Duration, +): boolean { + return fields.every((field) => left[field] === right[field]) +} + +type Case = { + name: string + stored: Temporal.Duration + probe: Temporal.Duration +} + +const sameDisplay: Case = { + name: `seconds versus milliseconds with identical display`, + stored: Temporal.Duration.from({ seconds: 1 }), + probe: Temporal.Duration.from({ milliseconds: 1000 }), +} +const oppositeSigns: Case = { + name: `opposite signs`, + stored: Temporal.Duration.from({ seconds: 1 }), + probe: Temporal.Duration.from({ seconds: -1 }), +} +const secondFieldChange: Case = { + name: `same first nonzero field with a changed second field`, + stored: Temporal.Duration.from({ hours: 1, minutes: 2 }), + probe: Temporal.Duration.from({ hours: 1, minutes: 3 }), +} + +const cases: ReadonlyArray = [ + ...fields.flatMap((field) => { + const stored = Temporal.Duration.from({ [field]: 1 }) + return [ + { + name: `${field} equal copy`, + stored, + probe: Temporal.Duration.from({ [field]: 1 }), + }, + { + name: `${field} changed`, + stored, + probe: Temporal.Duration.from({ [field]: 2 }), + }, + ] + }), + sameDisplay, + oppositeSigns, + secondFieldChange, +] + +function modelKeys({ stored, probe }: Case): Set { + return new Set(modelEqual(stored, probe) ? [`row`] : []) +} + +const indexTypes = [ + [`BasicIndex`, BasicIndex], + [`BTreeIndex`, BTreeIndex], +] as const + +// Every field changes the model's answer, while a separately constructed +// equal copy does not. Three pairs kill text, absolute-value, and first-field +// key mutants at the same exact-key-set comparison used below. +it(`calibrates all ten fields and three plausible wrong keys`, () => { + for (const [equal, changed] of Array.from({ length: 10 }, (_, index) => + cases.slice(index * 2, index * 2 + 2), + )) { + expect(modelEqual(equal!.stored, equal!.probe)).toBe(true) + expect(modelEqual(changed!.stored, changed!.probe)).toBe(false) + } + expect(String(sameDisplay.stored)).toBe(String(sameDisplay.probe)) + expect(modelEqual(sameDisplay.stored, sameDisplay.probe)).toBe(false) + const magnitudeKey = (value: Temporal.Duration) => + fields.map((field) => Math.abs(value[field])).join(`,`) + const firstNonzeroKey = (value: Temporal.Duration) => { + const field = fields.find((candidate) => value[candidate] !== 0)! + return `${field}:${value[field]}` + } + const wrongKeys = ( + testCase: Case, + key: (value: Temporal.Duration) => string, + ) => new Set(key(testCase.stored) === key(testCase.probe) ? [`row`] : []) + for (const [testCase, key] of [ + [sameDisplay, String], + [oppositeSigns, magnitudeKey], + [secondFieldChange, firstNonzeroKey], + ] as const) { + expect(() => + expect(wrongKeys(testCase, key)).toEqual(modelKeys(testCase)), + ).toThrow() + } +}) + +it.each(indexTypes)( + `%s exact lookup follows Duration fields`, + (_name, IndexType) => { + for (const { name, stored, probe } of cases) { + const index = new IndexType(1, new PropRef([`value`])) + index.add(`row`, { value: stored }) + expect(index.lookup(`eq`, probe), name).toEqual( + modelKeys({ name, stored, probe }), + ) + } + }, +) + +it.each(indexTypes)( + `public eq agrees before and after %s indexing`, + async (_name, IndexType) => { + for (const { name, stored, probe } of cases) { + const source = createCollection( + mockSyncCollectionOptions({ + id: `duration-query-equality-${name}`, + getKey: (row: { id: string; value: Temporal.Duration }) => row.id, + initialData: [{ id: `row`, value: stored }], + autoIndex: `off`, + defaultIndexType: IndexType, + }), + ) + try { + await source.stateWhenReady() + const where = eq(new PropRef([`value`]), probe) + const expected = [...modelKeys({ name, stored, probe })] + const readKeys = () => + source.currentStateAsChanges({ where })!.map((change) => change.key) + expect(readKeys(), `${name}: scan`).toEqual(expected) + const index = source.createIndex((row) => row.value) + const lookup = vi.spyOn(index, `lookup`) + expect(readKeys(), `${name}: indexed`).toEqual(expected) + expect(lookup, `${name}: index path`).toHaveBeenCalled() + } finally { + await source.cleanup() + } + } + }, +) diff --git a/packages/db/tests/index-update-oracle.property.test.ts b/packages/db/tests/index-update-oracle.property.test.ts index 8270f6214d..90ff363935 100644 --- a/packages/db/tests/index-update-oracle.property.test.ts +++ b/packages/db/tests/index-update-oracle.property.test.ts @@ -1,15 +1,22 @@ import { describe, expect, expectTypeOf, test, vi } from 'vitest' import { fc } from '@fast-check/vitest' +import { Temporal } from 'temporal-polyfill' import { createCollection } from '../src/collection/index.js' import { CollectionInErrorStateError } from '../src/errors.js' import { BasicIndex } from '../src/indexes/basic-index.js' import { BTreeIndex } from '../src/indexes/btree-index.js' import { PropRef } from '../src/query/ir.js' import { DEFAULT_COMPARE_OPTIONS } from '../src/utils.js' -import { indexedKeysSet, orderedEntriesArray, valueMapData } from './utils' +import { + indexedKeysSet, + mockSyncCollectionOptions, + orderedEntriesArray, + valueMapData, +} from './utils' import { oraclePropertyOptions, oracleRuns } from './oracle-config' import type { BaseIndex, IndexInterface } from '../src/indexes/base-index.js' import type { CompareOptions } from '../src/query/builder/types.js' +import type { StringCollationConfig } from '../src/types.js' /** * An index is a derived multimap from indexed value to source keys. @@ -40,6 +47,9 @@ import type { CompareOptions } from '../src/query/builder/types.js' * A build is one of those direct actions even when a source value is * `undefined`: default nulls-first order must hold after build just as it * does after incremental add. The nullish lane below exercises both paths. + * A separate mixed number/string lane checks key conservation only. A full + * ordered read and an indexed public snapshot must contain each live source + * key once, even when unlike values have no declared relative sort order. */ type IndexValue = number @@ -2160,7 +2170,7 @@ const collationCases = [ ] as const function createCollationCollection( - collation: (typeof collationCases)[number][`collation`], + collation: StringCollationConfig, autoIndex: `off` | `eager`, defaultIndexType?: typeof BasicIndex | typeof BTreeIndex, labels: ReadonlyArray = [`item10`, `item2`], @@ -2174,8 +2184,8 @@ function createCollationCollection( sync: { sync: ({ begin, write, commit, markReady }) => { begin() - for (const label of labels) { - write({ type: `insert`, value: { id: label, label } }) + for (const [ordinal, label] of labels.entries()) { + write({ type: `insert`, value: { id: String(ordinal), label } }) } commit() markReady() @@ -2281,3 +2291,251 @@ test(`ordered scan resolves inherited collation outside each comparison`, async expect(collection.indexes.size).toBe(0) }) }) + +/** + * Public ordered reads compare the user's string values. An index escapes a + * leading internal prefix for exact-key storage; that storage encoding must + * not change the Collection's declared string order. The en-US locale and a + * custom callback both place the prefixed value first. The scan and the + * bounded indexed read must return that same first row. + */ +const prefixedLabel = `\u0000tanstack-db:a` +const plainLabel = `tanstack-db:r` +const prefixedStringCases = [ + { + name: `locale`, + collation: { stringSort: `locale`, locale: `en-US` }, + }, + { + name: `custom collation`, + collation: { + stringSort: `custom`, + compare: (left: string, right: string) => + left.localeCompare(right, `en-US`), + }, + }, +] as const satisfies ReadonlyArray<{ + name: string + collation: StringCollationConfig +}> + +describe.each([ + [`BasicIndex`, BasicIndex], + [`BTreeIndex`, BTreeIndex], +] as const)(`%s prefixed-string public order`, (_name, IndexType) => { + test.each(prefixedStringCases)( + `$name scan and bounded index return the same first row`, + async ({ collation }) => { + const labels = [plainLabel, prefixedLabel] + expect(prefixedLabel.localeCompare(plainLabel, `en-US`)).toBeLessThan(0) + const orderBy = [ + { + expression: new PropRef([`label`]), + compareOptions: { + direction: `asc` as const, + nulls: `first` as const, + }, + }, + ] + + const scan = createCollationCollection( + collation, + `off`, + undefined, + labels, + ) + await withCollectionCleanup(scan, async () => { + await scan.stateWhenReady() + const changes = scan.currentStateAsChanges({ orderBy, limit: 1 }) + expect(changes?.map(({ value }) => value.label)).toEqual([ + prefixedLabel, + ]) + expect(scan.indexes.size, `scan path reached`).toBe(0) + }) + + const indexed = createCollationCollection( + collation, + `eager`, + IndexType, + labels, + ) + await withCollectionCleanup(indexed, async () => { + await indexed.stateWhenReady() + const changes = indexed.currentStateAsChanges({ + orderBy, + limit: 1, + optimizedOnly: true, + }) + expect(changes?.map(({ value }) => value.label)).toEqual([ + prefixedLabel, + ]) + expect(indexed.indexes.size, `index path reached`).toBe(1) + }) + }, + ) +}) + +/** + * Mixed primitive values can share a comparator position without sharing an + * exact value. The source Map owns each key once. This model intentionally + * predicts membership and multiplicity, not the relative order of a number + * and a string. Its fixed grammar adds 2, "2", and "10", then removes each + * possible owner in a fresh history. A second history deletes, updates across + * value kinds, then re-adds an earlier key. A third history adds two ordered + * Temporal values beside an ordinary string that resembles an internal key, + * then deletes one Temporal owner. After each returned action, both full index + * traversals must contain exactly the live source keys. + * + * In the public lane a ready Collection receives the same rows and deletes + * the number. A manual index serves a bounded `currentStateAsChanges` read + * with room for one more row than the source contains. The returned change + * messages must contain each live key once. `optimizedOnly` and the take spy + * establish that this observation came through the index. No mixed-kind + * `orderBy` sequence, custom callback, or modified built-in is claimed. + */ +const mixedRows: ReadonlyArray<{ id: string; value: number | string }> = [ + { id: `a`, value: 2 }, + { id: `b`, value: `2` }, + { id: `c`, value: `10` }, +] + +function expectConservedIndexKeys( + index: BaseIndex, + rows: ReadonlyMap, +): void { + const expected = new Set(rows.keys()) + expect(index.keyCount).toBe(expected.size) + for (const keys of [ + index.takeFromStart(rows.size + 1), + index.takeReversedFromEnd(rows.size + 1), + ]) { + expect(keys).toHaveLength(expected.size) + expect(new Set(keys)).toEqual(expected) + } +} + +describe.each(indexTypes)( + `%s mixed-value key conservation`, + (_indexName, IndexType) => { + test.each(mixedRows)( + `keeps every surviving key once after deleting $id`, + (removed) => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + for (const row of mixedRows) { + index.add(row.id, row) + rows.set(row.id, row.value) + expectConservedIndexKeys(index, rows) + } + index.remove(removed.id, removed) + rows.delete(removed.id) + expectConservedIndexKeys(index, rows) + }, + ) + + test(`keeps one copy of each key after a mixed-value delete, update, and add`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + const add = (id: string, value: number | string) => { + index.add(id, { id, value }) + rows.set(id, value) + expectConservedIndexKeys(index, rows) + } + + add(`a`, 2) + add(`b`, `2`) + add(`d`, `10`) + index.remove(`a`, { id: `a`, value: 2 }) + rows.delete(`a`) + expectConservedIndexKeys(index, rows) + index.update(`b`, { id: `b`, value: `2` }, { id: `b`, value: 2 }) + rows.set(`b`, 2) + expectConservedIndexKeys(index, rows) + add(`a`, `2`) + }) + + test.each([ + { + name: `ZonedDateTime`, + first: + Temporal.Instant.fromEpochNanoseconds(9n).toZonedDateTimeISO(`UTC`), + second: + Temporal.Instant.fromEpochNanoseconds(10n).toZonedDateTimeISO(`UTC`), + text: `tanstack-db:temporal:Temporal.ZonedDateTime:5:fake`, + }, + { + name: `Duration`, + first: Temporal.Duration.from({ nanoseconds: 9 }), + second: Temporal.Duration.from({ nanoseconds: 10 }), + text: `tanstack-db:temporal:Temporal.Duration:time:5:fake`, + }, + ])( + `keeps $name and ordinary-string owners after deletion`, + ({ first, second, text }) => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + for (const [id, value] of [ + [`a`, first], + [`b`, second], + [`c`, text], + ] as const) { + index.add(id, { id, value }) + rows.set(id, value) + expectConservedIndexKeys(index, rows) + } + index.remove(`b`, { id: `b`, value: second }) + rows.delete(`b`) + expectConservedIndexKeys(index, rows) + }, + ) + }, +) + +describe.each([ + [`BasicIndex`, BasicIndex], + [`BTreeIndex`, BTreeIndex], +] as const)(`%s mixed-value public snapshot`, (_indexName, IndexType) => { + test(`returns each surviving key once after a source deletion`, async () => { + const options = mockSyncCollectionOptions<{ + id: string + value: number | string + }>({ + id: `mixed-value-public-snapshot-${IndexType.name}`, + getKey: (row) => row.id, + initialData: mixedRows.map((row) => ({ ...row })), + autoIndex: `off`, + }) + const collection = createCollection(options) + await withCollectionCleanup(collection, async () => { + await collection.stateWhenReady() + const index = collection.createIndex((row) => row.value, { + indexType: IndexType, + }) + const take = vi.spyOn(index, `takeFromStart`) + const rows = new Map(mixedRows.map((row) => [row.id, row.value])) + + options.utils.begin() + options.utils.write({ type: `delete`, value: mixedRows[0]! }) + options.utils.commit() + rows.delete(mixedRows[0]!.id) + + take.mockClear() + + const changes = collection.currentStateAsChanges({ + orderBy: [ + { + expression: new PropRef([`value`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + limit: rows.size + 1, + optimizedOnly: true, + }) + expect(take).toHaveBeenCalled() + expect(changes).toBeDefined() + const keys = changes!.map((change) => change.key) + expect(keys).toHaveLength(rows.size) + expect(new Set(keys)).toEqual(new Set(rows.keys())) + }) + }) +}) diff --git a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts index f5ea6b05cb..bca8b0ed6e 100644 --- a/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts +++ b/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts @@ -19,7 +19,8 @@ import { oraclePropertyOptions, oracleRuns } from '../../oracle-config.js' * - A valid Date equals its millisecond timestamp, and another Date with the * same timestamp. * - Two Uint8Arrays (Buffers included) are equal when their bytes are equal. - * - A Temporal value equals a Temporal value of the same kind and string. + * - A Temporal value equals one of the same kind and string, except Duration: + * its ten fields determine equality even when two forms display alike. * - Every other pair compares with `===`, so `-0` equals `0`, `1n` does not * equal `1`, and objects compare by reference. * @@ -73,6 +74,9 @@ const pool: ReadonlyArray = [ Temporal.PlainDate.from(`2024-01-01`), Temporal.PlainDate.from(`2024-01-01`), Temporal.PlainTime.from(`10:00`), + Temporal.Duration.from({ seconds: 1 }), + Temporal.Duration.from({ seconds: 1 }), + Temporal.Duration.from({ milliseconds: 1000 }), shared, { id: 1 }, null, @@ -111,6 +115,19 @@ function asComparable(value: unknown): unknown { return value instanceof Date ? value.getTime() : value } +const durationFields = [ + `years`, + `months`, + `weeks`, + `days`, + `hours`, + `minutes`, + `seconds`, + `milliseconds`, + `microseconds`, + `nanoseconds`, +] as const + function referenceEqual(left: unknown, right: unknown): boolean { if (isNotANumber(left) || isNotANumber(right)) { return isNotANumber(left) && isNotANumber(right) @@ -123,6 +140,13 @@ function referenceEqual(left: unknown, right: unknown): boolean { Array.from(left).every((byte, index) => byte === right[index]) ) } + if (left instanceof Temporal.Duration || right instanceof Temporal.Duration) { + return ( + left instanceof Temporal.Duration && + right instanceof Temporal.Duration && + durationFields.every((field) => left[field] === right[field]) + ) + } if (isTemporalValue(left) || isTemporalValue(right)) { return ( isTemporalValue(left) && @@ -219,6 +243,18 @@ describe(`in evaluator`, () => { [`equal bytes`, Buffer.from([1, 2]), [new Uint8Array([1, 2])], true], [`bytes and their internal key`, PREFIX, [new Uint8Array([])], false], [`a structural copy`, { id: 1 }, [shared], false], + [ + `equal Duration fields`, + Temporal.Duration.from({ seconds: 1 }), + [Temporal.Duration.from({ seconds: 1 })], + true, + ], + [ + `equivalent Duration forms with different fields`, + Temporal.Duration.from({ seconds: 1 }), + [Temporal.Duration.from({ milliseconds: 1000 })], + false, + ], [`an empty list`, `a`, [], false], ] as const)(`pins %s`, (_name, value, list, expected) => { const testCase = { value, list: [...list], listFromRow: false } @@ -240,4 +276,18 @@ describe(`in evaluator`, () => { ), ).toThrow(/in mismatch/) }) + + it(`rejects display-text equality for equivalent Duration forms`, () => { + const seconds = Temporal.Duration.from({ seconds: 1 }) + const milliseconds = Temporal.Duration.from({ milliseconds: 1000 }) + const textual: Evaluate = ({ value, list }) => + list.some((entry) => String(entry) === String(value)) + expect(() => + check(textual, { + value: seconds, + list: [milliseconds], + listFromRow: false, + }), + ).toThrow(/in mismatch/) + }) }) diff --git a/packages/db/tests/query/custom-index-order-selection-oracle.test.ts b/packages/db/tests/query/custom-index-order-selection-oracle.test.ts new file mode 100644 index 0000000000..67b7196f5c --- /dev/null +++ b/packages/db/tests/query/custom-index-order-selection-oracle.test.ts @@ -0,0 +1,84 @@ +/** + * An index may choose how to store rows, but it cannot change a query's + * declared order. For a bounded `orderBy(rank)` query, the public window is + * the ascending ranks from the source rows, regardless of a manual index's + * callback. The live-query guide defines `orderBy` and `limit`; the live-query + * architecture treats index selection as physical planning. + * + * The Collection API also exposes an ordered, limited direct snapshot through + * `currentStateAsChanges`. The model sorts a plain array by numeric rank and + * slices it. Its two distinct ranks avoid any promise about callback ordering + * within an equality class. The legal history starts one ready, eager source, + * then reads a direct snapshot and compiles one limited live query. A scan and + * a source with a reverse-order BTree index exercise both public paths. The + * exact public row IDs must equal the model at the direct-read and post-preload + * checkpoints. The reverse index is the hostile planner control: taking its + * first entry as the first query row would expose rank 30 instead of rank 10. + * This witness does not cover on-demand acquisition or later source changes. + */ +import { expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' +import { createLiveQueryCollection } from '../../src/query/live-query-collection.js' +import { PropRef } from '../../src/query/ir.js' +import { mockSyncCollectionOptions } from '../utils.js' + +type Row = { id: number; rank: number } + +const rows: ReadonlyArray = [ + { id: 30, rank: 30 }, + { id: 10, rank: 10 }, +] + +// The query's rank order comes from its declaration, not index traversal. +const expectedIds = [...rows] + .sort((a, b) => a.rank - b.rank) + .slice(0, 1) + .map((row) => row.id) + +it.each([`scan`, `reverse manual index`] as const)( + `keeps bounded public rank order with %s`, + async (path) => { + const source = createCollection( + mockSyncCollectionOptions({ + id: `custom-index-order-selection-${path}`, + getKey: (row) => row.id, + initialData: [...rows], + autoIndex: `off`, + }), + ) + let live: ReturnType | undefined + try { + await source.stateWhenReady() + if (path === `reverse manual index`) { + source.createIndex((row) => row.rank, { + indexType: BTreeIndex, + options: { compareFn: (a: number, b: number) => b - a }, + }) + } + + const direct = source.currentStateAsChanges({ + orderBy: [ + { + expression: new PropRef([`rank`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + limit: 1, + })! + expect(direct.map((change) => change.value.id)).toEqual(expectedIds) + + live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + await live.preload() + expect([...live.values()].map((row) => row.id)).toEqual(expectedIds) + } finally { + await live?.cleanup() + await source.cleanup() + } + }, +) diff --git a/packages/db/tests/temporal-index-order-oracle.test.ts b/packages/db/tests/temporal-index-order-oracle.test.ts new file mode 100644 index 0000000000..b4d78974a8 --- /dev/null +++ b/packages/db/tests/temporal-index-order-oracle.test.ts @@ -0,0 +1,458 @@ +import { Temporal } from 'temporal-polyfill' +import { describe, expect, it, vi } from 'vitest' +import { BasicIndex } from '../src/indexes/basic-index' +import { BTreeIndex } from '../src/indexes/btree-index' +import { createCollection } from '../src/collection/index' +import { createLiveQueryCollection } from '../src/query/live-query-collection' +import { PropRef } from '../src/query/ir' +import { mockSyncCollectionOptions } from './utils' +import type { CompareOptions } from '../src/query/builder/types' + +/** + * # Does an index preserve the public order of valid ZonedDateTimes? + * + * Changelog #1519 promises instant order for ZonedDateTime `orderBy`. The + * supplied instants below put `early` first, while their display strings put + * `late` first. A plain two-row model therefore predicts `early, late` for an + * ascending direct index read and `early` for a one-row public query. + * + * The legal fixed grammar crosses BasicIndex and BTreeIndex with incremental + * adds and full builds, positive and negative epoch width, string collations, + * direction, and equal-instant ties, then compares exact ordered keys after + * the action. A ready source exercises scan and indexed live-query paths at + * preload; an on-demand Collection checks the first published ordered row. + * Index spies prove those paths read the index. The oracle does not claim mixed + * Temporal kinds, custom index callbacks, mutated built-ins, or order for + * Temporal kinds whose public comparison is undefined. + */ +type Row = { id: string; value: Temporal.ZonedDateTime } +const rows: ReadonlyArray = [ + { + id: `early`, + value: Temporal.ZonedDateTime.from(`2024-01-01T14:00+14:00[+14:00]`), + }, + { + id: `late`, + value: Temporal.ZonedDateTime.from(`2024-01-01T01:00+00:00[UTC]`), + }, +] +const [early, late] = rows +const expectedOrder = [`early`, `late`] +const widthBoundary: ReadonlyArray = [ + { + id: `early`, + value: + Temporal.Instant.fromEpochNanoseconds(9n).toZonedDateTimeISO(`+14:00`), + }, + { + id: `late`, + value: Temporal.Instant.fromEpochNanoseconds(10n).toZonedDateTimeISO(`UTC`), + }, +] +const negativeWidthBoundary: ReadonlyArray = [ + { + id: `early`, + value: + Temporal.Instant.fromEpochNanoseconds(-10n).toZonedDateTimeISO(`UTC`), + }, + { + id: `late`, + value: Temporal.Instant.fromEpochNanoseconds(-1n).toZonedDateTimeISO(`UTC`), + }, +] +const highMagnitudeNeighbors: ReadonlyArray = [ + { + id: `z-earlier`, + value: + Temporal.Instant.fromEpochNanoseconds( + 1_704_067_200_000_000_000n, + ).toZonedDateTimeISO(`+14:00`), + }, + { + id: `a-later`, + value: + Temporal.Instant.fromEpochNanoseconds( + 1_704_067_200_000_000_001n, + ).toZonedDateTimeISO(`UTC`), + }, +] +const tiedInstants: ReadonlyArray = [ + { + id: `late`, + value: Temporal.Instant.fromEpochNanoseconds(0n).toZonedDateTimeISO(`UTC`), + }, + { + id: `early`, + value: + Temporal.Instant.fromEpochNanoseconds(0n).toZonedDateTimeISO(`+14:00`), + }, +] +const comparisons: ReadonlyArray<{ + name: string + compareOptions?: CompareOptions + expected: ReadonlyArray +}> = [ + { name: `default`, expected: expectedOrder }, + { + name: `lexical strings`, + compareOptions: { direction: `asc`, nulls: `first`, stringSort: `lexical` }, + expected: expectedOrder, + }, + { + name: `locale strings`, + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `locale`, + locale: `en`, + }, + expected: expectedOrder, + }, + { + name: `custom string collation`, + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `custom`, + compare: (a, b) => b.localeCompare(a), + }, + expected: expectedOrder, + }, + { + name: `descending`, + compareOptions: { direction: `desc`, nulls: `first` }, + expected: [`late`, `early`], + }, +] + +describe(`ZonedDateTime indexed instant order`, () => { + it(`calibrates the opposing instant and display orders`, () => { + expect(early!.value.epochNanoseconds < late!.value.epochNanoseconds).toBe( + true, + ) + expect(early!.value.toString() > late!.value.toString()).toBe(true) + expect( + widthBoundary[0]!.value.toString() > widthBoundary[1]!.value.toString(), + ).toBe(true) + expect(highMagnitudeNeighbors[0]!.value.epochNanoseconds + 1n).toBe( + highMagnitudeNeighbors[1]!.value.epochNanoseconds, + ) + expect(Number(highMagnitudeNeighbors[0]!.value.epochNanoseconds)).toBe( + Number(highMagnitudeNeighbors[1]!.value.epochNanoseconds), + ) + expect( + highMagnitudeNeighbors[0]!.value.toString() > + highMagnitudeNeighbors[1]!.value.toString(), + ).toBe(true) + expect( + [...rows] + .sort((a, b) => + a.value.epochNanoseconds < b.value.epochNanoseconds ? -1 : 1, + ) + .map((row) => row.id), + ).toEqual(expectedOrder) + expect([`late`, `early`]).not.toEqual(expectedOrder) + }) + + for (const IndexType of [BasicIndex, BTreeIndex]) { + it.each([`add`, `build`] as const)( + `${IndexType.name} %s orders the supplied instants`, + (mode) => { + const index = new IndexType(1, new PropRef([`value`])) + if (mode === `build`) index.build(rows.map((row) => [row.id, row])) + else for (const row of rows) index.add(row.id, row) + expect(index.takeFromStart(2)).toEqual(expectedOrder) + expect(index.take(1, early!.value)).toEqual([`late`]) + }, + ) + + // Equal-width 2024 epochs do not expose a key that sorts decimal text. + // The adjacent 9ns/10ns pair does, with opposing display order too. + it.each(comparisons)( + `${IndexType.name} orders instants across an epoch digit-width boundary with $name`, + ({ compareOptions, expected }) => { + const index = new IndexType( + 1, + new PropRef([`value`]), + undefined, + { compareOptions }, + ) + for (const row of widthBoundary) index.add(row.id, row) + expect(index.takeFromStart(2)).toEqual(expected) + }, + ) + + it(`${IndexType.name} orders negative instants across an epoch digit-width boundary`, () => { + const index = new IndexType(1, new PropRef([`value`])) + for (const row of negativeWidthBoundary) index.add(row.id, row) + expect(index.takeFromStart(2)).toEqual(expectedOrder) + }) + + it(`${IndexType.name} preserves adjacent nanoseconds at a large epoch`, () => { + const index = new IndexType(1, new PropRef([`value`])) + for (const row of highMagnitudeNeighbors) index.add(row.id, row) + expect(index.takeFromStart(2)).toEqual([`z-earlier`, `a-later`]) + }) + + it(`${IndexType.name} orders equal instants by key while retaining exact membership`, () => { + const index = new IndexType(1, new PropRef([`value`])) + for (const row of tiedInstants) index.add(row.id, row) + expect(index.takeFromStart(2)).toEqual(expectedOrder) + expect(index.equalityLookup(tiedInstants[0]!.value)).toEqual( + new Set([`late`]), + ) + expect(index.equalityLookup(tiedInstants[1]!.value)).toEqual( + new Set([`early`]), + ) + }) + } + + for (const [scenario, sourceRows] of [ + [`opposing display text`, rows], + [`epoch digit width`, widthBoundary], + ] as const) { + it.each([`scan`, `BasicIndex`, `BTreeIndex`] as const)( + `keeps the first public row for ${scenario} with %s`, + async (path) => { + const source = createCollection( + mockSyncCollectionOptions({ + id: `temporal-index-order-${scenario}-${path}`, + getKey: (row) => row.id, + initialData: [...sourceRows], + autoIndex: `off`, + }), + ) + let live: ReturnType | undefined + try { + await source.stateWhenReady() + const index = + path === `scan` + ? undefined + : source.createIndex((row) => row.value, { + indexType: path === `BasicIndex` ? BasicIndex : BTreeIndex, + }) + const take = index && vi.spyOn(index, `takeFromStart`) + live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.value) + .limit(1), + ) + await live.preload() + expect([...live.values()].map((row) => row.id)).toEqual([`early`]) + if (take) expect(take).toHaveBeenCalled() + } finally { + await live?.cleanup() + await source.cleanup() + } + }, + ) + } + + for (const IndexType of [BasicIndex, BTreeIndex]) { + for (const [scenario, sourceRows, expectedFirst] of [ + [`epoch digit width`, widthBoundary, `early`], + [ + `adjacent high-magnitude nanoseconds`, + highMagnitudeNeighbors, + `z-earlier`, + ], + ] as const) { + it(`${IndexType.name} selects the earliest on-demand Collection row for ${scenario}`, async () => { + // A local ordered snapshot reads the index before asking its provider for + // more rows. The 9ns/10ns pair makes equality-key text order disagree + // with the public instant order at that synchronous publication cut. + const source = createCollection({ + id: `temporal-on-demand-${IndexType.name}-${scenario}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `off`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of sourceRows) { + write({ type: `insert`, value: row }) + } + commit() + markReady() + return { loadSubset: () => true as const, unloadSubset: () => {} } + }, + }, + }) + let subscription: ReturnType | undefined + try { + const index = source.createIndex((row) => row.value, { + indexType: IndexType, + }) + const take = vi.spyOn(index, `takeFromStart`) + const published: Array> = [] + subscription = source.subscribeChanges( + (changes) => + published.push(changes.map((change) => change.value.id)), + { includeInitialState: false }, + ) + subscription.setOrderByIndex(index) + subscription.requestLimitedSnapshot({ + orderBy: [ + { + expression: new PropRef([`value`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + limit: 1, + }) + expect(take).toHaveBeenCalled() + expect(published).toEqual([[expectedFirst]]) + } finally { + subscription?.unsubscribe() + await source.cleanup() + } + }) + } + } +}) + +/** + * # Does an index preserve the public order of day/time Durations? + * + * Changelog #1519 makes `orderBy` use Temporal's static compare. The external + * Temporal implementation supplies this oracle's expected sign: an hour is + * shorter than 120 minutes, while an hour and 60 minutes tie in order but + * remain distinct under #1519 query equality. The fixed legal domain below + * uses day/time fields whose comparison needs no `relativeTo`. Calendar-unit + * Durations, custom index callbacks, and a public order among tied rows are + * outside the claim. + * + * Direct adds check both index engines after each supplied pair. One on-demand + * Collection then checks the first published row at requestLimitedSnapshot's + * synchronous cut. Distinct IDs deliberately reverse key order, so treating + * unequal Durations as an index tie fails the ordered observation. Exclusive + * cursors on the equal-order pair must pass neither row, even though exact + * equality lookups keep their distinct representations separate. + */ +type DurationRow = { id: string; value: Temporal.Duration } +const durationOrderCases = [ + { + name: `hours and minutes`, + earlier: Temporal.Duration.from({ hours: 1 }), + later: Temporal.Duration.from({ minutes: 120 }), + }, + { + name: `negative hours and minutes`, + earlier: Temporal.Duration.from({ hours: -2 }), + later: Temporal.Duration.from({ minutes: -30 }), + }, + { + name: `days and hours`, + earlier: Temporal.Duration.from({ hours: 23 }), + later: Temporal.Duration.from({ days: 1 }), + }, + { + name: `minutes and seconds`, + earlier: Temporal.Duration.from({ seconds: 59 }), + later: Temporal.Duration.from({ minutes: 1 }), + }, + { + name: `seconds and milliseconds`, + earlier: Temporal.Duration.from({ milliseconds: 999 }), + later: Temporal.Duration.from({ seconds: 1 }), + }, + { + name: `milliseconds and microseconds`, + earlier: Temporal.Duration.from({ microseconds: 999 }), + later: Temporal.Duration.from({ milliseconds: 1 }), + }, + { + name: `microseconds and nanoseconds`, + earlier: Temporal.Duration.from({ nanoseconds: 999 }), + later: Temporal.Duration.from({ microseconds: 1 }), + }, +] +const publishedDurationRows: ReadonlyArray = [ + { id: `z-shorter`, value: Temporal.Duration.from({ hours: 1 }) }, + { id: `a-longer`, value: Temporal.Duration.from({ minutes: 120 }) }, +] + +describe(`Duration indexed time order`, () => { + it.each(durationOrderCases)( + `calibrates the $name order against Temporal`, + ({ earlier, later }) => { + expect(Temporal.Duration.compare(earlier, later)).toBeLessThan(0) + }, + ) + + for (const IndexType of [BasicIndex, BTreeIndex]) { + it.each(durationOrderCases)( + `${IndexType.name} preserves $name order`, + ({ earlier, later }) => { + const index = new IndexType(1, new PropRef([`value`])) + index.add(`z-earlier`, { value: earlier }) + index.add(`a-later`, { value: later }) + expect(index.takeFromStart(2)).toEqual([`z-earlier`, `a-later`]) + }, + ) + + it(`${IndexType.name} retains distinct exact Duration values at an order tie`, () => { + const hour = Temporal.Duration.from({ hours: 1 }) + const minutes = Temporal.Duration.from({ minutes: 60 }) + expect(Temporal.Duration.compare(hour, minutes)).toBe(0) + const index = new IndexType(1, new PropRef([`value`])) + index.add(`z-hour`, { value: hour }) + index.add(`a-minutes`, { value: minutes }) + expect(index.equalityLookup(hour)).toEqual(new Set([`z-hour`])) + expect(index.equalityLookup(minutes)).toEqual(new Set([`a-minutes`])) + expect(new Set(index.takeFromStart(2))).toEqual( + new Set([`z-hour`, `a-minutes`]), + ) + expect(index.take(2, minutes)).toEqual([]) + expect(index.takeReversed(2, hour)).toEqual([]) + }) + + it(`${IndexType.name} publishes the shorter Duration in an on-demand ordered snapshot`, async () => { + const source = createCollection({ + id: `duration-on-demand-${IndexType.name}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `off`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of publishedDurationRows) { + write({ type: `insert`, value: row }) + } + commit() + markReady() + return { loadSubset: () => true as const, unloadSubset: () => {} } + }, + }, + }) + let subscription: ReturnType | undefined + try { + const index = source.createIndex((row) => row.value, { + indexType: IndexType, + }) + const take = vi.spyOn(index, `takeFromStart`) + const published: Array> = [] + subscription = source.subscribeChanges( + (changes) => published.push(changes.map((change) => change.value.id)), + { includeInitialState: false }, + ) + subscription.setOrderByIndex(index) + subscription.requestLimitedSnapshot({ + orderBy: [ + { + expression: new PropRef([`value`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + limit: 1, + }) + expect(take).toHaveBeenCalled() + expect(published).toEqual([[`z-shorter`]]) + } finally { + subscription?.unsubscribe() + await source.cleanup() + } + }) + } +}) diff --git a/packages/db/tests/utils-oracle.property.test.ts b/packages/db/tests/utils-oracle.property.test.ts index 2bfaaa4f08..2ad77a4632 100644 --- a/packages/db/tests/utils-oracle.property.test.ts +++ b/packages/db/tests/utils-oracle.property.test.ts @@ -12,8 +12,12 @@ import { deepEquals, equalPersistedSnapshotValues } from '../src/utils' * structure, and changed-leaf inequality for primitives, arrays, records, * dates, regexes, typed bytes, and selected Temporal types. Named cross-type * pairs remain unequal, even when a primitive displays or encodes a Date or - * RegExp value. Separate bounded graph laws construct corresponding rings - * and acyclic copies without reusing the production walk. + * RegExp value. Published #1519 keeps equivalent Temporal.Duration forms + * distinct for query `eq`. The `deepEquals` API and #1901's change-message + * reconstruction law carry that field distinction to Collection events: a + * changed public Duration row cannot be suppressed because its display text + * matches. Separate bounded graph laws construct corresponding rings and + * acyclic copies without reusing the production walk. * * Arbitrary different cycle topologies are outside this model. Naming that * exclusion matters: more random examples cannot establish semantics the @@ -123,6 +127,30 @@ const arbitraryTemporalDuration = fc Temporal.Duration.from({ hours, minutes, seconds }), ) +// One nonzero intrinsic field at a time is a legal Duration. Every campaign +// enumerates all ten fields at generated magnitudes 1–9 and both signs. Each +// field has an equal fresh copy, a changed-field neighbor, and an opposite-sign +// value, so its exact value affects change-event equality independently of +// production comparison. +const durationFields = [ + `years`, + `months`, + `weeks`, + `days`, + `hours`, + `minutes`, + `seconds`, + `milliseconds`, + `microseconds`, + `nanoseconds`, +] as const + +const adjacentDurationUnits = [ + [`seconds`, `milliseconds`], + [`milliseconds`, `microseconds`], + [`microseconds`, `nanoseconds`], +] as const + // Same-type value arbitraries for testing equivalence properties // Mixed Date/Temporal inequality has its own generated and fixed witnesses below. const arbitrarySameTypePrimitive = fc.oneof( @@ -1232,6 +1260,56 @@ describe(`deepEquals property-based tests`, () => { `RegExp differs from its source text in both directions`, (regex) => assertDistinctTypes(regex, regex.source, deepEquals), ) + + // The source fields, rather than Temporal's display or ordered duration, + // form the reference identity. The production path is deepEquals in both + // directions; its direct result is compared after each constructed pair. + // Every field is exercised at a positive magnitude, its neighbor, and its + // negative counterpart. No one field may be ignored or compared by magnitude. + utilsProperty([fc.integer({ min: 1, max: 9 })])( + `each Temporal.Duration field determines change-event equality`, + (value) => { + for (const field of durationFields) { + const original = Temporal.Duration.from({ [field]: value }) + const copy = Temporal.Duration.from({ [field]: value }) + const changed = Temporal.Duration.from({ [field]: value + 1 }) + const opposite = Temporal.Duration.from({ [field]: -value }) + expectCopyAndChanged(original, copy, changed) + expectEqualityPair(original, opposite, false) + } + }, + ) + + it(`keeps one second distinct from 1000 milliseconds with the same display`, () => { + const seconds = Temporal.Duration.from({ seconds: 1 }) + const milliseconds = Temporal.Duration.from({ milliseconds: 1000 }) + expect(seconds.toString()).toBe(milliseconds.toString()) + expectEqualityPair(seconds, milliseconds, false) + }) + + // This generated grammar varies magnitude and sign while enumerating every + // adjacent subsecond unit boundary. Both constructors encode the same + // displayed elapsed time, but the single nonzero source field differs. + // #1519's query equality distinction and #1901's event reconstruction + // law together predict inequality in both directions here. This rejects + // a comparator that checks seconds alone before using display text. + utilsProperty([fc.integer({ min: 1, max: 9 })])( + `adjacent Duration units remain unequal despite matching display`, + (magnitude) => { + for (const sign of [-1, 1] as const) { + for (const [coarser, finer] of adjacentDurationUnits) { + const left = Temporal.Duration.from({ + [coarser]: sign * magnitude, + }) + const right = Temporal.Duration.from({ + [finer]: sign * magnitude * 1000, + }) + expect(left.toString()).toBe(right.toString()) + expectEqualityPair(left, right, false) + } + } + }, + ) }) }) From 78bd4946b1067cd637158b49f46c1b339ed4ad3c Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 12:38:46 -0600 Subject: [PATCH 38/46] refactor: simplify index comparison options and descending order --- .../oracle-reviews/pr-2116-confirmed-fixes.md | 8 ++++---- packages/db/src/indexes/base-index.ts | 9 ++------- packages/db/src/utils/comparison.ts | 7 +++---- 3 files changed, 9 insertions(+), 15 deletions(-) diff --git a/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md b/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md index e7ec02e82e..8f3902d4c5 100644 --- a/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md +++ b/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md @@ -37,13 +37,13 @@ supports only its legal histories, paths, and observations. ## Production-code weight -Against the recorded `origin/main` baseline, the final source diff under -`packages/db/src` adds 200 lines and removes 211, for **11 fewer production +Against the recorded `origin/main` baseline, the source diff under +`packages/db/src` adds 205 lines and removes 222, for **17 fewer production lines**. The built DB ESM entry bundled with esbuild (`bundle`, `minify`, `format: esm`, `platform: neutral`, `NODE_ENV=production`; DB IVM, pacer-lite, and standard-schema external) is 334,795 bytes / 100,227 gzip at baseline and -335,886 bytes / 100,644 gzip for this change. That is **1,091 more minified -bytes / 417 more gzip bytes**. The source-line budget is negative; the bundle +335,754 bytes / 100,551 gzip for this change. That is **959 more minified +bytes / 324 more gzip bytes**. The source-line budget is negative; the bundle budget is still positive. The remaining runtime addition is concentrated in distinct index-storage comparison and exact Temporal keys, which the bounded fixes above require. Do not treat source-line reduction as a bundle-size win. diff --git a/packages/db/src/indexes/base-index.ts b/packages/db/src/indexes/base-index.ts index 393618dcfd..c13715572e 100644 --- a/packages/db/src/indexes/base-index.ts +++ b/packages/db/src/indexes/base-index.ts @@ -304,17 +304,12 @@ export abstract class BaseIndex< return false } - if (indexStringSort === `custom` && requestedStringSort === `custom`) { + if (indexCompareOptions.stringSort === `custom`) return ( - indexCompareOptions.stringSort === `custom` && compareOptions.stringSort === `custom` && indexCompareOptions.compare === compareOptions.compare ) - } - - if (indexStringSort !== `locale` || requestedStringSort !== `locale`) { - return true - } + if (indexStringSort !== `locale`) return true const indexLocaleOptions = indexCompareOptions as LocaleCompareOptions const requestedLocaleOptions = compareOptions as LocaleCompareOptions diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index ec0c4ed344..085ba615a1 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -118,10 +118,9 @@ export const descComparator = ( b: unknown, opts: CompareOptions, ): number => { - return ascComparator(b, a, { - ...opts, - nulls: opts.nulls === `first` ? `last` : `first`, - }) + return a == null || b == null + ? ascComparator(a, b, opts) + : ascComparator(b, a, opts) } export function makeComparator( From 2f852dcd15bacaceefad1c307e3c6bb9d26397c7 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 14:34:00 -0600 Subject: [PATCH 39/46] fix: preserve index ownership and Temporal query equality --- docs/contributing/oracle-coverage.md | 21 +- packages/db/src/indexes/base-index.ts | 29 +- packages/db/src/indexes/basic-index.ts | 98 +++-- packages/db/src/indexes/btree-index.ts | 366 ++++++------------ packages/db/src/query/builder/functions.ts | 6 +- packages/db/src/utils/comparison.ts | 189 ++++----- packages/db/tests/collection-indexes.test.ts | 8 +- .../duration-query-equality-oracle.test.ts | 50 ++- .../index-update-oracle.property.test.ts | 293 +++++++++++--- packages/db/tests/utils.ts | 35 +- 10 files changed, 606 insertions(+), 489 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 107d146986..6bfba770ae 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -484,9 +484,24 @@ the first bounded public row through a scan and each index when an ordinary string begins the internal encoding prefix. An `origin/main` probe failed the default-locale public prefix case. The pre-classifier PR comparator failed all four public prefix cases and both BTree Temporal-deletion histories. This owner -does not prescribe mixed-kind order or cover arbitrary custom index callbacks, -all JavaScript value kinds, or -general Collection publication histories. +also compares complete forward and reverse reads through `IndexInterface`, +without prescribing index buckets or a BTree representative. A fixed churn +witness holds one live row while 16, 64, or 256 earlier distinct values are +added and removed; its public `takeFromStart(1)` result and comparator-call +growth reject retained empty buckets that add comparator work. This is a bounded +read-work check, not a memory bound or a general complexity proof. A deliberate +empty-bucket mutant failed at the intended comparator-count comparison. +The invalid-comparator owner checks rejection exactly when a comparison +returns a non-number or `NaN` and checks rejected removal atomicity; it does +not require a comparison merely because one comparator-equal row is removed. +The binary lane checks fresh equal-content removal, re-addition, and +replacement beside another byte value. It compares complete forward and +reverse key reads to a source Map and exact membership for short and wide +values that differ in their last byte. A pre-repair BTree lost a surviving key +after one equal-content owner left; the expanded lane rejected that result. +This owner does not prescribe mixed-kind order or cover arbitrary custom index +callbacks, binary range ordering, all JavaScript value kinds, or general +Collection publication histories. The [comparison oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts) owns the transferred same-realm NaN and invalid-Date default-order cases, diff --git a/packages/db/src/indexes/base-index.ts b/packages/db/src/indexes/base-index.ts index c13715572e..cc2ce9414c 100644 --- a/packages/db/src/indexes/base-index.ts +++ b/packages/db/src/indexes/base-index.ts @@ -1,5 +1,8 @@ import { compileSingleRowExpression } from '../query/compiler/evaluators.js' -import { comparisonFunctions } from '../query/builder/functions.js' +import { + comparisonFunctions, + indexableOperations, +} from '../query/builder/functions.js' import { DEFAULT_COMPARE_OPTIONS, deepEquals } from '../utils.js' import { makeCheckedComparator, makeComparator } from '../utils/comparison.js' import { codedMessage, codedWarning, devBuild } from '../error-message.js' @@ -35,6 +38,8 @@ function resolveStringSort( */ export const IndexOperation = comparisonFunctions +export { indexableOperations } from '../query/builder/functions.js' + export const builtInIndexResolverNames = new WeakMap() /** @@ -156,14 +161,7 @@ export abstract class BaseIndex< options?.compareFn ?? makeComparator(this.compareOptions, `index`), ) this.name = name - this.supportedOperations = new Set([ - `eq`, - `gt`, - `gte`, - `lt`, - `lte`, - `in`, - ]) + this.supportedOperations = new Set(indexableOperations) this.initialize(options) } @@ -194,7 +192,7 @@ export abstract class BaseIndex< ): Array abstract get keyCount(): number abstract equalityLookup(value: any): Set - protected exactBucketKeys(value: any): Set | undefined { + protected exactBucketKeys(value: any): Iterable | undefined { return this.equalityLookup(value) } abstract rangeQuery(options: RangeQueryOptions): Set @@ -221,7 +219,7 @@ export abstract class BaseIndex< const result = new Set() for (const value of values) { const keys = this.exactBucketKeys(value) - if (keys) keys.forEach((key) => result.add(key)) + if (keys) for (const key of keys) result.add(key) } return result } @@ -304,12 +302,17 @@ export abstract class BaseIndex< return false } - if (indexCompareOptions.stringSort === `custom`) + if (indexStringSort === `custom` && requestedStringSort === `custom`) { return ( + indexCompareOptions.stringSort === `custom` && compareOptions.stringSort === `custom` && indexCompareOptions.compare === compareOptions.compare ) - if (indexStringSort !== `locale`) return true + } + + if (indexStringSort !== `locale` || requestedStringSort !== `locale`) { + return true + } const indexLocaleOptions = indexCompareOptions as LocaleCompareOptions const requestedLocaleOptions = compareOptions as LocaleCompareOptions diff --git a/packages/db/src/indexes/basic-index.ts b/packages/db/src/indexes/basic-index.ts index b4ce5ba31f..22e30c5365 100644 --- a/packages/db/src/indexes/basic-index.ts +++ b/packages/db/src/indexes/basic-index.ts @@ -1,10 +1,15 @@ import { compareKeys } from '@tanstack/db-ivm' -import { areSameValueZeroEqual, normalizeValue } from '../utils/comparison.js' +import { + areSameValueZeroEqual, + cacheBinaryKey, + isUint8Array, + normalizeValue, +} from '../utils/comparison.js' +import { codedMessage, devBuild } from '../error-message.js' import { compareKeysReversed, findInsertPositionInArray, } from '../utils/array-utils.js' -import { codedMessage, devBuild } from '../error-message.js' import { BaseIndex, builtInIndexResolverNames, @@ -20,6 +25,19 @@ export type { RangeQueryOptions } from './btree-index.js' */ export type BasicIndexOptions = IndexComparisonOptions +function orderValue(value: unknown): unknown { + return value instanceof Date ? value.getTime() : value +} + +function storedOrderValue(value: unknown, exact: unknown): unknown { + if (isUint8Array(value)) { + const copy = new Uint8Array(value) + cacheBinaryKey(copy, exact as string) + return copy + } + return orderValue(value) +} + /** * Basic index using Map + sorted Array. * @@ -47,13 +65,17 @@ export class BasicIndex< const indexedValue = this.evaluateAddedValue(key, item) const normalizedValue = normalizeValue(indexedValue) - this.addToBucket(key, normalizedValue) + this.addToBucket(key, normalizedValue, indexedValue) this.addRangeValue(indexedValue) this.indexedKeys.add(key) } - private addToBucket(key: TKey, normalizedValue: unknown): void { + private addToBucket( + key: TKey, + normalizedValue: unknown, + value: unknown, + ): void { const keySet = this.valueMap.get(normalizedValue) if (keySet) { // Value already exists, just add the key to the set @@ -61,13 +83,14 @@ export class BasicIndex< } else { // New value. Find its sorted position first: a comparator failure must // throw before the index changes. + const orderedValue = storedOrderValue(value, normalizedValue) const insertIdx = findInsertPositionInArray( this.sortedValues, - normalizedValue, + orderedValue, this.compareFn, ) this.valueMap.set(normalizedValue, new Set([key])) - this.sortedValues.splice(insertIdx, 0, normalizedValue) + this.sortedValues.splice(insertIdx, 0, orderedValue) } } @@ -86,31 +109,36 @@ export class BasicIndex< const normalizedValue = normalizeValue(indexedValue) - this.removeFromBucket(key, normalizedValue) + this.removeFromBucket(key, normalizedValue, indexedValue) this.removeRangeValue(indexedValue) this.indexedKeys.delete(key) } - private removeFromBucket(key: TKey, normalizedValue: unknown): void { + private removeFromBucket( + key: TKey, + normalizedValue: unknown, + value: unknown, + ): void { const keySet = this.valueMap.get(normalizedValue) if (keySet?.has(key)) { // The last key for this value leaves the sorted array too. Locate it // before any change, so a comparator failure leaves the index unchanged. if (keySet.size === 1) { + const orderedValue = orderValue(value) let sortedIndex = findInsertPositionInArray( this.sortedValues, - normalizedValue, + orderedValue, this.compareFn, ) // Distinct equality keys may share one comparator position. while ( sortedIndex < this.sortedValues.length && - this.compareFn(this.sortedValues[sortedIndex], normalizedValue) === 0 + this.compareFn(this.sortedValues[sortedIndex], orderedValue) === 0 ) { if ( areSameValueZeroEqual( - this.sortedValues[sortedIndex], + normalizeValue(this.sortedValues[sortedIndex]), normalizedValue, ) ) { @@ -152,9 +180,9 @@ export class BasicIndex< return } - this.removeFromBucket(key, oldValue) + this.removeFromBucket(key, oldValue, oldIndexedValue) this.removeRangeValue(oldIndexedValue) - this.addToBucket(key, newValue) + this.addToBucket(key, newValue, newIndexedValue) this.addRangeValue(newIndexedValue) this.indexedKeys.add(key) } @@ -170,14 +198,17 @@ export class BasicIndex< const value = normalizeValue(indexedValue) const keys = this.valueMap.get(value) if (keys) keys.add(key) - else this.valueMap.set(value, new Set([key])) + else { + this.valueMap.set(value, new Set([key])) + this.sortedValues.push(storedOrderValue(indexedValue, value)) + } this.addRangeValue(indexedValue) this.indexedKeys.add(key) } // Array.sort moves raw undefined values last without comparing them. // Wrap values so nulls-first comparison runs during a full build. - const sortable = Array.from(this.valueMap.keys(), (value) => ({ value })) + const sortable = this.sortedValues.map((value) => ({ value })) sortable.sort((left, right) => this.compareFn(left.value, right.value)) this.sortedValues = sortable.map(({ value }) => value) } @@ -192,9 +223,6 @@ export class BasicIndex< this.clearRangeValues() } - /** - * Performs a lookup operation - */ lookup(operation: IndexOperation, value: any): Set { const result = this.lookupSupportedOperation(operation, value) if (result !== undefined) return result @@ -231,8 +259,8 @@ export class BasicIndex< return result } - const normalizedFrom = normalizeValue(from) - const normalizedTo = normalizeValue(to) + const orderedFrom = orderValue(from) + const orderedTo = orderValue(to) const hasFrom = `from` in options const hasTo = `to` in options @@ -241,7 +269,7 @@ export class BasicIndex< if (hasFrom) { startIdx = findInsertPositionInArray( this.sortedValues, - normalizedFrom, + orderedFrom, this.compareFn, ) // Comparator-equal values form one range boundary even when they are @@ -249,7 +277,7 @@ export class BasicIndex< while ( !fromInclusive && startIdx < this.sortedValues.length && - this.compareFn(this.sortedValues[startIdx], normalizedFrom) === 0 + this.compareFn(this.sortedValues[startIdx], orderedFrom) === 0 ) { startIdx++ } @@ -260,14 +288,14 @@ export class BasicIndex< if (hasTo) { endIdx = findInsertPositionInArray( this.sortedValues, - normalizedTo, + orderedTo, this.compareFn, ) // Include the whole comparator group at an inclusive upper boundary. while ( toInclusive && endIdx < this.sortedValues.length && - this.compareFn(this.sortedValues[endIdx], normalizedTo) === 0 + this.compareFn(this.sortedValues[endIdx], orderedTo) === 0 ) { endIdx++ } @@ -275,7 +303,7 @@ export class BasicIndex< // Collect all keys in range for (let i = startIdx; i < endIdx; i++) { - const keys = this.valueMap.get(this.sortedValues[i]) + const keys = this.valueMap.get(normalizeValue(this.sortedValues[i])) if (keys) { keys.forEach((key) => result.add(key)) } @@ -288,16 +316,16 @@ export class BasicIndex< * Returns the next n items in sorted order */ take(n: number, from: any, filterFn?: (key: TKey) => boolean): Array { - const normalizedFrom = normalizeValue(from) + const orderedFrom = orderValue(from) let startIdx = findInsertPositionInArray( this.sortedValues, - normalizedFrom, + orderedFrom, this.compareFn, ) // Skip past the 'from' value (exclusive) while ( startIdx < this.sortedValues.length && - this.compareFn(this.sortedValues[startIdx], normalizedFrom) <= 0 + this.compareFn(this.sortedValues[startIdx], orderedFrom) <= 0 ) { startIdx++ } @@ -313,17 +341,17 @@ export class BasicIndex< from: any, filterFn?: (key: TKey) => boolean, ): Array { - const normalizedFrom = normalizeValue(from) + const orderedFrom = orderValue(from) let startIdx = findInsertPositionInArray( this.sortedValues, - normalizedFrom, + orderedFrom, this.compareFn, ) - 1 // Skip past the 'from' value (exclusive) while ( startIdx >= 0 && - this.compareFn(this.sortedValues[startIdx], normalizedFrom) >= 0 + this.compareFn(this.sortedValues[startIdx], orderedFrom) >= 0 ) { startIdx-- } @@ -364,7 +392,9 @@ export class BasicIndex< const groupValue = this.sortedValues[index] const groupKeys: Array = [] do { - for (const key of this.valueMap.get(this.sortedValues[index]) ?? []) { + for (const key of this.valueMap.get( + normalizeValue(this.sortedValues[index]), + ) ?? []) { groupKeys.push(key) } index += step @@ -381,10 +411,6 @@ export class BasicIndex< } return result } - - protected exactBucketKeys(value: any): Set | undefined { - return this.valueMap.get(normalizeValue(value)) - } } builtInIndexResolverNames.set(BasicIndex, `BasicIndex`) diff --git a/packages/db/src/indexes/btree-index.ts b/packages/db/src/indexes/btree-index.ts index c94f8a0610..3bb6952f6d 100644 --- a/packages/db/src/indexes/btree-index.ts +++ b/packages/db/src/indexes/btree-index.ts @@ -1,11 +1,7 @@ import { compareKeys } from '@tanstack/db-ivm' import { compareKeysReversed } from '../utils/array-utils.js' import { BTree } from '../utils/btree.js' -import { - areSameValueZeroEqual, - denormalizeUndefined, - normalizeForBTree, -} from '../utils/comparison.js' +import { areSameValueZeroEqual, normalizeValue } from '../utils/comparison.js' import { codedMessage, devBuild } from '../error-message.js' import { BaseIndex, @@ -14,14 +10,7 @@ import { } from './base-index.js' import type { IndexComparisonOptions, IndexOperation } from './base-index.js' -/** - * Options for Ordered index - */ export type BTreeIndexOptions = IndexComparisonOptions - -/** - * Options for range queries - */ export interface RangeQueryOptions { from?: any to?: any @@ -29,175 +18,131 @@ export interface RangeQueryOptions { toInclusive?: boolean } -type OrderedBucket = { - representative: unknown - exactValues: Set - keys: Set +type SortKey = { value: any } +type Group = SortKey & { + keys: Map + buckets?: Map> + representativeKey: TKey + nextRepresentative?: IterableIterator<[TKey, any]> } - -/** - * B+Tree index for sorted data with range queries - * This maintains items in sorted order and provides efficient range operations - */ export class BTreeIndex< TKey extends string | number = string | number, > extends BaseIndex { - // Internal data structures - private to hide implementation details - // The `orderedEntries` B+ tree groups values that occupy the same comparator - // position. The `valueMap` keeps exact values separate for equality lookups. - private orderedEntries = new BTree>((a, b) => - this.compareStored(a, b), + private orderedEntries = new BTree>((a, b) => + this.compareFn(a.value, b.value), ) - private valueMap = new Map< - unknown, - { keys: Set; ordered: OrderedBucket } - >() + private valueMap = new Map>() private indexedKeys = new Set() - // Stored keys use UNDEFINED_SENTINEL; the comparator, which may be - // user-supplied, expects undefined. - private compareStored(a: any, b: any): number { - return this.compareFn(denormalizeUndefined(a), denormalizeUndefined(b)) - } - - /** - * Adds a value to the index - */ add(key: TKey, item: any): void { - const indexedValue = this.evaluateAddedValue(key, item) - - // Normalize the value for Map key usage - const normalizedValue = normalizeForBTree(indexedValue) - - this.addToBucket(key, normalizedValue) - this.addRangeValue(indexedValue) - + const value = this.evaluateAddedValue(key, item) + const exact = normalizeValue(value) + this.addToBucket(key, value, exact) + this.addRangeValue(value) this.indexedKeys.add(key) } - private addToBucket(key: TKey, normalizedValue: unknown): void { - const exact = this.valueMap.get(normalizedValue) - if (exact) { - exact.keys.add(key) - exact.ordered.keys.add(key) + private addToBucket(key: TKey, value: any, exact: unknown): void { + const group = this.valueMap.get(exact) + if (group) { + group.keys.set(key, value) + group.buckets?.get(exact)!.add(key) return } - - let orderedBucket = this.orderedEntries.get(normalizedValue) - if (orderedBucket) { - orderedBucket.keys.add(key) - orderedBucket.exactValues.add(normalizedValue) + const ordered = this.orderedEntries.get({ value }) + if (ordered) { + ordered.buckets ??= new Map([ + [normalizeValue(ordered.value), new Set(ordered.keys.keys())], + ]) + ordered.buckets.set(exact, new Set([key])) + ordered.keys.set(key, value) + this.valueMap.set(exact, ordered) } else { - orderedBucket = { - representative: normalizedValue, - exactValues: new Set([normalizedValue]), - keys: new Set([key]), + const created = { + value, + keys: new Map([[key, value]]), + representativeKey: key, } - this.orderedEntries.set(normalizedValue, orderedBucket) + // The tree compares before the exact lookup map changes. + this.orderedEntries.set(created, created) + this.valueMap.set(exact, created) } - this.valueMap.set(normalizedValue, { - keys: new Set([key]), - ordered: orderedBucket, - }) } - /** - * Removes a value from the index - */ remove(key: TKey, item: any): void { - let indexedValue: any + let value: any try { - indexedValue = this.evaluateIndexExpression(item) + value = this.evaluateIndexExpression(item) } catch (error) { warnRemovalFailure(key, error) return } - - // Normalize the value for Map key usage - const normalizedValue = normalizeForBTree(indexedValue) - - this.removeFromBucket(key, normalizedValue) - this.removeRangeValue(indexedValue) - + const exact = normalizeValue(value) + this.removeFromBucket(key, exact) + this.removeRangeValue(value) this.indexedKeys.delete(key) } - private removeFromBucket(key: TKey, normalizedValue: unknown): void { - const exact = this.valueMap.get(normalizedValue) - if (!exact?.keys.has(key)) return - const orderedBucket = exact.ordered - const removedExactValue = exact.keys.size === 1 - const replacesRepresentative = - removedExactValue && - orderedBucket.keys.size > 1 && - areSameValueZeroEqual(orderedBucket.representative, normalizedValue) - - // Tree writes compare first, so a comparator failure throws before the - // index changes. A comparator-equal `set` overwrites the entry's key. - if (orderedBucket.keys.size === 1) { - this.orderedEntries.delete(normalizedValue) - } else if (replacesRepresentative) { - const representative = [...orderedBucket.exactValues].find( - (value) => !areSameValueZeroEqual(value, normalizedValue), - ) - this.orderedEntries.set(representative, orderedBucket) - orderedBucket.representative = representative + private removeFromBucket(key: TKey, exact: unknown): void { + const group = this.valueMap.get(exact) + if (!group?.keys.has(key)) return + const bucket = group.buckets?.get(exact) + if (group.buckets && !bucket?.has(key)) return + if (group.keys.size === 1) { + this.orderedEntries.delete(group) } - exact.keys.delete(key) - orderedBucket.keys.delete(key) - if (removedExactValue) { - this.valueMap.delete(normalizedValue) - orderedBucket.exactValues.delete(normalizedValue) + bucket?.delete(key) + group.keys.delete(key) + if (bucket?.size === 0 || group.keys.size === 0) { + this.valueMap.delete(exact) + group.buckets?.delete(exact) + } + if (group.buckets?.size === 1) group.buckets = undefined + if (group.keys.size && group.representativeKey === key) { + // Advancing one iterator skips each removed Map slot at most once. + const [nextKey, nextValue] = (group.nextRepresentative ??= + group.keys.entries()).next().value as [TKey, any] + group.representativeKey = nextKey + group.value = nextValue } } - /** - * Updates a value in the index - */ update(key: TKey, oldItem: any, newItem: any): void { - let oldIndexedValue: unknown - let newIndexedValue: unknown + let oldValue: any + let newValue: any try { - oldIndexedValue = this.evaluateIndexExpression(oldItem) - newIndexedValue = this.evaluateIndexExpression(newItem) + oldValue = this.evaluateIndexExpression(oldItem) + newValue = this.evaluateIndexExpression(newItem) } catch { this.remove(key, oldItem) this.add(key, newItem) return } - - const oldValue = normalizeForBTree(oldIndexedValue) - const newValue = normalizeForBTree(newIndexedValue) - if ( - areSameValueZeroEqual(oldValue, newValue) && - this.valueMap.get(newValue)?.keys.has(key) - ) { - this.removeRangeValue(oldIndexedValue) - this.addRangeValue(newIndexedValue) + const oldExact = normalizeValue(oldValue) + const newExact = normalizeValue(newValue) + const group = this.valueMap.get(newExact) + const containsKey = group?.buckets + ? group.buckets.get(newExact)?.has(key) + : group?.keys.has(key) + if (areSameValueZeroEqual(oldExact, newExact) && containsKey) { + group!.keys.set(key, newValue) + if (group!.representativeKey === key) group!.value = newValue + this.removeRangeValue(oldValue) + this.addRangeValue(newValue) return } - - this.removeFromBucket(key, oldValue) - this.removeRangeValue(oldIndexedValue) - this.addToBucket(key, newValue) - this.addRangeValue(newIndexedValue) + this.removeFromBucket(key, oldExact) + this.removeRangeValue(oldValue) + this.addToBucket(key, newValue, newExact) + this.addRangeValue(newValue) this.indexedKeys.add(key) } - /** - * Builds the index from a collection of entries - */ build(entries: Iterable<[TKey, any]>): void { this.clear() - - for (const [key, item] of entries) { - this.add(key, item) - } + for (const [key, item] of entries) this.add(key, item) } - /** - * Clears all data from the index - */ clear(): void { this.orderedEntries.clear() this.valueMap.clear() @@ -205,9 +150,6 @@ export class BTreeIndex< this.clearRangeValues() } - /** - * Performs a lookup operation - */ lookup(operation: IndexOperation, value: any): Set { const result = this.lookupSupportedOperation(operation, value) if (result !== undefined) return result @@ -218,163 +160,91 @@ export class BTreeIndex< ) } - /** - * Gets the number of indexed keys - */ get keyCount(): number { return this.indexedKeys.size } - // Public methods for backward compatibility (used by tests) - - /** - * Performs an equality lookup - */ equalityLookup(value: any): Set { - const normalizedValue = normalizeForBTree(value) - return new Set(this.valueMap.get(normalizedValue)?.keys ?? []) + return new Set(this.exactBucketKeys(value)) + } + + protected exactBucketKeys(value: any): Iterable | undefined { + const exact = normalizeValue(value) + const group = this.valueMap.get(exact) + return group?.buckets ? group.buckets.get(exact) : group?.keys.keys() } - /** - * Performs a range query with options - * This is more efficient for compound queries like "WHERE a > 5 AND a < 10" - */ rangeQuery(options: RangeQueryOptions = {}): Set { - const { from, to, fromInclusive = true, toInclusive = true } = options const result = new Set() - - // Check if from/to were explicitly provided (even if undefined) - // vs not provided at all (should use min/max key) - const hasFrom = `from` in options - const hasTo = `to` in options - - const fromKey = hasFrom - ? normalizeForBTree(from) - : this.orderedEntries.minKey() - const toKey = hasTo ? normalizeForBTree(to) : this.orderedEntries.maxKey() - + if (this.orderedEntries.size === 0) return result + const { from, to, fromInclusive = true, toInclusive = true } = options + const low = + 'from' in options ? { value: from } : this.orderedEntries.minKey()! + const high = 'to' in options ? { value: to } : this.orderedEntries.maxKey()! this.orderedEntries.forRange( - fromKey, - toKey, - toInclusive, - (indexedValue, bucket) => { - // Only exclude the boundary when an exclusive lower bound was - // actually provided. Without a `from` bound, `fromKey` defaults to - // the minimum key and must not be dropped. Compare against the - // normalized key since indexed values are stored normalized - // (e.g. dates as timestamps), so the raw `from` would never match. + low, + high, + !('to' in options) || toInclusive, + (sortKey, group) => { if ( - hasFrom && + 'from' in options && !fromInclusive && - this.compareStored(indexedValue, fromKey) === 0 - ) { - // the B+ tree `forRange` method does not support exclusive lower bounds - // so we need to exclude it manually + this.compareFn(sortKey.value, from) === 0 + ) return - } - - bucket.keys.forEach((key) => result.add(key)) + for (const key of group.keys.keys()) result.add(key) }, ) - return result } - /** - * Internal method for taking items from the index. - * @param n - The number of items to return - * @param nextPair - Function to get the next pair from the BTree - * @param from - Already normalized! undefined means "start from beginning/end", sentinel means "start from the key undefined" - * @param filterFn - Optional filter function - * @param reversed - Whether to reverse the order of keys within each value - */ private takeInternal( n: number, - nextPair: (k?: any) => [any, OrderedBucket] | undefined, - from: any, - filterFn?: (key: TKey) => boolean, - reversed: boolean = false, + from: SortKey | undefined, + filterFn: ((key: TKey) => boolean) | undefined, + reversed: boolean, ): Array { const result: Array = [] - let pair: [any, OrderedBucket] | undefined - let key = from // Use as-is - it's already normalized by the caller - - // Every key owns exactly one bucket, so the walk never repeats a key. - while ((pair = nextPair(key)) !== undefined && result.length < n) { - key = pair[0] - // Sort keys for deterministic order within a comparator position. - const sorted = Array.from(pair[1].keys).sort( + let pair = reversed + ? this.orderedEntries.nextLowerPair(from) + : this.orderedEntries.nextHigherPair(from) + while (pair && result.length < n) { + for (const key of [...pair[1].keys.keys()].sort( reversed ? compareKeysReversed : compareKeys, - ) - for (const ks of sorted) { + )) { + if (filterFn?.(key) ?? true) result.push(key) if (result.length >= n) break - if (filterFn?.(ks) ?? true) result.push(ks) } + if (result.length >= n) break + pair = reversed + ? this.orderedEntries.nextLowerPair(pair[0]) + : this.orderedEntries.nextHigherPair(pair[0]) } - return result } - /** - * Returns the next n items after the provided item. - * @param n - The number of items to return - * @param from - The item to start from (exclusive). - * @returns The next n items after the provided key. - */ take(n: number, from: any, filterFn?: (key: TKey) => boolean): Array { - const nextPair = (k?: any) => this.orderedEntries.nextHigherPair(k) - // Normalize the from value - const normalizedFrom = normalizeForBTree(from) - return this.takeInternal(n, nextPair, normalizedFrom, filterFn) + return this.takeInternal(n, { value: from }, filterFn, false) } - /** - * Returns the first n items from the beginning. - * @param n - The number of items to return - * @param filterFn - Optional filter function - * @returns The first n items - */ takeFromStart(n: number, filterFn?: (key: TKey) => boolean): Array { - const nextPair = (k?: any) => this.orderedEntries.nextHigherPair(k) - // Pass undefined to mean "start from beginning" (BTree's native behavior) - return this.takeInternal(n, nextPair, undefined, filterFn) + return this.takeInternal(n, undefined, filterFn, false) } - /** - * Returns the next n items **before** the provided item (in descending order). - * @param n - The number of items to return - * @param from - The item to start from (exclusive). Required. - * @returns The next n items **before** the provided key. - */ takeReversed( n: number, from: any, filterFn?: (key: TKey) => boolean, ): Array { - const nextPair = (k?: any) => this.orderedEntries.nextLowerPair(k) - // Normalize the from value - const normalizedFrom = normalizeForBTree(from) - return this.takeInternal(n, nextPair, normalizedFrom, filterFn, true) + return this.takeInternal(n, { value: from }, filterFn, true) } - /** - * Returns the last n items from the end. - * @param n - The number of items to return - * @param filterFn - Optional filter function - * @returns The last n items - */ takeReversedFromEnd( n: number, filterFn?: (key: TKey) => boolean, ): Array { - const nextPair = (k?: any) => this.orderedEntries.nextLowerPair(k) - // Pass undefined to mean "start from end" (BTree's native behavior) - return this.takeInternal(n, nextPair, undefined, filterFn, true) - } - - protected exactBucketKeys(value: any): Set | undefined { - return this.valueMap.get(normalizeForBTree(value))?.keys + return this.takeInternal(n, undefined, filterFn, true) } } -builtInIndexResolverNames.set(BTreeIndex, `BTreeIndex`) +builtInIndexResolverNames.set(BTreeIndex, 'BTreeIndex') diff --git a/packages/db/src/query/builder/functions.ts b/packages/db/src/query/builder/functions.ts index c2095a5cb6..c25c0bf772 100644 --- a/packages/db/src/query/builder/functions.ts +++ b/packages/db/src/query/builder/functions.ts @@ -704,13 +704,17 @@ export function max(arg: ExpressionLike): Aggregate { /** * List of comparison function names that can be used with indexes */ -export const comparisonFunctions = [ +export const indexableOperations = [ `eq`, `gt`, `gte`, `lt`, `lte`, `in`, +] as const + +export const comparisonFunctions = [ + ...indexableOperations, `like`, `ilike`, ] as const diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index 085ba615a1..e40809d305 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -123,51 +123,55 @@ export const descComparator = ( : ascComparator(b, a, opts) } +function indexObjectKind(value: any): string { + if (isUint8Array(value)) return `Binary` + if (isTemporal(value)) { + const tag = value[Symbol.toStringTag] + const fields = value as unknown as DurationFields + return tag === `Temporal.Duration` && + (fields.years || fields.months || fields.weeks) + ? `Temporal.Duration.calendar` + : tag + } + if (Array.isArray(value)) return `Array` + return `Object` +} + export function makeComparator( opts: CompareOptions, scope: `values` | `index` = `values`, ): (a: any, b: any) => number { + const compare = opts.direction === `asc` ? ascComparator : descComparator + if (scope !== `index`) return (a, b) => compare(a, b, opts) + const direction = opts.direction === `asc` ? 1 : -1 return (a, b) => { - // Index storage needs a transitive order across primitive kinds. Native - // coercion can make 2 tie "2", yet put them on opposite sides of "10". - if (scope === `index` && a != null && b != null && typeof a !== typeof b) { - const order = typeof a < typeof b ? -1 : 1 - return order * direction - } - if (scope === `index` && typeof a === `string` && typeof b === `string`) { - const aKind = indexStringKind(a) - const bKind = indexStringKind(b) - if (aKind !== bKind) { - const order = aKind < bKind ? -1 : 1 - return order * direction - } - if (aKind === `string`) { - a = a.startsWith(ESCAPED_STRING_PREFIX) - ? a.slice(ESCAPED_STRING_PREFIX.length) - : a - b = b.startsWith(ESCAPED_STRING_PREFIX) - ? b.slice(ESCAPED_STRING_PREFIX.length) - : b - } else { - if (aKind === ZONED_EPOCH_PREFIX || aKind === DURATION_TIME_PREFIX) { - // Exact index keys retain identity; these ordinals define order ties. - const aOrdinal = BigInt( - a.slice(aKind.length, a.indexOf(`:`, aKind.length)), - ) - const bOrdinal = BigInt( - b.slice(aKind.length, b.indexOf(`:`, aKind.length)), - ) - const order = aOrdinal < bOrdinal ? -1 : aOrdinal > bOrdinal ? 1 : 0 - return order * direction + // An index needs one stable domain for each native comparison law. In + // particular, Date order cannot be mixed with reference-object order. + if (a != null && b != null) { + if (a instanceof Date) a = a.getTime() + if (b instanceof Date) b = b.getTime() + const aType = typeof a + const bType = typeof b + if (aType !== bType) return (aType < bType ? -1 : 1) * direction + if (aType === `object`) { + const aKind = indexObjectKind(a) + const bKind = indexObjectKind(b) + if (aKind !== bKind) return (aKind < bKind ? -1 : 1) * direction + // Binary content and unordered Temporal values use their exact keys + // for stable storage order. + if ( + aKind === `Binary` || + aKind === `Temporal.Duration.calendar` || + aKind === `Temporal.PlainMonthDay` + ) { + const left = normalizeValue(a) + const right = normalizeValue(b) + return (left < right ? -1 : left > right ? 1 : 0) * direction } } } - if (opts.direction === `asc`) { - return ascComparator(a, b, opts) - } else { - return descComparator(a, b, opts) - } + return compare(a, b, opts) } } @@ -219,65 +223,45 @@ function areUint8ArraysEqual(a: Uint8Array, b: Uint8Array): boolean { } const NORMALIZED_KEY_PREFIX = `\u0000tanstack-db:` -const ESCAPED_STRING_PREFIX = `${NORMALIZED_KEY_PREFIX}string:` -const TEMPORAL_PREFIX = `${NORMALIZED_KEY_PREFIX}temporal:` -const ZONED_EPOCH_PREFIX = `${NORMALIZED_KEY_PREFIX}temporal:Temporal.ZonedDateTime:` -const DURATION_TIME_PREFIX = `${NORMALIZED_KEY_PREFIX}temporal:Temporal.Duration:time:` - -function indexStringKind(value: string): string { - if ( - !value.startsWith(NORMALIZED_KEY_PREFIX) || - value.startsWith(ESCAPED_STRING_PREFIX) - ) { - return `string` - } - if (value.startsWith(DURATION_TIME_PREFIX)) return DURATION_TIME_PREFIX - const afterKind = value.startsWith(TEMPORAL_PREFIX) - ? TEMPORAL_PREFIX.length - : NORMALIZED_KEY_PREFIX.length - return value.slice(0, value.indexOf(`:`, afterKind) + 1) -} - function normalizedKey(kind: string, value: string): string { return `${NORMALIZED_KEY_PREFIX}${kind}:${value}` } +const binaryKeys = new WeakMap() + +export function cacheBinaryKey(value: Uint8Array, key: string): void { + binaryKeys.set(value, key) +} + function normalizeBinary(value: Uint8Array): string { - let bytes = `` - for (let index = 0; index < value.byteLength; index++) { - bytes += String.fromCharCode(value[index]!) + const cached = binaryKeys.get(value) + if (cached) return cached + let bytes: string + if (value.byteLength <= 8192) { + bytes = String.fromCharCode.apply(null, value as unknown as Array) + } else { + const chunks: Array = [] + for (let index = 0; index < value.byteLength; index += 8192) { + chunks.push( + String.fromCharCode.apply( + null, + value.subarray(index, index + 8192) as unknown as Array, + ), + ) + } + bytes = chunks.join(``) } - return normalizedKey(`binary`, bytes) + const key = normalizedKey(`binary`, bytes) + binaryKeys.set(value, key) + return key } type DurationFields = Record<(typeof durationFields)[number], number> -function durationTimeNanoseconds(value: DurationFields): bigint { - return ( - BigInt(value.days) * 86_400_000_000_000n + - BigInt(value.hours) * 3_600_000_000_000n + - BigInt(value.minutes) * 60_000_000_000n + - BigInt(value.seconds) * 1_000_000_000n + - BigInt(value.milliseconds) * 1_000_000n + - BigInt(value.microseconds) * 1_000n + - BigInt(value.nanoseconds) - ) -} - -/** - * Sentinel value representing undefined in normalized form. - * This allows distinguishing between "start from beginning" (undefined parameter) - * and "start from the key undefined" (actual undefined value in the tree). - */ -export const UNDEFINED_SENTINEL = normalizedKey(`undefined`, ``) - /** * Normalize a value for comparison and Map key usage * Converts values that can't be directly compared or used as Map keys * into comparable primitive representations - * - * Note: This does NOT convert undefined to a sentinel. Use normalizeForBTree - * for BTree index operations that need to distinguish undefined values. */ export function normalizeValue(value: any): any { if (typeof value === `string`) { @@ -296,21 +280,14 @@ export function normalizeValue(value: any): any { if (isTemporal(value)) { const tag = value[Symbol.toStringTag] - if (tag === `Temporal.Duration`) { + if (tag === 'Temporal.Duration') { const fields = value as unknown as DurationFields - const exact = durationFields.map((field) => fields[field]).join(`,`) - const order = - fields.years || fields.months || fields.weeks - ? `` - : `time:${durationTimeNanoseconds(fields)}:` - return normalizedKey(`temporal`, `${tag}:${order}${exact}`) - } - if (tag === `Temporal.ZonedDateTime`) { - const epoch = (value as unknown as { epochNanoseconds: bigint }) - .epochNanoseconds - return normalizedKey(`temporal`, `${tag}:${epoch}:${value.toString()}`) + return normalizedKey( + tag, + durationFields.map((field) => fields[field]).join(','), + ) } - return normalizedKey(`temporal`, `${tag}:${value.toString()}`) + return normalizedKey(tag, value.toString()) } // Normalize Uint8Arrays/Buffers to a string representation for Map key usage @@ -322,19 +299,6 @@ export function normalizeValue(value: any): any { return value } -/** - * Normalize a value for BTree index usage. - * Extends normalizeValue to also convert undefined to a sentinel value. - * This is needed because the BTree does not properly support `undefined` as a key - * (it interprets undefined as "start from beginning" in nextHigherPair/nextLowerPair). - */ -export function normalizeForBTree(value: any): any { - if (value === undefined) { - return UNDEFINED_SENTINEL - } - return normalizeValue(value) -} - /** * Compare values using the equality semantics used by Map keys. */ @@ -342,17 +306,6 @@ export function areSameValueZeroEqual(a: unknown, b: unknown): boolean { return a === b || (Number.isNaN(a) && Number.isNaN(b)) } -/** - * Converts the `UNDEFINED_SENTINEL` back to `undefined`. - * Needed such that the sentinel is converted back to `undefined` before comparison. - */ -export function denormalizeUndefined(value: any): any { - if (value === UNDEFINED_SENTINEL) { - return undefined - } - return value -} - /** * Compare two Temporal values of the same type, returning -1, 0, or 1. * diff --git a/packages/db/tests/collection-indexes.test.ts b/packages/db/tests/collection-indexes.test.ts index db5ffee2b7..d08920b07d 100644 --- a/packages/db/tests/collection-indexes.test.ts +++ b/packages/db/tests/collection-indexes.test.ts @@ -23,7 +23,6 @@ import { expectIndexUsage, indexedKeysSet, orderedEntriesArray, - orderedEntriesArrayReversed, stripVirtualProps, valueMapData, withIndexTracking, @@ -1738,12 +1737,7 @@ describe(`Collection Indexes`, () => { const index = groupedCollection.createIndex((row) => row.value) expect(index.takeFromStart(2)).toEqual([`first`, `second`]) - expect(orderedEntriesArray(index)[0]?.[1]).toEqual( - new Set([`first`, `second`]), - ) - expect(orderedEntriesArrayReversed(index)[0]?.[1]).toEqual( - new Set([`first`, `second`]), - ) + expect(index.takeReversedFromEnd(2)).toEqual([`second`, `first`]) }) it(`should return all matching rows for a range predicate on a custom-comparator index`, async () => { diff --git a/packages/db/tests/duration-query-equality-oracle.test.ts b/packages/db/tests/duration-query-equality-oracle.test.ts index cc9c89becf..1651188f88 100644 --- a/packages/db/tests/duration-query-equality-oracle.test.ts +++ b/packages/db/tests/duration-query-equality-oracle.test.ts @@ -6,8 +6,10 @@ * The independent model reads the ten public fields rather than formatting a * Duration or importing query normalization. The fixed input grammar inserts * one ordinary Duration row, then probes an equal copy and a changed value - * for every field. Three further probes distinguish a same-display unit - * change, opposite signs, and a changed second nonzero field. + * for every field. A two-row calendar-unit history checks that equality + * indexing remains available when Duration has no intrinsic order without + * `relativeTo`. Three further probes distinguish a same-display unit change, + * opposite signs, and a changed second nonzero field. * Each direct BasicIndex/BTreeIndex exact lookup and each public Collection * `eq` read before and after indexing must return the model's exact key set at * that read. An index spy confirms the indexed public route ran. These checks @@ -171,3 +173,47 @@ it.each(indexTypes)( } }, ) + +// The model distinguishes year counts by their public fields. Both rows are +// legal equality inputs even though comparing their duration requires a +// relative date. The scan, direct index, and indexed Collection read observe +// exact key sets; this test makes no claim about their relative order. +it.each(indexTypes)( + `%s preserves equality indexing for multiple calendar Durations`, + async (_name, IndexType) => { + const rows = [ + { id: `one`, value: Temporal.Duration.from({ years: 1 }) }, + { id: `two`, value: Temporal.Duration.from({ years: 2 }) }, + ] + const probe = Temporal.Duration.from({ years: 2 }) + const expected = new Set( + rows.filter((row) => modelEqual(row.value, probe)).map((row) => row.id), + ) + const index = new IndexType(1, new PropRef([`value`])) + for (const row of rows) index.add(row.id, row) + expect(index.lookup(`eq`, probe)).toEqual(expected) + + const source = createCollection( + mockSyncCollectionOptions({ + id: `calendar-duration-equality-${IndexType.name}`, + getKey: (row: (typeof rows)[number]) => row.id, + initialData: rows, + autoIndex: `off`, + defaultIndexType: IndexType, + }), + ) + try { + await source.stateWhenReady() + const where = eq(new PropRef([`value`]), probe) + const read = () => + new Set(source.currentStateAsChanges({ where })!.map((row) => row.key)) + expect(read()).toEqual(expected) + const publicIndex = source.createIndex((row) => row.value) + const lookup = vi.spyOn(publicIndex, `lookup`) + expect(read()).toEqual(expected) + expect(lookup).toHaveBeenCalled() + } finally { + await source.cleanup() + } + }, +) diff --git a/packages/db/tests/index-update-oracle.property.test.ts b/packages/db/tests/index-update-oracle.property.test.ts index 90ff363935..8822b17223 100644 --- a/packages/db/tests/index-update-oracle.property.test.ts +++ b/packages/db/tests/index-update-oracle.property.test.ts @@ -5,17 +5,14 @@ import { createCollection } from '../src/collection/index.js' import { CollectionInErrorStateError } from '../src/errors.js' import { BasicIndex } from '../src/indexes/basic-index.js' import { BTreeIndex } from '../src/indexes/btree-index.js' +import { eq } from '../src/query/builder/functions.js' import { PropRef } from '../src/query/ir.js' import { DEFAULT_COMPARE_OPTIONS } from '../src/utils.js' -import { - indexedKeysSet, - mockSyncCollectionOptions, - orderedEntriesArray, - valueMapData, -} from './utils' +import { mockSyncCollectionOptions } from './utils' import { oraclePropertyOptions, oracleRuns } from './oracle-config' import type { BaseIndex, IndexInterface } from '../src/indexes/base-index.js' import type { CompareOptions } from '../src/query/builder/types.js' +import type { BasicExpression } from '../src/query/ir.js' import type { StringCollationConfig } from '../src/types.js' /** @@ -30,7 +27,7 @@ import type { StringCollationConfig } from '../src/types.js' * A native Map is the sole ownership model. After put, delete, or full build, * the oracle groups that Map by value and freshly sorts the groups. BasicIndex * and BTreeIndex must agree on key count, key membership, equality/range lookup, - * ordered entries, and rebuild behavior. Small duplicate values and signed zero + * full ordered reads, and rebuild behavior. Small duplicate values and signed zero * force collisions in the value groups without copying either index structure. * The comparison runs after every direct index action, after fresh build, and * after each named Collection read. Collection rows and their order are public; @@ -56,7 +53,7 @@ type IndexValue = number type IndexConstructor = new ( id: number, - expression: PropRef, + expression: BasicExpression, name?: string, options?: { compareFn?: (left: unknown, right: unknown) => number @@ -142,21 +139,22 @@ function expectIndexMatchesModel( rows: Map, ): void { const groups = groupKeysByValue(rows) + const orderedKeys = [...groups] + .sort(([left], [right]) => left - right) + .flatMap(([, keys]) => [...keys].sort()) expect(index.keyCount).toBe(rows.size) - expect(indexedKeysSet(index)).toEqual(new Set(rows.keys())) - expect(valueMapData(index)).toEqual(groups) - // The index may retain either zero sign as a bucket representative. Its - // comparator and the source Map place both signs in the same value group. - expect( - orderedEntriesArray(index).map(([value, keys]) => [ - value === 0 ? 0 : value, - keys, - ]), - ).toEqual([...groups].sort(([left], [right]) => left - right)) + // Complete returned arrays preserve key multiplicity and comparator order + // without requiring a particular bucket or ordered-storage representation. + expect(index.takeFromStart(rows.size + 1)).toEqual(orderedKeys) + expect(index.takeReversedFromEnd(rows.size + 1)).toEqual( + [...orderedKeys].reverse(), + ) for (const value of probeValues) { - expect(index.lookup(`eq`, value)).toEqual(groups.get(value) ?? new Set()) + const expected = groups.get(value) ?? new Set() + expect(index.equalityLookup(value)).toEqual(expected) + expect(index.lookup(`eq`, value)).toEqual(expected) } for (const boundary of rangeBoundaries) { @@ -348,6 +346,30 @@ describe.each(indexTypes)(`%s update properties`, (_indexName, IndexType) => { expect(index.canOptimizeRangeFor(100)).toBe(false) }) + test.each([ + [`duplicate key`, `takeFromStart`, [`a`, `a`]], + [`wrong reverse order`, `takeReversedFromEnd`, [`a`, `b`]], + ] as const)(`full traversal rejects a %s`, (_name, method, wrongKeys) => { + const index = new IndexType(1, new PropRef([`value`])) + index.add(`a`, { value: 1 }) + index.add(`b`, { value: 2 }) + const wrongOutput = vi.spyOn(index, method).mockReturnValue([...wrongKeys]) + try { + expect(() => + expectIndexMatchesModel( + index, + new Map([ + [`a`, 1], + [`b`, 2], + ]), + ), + ).toThrowError(/expected/) + expect(wrongOutput).toHaveBeenCalled() + } finally { + wrongOutput.mockRestore() + } + }) + test(`inclusive lower bound rejects an exclusive-boundary result`, () => { const index = new IndexType(1, new PropRef([`value`])) index.add(`boundary`, { value: -2 }) @@ -585,7 +607,9 @@ function expectExactMembership( // from source-value membership in the operand list, including an empty list. const groups = groupKeysByValue(rows) expect(index.keyCount).toBe(rows.size) - expect(indexedKeysSet(index)).toEqual(new Set(rows.keys())) + const traversedKeys = index.takeFromStart(rows.size + 1) + expect(traversedKeys).toHaveLength(rows.size) + expect(new Set(traversedKeys)).toEqual(new Set(rows.keys())) for (const value of exactProbes) { const expected = groups.get(value) ?? new Set() expect(index.equalityLookup(value)).toEqual(expected) @@ -1307,7 +1331,10 @@ describe.each(indexTypes)( * operation. The probe inserts or reads the non-numeric value `ann`. An * empty prefix is included: its first write compares nothing, so it must * succeed, and the next comparing operation must throw. Comparator - * transitivity is outside this owner. + * transitivity is outside this owner. A removal need not compare a surviving + * value with its neighbors: the former representative-promotion example + * prescribed one storage layout, while the rejected-removal witness above + * retains the atomicity law for comparisons that actually occur. */ describe.each(indexTypes)( `%s invalid comparator results`, @@ -1445,25 +1472,19 @@ describe.each(indexTypes)( expect(index.lookup(`eq`, `x`)).toEqual(new Set([`x`])) }) - // `A` and `a` share one comparator position. Removing the representative - // `A` promotes `a`, whose comparison with `z` is the only invalid pair. - test(`a rejected representative replacement leaves the index unchanged`, () => { - const { index, step } = createRecordedIndex((left, right) => { - if ([left, right].includes(`a`) && [left, right].includes(`z`)) - return NaN - const l = String(left).toLowerCase() - const r = String(right).toLowerCase() - return l === r ? 0 : l < r ? -1 : 1 - }) - expect(step(() => index.add(`upper`, { value: `A` }))).toBe(false) - expect(step(() => index.add(`lower`, { value: `a` }))).toBe(false) - expect(step(() => index.add(`zed`, { value: `z` }))).toBe(false) - // Equality lookups and key count read the index without comparing. - if (step(() => index.remove(`upper`, { value: `A` }))) { - expect(index.keyCount).toBe(3) - expect(index.lookup(`eq`, `A`)).toEqual(new Set([`upper`])) - expect(index.lookup(`eq`, `a`)).toEqual(new Set([`lower`])) - expect(index.lookup(`eq`, `z`)).toEqual(new Set([`zed`])) + test(`a rejected remove preserves other exact values`, () => { + const { index, step } = createRecordedIndex((left, right) => + left === `x` && right === `x` + ? NaN + : String(left).localeCompare(String(right)), + ) + for (const value of [`a`, `x`, `z`]) { + expect(step(() => index.add(value, { value }))).toBe(false) + } + expect(step(() => index.remove(`x`, { value: `x` }))).toBe(true) + expect(index.keyCount).toBe(3) + for (const value of [`a`, `x`, `z`]) { + expect(index.lookup(`eq`, value)).toEqual(new Set([value])) } }) @@ -1884,15 +1905,6 @@ describe.each(indexTypes)(`%s comparator groups`, (_indexName, IndexType) => { expect( subject.takeReversedFromEnd(currentRows.length + 1), ).toEqual(reversed) - for (const [representative, keys] of orderedEntriesArray( - subject, - )) { - expect( - currentRows.some( - (row) => row.value === representative && keys.has(row.key), - ), - ).toBe(true) - } for (const row of currentRows) { expect(subject.equalityLookup(row.value)).toEqual( new Set([row.key]), @@ -2539,3 +2551,186 @@ describe.each([ }) }) }) + +/** + * Binary equality is by byte content, so a fresh byte array can remove an + * earlier equal value. The source Map predicts exact membership after removal, + * re-addition, and replacement while another byte value remains. Complete + * index reads must contain each live key once; byte ordering is unspecified. + */ +describe.each(indexTypes)(`%s binary-value ownership`, (_name, IndexType) => { + test(`does not retain an ordered entry after equal-content removal`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + const bytes = (value = 1) => new Uint8Array([value]) + const check = () => { + const expected = new Set(rows.keys()) + expect(index.keyCount).toBe(rows.size) + expect(index.equalityLookup(bytes())).toEqual( + new Set( + [...rows].filter(([, value]) => value[0] === 1).map(([key]) => key), + ), + ) + for (const read of [ + index.takeFromStart(rows.size + 1), + index.takeReversedFromEnd(rows.size + 1), + ]) { + expect(read).toHaveLength(rows.size) + expect(new Set(read)).toEqual(expected) + } + } + + index.add(`a`, { value: bytes() }) + rows.set(`a`, bytes()) + check() + index.remove(`a`, { value: bytes() }) + rows.delete(`a`) + check() + index.add(`b`, { value: bytes() }) + rows.set(`b`, bytes()) + check() + + const oldBytes = bytes() + index.add(`c`, { value: bytes(2) }) + rows.set(`c`, bytes(2)) + index.add(`d`, { value: oldBytes }) + rows.set(`d`, oldBytes) + check() + index.remove(`b`, { value: bytes() }) + rows.delete(`b`) + check() + index.update(`d`, { value: oldBytes }, { value: bytes() }) + rows.set(`d`, bytes()) + check() + }) + + test.each([2, 8193])( + `compares every byte for exact membership (%i bytes)`, + (length) => { + const index = new IndexType(1, new PropRef([`value`])) + const left = new Uint8Array(length) + left[0] = 1 + const right = left.slice() + right[length - 1] = 1 + index.add(`left`, { value: left }) + index.add(`right`, { value: right }) + expect(index.equalityLookup(left.slice())).toEqual(new Set([`left`])) + expect(index.equalityLookup(right.slice())).toEqual(new Set([`right`])) + }, + ) +}) + +/** + * PlainMonthDay has exact equality but no intrinsic order. The two-row source + * Map predicts one key for a copy of February 1, both before and after index + * creation. The index may choose any storage order; a public equality read + * must not fail merely because both values are present. + */ +describe.each(indexTypes)( + `%s unordered Temporal equality`, + (_name, IndexType) => { + test(`accepts distinct PlainMonthDay values for exact lookup`, async () => { + const rows = [ + { id: `jan`, value: Temporal.PlainMonthDay.from(`--01-01`) }, + { id: `feb`, value: Temporal.PlainMonthDay.from(`--02-01`) }, + ] + const probe = Temporal.PlainMonthDay.from(`--02-01`) + const expected = new Set( + rows + .filter( + (row) => + row.value.monthCode === probe.monthCode && + row.value.day === probe.day, + ) + .map((row) => row.id), + ) + const index = new IndexType(1, new PropRef([`value`])) + for (const row of rows) index.add(row.id, row) + expect(index.lookup(`eq`, probe)).toEqual(expected) + + const collection = createCollection( + mockSyncCollectionOptions({ + id: `month-day-equality-${IndexType.name}`, + getKey: (row: (typeof rows)[number]) => row.id, + initialData: rows, + autoIndex: `off`, + defaultIndexType: IndexType, + }), + ) + await withCollectionCleanup(collection, async () => { + await collection.stateWhenReady() + const where = eq(new PropRef([`value`]), probe) + const read = () => + collection + .currentStateAsChanges({ where })! + .map((row) => row.key) + .sort() + expect(read()).toEqual([...expected].sort()) + const publicIndex = collection.createIndex((row) => row.value) + const lookup = vi.spyOn(publicIndex, `lookup`) + expect(read()).toEqual([...expected].sort()) + expect(lookup).toHaveBeenCalled() + }) + }) + }, +) + +/** + * A bounded read should not make more comparator calls as retired values + * accumulate. A fixed history retains one row while adding and removing 16, + * 64, or 256 earlier values. A user comparator records calls made by the + * public takeFromStart method after each history. This witnesses comparator + * work only: a traversal over tombstones that avoids the comparator could + * pass, and retained memory is measured separately. + */ +describe.each(indexTypes)(`%s retired-value read work`, (_name, IndexType) => { + test(`takeFromStart comparator work does not grow with retired values`, () => { + const comparisons = [16, 64, 256].map((retiredCount) => { + let calls = 0 + const index = new IndexType(1, new PropRef([`value`]), undefined, { + compareFn: (left: unknown, right: unknown) => { + calls++ + return (left as number) - (right as number) + }, + }) + index.add(`live`, { value: 1000 }) + for (let value = 0; value < retiredCount; value++) { + const key = String(value) + index.add(key, { value }) + index.remove(key, { value }) + } + expect(index.keyCount).toBe(1) + calls = 0 + expect(index.takeFromStart(1)).toEqual([`live`]) + return calls + }) + + expect(comparisons[2]).toBeLessThanOrEqual(comparisons[0]! + 4) + }) +}) + +/** + * Distinct exact values can share one comparator position. An ordered first + * row still requires sorting that group's keys, but it need not recompare + * every value in the group. The fixed 16/64/256 sizes distinguish a grouped + * read from one tree search per tied exact value. This bounds comparator work + * at the direct read checkpoint, not elapsed time or retained memory. + */ +test(`BTreeIndex bounded reads do not recompare every live tie`, () => { + const comparisons = [16, 64, 256].map((count) => { + let calls = 0 + const index = new BTreeIndex(1, new PropRef([`value`]), undefined, { + compareFn: () => { + calls++ + return 0 + }, + }) + for (let value = 0; value < count; value++) { + index.add(String(value), { value }) + } + calls = 0 + expect(index.takeFromStart(1)).toEqual([`0`]) + return calls + }) + expect(comparisons[2]).toBeLessThanOrEqual(comparisons[0]! + 4) +}) diff --git a/packages/db/tests/utils.ts b/packages/db/tests/utils.ts index c952b7674e..54267feecc 100644 --- a/packages/db/tests/utils.ts +++ b/packages/db/tests/utils.ts @@ -2,7 +2,7 @@ import { expect } from 'vitest' import { createCollection } from '../src/collection/index.js' import { BTreeIndex } from '../src/indexes/btree-index' import { withCollectionConfigFactory } from '../src/client' -import { denormalizeUndefined } from '../src/utils/comparison.js' +import { normalizeValue } from '../src/utils/comparison.js' import { CleanupQueue } from '../src/collection/cleanup-queue.js' import type { CollectionConfig, @@ -601,16 +601,25 @@ export function withExpectedRejection( type IndexInternals = { indexedKeys: Set } & ( | { sortedValues: Array; valueMap: Map> } | { - valueMap: Map }> + valueMap: Map< + unknown, + { + keys: Map + buckets?: Map> + } + > orderedEntries: { size: number - minKey: () => unknown - maxKey: () => unknown + minKey: () => { value: unknown } + maxKey: () => { value: unknown } forRange: ( low: unknown, high: unknown, includeHigh: boolean, - onFound: (key: unknown, bucket: { keys: Set }) => void, + onFound: ( + key: { value: unknown }, + group: { keys: Map }, + ) => void, ) => void } } @@ -632,12 +641,14 @@ export function indexedKeysSet(index: object): Set { } /** Test inspection of an index's value buckets keyed by indexed value. */ -export function valueMapData(index: object): Map> { +export function valueMapData( + index: object, +): Map, 'size' | 'has'>> { const [internals] = indexInternals(index) if (`sortedValues` in internals) return internals.valueMap - const result = new Map>() - for (const [key, bucket] of internals.valueMap) { - result.set(denormalizeUndefined(key), bucket.keys) + const result = new Map, 'size' | 'has'>>() + for (const [key, group] of internals.valueMap) { + result.set(key, group.buckets?.get(key) ?? group.keys) } return result } @@ -651,14 +662,14 @@ export function orderedEntriesArray( if (`sortedValues` in internals) { entries = internals.sortedValues.map((value) => [ value, - internals.valueMap.get(value) ?? new Set(), + internals.valueMap.get(normalizeValue(value)) ?? new Set(), ]) } else { const tree = internals.orderedEntries entries = [] if (tree.size > 0) { - tree.forRange(tree.minKey(), tree.maxKey(), true, (key, bucket) => { - entries.push([denormalizeUndefined(key), bucket.keys]) + tree.forRange(tree.minKey(), tree.maxKey(), true, (key, group) => { + entries.push([key.value, new Set(group.keys.keys())]) }) } } From 905b933231642f016a948aef8828abd1f6aa4900 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 14:48:20 -0600 Subject: [PATCH 40/46] test: close binary oracle reach and trim index storage --- docs/contributing/oracle-coverage.md | 5 +++-- .../pr-2116-deferred-findings.md | 1 + packages/db/src/indexes/basic-index.ts | 20 +++---------------- packages/db/src/utils/comparison.ts | 4 ---- .../index-update-oracle.property.test.ts | 7 +++++++ 5 files changed, 14 insertions(+), 23 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 6bfba770ae..5e1c0260ca 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -496,8 +496,9 @@ returns a non-number or `NaN` and checks rejected removal atomicity; it does not require a comparison merely because one comparator-equal row is removed. The binary lane checks fresh equal-content removal, re-addition, and replacement beside another byte value. It compares complete forward and -reverse key reads to a source Map and exact membership for short and wide -values that differ in their last byte. A pre-repair BTree lost a surviving key +reverse key reads to a source Map. Separate short and 8193-byte pairs check +exact membership and complete forward and reverse key conservation when only +the last byte differs. An intermediate PR BTree rewrite lost a surviving key after one equal-content owner left; the expanded lane rejected that result. This owner does not prescribe mixed-kind order or cover arbitrary custom index callbacks, binary range ordering, all JavaScript value kinds, or general diff --git a/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md b/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md index 8bf29842c7..dc6dbad3e3 100644 --- a/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md +++ b/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md @@ -12,6 +12,7 @@ not closed by passing a neighboring transfer oracle. | R29: two valid `Temporal.Duration` values can have the same display text but different fields. A changed field can be suppressed as an equal Collection value. | Closed in this PR for the tested direct and Collection paths. The Duration field model checks all ten fields; a same-display replacement and adjacent-unit histories make the former `deepEquals` and change-event behavior fail before the repair. The query equality owner and `in` evaluator owner separately check Duration field identity. | General multirow and live-query publication histories remain outside these owners. Add them to the relevant owner if a new failure reaches those paths. | | R16: an indexed Collection can admit a valid Temporal row whose index comparator has no public order, then throw after partial state changes. | Earlier PR revisions recorded RED admission and publication traces for Duration, PlainMonthDay, and mixed-kind rows. Source inspection suggests the risk remains on `origin/main`, but this narrowed PR has not executed a current-base RED witness. The public order law and internal index storage order must be distinguished. | The direct-index and Collection publication owners should pair a valid-row admission history with exact membership, ordered storage, stored rows, and subscriber observations. Choose the smallest internal ordering or atomic-admission repair that satisfies the accepted contract. | | Mixed-kind indexed order and custom-index query selection may disagree with public query results. | The mixed number/string direct-index history confirms key duplication after deletion on `origin/main`; the new source-Map lane closes that bounded key-conservation bug. A reverse manual index also returned the wrong first row through `currentStateAsChanges`; the new selection oracle closes that bounded public bug. The user withdrew a per-row custom callback position guarantee for comparator-equal exact values. | A general public order policy for unlike kinds remains open. The current owners prove key conservation and the two-rank manual-index selection case, not arbitrary mixed-kind order, custom callback ties, or later live updates. | +| Indexed ordering can suppress a required Temporal comparison error. | Changelog #1519 says ordering mixed Temporal kinds throws `TypeError`. At executable head `2f852dcd1` and clean `origin/main` `bbbf017ef5`, a ready two-row Collection scan throws for `PlainDate` beside `PlainDateTime`; after creating either BasicIndex or BTreeIndex, the same ascending `currentStateAsChanges` read returns both rows. Two distinct `PlainMonthDay` values give the same scan/index disagreement because that type has no static order. This is a pre-existing public-query defect, not a regression caused by this rewrite. | The Temporal index-order owner should add a scan/index exception witness for mixed or unordered Temporal values, then the ordered-query index selection path should preserve the public error while retaining exact equality indexing. The present oracle deliberately covers ordered ZonedDateTimes and time-only Durations only. On-demand acquisition and later updates need separate witnesses. | | R17: an equal-time Date removal left an ordered ghost under added index machinery. | The recorded RED trace depended on code introduced during this PR. It does not establish an `origin/main` defect. The stronger rule for a substituted old row with a different callback position was withdrawn. | If ordinary direct remove/update still has a current-base counterexample, add a source-Map history to the direct-index owner before changing production. | | PlainDate extended-year order under lexical string collation may diverge between scans and indexes. | On both the recorded `origin/main` baseline and this branch, a direct scan comparator orders `-100000-01-01` before `+010000-01-01`, while direct BasicIndex and BTreeIndex traversal returns the positive date first with `stringSort: 'lexical'`. This was found during hostile review of the final kind classifier, not in the retired unit-test ledger. A public indexed query witness has not run. | Extend the Temporal index-order owner with a scan/indexed public read under lexical collation, confirm the contract for this other Temporal kind, and reproduce RED before a separate repair. The present owner is bounded to ZonedDateTime and time-only Duration. | | Cross-realm native equality was asserted by a surviving request-transport unit test. | PR #1797 captured the then-current `instanceof` result. No public realm-distinctness promise was found. The maintainer questioned that distinction. This PR removes that incidental test expectation and makes no cross-realm equality claim. | A future public contract decision needs a host-boundary witness before either equality policy becomes an oracle law. | diff --git a/packages/db/src/indexes/basic-index.ts b/packages/db/src/indexes/basic-index.ts index 22e30c5365..110892a821 100644 --- a/packages/db/src/indexes/basic-index.ts +++ b/packages/db/src/indexes/basic-index.ts @@ -1,10 +1,5 @@ import { compareKeys } from '@tanstack/db-ivm' -import { - areSameValueZeroEqual, - cacheBinaryKey, - isUint8Array, - normalizeValue, -} from '../utils/comparison.js' +import { areSameValueZeroEqual, normalizeValue } from '../utils/comparison.js' import { codedMessage, devBuild } from '../error-message.js' import { compareKeysReversed, @@ -29,15 +24,6 @@ function orderValue(value: unknown): unknown { return value instanceof Date ? value.getTime() : value } -function storedOrderValue(value: unknown, exact: unknown): unknown { - if (isUint8Array(value)) { - const copy = new Uint8Array(value) - cacheBinaryKey(copy, exact as string) - return copy - } - return orderValue(value) -} - /** * Basic index using Map + sorted Array. * @@ -83,7 +69,7 @@ export class BasicIndex< } else { // New value. Find its sorted position first: a comparator failure must // throw before the index changes. - const orderedValue = storedOrderValue(value, normalizedValue) + const orderedValue = orderValue(value) const insertIdx = findInsertPositionInArray( this.sortedValues, orderedValue, @@ -200,7 +186,7 @@ export class BasicIndex< if (keys) keys.add(key) else { this.valueMap.set(value, new Set([key])) - this.sortedValues.push(storedOrderValue(indexedValue, value)) + this.sortedValues.push(orderValue(indexedValue)) } this.addRangeValue(indexedValue) this.indexedKeys.add(key) diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index e40809d305..4cad6ffb81 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -229,10 +229,6 @@ function normalizedKey(kind: string, value: string): string { const binaryKeys = new WeakMap() -export function cacheBinaryKey(value: Uint8Array, key: string): void { - binaryKeys.set(value, key) -} - function normalizeBinary(value: Uint8Array): string { const cached = binaryKeys.get(value) if (cached) return cached diff --git a/packages/db/tests/index-update-oracle.property.test.ts b/packages/db/tests/index-update-oracle.property.test.ts index 8822b17223..bf11bd515c 100644 --- a/packages/db/tests/index-update-oracle.property.test.ts +++ b/packages/db/tests/index-update-oracle.property.test.ts @@ -2616,6 +2616,13 @@ describe.each(indexTypes)(`%s binary-value ownership`, (_name, IndexType) => { index.add(`right`, { value: right }) expect(index.equalityLookup(left.slice())).toEqual(new Set([`left`])) expect(index.equalityLookup(right.slice())).toEqual(new Set([`right`])) + for (const read of [ + index.takeFromStart(3), + index.takeReversedFromEnd(3), + ]) { + expect(read).toHaveLength(2) + expect(new Set(read)).toEqual(new Set([`left`, `right`])) + } }, ) }) From 4216a5783a5c31d2358a5d1ce0113d9125d59b83 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 14:49:44 -0600 Subject: [PATCH 41/46] docs: record exact-head hostile oracle review and code weight --- .../oracle-reviews/pr-2116-confirmed-fixes.md | 82 ++++++++++++++++--- 1 file changed, 72 insertions(+), 10 deletions(-) diff --git a/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md b/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md index 8f3902d4c5..ceed155254 100644 --- a/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md +++ b/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md @@ -24,8 +24,8 @@ were proposed extensions, then withdrawn by the maintainer. This PR does not promise behavior after callers modify standard data objects in place, unsigned byte ordering, or special handling for malformed Temporal lookalikes. The [deferred record](pr-2116-deferred-findings.md) keeps valid Temporal admission, -mixed-kind query order, and earlier-branch observations separate from the -confirmed fixes above. +mixed-kind query order, the confirmed pre-existing indexed Temporal exception +gap, and earlier-branch observations separate from the confirmed fixes above. The repeatable workflow is: freeze source assertions in a loss ledger; name the receiving law and its limits; obtain a baseline failure before production @@ -38,12 +38,74 @@ supports only its legal histories, paths, and observations. ## Production-code weight Against the recorded `origin/main` baseline, the source diff under -`packages/db/src` adds 205 lines and removes 222, for **17 fewer production +`packages/db/src` adds 360 lines and removes 536, for **176 fewer production lines**. The built DB ESM entry bundled with esbuild (`bundle`, `minify`, -`format: esm`, `platform: neutral`, `NODE_ENV=production`; DB IVM, pacer-lite, -and standard-schema external) is 334,795 bytes / 100,227 gzip at baseline and -335,754 bytes / 100,551 gzip for this change. That is **959 more minified -bytes / 324 more gzip bytes**. The source-line budget is negative; the bundle -budget is still positive. The remaining runtime addition is concentrated in -distinct index-storage comparison and exact Temporal keys, which the bounded -fixes above require. Do not treat source-line reduction as a bundle-size win. +`format: esm`, `platform: neutral`, `target: es2020`, +`NODE_ENV=production`; DB IVM, pacer-lite, and standard-schema external) is +335,613 bytes / 100,492 gzip at baseline and 336,504 bytes / 100,889 gzip at +executable head `905b93323`. That is **891 more minified bytes / 397 more gzip +bytes**. Both outputs use Node's `zlib.gzipSync`. The source-line budget is +negative; the consumer-bundle budget remains positive. + +## Exact-head hostile review of the index rewrite + +The executable revision is `905b93323` (following `2f852dcd1`); the baseline +is `origin/main` `bbbf017ef5b3884515295a4225faacdfd814499e`. The rewrite +replaces BTree's per-exact ordered entries with comparator groups while keeping +exact membership separate. Its binary ownership history compares a source Map +with complete forward and reverse index reads. An intermediate PR rewrite lost +a surviving byte owner after a fresh equal-content removal. The published +parent `78bd4946b` passes the six final binary cases, so that particular RED +is **not** evidence of a pre-existing defect. + +The hostile reviewer ran two wrong-output controls against the rewritten +BTree. One suppressed a surviving short-byte key after equal-content removal; +the oracle failed at the ordered-read length comparison. A second suppressed +one key after inserting two 8193-byte values that differ only in their last +byte. That mutant survived the first version of the binary lane because the +wide-byte case checked exact lookup only. The final oracle adds full forward +and reverse key conservation there; the same mutant then failed at that +comparison. The unmodified six cases passed. These controls are output +mutants, not claims that `origin/main` failed. + +The final BasicIndex retains the original stable byte value in its sorted +array and reuses the normalization cache populated at admission. A direct +offset-view and Buffer probe preserved exact membership and ordered keys after +fresh equal-content removal. Callers who edit the retained byte array in +place have no promise. This removes a redundant byte copy without changing +the accepted immutable-value law. Duplicate-heavy insert and removal timings +were near parity in the bounded A/B probe. An attempted merge of Array and +ordinary-object index kinds was discarded: it produced a strict three-value +comparator cycle, so the separate kind remains. + +The fresh review also reconstructed a public scan/index mismatch for mixed +`PlainDate`/`PlainDateTime` and for two `PlainMonthDay` values. The scan throws +as changelog #1519 requires, while either index returns rows. It reproduces +on clean `origin/main` and this head. The [deferred record](pr-2116-deferred-findings.md) +assigns the exception witness and ordered-query repair to the Temporal owner; +the bounded ZonedDateTime and time-only Duration claims above do not include +this case. + +| Guide requirement | Exact-head result | +| --- | --- | +| ORC-001 contract and limits | Transfer ledgers trace the retired assertions; changelog #1519 and the surviving index interface give the separate Temporal and key-conservation laws. The coverage map names excluded mixed-kind order, mutation of standard values, and wider histories. | +| ORC-002 independent judgment | Source Maps, supplied Duration fields, and supplied Temporal instants predict results without importing index grouping, normalization, or comparator classifications. | +| ORC-003 literate responsibilities | The owners state each law before mechanics and explain the model, legal actions, direct or public production path, observation, and checkpoint beside the code. | +| ORC-004 generated grammar | The transfer ledgers account for reconstruction, ablation, bounded ranges, and excluded histories of the existing generated index lanes. The new binary lane is a fixed history, so the generated-grammar trigger does not apply to that lane. | +| ORC-005 path and observation | Complete returned arrays expose missing or duplicate keys in both directions; exact lookups expose byte-content errors. Collection reads and index spies establish the named public routes. No earlier-publication information claim is made. | +| ORC-006 checker calibration | Short-byte and 8193-byte wrong-output mutants failed at ordered-read assertions; the wide mutant first survived, motivating the final assertion. Existing transfer ledgers retain the other hostile controls. | +| ORC-007 fixed and random replay | The important generated index properties retain their documented fixed and seedless campaigns and direct seed/path replay. The fixed binary examples do not trigger this requirement. | +| ORC-008 model state | The source Map retains key ownership because a later delete or replacement can distinguish live rows; the binary fixed history adds no production-shaped index state. | +| ORC-009 vocabulary | Model puts, deletes, source keys, and ordered reads map to direct index actions and public observations. Comparator groups remain production storage, not reference-model state. | +| ORC-010 failure and cleanup | Existing generated drivers retain their replay context and failure cause; public Collection witnesses clean up in `finally`. The fixed direct binary cases acquire no external resource. | +| ORC-011 second formulation | No plausible fault shared by the source Map and index ownership model was identified inside the bounded transfer. The separately confirmed mixed-Temporal public error is recorded as an unresolved neighboring law, rather than inferred from direct index equality. | +| ORC-012 review record | This versioned record identifies the executable head, baseline, mutant outcomes, tests, size, and unresolved public path. | +| ORC-013 boundary witness | Nullish boundary controls, opposing 9ns/10ns display and instant order, adjacent large-epoch nanoseconds, and Duration display ties reject nearby wrong rules at their named checkpoints. | +| ORC-014 controlled premise | The controlled Collection fixtures support only their named Collection observations; no real-provider behavior is claimed, so the cross-provider handoff trigger does not apply. | + +The DB source suite passed 13,192 tests in 282 files before the no-copy +follow-up. After that follow-up, the affected index and comparison files passed +623 tests, the built-package lane passed 291 tests, the minified DB check +passed, and the DB source TypeScript and affected-file lint checks passed. +Those runs cover the changed immutable-value paths; they do not establish +arbitrary later Collection histories or the deferred mixed-Temporal error. From ad6f22105d2fe9e619e3c4079cc68d618af2920f Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 14:57:11 -0600 Subject: [PATCH 42/46] build(db): refresh safe private-member mangle cache --- packages/db/mangle-cache.json | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/packages/db/mangle-cache.json b/packages/db/mangle-cache.json index a786f671cd..ab5a0ab8cc 100644 --- a/packages/db/mangle-cache.json +++ b/packages/db/mangle-cache.json @@ -297,7 +297,6 @@ "loadPrefix": "e$", "activateLease": "e0", "calculateSize": "e1", - "compareStored": "e2", "invokeCleanup": "e3", "onceCallbacks": "e4", "publishEvents": "e5", @@ -380,5 +379,28 @@ "readyFailureSink": "gn", "writeExplicitRowMetadata": "a8", "applyOwnedMutations": "bX", - "commitDirect": "cn" + "commitDirect": "cn", + "defersAcquisition": "c_", + "_getCurrentBindings": "dH", + "subscriberOrPreloadListeners": "ex", + "changeAcquiringSubscribers": "e2", + "discardPreloadedLiveQuery": "fU", + "pendingInitialLoads": "f1", + "subscriberOrPreload": "f2", + "readsBoundedPrefix": "gp", + "captureMutations": "gq", + "startSyncIfStopped": "gr", + "trackInitialLoad": "gs", + "nullsFirst": "gu", + "sourcesPermitInitialReadiness": "gv", + "startLiveQueryPreload": "gw", + "getSourceInfo": "gx", + "publishReplayRows": "gy", + "afterPreloadMark": "gz", + "resumeSubscriber": "gA", + "beginWrite": "gB", + "detach": "gC", + "copyRows": "gD", + "restore": "gE", + "attach": "gF" } From a5c7e43733b81e807a348e4aaf436daea4bbb1a5 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 15:02:59 -0600 Subject: [PATCH 43/46] docs(db): record final package and consumer size measures --- .../oracle-reviews/pr-2116-confirmed-fixes.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md b/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md index ceed155254..c833e7b4a1 100644 --- a/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md +++ b/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md @@ -47,6 +47,16 @@ executable head `905b93323`. That is **891 more minified bytes / 397 more gzip bytes**. Both outputs use Node's `zlib.gzipSync`. The source-line budget is negative; the consumer-bundle budget remains positive. +The later build-cache commit `ad6f22105` removes the obsolete `compareStored` +minifier entry and registers 23 private members that the cache check recognizes +as safe. Production TypeScript is unchanged from the reviewed executable head. +After a fresh build, the same consumer-bundle measurement is 335,182 bytes / +100,737 gzip: **431 fewer minified bytes / 245 more gzip bytes** than baseline. +The local cache check and minified public API check pass. This narrows, but does +not close, the consumer-bundle gap. The repository's separate compressed-size +check reports **395 fewer bytes** across its published ESM package files at +`ad6f22105`; that unbundled package measure is net negative. + ## Exact-head hostile review of the index rewrite The executable revision is `905b93323` (following `2f852dcd1`); the baseline From 16b2192243978dfc1c0df4ab70837d1ba9078ed9 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 15:06:00 -0600 Subject: [PATCH 44/46] docs(db): correct production diff count --- docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md b/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md index c833e7b4a1..783adae0c9 100644 --- a/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md +++ b/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md @@ -38,7 +38,7 @@ supports only its legal histories, paths, and observations. ## Production-code weight Against the recorded `origin/main` baseline, the source diff under -`packages/db/src` adds 360 lines and removes 536, for **176 fewer production +`packages/db/src` adds 356 lines and removes 535, for **179 fewer production lines**. The built DB ESM entry bundled with esbuild (`bundle`, `minify`, `format: esm`, `platform: neutral`, `target: es2020`, `NODE_ENV=production`; DB IVM, pacer-lite, and standard-schema external) is From 0c10ae8e870cee576e6c61b83ef36980725b4e90 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 16:54:34 -0600 Subject: [PATCH 45/46] fix(db): preserve ordered query results across index value domains --- .changeset/fix-basic-index-nullish-build.md | 2 +- docs/contributing/oracle-coverage.md | 34 +- .../oracle-reviews/pr-2116-confirmed-fixes.md | 66 ++- .../pr-2116-deferred-findings.md | 19 +- packages/db/mangle-cache.json | 3 +- packages/db/package.json | 2 +- packages/db/src/collection/change-events.ts | 7 +- packages/db/src/indexes/auto-index.ts | 4 +- packages/db/src/indexes/base-index.ts | 33 +- packages/db/src/indexes/basic-index.ts | 20 +- packages/db/src/indexes/btree-index.ts | 1 - packages/db/src/indexes/reverse-index.ts | 4 + packages/db/src/query/compiler/order-by.ts | 4 +- packages/db/src/utils.ts | 2 +- packages/db/src/utils/comparison.ts | 5 +- packages/db/tests/basic-index-work.test.ts | 28 ++ packages/db/tests/btree-index-work.test.ts | 39 +- .../index-update-oracle.property.test.ts | 63 +++ ...ustom-index-order-selection-oracle.test.ts | 80 +++- .../query/mixed-index-order-oracle.test.ts | 449 ++++++++++++++++++ .../tests/temporal-index-order-oracle.test.ts | 194 +++++++- 21 files changed, 996 insertions(+), 63 deletions(-) create mode 100644 packages/db/tests/query/mixed-index-order-oracle.test.ts diff --git a/.changeset/fix-basic-index-nullish-build.md b/.changeset/fix-basic-index-nullish-build.md index c0610889b1..e5553a1c33 100644 --- a/.changeset/fix-basic-index-nullish-build.md +++ b/.changeset/fix-basic-index-nullish-build.md @@ -2,4 +2,4 @@ '@tanstack/db': patch --- -Keep `undefined` in the configured nulls-first position when BasicIndex builds from existing rows. Preserve indexed key ownership and public string order across mixed and prefixed values. Make Duration equality follow its fields, order indexed ZonedDateTimes by instant and time-only Durations by elapsed time, and ignore a manual index whose order conflicts with a limited public read. +Keep `undefined` in the configured nulls-first position when BasicIndex builds from existing rows. Preserve indexed key ownership and public query order across mixed and prefixed values. Make Duration equality follow its fields; order indexed ZonedDateTimes by instant, PlainDates by calendar, and time-only Durations by elapsed time; and preserve required errors for mixed or unordered Temporal values. Keep custom index callback inputs consistent across index types and ignore a manual index whose order conflicts with a limited public read. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 5e1c0260ca..f98064768e 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -83,6 +83,7 @@ that test identifiers must copy production's private data structures. | Offline execution | Complete | FIFO retry, scheduler eligibility, leadership replay, transaction settlement, and typed wire serialization are literate. The settlement oracle's generated FIFO histories draw each transaction as one record (outcome; manual commit, auto-commit with the option set or omitted, or an `OfflineAction`; receipt read or not), use `isPersisted` as an auto-commit's receipt, and require that no failure escapes as an unhandled rejection, also when nobody reads the receipt, and that settlement logs no failure again and writes no debug output. An empty transaction on any commit path may precede each one; its receipt and completion waiter must fulfil without reaching the provider or the outbox. A manual transaction that writes a row and rolls back before it commits may also precede each one; its waiter must reject with an Error that names the rollback. A focused witness rolls a transaction back while it persists, for both provider outcomes: its waiter must wait for the executor and report the executor's outcome. After every transaction settles, a new waiter for each one must reject within a turn rather than hang. An unread receipt still settles with its own outcome when read afterwards, and a caller may register its completion waiter before admission on every path but `OfflineAction`. A focused witness fails the admission outbox write on every commit path and requires the receipt to reject with the storage error, without an unhandled rejection or a repeated log; a failed outbox write stops the executor, so generated histories do not continue past one. Open: the generated failure source is the mutation function only; manual commits always follow their own `mutate()` at once, so a pending manual transaction behind a failing shared-key sibling is not generated; a throwing `mutate()` callback is not generated, because core keeps its transaction pending; a rolled-back-before-commit transaction is generated only on the manual path, with its own key and its waiter registered after `mutate()`, so a secondary rollback of a pending transaction by a failing sibling, whose waiter gets the generic rollback Error, is not generated. | | Frameworks | Complete | Shared live-query and infinite-query models are literate. Each framework keeps its own realm, ownership, and scheduling driver. | | Structural values and ordered primitives | Complete | Structural hashing, deep equality, comparison, cursor denotation, index refinement, and query-identity output equivalence are literate. | +| Mixed-kind ordered prefixes | Bounded | Eager BasicIndex and BTreeIndex paths compare ascending and descending direct and live-query windows with an unindexed scan across mixed value kinds and paired source changes. One retained on-demand BTree history checks settled windows after a number/bigint transition, a finite request for a missing row, and a cursor request after index eligibility changes. Other provider paging histories remain open. | | Boundary refinements | Complete | Cleanup/restart admission, metadata publication, retained state, acquisition cells, D2 source reconciliation, top-K support windows, and nested Query work bounds are literate. | | Small structures and test mechanics | Complete | SortedMap, cleanup appointments, and guarded replay are literate. | @@ -443,6 +444,7 @@ generic visitor behavior outside the named collectors. | Window-controller overlap | [shared infinite-query conformance](https://github.com/TanStack/db/blob/main/packages/db/tests/conformance/infinite-suite-oracle.ts), [React driver](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-conformance.test.tsx), [Vue driver](https://github.com/TanStack/db/blob/main/packages/vue-db/tests/infinite-query-conformance.test.ts), [Svelte driver](https://github.com/TanStack/db/blob/main/packages/svelte-db/tests/infinite-query-conformance.svelte.test.ts), [review record](oracle-reviews/dec-03-window-overlap.md) | Four or five ordered source rows, two window-settlement orders, and public snapshots after each settlement, a subscribed observer update, a detached controller read, and explicit recovery. Insertion and removal of the fifth row distinguish both continuation directions while an earlier error remains visible. The detached read follows a source change with no controller subscriber; its preloaded Collection retains the five-row window. The exact overlap uses the exported DB controller in each package realm. Framework hooks expose no controller preload, so this cell does not establish a hook scheduling path; direct hook paging remains in the adjacent shared scenarios. An unsubscribed controller has no notification claim. | | Infinite-query publication | [shared publication laws](https://github.com/TanStack/db/blob/main/packages/db/tests/conformance/infinite-suite-oracle.ts), [React render cuts](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-render-cuts-oracle.test.tsx), [Svelte hook tests](https://github.com/TanStack/db/blob/main/packages/svelte-db/tests/useLiveInfiniteQuery.svelte.test.ts), [review record](oracle-reviews/2026-10-05-infinite-first-paint.md) | Each driver records every value its framework published: React layout commits, Vue pre-flushes, and the Svelte construction value plus pre-effect runs. The shared suite checks that a synchronous source shows its ready first page on the first value, for a query callback and for a supplied collection that has not started, that every ready value in a fixed-source, fixed-query scenario equals the source prefix for its own page count, including an equal dependency replacement that keeps page depth, and that a mount requests an on-demand first window once, for a query callback and for a supplied collection whose finite window is wider than or equal to the one the hook needs, judged by the rows the on-demand source holds after mount. Under deferred acquisition, an on-demand first value is either the ready first page or a value with no rows, by framework: React and Svelte publish before the hook subscribes, Vue after. `preloaded-collection-first-paint` checks the documented route-loader pattern: a limited live query preloaded before mount shows its ready first page on the first value in every framework and asks the source for nothing more; a preload that does not count as demand never settles there. Pre-commit work has no shared form, because Vue setup is its commit. The React driver owns renders that never commit, StrictMode double renders (one first-window request on React 19), retries after GC, a mounted replacement whose startup error must reach the error boundary, and a collection kept from an abandoned render that enters terminal error before its query is selected again, each as under `useLiveQuery`. The React source ID reuse oracle drives the infinite hook too. Svelte owns construction and superseded recomputes before the subscribing effect. In both, the hook acquires an on-demand source as often as `useLiveQuery` does under the same history, for either input form, direct or behind a live-query Collection. On-demand requests from renders that never commit are now owned by the live-query deferred acquisition oracle; the React render-cut oracle checks that an abandoned render makes no acquisition. Open cells, owned here: React 18 StrictMode, where neither hook keeps refs across the double render; a React commit held after the 50 ms unsubscribed GC floor, where a controlled-premise controller witness shows no non-ready value but no React scheduling witness exists; ready values in scenarios that write the source or replace the query, which need a per-step source model; a join of eager and on-demand sources, which needs a zero-acquisition check before commit; a supplied collection with no `.limit`, which sends one unbounded request on `main` too, because the controller cannot narrow it before the source request; and Svelte `state_unsafe_mutation` when a start inside `$derived.by` synchronously writes rows a peer is waiting for, shared with Svelte's `useLiveQuery`, which a two-component probe reproduces and the deferred-acquisition follow-up removes. | | Structural values and ordered primitives | [hash values](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-oracle.property.test.ts), [hash identity](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-identity-oracle.property.test.ts), [MultiSet consolidation](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/multiset-consolidate-oracle.property.test.ts), [hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-graph-oracle.property.test.ts), [mixed hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-mixed-graph-oracle.property.test.ts), [hash retry](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-failure-retry-oracle.property.test.ts), [murmur stream](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/murmur-stream-oracle.property.test.ts), [comparison](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts), [deep equality](https://github.com/TanStack/db/blob/main/packages/db/tests/utils-oracle.property.test.ts), [cursor](https://github.com/TanStack/db/blob/main/packages/db/tests/cursor-oracle.property.test.ts), [indexes](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update-oracle.property.test.ts), [BTree Map](https://github.com/TanStack/db/blob/main/packages/db/tests/btree-map-oracle.test.ts), [query identity](https://github.com/TanStack/db/blob/main/packages/db/tests/query/identity-output-shape-oracle.test.ts), [LIKE semantics](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/evaluators-oracle.test.ts) | Independent flat values, graph topology, algebraic laws, structural object keys in code-unit order (the flat-value, identity and consolidation grammars include keys that tie under locale collation), murmur byte framing against a byte-at-a-time reference (block mixing, the finalizer and symbol framing have no owner there), Map/group/sort recomputation, expression denotation, custom comparator dispatch/reference identity/index compatibility, LIKE wildcard refinement, compiled output bags, declared value-pair agreement between D2 equality keys and IR equality operands, and Collection collation with exact public scan/index order and one-index reuse across repeated reads. The auto-index cells cross BasicIndex and BTreeIndex; scan cells omit indexes. Both paths cover lexical and numeric `en-US` locale defaults plus an explicit ordinary locale override of a numeric default. An unindexed six-row work cell bounds Collection collation resolution to at most two reads per ordered scan: one index-path check and one scan clause. Other locales and ordered histories are outside this owner. The identity pair grammar covers primitives, signed zero, NaN/invalid Date, valid Date/timestamp, binary/Buffer, Temporal, nested references, symbols, and functions; fixed and sampled pairs do not prove every JavaScript value or hash collision freedom. Ref proxies are expressions rather than literal values in this lane. The deep-equality owner also pins that `deepEquals` ignores Map and Set insertion order, RegExp `lastIndex`, array holes, and typed-array class, the rules draft equality keeps. It also pins that objects of another class differ, that plain and null-prototype objects are one class, that a class instance without enumerable keys equals only itself, that URLs compare by `href`, and that typed-array elements compare like numbers. The hash-values owner checks equal-hash consequences for independently constructed flat values. Distinct type identities are checked by the hash-identity owner through `equalHashValues`, including forced digest collisions; distinct 32-bit digests are not promised. The hash-identity owner checks `hash` and `equalHashValues` against one spec-level identity model: primitives with signed zero, NaN, and bigint; reference leaves (functions, registered handles, `File`, binary over 128 bytes); Date, binary, Temporal, and RegExp headers; array length and holes; Map and Set insertion order; and object keys in any order, for acyclic values and for cycles through arrays, Map values, Sets, and plain objects. Map/Set order sensitivity is pinned current behavior, not a promise. Arrays from another realm, RegExps with a `NaN` `lastIndex`, getters, proxies, other cycle carriers, and pairs that differ only in a back-edge target are outside its grammar; pinned cases keep equality's cross-realm length check and strict RegExp field comparison. `hash-work.test.ts` pins how visited values and array/RegExp headers count toward the structural work cap, and that equality copies a shared Map's entries once per distinct pair ([code-weight hash dispatch review](oracle-reviews/code-weight-hash-dispatch.md)). Custom string indexes do not optimize ordinary ranges, and custom cursors remain unsupported. The LIKE owner covers boolean string matching and bounded work, not nullish three-valued logic or a general Unicode collation contract. Unsupported composite cursors reject. The MultiSet consolidation owner checks keyed, single-type, and structural identity, signed zero and NaN in keyed and single-number data, the retained first record, zero-sum removal, and input record contents. It does not assert output order. Its keyed grammar includes cross-type key and value collisions, non-finite numeric keys, the `\|` delimiter, symbol and function identity, and a direct-object/object-tuple delimiter control from #1948. Eight named collision witnesses and both generated campaigns are RED on `c879ba6d` and GREEN with the repair ([#1948 review record](oracle-reviews/issue-1948-keyed-consolidation.md)); the [code-weight consolidation review](oracle-reviews/code-weight-multiset-consolidation.md) records the earlier exclusion. The keyed reference uses direct value/reference equality, including SameValueZero for numeric keys. The unkeyed fallback keeps its original key and value domains. The direct `MultiSet` owner does not prove which `@tanstack/db` query shapes produce primitive keyed values or mixed-type source keys; that needs a live-query production witness. The index owner enforces one invalid-comparator law across both BasicIndex and BTreeIndex: for each comparator, accepted numeric prefix (including the empty prefix), and probe operation (add, update, eq/range lookup, take, build), the operation throws exactly when it receives a `NaN` or non-number result and never when every comparison is valid. A `signed infinity` comparator is the valid control, and a custom collation `compare` supplied through `compareOptions` is checked the same way. Two BTree Map split histories check inserted-key reads, exact whole-range payloads/order, size, and extrema, proving split placement follows the insertion index without a post-mutation comparison. A rejected index add or remove leaves the index refining its accepted rows; update and build are not atomic. At the Collection boundary, both index types and both write paths (optimistic insert, sync commit) crash the collection: status becomes `error` and the next mutation throws, so a usable collection never holds rows its subscribers were not told about. Open: a sync source can still commit writes on a collection in `error` state, which stores unpublished rows; this is existing `markError` behavior and needs a lifecycle-owner witness. Comparator transitivity is outside this evidence. | +| Mixed-kind ordered prefixes | [mixed-kind public-order oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/mixed-index-order-oracle.test.ts) | A separate unindexed ready Collection computes the public ordered two-row prefix for the same keyed rows as an eager BasicIndex or BTreeIndex Collection. Five histories cross number/bigint, boolean/number, object/string, and array/object values, with insert, update, and delete transitions into and out of mixed domains. Ascending and descending reads use one ascending physical index, so descending checks exercise ReverseIndex. After preload and after both source copies commit each step, exact direct and live-query IDs must match the scan. Same-kind checkpoints verify that safe indexes still serve ordered reads. The original type-name index order failed eight ascending public-prefix checks. A second numeric source Map models a finite on-demand provider: a bigint enters and leaves the visible prefix, deletion requires an uninstalled row, and an offset window reaches a cursor after the retained BTree index loses ordered-plan eligibility. Public IDs match the independent provider order at each settled window; controls withholding either requested row fail those assertions. This owner does not define a global precedence among unlike values, cover in-place value mutation or arbitrary mixed-kind pairs, or prove that every retained on-demand traversal is safe after its source domain changes. | | WHERE predicate publication | [WHERE predicate publication oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-predicate-publication-oracle.property.test.ts), [`in` evaluator](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/in-evaluator-oracle.property.test.ts) ([review](oracle-reviews/perf-in-constant-set.md)) | An independent Kleene evaluator judges `eq`/`not`/`and`/`or` predicates over strings, a normalization-prefixed string, booleans, numbers, `NaN`, a valid Date, `null`, a missing field, and virtual fields. Snapshot histories with pending optimistic inserts compare a live query, direct subscribers with and without initial state, and `currentStateAsChanges`; change histories apply multi-key sync transactions, including reinsertion of a key deleted earlier, and compare every consumer's key set after each commit, on scan and `BasicIndex` paths. Peer subscribers share one field with different literals, plus one on a virtual field, so every published change must reach exactly the subscribers whose predicate it can satisfy; routing mutants that ignored the previous value, ignored the field, delivered a change twice, or routed while stale published rows awaited reconciliation fail here. Fixed witnesses cover a layout-only publication reaching a filtered subscriber as one empty batch, a filtered subscriber's empty Collection-readiness batch, retraction of a vanished row after eager cleanup and restart, and the new row plus exact subscriber update payload when a same-key row stays TRUE. A stale-live-value mutant survives the generated membership checks but fails the payload witness. A focused [property-visibility test](https://github.com/TanStack/db/blob/main/packages/db/tests/query/where-prefilter-property-visibility.test.ts) compares the public unindexed snapshot with the enriched-row contract for inherited, non-enumerable, enumerable-own, and nested getter paths; the pre-fix stored-row shortcut failed three of its four controls. Other property-descriptor and stateful-getter histories remain outside those fixed controls. Hostile mutants for FALSE-for-UNKNOWN `eq`, `or` or number-literal subscription prefilters, a prefilter that ignores the previous value, a skipped readiness batch, a skip while stale published rows await reconciliation, and an unindexed snapshot scan that reads only synced rows while an optimistic insert or delete changes visibility passed the prior `@tanstack/db` suite and fail here. Change routing withholds a dropped row from an `eq` subscription before its sent key could be recorded, so a mutant that records dropped rows as sent is equivalent for routed predicates in this oracle. A focused `collection-subscription.test.ts` witness checks that a dropped insert or update, alone or beside a matching change, cannot advance a limited subscription's page offset under a routed `eq` and an unrouted `or` predicate; the unrouted cases beside a matching change kill that mutant, and the unrouted `alone` case does not. A loose-equality prefilter is an equivalent mutant: it only skips less. Skipping during truncate replay also survives; the replay's baseline diff re-derives the same retraction, so the guard keeps the prior dataflow without its own witness. Generated cleanup and restart histories for filtered subscribers remain open for the lifecycle publication owner. No-op updates, other comparison operators, Temporal and binary operands, joins, ordering, optimistic updates, and truncate are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. Change routing was later removed in favor of pooled live queries; the routing mutants above are historical, and the same peer laws now check plain subscription dispatch. The unindexed snapshot prefilter was restored for compiled queries as one stored-row `eq` test on a string or boolean literal. Its property-visibility test also covers the reverse direction: a stored row with an inherited or non-enumerable field must still match `isUndefined` on that field, which a scan that evaluated the full predicate on stored rows fails. The `in` evaluator owner checks `in` against an independent linear reference over Dates, `NaN`, `-0`, BigInt, bytes, Temporal values, objects and nullish items, for constant and row-supplied lists. | | Joined result keys | [Joined result key oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/join-result-key-oracle.property.test.ts) | A nested-loop model recomputes the pair set of an inner, left, or full join over source keys drawn from plain, comma-bearing, bracket-bearing, and quoted strings, numbers beside the strings that print the same, both infinities, and `NaN`. After preload and each synced group change, the published rows must equal the model's pairs and the result key count must equal the row count. The comma-joined key encoding fails its two pinned histories and both campaigns; plain `JSON.stringify`, which prints infinities as the missing-side `null`, fails the pinned infinity history and both campaigns. A pinned history in which a `NaN`-keyed pair leaves and re-forms fails when the join index compares source-key prefixes with `===`. Joins over subqueries, more than two sources, custom `getKey`, and optimistic mutations are outside this owner. It runs in `@tanstack/db`'s `test:oracles` campaign. The [2026-09-30 review](https://github.com/TanStack/db/blob/main/docs/contributing/oracle-reviews/2026-09-30-where-predicate-and-join-keys.md) records each ORC outcome. | | D2 Index storage | [Index refinement oracle](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/index-refinement-oracle.property.test.ts) | A plain-`Map` model sums multiplicities under a type-and-value identity in which `NaN` equals `NaN` and `-0` equals `0`. Histories of up to twelve additions to two keys mix prefixed arrays with `NaN`, `0`, `-0`, `1`, `'1'`, `1n`, and `'a'` prefixes and unprefixed values, with cancellation and reappearance. After each addition, `get` and `has` must equal the model. Each step also checks that `get` returns what `getIterator` yields, in its order, since the two walk the storage separately. Comparing prefixes with `===`, which disagrees with the `Map` that holds them, fails three pinned histories and both campaigns; the `-0` control passes under both. Every history also runs with `prefixIdentity`, where values under one key with the same prefix are one value; mutants that merge unprefixed values or ignore the option in a prefix map fail. The [structural identity oracle](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/structural-identity-oracle.property.test.ts) checks that `structuralIdentity` agrees with D2 value identity for equal representations (array subclasses, class instances, null prototypes, key order, non-enumerable symbols) and independent values, including separator-bearing strings and keys, enumerable symbol keys, Dates, RegExps, Maps, Sets, binary values, holes, deep nesting, and a cyclic value; an independent normal-form model decides equality and is checked against `equalHashValues`. Mutants that drop a string's or key's length, an array's length, or the Date and RegExp headers fail it; lowering the depth cap is equivalent, because the cap only bounds work. groupBy's use of it is owned by the incrementalization law: its string-extrema lane fails a string identity without a length prefix, and a focused groupBy witness fails when the group key is left out of the identity. Compaction, presence tracking, and structural payloads are outside this owner; a focused test checks that an index built without presence (as reduce builds its indexes) throws when presence is read; the [reduce fast-path oracle](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/operators/reduce-fast-path-oracle.property.test.ts) compares the fast path's emitted entries one by one with the general path's, over keys that move between the paths, hash-merged outputs, -0, and zero and negative multiplicities, while an unchanged `NaN` output is owned by a focused reduce test; the join operator tests and incrementalization law own operator behavior. | @@ -500,9 +502,11 @@ reverse key reads to a source Map. Separate short and 8193-byte pairs check exact membership and complete forward and reverse key conservation when only the last byte differs. An intermediate PR BTree rewrite lost a surviving key after one equal-content owner left; the expanded lane rejected that result. -This owner does not prescribe mixed-kind order or cover arbitrary custom index -callbacks, binary range ordering, all JavaScript value kinds, or general -Collection publication histories. +A separate two-value Date, Temporal.Instant, and Uint8Array callback lane +checks legacy scalar input kinds on direct add only. Arbitrary callbacks and +update, build, and range paths are outside that lane. +This owner does not prescribe mixed-kind order or cover binary range ordering, +all JavaScript value kinds, or general Collection publication histories. The [comparison oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison-oracle.property.test.ts) owns the transferred same-realm NaN and invalid-Date default-order cases, @@ -532,21 +536,31 @@ The [Temporal index order oracle](https://github.com/TanStack/db/blob/main/packa owns ZonedDateTime instant ordering and time-only Duration ordering for direct BasicIndex/BTreeIndex reads and bounded public first-row reads. It separates exact Duration fields from order ties, including exclusive cursor behavior. -Calendar-unit Duration order, mixed Temporal kinds, arbitrary updates, and -custom index callbacks remain outside this owner. +An extended-year PlainDate lane checks calendar order under lexical string +collation through ready scan and direct indexed snapshots, with eager live-query +preload as parity evidence. The old indexed direct path chose the wrong year. +An exception lane requires ready scan, BasicIndex, and BTreeIndex direct reads +and eager live-query preloads to reject mixed PlainDate/PlainDateTime values and +distinct PlainMonthDays, as changelog #1519 specifies. The old indexed direct +path returned a row instead. Calendar-unit Duration order, other extended-year +histories, on-demand exception publication, arbitrary updates, and custom index +callbacks remain outside this owner. The [custom-index selection oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/custom-index-order-selection-oracle.test.ts) checks that a reverse manual index cannot change an ascending limited public -snapshot or eager live-query result. Its two distinct numeric ranks avoid any -claim about per-row custom callback order inside a comparator tie. On-demand -acquisition and later updates remain outside this witness. +snapshot or live-query result. Eager and prepopulated on-demand sources with +three distinct numeric ranks and a two-row window compare public IDs after +preload. The reviewer withdrew a claimed wrong live-query result after fixing +its scratch test's cross-checkout `@tanstack/db-ivm` resolution. The witness +makes no claim about the chosen physical plan or per-row custom callback order +inside a comparator tie. Provider paging from an incomplete on-demand source +and later updates remain outside this witness. The [PR #2116 confirmed-fixes review](oracle-reviews/pr-2116-confirmed-fixes.md) records baseline failures, hostile controls, and the bounded scope of these repairs. The [deferred findings record](oracle-reviews/pr-2116-deferred-findings.md) tracks remaining design and coverage questions, including valid Temporal kinds -without public order, extended-year PlainDate under lexical string collation, -and mixed-kind query order. Earlier PR-branch +without public order and mixed-kind query order. Earlier PR-branch counterexamples do not by themselves establish an `origin/main` failure. The Query DB ownership oracle owns the direct-write no-automatic-refetch law diff --git a/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md b/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md index 783adae0c9..4869509a3a 100644 --- a/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md +++ b/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md @@ -15,10 +15,25 @@ receiving witness, and a bounded hostile challenge. `origin/main` was | An indexed ZonedDateTime `orderBy` follows instant order, as promised by changelog #1519. A time-only Duration uses its elapsed-time order while keeping exact field identity. | Indexed first-row reads placed a later ZonedDateTime first when display text opposed epoch order, including a 9ns/10ns digit-width pair. PT120M also preceded PT1H under string-key order. | The Temporal index-order oracle compares exact direct BasicIndex/BTreeIndex reads and bounded public first rows, including an on-demand Collection. Adjacent nanoseconds at a large epoch reject `Number` precision loss. PT1H/PT60M exclusive cursors reject an exact-suffix tie-break; negative durations reject absolute-value order. The hostile reviewer found no sign mismatch in 2,000 seeded valid day/time pairs. | ZonedDateTime instant and time-only Duration order in the named fixed histories. Calendar units needing `relativeTo`, mixed Temporal kinds, custom index callbacks, and arbitrary update histories remain open. | | A manual index cannot change the declared ascending `orderBy(rank).limit(1)` answer. Query order is public; index selection is physical planning. | A reverse manual BTree index made `currentStateAsChanges` return rank 30 instead of rank 10. | The custom-index selection oracle independently sorts two distinct numeric ranks and checks the direct public snapshot and an eager live-query result. The hostile reviewer found the direct path that the first witness missed; both paths now pass. An added compiler guard had no demonstrated effect on this witness and was removed. | Eager initial reads with distinct ranks; no callback-tie, on-demand, later-update, or work-bound claim. | | Index updates conserve each source key even when number and string values occupy different comparison domains. | The old comparator could make `2` tie `"2"` while putting them on opposite sides of `"10"`; a direct deletion then duplicated or lost a key. | A source Map predicts full direct traversals after bounded delete/update/re-add actions, and a bounded indexed Collection read checks the public key set. The hostile reviewer confirmed the original comparator fails the source-Map comparison. | Key conservation, not a mixed-kind public order policy or all Collection histories. | -| An ordinary string beginning `\0tanstack-db:` obeys its declared string collation in indexed and scan reads. Internal Temporal order keys must not cause a tree to lose a neighboring ordinary string. | On `origin/main`, an en-US locale scan selected `\0tanstack-db:a` first while BasicIndex/BTreeIndex selected `tanstack-db:r`. Before the final kind classifier, deleting a neighboring ZonedDateTime or Duration also made BTreeIndex omit the surviving string. | The direct-index owner compares first public rows through scan and indexed paths under locale and custom string collations, with row IDs independent of label order. Two three-key BTree histories compare all surviving keys. The hostile reviewer ran 24 focused tests, 108 add/delete permutations, and a 27-value comparator transitivity sweep across five option variants. After the final classifier simplification, it also ran 3,456 four-key BasicIndex/BTreeIndex insertion/deletion histories (27,648 checkpoints) and a 16-value comparator sweep; no in-domain counterexample remained. | Named scalar, binary, Temporal, and string-kind histories. No arbitrary nested values, mixed-kind relative order, or custom index `compareFn` guarantee. | +| An ordinary string beginning `\0tanstack-db:` obeys its declared string collation in indexed and scan reads. Internal Temporal order keys must not cause a tree to lose a neighboring ordinary string. | On `origin/main`, an en-US locale scan selected `\0tanstack-db:a` first while BasicIndex/BTreeIndex selected `tanstack-db:r`. Before the final kind classifier, deleting a neighboring ZonedDateTime or Duration also made BTreeIndex omit the surviving string. | The direct-index owner compares first public rows through scan and indexed paths under locale and custom string collations, with row IDs independent of label order. Two three-key BTree histories compare all surviving keys. The hostile reviewer ran 24 focused tests, 108 add/delete permutations, and a 27-value comparator transitivity sweep across five option variants. After the final classifier simplification, it also ran 3,456 four-key BasicIndex/BTreeIndex insertion/deletion histories (27,648 checkpoints) and a 16-value comparator sweep; no in-domain counterexample remained. | Named scalar, binary, Temporal, and string-kind histories. No arbitrary nested values, mixed-kind relative order, or arbitrary custom index `compareFn` guarantee. A separate two-value direct callback lane checks legacy scalar input kinds for immutable Date, Temporal.Instant, and Uint8Array values. | ## Design decisions and open work +The removed `subset-dedupe.test.ts` case that overrode `Date.getTime` and +`Uint8Array.slice` was omitted from the earlier retirement ledger. Its frozen +assertion checked that request transport kept the same `where` object and that +the evaluator produced the same answers before and after transport. A probe on +this PR still passes that case. The maintainer excluded guarantees after callers +modify standard data objects, so the overridden-method case has no receiving +oracle law. Immutable Date, byte-array, and Buffer request identity and matching +remain checked in `subset-dedupe.test.ts`; `immutable-demand-boundary.test.ts` +also checks the direct and deferred request handoff. This records the excluded +assertion explicitly rather than treating its deletion as a transfer. +The rewritten `binary-equality-work.test.ts` also stopped editing a compiled +byte literal in place. The evaluator still passes that old trace on this PR, +but in-place edits to standard values are outside the accepted contract. Its +fresh-value checks remain the supported equality and work witnesses. + The captured-`inArray` list rebuild and per-row custom index callback ordering were proposed extensions, then withdrawn by the maintainer. This PR does not promise behavior after callers modify standard data objects in place, unsigned @@ -119,3 +134,52 @@ follow-up. After that follow-up, the affected index and comparison files passed passed, and the DB source TypeScript and affected-file lint checks passed. Those runs cover the changed immutable-value paths; they do not establish arbitrary later Collection histories or the deferred mixed-Temporal error. + +## 2026-10-13 high-effort review of PR head `16b219224` + +This addendum supersedes the earlier manual-index row's two-rank limit. The +earlier observations remain historical evidence. The follow-up tree has a +three-rank/two-row owner and one ordered-index eligibility check shared with +the mixed-kind repair. The reviewer withdrew the claimed wrong live-query +result after finding that its scratch test resolved `@tanstack/db-ivm` from a +different checkout. With the correct package, both the frozen PR head and +`origin/main` published the expected prefix before and after the reported +writes. + +| Review IDs | Law, experiment, and action | Remaining limit | +| --- | --- | --- | +| R01, R10 | Index choice cannot change an ordered public prefix. The new scan-versus-index owner rejected the frozen type-name order in eight initial ascending cases. Its repaired 20 ascending and descending BasicIndex/BTreeIndex cases include paired source updates and a ReverseIndex route. Forcing the index-eligibility getter true rejects all ten original ascending histories. A retained on-demand BTree history starts with numbers, inserts and moves a bigint, deletes a prefix row, then requests a later window. It asserts each settled public prefix against independently sorted provider rows. The finite request supplies a missing row; the later request uses a cursor after the retained index has become ineligible for new ordered plans. Withholding either requested row fails the corresponding public assertion. A single physical and query comparator is unsafe here: the query comparator does not form a transitive total order for all mixed JavaScript values. | The eager histories and this retained on-demand paging history pass. The withheld-row controls challenge the provider result, not production code. Other provider paging sequences and repeated domain changes remain untested; the retained loader still traverses the index after eligibility changes. | +| R03 | The reviewer withdrew the `[20,30]` public-result claim after fixing the scratch test's cross-checkout `@tanstack/db-ivm` resolution. The corrected three-row eager fixture returns `[10,20]` on both the frozen PR and `origin/main`; the reviewer also checked the insert and delete checkpoints. Our widened owner independently sorts three ranks and compares eager and prepopulated on-demand public rows after preload. The compiler's ordered-index eligibility check stays because mixed-kind finding R01 needs it; the R03-only planner assertion was removed. | No confirmed live-query correctness defect remains for this fixture. Incomplete-provider paging and later updates are outside this owner. | +| R04 | Direct index callbacks retain the legacy scalar input kinds for immutable Date, Temporal.Instant, and Uint8Array values. Five of six new callback cells failed on frozen head; callback-only normalization makes all six pass, with raw-object and collapsed-position hostile controls rejected. | Two-value direct-add histories only; arbitrary callbacks and later updates are not claimed. | +| R06, R07 | BasicIndex now keeps admitted exact keys beside ordered representatives; a 64-row Temporal range and ordered read made zero `toString()` calls, down from 128. BTreeIndex keeps a comparator-tie bucket map after one exact bucket leaves; a return to the tie no longer copies every peer. | BasicIndex may retain one former representative per live exact group. No general heap limit is claimed beyond existing retained-index state. | +| R08, R09 | Object comparison does allocate temporary reference-identity tuples, but a bounded timing probe found no object throughput regression; no extra hot-path code was added without a demonstrated cost. Auto-index now shares `indexableOperations`, and `DurationFields` is defined once. | An allocation profile under a representative workload would be needed to establish a material R08 performance defect. | +| R02, R05, R11 | In-place byte edits can stale the normalized-key cache, and the overridden-method request test was omitted from the earlier retirement record. Both concern caller modification of standard data objects, which the maintainer excluded; fresh immutable matching and request identity remain covered. The reviewer's random index run supplied no seed, grammar, or count. | The cache holds one key per live byte array. The random run is stress evidence, not a witness for mixed public order or callback inputs. | + +An adjacent previously recorded defect also changed under the mixed-domain +guard. On `origin/main`, ready BasicIndex and BTreeIndex direct ordered reads +returned a row for PlainDate beside PlainDateTime, or for two PlainMonthDays, +while the scan threw the `TypeError` promised by changelog #1519. The expanded +Temporal index-order owner now checks six scan/index cases at direct-read and +eager live-query checkpoints. All 68 tests in that file pass on this worktree. +The eager live-query paths already threw on `origin/main`; the repair evidence +is the indexed direct-read mismatch. On-demand and later-update exception +histories are not claimed. + +The same rewrite repairs a separate PlainDate boundary under lexical string +collation. The supplied year fields require `-100000-01-01` before +`+010000-01-01`, while their display strings have the reverse lexical order. +The expanded Temporal owner rejects the two `origin/main` indexed direct-read +results, which selected the positive year; scan and eager live-query preload +already selected the negative year. Current BasicIndex and BTreeIndex direct +reads pass. This establishes the fixed ready-source history, not every +extended-year pair or on-demand publication. + +The follow-up tree adds 406 and removes 541 production source lines against +`origin/main`, for **135 fewer lines**. A paired local minified build of the 105 +DB JavaScript package files totals 205,973 gzip bytes, versus 206,122 for the +same files on `origin/main`: **149 fewer gzip bytes**. Bundling each built DB +ESM entry into one consumer file with esbuild instead yields 100,746 versus +100,544 gzip bytes: **202 more gzip bytes**. Separate-file and whole-bundle +compression have different dictionaries and tree-shaking boundaries. React DB +has no source change in this follow-up; the CI size action has not yet run on +the final pushed commit. diff --git a/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md b/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md index dc6dbad3e3..bf20f476bf 100644 --- a/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md +++ b/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md @@ -11,12 +11,25 @@ not closed by passing a neighboring transfer oracle. | --- | --- | --- | | R29: two valid `Temporal.Duration` values can have the same display text but different fields. A changed field can be suppressed as an equal Collection value. | Closed in this PR for the tested direct and Collection paths. The Duration field model checks all ten fields; a same-display replacement and adjacent-unit histories make the former `deepEquals` and change-event behavior fail before the repair. The query equality owner and `in` evaluator owner separately check Duration field identity. | General multirow and live-query publication histories remain outside these owners. Add them to the relevant owner if a new failure reaches those paths. | | R16: an indexed Collection can admit a valid Temporal row whose index comparator has no public order, then throw after partial state changes. | Earlier PR revisions recorded RED admission and publication traces for Duration, PlainMonthDay, and mixed-kind rows. Source inspection suggests the risk remains on `origin/main`, but this narrowed PR has not executed a current-base RED witness. The public order law and internal index storage order must be distinguished. | The direct-index and Collection publication owners should pair a valid-row admission history with exact membership, ordered storage, stored rows, and subscriber observations. Choose the smallest internal ordering or atomic-admission repair that satisfies the accepted contract. | -| Mixed-kind indexed order and custom-index query selection may disagree with public query results. | The mixed number/string direct-index history confirms key duplication after deletion on `origin/main`; the new source-Map lane closes that bounded key-conservation bug. A reverse manual index also returned the wrong first row through `currentStateAsChanges`; the new selection oracle closes that bounded public bug. The user withdrew a per-row custom callback position guarantee for comparator-equal exact values. | A general public order policy for unlike kinds remains open. The current owners prove key conservation and the two-rank manual-index selection case, not arbitrary mixed-kind order, custom callback ties, or later live updates. | -| Indexed ordering can suppress a required Temporal comparison error. | Changelog #1519 says ordering mixed Temporal kinds throws `TypeError`. At executable head `2f852dcd1` and clean `origin/main` `bbbf017ef5`, a ready two-row Collection scan throws for `PlainDate` beside `PlainDateTime`; after creating either BasicIndex or BTreeIndex, the same ascending `currentStateAsChanges` read returns both rows. Two distinct `PlainMonthDay` values give the same scan/index disagreement because that type has no static order. This is a pre-existing public-query defect, not a regression caused by this rewrite. | The Temporal index-order owner should add a scan/index exception witness for mixed or unordered Temporal values, then the ordered-query index selection path should preserve the public error while retaining exact equality indexing. The present oracle deliberately covers ordered ZonedDateTimes and time-only Durations only. On-demand acquisition and later updates need separate witnesses. | +| Mixed-kind indexed order and custom-index query selection may disagree with public query results. | The source-Map owner closes the mixed number/string direct-index key loss. The new scan-versus-index owner rejected the frozen PR's wrong mixed-kind public prefixes, and the current guard makes the tested eager paths use the query's order. A retained on-demand BTree history checks visible prefixes after a number/bigint transition, a missing-row finite request, and a later cursor request that still traverses the index; all settled public windows match independently sorted provider rows. A reverse manual index produced a wrong direct prefix on an earlier branch revision; the current three-rank owner checks direct, eager, and prepopulated on-demand public results. The reviewer withdrew its claimed live-query failure after fixing a cross-checkout `@tanstack/db-ivm` resolution error. The maintainer withdrew a per-row custom callback position guarantee for comparator-equal exact values. | Other retained on-demand paging sequences, different incomplete-source patterns, and later live updates need public-result witnesses. The owners do not define a universal precedence among unlike kinds. | +| Indexed ordering can suppress a required Temporal comparison error. | Changelog #1519 requires `TypeError` for mixed or unordered Temporal values. On clean `origin/main`, indexed direct reads returned rows for `PlainDate` beside `PlainDateTime` and for two distinct `PlainMonthDay` values while the scan threw. The new Temporal exception lane fails on those four baseline indexed reads and passes on this branch; eager live queries already threw on main. | The bounded direct-read repair is complete for these pairs. On-demand acquisition, later updates, and other mixed Temporal pairs need separate witnesses before claiming the whole exception class. | | R17: an equal-time Date removal left an ordered ghost under added index machinery. | The recorded RED trace depended on code introduced during this PR. It does not establish an `origin/main` defect. The stronger rule for a substituted old row with a different callback position was withdrawn. | If ordinary direct remove/update still has a current-base counterexample, add a source-Map history to the direct-index owner before changing production. | -| PlainDate extended-year order under lexical string collation may diverge between scans and indexes. | On both the recorded `origin/main` baseline and this branch, a direct scan comparator orders `-100000-01-01` before `+010000-01-01`, while direct BasicIndex and BTreeIndex traversal returns the positive date first with `stringSort: 'lexical'`. This was found during hostile review of the final kind classifier, not in the retired unit-test ledger. A public indexed query witness has not run. | Extend the Temporal index-order owner with a scan/indexed public read under lexical collation, confirm the contract for this other Temporal kind, and reproduce RED before a separate repair. The present owner is bounded to ZonedDateTime and time-only Duration. | +| PlainDate extended-year order under lexical string collation can diverge between scans and indexes. | On `origin/main`, a ready scan returns `-100000-01-01` first while BasicIndex and BTreeIndex direct public reads return `+010000-01-01`. The new Temporal owner checks calendar order from the supplied years and fails on those two baseline direct reads. Both direct indexed reads pass on this branch. Baseline eager live queries already returned the right first row. | The fixed ready-source direct-read history is closed. Other extended-year pairs, later updates, and on-demand publication remain outside the current witness. | | Cross-realm native equality was asserted by a surviving request-transport unit test. | PR #1797 captured the then-current `instanceof` result. No public realm-distinctness promise was found. The maintainer questioned that distinction. This PR removes that incidental test expectation and makes no cross-realm equality claim. | A future public contract decision needs a host-boundary witness before either equality policy becomes an oracle law. | +The 2026-10-13 review expanded the custom-index owner to three ranks and a +two-row window. The reported `[20,30]` public result came from a scratch test +that resolved `@tanstack/db-ivm` from another checkout; with the correct +package, both frozen PR head `16b219224` and `origin/main` returned `[10,20]` +at the reviewer's read checkpoints. The reviewer withdrew that finding. A +separate paged-provider witness now reaches a retained +on-demand mixed-kind index after its domain changes. It checks public rows +after a finite request fills a missing prefix row and after a later cursor +request traverses the retained index. Wrong-provider controls that withhold +either requested row fail the public comparison. Other provider histories +remain untested; this witness does not establish general safe traversal of an +ineligible retained index. + Captured `inArray` list-entry refresh and whole-live-query reconstruction were explored as new features. The pre-PR evaluator and live graph did not already deliver the whole-result behavior. The maintainer withdrew that expansion for diff --git a/packages/db/mangle-cache.json b/packages/db/mangle-cache.json index ab5a0ab8cc..0673dd2f48 100644 --- a/packages/db/mangle-cache.json +++ b/packages/db/mangle-cache.json @@ -402,5 +402,6 @@ "detach": "gC", "copyRows": "gD", "restore": "gE", - "attach": "gF" + "attach": "gF", + "sortedExactValues": "gG" } diff --git a/packages/db/package.json b/packages/db/package.json index 80d4c5eb59..92d04b8bb1 100644 --- a/packages/db/package.json +++ b/packages/db/package.json @@ -23,7 +23,7 @@ "test": "vitest --run", "test:dist": "vitest run --config vitest.dist.config.ts", "test:facade-retention": "node --expose-gc --import tsx tests/facade-retention.probe.ts", - "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/duration-query-equality-oracle.test.ts tests/temporal-index-order-oracle.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/custom-index-order-selection-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", + "test:oracles": "vitest --run --coverage.enabled=false tests/reverse-index-oracle.property.test.ts tests/query/compiler/in-evaluator-oracle.property.test.ts tests/cleanup-queue-oracle.property.test.ts tests/comparison-oracle.property.test.ts tests/duration-query-equality-oracle.test.ts tests/temporal-index-order-oracle.test.ts tests/cursor-oracle.property.test.ts tests/index-update-oracle.property.test.ts tests/utils-oracle.property.test.ts tests/change-event-history-oracle.test.ts tests/index-suggestion-oracle.test.ts tests/btree-map-oracle.test.ts tests/paced-mutations-oracle.test.ts tests/db-client-hydration-authority-oracle.test.ts tests/local-storage-order-oracle.test.ts tests/local-storage-peer-oracle.test.ts tests/collection-mutation-startup-oracle.test.ts tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/optimistic-transaction-oracle.property.test.ts tests/transaction-ownership-oracle.property.test.ts tests/optimistic-settlement-boundaries-oracle.test.ts tests/optimistic-history-publication-oracle.test.ts tests/optimistic-history-outcomes-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-index-value-oracle.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history-oracle.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-reentrancy-oracle.test.ts tests/collection-subscription-lifecycle-publication-oracle.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/live-query-observer-history-oracle.property.test.ts tests/live-query-observer-snapshot-oracle.property.test.ts tests/query/cold-join-reconciliation-oracle.test.ts tests/query/identity-output-shape-oracle.test.ts tests/query/optimizer-semantics-oracle.test.ts tests/query/index-path-collision-oracle.test.ts tests/query/custom-index-order-selection-oracle.test.ts tests/query/mixed-index-order-oracle.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-alias-shadowing-oracle.test.ts tests/query/includes-outer-join-route-work-oracle.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/subquery-user-value-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state-oracle.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts tests/query/virtual-row-fields-oracle.test.ts tests/query/where-predicate-publication-oracle.property.test.ts tests/query/join-result-key-oracle.property.test.ts tests/query/pooled-live-query-oracle.property.test.ts tests/live-query-deferred-acquisition-oracle.test.ts tests/flat-change-tracking-oracle.property.test.ts tests/truncate-readiness-oracle.test.ts tests/proxy-detachment-contract-oracle.test.ts tests/production-error-messages-oracle.test.ts tests/query/bucket-facade-rollback-oracle.property.test.ts tests/query/aggregate-select-oracle.property.test.ts", "bench:nested-includes": "vitest bench tests/query/includes-performance.bench.ts --run", "test:indexed-db:package": "pnpm --filter @tanstack/db... build && vitest run --config vitest.indexed-db.package.config.ts", "test:indexed-db:browser": "playwright test --config playwright.indexed-db.config.ts", diff --git a/packages/db/src/collection/change-events.ts b/packages/db/src/collection/change-events.ts index ec9a2cc359..4ad8e8aa56 100644 --- a/packages/db/src/collection/change-events.ts +++ b/packages/db/src/collection/change-events.ts @@ -356,7 +356,12 @@ function getOrderedKeys( // Find the index const index = findIndexForField(collection, fieldPath, compareOpts) - if (index && index.supports(`gt`) && index.supportsRangeOptimization) { + if ( + index && + index.supports(`gt`) && + index.supportsRangeOptimization && + index.supportsOrderOptimization + ) { // Use index optimization const filterFn = (key: TKey): boolean => { const value = collection.get(key) diff --git a/packages/db/src/indexes/auto-index.ts b/packages/db/src/indexes/auto-index.ts index 2098e4f3cc..0f1c9bb6f2 100644 --- a/packages/db/src/indexes/auto-index.ts +++ b/packages/db/src/indexes/auto-index.ts @@ -1,4 +1,5 @@ import { DEFAULT_COMPARE_OPTIONS } from '../utils' +import { indexableOperations } from '../query/builder/functions.js' import { hasVirtualPropPath } from '../virtual-props' import { codedWarning, devBuild } from '../error-message.js' import { checkCollectionSizeForIndex, isDevModeEnabled } from './index-registry' @@ -122,8 +123,7 @@ function extractIndexableExpressions( } // Check if this is a supported operation - const supportedOperations = [`eq`, `gt`, `gte`, `lt`, `lte`, `in`] - if (!supportedOperations.includes(func.name)) { + if (!indexableOperations.includes(func.name)) { return } diff --git a/packages/db/src/indexes/base-index.ts b/packages/db/src/indexes/base-index.ts index cc2ce9414c..4c1c80f6b5 100644 --- a/packages/db/src/indexes/base-index.ts +++ b/packages/db/src/indexes/base-index.ts @@ -4,7 +4,12 @@ import { indexableOperations, } from '../query/builder/functions.js' import { DEFAULT_COMPARE_OPTIONS, deepEquals } from '../utils.js' -import { makeCheckedComparator, makeComparator } from '../utils/comparison.js' +import { + indexObjectKind, + makeCheckedComparator, + makeComparator, + normalizeValue, +} from '../utils/comparison.js' import { codedMessage, codedWarning, devBuild } from '../error-message.js' import type { CompiledSingleRowExpression } from '../query/compiler/evaluators.js' import type { RangeQueryOptions } from './btree-index.js' @@ -62,6 +67,7 @@ export type IndexReader = Pick< | `keyCount` | `supports` | `supportsRangeOptimization` + | `supportsOrderOptimization` | `canOptimizeRangeFor` > @@ -110,6 +116,8 @@ export interface IndexInterface< * a full scan when this is `false`. */ get supportsRangeOptimization(): boolean + /** Whether a query can use this index for its declared ordered prefix. */ + readonly supportsOrderOptimization?: boolean /** * Whether the live values in this index share the predicate operand's @@ -158,7 +166,9 @@ export abstract class BaseIndex< this.compareOptions = options?.compareOptions ?? DEFAULT_COMPARE_OPTIONS this.hasCustomComparator = options?.compareFn != null this.compareFn = makeCheckedComparator( - options?.compareFn ?? makeComparator(this.compareOptions, `index`), + options?.compareFn + ? (a, b) => options.compareFn(normalizeValue(a), normalizeValue(b)) + : makeComparator(this.compareOptions, `index`), ) this.name = name this.supportedOperations = new Set(indexableOperations) @@ -246,6 +256,24 @@ export abstract class BaseIndex< return !this.hasCustomComparator } + get supportsOrderOptimization(): boolean { + if (this.hasCustomComparator || this.rangeValueDomains.size > 1) + return false + // Within these domains the index orders by a storage key (or lacks a + // defined query order), rather than by the query's value comparator. + const domain = this.rangeValueDomains.keys().next().value + return ( + domain === undefined || + ![ + `Array`, + `Binary`, + `Temporal.Duration.calendar`, + `Temporal.PlainMonthDay`, + `function`, + ].includes(domain) + ) + } + protected addRangeValue(value: unknown): void { const domain = rangeValueDomain(value) if (domain === undefined) return @@ -360,6 +388,7 @@ export abstract class BaseIndex< function rangeValueDomain(value: unknown): string | undefined { if (value == null) return undefined if (value instanceof Date) return `date` + if (typeof value === `object`) return indexObjectKind(value) return typeof value } diff --git a/packages/db/src/indexes/basic-index.ts b/packages/db/src/indexes/basic-index.ts index 110892a821..911959171d 100644 --- a/packages/db/src/indexes/basic-index.ts +++ b/packages/db/src/indexes/basic-index.ts @@ -41,6 +41,7 @@ export class BasicIndex< private valueMap = new Map>() // Sorted array of unique indexed values for range queries private sortedValues: Array = [] + private sortedExactValues: Array = [] // Set of all indexed PKs private indexedKeys = new Set() @@ -77,6 +78,7 @@ export class BasicIndex< ) this.valueMap.set(normalizedValue, new Set([key])) this.sortedValues.splice(insertIdx, 0, orderedValue) + this.sortedExactValues.splice(insertIdx, 0, normalizedValue) } } @@ -124,11 +126,12 @@ export class BasicIndex< ) { if ( areSameValueZeroEqual( - normalizeValue(this.sortedValues[sortedIndex]), + this.sortedExactValues[sortedIndex], normalizedValue, ) ) { this.sortedValues.splice(sortedIndex, 1) + this.sortedExactValues.splice(sortedIndex, 1) break } sortedIndex++ @@ -187,6 +190,7 @@ export class BasicIndex< else { this.valueMap.set(value, new Set([key])) this.sortedValues.push(orderValue(indexedValue)) + this.sortedExactValues.push(value) } this.addRangeValue(indexedValue) this.indexedKeys.add(key) @@ -194,9 +198,13 @@ export class BasicIndex< // Array.sort moves raw undefined values last without comparing them. // Wrap values so nulls-first comparison runs during a full build. - const sortable = this.sortedValues.map((value) => ({ value })) + const sortable = this.sortedValues.map((value, index) => ({ + value, + exact: this.sortedExactValues[index], + })) sortable.sort((left, right) => this.compareFn(left.value, right.value)) this.sortedValues = sortable.map(({ value }) => value) + this.sortedExactValues = sortable.map(({ exact }) => exact) } /** @@ -205,6 +213,7 @@ export class BasicIndex< clear(): void { this.valueMap.clear() this.sortedValues = [] + this.sortedExactValues = [] this.indexedKeys.clear() this.clearRangeValues() } @@ -289,7 +298,7 @@ export class BasicIndex< // Collect all keys in range for (let i = startIdx; i < endIdx; i++) { - const keys = this.valueMap.get(normalizeValue(this.sortedValues[i])) + const keys = this.valueMap.get(this.sortedExactValues[i]) if (keys) { keys.forEach((key) => result.add(key)) } @@ -378,9 +387,8 @@ export class BasicIndex< const groupValue = this.sortedValues[index] const groupKeys: Array = [] do { - for (const key of this.valueMap.get( - normalizeValue(this.sortedValues[index]), - ) ?? []) { + for (const key of this.valueMap.get(this.sortedExactValues[index]) ?? + []) { groupKeys.push(key) } index += step diff --git a/packages/db/src/indexes/btree-index.ts b/packages/db/src/indexes/btree-index.ts index 3bb6952f6d..84d5a7acae 100644 --- a/packages/db/src/indexes/btree-index.ts +++ b/packages/db/src/indexes/btree-index.ts @@ -97,7 +97,6 @@ export class BTreeIndex< this.valueMap.delete(exact) group.buckets?.delete(exact) } - if (group.buckets?.size === 1) group.buckets = undefined if (group.keys.size && group.representativeKey === key) { // Advancing one iterator skips each removed Map slot at most once. const [nextKey, nextValue] = (group.nextRepresentative ??= diff --git a/packages/db/src/indexes/reverse-index.ts b/packages/db/src/indexes/reverse-index.ts index 9e018bc07f..181b5bd475 100644 --- a/packages/db/src/indexes/reverse-index.ts +++ b/packages/db/src/indexes/reverse-index.ts @@ -103,6 +103,10 @@ export class ReverseIndex< return this.originalIndex.supportsRangeOptimization } + get supportsOrderOptimization(): boolean { + return this.originalIndex.supportsOrderOptimization === true + } + canOptimizeRangeFor(value: unknown): boolean { return this.originalIndex.canOptimizeRangeFor?.(value) ?? true } diff --git a/packages/db/src/query/compiler/order-by.ts b/packages/db/src/query/compiler/order-by.ts index 47004990e4..b7e297dba6 100644 --- a/packages/db/src/query/compiler/order-by.ts +++ b/packages/db/src/query/compiler/order-by.ts @@ -203,8 +203,8 @@ export function processOrderBy( compareOpts, ) - // Only use the index if it supports range queries - if (!index?.supports(`gt`)) { + // An ordered source prefix must agree with the query's declared order. + if (!index?.supports(`gt`) || !index.supportsOrderOptimization) { index = undefined } diff --git a/packages/db/src/utils.ts b/packages/db/src/utils.ts index 0c6cd53d01..ec99c9311b 100644 --- a/packages/db/src/utils.ts +++ b/packages/db/src/utils.ts @@ -21,7 +21,7 @@ export const durationFields = [ `microseconds`, `nanoseconds`, ] as const -type DurationFields = Record<(typeof durationFields)[number], number> +export type DurationFields = Record<(typeof durationFields)[number], number> /** * Deep equality function that compares two values recursively diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index 4cad6ffb81..1a4d79bac0 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -1,6 +1,7 @@ import { durationFields, isTemporal } from '../utils' import { getRuntimeReferenceIdentity } from '../query/runtime-reference-identity' import { codedMessage, devBuild } from '../error-message' +import type { DurationFields } from '../utils' import type { CompareOptions } from '../query/builder/types' /** @@ -123,7 +124,7 @@ export const descComparator = ( : ascComparator(b, a, opts) } -function indexObjectKind(value: any): string { +export function indexObjectKind(value: any): string { if (isUint8Array(value)) return `Binary` if (isTemporal(value)) { const tag = value[Symbol.toStringTag] @@ -252,8 +253,6 @@ function normalizeBinary(value: Uint8Array): string { return key } -type DurationFields = Record<(typeof durationFields)[number], number> - /** * Normalize a value for comparison and Map key usage * Converts values that can't be directly compared or used as Map keys diff --git a/packages/db/tests/basic-index-work.test.ts b/packages/db/tests/basic-index-work.test.ts index 438a9bfede..c513393795 100644 --- a/packages/db/tests/basic-index-work.test.ts +++ b/packages/db/tests/basic-index-work.test.ts @@ -1,7 +1,35 @@ import { describe, expect, it, vi } from 'vitest' +import { Temporal } from 'temporal-polyfill' import { BasicIndex } from '../src/indexes/basic-index' import { PropRef } from '../src/query/ir' +describe(`BasicIndex ordered read work`, () => { + it(`does not reserialize admitted Temporal values while reading`, () => { + const index = new BasicIndex(1, new PropRef([`value`])) + for (let key = 0; key < 64; key++) { + index.add(key, { + value: Temporal.Instant.fromEpochNanoseconds(BigInt(key)), + }) + } + + const toString = vi.spyOn(Temporal.Instant.prototype, `toString`) + let range = new Set() + let ordered: Array = [] + let stringifyCalls = 0 + try { + range = index.rangeQuery({}) + ordered = index.takeFromStart(64) + stringifyCalls = toString.mock.calls.length + } finally { + toString.mockRestore() + } + + expect(range.size).toBe(64) + expect(ordered).toEqual(Array.from({ length: 64 }, (_, key) => key)) + expect(stringifyCalls).toBe(0) + }) +}) + describe(`BasicIndex removal work`, () => { it.each([1, 4])( `searches only the comparator group of size %s`, diff --git a/packages/db/tests/btree-index-work.test.ts b/packages/db/tests/btree-index-work.test.ts index d907df5694..ee15d5e808 100644 --- a/packages/db/tests/btree-index-work.test.ts +++ b/packages/db/tests/btree-index-work.test.ts @@ -1,7 +1,44 @@ -import { describe, expect, it } from 'vitest' +import { describe, expect, it, vi } from 'vitest' +import { Temporal } from 'temporal-polyfill' import { BTreeIndex } from '../src/indexes/btree-index.js' import { PropRef } from '../src/query/ir.js' +describe(`BTree comparator-tie update work`, () => { + it(`does not copy every owner when an exact value returns to a tie`, () => { + const instant = Temporal.Instant.fromEpochNanoseconds(0n) + const first = instant.toZonedDateTimeISO(`UTC`) + const second = instant.toZonedDateTimeISO(`+14:00`) + const additions = [16, 256].map((size) => { + const index = new BTreeIndex(1, new PropRef([`value`])) + for (let key = 0; key < size; key++) { + index.add(key, { value: first }) + } + index.add(-1, { value: second }) + index.remove(-1, { value: second }) + + const originalAdd = Set.prototype.add + const add = vi.spyOn(Set.prototype, `add`).mockImplementation(function ( + this: Set, + value: unknown, + ) { + return originalAdd.call(this, value) + }) + let calls = 0 + try { + index.add(-1, { value: second }) + calls = add.mock.calls.length + } finally { + add.mockRestore() + } + expect(index.equalityLookup(first).size).toBe(size) + expect(index.equalityLookup(second)).toEqual(new Set([-1])) + expect(index.takeFromStart(1)).toEqual([-1]) + return calls + }) + expect(additions[1]).toBeLessThanOrEqual(additions[0]! + 4) + }) +}) + describe(`BTree exact-bucket ownership work`, () => { it.each( [300, 100000].flatMap((size) => diff --git a/packages/db/tests/index-update-oracle.property.test.ts b/packages/db/tests/index-update-oracle.property.test.ts index bf11bd515c..bafc931160 100644 --- a/packages/db/tests/index-update-oracle.property.test.ts +++ b/packages/db/tests/index-update-oracle.property.test.ts @@ -47,6 +47,8 @@ import type { StringCollationConfig } from '../src/types.js' * A separate mixed number/string lane checks key conservation only. A full * ordered read and an indexed public snapshot must contain each live source * key once, even when unlike values have no declared relative sort order. + * A separate two-value callback lane checks legacy scalar input kinds for + * immutable Date, Temporal.Instant, and byte values across both index types. */ type IndexValue = number @@ -2741,3 +2743,64 @@ test(`BTreeIndex bounded reads do not recompare every live tie`, () => { }) expect(comparisons[2]).toBeLessThanOrEqual(comparisons[0]! + 4) }) + +/** + * Preserve a bounded legacy callback input: older indexes supplied a Date's + * timestamp and a string for Temporal and binary values. For two immutable + * values of each named kind, a direct add comparison supplies that same scalar + * kind in both index types. The model uses two values with a known order; it + * does not copy or specify the index's string encoding. A callback that accepts + * only the old scalar kind distinguishes a raw-object leak. After both adds, + * full ordered keys and exact owners must match the supplied facts. This does + * not establish arbitrary callbacks, build/update/range paths, or mutation of + * the value objects after admission. + */ +describe.each(indexTypes)( + `%s comparator input compatibility`, + (_name, IndexType) => { + test.each([ + { + name: `Date`, + values: [new Date(1), new Date(2)], + kind: `number`, + }, + { + name: `Temporal.Instant`, + values: [ + Temporal.Instant.from(`2020-01-01T00:00:00Z`), + Temporal.Instant.from(`2021-01-01T00:00:00Z`), + ], + kind: `string`, + }, + { + name: `Uint8Array`, + values: [new Uint8Array([1]), new Uint8Array([2])], + kind: `string`, + }, + ])( + `passes $name to callbacks in its legacy scalar kind`, + ({ values, kind }) => { + let comparisons = 0 + const index = new IndexType(1, new PropRef([`value`]), undefined, { + compareFn: (left, right) => { + comparisons++ + expect(typeof left).toBe(kind) + expect(typeof right).toBe(kind) + if (kind === `number`) return (left as number) - (right as number) + const a = left as string + const b = right as string + return a < b ? -1 : a > b ? 1 : 0 + }, + }) + // Keys oppose value order, so collapsing both inputs to one comparator + // position cannot pass by key-tie ordering alone. + index.add(`a`, { value: values[1] }) + index.add(`z`, { value: values[0] }) + expect(comparisons).toBeGreaterThan(0) + expect(index.takeFromStart(2)).toEqual([`z`, `a`]) + expect(index.equalityLookup(values[0])).toEqual(new Set([`z`])) + expect(index.equalityLookup(values[1])).toEqual(new Set([`a`])) + }, + ) + }, +) diff --git a/packages/db/tests/query/custom-index-order-selection-oracle.test.ts b/packages/db/tests/query/custom-index-order-selection-oracle.test.ts index 67b7196f5c..622e421864 100644 --- a/packages/db/tests/query/custom-index-order-selection-oracle.test.ts +++ b/packages/db/tests/query/custom-index-order-selection-oracle.test.ts @@ -7,49 +7,86 @@ * * The Collection API also exposes an ordered, limited direct snapshot through * `currentStateAsChanges`. The model sorts a plain array by numeric rank and - * slices it. Its two distinct ranks avoid any promise about callback ordering - * within an equality class. The legal history starts one ready, eager source, - * then reads a direct snapshot and compiles one limited live query. A scan and - * a source with a reverse-order BTree index exercise both public paths. The - * exact public row IDs must equal the model at the direct-read and post-preload - * checkpoints. The reverse index is the hostile planner control: taking its - * first entry as the first query row would expose rank 30 instead of rank 10. - * This witness does not cover on-demand acquisition or later source changes. + * slices it. Its three distinct ranks avoid any promise about callback ordering + * within an equality class. The legal history starts one ready source, then + * reads a direct snapshot and compiles one limited live query. Eager and + * prepopulated on-demand sources each run with a scan and a reverse-order + * BTree index. Three distinct ranks and a two-row window can expose a reversed + * prefix whose membership and order differ from the query's prefix. The public row + * IDs must equal the model at the direct-read and post-preload checkpoints. + * Taking the reverse index's first two entries as the query window would + * expose ranks 30 and 20 instead of 10 and 20. The oracle checks the public + * result and allows any physical plan that produces it. + * The on-demand adapter confirms a finite request without adding rows because + * all three are already installed. This witness does not cover provider paging + * from an incomplete source or later source changes. */ import { expect, it } from 'vitest' import { createCollection } from '../../src/collection/index.js' import { BTreeIndex } from '../../src/indexes/btree-index.js' import { createLiveQueryCollection } from '../../src/query/live-query-collection.js' import { PropRef } from '../../src/query/ir.js' -import { mockSyncCollectionOptions } from '../utils.js' +import { mockSyncCollectionOptions, withOracleCleanup } from '../utils.js' type Row = { id: number; rank: number } const rows: ReadonlyArray = [ { id: 30, rank: 30 }, + { id: 20, rank: 20 }, { id: 10, rank: 10 }, ] // The query's rank order comes from its declaration, not index traversal. const expectedIds = [...rows] .sort((a, b) => a.rank - b.rank) - .slice(0, 1) + .slice(0, 2) .map((row) => row.id) -it.each([`scan`, `reverse manual index`] as const)( - `keeps bounded public rank order with %s`, - async (path) => { - const source = createCollection( +function makeSource( + mode: `eager` | `on-demand`, + path: `scan` | `reverse manual index`, +) { + const id = `custom-index-order-selection-${mode}-${path}` + if (mode === `eager`) { + return createCollection( mockSyncCollectionOptions({ - id: `custom-index-order-selection-${path}`, + id, getKey: (row) => row.id, initialData: [...rows], autoIndex: `off`, }), ) + } + return createCollection({ + id, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `off`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of rows) write({ type: `insert`, value: row }) + commit() + markReady() + return { loadSubset: () => true, unloadSubset: () => {} } + }, + }, + }) +} + +it.each( + ([`eager`, `on-demand`] as const).flatMap((mode) => + ([`scan`, `reverse manual index`] as const).map((path) => ({ mode, path })), + ), +)( + `keeps bounded public rank order with $mode $path`, + async ({ mode, path }) => { + const source = makeSource(mode, path) let live: ReturnType | undefined - try { - await source.stateWhenReady() + await withOracleCleanup(async () => { + if (mode === `eager`) await source.stateWhenReady() + else expect(source.isReady()).toBe(true) if (path === `reverse manual index`) { source.createIndex((row) => row.rank, { indexType: BTreeIndex, @@ -64,7 +101,7 @@ it.each([`scan`, `reverse manual index`] as const)( compareOptions: { direction: `asc`, nulls: `first` }, }, ], - limit: 1, + limit: 2, })! expect(direct.map((change) => change.value.id)).toEqual(expectedIds) @@ -72,13 +109,10 @@ it.each([`scan`, `reverse manual index`] as const)( q .from({ row: source }) .orderBy(({ row }) => row.rank) - .limit(1), + .limit(2), ) await live.preload() expect([...live.values()].map((row) => row.id)).toEqual(expectedIds) - } finally { - await live?.cleanup() - await source.cleanup() - } + }, [() => live?.cleanup(), () => source.cleanup()]) }, ) diff --git a/packages/db/tests/query/mixed-index-order-oracle.test.ts b/packages/db/tests/query/mixed-index-order-oracle.test.ts new file mode 100644 index 0000000000..6d3ba8489e --- /dev/null +++ b/packages/db/tests/query/mixed-index-order-oracle.test.ts @@ -0,0 +1,449 @@ +/** + * An index is a physical route for an `orderBy` query. It cannot choose a + * different public prefix from a full-source scan of the same query, either + * on the first read or after a source value changes kind. This follows the + * `orderBy`/`limit` contract in docs/guides/live-queries.md; it does not define + * a new global precedence among unlike JavaScript values. + * + * The reference is a separate ready Collection with auto-indexing disabled. + * It holds the same keyed source rows as the indexed Collection and executes + * the same public query. Its full scan is independent of index storage and the + * index-only comparator being judged. The finite histories cross number, + * bigint, boolean, string, ordinary-object, and array kinds, then update, + * insert, or delete rows to enter and leave mixed domains. Ascending and + * descending reads also exercise the reverse index adapter. Values are not + * mutated in place. Every value set has a distinct first and second row, so + * the law does not depend on an unspecified tie order. These witnesses do not + * prescribe order for every mixed-kind combination or provider paging. + * + * After preload and after both source copies commit each step, direct ordered snapshots + * and both live-query Collections must return the same exact ordered IDs. + * The scan and indexed sources publish identical sync transactions. Running + * both BasicIndex and BTreeIndex proves the optimizer cannot silently change + * the query result. The original type-name index order loses at the exact + * public-prefix comparison; a same-type step checks that the repair still + * permits indexed traversal when its order agrees with the query. + */ +import { expect, it, vi } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { BasicIndex } from '../../src/indexes/basic-index.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' +import { createLiveQueryCollection } from '../../src/query/live-query-collection.js' +import { PropRef } from '../../src/query/ir.js' +import { evaluateReferenceExpression } from '../reference-expression-oracle.js' +import { mockSyncCollectionOptions, withOracleCleanup } from '../utils.js' +import type { Collection } from '../../src/collection/index.js' +import type { + LoadSubsetOptions, + SyncConfig, + UtilsRecord, +} from '../../src/types.js' +import type { BasicExpression } from '../../src/query/ir.js' +import type { BaseIndex } from '../../src/indexes/base-index.js' + +type Value = number | bigint | boolean | string | { x: number } | Array +type Row = { id: string; value: Value } +type Step = + { type: `insert` | `update`; row: Row } | { type: `delete`; id: string } +type Scenario = { + name: string + initial: Array + steps: Array + indexedReads: Array +} + +const histories: Array = [ + { + name: `number and bigint`, + initial: [ + { id: `five`, value: 5n }, + { id: `ten`, value: 10n }, + { id: `three`, value: 3 }, + ], + steps: [ + { type: `update`, row: { id: `three`, value: 12n } }, + { type: `insert`, row: { id: `minus`, value: -4 } }, + { type: `delete`, id: `minus` }, + ], + indexedReads: [false, true, false, true], + }, + { + name: `number index gains and loses a bigint`, + initial: [ + { id: `three`, value: 3 }, + { id: `five`, value: 5 }, + ], + steps: [ + { type: `insert`, row: { id: `minus`, value: -1n } }, + { type: `update`, row: { id: `minus`, value: 7 } }, + { type: `update`, row: { id: `minus`, value: -2n } }, + ], + indexedReads: [true, false, true, false], + }, + { + name: `boolean and number`, + initial: [ + { id: `false`, value: false }, + { id: `five`, value: 5 }, + { id: `minus`, value: -1 }, + ], + steps: [ + { type: `update`, row: { id: `false`, value: 7 } }, + { type: `insert`, row: { id: `true`, value: true } }, + { type: `delete`, id: `true` }, + ], + indexedReads: [false, true, false, true], + }, + { + name: `object and string`, + initial: [ + { id: `object`, value: { x: 1 } }, + { id: `b`, value: `b` }, + { id: `a`, value: `a` }, + ], + steps: [ + { type: `update`, row: { id: `object`, value: `c` } }, + { type: `insert`, row: { id: `other`, value: { x: 2 } } }, + { type: `delete`, id: `other` }, + ], + indexedReads: [false, true, false, true], + }, + { + name: `array and ordinary object`, + initial: [ + { id: `array`, value: [1] }, + { id: `object`, value: { x: 1 } }, + ], + steps: [ + { type: `update`, row: { id: `array`, value: { x: 2 } } }, + { type: `insert`, row: { id: `array-again`, value: [2] } }, + { type: `delete`, id: `array-again` }, + ], + indexedReads: [false, true, false, true], + }, +] + +function orderedSnapshot( + source: Pick< + Collection, + `currentStateAsChanges` + >, + direction: `asc` | `desc`, +): Array { + return source + .currentStateAsChanges({ + orderBy: [ + { + expression: new PropRef([`value`]), + compareOptions: { direction, nulls: `first` }, + }, + ], + limit: 2, + })! + .map((change) => change.value.id) +} + +function applyStep( + source: ReturnType>, + rows: Map, + step: Step, +): void { + source.utils.begin() + if (step.type === `delete`) { + source.utils.write({ type: `delete`, value: rows.get(step.id)! }) + rows.delete(step.id) + } else { + source.utils.write({ type: step.type, value: step.row }) + rows.set(step.row.id, step.row) + } + source.utils.commit() +} + +for (const scenario of histories) { + for (const IndexType of [BasicIndex, BTreeIndex]) { + for (const direction of [`asc`, `desc`] as const) { + it(`${IndexType.name} preserves ${scenario.name} ${direction} public order across kind changes`, async () => { + const scanOptions = mockSyncCollectionOptions({ + id: `mixed-order-scan-${scenario.name}-${IndexType.name}`, + getKey: (row) => row.id, + initialData: scenario.initial, + autoIndex: `off`, + }) + const indexedOptions = mockSyncCollectionOptions({ + id: `mixed-order-indexed-${scenario.name}-${IndexType.name}`, + getKey: (row) => row.id, + initialData: scenario.initial, + autoIndex: `eager`, + defaultIndexType: IndexType, + }) + const scan = createCollection(scanOptions) + const indexed = createCollection(indexedOptions) + const scanRows = new Map(scenario.initial.map((row) => [row.id, row])) + const indexedRows = new Map(scanRows) + let scanLive: ReturnType | undefined + let indexedLive: + ReturnType | undefined + await withOracleCleanup(async () => { + await Promise.all([scan.stateWhenReady(), indexed.stateWhenReady()]) + // Keep one ascending physical index so descending reads must use + // ReverseIndex rather than auto-creating a descending index. + indexed.createIndex((row) => row.value, { indexType: IndexType }) + const check = (checkpoint: string, expectIndexRead: boolean) => { + const expected = orderedSnapshot(scan, direction) + const indexes = [...indexed.indexes.values()] + expect(indexes, `${checkpoint}: index exists`).toHaveLength(1) + const index = indexes[0] as BaseIndex + expect(index, `${checkpoint}: index type`).toBeInstanceOf(IndexType) + expect(index.matchesDirection(`asc`)).toBe(true) + const take = vi.spyOn( + index, + direction === `asc` ? `takeFromStart` : `takeReversedFromEnd`, + ) + try { + expect( + orderedSnapshot(indexed, direction), + `${checkpoint}: direct prefix`, + ).toEqual(expected) + if (expectIndexRead) + expect(take, `${checkpoint}: indexed route`).toHaveBeenCalled() + else + expect( + take, + `${checkpoint}: full-scan route`, + ).not.toHaveBeenCalled() + } finally { + take.mockRestore() + } + expect( + [...scanLive!.values()].map((row) => row.id), + `${checkpoint}: scan live prefix`, + ).toEqual(expected) + expect( + [...indexedLive!.values()].map((row) => row.id), + `${checkpoint}: indexed live prefix`, + ).toEqual(expected) + } + // Compile both live queries while the initial source domain is mixed. + scanLive = createLiveQueryCollection((q) => + q + .from({ row: scan }) + .orderBy(({ row }) => row.value, direction) + .limit(2), + ) + indexedLive = createLiveQueryCollection((q) => + q + .from({ row: indexed }) + .orderBy(({ row }) => row.value, direction) + .limit(2), + ) + await Promise.all([scanLive.preload(), indexedLive.preload()]) + check(`initial domain`, scenario.indexedReads[0]!) + for (const [position, step] of scenario.steps.entries()) { + applyStep(scanOptions, scanRows, step) + applyStep(indexedOptions, indexedRows, step) + check( + `step ${position + 1}: ${step.type}`, + scenario.indexedReads[position + 1]!, + ) + } + }, [ + () => indexedLive?.cleanup(), + () => scanLive?.cleanup(), + () => indexed.cleanup(), + () => scan.cleanup(), + ]) + }) + } + } +} + +/** + * The eager histories above do not cross provider demand. This receiving + * history begins with a homogeneous numeric index and a finite on-demand + * provider. A later bigint insert makes that retained index ineligible for + * a new ordered plan. The independent provider filters requested predicates, + * applies cursors, and returns only the requested page from its own source + * rows. At each settled window, the model sorts those rows by numeric value + * and compares the exact public IDs. The bigint first enters the visible + * prefix, then moves beyond it; a subsequent deletion requires a missing + * provider row to enter the bounded result. A later page reaches a cursor. + * This checks the public result, not whether an existing loader avoids every + * unnecessary index read. + */ +type NumericRow = { id: string; value: number | bigint } + +// The shared request reference intentionally accepts only like-typed numeric +// operands. This local extension gives the provider its legal number/bigint +// cursor semantics using JavaScript's direct relational order. +function evaluateNumericRequest( + expression: BasicExpression, + row: NumericRow, +): unknown { + if (expression.type !== `func`) + return evaluateReferenceExpression(expression, row) + if (expression.name === `and` || expression.name === `or`) { + const values = expression.args.map((arg) => + evaluateNumericRequest(arg, row), + ) + return expression.name === `and` + ? values.every(Boolean) + : values.some(Boolean) + } + if ([`gt`, `gte`, `lt`, `lte`].includes(expression.name)) { + const [left, right] = expression.args.map((arg) => + evaluateNumericRequest(arg, row), + ) + if ( + (typeof left === `number` || typeof left === `bigint`) && + (typeof right === `number` || typeof right === `bigint`) + ) { + const sign = left < right ? -1 : left > right ? 1 : 0 + switch (expression.name) { + case `gt`: + return sign > 0 + case `gte`: + return sign >= 0 + case `lt`: + return sign < 0 + case `lte`: + return sign <= 0 + } + } + } + return evaluateReferenceExpression(expression, row) +} + +it(`preserves a retained on-demand window after its index domain becomes mixed`, async () => { + const initial: Array = [ + { id: `three`, value: 3 }, + { id: `five`, value: 5 }, + { id: `ten`, value: 10 }, + { id: `twenty`, value: 20 }, + { id: `thirty`, value: 30 }, + ] + const truth = new Map(initial.map((row) => [row.id, row])) + const installed = new Map(initial.slice(0, 2).map((row) => [row.id, row])) + const requests: Array = [] + let sync!: Parameters[`sync`]>[0] + const compare = (a: NumericRow, b: NumericRow) => + a.value < b.value ? -1 : a.value > b.value ? 1 : 0 + const expected = (offset = 0, limit = 2) => + [...truth.values()] + .sort(compare) + .slice(offset, offset + limit) + .map((row) => row.id) + + const source = createCollection({ + id: `mixed-index-order-on-demand`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `off`, + sync: { + sync: (operations) => { + sync = operations + sync.begin() + for (const row of installed.values()) { + sync.write({ type: `insert`, value: row }) + } + sync.commit() + sync.markReady() + return { + loadSubset: async (options) => { + requests.push(options) + const matched = [...truth.values()] + .filter( + (row) => + (!options.where || + evaluateNumericRequest(options.where, row) === true) && + (!options.cursor || + evaluateNumericRequest(options.cursor.whereFrom, row) === + true), + ) + .sort(compare) + const offset = options.cursor ? 0 : (options.offset ?? 0) + const page = matched.slice( + offset, + options.limit === undefined ? undefined : offset + options.limit, + ) + sync.begin() + for (const row of page) { + if (installed.get(row.id) === row) continue + sync.write({ + type: installed.has(row.id) ? `update` : `insert`, + value: row, + }) + installed.set(row.id, row) + } + await sync.commit() + }, + unloadSubset: () => {}, + } + }, + }, + }) + let live: ReturnType | undefined + let take: { mockRestore: () => void } | undefined + await withOracleCleanup(async () => { + await source.stateWhenReady() + const index = source.createIndex((row) => row.value, { + indexType: BTreeIndex, + }) + expect(index.supportsOrderOptimization).toBe(true) + take = vi.spyOn(index, `takeFromStart`) + live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.value) + .limit(2), + ) + await live.preload() + expect([...live.values()].map((row) => row.id)).toEqual(expected()) + expect(take).toHaveBeenCalled() + + const mixed: NumericRow = { id: `four`, value: 4n } + truth.set(mixed.id, mixed) + sync.begin() + sync.write({ type: `insert`, value: mixed }) + installed.set(mixed.id, mixed) + await sync.commit() + expect(index.supportsOrderOptimization).toBe(false) + await live.utils.setWindow({ offset: 0, limit: 2 }) + expect(expected()).toEqual([`three`, `four`]) + expect([...live.values()].map((row) => row.id)).toEqual(expected()) + + const moved: NumericRow = { id: `four`, value: 15n } + truth.set(moved.id, moved) + sync.begin() + sync.write({ type: `update`, value: moved }) + installed.set(moved.id, moved) + await sync.commit() + await live.utils.setWindow({ offset: 0, limit: 2 }) + expect(expected()).toEqual([`three`, `five`]) + expect([...live.values()].map((row) => row.id)).toEqual(expected()) + expect(installed.has(`ten`)).toBe(false) + + const requestsBeforeDeletion = requests.length + truth.delete(`three`) + sync.begin() + sync.write({ type: `delete`, value: initial[0]! }) + installed.delete(`three`) + await sync.commit() + await live.utils.setWindow({ offset: 0, limit: 2 }) + expect(expected()).toEqual([`five`, `ten`]) + expect([...live.values()].map((row) => row.id)).toEqual(expected()) + expect(installed.has(`ten`)).toBe(true) + expect( + requests + .slice(requestsBeforeDeletion) + .some((request) => request.orderBy && request.limit === 2), + ).toBe(true) + + const requestsBeforePage = requests.length + await live.utils.setWindow({ offset: 2, limit: 2 }) + expect([...live.values()].map((row) => row.id)).toEqual(expected(2, 2)) + expect( + requests.slice(requestsBeforePage).some((request) => request.cursor), + ).toBe(true) + }, [() => take?.mockRestore(), () => live?.cleanup(), () => source.cleanup()]) +}) diff --git a/packages/db/tests/temporal-index-order-oracle.test.ts b/packages/db/tests/temporal-index-order-oracle.test.ts index b4d78974a8..3e041f0022 100644 --- a/packages/db/tests/temporal-index-order-oracle.test.ts +++ b/packages/db/tests/temporal-index-order-oracle.test.ts @@ -5,7 +5,7 @@ import { BTreeIndex } from '../src/indexes/btree-index' import { createCollection } from '../src/collection/index' import { createLiveQueryCollection } from '../src/query/live-query-collection' import { PropRef } from '../src/query/ir' -import { mockSyncCollectionOptions } from './utils' +import { mockSyncCollectionOptions, withOracleCleanup } from './utils' import type { CompareOptions } from '../src/query/builder/types' /** @@ -21,9 +21,11 @@ import type { CompareOptions } from '../src/query/builder/types' * direction, and equal-instant ties, then compares exact ordered keys after * the action. A ready source exercises scan and indexed live-query paths at * preload; an on-demand Collection checks the first published ordered row. - * Index spies prove those paths read the index. The oracle does not claim mixed - * Temporal kinds, custom index callbacks, mutated built-ins, or order for - * Temporal kinds whose public comparison is undefined. + * Index spies prove the named indexed reads. The PlainDate lane checks one + * extended-year calendar boundary; the exception lane requires `TypeError` + * for its named mixed or unordered Temporal pairs and claims no ordered + * result for them. Custom index callbacks, mutated built-ins, and other + * mixed Temporal pairs remain outside this owner. */ type Row = { id: string; value: Temporal.ZonedDateTime } const rows: ReadonlyArray = [ @@ -456,3 +458,187 @@ describe(`Duration indexed time order`, () => { }) } }) + +/** + * # Does string collation change PlainDate calendar order? + * + * Changelog #1519 promises calendar order for PlainDate. With equal month and + * day, the year decides: -100000 precedes +10000. Their ISO display strings + * have the opposite lexical order, so sorting normalized text would choose the + * wrong first row. String collation applies to strings, not Temporal values. + * + * A ready two-row source is read through a scan, BasicIndex, or BTreeIndex. + * The model reads the supplied years, while direct ordered snapshots and + * live-query preload expose the public first row. Index spies establish the + * direct indexed reads; preload is a parity check. This fixed history does + * not claim all extended-year pairs or on-demand publication. + */ +describe(`PlainDate extended-year calendar order`, () => { + const calendarRows = [ + { id: `negative`, value: Temporal.PlainDate.from(`-100000-01-01`) }, + { id: `positive`, value: Temporal.PlainDate.from(`+010000-01-01`) }, + ] + const lexicalOrder = { + direction: `asc` as const, + nulls: `first` as const, + stringSort: `lexical` as const, + } + + it(`calibrates opposing year and display orders`, () => { + expect(calendarRows[0]!.value.year).toBeLessThan( + calendarRows[1]!.value.year, + ) + expect( + calendarRows[0]!.value.toString() > calendarRows[1]!.value.toString(), + ).toBe(true) + }) + + it.each([`scan`, `BasicIndex`, `BTreeIndex`] as const)( + `keeps calendar order with %s`, + async (path) => { + const source = createCollection( + mockSyncCollectionOptions({ + id: `plain-date-extended-year-${path}`, + getKey: (row: (typeof calendarRows)[number]) => row.id, + initialData: calendarRows, + autoIndex: `off`, + }), + ) + let live: ReturnType | undefined + await withOracleCleanup(async () => { + await source.stateWhenReady() + const index = + path === `scan` + ? undefined + : source.createIndex((row) => row.value, { + indexType: path === `BasicIndex` ? BasicIndex : BTreeIndex, + options: { compareOptions: lexicalOrder }, + }) + const take = index && vi.spyOn(index, `takeFromStart`) + try { + const direct = source.currentStateAsChanges({ + orderBy: [ + { + expression: new PropRef([`value`]), + compareOptions: lexicalOrder, + }, + ], + limit: 1, + })! + expect(direct.map((change) => change.value.id)).toEqual([`negative`]) + if (take) expect(take).toHaveBeenCalled() + + live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.value, lexicalOrder) + .limit(1), + ) + await live.preload() + expect([...live.values()].map((row) => row.id)).toEqual([`negative`]) + } finally { + take?.mockRestore() + } + }, [() => live?.cleanup(), () => source.cleanup()]) + }, + ) +}) + +/** + * # Can an index hide an undefined public Temporal order? + * + * Changelog #1519 requires TanStack DB to reject ordering two different + * Temporal kinds; PlainMonthDay has no static comparison. The model therefore + * predicts a TypeError for either two-row ordering, regardless of which row + * would be first. Index storage may assign a physical position to both values, + * but that position cannot turn this query into a successful public read. + * + * Each fixed history starts a ready source with both rows, optionally creates + * a BasicIndex or BTreeIndex, then checks the direct ordered snapshot and a + * live-query preload. These are public observation cuts; the oracle does not + * claim that a later invalid update leaves a usable Collection or that an + * on-demand provider has a matching error boundary. + */ +type UnorderedTemporalRow = { + id: string + value: Temporal.PlainDate | Temporal.PlainDateTime | Temporal.PlainMonthDay +} +const unorderedTemporalCases: ReadonlyArray<{ + name: string + rows: Array +}> = [ + { + name: `different Temporal kinds`, + rows: [ + { id: `date`, value: Temporal.PlainDate.from(`2024-01-01`) }, + { + id: `date-time`, + value: Temporal.PlainDateTime.from(`2024-01-02T00:00`), + }, + ], + }, + { + name: `PlainMonthDay without an order`, + rows: [ + { id: `january`, value: Temporal.PlainMonthDay.from(`01-01`) }, + { id: `february`, value: Temporal.PlainMonthDay.from(`02-01`) }, + ], + }, +] + +describe(`undefined Temporal public order`, () => { + it.each(unorderedTemporalCases)( + `calibrates the $name premise`, + ({ name, rows: sampleRows }) => { + if (name === `different Temporal kinds`) { + expect(Object.prototype.toString.call(sampleRows[0]!.value)).not.toBe( + Object.prototype.toString.call(sampleRows[1]!.value), + ) + } else { + expect(Reflect.get(Temporal.PlainMonthDay, `compare`)).toBeUndefined() + } + }, + ) + + for (const scenario of unorderedTemporalCases) { + for (const path of [`scan`, `BasicIndex`, `BTreeIndex`] as const) { + it(`${path} keeps the ${scenario.name} public error`, async () => { + const source = createCollection( + mockSyncCollectionOptions({ + id: `undefined-temporal-order-${scenario.name}-${path}`, + getKey: (row) => row.id, + initialData: scenario.rows, + autoIndex: `off`, + }), + ) + let live: ReturnType | undefined + await withOracleCleanup(async () => { + await source.stateWhenReady() + if (path !== `scan`) { + source.createIndex((row) => row.value, { + indexType: path === `BasicIndex` ? BasicIndex : BTreeIndex, + }) + } + expect(() => + source.currentStateAsChanges({ + orderBy: [ + { + expression: new PropRef([`value`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + limit: 1, + }), + ).toThrow(TypeError) + live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.value) + .limit(1), + ) + await expect(live.preload()).rejects.toThrow(TypeError) + }, [() => live?.cleanup(), () => source.cleanup()]) + }) + } + } +}) From 85074b7b674f58043908c02701cca1be3af79bc7 Mon Sep 17 00:00:00 2001 From: Isaac Date: Tue, 13 Oct 2026 17:03:55 -0600 Subject: [PATCH 46/46] docs(db): correct Temporal admission review record --- .../oracle-reviews/pr-2116-confirmed-fixes.md | 7 ++++--- .../oracle-reviews/pr-2116-deferred-findings.md | 8 ++++---- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md b/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md index 4869509a3a..676213d656 100644 --- a/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md +++ b/docs/contributing/oracle-reviews/pr-2116-confirmed-fixes.md @@ -38,9 +38,10 @@ The captured-`inArray` list rebuild and per-row custom index callback ordering were proposed extensions, then withdrawn by the maintainer. This PR does not promise behavior after callers modify standard data objects in place, unsigned byte ordering, or special handling for malformed Temporal lookalikes. The -[deferred record](pr-2116-deferred-findings.md) keeps valid Temporal admission, -mixed-kind query order, the confirmed pre-existing indexed Temporal exception -gap, and earlier-branch observations separate from the confirmed fixes above. +[earlier-findings record](pr-2116-deferred-findings.md) keeps Temporal admission +observations, mixed-kind query order, the confirmed pre-existing indexed +Temporal exception gap, and earlier-branch observations separate from the +confirmed fixes above. The repeatable workflow is: freeze source assertions in a loss ledger; name the receiving law and its limits; obtain a baseline failure before production diff --git a/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md b/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md index bf20f476bf..ebb483c161 100644 --- a/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md +++ b/docs/contributing/oracle-reviews/pr-2116-deferred-findings.md @@ -1,16 +1,16 @@ -# PR #2116: deferred findings and guarantee decisions +# PR #2116: earlier findings and guarantee decisions This record separates findings discovered during the unit-test retirement workflow from the bounded fixes retained in PR #2116. The transfer ledgers remain the authority for retired assertions. The [confirmed-fixes review](pr-2116-confirmed-fixes.md) records the baseline and -hostile evidence for fixes that this PR now keeps. A deferred finding below is -not closed by passing a neighboring transfer oracle. +hostile evidence for fixes that this PR now keeps. A finding below is not +closed by passing a neighboring transfer oracle. | Finding | Authority and evidence | Next owner and witness | | --- | --- | --- | | R29: two valid `Temporal.Duration` values can have the same display text but different fields. A changed field can be suppressed as an equal Collection value. | Closed in this PR for the tested direct and Collection paths. The Duration field model checks all ten fields; a same-display replacement and adjacent-unit histories make the former `deepEquals` and change-event behavior fail before the repair. The query equality owner and `in` evaluator owner separately check Duration field identity. | General multirow and live-query publication histories remain outside these owners. Add them to the relevant owner if a new failure reaches those paths. | -| R16: an indexed Collection can admit a valid Temporal row whose index comparator has no public order, then throw after partial state changes. | Earlier PR revisions recorded RED admission and publication traces for Duration, PlainMonthDay, and mixed-kind rows. Source inspection suggests the risk remains on `origin/main`, but this narrowed PR has not executed a current-base RED witness. The public order law and internal index storage order must be distinguished. | The direct-index and Collection publication owners should pair a valid-row admission history with exact membership, ordered storage, stored rows, and subscriber observations. Choose the smallest internal ordering or atomic-admission repair that satisfies the accepted contract. | +| R16: an indexed Collection can admit a valid Temporal row whose index comparator has no public order, then throw after partial state changes. | Earlier PR revisions recorded RED admission and publication traces for Duration, PlainMonthDay, and mixed-kind rows. A paired probe on `origin/main` `bbbf017ef5` and PR head `0c10ae8e8` found no failure: 190 valid Temporal pairs per index type and revision passed direct insertion; 24 two-row sync and optimistic Collection histories retained both public rows, exact memberships, ordered storage, ready status, and both insert publications at the second write return. The earlier RED traces establish an intermediate-branch failure, not a current-base or current-head defect. Public query comparison errors remain separate from internal index storage order. | No admission repair follows from this evidence. If a valid current-head admission history fails, extend the direct-index and Collection publication owners with exact membership, ordered storage, stored rows, and subscriber observations before changing production. The probe does not cover all valid Temporal values, native-host implementations, updates, builds, or longer histories. | | Mixed-kind indexed order and custom-index query selection may disagree with public query results. | The source-Map owner closes the mixed number/string direct-index key loss. The new scan-versus-index owner rejected the frozen PR's wrong mixed-kind public prefixes, and the current guard makes the tested eager paths use the query's order. A retained on-demand BTree history checks visible prefixes after a number/bigint transition, a missing-row finite request, and a later cursor request that still traverses the index; all settled public windows match independently sorted provider rows. A reverse manual index produced a wrong direct prefix on an earlier branch revision; the current three-rank owner checks direct, eager, and prepopulated on-demand public results. The reviewer withdrew its claimed live-query failure after fixing a cross-checkout `@tanstack/db-ivm` resolution error. The maintainer withdrew a per-row custom callback position guarantee for comparator-equal exact values. | Other retained on-demand paging sequences, different incomplete-source patterns, and later live updates need public-result witnesses. The owners do not define a universal precedence among unlike kinds. | | Indexed ordering can suppress a required Temporal comparison error. | Changelog #1519 requires `TypeError` for mixed or unordered Temporal values. On clean `origin/main`, indexed direct reads returned rows for `PlainDate` beside `PlainDateTime` and for two distinct `PlainMonthDay` values while the scan threw. The new Temporal exception lane fails on those four baseline indexed reads and passes on this branch; eager live queries already threw on main. | The bounded direct-read repair is complete for these pairs. On-demand acquisition, later updates, and other mixed Temporal pairs need separate witnesses before claiming the whole exception class. | | R17: an equal-time Date removal left an ordered ghost under added index machinery. | The recorded RED trace depended on code introduced during this PR. It does not establish an `origin/main` defect. The stronger rule for a substituted old row with a different callback position was withdrawn. | If ordinary direct remove/update still has a current-base counterexample, add a source-Map history to the direct-index owner before changing production. |