Skip to content

Phase 6c: Pagination engine, built-in strategies, and verbatim query splice (#17) - #46

Merged
Wahbeh-Mohammad merged 1 commit into
14-phase-6from
17-phase-6c-pagination
Aug 27, 2026
Merged

Phase 6c: Pagination engine, built-in strategies, and verbatim query splice (#17)#46
Wahbeh-Mohammad merged 1 commit into
14-phase-6from
17-phase-6c-pagination

Conversation

@Wahbeh-Mohammad

@Wahbeh-Mohammad Wahbeh-Mohammad commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

What Changed

  • Implemented core pagination primitives in @dexpace/core: Page<T>, PageInfo<T>, PaginationStrategy<T>, and Paginator<T> (PAGE-1..PAGE-15).
  • Dual consumption views: items() (reusable, closes responses before yielding items to prevent stranded connections) and pages() (single-use, raw response access).
  • Explicit Resource Management: Page<T> implements AsyncDisposable ([Symbol.asyncDispose]) for await using workflows (PAGE-3, PAGE-12).
  • Built-in pagination strategies: cursorStrategy, pageNumberStrategy, and linkHeaderStrategy with RFC 8288 multi-header Link parsing (PAGE-16..PAGE-20).
  • Internal verbatim query splicing (spliceQueryParam, readQueryParam) operating directly on raw query strings without URLSearchParams re-encoding (PAGE-21..PAGE-24).
  • Functional paginateWithFetchers() front-end threading mutable PagingOptions across first() and next() calls (PAGE-34, PAGE-35).
  • Pre-dispatch abort checks and post-abort response teardown for cancellation safety (PAGE-25, PAGE-26, PAGE-33).
  • Added Node conformance suite (test/node-conformance/pagination.test.mjs) validating AsyncDisposable, abort signal threading, and stream teardown on real Node Web Streams.

Reviews Done

  • Three-pass review protocol completed (Pass 1 Shape, Pass 2 Adversarial, Pass 3 Reader).
  • Addressed 12 code review findings:
    1. Unconditional AsyncDisposable on Page<T> and compiler lib documentation.
    2. Pre-dispatch response release preventing connection overlap across network exchanges.
    3. Pre-dispatch abort checks preventing discarded requests after cancellation.
    4. Population of PagingOptions.nextLink and continuationToken in fetchers.
    5. Exact close-count leak assertions in lifecycle test matrix.
    6. Line 1 SPDX-License-Identifier headers across all pagination files.
    7. Accurate docstrings for ECMAScript AsyncIteratorClose .return() unwinding.
    8. Coverage for non-positive maxPages validation branches.
    9. Clean PaginationError inheritance with forwarding constructor for coverage instrumentation.
    10. Node conformance tests for pagination.
    11. Traceability checklist (PAGE-1..PAGE-36) and citations for PAGE-29, PAGE-30, PAGE-32.
    12. Updated knowledge docs regarding held page release lifecycle.
  • Full verification gates passing: bun test (1527/1527 pass, 100% line/branch coverage across all 7 pagination files), test:node (82/82), typecheck, lint, api, verify:dual-consumption, verify:consumer-types, verify:seam-1, verify:runtime-floor, and lint:publish.

Open Items / Deferred Items

  • Recorded in docs/open-items.md Section I (I1–I8):
    • I1: PAGE-11 close-before-yield precedence over §7.1 illustrative snippet (resolved with erratum).
    • I2: PAGE-5 / PAGE-29 async PaginationStrategy.parse signature for stream compatibility (resolved by spec clarification).
    • I3: Page AsyncDisposable compiler lib requirement (ESNext.Disposable) documented.
    • I4: WHATWG query encode-set boundary handled via custom tokenizer (resolved by design).
    • I5: Transport-direct paginator execution without internal retry loops (resolved by design).
    • I6: items() vs pages() single-use asymmetry (resolved by design).
    • I7: Iterative generator drive without trampoline recursion (resolved by design).
    • I8: Error unwrapping preserving original transport/parse causes (resolved by design).
  • Requirements tracking: all requirements (PAGE-1 through PAGE-36) verified and marked executed in docs/superpowers/plans/2026-07-28-phase6c-pagination-checklist.md. No blocking open items.

@Wahbeh-Mohammad Wahbeh-Mohammad self-assigned this Aug 27, 2026
@Wahbeh-Mohammad
feat(core): phase 6c — pagination engine and built-in strategies (#17).
b3fd43a 
Implement the Phase 6c pagination subsystem, delivering transport- and serde-agnostic
lazy pagination walks with dual consumption views, built-in strategies, RFC 8288 Link
parsing, verbatim query parameter splicing, and deterministic response lifecycle management.

Closes PAGE-1 through PAGE-36:

@dexpace/core:
- `Page<T>`: wraps a live transport response and frozen materialized items (`PAGE-1`,
  `PAGE-2`, `PAGE-30`); implements `AsyncDisposable` (`[Symbol.asyncDispose]`) for
  `await using` explicit resource management; metadata and items survive `close()` (`PAGE-2`).
- `PageInfo<T>` & `PaginationStrategy<T>`: stateless async parser interface returning
  `Promise<PageInfo<T>>` with `{items, nextRequest}`; `undefined` signals end of stream
  (`PAGE-4`, `PAGE-5`, `PAGE-29`).
- `Paginator<T>`: lazy generator driver (`PAGE-6`, `PAGE-7`) exposing:
  - `items()`: reusable sequence view yielding server-order items, closing each page's
    underlying response before yielding any items to eliminate stranded connections (`PAGE-8`,
    `PAGE-11`).
  - `pages()`: single-use sequence view yielding whole `Page` objects with raw response
    access (`PAGE-14`).
  - `maxPages` cap enforcement evaluated before wire dispatch (`PAGE-9`, `PAGE-10`).
  - Strict cancellation safety with pre-dispatch abort checks and drop-and-close handling
    for in-flight responses arriving after cancellation (`PAGE-25`, `PAGE-26`, `PAGE-33`).
  - Iterative generator loop guaranteeing stack safety across thousands of pages (`PAGE-31`).
- Built-in pagination strategies (`PAGE-16`–`PAGE-20`):
  - `cursorStrategy()`: single body read with configurable cursor query parameter.
  - `pageNumberStrategy()`: 1-based (or configured) start-page fallback with next-page advance.
  - `linkHeaderStrategy()`: RFC 8288 Link header parser supporting multi-header concatenation,
    unquoted/quoted/case-insensitive `rel="next"`, and RFC 3986 reference resolution.
- `query-splice` (internal):
  - Hand-rolled query substring tokenizer splicing targeted parameters without `URLSearchParams`
    canonicalization, preserving untargeted query bytes and non-query components byte-for-byte
    (`PAGE-21`, `PAGE-22`, `PAGE-23`, `PAGE-24`).
- `paginateWithFetchers()`:
  - Higher-level functional front-end threading a single shared mutable `PagingOptions` bag
    across `first()` and `next()` invocations (`PAGE-34`, `PAGE-35`).
- `PaginationError`:
  - Flat leaf under `DexpaceError` for precondition and engine misuse; underlying network, I/O,
    and parse causes propagate unwrapped (`PAGE-28`).

Node conformance:
- `test/node-conformance/pagination.test.mjs`: validates `Page` explicit resource management,
  `Paginator` items and pages walks, `AbortSignal` thread-through, and response stream cancellation
  against real Node Web Streams.

Traceability & ledger:
- `docs/superpowers/plans/2026-07-28-phase6c-pagination-checklist.md` tracking `PAGE-1`..`PAGE-36`.
- `docs/open-items.md` Section I updated with ledger entries I1–I8 (close-before-yield precedence,
  async parse boundary, `AsyncDisposable` lib requirement, WHATWG query encode-set boundary,
  transport-direct execution, single-use view asymmetry, iterative loop drive, error unwrapping).

Full gate sequence green (1527 unit tests with 100% pagination line/branch coverage, 82 Node
conformance tests, publint, attw, dual-consumption, consumer-types, seam-1, runtime-floor).
@Wahbeh-Mohammad
Wahbeh-Mohammad merged commit 7c29255 into 14-phase-6 Aug 27, 2026
3 checks passed
@Wahbeh-Mohammad
Wahbeh-Mohammad deleted the 17-phase-6c-pagination branch August 27, 2026 16:44
Wahbeh-Mohammad added a commit that referenced this pull request Aug 27, 2026
@Wahbeh-Mohammad
Phase 6 — the serde seam, Server-Sent Events, and pagination (#48)
743f316 
* Phase 6a: Serde seam and @dexpace/codec-json (#45)

* feat: phase 6a — the serde seam and @dexpace/codec-json

Reshape the serialization seam around an explicit runtime type witness and ship
the workspace's second package.

Every decode now takes a caller-supplied `Schema<T>` value — the structural
`{parse(input: unknown): T}` shape Zod, Valibot, ArkType and effect/schema all
satisfy without an adapter. That closes SEAM-21: TypeScript erases types
completely, so the schema value *is* the reification, and because the compiler
infers `T` from it, the runtime witness and the static type are one artifact
rather than two kept in sync by convention. `Serde` consequently drops its type
parameter — a bundle is per wire format, not per DTO, which is what SERDE-1
actually says — so one `jsonSerde()` serves every DTO in an application.

Phase 2 kept `Serde<T>` out of the public barrel precisely so this reshape would
not break a published API. It is promoted here, forced rather than chosen:
`@dexpace/codec-json` is a separate package and can reach core only through its
public entry point.

@dexpace/core:
- `Schema`, `Serializer`, `Deserializer`, `Serde` — all four SEAM-20 allocation
  profiles, including the fresh-string one an earlier draft dropped
- `Tristate<T>` with PATCH three-state semantics; `present()` takes
  `NonNullable<T>`, so SERDE-14's illegal fourth state is unrepresentable at the
  type level rather than rejected at construction
- `SerializationError` / `DeserializationError` as two flat leaves under
  `DexpaceError` plus an `isSerdeError` guard — the tree stays two levels
- `serdeBody()`: the serde's declared media type is the default `Content-Type`,
  with no format-agnostic fallback anywhere on the path (SERDE-2)
- `decodeResponse()` / `decodeSuccessResponse()`, closing the response on every
  path via Phase 4b's `releaseQuietly`/`withReleaseFailure` so a teardown failure
  rides along as suppressed instead of displacing the decode failure

@dexpace/codec-json (new, zero external dependencies — NFR-2):
- `jsonSerde()`, the Tristate replacer installed by default with a named opt-out,
  and the `tristate()` / `tristateObject()` decode combinators

Workspace, the three Phase-0 deferrals a second package makes live:
- Bun `workspaces.catalog` as the single source of tool versions (NFR-14)
- `@dexpace/core` peer + `peerDependenciesMeta`, with a cross-package test that
  proves the consequence rather than the declaration: `TRISTATE_BRAND` is a
  registry-global `Symbol.for`, so the codec keeps recognizing a caller's
  Tristate values even across two non-identical copies of core
- `verify:seam-1`, `verify:consumer-types` and `verify:dual-consumption`
  generalized from core-only to every package

Three adversarial-pass bugs worth naming, all reproduced against the built
artifacts before fixing:
- a stream failure was re-typed as a payload failure — the guard tested
  `instanceof IoError` while core's I/O tree is flat, so four of five classes
  were re-stamped as `DeserializationError`, inverting SERDE-12
- a JSON key named `""` collided with the replacer's top-level detection and
  emitted `null` where the key should have been omitted, silently turning a PATCH
  "leave unchanged" into "clear"
- `key in source` walked the prototype chain, so all eleven `Object.prototype`
  member names decoded as Present when the wire omitted them

Deviations, deferrals and the two open questions this phase could not settle
(`IoError`'s reachability, `toHttpError`'s teardown masking) are recorded in
docs/open-items.md §H1–H17.

* fix: ci checks.

* fix: ci checks.

* feat(core): phase 6b — Server-Sent Events (SSE-1..41). (#47)

Implement the full Server-Sent Events subsystem in `@dexpace/core` per
`docs/product-spec/13-server-sent-events-and-streaming.md` (SSE-1 through SSE-41).

The implementation is strictly pull-based, single-pass, and zero-dependency:
events are parsed 1:1 on demand as the consumer polls the stream, with no
unbounded buffering and no auto-reconnection logic.

@dexpace/core additions:
- `SseEvent`, `makeSseEvent`: Immutable event representation with defensively
  copied and frozen `data` lines (`SSE-20`). Structural equality (`sseEventsEqual`),
  string representation (`sseEventToString`), and content predicate
  (`isSseEventEmpty`, where comments count as content per `SSE-22`).
- `SseLineReader` (@internal): Hand-rolled line framing supporting `\n`, `\r`, and
  `\r\n` line terminators across chunk boundaries (`SSE-2`), start-only UTF-8 BOM
  stripping via non-consuming `peek()` lookahead (`SSE-12`), and an optional
  configurable line length cap (`maxLineBytes` / `SseLineTooLongError`, `SSE-19`).
- `SseParser` (@internal): Single-pass state machine implementing WHATWG SSE
  grammar with the three reference spec deviations: comments captured and
  dispatched (`SSE-6`), permissive dispatch when any of the 5 fields are set
  (`SSE-13`), and EOF dispatch of pending fields (`SSE-14`). Ignores NUL in IDs
  (`SSE-9`), unknown fields (`SSE-7`), and non-digit / overflow retries (`SSE-11`).
- `SseStream` / `sseStreamFrom`: Resource-owning single-pass AsyncGenerator
  facade (`SSE-18`, `SSE-23`–`SSE-32`). Guarantees exactly-once release of the
  underlying Response body and BufferedSource across all termination routes (clean
  EOF, explicit `close()`, early `break`, consumer error, mid-stream read error,
  or abort signal). Teardown promise memoization ensures concurrent `close()`
  awaits in-flight releases and propagates failures (`SSE-30`). In-flight reader
  teardown on close is mapped to `IoError` (`SSE-31`). Abort listeners are
  cleaned up on normal completion.
- `typedSseStream`: Lazy per-element stream adapter passing raw event name and
  newline-joined data (`SSE-33`, `SSE-35`). Dispatches `MapperOutcome<T>` union
  (`mapperValue`, `MAPPER_SKIP`, `MAPPER_DONE`, `SSE-34`). Releases stream
  resource before propagating any mapper error, attaching close failure as
  suppressed (`SSE-36`).

Tooling, gates, and conformance:
- `scripts/verify-sse-37.mjs` & `test:scripts`: Recursive AST/regex gate enforcing
  zero serde dependencies in core SSE (`SSE-37`) and no reconnect / `Last-Event-ID`
  paths (`SSE-38`).
- `test/node-conformance/sse.test.mjs`: 8 new conformance test cases running over
  real Node Web Streams and TextDecoder under `node --test` across Node 20.3.0 and
  LTS (`test:node`).
- `docs/superpowers/plans/2026-07-28-phase6b-sse-checklist.md`: Full requirement
  traceability checklist mapping all 41 SSE requirements to code and tests.
- `docs/open-items.md`: Recorded Section I entries for `SSE-41` reactive adapter
  deferral (Phase 8b `@dexpace/rx`), line reader separation rationale (`IO-14` vs
  `SSE-2`), `Symbol.asyncDispose` runtime floor guard, and JavaScript hash equality.

Gates verified: typecheck, lint, build, bun test (1,514 passing, 100% coverage on
src/sse/*), api, lint:publish (publint + attw), verify:dual-consumption,
verify:consumer-types, verify:seam-1, verify:sse-37, test:scripts (40 passing),
verify:runtime-floor, test:node (87 passing), and audit.

* feat(core): phase 6c — pagination engine and built-in strategies (#17). (#46)

Implement the Phase 6c pagination subsystem, delivering transport- and serde-agnostic
lazy pagination walks with dual consumption views, built-in strategies, RFC 8288 Link
parsing, verbatim query parameter splicing, and deterministic response lifecycle management.

Closes PAGE-1 through PAGE-36:

@dexpace/core:
- `Page<T>`: wraps a live transport response and frozen materialized items (`PAGE-1`,
  `PAGE-2`, `PAGE-30`); implements `AsyncDisposable` (`[Symbol.asyncDispose]`) for
  `await using` explicit resource management; metadata and items survive `close()` (`PAGE-2`).
- `PageInfo<T>` & `PaginationStrategy<T>`: stateless async parser interface returning
  `Promise<PageInfo<T>>` with `{items, nextRequest}`; `undefined` signals end of stream
  (`PAGE-4`, `PAGE-5`, `PAGE-29`).
- `Paginator<T>`: lazy generator driver (`PAGE-6`, `PAGE-7`) exposing:
  - `items()`: reusable sequence view yielding server-order items, closing each page's
    underlying response before yielding any items to eliminate stranded connections (`PAGE-8`,
    `PAGE-11`).
  - `pages()`: single-use sequence view yielding whole `Page` objects with raw response
    access (`PAGE-14`).
  - `maxPages` cap enforcement evaluated before wire dispatch (`PAGE-9`, `PAGE-10`).
  - Strict cancellation safety with pre-dispatch abort checks and drop-and-close handling
    for in-flight responses arriving after cancellation (`PAGE-25`, `PAGE-26`, `PAGE-33`).
  - Iterative generator loop guaranteeing stack safety across thousands of pages (`PAGE-31`).
- Built-in pagination strategies (`PAGE-16`–`PAGE-20`):
  - `cursorStrategy()`: single body read with configurable cursor query parameter.
  - `pageNumberStrategy()`: 1-based (or configured) start-page fallback with next-page advance.
  - `linkHeaderStrategy()`: RFC 8288 Link header parser supporting multi-header concatenation,
    unquoted/quoted/case-insensitive `rel="next"`, and RFC 3986 reference resolution.
- `query-splice` (internal):
  - Hand-rolled query substring tokenizer splicing targeted parameters without `URLSearchParams`
    canonicalization, preserving untargeted query bytes and non-query components byte-for-byte
    (`PAGE-21`, `PAGE-22`, `PAGE-23`, `PAGE-24`).
- `paginateWithFetchers()`:
  - Higher-level functional front-end threading a single shared mutable `PagingOptions` bag
    across `first()` and `next()` invocations (`PAGE-34`, `PAGE-35`).
- `PaginationError`:
  - Flat leaf under `DexpaceError` for precondition and engine misuse; underlying network, I/O,
    and parse causes propagate unwrapped (`PAGE-28`).

Node conformance:
- `test/node-conformance/pagination.test.mjs`: validates `Page` explicit resource management,
  `Paginator` items and pages walks, `AbortSignal` thread-through, and response stream cancellation
  against real Node Web Streams.

Traceability & ledger:
- `docs/superpowers/plans/2026-07-28-phase6c-pagination-checklist.md` tracking `PAGE-1`..`PAGE-36`.
- `docs/open-items.md` Section I updated with ledger entries I1–I8 (close-before-yield precedence,
  async parse boundary, `AsyncDisposable` lib requirement, WHATWG query encode-set boundary,
  transport-direct execution, single-use view asymmetry, iterative loop drive, error unwrapping).

Full gate sequence green (1527 unit tests with 100% pagination line/branch coverage, 82 Node
conformance tests, publint, attw, dual-consumption, consumer-types, seam-1, runtime-floor).

* docs: update changesets for phase6.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@Wahbeh-Mohammad Wahbeh-Mohammad

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Wahbeh-Mohammad