feat(transport): bound streaming, framing and queued work #244

Own data flow across public APIs and transports: raw request/response boundaries, pending input admission, terminal channel wakeups and bounded destination output.

This is the sole active owner of audit IDs 04, 09, 19, 24, 55 from the archived #203. Original acceptance and evidence remain there. Session open/close/cancel deadlines and mobile lifecycle belong to #269; guest SQL recovery to #201; persistent database publication to #249. This issue owns tool-output destination publication; #249 owns PGDATA/root publication. Each item below includes its own behavioral verification; shared packaged-product/integration evidence is owned by #213.

Reviewed 8 October 2026: main 293f31f4; integration #218 at 3b1f7cd4. #255 merged through #247 into #218; #268 is also in #218. #218 remains open, so candidate changes are not on main or published.

Current status

Main #209 moved ordinary native broker SQL onto pgwire and #219 added mobile file/stream physical backup. #218 removes the introduced 64 MiB aggregate tool-output quota, but whole-result APIs still collect output. Destination APIs and bounded end-to-end flow remain unfinished; do not reimplement already-landed mobile archive streaming or describe the old custom SQL protocol as current.

Transport correctness before adding destinations

• Raw exchange boundary (04). Define and enforce one admitted exchange, or drain every admitted Query/Sync boundary coherently. Current validation accepts multiple boundaries while response scanning stops at the first ReadyForQuery. Test multi-Q/multi-Sync, partial frames, SQL errors and subsequent-query ownership.
• Terminal channel wakeups (19). Make close/failure durable in the state on which readers/writers wait. The current candidate misses a notification injected between terminal-flag inspection and Atomics.wait/waitAsync registration on an unchanged cursor. Two deterministic probes reproduced it; existing unit success does not cover that interleaving. Validate sync/async readers and writers, cancellation, worker exit and exactly-once settlement.
• Pending input admission (24). Bound queued copied work before snapshotting/planning inputs. A serialized promise queue or bounded internal output channel alone does not bound public admission. Preserve ownership, ordering and cancellation without silently dropping accepted work.

The destination and broker acceptance below owns audit items 09 and 55 as well as the original #244 proposal. The full pipeline must stay bounded under a stalled sink; removing a size constant alone does not complete it.

Destination output: files and Rust writers first

• Add writer/file output using each existing tool runner's execution, reservation and cleanup machinery. Its current complete-output API remains a collector adapter. Avoid a universal cross-language runner framework.
• Native Rust may borrow a scoped writer; Rust WASIX needs an owned Send + 'static destination where its tool-thread/VirtualFile ownership requires it. Host JS file destinations must work without awaiting callbacks on a blocked JS thread, including /direct.
• Publish a staged file only after tool execution and database-side cleanup succeed. Specify replacement and permissions behavior; do not call ordinary flush crash-durable. Treat destinations as host paths, not arbitrary guest --file arguments.
• Drain stdout and stderr concurrently. For the new bounded destination API, reuse the existing 8 KiB diagnostic tail, mark truncation, and allow a full diagnostic destination. Preserve existing complete-capture error semantics.
• Preserve plain SQL/COPY dumps, configured login roles, transaction outcome reporting and session ownership. No SQL retries or implicit rollback. psql(script: string) remains buffered input; an unbounded restore source is a separate API decision.

Async JS/browser destinations with real backpressure

• Use standard WritableStream<Uint8Array> with thin Node interoperability. Producer credit is released only after the destination accepts the write, not when a callback merely returns a Promise.
• N-API actor mode needs a bounded channel; /worker must carry acknowledgements across its worker boundary. Sink failure, cancellation and worker termination must wake both ends and settle once.
• /direct supports native file output; reject unsupported async JS destinations before execution. Never silently migrate/reopen a live database.
• Browser /worker must run the tool independently of the database worker and replace both the internal whole-response queue and final output collector on this path. Reuse the existing byte-channel design. Browser root keeps buffered convenience output and gives an early, actionable capability error for unsupported bounded async output.
• JS chunks have owned lifetimes; Rust borrows stay scoped. Reuse current 64 KiB batching and 256 KiB worker queues initially, then measure. These are transport choices, not total-output quotas or proven optima. No automatic temporary-file spooling or new tuning flags.

Native broker/mobile buffering and cross-path validation

• Separate native desktop broker aggregate-result admission from per-message framing. Incrementally validate and forward message bodies, including one huge DataRow, without first allocating the whole response. Buffered convenience APIs still collect. Remove or justify aggregate policy separately; do not simply enlarge the existing ~128 MiB limits.
• Stream desktop backup archives through existing writer/reader machinery rather than a larger management frame. Reuse mobile chunk-transfer patterns only where the contracts match.
• Preserve Swift/Kotlin file-backed backup. Evaluate React Native file output through the same native writer so entire archives need not cross JS. Mobile logical tools and changes to PostgreSQL's own per-value memory use are out of scope.
• Postmaster uses client-tool improvements where applicable; it does not need an embedded execution redesign. Guest recovery/compiler semantics are owned by fix(runtime): correct guest execution, recovery and host isolation #201; fsync/publication contracts by fix(storage): complete persistence, publication and maintenance contracts #249; SDK session/control lifecycle by fix(sdk): complete session lifecycle and mobile ownership contracts #269.

Acceptance

• Exercise all four current collectors with an actual >64 MiB dump and supported restore, not just synthetic buffer tests. Cover tiny output, many rows, one huge value, split UTF-8, concurrent stdout/stderr, tool failure and custom login roles.
• For destinations, test slow/stalled sinks, allocation/write failures, cancellation and worker termination. Verify a usable next query after recoverable failure, or an explicit terminal state when the outcome is unknown. Never report truncated success.
• Cover each affected implementation: native/Rust WASIX; Node/Bun/Deno/Electron root, /direct, /worker; browser root/worker and supported memory/IndexedDB/OPFS storage; memory and directory databases. Run installed consumers on supported desktop targets and affected mobile paths, and standard/ICU carrier checks when packaging changes. Unsupported combinations must fail before execution rather than deadlock.
• Compare first-byte latency, throughput and peak host/Wasm memory as output size grows. A slow sink must bound the whole transport pipeline, not relocate an accumulating queue. PostgreSQL and complete-result APIs still have their own memory costs. Check small-query RTT for regressions; no query-speedup claim is inherent in this work.

PGlite is useful inspiration for a small convenience API, but complete File output and raw callbacks are not evidence of bounded async backpressure. Preserve our existing COPY/restore and configured-role behavior.

Detailed transport triggers and fresh race reproduction remain in archived #203. Lifecycle/device matrix bookkeeping is centralized in #269 and packaged identity in #213; this issue still owns the actual streaming/error/backpressure assertions for each affected path. Close after the transport contracts above are integrated and their semantic/memory checks pass.