Repository navigation
feat!: bring the large-project server work onto main - #1566
Merged
Merged
Conversation
Integrates the large-project branch: the code location runtime, warehouse history sync, plan previews over the synced warehouse state, job pools, ignored audit failures, job-group waits, the in-memory node list and node state cache, and the UI for them. - Locations sync warehouse history in bounded pages (2,000 rows, 100 statistics relations) and report each page's health. A time-out or memory failure halves the page; other read failures back off 30, 60, 120 and 240 s, and the fifth in a row halts the sync until a run, a request, a publication or a due safety sync. The health report carries the closed cause, the redacted error and the halted state, and the UI shows it. - Plan previews read the held warehouse state instead of an emulating synced-copy adapter. After each publication and completed sync a location plans "Build what changed" and reports it; a failed plan is reported with its own error. - The server keeps each target's node list and node states in memory and pushes the changed states on the change stream. A listener resync pushes only real differences, since clients refetch on `all`. - Job-group waits use the nearest-member rule inside the group wait graph. - Ignored audit failures are stored in real columns (`sqb_runs.ignored_audit_failures`, `sqb_warehouse_audit_results.ignored`); the Recall `audit_results` view gains `ignored` (Recall schema version 7). - A SQLite control store gives freed pages back with incremental vacuum until it finishes. Job and job group files declare `pools` instead of `concurrency_groups`, and server.toml launchers lower them with `pool_caps` instead of `concurrency_limits`; the old names are now unknown keys. The wide node-listing guard allows 12 fixed statements instead of 9: a cold node-list cache authorizes in its own transaction, which adds a clock read and a second current-snapshot read, and reads the target's changed build. Builds are still read 100 nodes per statement. Code this change duplicated is shared instead: one adapter read and one state-history read on WarehouseReads, a listing-read helper, shared thousands grouping and row-count estimates. The Python bindings crate cap rises to 51,200 code lines. BREAKING CHANGE: the control store schema changes: runs record ignored audit failures in `sqb_runs.ignored_audit_failures` instead of a reserved node-count key, and stored check results record them in `sqb_warehouse_audit_results.ignored` instead of the result metadata. Job and job group files rename `concurrency_groups` to `pools`, and server.toml launchers rename `concurrency_limits` to `pool_caps`.
The location execution helpers grew past the 2,000-line file limit once main's store migration command joined them; the command has nothing to do with locations.
The scenario pipeline now passes the native plan builder a dialect instead of the adapter; the test double keeps the same keywords.
Control-store migration 0002 brings a version 1 store to this branch's schema: skipped runs, an ignored audit failure count on runs and an ignored flag on audit results, connection snapshot documents, concurrency slots without the one-slot-per-run limit, changed-build reports, plan diffs, the received-time history indexes and the ignored column of the audit_results view. SQLite rebuilds sqb_runs, sqb_snapshot_documents and sqb_concurrency_slots and recreates their indexes; PostgreSQL alters them in place. No earlier release recorded ignored audit failures, so existing rows keep the new columns' defaults. A version 1 store holding a running child run, a concurrency slot, a snapshot document and an audit result upgrades with no broken references, and the baseline upgrade compares only the columns each row already had.
An idle location skips the interval safety sync when its adapter's history reads wake billed compute, but the flag was copied from every history page. The statistics page a location builds locally said false, and statistics is the last kind read, so a finished sync stopped deferring; a sync that stopped before any page succeeded never set it. Either way the safety sync still woke Databricks, MotherDuck or ClickHouse. sqb debug --json now reports history_reads_wake_compute, the connection check after each publication sets it on the sync that publication starts, and history pages no longer carry it.
The selection and choice-filter checks read the page URL once, right after the message or chip they waited for appeared, so a run could read it before the app wrote the new query. They now retry until it matches.
A new control store is created at the current schema version, which migration 0002 made 2.
kvlonge
marked this pull request as ready for review
November 18, 2026 05:14
kvlonge
enabled auto-merge
November 18, 2026 05:17
…key goldens ignore_audit_failures is a new settings and target key, and pools a new target key, so the allowed-keys list in the two unknown-key diagnostics grows by them.
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 18, 2026
The change-stream test issued a token that expired one second later and then had to open a stream with it. On a loaded Windows runner issuing and connecting took longer than that, so the stream was refused with 401 before the expiry could be observed. Give the token five seconds; the test still waits it out before checking that the stream stops.
kvlonge
enabled auto-merge
November 18, 2026 06:22
…ecting Each replica announces node states after its own debounce, and only for targets a stream holds. The change-stream test reconnected to the second replica as soon as the first replica's observer saw the change, so the second replica's announcement for that change could reach the new stream and look like a replay. Watch the target on the second replica too and wait for its announcement before reconnecting.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The large-project server work (location runtime, history sync, plan previews over synced state, pools, group waits, node state cache, UI) lived on a long branch while main split the Python bindings into crates. This brings it onto main as one change.
Changes
concurrency_groups/concurrency_limitsrenamed topools/pool_caps.sqb debug --json) reports whether history reads wake compute, not each page.changed_builds.rs,person_selection.rs(utoipa routes);ingest_resource_costs.rs(typed events).ignore_audit_failuresandpoolskeys.Verification
make check-cion the final head.