Skip to content

feat!: bring the large-project server work onto main - #1566

Merged
chio-labs-delivery[bot] merged 11 commits into
mainfrom
feat/large-project-integration
Nov 18, 2026
Merged

chio-labs-delivery[bot] merged 11 commits into
mainfrom
feat/large-project-integration

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Why

The large-project server work (location runtime, history sync, plan previews over synced state, pools, group waits, node state cache, UI) lived on a long branch while main split the Python bindings into crates. This brings it onto main as one change.

Changes

  • Sync read time-outs halve the page; other failures back off 30–240 s and the fifth halts the sync. Health and the UI show the cause.
  • Plan previews read the held warehouse state; "Build what changed" is planned after each publication and completed sync.
  • In-memory node list and node states; a resync pushes only real differences. Group waits use the nearest member.
  • Breaking: ignored audit failures in real store columns; concurrency_groups/concurrency_limits renamed to pools/pool_caps.
  • Control-store migration 0002 upgrades version 1 stores in place.
  • Billed warehouses keep deferring safety syncs: the publication's connection check (sqb debug --json) reports whether history reads wake compute, not each page.
  • Wide-listing guard: 12 fixed statements (approved). Python bindings crate cap 51,200 (approved).
  • UI CI: layout project required; perf timings in a non-required job.
  • New dupes allowlist paths: changed_builds.rs, person_selection.rs (utoipa routes); ingest_resource_costs.rs (typed events).
  • Goldens: both unknown-key diagnostics now list the new ignore_audit_failures and pools keys.
  • No DuckShop case: server, store and runtime behaviour, not project shapes.

Verification

  • Rust: clippy; orchestration on SQLite and PostgreSQL; CLI, state, core; billed-deferral test fails without the fix.
  • Store: stores created by the 0.243.0 wheel upgrade on SQLite and PostgreSQL to a fresh store's schema, rows and references intact.
  • Python: freshness, publication, migrations, audit, scenario and debug tests; adapter unit tests; compiler differential.
  • UI: svelte-check; perf project.
  • make check-ci on the final head.

Integrates the large-project branch: the code location runtime, warehouse
history sync, plan previews over the synced warehouse state, job pools,
ignored audit failures, job-group waits, the in-memory node list and node
state cache, and the UI for them.

- Locations sync warehouse history in bounded pages (2,000 rows, 100
  statistics relations) and report each page's health. A time-out or
  memory failure halves the page; other read failures back off 30, 60,
  120 and 240 s, and the fifth in a row halts the sync until a run, a
  request, a publication or a due safety sync. The health report carries
  the closed cause, the redacted error and the halted state, and the UI
  shows it.
- Plan previews read the held warehouse state instead of an emulating
  synced-copy adapter. After each publication and completed sync a
  location plans "Build what changed" and reports it; a failed plan is
  reported with its own error.
- The server keeps each target's node list and node states in memory and
  pushes the changed states on the change stream. A listener resync
  pushes only real differences, since clients refetch on `all`.
- Job-group waits use the nearest-member rule inside the group wait
  graph.
- Ignored audit failures are stored in real columns
  (`sqb_runs.ignored_audit_failures`,
  `sqb_warehouse_audit_results.ignored`); the Recall `audit_results` view
  gains `ignored` (Recall schema version 7).
- A SQLite control store gives freed pages back with incremental vacuum
  until it finishes.

Job and job group files declare `pools` instead of `concurrency_groups`,
and server.toml launchers lower them with `pool_caps` instead of
`concurrency_limits`; the old names are now unknown keys.

The wide node-listing guard allows 12 fixed statements instead of 9: a
cold node-list cache authorizes in its own transaction, which adds a
clock read and a second current-snapshot read, and reads the target's
changed build. Builds are still read 100 nodes per statement.

Code this change duplicated is shared instead: one adapter read and one
state-history read on WarehouseReads, a listing-read helper, shared
thousands grouping and row-count estimates. The Python bindings crate cap
rises to 51,200 code lines.

BREAKING CHANGE: the control store schema changes: runs record ignored
audit failures in `sqb_runs.ignored_audit_failures` instead of a reserved
node-count key, and stored check results record them in
`sqb_warehouse_audit_results.ignored` instead of the result metadata. Job
and job group files rename `concurrency_groups` to `pools`, and
server.toml launchers rename `concurrency_limits` to `pool_caps`.
The location execution helpers grew past the 2,000-line file limit once
main's store migration command joined them; the command has nothing to
do with locations.
The scenario pipeline now passes the native plan builder a dialect
instead of the adapter; the test double keeps the same keywords.
Control-store migration 0002 brings a version 1 store to this branch's
schema: skipped runs, an ignored audit failure count on runs and an
ignored flag on audit results, connection snapshot documents,
concurrency slots without the one-slot-per-run limit, changed-build
reports, plan diffs, the received-time history indexes and the ignored
column of the audit_results view.

SQLite rebuilds sqb_runs, sqb_snapshot_documents and
sqb_concurrency_slots and recreates their indexes; PostgreSQL alters
them in place. No earlier release recorded ignored audit failures, so
existing rows keep the new columns' defaults.

A version 1 store holding a running child run, a concurrency slot, a
snapshot document and an audit result upgrades with no broken
references, and the baseline upgrade compares only the columns each row
already had.
An idle location skips the interval safety sync when its adapter's
history reads wake billed compute, but the flag was copied from every
history page. The statistics page a location builds locally said
false, and statistics is the last kind read, so a finished sync stopped
deferring; a sync that stopped before any page succeeded never set it.
Either way the safety sync still woke Databricks, MotherDuck or
ClickHouse.

sqb debug --json now reports history_reads_wake_compute, the connection
check after each publication sets it on the sync that publication
starts, and history pages no longer carry it.
The selection and choice-filter checks read the page URL once, right
after the message or chip they waited for appeared, so a run could read
it before the app wrote the new query. They now retry until it matches.
A new control store is created at the current schema version, which
migration 0002 made 2.
@kvlonge
kvlonge marked this pull request as ready for review November 18, 2026 05:14
@kvlonge
kvlonge enabled auto-merge November 18, 2026 05:17
@kvlonge
kvlonge added this pull request to the merge queue Nov 18, 2026
…key goldens

ignore_audit_failures is a new settings and target key, and pools a new
target key, so the allowed-keys list in the two unknown-key diagnostics
grows by them.
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 18, 2026
The change-stream test issued a token that expired one second later and
then had to open a stream with it. On a loaded Windows runner issuing and
connecting took longer than that, so the stream was refused with 401
before the expiry could be observed. Give the token five seconds; the
test still waits it out before checking that the stream stops.
@kvlonge
kvlonge enabled auto-merge November 18, 2026 06:22
@kvlonge
kvlonge added this pull request to the merge queue Nov 18, 2026
…ecting

Each replica announces node states after its own debounce, and only for
targets a stream holds. The change-stream test reconnected to the second
replica as soon as the first replica's observer saw the change, so the
second replica's announcement for that change could reach the new stream
and look like a replay. Watch the target on the second replica too and
wait for its announcement before reconnecting.
@kvlonge
kvlonge removed this pull request from the merge queue due to a manual request Nov 18, 2026
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 3875c56 Nov 18, 2026
35 checks passed
@chio-labs-delivery
chio-labs-delivery Bot deleted the feat/large-project-integration branch November 18, 2026 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant