Skip to content

feat(location): sync warehouse history after builds, while idle and around a busy DuckDB file - #1449

Merged
chio-labs-delivery[bot] merged 10 commits into
mainfrom
feat/location-sync-bundle
Nov 13, 2026
Merged

chio-labs-delivery[bot] merged 10 commits into
mainfrom
feat/location-sync-bundle

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Why

The console's warehouse history went stale: a person's sqb build did not tell the server, an idle location never synced, a build holding the DuckDB file made syncs fail, and audit result history grew without bound.

Changes

  • Janitor prunes _sqlbuild_audit_results to the newest rows per audit binding on every adapter.
  • After a signed-in, non-worker sqb build, the CLI asks the server to sync the configured location. The location comes from a new [control] section or SQLBUILD_CONTROL_* variables. Bounded to 5 seconds, it prints one line after the build summary and never changes the build's result.
  • The server coalesces sync requests while one is pending for a location target.
  • sqb location serve syncs at least every 30 minutes while idle (--sync-at-least-every, environment variable or [control]; minimum 1 minute, off disables). Any sync restarts the interval.
  • When a build holds the DuckDB warehouse file, the location reports a busy sync, retries after one poll interval, and the console shows the table as waiting, not failing.
  • Docs: the server page covers the settings.

Verification

  • Integration: a signed-in build syncs once and concurrent builds coalesce (test_person_sql.py); a served DuckDB location reports busy while a write lock is held and syncs once it is released (test_warehouse_metadata.py).
  • E2E: the janitor prunes audit results per binding, including a tie at the keep boundary (test_janitor.py).
  • Native: coalescing on SQLite and PostgreSQL; the safety timer's coalescing, idle firing, off and precedence.
  • Live Snowflake: an idle location ran two 1-minute safety syncs with the warehouse suspended; none of the 58 statements used warehouse compute, and the next safety sync picked up a changed state table.
  • make check-ci passed on the final head.

DuckShop: adds the held-lock stage to incremental_warehouse_history.json and the pruning timeline with its boundary tie as direct_state_pruning.json.

…nutes

A [control] server and location in sqlbuild_project.toml, or SQLBUILD_CONTROL_*
variables, let a signed-in sqb build ask that location to sync the target it
built. The request is best-effort, bounded to five seconds and never changes the
exit code. A sync request while one is pending for the target joins it instead
of queueing another. sqb location serve also syncs an idle location at least
every sync_at_least_every (default 30m, 0 or off disables it), as one more
trigger into the same outstanding sync.
A metadata read that finds the DuckDB file locked by another process now
reports a busy sync instead of a failed read. The location backs off one
poll interval without shrinking its page, logs one line when the wait
starts, and the console shows the table as waiting rather than failing.
…ync on its own thread

A failed read after a busy reply no longer stays on the busy path, the
post-build request runs entirely within its bound (keyring, DNS and
server), its errors never change the build's result, and the console
shows a waiting table in the warning tone.
…h-free

Satisfies the repository's structure and test rules: the build sync's
phases live in a helper module, the binding imports it directly,
environment reads match their errors, and the safety and lock tests
use case data instead of branches.
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 13, 2026
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit 4510519 Nov 13, 2026
53 checks passed
@chio-labs-delivery
chio-labs-delivery Bot deleted the feat/location-sync-bundle branch November 13, 2026 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant