Skip to content

perf(location): read warehouse history through one warm reader per sync - #1431

Merged
chio-labs-delivery[bot] merged 1 commit into
mainfrom
feat/streamed-location-sync
Nov 12, 2026
Merged

chio-labs-delivery[bot] merged 1 commit into
mainfrom
feat/streamed-location-sync

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Why

A code location's warehouse history sync started a new sqb interpreter, and with it a new warehouse login, for every 100-row page of every state table. On a Snowflake location with about 10,000 state rows, the first sync took about 400 s and an unchanged resync about 36 s, almost all of it process start-up and login.

Changes

  • One warm runtime interpreter now serves every page of a synchronization. It is closed when the synchronization completes, when project inputs change, and after a failed delivery, so no warehouse session is held while the location is idle.
  • The interpreter keeps one read-only adapter session for remote warehouses and drops it after any failed read, so the next page reconnects. Local DuckDB and SQLite files reconnect for every page, so their lock never blocks other writers.
  • Each remote state-table scan streams one ordered SELECT, bounded to 50 pages, and serves later pages from the same result. When the window ends, or a request does not continue the open result, the scan resumes from its keyset cursor as before.
  • No DuckShop case: this changes how the sync reads state tables, not a project shape, data value or build lifecycle.

Verification

  • The change-token E2E test now asserts that each resync uses a single interpreter, that a local file is never held open between pages, and that a 250-row change syncs across several pages. All 42 warehouse-history tests and the catalog and statistics tests pass.
  • Snowflake, same machine, 10,061 state rows. PyPI 0.214.0, medians of 3 runs: first sync 399 s, unchanged resync 36 s, changed resync 43 s. This branch, medians of 3 runs: 64 s, 12 s and 12 s. Per first sync, warehouse sessions fall from 114 to 2–3, queries from about 850 to about 110, and bytes scanned from 108 MB to 3.2 MB, with the same 10,223 rows read.
  • make check-ci passes.

A code location's warehouse history sync started a new interpreter, and a
new warehouse login, for every 100-row page of every state table.

One warm runtime interpreter now serves a whole synchronization and is
closed when it completes, when project inputs change, or after a failed
delivery. Remote warehouses keep one read-only session for the sync and
stream each state-table scan through a bounded ordered SELECT, resuming
from the keyset cursor when the window ends. Local DuckDB and SQLite files
reconnect for every page so their lock never blocks other writers.
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 12, 2026
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit f9b8697 Nov 12, 2026
58 of 79 checks passed
@chio-labs-delivery
chio-labs-delivery Bot deleted the feat/streamed-location-sync branch November 12, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant