Skip to content

perf(location): read sync pages back to back and check git HEAD at most every 2 s - #1454

Merged
chio-labs-delivery[bot] merged 2 commits into
mainfrom
perf/sync-pacing
Nov 13, 2026
Merged

chio-labs-delivery[bot] merged 2 commits into
mainfrom
perf/sync-pacing

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Why

A served location sent one 100-row warehouse-metadata page per 250 ms watch tick, so a sync of 10,000 fingerprints spent most of its time waiting for the next tick. Each idle tick also spawned git rev-parse three times (once per fence), which is wasted work on an idle server.

Changes

  • While a metadata sync has a page due and the project inputs are settled, the location loop runs the next page straight away instead of waiting for the tick. Idle work polling is unchanged.
  • Only the watch tick asks git for HEAD, at most every 2 s. The work and metadata fences compare project file stamps only.
  • The SQLite consumption-history test helper now inserts recorded_at as ISO text, the way the store writer does, and the consumption lifecycle test covers SQLite as well as DuckDB.
  • The DuckDB held-lock case takes its lock between syncs, so its busy report no longer depends on how far a paced sync had read.

DuckShop: no new edge case. The change is to how often a served location polls and pages, not to project shapes, data values or a build lifecycle sequence.

Verification

  • New E2E test: a served SQLite location with 4,000 consumption rows finishes its sync within 9.5 s (one page per tick would need at least 10 s) and makes at most 4 git calls in 4 s idle. Head: 3.7 s and 2 git calls. Base: fails at 13.0 s with 17 git calls.
  • Snowflake, about 10,000 fingerprints, same machine, base and head alternated, medians of 3: first sync 84.7 s → 46.2 s; reconcile with 100 new rows 36.2 s → 15.4 s.
  • Reconcile with nothing new (one page, so pacing does not apply), 5 more alternating pairs with base built from main, two reconciles each: median 8.35 s base, 8.2 s head. Both issue the same 27 catalog queries. The earlier 13.2 s head median was account noise.
  • Consumption lifecycle (SQLite and DuckDB) and change-token tests pass; fensu check is clean; make check-ci passes.

…most every 2 s

A synchronization sent one 100-row page per 250 ms watch tick, so a 10,000-row
reconcile spent most of its time waiting for the next tick. While a sync has a
page due, the loop now skips the tick; idle work polling is unchanged.

Only the watch tick asks git for HEAD, at most every 2 s; the work and metadata
fences compare project files only. The SQLite consumption lifecycle case now
inserts recorded_at as text, the way the writer does.
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 13, 2026
The DuckDB overlap case took its write lock while the restored sync was still
reading, so the busy report landed on whichever table that sync had reached.
With pages read back to back that is no longer source_freshness. The case now
waits for the sync to finish and expects the next sync's first table to report
the lock.
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit 8864c22 Nov 13, 2026
53 checks passed
@chio-labs-delivery
chio-labs-delivery Bot deleted the perf/sync-pacing branch November 13, 2026 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant