Skip to content

fix(server): commit SQLite control-store writes durably without a checkpoint on every close - #1481

Merged
kvlonge merged 6 commits into
mainfrom
fix/sqlite-store-durability
Nov 14, 2026
Merged

kvlonge merged 6 commits into
mainfrom
fix/sqlite-store-durability

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Why

The SQLite control store opens a connection per transaction. Closing the last one checkpointed the WAL and synced the database on every commit, so a slow disk stalled writes and scheduler ticks. On a slow CI runner a renewal ran past the lease, the cycle returned Fenced, and the scheduler loop stopped the server. Commits also relied on the default sync level.

Changes

  • Transaction connections no longer checkpoint on close, and writes commit with synchronous=FULL.
  • The store keeps one idle connection open for its lifetime, so each transaction reuses the WAL index instead of re-reading the whole WAL. Without it, 600 non-overlapping commits grew the WAL to 15 MB and took 11 s; with it they take 0.6 s.
  • Platform maintenance truncates the WAL every 30 s, and a commit truncates it best-effort past 64 MiB.
  • A cycle that loses the lease mid-way retries as a follower instead of stopping the server; the log says "Scheduler fenced".
  • Server docs: back up SQLite with .backup, or stop the server first; never copy the database file alone.

Verification

  • New tests: a writer killed after 5 and 200 acknowledged commits loses none; maintenance empties the WAL and keeps rows; non-overlapping commits keep the WAL bounded (fails on the old code); a leader fenced mid-cycle keeps running and leads again. The tick retention test writes its backlog in one transaction.
  • Same machine, three alternating runs each, load 1.7 to 1.9 on 8 cores, medians:
main this PR
Write 4.21 ms 2.98 ms
Read 2.28 ms 1.95 ms
Scheduler cycle 14.5 ms 12.3 ms
Wall / CPU (user+sys) 3.05 s / 2.33 s 2.40 s / 2.06 s
fsync calls (strace) 1,839 742
  • Orchestration tests on SQLite and PostgreSQL and make check-ci pass. The review findings were each re-checked after fixing.

…ckpoint on every close

Each control-store transaction opens its own connection, and closing the last
one checkpointed the WAL and synced the database file on every commit, which
stalled writes and scheduler ticks on slow disks. Connections now keep the WAL
when they close, writes commit with synchronous=FULL, platform maintenance
truncates the WAL every 30 seconds so constant readers cannot grow it without
bound, and a commit that leaves a WAL over 64 MiB resets it at once. The docs
say to back SQLite up with .backup or with the -wal file, never the database
file alone.
A renewal that found the lease expired between trigger evaluations or in the
launcher's poll ended the scheduler loop, which stopped the whole server. The
loop now retries, and the next cycle rejoins the election. The tick retention
test writes its backlog in one transaction, and its background lease holder
reports why it stopped renewing.
WIP: review fixes. Hold the store's setup connection for its lifetime so
transactions that never overlap don't rebuild the WAL index and restart
checkpointing; stop the fenced-loop test once it rejoins; neutral wording
for a fenced cycle; bound the crash test from below only.
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 14, 2026
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 14, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 14, 2026
@kvlonge
kvlonge added this pull request to the merge queue Nov 14, 2026
Merged via the queue into main with commit 4c5d5f0 Nov 14, 2026
50 of 75 checks passed
@kvlonge
kvlonge deleted the fix/sqlite-store-durability branch November 14, 2026 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant