Repository navigation
perf(control)!: cut per-build control store growth by 40% and prune run logs - #1459
Merged
Merged
Conversation
… a batch Incomplete: part 1 of the per-run footprint change. Recall schema version 6 adds recall_events.started_at; the statements views read it. Verified with cargo test -p sqlbuild-recall only.
Relational stores drop payload fields that already have their own column from sqlbuild lifecycle events; the events view merges them back so the payload reads as written. Server-produced events keep their payloads.
Worker IDs namespaced by invocation are stored as a 128-bit digest of the pair instead of the JSON pair. Content digests keep their first 128 bits; a full-length digest stored earlier still matches. Each invocation writes one reference per artifact digest; statements and audits find their SQL and evidence by digest.
…d report store size A finished run's worker log (events, statements, SQL references and output) goes whole once it is older than --run-log-retention-days or more than --run-log-retention-count newer runs finished in its location target. The run, its node results and the scheduler's events stay. Administrators see both limits and the store's size; heartbeats report the size too.
…tements The Settings page shows the control store's size and the run log limits; the control client carries the new fields.
…rol databases and run log flag limits
…t predates this release A control store created by an earlier release failed with a generic checksum message. The error now names the SQLite files or the PostgreSQL database and prints the one command that discards them, so the next start creates a new store. BREAKING CHANGE: control stores created by earlier releases are not upgraded, so sqb server and sqb dev start with no run history after upgrading. The server refuses an old store and prints the command that recreates it: for sqb dev, rm -f .sqlbuild/control.sqlite3 .sqlbuild/control.sqlite3-wal .sqlbuild/control.sqlite3-shm in the project; for a PostgreSQL control database, dropdb and createdb on that database.
…enance role Run log pruning and the store size read move into the platform helpers that use them, retention settings travel as one named struct, and the new tests follow the parametrized, branch-free test layout.
A finished run clears its worker instance, so run log retention matched no run on a real store. Retention now links a run to its worker log through the scheduler's run_claimed event, and the test claims and finishes runs through the real lifecycle. PostgreSQL control stores order invocation_id by bytes, so a worker's prefix range finds its nested invocations through the index under any database collation. Real payload fields stay stored, because SQLite and PostgreSQL JSON functions do not print them back digit for digit. The recreate message says a different release made the store, names the socket directory for socket connections and says that dropdb removes the whole database. The bundled skill joins SQL and audit evidence by digest.
# Conflicts: # crates/sqlbuild-orchestration/tests/runs/test_contracts.rs # crates/sqlbuild-orchestration/tests/runs/test_types.rs # src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/server.md # website/src/content/docs/docs/cli/server.mdx
…use statements view
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A control store grew about 35 MB per DuckShop build and nothing pruned run logs.
Upgrading discards run history. Control stores created by earlier releases are not upgraded:
sqb serverandsqb devrefuse them and print the command that starts a new, empty store, for examplerm -f .sqlbuild/control.sqlite3*forsqb dev, ordropdb … && createdb …for PostgreSQL.Changes
started_atsits on that row. Recall schema 6.eventsview merges them back.--run-log-retention-days,--run-log-retention-count). Runs, nodes and scheduler history stay. PostgreSQL stores orderinvocation_idby bytes.store_bytesin the admin server read, the heartbeat event and Settings.Verification
sqb dev, same harness: 34.61 → 20.65 MB per steady build. Remaining growth per build:~1,240
audit_completedevents per build carry most payload bytes: audit results are stored twice, as events and in the audit mirror.edge_cases/orchestration/run_log_retention.jsondrives the retention test.