You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e03dc83
Browse filesBrowse the repository at this point in the historyBrowse files
perf(control)!: cut per-build control store growth by 40% and prune run logs (#1459)
* wip(recall): write one row per statement when its start and end share a batch
Incomplete: part 1 of the per-run footprint change. Recall schema version 6 adds
recall_events.started_at; the statements views read it. Verified with
cargo test -p sqlbuild-recall only.
* perf(recall): keep promoted lifecycle fields out of stored payloads
Relational stores drop payload fields that already have their own column
from sqlbuild lifecycle events; the events view merges them back so the
payload reads as written. Server-produced events keep their payloads.
* perf(recall): compact worker IDs, content digests and SQL references
Worker IDs namespaced by invocation are stored as a 128-bit digest of the
pair instead of the JSON pair. Content digests keep their first 128 bits;
a full-length digest stored earlier still matches. Each invocation writes
one reference per artifact digest; statements and audits find their SQL
and evidence by digest.
* feat(control): prune finished runs' logs after 14 days or 200 runs and report store size
A finished run's worker log (events, statements, SQL references and output)
goes whole once it is older than --run-log-retention-days or more than
--run-log-retention-count newer runs finished in its location target. The
run, its node results and the scheduler's events stay. Administrators see
both limits and the store's size; heartbeats report the size too.
* docs(control): document run log retention, store size and one-row statements
The Settings page shows the control store's size and the run log limits;
the control client carries the new fields.
* test(recall): check stored and merged lifecycle payloads on both control databases and run log flag limits
* fix(control): name the store and the command that recreates it when it predates this release
A control store created by an earlier release failed with a generic
checksum message. The error now names the SQLite files or the PostgreSQL
database and prints the one command that discards them, so the next
start creates a new store.
BREAKING CHANGE: control stores created by earlier releases are not upgraded, so sqb server and sqb dev start with no run history after upgrading. The server refuses an old store and prints the command that recreates it: for sqb dev, rm -f .sqlbuild/control.sqlite3 .sqlbuild/control.sqlite3-wal .sqlbuild/control.sqlite3-shm in the project; for a PostgreSQL control database, dropdb and createdb on that database.
* refactor(control): keep run log pruning and store size with the maintenance role
Run log pruning and the store size read move into the platform helpers
that use them, retention settings travel as one named struct, and the
new tests follow the parametrized, branch-free test layout.
* fix(control): find finished runs' worker logs through their claim events
A finished run clears its worker instance, so run log retention matched
no run on a real store. Retention now links a run to its worker log
through the scheduler's run_claimed event, and the test claims and
finishes runs through the real lifecycle. PostgreSQL control stores
order invocation_id by bytes, so a worker's prefix range finds its nested
invocations through the index under any database collation.
Real payload fields stay stored, because SQLite and PostgreSQL JSON
functions do not print them back digit for digit. The recreate message
says a different release made the store, names the socket directory for
socket connections and says that dropdb removes the whole database.
The bundled skill joins SQL and audit evidence by digest.
* test(recall): read the folded statement start in the previous ClickHouse statements view
"description": "Four ad-hoc runs of one location target, each with a worker log at its claim and a nested session that shares one SQL text with the others and keeps one of its own. Runs ended 20 days ago, 2 hours ago and 1 hour ago; the fourth is still running. Keeping 14 days and 1 run drops the first two logs whole, keeps the shared SQL, and leaves every run, its nodes and the scheduler's events.",
Copy file name to clipboardExpand all lines: crates/sqlbuild-cli/src/argv/command_table.json
+22Lines changed: 22 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -1124,6 +1124,28 @@
1124
1124
"help": "published snapshots to keep per location and target; older snapshots are deleted unless a run, trigger, job or other retained record still references them (default: 10)",
1125
1125
"group": 1
1126
1126
},
1127
+
{
1128
+
"kind": "store",
1129
+
"options": [
1130
+
"--run-log-retention-days"
1131
+
],
1132
+
"dest": "control_run_log_retention_days",
1133
+
"default": 14,
1134
+
"type": "int",
1135
+
"help": "days a finished run keeps its log of events, statements, SQL and output; the run, its node results and history stay (default: 14)",
1136
+
"group": 1
1137
+
},
1138
+
{
1139
+
"kind": "store",
1140
+
"options": [
1141
+
"--run-log-retention-count"
1142
+
],
1143
+
"dest": "control_run_log_retention_count",
1144
+
"default": 200,
1145
+
"type": "int",
1146
+
"help": "finished runs per location and target that keep their logs; older runs lose theirs, whichever limit comes first (default: 200)",
/// Old evaluation phases go once a later attempt of the same sensor exists.
21
25
constSENSOR_EVALUATIONS:&str = "DELETE FROM sqb_sensor_evaluation_events WHERE event_id IN (SELECT e.event_id \
@@ -37,12 +41,35 @@ const LOCATION_INSTANCES: &str = "DELETE FROM sqb_location_instances WHERE insta
37
41
FROM sqb_location_work w WHERE w.session_id=i.instance_id) \
38
42
ORDER BY i.last_seen_at LIMIT $2) RETURNING instance_id";
39
43
44
+
/// Claims of finished runs past either limit whose worker log is still stored, oldest first. Worker invocations are `worker/<claim>` and `worker/<claim>/<session>`; the stored logs, not every run ever finished, bound the join. A finished run no longer names its worker, so the scheduler's `run_claimed` event, whose ID is the claim, links the two.
45
+
constEXPIRED_CLAIMS:&str = "WITH logged AS (\
46
+
SELECT DISTINCT substr(invocation_id, 8, 36) AS claim FROM recall_invocations \
47
+
WHERE invocation_id > 'worker/' AND invocation_id < 'worker0' \
48
+
AND (length(invocation_id) = 43 OR substr(invocation_id, 44, 1) = '/') \
49
+
UNION SELECT DISTINCT substr(invocation_id, 8, 36) AS claim FROM recall_output \
50
+
WHERE invocation_id > 'worker/' AND invocation_id < 'worker0' \
51
+
AND (length(invocation_id) = 43 OR substr(invocation_id, 44, 1) = '/')), \
52
+
claimed AS (SELECT DISTINCT event_id AS claim, run_id FROM recall_events \
53
+
WHERE event_type = 'run_claimed' AND producer = 'scheduler'), \
54
+
ranked AS (SELECT CAST(run_id AS TEXT) AS run_id, ended_at, ROW_NUMBER() OVER (PARTITION BY location_id, target \
55
+
ORDER BY ended_at DESC, run_id DESC) AS newest FROM sqb_runs \
56
+
WHERE ended_at IS NOT NULL AND status IN ('success', 'warn', 'partial', 'failed', 'cancelled')) \
57
+
SELECT logged.claim FROM ranked JOIN claimed ON claimed.run_id = ranked.run_id \
58
+
JOIN logged ON logged.claim = claimed.claim \
59
+
WHERE ranked.newest > $1 OR ranked.ended_at < $2 ORDER BY ranked.ended_at LIMIT $3";
60
+
40
61
/// Returns how many audit rows outlive their horizon because a sink has not acknowledged them.
/// A finished run's worker log goes whole once it is too old or too many newer runs finished; the run, its nodes and the scheduler's node history stay. Batches commit independently; a repeated or interrupted pass finds only logs still stored.
123
+
pub(crate)asyncfnprune_run_logs(
124
+
store:&ControlStore,
125
+
retention:RunLogRetention,
126
+
) -> Result<usize,ControlError>{
127
+
letmut deleted = 0;
128
+
loop{
129
+
let transaction = begin(store,TransactionMode::Write).await?;
let transaction = delete_logs(transaction,&claims).await?;
146
+
commit(transaction).await?;
147
+
deleted += claims.len();
148
+
if claims.len() < RUN_LOG_RETENTION_BATCH{
149
+
returnOk(deleted);
150
+
}
151
+
}
152
+
}
153
+
154
+
/// Every Recall row of the claims' invocations, then stored artifacts no remaining reference names. `worker/<claim>/<session>` sorts between `worker/<claim>/` and `worker/<claim>0` in byte order, which SQLite uses and PostgreSQL control stores declare for `invocation_id`, so each range reads the index.
0 commit comments