Skip to content

Commit e03dc83

Browse files
authored
perf(control)!: cut per-build control store growth by 40% and prune run logs (#1459)
* wip(recall): write one row per statement when its start and end share a batch Incomplete: part 1 of the per-run footprint change. Recall schema version 6 adds recall_events.started_at; the statements views read it. Verified with cargo test -p sqlbuild-recall only. * perf(recall): keep promoted lifecycle fields out of stored payloads Relational stores drop payload fields that already have their own column from sqlbuild lifecycle events; the events view merges them back so the payload reads as written. Server-produced events keep their payloads. * perf(recall): compact worker IDs, content digests and SQL references Worker IDs namespaced by invocation are stored as a 128-bit digest of the pair instead of the JSON pair. Content digests keep their first 128 bits; a full-length digest stored earlier still matches. Each invocation writes one reference per artifact digest; statements and audits find their SQL and evidence by digest. * feat(control): prune finished runs' logs after 14 days or 200 runs and report store size A finished run's worker log (events, statements, SQL references and output) goes whole once it is older than --run-log-retention-days or more than --run-log-retention-count newer runs finished in its location target. The run, its node results and the scheduler's events stay. Administrators see both limits and the store's size; heartbeats report the size too. * docs(control): document run log retention, store size and one-row statements The Settings page shows the control store's size and the run log limits; the control client carries the new fields. * test(recall): check stored and merged lifecycle payloads on both control databases and run log flag limits * fix(control): name the store and the command that recreates it when it predates this release A control store created by an earlier release failed with a generic checksum message. The error now names the SQLite files or the PostgreSQL database and prints the one command that discards them, so the next start creates a new store. BREAKING CHANGE: control stores created by earlier releases are not upgraded, so sqb server and sqb dev start with no run history after upgrading. The server refuses an old store and prints the command that recreates it: for sqb dev, rm -f .sqlbuild/control.sqlite3 .sqlbuild/control.sqlite3-wal .sqlbuild/control.sqlite3-shm in the project; for a PostgreSQL control database, dropdb and createdb on that database. * refactor(control): keep run log pruning and store size with the maintenance role Run log pruning and the store size read move into the platform helpers that use them, retention settings travel as one named struct, and the new tests follow the parametrized, branch-free test layout. * fix(control): find finished runs' worker logs through their claim events A finished run clears its worker instance, so run log retention matched no run on a real store. Retention now links a run to its worker log through the scheduler's run_claimed event, and the test claims and finishes runs through the real lifecycle. PostgreSQL control stores order invocation_id by bytes, so a worker's prefix range finds its nested invocations through the index under any database collation. Real payload fields stay stored, because SQLite and PostgreSQL JSON functions do not print them back digit for digit. The recreate message says a different release made the store, names the socket directory for socket connections and says that dropdb removes the whole database. The bundled skill joins SQL and audit evidence by digest. * test(recall): read the folded statement start in the previous ClickHouse statements view
1 parent fd2dcc6 commit e03dc83

67 files changed

Lines changed: 1220 additions & 148 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"description": "Four ad-hoc runs of one location target, each with a worker log at its claim and a nested session that shares one SQL text with the others and keeps one of its own. Runs ended 20 days ago, 2 hours ago and 1 hour ago; the fourth is still running. Keeping 14 days and 1 run drops the first two logs whole, keeps the shared SQL, and leaves every run, its nodes and the scheduler's events.",
3+
"keep_days": 14,
4+
"keep_runs": 1,
5+
"ended_hours_ago": [480, 2, 1, null],
6+
"expected_pruned": 2,
7+
"expected_logged_runs": [3, 4]
8+
}

‎clients/typescript/src/schema.d.ts‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1726,6 +1726,11 @@ export interface components {
17261726
/** @description Digest of the schema; a server starts only on the exact schema it creates. */
17271727
schema_checksum: string;
17281728
storage: components["schemas"]["StorageKind"];
1729+
/**
1730+
* Format: int64
1731+
* @description The database's size on disk now: SQLite's main file, or the whole PostgreSQL database.
1732+
*/
1733+
store_bytes: number;
17291734
};
17301735
/** @description One job or group as declared, for a person who may list the target; free-form setting values are withheld. */
17311736
JobDetailRecord: {
@@ -2330,6 +2335,11 @@ export interface components {
23302335
kind: "heartbeat";
23312336
/** Format: uuid */
23322337
replica_id: string;
2338+
/**
2339+
* Format: int64
2340+
* @description The control store's database size on disk when the replica checked in.
2341+
*/
2342+
store_bytes: number;
23332343
} | {
23342344
/** @enum {string} */
23352345
kind: "scheduler_no_leader";
@@ -3041,6 +3051,16 @@ export interface components {
30413051
location_health_days: number;
30423052
/** Format: int64 */
30433053
platform_alert_days: number;
3054+
/**
3055+
* Format: int64
3056+
* @description Days a finished run keeps its log of events, statements and output.
3057+
*/
3058+
run_log_days: number;
3059+
/**
3060+
* Format: int64
3061+
* @description Finished runs per location target that keep their logs.
3062+
*/
3063+
run_logs_kept: number;
30443064
/** Format: int64 */
30453065
sensor_evaluation_days: number;
30463066
/**

‎crates/sqlbuild-cli/src/argv/command_table.json‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1124,6 +1124,28 @@
11241124
"help": "published snapshots to keep per location and target; older snapshots are deleted unless a run, trigger, job or other retained record still references them (default: 10)",
11251125
"group": 1
11261126
},
1127+
{
1128+
"kind": "store",
1129+
"options": [
1130+
"--run-log-retention-days"
1131+
],
1132+
"dest": "control_run_log_retention_days",
1133+
"default": 14,
1134+
"type": "int",
1135+
"help": "days a finished run keeps its log of events, statements, SQL and output; the run, its node results and history stay (default: 14)",
1136+
"group": 1
1137+
},
1138+
{
1139+
"kind": "store",
1140+
"options": [
1141+
"--run-log-retention-count"
1142+
],
1143+
"dest": "control_run_log_retention_count",
1144+
"default": 200,
1145+
"type": "int",
1146+
"help": "finished runs per location and target that keep their logs; older runs lose theirs, whichever limit comes first (default: 200)",
1147+
"group": 1
1148+
},
11271149
{
11281150
"kind": "store",
11291151
"options": ["--trusted-proxy-config"],

‎crates/sqlbuild-cli/src/native_commands/_helpers/decision.rs‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ use sqlbuild_orchestration::api::models::{ApiAuthentication, HumanLogin, ServerF
99
use sqlbuild_orchestration::credentials::models::{AuditRetention, CredentialScope};
1010
use sqlbuild_orchestration::declarations::models::TriggerChange;
1111
use sqlbuild_orchestration::platform::models::{LocationWork, LocationWorkRequest};
12+
use sqlbuild_orchestration::runs::models::RunLogRetention;
1213
use sqlbuild_orchestration::runs::models::{JobRunRequest, RerunRequest, RunWindow};
1314
use sqlbuild_orchestration::snapshots::models::SnapshotRetention;
1415
use sqlbuild_orchestration::store::errors::ControlError;
@@ -280,6 +281,11 @@ pub(crate) fn control_api_settings(
280281
Some(count) => SnapshotRetention::count(count)?,
281282
None => SnapshotRetention::default(),
282283
};
284+
let defaults = RunLogRetention::default();
285+
let run_logs = RunLogRetention::new(
286+
retention_setting(namespace, "run-log-retention-days")?.unwrap_or(defaults.days()),
287+
retention_setting(namespace, "run-log-retention-count")?.unwrap_or(defaults.count()),
288+
)?;
283289
Ok((
284290
control_authentication(namespace)?,
285291
ServerFacts {
@@ -288,6 +294,7 @@ pub(crate) fn control_api_settings(
288294
audit,
289295
statistics,
290296
snapshots,
297+
run_logs,
291298
},
292299
))
293300
}

‎crates/sqlbuild-cli/src/native_commands/_helpers/execution.rs‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -21,29 +21,26 @@ use sqlbuild_config::project::models::ControlConfig;
2121
use sqlbuild_orchestration::api::models::{
2222
ApiAuthentication, DevelopmentScope, HumanLogin, ServerFacts,
2323
};
24-
use sqlbuild_orchestration::credentials::models::AuditRetention;
2524
use sqlbuild_orchestration::platform::constants::PLATFORM_WEBHOOK_VARIABLE;
2625
use sqlbuild_orchestration::platform::main::report_remote_location_health::report_remote_location_health;
2726
use sqlbuild_orchestration::platform::main::run_maintenance::run_maintenance;
2827
use sqlbuild_orchestration::platform::models::{
29-
LocationHealth, LocationHealthReport, LocationWorkClient, LocationWorkPoll,
28+
LocationHealth, LocationHealthReport, LocationWorkClient, LocationWorkPoll, RetentionPolicies,
3029
WarehouseSyncFailure,
3130
};
3231
use sqlbuild_orchestration::runs::types::RunHandoff;
3332
use sqlbuild_orchestration::scheduler::main::run_scheduler::run_scheduler;
3433
use sqlbuild_orchestration::scheduler::models::SchedulerState;
3534
use sqlbuild_orchestration::snapshots::main::next_remote_generation::next_remote_generation;
36-
use sqlbuild_orchestration::snapshots::models::SnapshotRetention;
3735
use sqlbuild_orchestration::store::errors::ControlError;
3836
use sqlbuild_orchestration::store::main::migrate_store::migrate_store;
3937
use sqlbuild_orchestration::store::main::open_store::open_store;
4038
use sqlbuild_orchestration::store::main::set_release_version::set_release_version;
4139
use sqlbuild_orchestration::store::models::ControlStore;
4240
use sqlbuild_orchestration::warehouse_metadata::constants::MAX_HISTORY_RECORDS;
4341
use sqlbuild_orchestration::warehouse_metadata::models::{
44-
MetadataBatch, MetadataScan, StatisticsDueRead, StatisticsRead, StatisticsRetention,
45-
WarehouseHistoryKind, WarehouseHistoryPage, WarehouseStateLocation, WarehouseSyncDueRead,
46-
WarehouseWatermarkRead,
42+
MetadataBatch, MetadataScan, StatisticsDueRead, StatisticsRead, WarehouseHistoryKind,
43+
WarehouseHistoryPage, WarehouseStateLocation, WarehouseSyncDueRead, WarehouseWatermarkRead,
4744
};
4845
use sqlbuild_orchestration::webhooks::constants::SECURITY_AUDIT_WEBHOOK_VARIABLE;
4946
use sqlbuild_orchestration::webhooks::main::{
@@ -891,7 +888,7 @@ pub(crate) fn location_child(options: &LocationServe, script: &Path) -> Command
891888
pub(crate) fn start_platform_roles(
892889
tasks: &mut JoinSet<Result<(), ControlError>>,
893890
(store, shutdown): (&ControlStore, &watch::Receiver<bool>),
894-
retention: (AuditRetention, StatisticsRetention, SnapshotRetention),
891+
retention: RetentionPolicies,
895892
) -> Result<(), ControlError> {
896893
let mut sinks: Vec<(Webhook, &str)> = Vec::new();
897894
for (variable, source, label) in [

‎crates/sqlbuild-cli/src/native_commands/main/_run_control.rs‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
//! Supervise native API/scheduler roles together; a failed role stops its siblings.
22
33
use sqlbuild_orchestration::api::main::serve_api::serve_api;
4+
use sqlbuild_orchestration::platform::models::RetentionPolicies;
45
use sqlbuild_orchestration::scheduler::main::run_scheduler::run_scheduler;
56
use sqlbuild_orchestration::scheduler::models::SchedulerState;
67
use sqlbuild_orchestration::store::errors::ControlError;
@@ -56,7 +57,12 @@ pub(crate) async fn run_control(
5657
start_platform_roles(
5758
&mut tasks,
5859
(&store, &shutdown),
59-
(facts.audit, facts.statistics, facts.snapshots),
60+
RetentionPolicies {
61+
audit: facts.audit,
62+
statistics: facts.statistics,
63+
snapshots: facts.snapshots,
64+
run_logs: facts.run_logs,
65+
},
6066
)?;
6167
}
6268
if facts.scheduler {

‎crates/sqlbuild-orchestration/src/api/_helpers/resources/administrators.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,8 @@ pub(crate) async fn server(
113113
audit_days: i64::from(facts.audit.kept_days()),
114114
statistics_days: i64::from(facts.statistics.kept_days()),
115115
snapshots_kept: i64::from(facts.snapshots.kept_count()),
116+
run_log_days: i64::from(facts.run_logs.days()),
117+
run_logs_kept: i64::from(facts.run_logs.count()),
116118
platform_alert_days: PLATFORM_ALERT_RETENTION_DAYS,
117119
heartbeat_days: HEARTBEAT_RETENTION_DAYS,
118120
location_health_days: LOCATION_RETENTION_DAYS,

‎crates/sqlbuild-orchestration/src/api/models.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -521,6 +521,7 @@ pub struct ServerFacts {
521521
pub audit: crate::credentials::models::AuditRetention,
522522
pub statistics: crate::warehouse_metadata::models::StatisticsRetention,
523523
pub snapshots: crate::snapshots::models::SnapshotRetention,
524+
pub run_logs: crate::runs::models::RunLogRetention,
524525
}
525526

526527
/// The server's version, storage, sign-in and retention, for administrators.
@@ -541,6 +542,10 @@ pub struct ServerRetention {
541542
pub statistics_days: i64,
542543
/// Publications kept per location target.
543544
pub snapshots_kept: i64,
545+
/// Days a finished run keeps its log of events, statements and output.
546+
pub run_log_days: i64,
547+
/// Finished runs per location target that keep their logs.
548+
pub run_logs_kept: i64,
544549
pub platform_alert_days: i64,
545550
pub heartbeat_days: i64,
546551
pub location_health_days: i64,

‎crates/sqlbuild-orchestration/src/platform/_helpers/retention.rs‎

Lines changed: 127 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,25 @@
11
//! Every retention policy runs in one bounded pass; each part is safe to repeat.
22
3+
use std::collections::BTreeSet;
4+
35
use chrono::{DateTime, TimeDelta, Utc};
6+
use sqlbuild_recall::schema::constants::INVOCATION_TABLES;
7+
use sqlbuild_recall::schema::main::table_spec::table_spec;
48

59
use crate::credentials::main::audit::prune_audit::prune_audit;
6-
use crate::credentials::models::AuditRetention;
710
use crate::platform::constants::{
811
HEARTBEAT_PAGE, HEARTBEAT_RETENTION_DAYS, LOCATION_RETENTION_DAYS,
912
PLATFORM_ALERT_RETENTION_DAYS, SENSOR_EVALUATION_RETENTION_DAYS,
1013
};
14+
use crate::platform::models::RetentionPolicies;
15+
use crate::runs::constants::{RUN_LOG_DIGEST_CHUNK, RUN_LOG_RETENTION_BATCH};
16+
use crate::runs::models::RunLogRetention;
1117
use crate::snapshots::main::prune_snapshots::prune_snapshots;
12-
use crate::snapshots::models::SnapshotRetention;
1318
use crate::store::errors::ControlError;
14-
use crate::store::main::{query::query, write::write};
15-
use crate::store::models::{ControlStore, Parameter};
19+
use crate::store::main::{begin::begin, commit::commit, query::query, write::write};
20+
use crate::store::models::{ControlStore, ControlTransaction, Parameter, TransactionMode};
1621
use crate::warehouse_metadata::constants::MAX_HISTORY_RECORDS;
1722
use crate::warehouse_metadata::main::ingest::prune_statistics::prune_statistics;
18-
use crate::warehouse_metadata::models::StatisticsRetention;
1923

2024
/// Old evaluation phases go once a later attempt of the same sensor exists.
2125
const SENSOR_EVALUATIONS: &str = "DELETE FROM sqb_sensor_evaluation_events WHERE event_id IN (SELECT e.event_id \
@@ -37,12 +41,35 @@ const LOCATION_INSTANCES: &str = "DELETE FROM sqb_location_instances WHERE insta
3741
FROM sqb_location_work w WHERE w.session_id=i.instance_id) \
3842
ORDER BY i.last_seen_at LIMIT $2) RETURNING instance_id";
3943

44+
/// Claims of finished runs past either limit whose worker log is still stored, oldest first. Worker invocations are `worker/<claim>` and `worker/<claim>/<session>`; the stored logs, not every run ever finished, bound the join. A finished run no longer names its worker, so the scheduler's `run_claimed` event, whose ID is the claim, links the two.
45+
const EXPIRED_CLAIMS: &str = "WITH logged AS (\
46+
SELECT DISTINCT substr(invocation_id, 8, 36) AS claim FROM recall_invocations \
47+
WHERE invocation_id > 'worker/' AND invocation_id < 'worker0' \
48+
AND (length(invocation_id) = 43 OR substr(invocation_id, 44, 1) = '/') \
49+
UNION SELECT DISTINCT substr(invocation_id, 8, 36) AS claim FROM recall_output \
50+
WHERE invocation_id > 'worker/' AND invocation_id < 'worker0' \
51+
AND (length(invocation_id) = 43 OR substr(invocation_id, 44, 1) = '/')), \
52+
claimed AS (SELECT DISTINCT event_id AS claim, run_id FROM recall_events \
53+
WHERE event_type = 'run_claimed' AND producer = 'scheduler'), \
54+
ranked AS (SELECT CAST(run_id AS TEXT) AS run_id, ended_at, ROW_NUMBER() OVER (PARTITION BY location_id, target \
55+
ORDER BY ended_at DESC, run_id DESC) AS newest FROM sqb_runs \
56+
WHERE ended_at IS NOT NULL AND status IN ('success', 'warn', 'partial', 'failed', 'cancelled')) \
57+
SELECT logged.claim FROM ranked JOIN claimed ON claimed.run_id = ranked.run_id \
58+
JOIN logged ON logged.claim = claimed.claim \
59+
WHERE ranked.newest > $1 OR ranked.ended_at < $2 ORDER BY ranked.ended_at LIMIT $3";
60+
4061
/// Returns how many audit rows outlive their horizon because a sink has not acknowledged them.
4162
pub(crate) async fn retain(
4263
store: &ControlStore,
43-
(audit, statistics, snapshots): (AuditRetention, StatisticsRetention, SnapshotRetention),
64+
RetentionPolicies {
65+
audit,
66+
statistics,
67+
snapshots,
68+
run_logs,
69+
}: RetentionPolicies,
4470
sinks: &[String],
4571
) -> Result<i64, ControlError> {
72+
prune_run_logs(store, run_logs).await?;
4673
prune_snapshots(store, snapshots).await?;
4774
while prune_statistics(store, statistics).await? == MAX_HISTORY_RECORDS {
4875
tokio::task::yield_now().await;
@@ -91,3 +118,97 @@ async fn drain(
91118
tokio::task::yield_now().await;
92119
}
93120
}
121+
122+
/// A finished run's worker log goes whole once it is too old or too many newer runs finished; the run, its nodes and the scheduler's node history stay. Batches commit independently; a repeated or interrupted pass finds only logs still stored.
123+
pub(crate) async fn prune_run_logs(
124+
store: &ControlStore,
125+
retention: RunLogRetention,
126+
) -> Result<usize, ControlError> {
127+
let mut deleted = 0;
128+
loop {
129+
let transaction = begin(store, TransactionMode::Write).await?;
130+
let now = transaction.now;
131+
let (transaction, rows) = query(
132+
transaction,
133+
EXPIRED_CLAIMS,
134+
vec![
135+
Parameter::Integer(i64::from(retention.count())),
136+
Parameter::Time(retention.cutoff(now)),
137+
Parameter::Integer(RUN_LOG_RETENTION_BATCH as i64),
138+
],
139+
)
140+
.await?;
141+
let claims: Vec<String> = rows
142+
.iter()
143+
.filter_map(|row| row["claim"].as_str().map(str::to_owned))
144+
.collect();
145+
let transaction = delete_logs(transaction, &claims).await?;
146+
commit(transaction).await?;
147+
deleted += claims.len();
148+
if claims.len() < RUN_LOG_RETENTION_BATCH {
149+
return Ok(deleted);
150+
}
151+
}
152+
}
153+
154+
/// Every Recall row of the claims' invocations, then stored artifacts no remaining reference names. `worker/<claim>/<session>` sorts between `worker/<claim>/` and `worker/<claim>0` in byte order, which SQLite uses and PostgreSQL control stores declare for `invocation_id`, so each range reads the index.
155+
async fn delete_logs(
156+
mut transaction: ControlTransaction,
157+
claims: &[String],
158+
) -> Result<ControlTransaction, ControlError> {
159+
let parameters: Vec<Parameter> = claims
160+
.iter()
161+
.flat_map(|claim| {
162+
let namespace = format!("worker/{claim}");
163+
[format!("{namespace}/"), format!("{namespace}0"), namespace]
164+
})
165+
.map(Parameter::Text)
166+
.collect();
167+
let ranges: Vec<String> = (0..claims.len())
168+
.map(|index| {
169+
let first = 3 * index + 1;
170+
format!(
171+
"invocation_id=${} OR (invocation_id>${first} AND invocation_id<${})",
172+
first + 2,
173+
first + 1
174+
)
175+
})
176+
.collect();
177+
if ranges.is_empty() {
178+
return Ok(transaction);
179+
}
180+
let owned = format!("({})", ranges.join(" OR "));
181+
let rows;
182+
(transaction, rows) = query(
183+
transaction,
184+
&format!("SELECT DISTINCT digest FROM recall_artifact_refs WHERE {owned}"),
185+
parameters.clone(),
186+
)
187+
.await?;
188+
let digests: BTreeSet<String> = rows
189+
.iter()
190+
.filter_map(|row| row["digest"].as_str().map(str::to_owned))
191+
.collect();
192+
for table in INVOCATION_TABLES {
193+
(transaction, _) = query(
194+
transaction,
195+
&format!("DELETE FROM {} WHERE {owned}", table_spec(table).name),
196+
parameters.clone(),
197+
)
198+
.await?;
199+
}
200+
let digests: Vec<String> = digests.into_iter().collect();
201+
for chunk in digests.chunks(RUN_LOG_DIGEST_CHUNK) {
202+
(transaction, _) = query(
203+
transaction,
204+
&format!(
205+
"DELETE FROM recall_artifacts WHERE digest IN ({}) AND NOT EXISTS \
206+
(SELECT 1 FROM recall_artifact_refs AS ref WHERE ref.digest = recall_artifacts.digest)",
207+
Parameter::placeholders(1..=chunk.len())
208+
),
209+
chunk.iter().cloned().map(Parameter::Text).collect(),
210+
)
211+
.await?;
212+
}
213+
Ok(transaction)
214+
}

0 commit comments

Comments
 (0)