Skip to content

Commit 53d8395

Browse files
authored
fix!: atomic keyed writes, indexed control store and matching loader locations (#1478)
* fix(server): index every control-store foreign key * fix: write keyed delete_insert atomically, add store indexes in place and expand loader location templates * style: shorten the Databricks keyed MERGE docstring * fix: stop a loader from writing to a schema named preserve * fix: say a connection schema counts only for a project's only top-level connection * fix: show the S104 hint's connection schema with command-line vars applied * fix(debug): report no database for DuckDB targets Debug passed the DuckDB connection file path as the target database, while models build in the file's own catalog. Debug now drops it for namespaceless adapters, as diff already does. * fix: check only intermediate loaders under a preserved target schema A managed source's loader writes to the source's own schema or the target's loader_schema, so a target schema of preserve no longer stops it. The check now applies when the selected target's own schema is preserve and the loader's destination names no schema, the same way for compile, build and load. * refactor: read the loader node flag without a Python call * fix(load): load a managed source into the target's loader_schema sqb load filled a managed source's missing schema from the target's schema, while build uses the target's loader_schema first, so the two commands wrote the same source to different schemas. Load now resolves it the same way, and its missing-schema error names a managed source as one. * refactor(load): pick a managed source's schema without comparing field names * fix(load): stop a managed source that would load into a schema named preserve With loader_schema or schema set to preserve and no schema on the source, build stops with P001 while load wrote into a schema literally named preserve. Load now raises the same error. * fix(load): expand templated target locations for managed sources sqb load read the target's database, schema and loader_schema as written when it placed a managed source, so a value such as ${raw_schema} failed with a parser error while build expanded it. Load now expands them with the project and command-line vars, as build does. * fix(load): expand only the target location a managed source falls back to Expanding the whole target up front failed load on unused templates that build ignores, and accepted run context templates that build refuses. Load now expands just the fallback value, with build's own options. * fix(load): read an expanded target location as text, as build does * docs(deploy): run the server and a location as Podman Quadlet units * fix(scheduler): wake on a due time however long the store took, and log slow cycles The loop slept for the next due time's distance measured before the cycle's last commit, so a slow commit or WAL checkpoint pushed every wake back by its own length. It now sleeps until that deadline. Each cycle records per-phase timings, the worker supervisor logs cycles slower than 500 ms with them, and the latency test bounds how late the scheduler wakes while printing store time separately. * refactor: share target lookup and load-selection request reading Named-target checks reuse the compile attachment's resolved_target instead of a copy, the named-target schema binding declares its keyword arguments directly, and both load-selection bindings read their request through one helper.
1 parent 1b98482 commit 53d8395

50 files changed

Lines changed: 1370 additions & 199 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎crates/sqlbuild-analysis/src/assembly/target_checks/errors.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,4 +13,6 @@ pub enum TargetError {
1313
ManagedWriteSchema(String),
1414
/// A named target without an explicit schema.
1515
NamedTargetSchema(String),
16+
/// An intermediate loader, by name, whose schema resolves to the target's `preserve`.
17+
PreservedLoaderSchema(String),
1618
}

‎crates/sqlbuild-analysis/src/assembly/target_checks/main/target_error_message.rs‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,12 +35,18 @@ pub fn target_error_message(error: &TargetError) -> String {
3535
),
3636
TargetError::ManagedWriteSchema(label) => format!(
3737
"{label} has no explicitly resolved physical write schema. Set schema on the \
38-
resource or its defaults, selected target, loader_schema, or connection."
38+
resource or its defaults, the selected target or its loader_schema, or the \
39+
project's only top-level [connection]."
3940
),
4041
TargetError::NamedTargetSchema(target) => format!(
4142
"Named target '{target}' must explicitly set schema to a nonblank literal or \
4243
'{PRESERVE_TARGET_VALUE}'. Resource schemas, defaults, and connection schemas do \
4344
not select a named target schema strategy."
4445
),
46+
TargetError::PreservedLoaderSchema(loader) => format!(
47+
"Intermediate loader '{loader}' would write to a schema named \
48+
'{PRESERVE_TARGET_VALUE}'. The target's schema '{PRESERVE_TARGET_VALUE}' keeps each \
49+
model's own schema, and a loader has none, so its destination must name a schema."
50+
),
4551
}
4652
}

‎crates/sqlbuild-analysis/src/assembly/target_checks/tests/test_target_checks.rs‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,13 +46,18 @@ fn given_target_error_when_formatting_then_matches_python_message() {
4646
MessageTestCase {
4747
description: "a managed write without a schema",
4848
error: TargetError::ManagedWriteSchema("Model 'orders'".to_owned()),
49-
expected_message: "Model 'orders' has no explicitly resolved physical write schema. Set schema on the resource or its defaults, selected target, loader_schema, or connection.",
49+
expected_message: "Model 'orders' has no explicitly resolved physical write schema. Set schema on the resource or its defaults, the selected target or its loader_schema, or the project's only top-level [connection].",
5050
},
5151
MessageTestCase {
5252
description: "a named target without a schema",
5353
error: TargetError::NamedTargetSchema("ci".to_owned()),
5454
expected_message: "Named target 'ci' must explicitly set schema to a nonblank literal or 'preserve'. Resource schemas, defaults, and connection schemas do not select a named target schema strategy.",
5555
},
56+
MessageTestCase {
57+
description: "an intermediate loader under a preserved target schema",
58+
error: TargetError::PreservedLoaderSchema("raw_orders".to_owned()),
59+
expected_message: "Intermediate loader 'raw_orders' would write to a schema named 'preserve'. The target's schema 'preserve' keeps each model's own schema, and a loader has none, so its destination must name a schema.",
60+
},
5661
];
5762

5863
for test_case in test_cases {

‎crates/sqlbuild-cli/src/workers/_helpers/supervisor.rs‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ use sqlbuild_orchestration::runs::models::{
1616
};
1717
use sqlbuild_orchestration::scheduler::constants::RENEW_INTERVAL_SECONDS;
1818
use sqlbuild_orchestration::scheduler::main::renew_lease::renew_lease;
19-
use sqlbuild_orchestration::scheduler::models::SchedulerLease;
19+
use sqlbuild_orchestration::scheduler::models::{SchedulerCycle, SchedulerLease};
2020
use sqlbuild_orchestration::store::errors::ControlError;
2121
use sqlbuild_orchestration::store::models::ControlStore;
2222
use sqlbuild_process_ownership::ownership::main::{
@@ -26,6 +26,7 @@ use tokio::process::Command;
2626
use uuid::Uuid;
2727

2828
use crate::workers::_helpers::invocation::{arguments, environment, excluded};
29+
use crate::workers::constants::SLOW_SCHEDULER_CYCLE;
2930
use crate::workers::models::{
3031
ClaimedWorker, DockerWorker, Launcher, LocalLauncher, LocalSpawn, WorkerChild, WorkerExit,
3132
Workers,
@@ -149,6 +150,22 @@ impl sqlbuild_orchestration::scheduler::types::RunLauncher for Workers {
149150
self.shutdown_workers(store, lease).await
150151
}
151152

153+
fn cycled(&mut self, cycle: &SchedulerCycle) {
154+
let timings = &cycle.timings;
155+
if timings.total() >= SLOW_SCHEDULER_CYCLE {
156+
drop(writeln!(
157+
std::io::stderr().lock(),
158+
"Slow scheduler cycle: {} ms (lease {} ms, triggers {} ms, run monitor {} ms, tick cleanup {} ms, next due {} ms)",
159+
timings.total().as_millis(),
160+
timings.lease.as_millis(),
161+
timings.triggers.as_millis(),
162+
timings.monitor.as_millis(),
163+
timings.prune.as_millis(),
164+
timings.next_due.as_millis(),
165+
));
166+
}
167+
}
168+
152169
/// A claim whose commit outcome is unknown is adopted by the next recovery pass.
153170
fn retrying(&mut self, error: &ControlError) {
154171
self.recovered_epoch = None;
Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1-
//! Immutable Docker image identities use the full SHA-256 hexadecimal digest.
1+
//! Worker supervision limits.
22
3+
/// Immutable Docker image identities use the full SHA-256 hexadecimal digest.
34
pub(super) const IMAGE_ID_HEX_LENGTH: usize = 64;
5+
6+
/// Healthy scheduler cycles take tens of milliseconds; slower ones are logged with phase timings.
7+
pub(super) const SLOW_SCHEDULER_CYCLE: std::time::Duration = std::time::Duration::from_millis(500);

‎crates/sqlbuild-orchestration/src/scheduler/_helpers/drain_due_triggers.rs‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
//! Drain a bounded due-trigger page, renewing the lease between persisted evaluations.
22
3-
use std::time::Duration;
3+
use std::time::{Duration, Instant};
44

55
use chrono::{DateTime, Utc};
66
use uuid::Uuid;
@@ -42,24 +42,26 @@ pub(crate) async fn drain_due_triggers(
4242
Ok((lease, cycle))
4343
}
4444

45-
/// Shorten the cycle wait to the next running trigger's due time, never below zero.
45+
/// Shorten the wait to the next due time, never below zero, measured from the returned instant.
4646
pub(crate) async fn wait_for_next_due(
4747
store: &ControlStore,
4848
wait_for: Duration,
49-
) -> Result<Duration, ControlError> {
49+
) -> Result<(Duration, Instant), ControlError> {
5050
let transaction = begin(store, TransactionMode::Read).await?;
5151
let now = transaction.now;
52+
let read_at = Instant::now();
5253
let (transaction, rows) = query(transaction,
5354
"SELECT MIN(s.next_due_at) AS next_due_at FROM sqb_trigger_state s JOIN sqb_triggers t ON t.trigger_id=s.trigger_id \
5455
JOIN sqb_locations l ON l.location_id=t.location_id \
5556
WHERE s.status='running' AND t.removed_in_snapshot_id IS NULL AND l.archived_at IS NULL", vec![]).await?;
5657
let next: Option<DateTime<Utc>> = serde_json::from_value(rows[0]["next_due_at"].clone())
5758
.map_err(|error| ControlError::Invalid(error.to_string()))?;
5859
commit(transaction).await?;
59-
Ok(next.map_or(wait_for, |next| {
60+
let wait_for = next.map_or(wait_for, |next| {
6061
(next - now)
6162
.to_std()
6263
.unwrap_or(Duration::ZERO)
6364
.min(wait_for)
64-
}))
65+
});
66+
Ok((wait_for, read_at))
6567
}
Lines changed: 25 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
//! A cycle renews leadership and drains a bounded set of persisted due triggers.
22
3-
use std::time::Duration;
3+
use std::time::{Duration, Instant};
4+
5+
use chrono::Utc;
46

57
use crate::runs::constants::MONITOR_BATCH_SIZE;
68
use crate::runs::main::monitor_runs::monitor_runs;
@@ -9,14 +11,23 @@ use crate::scheduler::_helpers::tick_retention::prune_ticks_when_due;
911
use crate::scheduler::constants::RENEW_INTERVAL_SECONDS;
1012
use crate::scheduler::main::_acquire_lease::acquire_lease;
1113
use crate::scheduler::main::renew_lease::renew_lease;
12-
use crate::scheduler::models::{SchedulerCycle, SchedulerState};
14+
use crate::scheduler::models::{CycleTimings, SchedulerCycle, SchedulerState};
1315
use crate::store::errors::ControlError;
1416
use crate::store::models::ControlStore;
1517

1618
pub(crate) async fn scheduler_cycle(
1719
store: &ControlStore,
1820
mut state: SchedulerState,
1921
) -> Result<(SchedulerState, SchedulerCycle), ControlError> {
22+
let mut timings = CycleTimings {
23+
started_at: Utc::now(),
24+
lease: Duration::ZERO,
25+
triggers: Duration::ZERO,
26+
monitor: Duration::ZERO,
27+
prune: Duration::ZERO,
28+
next_due: Duration::ZERO,
29+
};
30+
let phase = Instant::now();
2031
let renewal = match state.lease.as_ref() {
2132
Some(lease) => renew_lease(store, lease).await.map(Some),
2233
None => acquire_lease(store, state.holder).await,
@@ -26,23 +37,34 @@ pub(crate) async fn scheduler_cycle(
2637
Err(ControlError::Fenced) => None,
2738
Err(error) => return Err(error),
2839
};
40+
timings.lease = phase.elapsed();
2941
let cycle = SchedulerCycle {
3042
leader: state.lease.is_some(),
3143
evaluated: 0,
3244
ticks: 0,
3345
unresponsive: 0,
3446
wait_for: Duration::from_secs(RENEW_INTERVAL_SECONDS),
47+
wait_from: Instant::now(),
48+
timings,
3549
};
3650
let Some(lease) = state.lease.clone() else {
3751
return Ok((state, cycle));
3852
};
53+
let phase = Instant::now();
3954
let (lease, mut cycle) = drain_due_triggers(store, lease, cycle).await?;
55+
cycle.timings.triggers = phase.elapsed();
56+
let phase = Instant::now();
4057
cycle.unresponsive = monitor_runs(store, &lease).await?;
4158
if cycle.unresponsive == MONITOR_BATCH_SIZE {
4259
cycle.wait_for = Duration::ZERO;
4360
}
61+
cycle.timings.monitor = phase.elapsed();
62+
let phase = Instant::now();
4463
state.pruned_at = prune_ticks_when_due(store, &lease, state.pruned_at).await?;
4564
state.lease = Some(lease);
46-
cycle.wait_for = wait_for_next_due(store, cycle.wait_for).await?;
65+
cycle.timings.prune = phase.elapsed();
66+
let phase = Instant::now();
67+
(cycle.wait_for, cycle.wait_from) = wait_for_next_due(store, cycle.wait_for).await?;
68+
cycle.timings.next_due = phase.elapsed();
4769
Ok((state, cycle))
4870
}

‎crates/sqlbuild-orchestration/src/scheduler/main/run_scheduler.rs‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
use std::time::Duration;
44

55
use tokio::sync::watch;
6+
use tokio::time::Instant;
67

78
use crate::scheduler::main::_scheduler_cycle::scheduler_cycle;
89
use crate::scheduler::models::SchedulerState;
@@ -24,23 +25,25 @@ pub async fn run_scheduler(
2425
Ok((next_state, cycle)) => {
2526
state = next_state;
2627
last_lease = state.lease.clone();
28+
launcher.cycled(&cycle);
2729
launcher
2830
.poll(store, state.lease.as_ref())
2931
.await
30-
.map(|()| cycle.wait_for)
32+
.map(|()| Instant::from_std(cycle.wait_from + cycle.wait_for))
3133
}
3234
Err(error) => Err(error),
3335
};
34-
let wait_for = match polled {
35-
Ok(wait_for) => wait_for,
36+
let retry = Instant::now() + Duration::from_secs(1);
37+
let due = match polled {
38+
Ok(due) => due.min(retry),
3639
Err(error) if error.transient() => {
3740
launcher.retrying(&error);
38-
Duration::from_secs(1)
41+
retry
3942
}
4043
Err(error) => return Err(error),
4144
};
4245
tokio::select! {
43-
_ = tokio::time::sleep(wait_for.min(Duration::from_secs(1))) => {},
46+
_ = tokio::time::sleep_until(due) => {},
4447
result = shutdown.changed() => { if result.is_err() { break; } },
4548
}
4649
}

‎crates/sqlbuild-orchestration/src/scheduler/models.rs‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,26 @@ pub struct SchedulerCycle {
4040
pub ticks: usize,
4141
pub unresponsive: usize,
4242
pub wait_for: std::time::Duration,
43+
/// When `wait_for` was read, so the loop wakes on time however long the store takes afterwards.
44+
pub wait_from: std::time::Instant,
45+
pub timings: CycleTimings,
46+
}
47+
48+
/// Wall time per cycle phase, which separates a late wake from slow store writes.
49+
#[derive(Clone, Debug)]
50+
pub struct CycleTimings {
51+
pub started_at: DateTime<Utc>,
52+
pub lease: std::time::Duration,
53+
pub triggers: std::time::Duration,
54+
pub monitor: std::time::Duration,
55+
pub prune: std::time::Duration,
56+
pub next_due: std::time::Duration,
57+
}
58+
59+
impl CycleTimings {
60+
pub fn total(&self) -> std::time::Duration {
61+
self.lease + self.triggers + self.monitor + self.prune + self.next_due
62+
}
4363
}
4464

4565
/// Validated at the native boundary and evaluated as wall-clock times with SQLBuild's DST policy.

‎crates/sqlbuild-orchestration/src/scheduler/types.rs‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
//! Compound scheduler return values.
22
3-
use crate::scheduler::models::SchedulerLease;
3+
use crate::scheduler::models::{SchedulerCycle, SchedulerLease};
44
use crate::store::errors::ControlError;
55
use crate::store::models::ControlStore;
66

@@ -18,6 +18,9 @@ pub trait RunLauncher {
1818
lease: Option<&SchedulerLease>,
1919
) -> impl Future<Output = Result<(), ControlError>> + Send;
2020

21+
/// Hears each completed cycle, with its phase timings, before the launcher polls.
22+
fn cycled(&mut self, _cycle: &SchedulerCycle) {}
23+
2124
/// Hears each transient store failure the scheduler retries instead of stopping.
2225
fn retrying(&mut self, _error: &ControlError) {}
2326
}

0 commit comments

Comments
 (0)