Skip to content

Commit d34f0ab

Browse files
committed
docs(server)!: upgrade in order, keeping the scheduler running to drain runs
The server upgrade note no longer offers `scheduler = false`: the scheduler loop launches queued runs and advances group runs, so with it off nothing drains. It now gives the order to upgrade in and when to resume schedules and sensors, and says that until a location has published again a group's schedule or sensor stops the scheduler and a job run that fails can't record its failure. A test shows that failure: the worker's report that a job run failed is refused, because deciding on a retry reads the snapshot's documents before it checks for a retry policy, and the run stays running. BREAKING CHANGE: a published job's `resolved_concurrency` is now `{limit}`; its `source` field and the `ConcurrencySource` schema are removed from the control API, and `sqb build --verbose` no longer names the setting the concurrency came from. Control store migration 0004 removes the field from each stored job definition, but not from the snapshot documents jobs were published in. Upgrade in this order: 1. Pause every schedule and sensor, and start no manual runs. Keep the scheduler running. 2. Let queued and running runs finish, or cancel them. 3. Upgrade the server. 4. Upgrade and restart every location. 5. Once every location has published again, resume schedules and sensors. Until a location has published again, its job pages, group runs and ad hoc runs fail, a group's schedule or sensor stops the scheduler, and a job run that fails can't record its failure.
1 parent 5531575 commit d34f0ab

6 files changed

Lines changed: 167 additions & 26 deletions

File tree

‎crates/sqlbuild-orchestration/tests/scheduler/helpers.rs‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -227,3 +227,56 @@ pub(crate) fn upgrade_snapshot(
227227
input.documents.insert("triggers".into(), triggers);
228228
input
229229
}
230+
231+
/// Report a claimed run as its worker does: started, each node with `node`, then finished with `run`; each report's outcome.
232+
pub(crate) async fn report_run(
233+
store: &ControlStore,
234+
claim: &crate::runs::models::RunClaim,
235+
(node, run): (
236+
crate::runs::models::NodeOutcome,
237+
crate::runs::models::RunOutcome,
238+
),
239+
) -> Vec<String> {
240+
use crate::runs::main::execution::change_run::change_run;
241+
use crate::runs::models::{RunAuthority, RunChange, RunChangeRequest};
242+
243+
let mut changes = vec![RunChange::Started];
244+
changes.extend(
245+
claim
246+
.request
247+
.selection
248+
.iter()
249+
.map(|node_key| RunChange::NodeFinished {
250+
node_key: node_key.clone(),
251+
outcome: node,
252+
rows_affected: Some(1),
253+
error_code: None,
254+
skipped_because_of: None,
255+
error_message: None,
256+
}),
257+
);
258+
changes.push(RunChange::Finished {
259+
outcome: run,
260+
reason: "the worker finished".into(),
261+
exit_code: Some(1),
262+
});
263+
let mut outcomes = Vec::new();
264+
for change in changes {
265+
let reported = change_run(
266+
store,
267+
RunAuthority::Worker {
268+
run_id: claim.request.run_id,
269+
claim_id: claim.claim_id,
270+
},
271+
&RunChangeRequest {
272+
event_id: Uuid::now_v7(),
273+
run_id: claim.request.run_id,
274+
claim_id: Some(claim.claim_id),
275+
change,
276+
},
277+
)
278+
.await;
279+
outcomes.push(outcome(reported));
280+
}
281+
outcomes
282+
}

‎crates/sqlbuild-orchestration/tests/scheduler/test_concurrency_source.rs‎

Lines changed: 76 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,16 +12,19 @@ use crate::credentials::tests::role_helpers::{credentials, role_fixture};
1212
use crate::declarations::main::read_person_job::read_person_job;
1313
use crate::declarations::models::ServerPoolCaps;
1414
use crate::runs::main::{
15-
execution::advance_groups::advance_groups, requests::request_job_run::request_job_run,
15+
execution::advance_groups::advance_groups, execution::claim_next_run::claim_next_run,
16+
requests::request_job_run::request_job_run,
1617
};
18+
use crate::runs::models::{NodeOutcome, RunOutcome};
1719
use crate::runs::tests::helpers::{json_column, rows, snapshot};
1820
use crate::scheduler::models::SchedulerState;
1921
use crate::scheduler::tests::contracts::helpers::{
20-
acquire_lease, manual_run, operator, outcome, scheduler_cycle, store_old_documents,
22+
acquire_lease, manual_run, operator, outcome, report_run, scheduler_cycle, store_old_documents,
2123
store_old_job_definitions, upgrade_snapshot,
2224
};
2325
use crate::scheduler::tests::contracts::test_types::{
24-
GroupScheduleOldDocumentTestCase, MigratedConcurrencySourceTestCase, PinnedOldDocumentTestCase,
26+
FailedJobRunOldDocumentTestCase, GroupScheduleOldDocumentTestCase,
27+
MigratedConcurrencySourceTestCase, PinnedOldDocumentTestCase,
2528
};
2629
use crate::scheduler::tests::contracts::ticks::helpers::calendar_text;
2730
use crate::sensors::tests::helpers::append_events;
@@ -254,6 +257,76 @@ fn given_a_group_schedule_published_before_0004_when_the_scheduler_cycles_then_i
254257
}
255258
}
256259

260+
// A job run that fails reads its snapshot's documents to decide on a retry before it checks for a
261+
// retry policy, so until the location publishes again its failure report is refused and the run
262+
// stays running until it is found unresponsive. Tracked separately with the other reads, and that
263+
// change replaces this expectation.
264+
#[test]
265+
fn given_a_job_published_before_0004_when_its_run_fails_before_a_republish_then_the_failure_is_refused()
266+
{
267+
let test_cases = [FailedJobRunOldDocumentTestCase {
268+
description: "SQLite store",
269+
backend: Backend::Sqlite,
270+
stored_source: "project",
271+
expected_reports: ["ok", "ok", OLD_FORM],
272+
expected_status: "running",
273+
}];
274+
for test_case in test_cases {
275+
run(async {
276+
let fixture = open_test_store(test_case.backend)
277+
.await
278+
.expect("required backend");
279+
let store = &fixture.store;
280+
migrate_with(store, &MIGRATIONS[..2])
281+
.await
282+
.expect("a store at version 3");
283+
let location = create_location(store, "orders", "test-admin")
284+
.await
285+
.expect("location");
286+
publish_snapshot(store, &upgrade_snapshot(location, json!({})))
287+
.await
288+
.expect("publish");
289+
store_old_job_definitions(store, test_case.stored_source).await;
290+
store_old_documents(store, test_case.stored_source).await;
291+
migrate_store(store).await.expect("migrate to version 4");
292+
request_job_run(store, (&operator(), None), &manual_run(location, "orders"))
293+
.await
294+
.expect("a manual run of the migrated job");
295+
let lease = acquire_lease(store, Uuid::now_v7())
296+
.await
297+
.expect("elect")
298+
.expect("leader");
299+
let claim = claim_next_run(
300+
store,
301+
&lease,
302+
&BTreeMap::new(),
303+
Some((location, "development")),
304+
)
305+
.await
306+
.expect("claim")
307+
.expect("the queued run");
308+
309+
let reports =
310+
report_run(store, &claim, (NodeOutcome::Failed, RunOutcome::Failed)).await;
311+
let status = rows(store, "SELECT status FROM sqb_runs", vec![]).await;
312+
313+
assert_eq!(
314+
(
315+
reports,
316+
status[0]["status"].as_str().unwrap_or_default().to_owned()
317+
),
318+
(
319+
test_case.expected_reports.map(str::to_owned).to_vec(),
320+
test_case.expected_status.to_owned()
321+
),
322+
"{}: the failure report is refused and the run stays running",
323+
test_case.description
324+
);
325+
close_test_store(fixture).await;
326+
});
327+
}
328+
}
329+
257330
// Runs pinned to a snapshot published before 0004 read its immutable documents, which 0004 can't
258331
// rewrite. They still fail with `ControlError::Invalid`, which stops the scheduler; operators let
259332
// such runs finish or cancel them before upgrading. Turning this into a per-run failure is tracked

‎crates/sqlbuild-orchestration/tests/scheduler/test_types.rs‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,3 +45,14 @@ pub(crate) struct PinnedOldDocumentTestCase {
4545
/// The `ControlError::Invalid` message advancing the group returns.
4646
pub(crate) expected_error: &'static str,
4747
}
48+
49+
pub(crate) struct FailedJobRunOldDocumentTestCase {
50+
pub(crate) description: &'static str,
51+
pub(crate) backend: Backend,
52+
/// The concurrency source an older release stored with each job.
53+
pub(crate) stored_source: &'static str,
54+
/// The outcome of each worker report: started, the node finished, the run failed.
55+
pub(crate) expected_reports: [&'static str; 3],
56+
/// The run's status afterwards.
57+
pub(crate) expected_status: &'static str,
58+
}

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/server.md‎

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -314,17 +314,19 @@ afterwards, so back it up before you upgrade (see
314314
[Back up the control database](#back-up-the-control-database)). To roll back, restore that backup;
315315
runs recorded since the upgrade are lost.
316316

317-
Before you upgrade, pause your schedules and sensors, or keep the scheduler off (set
318-
`scheduler = false` under `[server]` and stop any `sqb scheduler`), then let queued and running
319-
runs finish, or cancel them. A run stays pinned to the snapshot it started on, and a snapshot an
320-
earlier release published can hold job definitions this release no longer reads: a group run or a
321-
retry from it fails with `` jobs: unknown field `source`, expected `limit` ``, which stops the
322-
scheduler.
323-
324-
After upgrading the server, upgrade and restart every location, so each publishes its project again
325-
with the new release. Until a location has published again, its job pages, group runs and ad hoc
326-
runs fail with the same error, and a group's schedule stops the scheduler; its job runs from
327-
schedules, sensors and manual requests work.
317+
A run stays pinned to the snapshot it started on, and a snapshot an earlier release published can
318+
hold job definitions this release no longer reads, which fail with
319+
`` jobs: unknown field `source`, expected `limit` ``. Upgrade in this order:
320+
321+
1. Pause every schedule and sensor, and start no manual runs. Keep the scheduler running: it
322+
launches queued runs and advances group runs.
323+
2. Let queued and running runs finish, or cancel them.
324+
3. Upgrade the server.
325+
4. Upgrade and restart every location, so each publishes its project again with the new release.
326+
5. Once every location has published again, resume schedules and sensors.
327+
328+
Until a location has published again, its job pages, group runs and ad hoc runs fail, a group's
329+
schedule or sensor stops the scheduler, and a job run that fails can't record its failure.
328330

329331
To see what an upgrade changes before you start the server, run the new release with `--dry-run`:
330332

‎ui/src/routes/jobs/[name]/job-view.svelte‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -531,7 +531,7 @@
531531
.picked li {
532532
padding: 2px 0;
533533
}
534-
/* The Concurrency panel: one plain fact per line, where it came from quieter beside it. */
534+
/* The Concurrency panel: one plain fact per line, any detail quieter beside it. */
535535
.concurrency {
536536
display: flex;
537537
flex-direction: column;

‎website/src/content/docs/docs/cli/server.mdx‎

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -293,17 +293,19 @@ afterwards, so back it up before you upgrade (see
293293
[Back up the control database](#back-up-the-control-database)). To roll back, restore that backup;
294294
runs recorded since the upgrade are lost.
295295

296-
Before you upgrade, pause your schedules and sensors, or keep the scheduler off (set
297-
`scheduler = false` under `[server]` and stop any `sqb scheduler`), then let queued and running
298-
runs finish, or cancel them. A run stays pinned to the snapshot it started on, and a snapshot an
299-
earlier release published can hold job definitions this release no longer reads: a group run or a
300-
retry from it fails with `` jobs: unknown field `source`, expected `limit` ``, which stops the
301-
scheduler.
302-
303-
After upgrading the server, upgrade and restart every location, so each publishes its project again
304-
with the new release. Until a location has published again, its job pages, group runs and ad hoc
305-
runs fail with the same error, and a group's schedule stops the scheduler; its job runs from
306-
schedules, sensors and manual requests work.
296+
A run stays pinned to the snapshot it started on, and a snapshot an earlier release published can
297+
hold job definitions this release no longer reads, which fail with
298+
`` jobs: unknown field `source`, expected `limit` ``. Upgrade in this order:
299+
300+
1. Pause every schedule and sensor, and start no manual runs. Keep the scheduler running: it
301+
launches queued runs and advances group runs.
302+
2. Let queued and running runs finish, or cancel them.
303+
3. Upgrade the server.
304+
4. Upgrade and restart every location, so each publishes its project again with the new release.
305+
5. Once every location has published again, resume schedules and sensors.
306+
307+
Until a location has published again, its job pages, group runs and ad hoc runs fail, a group's
308+
schedule or sensor stops the scheduler, and a job run that fails can't record its failure.
307309

308310
To see what an upgrade changes before you start the server, run the new release with `--dry-run`:
309311

0 commit comments

Comments
 (0)