Skip to content

Commit d715c7e

Browse files
authored
feat(plan): will-run / won't-run summary and a clearer Renames section (#1602)
* feat(plan): will run and won't run in the plan index, Renames and migrations apart, and renames marked in the plan JSON (CHI-1563) * fix(plan): bring the summary's migration step count through the server client, and keep read failures to their group - The server client copied a plan summary's fields one by one and dropped migration_steps, so the Renames and migrations section lost its data moves and its step count on a real server. It is now required on every summary and copied; a summary without it is refused. - A browser test runs the plan page through the real server client against a faked control server (e2e/support/fake-plans.ts), covering migration_steps, origin_model, old_name and is_rename; it fails when the client drops the step count again. - A failed read is kept to its group, warnings included, and to paging or filtering in Renames and migrations; a re-plan clears them all. - The static demo's plan previews predated plan summaries and paged sections; they are recorded again from DuckShop on sqb dev through the summary, rows and diff reads (scripts/record-demo-plans.mjs). * docs(plan): regenerate the docs skill pages for the plan and migrations docs * refactor(plan): one-line doc comments, the rename reason with the plan output constants, and branch-free test helpers
1 parent 1ed23cf commit d715c7e

39 files changed

Lines changed: 1740 additions & 100 deletions

File tree

‎crates/sqlbuild-core/src/plan_display/_helpers/plan_pages.rs‎

Lines changed: 33 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ use serde_json::Value;
77
use crate::plan_display::_helpers::diff_lines::diff_stats;
88
use crate::plan_display::_helpers::warning_groups::warning_groups;
99
use crate::plan_display::constants::{
10-
DIFF_STATS_FIELD, FIRST_RUN_REASON, NO_CHANGE_REASON, SKIP_ACTION,
10+
DIFF_STATS_FIELD, FIRST_RUN_REASON, IS_RENAME_FIELD, NO_CHANGE_REASON, ORIGIN_MODEL_FIELD,
11+
SKIP_ACTION,
1112
};
1213
use crate::plan_display::models::PlanPageSection;
1314

@@ -81,6 +82,21 @@ impl<'document> PlanRows<'document> {
8182
Self { document, runs }
8283
}
8384

85+
/// The migration steps: every migration-like row except renamed models' own migrations.
86+
pub(crate) fn migration_steps(&self) -> usize {
87+
let migrations = self
88+
.list("migrations")
89+
.into_iter()
90+
.filter(|migration| {
91+
migration.get(IS_RENAME_FIELD).and_then(Value::as_bool) != Some(true)
92+
})
93+
.count();
94+
migrations
95+
+ self.list("column_migrations").len()
96+
+ self.list("table_type_conversions").len()
97+
+ self.list("retention").len()
98+
}
99+
84100
/// The section's rows in plan order.
85101
pub(crate) fn rows(&self, section: PlanPageSection) -> Vec<&'document Value> {
86102
match section {
@@ -345,13 +361,26 @@ fn model_section(model: &Value) -> PlanPageSection {
345361
.map_or(PlanPageSection::RoutineModels, |(_, section)| *section)
346362
}
347363

348-
/// The name a row is found by; a warning is found by its model and message.
364+
/// The name a row is found by; renames match old names and retention changes their models.
349365
pub(crate) fn row_name(section: PlanPageSection, row: &Value) -> String {
350366
match section {
367+
PlanPageSection::Renamed => {
368+
format!("{} {}", text(row, "name"), text(row, ORIGIN_MODEL_FIELD))
369+
}
351370
PlanPageSection::Migrations
352371
| PlanPageSection::ColumnMigrations
353-
| PlanPageSection::TableTypeConversions
354-
| PlanPageSection::Retention => text(row, "model").to_owned(),
372+
| PlanPageSection::TableTypeConversions => text(row, "model").to_owned(),
373+
PlanPageSection::Retention => {
374+
let mut names: Vec<&str> = row
375+
.get("models")
376+
.and_then(Value::as_array)
377+
.into_iter()
378+
.flatten()
379+
.filter_map(Value::as_str)
380+
.collect();
381+
names.push(text(row, "name"));
382+
names.join(" ")
383+
}
355384
PlanPageSection::HeldBack => key_name(text(row, "node")).to_owned(),
356385
PlanPageSection::Unchanged => key_name(row.as_str().unwrap_or_default()).to_owned(),
357386
PlanPageSection::Warnings => {

‎crates/sqlbuild-core/src/plan_display/constants.rs‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,14 @@ pub(crate) const SKIP_ACTION: &str = "skip";
4949
pub(crate) const NO_CHANGE_REASON: &str = "no_change";
5050
/// The reason of a model or seed without a recorded build.
5151
pub(crate) const FIRST_RUN_REASON: &str = "first_run";
52+
/// The field naming the model a renamed model was renamed from, on its row and on its migration's.
53+
pub const ORIGIN_MODEL_FIELD: &str = "origin_model";
54+
/// The field saying what stays at a moved model's old name, on its migration's row and a renamed model's.
55+
pub const OLD_NAME_FIELD: &str = "old_name";
56+
/// The migration row field saying the migration is a renamed model's own.
57+
pub const IS_RENAME_FIELD: &str = "is_rename";
58+
/// The summary field counting migration steps other than renamed models' own migrations.
59+
pub(crate) const MIGRATION_STEPS_FIELD: &str = "migration_steps";
5260
/// The plan document field of per-model metadata, which a summary leaves out.
5361
pub(crate) const METADATA_FIELD: &str = "metadata";
5462
/// The plan document's row lists, which a summary empties and counts by section instead.

‎crates/sqlbuild-core/src/plan_display/main/plan_summary.rs‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,9 @@ use serde_json::Value;
44

55
use crate::plan_display::_helpers::plan_pages::PlanRows;
66
use crate::plan_display::_helpers::plan_sections::{emptied, section_counts, with_warning_groups};
7+
use crate::plan_display::constants::MIGRATION_STEPS_FIELD;
78

8-
/// The document with every row list emptied, each section counted and its warnings grouped; a failure report is returned unchanged.
9+
/// The document with its row lists emptied and counted by section; a failure report is unchanged.
910
pub fn plan_summary(document: &Value) -> Value {
1011
let Some(fields) = document
1112
.as_object()
@@ -18,5 +19,6 @@ pub fn plan_summary(document: &Value) -> Value {
1819
let mut summary = with_warning_groups(&plan, emptied(fields));
1920
summary.insert("sections".into(), Value::Array(sections));
2021
summary.insert("pages".into(), Value::Object(pages));
22+
summary.insert(MIGRATION_STEPS_FIELD.into(), plan.migration_steps().into());
2123
Value::Object(summary)
2224
}

‎crates/sqlbuild-core/src/plan_display/tests/helpers.rs‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,9 @@ pub(super) fn full_plan() -> serde_json::Value {
3838
let mut payments = planned_model("payments", ("create_table", "query_changed"));
3939
payments["query_diff"] = serde_json::json!(["-select id", "+SELECT id"]);
4040
payments["config_diff"] = serde_json::json!(["- \"tags\": []", "+ \"tags\": [\"finance\"]"]);
41+
let mut old_orders = planned_model("old_orders", ("create_table", "renamed"));
42+
old_orders["origin_model"] = serde_json::json!("orders_v1");
43+
old_orders["old_name"] = serde_json::json!({"action": "view_only", "view": "main.orders_v1", "expires_at": "2026-12-18T00:00:00Z"});
4144
let mut totals = planned_model("totals", ("create_table", "config_changed"));
4245
totals["config_diff"] = serde_json::json!([
4346
"--- a",
@@ -85,7 +88,7 @@ pub(super) fn full_plan() -> serde_json::Value {
8588
ledger,
8689
daily_orders,
8790
daily_payments,
88-
planned_model("old_orders", ("create_table", "renamed")),
91+
old_orders,
8992
planned_model("archive", ("create_table", "full_refresh")),
9093
planned_model("customers", ("create_view", "no_change")),
9194
planned_model("Customer_Notes", ("create_view", "no_change")),
@@ -118,10 +121,13 @@ pub(super) fn full_plan() -> serde_json::Value {
118121
{"severity": "warning", "message": "model stg__customers_3 has no tests."},
119122
{"severity": "warning", "message": "scenario 'returns' step `refund` is stale", "code": "W1"},
120123
],
121-
"migrations": [{"model": "old_orders", "origin_model": "orders_v1", "decision": "rename", "completed_at": null}],
124+
"migrations": [
125+
{"model": "old_orders", "origin_model": "orders_v1", "decision": "renamed", "completed_at": null, "is_rename": true},
126+
{"model": "totals", "origin_model": "totals_legacy", "decision": "forced_replace", "completed_at": null, "is_rename": false},
127+
],
122128
"column_migrations": [{"model": "ledger", "origin_column": "amt", "destination_column": "amount", "decision": "rename"}],
123129
"table_type_conversions": [{"model": "totals", "actual_type": "view", "desired_type": "table"}],
124-
"retention": [{"model": "events", "action": "set"}],
130+
"retention": [{"scope": "table", "name": "main.events", "models": ["events"], "desired_days": 7, "actual_days": 1, "direction": "increase"}],
125131
"metadata": {"orders": {"columns": 3}},
126132
"synced_copy": {"synced_at": null, "cursor_window_starts": {"events": "2026-11-01"}, "unknown_until_run": []},
127133
"changed": {

‎crates/sqlbuild-core/src/plan_display/tests/test_plan_pages.rs‎

Lines changed: 110 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ fn given_a_plan_when_paging_each_section_then_every_section_pages_its_rows_in_pl
5757
SectionTestCase {
5858
description: "migrations",
5959
section: PlanPageSection::Migrations,
60-
expected_rows: &["old_orders"],
60+
expected_rows: &["old_orders", "totals"],
6161
},
6262
SectionTestCase {
6363
description: "column migrations",
@@ -72,7 +72,7 @@ fn given_a_plan_when_paging_each_section_then_every_section_pages_its_rows_in_pl
7272
SectionTestCase {
7373
description: "retention",
7474
section: PlanPageSection::Retention,
75-
expected_rows: &["events"],
75+
expected_rows: &["main.events"],
7676
},
7777
SectionTestCase {
7878
description: "loader reloads",
@@ -220,6 +220,24 @@ fn given_a_name_filter_when_paging_then_matching_rows_page_with_their_count() {
220220
expected_total: 2,
221221
expected_section_total: 6,
222222
},
223+
PageTestCase {
224+
description: "a retention change matches by its models",
225+
section: PlanPageSection::Retention,
226+
window: (0, 50),
227+
name: Some("EVENTS"),
228+
expected_rows: &["main.events"],
229+
expected_total: 1,
230+
expected_section_total: 1,
231+
},
232+
PageTestCase {
233+
description: "a renamed model matches by its old name",
234+
section: PlanPageSection::Renamed,
235+
window: (0, 50),
236+
name: Some("orders_v1"),
237+
expected_rows: &["old_orders"],
238+
expected_total: 1,
239+
expected_section_total: 1,
240+
},
223241
PageTestCase {
224242
description: "nothing matching is an empty page",
225243
section: PlanPageSection::FirstRun,
@@ -404,6 +422,11 @@ fn given_a_plan_when_summarised_then_sections_are_counts_and_no_row_list_is_left
404422
{"template": "already-built dates keep their old results", "count": 1, "subject": null},
405423
]),
406424
},
425+
SummaryFieldTestCase {
426+
description: "migration steps leave out the renamed model's own migration",
427+
pointer: "/migration_steps",
428+
expected_value: json!(4),
429+
},
407430
SummaryFieldTestCase {
408431
description: "--changed facts stay",
409432
pointer: "/changed/reach",
@@ -712,3 +735,88 @@ fn given_config_diffs_when_paged_whole_or_detached_then_short_ones_read_as_one_c
712735
);
713736
}
714737
}
738+
739+
#[test]
740+
fn given_renames_when_paged_or_summarised_then_rows_keep_their_marks_and_steps_leave_renames_out() {
741+
let test_cases = [
742+
SummaryFieldTestCase {
743+
description: "a renamed row names the model it was renamed from",
744+
pointer: "/renamed/origin_model",
745+
expected_value: json!("orders_v1"),
746+
},
747+
SummaryFieldTestCase {
748+
description: "a renamed row says what stays at its old name",
749+
pointer: "/renamed/old_name/action",
750+
expected_value: json!("view_only"),
751+
},
752+
SummaryFieldTestCase {
753+
description: "a rename's own migration is marked",
754+
pointer: "/migrations/0/is_rename",
755+
expected_value: json!(true),
756+
},
757+
SummaryFieldTestCase {
758+
description: "another migration is not",
759+
pointer: "/migrations/1/is_rename",
760+
expected_value: json!(false),
761+
},
762+
];
763+
let plan = full_plan();
764+
let observed = json!({
765+
"renamed": plan_section_page(&plan, PlanPageSection::Renamed, ((0, 1), None)).rows[0],
766+
"migrations": plan_section_page(&plan, PlanPageSection::Migrations, ((0, 50), None)).rows,
767+
});
768+
for test_case in test_cases {
769+
assert_eq!(
770+
observed
771+
.pointer(test_case.pointer)
772+
.cloned()
773+
.unwrap_or_default(),
774+
test_case.expected_value,
775+
"{}",
776+
test_case.description
777+
);
778+
}
779+
}
780+
781+
#[test]
782+
fn given_a_schema_retention_change_without_models_when_filtered_then_its_schema_name_matches() {
783+
let test_cases = [
784+
PageTestCase {
785+
description: "a schema's retention change matches by the schema's name",
786+
section: PlanPageSection::Retention,
787+
window: (0, 50),
788+
name: Some("ARCHIVE"),
789+
expected_rows: &["analytics_archive"],
790+
expected_total: 1,
791+
expected_section_total: 2,
792+
},
793+
PageTestCase {
794+
description: "a table's retention change still matches by its models",
795+
section: PlanPageSection::Retention,
796+
window: (0, 50),
797+
name: Some("events"),
798+
expected_rows: &["main.events"],
799+
expected_total: 1,
800+
expected_section_total: 2,
801+
},
802+
];
803+
let mut plan = full_plan();
804+
plan["retention"]
805+
.as_array_mut()
806+
.expect("retention rows")
807+
.push(json!({"scope": "schema", "name": "analytics_archive", "models": [], "desired_days": 30, "actual_days": 7, "direction": "increase"}));
808+
for test_case in test_cases {
809+
let page = plan_section_page(&plan, test_case.section, (test_case.window, test_case.name));
810+
811+
assert_eq!(
812+
(labels(&page.rows), page.total, page.section_total),
813+
(
814+
owned(test_case.expected_rows),
815+
test_case.expected_total,
816+
test_case.expected_section_total
817+
),
818+
"{}",
819+
test_case.description
820+
);
821+
}
822+
}

‎crates/sqlbuild-python-planner/src/plan_output/constants.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,5 @@ pub const IGNORE_AUDIT_FAILURES_JOBS_KEY: &str = "ignore_audit_failures_jobs";
88
pub const FULL_REFRESH_JOBS_KEY: &str = "full_refresh_jobs";
99
/// The plan JSON mapping each named job to the warehouse its nodes run on.
1010
pub const WAREHOUSE_JOBS_KEY: &str = "job_warehouses";
11+
/// The plan JSON reason of a model built under a new name.
12+
pub const RENAMED_REASON: &str = "renamed";

‎crates/sqlbuild-python-planner/src/plan_output/main/format_plan_json.rs‎

Lines changed: 48 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,22 +15,26 @@ use crate::plan_output::_helpers::plan_json::table_type_json;
1515
use crate::plan_output::_helpers::plan_json::warning_json;
1616
use crate::plan_output::_helpers::values::mapped;
1717
use crate::plan_output::constants::{
18-
CHANGED_KEY, FULL_REFRESH_JOBS_KEY, IGNORE_AUDIT_FAILURES_JOBS_KEY, WAREHOUSE_JOBS_KEY,
18+
CHANGED_KEY, FULL_REFRESH_JOBS_KEY, IGNORE_AUDIT_FAILURES_JOBS_KEY, RENAMED_REASON,
19+
WAREHOUSE_JOBS_KEY,
1920
};
2021
use crate::plan_output::main::any::any;
2122
use crate::plan_output::main::dict::dict;
2223
use crate::plan_output::main::items::items;
2324
use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyResult, Python};
2425
use pyo3::pyfunction;
25-
use pyo3::types::{PyDict, PyDictMethods, PyList};
26-
use sqlbuild_core::plan_display::constants::SYNCED_COPY_KEY;
26+
use pyo3::types::{PyDict, PyDictMethods, PyList, PyListMethods};
27+
use sqlbuild_core::plan_display::constants::{
28+
IS_RENAME_FIELD, OLD_NAME_FIELD, ORIGIN_MODEL_FIELD, SYNCED_COPY_KEY,
29+
};
2730
use sqlbuild_python_base::boundary::models::KeywordArguments;
2831
use sqlbuild_python_base::compile_outputs::main::indented_json::indented_json;
2932
use sqlbuild_python_base::records::main::fields::field::field as native_field;
3033
use sqlbuild_python_base::records::main::fields::mapping_get::mapping_get;
3134
use sqlbuild_python_project::compiled_project::constants::{
3235
JOB_FULL_REFRESH_KEY, JOB_IGNORED_AUDIT_FAILURES_KEY, JOB_WAREHOUSES_KEY,
3336
};
37+
use std::collections::HashMap;
3438

3539
/// The plan as one JSON document, `json.dumps(indent=2)`.
3640
#[pyfunction]
@@ -111,6 +115,7 @@ pub fn format_plan_json(py: Python<'_>, options: Option<Bound<'_, PyDict>>) -> P
111115
),
112116
],
113117
)?;
118+
mark_renames(&result)?;
114119
let metadata: Bound<'_, PyAny> = native_field(&plan, "metadata")?;
115120
if metadata.is_truthy()? {
116121
result.set_item("metadata", &metadata)?;
@@ -146,3 +151,43 @@ pub fn format_plan_json(py: Python<'_>, options: Option<Bound<'_, PyDict>>) -> P
146151
}
147152
indented_json(result.as_any())
148153
}
154+
155+
/// Marks renamed models' own migrations and copies their origin and old name onto the model rows.
156+
fn mark_renames(result: &Bound<'_, PyDict>) -> PyResult<()> {
157+
let (Some(models), Some(migrations)) =
158+
(result.get_item("models")?, result.get_item("migrations")?)
159+
else {
160+
return Ok(());
161+
};
162+
let mut renamed: HashMap<String, Bound<'_, PyDict>> = HashMap::new();
163+
for model in models.downcast::<PyList>()?.iter() {
164+
let model: Bound<'_, PyDict> = model.downcast_into::<PyDict>()?;
165+
let reason: Option<String> = model.get_item("reason")?.map(|r| r.extract()).transpose()?;
166+
if reason.as_deref() == Some(RENAMED_REASON)
167+
&& let Some(name) = model.get_item("name")?
168+
{
169+
renamed.insert(name.extract()?, model);
170+
}
171+
}
172+
for migration in migrations.downcast::<PyList>()?.iter() {
173+
let migration: Bound<'_, PyDict> = migration.downcast_into::<PyDict>()?;
174+
let model: Option<String> = migration
175+
.get_item("model")?
176+
.map(|m| m.extract())
177+
.transpose()?;
178+
let row: Option<&Bound<'_, PyDict>> = model.as_deref().and_then(|name| renamed.get(name));
179+
migration.set_item(IS_RENAME_FIELD, row.is_some())?;
180+
let Some(row) = row else {
181+
continue;
182+
};
183+
for field in [ORIGIN_MODEL_FIELD, OLD_NAME_FIELD] {
184+
if let Some(value) = migration.get_item(field)?
185+
&& !value.is_none()
186+
&& !row.contains(field)?
187+
{
188+
row.set_item(field, value)?;
189+
}
190+
}
191+
}
192+
Ok(())
193+
}

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ Seeds (1)
125125

126126
When query, function, or schema changes are detected, the plan lists each affected model under its own change with its backfill action. A model that calls a changed function directly appears under `Function changed` with `cause: function <name> changed`; models further downstream continue forward. See [Replay decisions](../concepts/planning/replay-decisions.md).
127127

128-
In `sqb plan --json`, each model's `reason` is its own change. A direct caller of a changed function has `"reason": "function_changed"` and a `changed_functions` list, and a renamed model whose query also changed has `"reason": "renamed"` and `"query_changed": true`. A changed model also carries the detail the text plan prints under it: `query_diff` when its query changed and the previous query is known, and `config_diff` when its config changed, each as unified diff lines, and `schema_findings` for added, removed, or retyped columns.
128+
In `sqb plan --json`, each model's `reason` is its own change. A direct caller of a changed function has `"reason": "function_changed"` and a `changed_functions` list, and a renamed model whose query also changed has `"reason": "renamed"` and `"query_changed": true`. A renamed model's row names the model it was renamed from in `origin_model` and what stays at the old name in `old_name` (the same object as its migration's; see [Model migrations](../concepts/models/migrations.md); absent when the migration has none), and that rename's own entry in `migrations` has `"is_rename": true`; every other migration has `"is_rename": false`. A changed model also carries the detail the text plan prints under it: `query_diff` when its query changed and the previous query is known, and `config_diff` when its config changed, each as unified diff lines, and `schema_findings` for added, removed, or retyped columns.
129129

130130
Warnings and maintenance notices, such as `migrate_from can be removed`, are listed in a `Warnings` section before the final `Plan complete` line. `sqb plan --json` output is unchanged.
131131

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -246,6 +246,9 @@ migration decisions using your current connection. It only inspects; nothing is
246246
- A recorded view that no longer exists is not re-created by a build; janitor records it as dropped.
247247
- Run the same SQLBuild version for `sqb janitor` as for builds. An older janitor doesn't recognise
248248
compatibility views and archives them as stale relations.
249+
- In `sqb plan --json`, a migration that moves a renamed model has `"is_rename": true`, and the renamed
250+
model's own row names its old model in `origin_model` and carries the same `old_name` object as that
251+
migration, or no `old_name` when the migration has none. Other migrations have `"is_rename": false`.
249252
- In `sqb plan --json`, each migration has an `old_name` object with `action` (`archive_and_view`,
250253
`view_only`, `live` or `none`), `view`, `reads`, `expires_at`, `column_aliases`, `grants_copied`
251254
and `reason`. The top-level `old_names` list also includes steps that resume an earlier move.

‎tests/integration/src/sqlbuild/cli/commands/main/model_migrations/_test_types.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ class AutomaticMigrationTestCase:
6060
description: str
6161
expected_migrations: tuple[tuple[str | None, str, str, str], ...] = ()
6262
expected_events: tuple[tuple[str, str, str], ...] = ()
63+
expected_renames: tuple[tuple[str, str], ...] = ()
6364
expected_reason: str = ""
6465
expected_warning: str = ""
6566

0 commit comments

Comments
 (0)