Skip to content

Commit e6b04c2

Browse files
committed
fix(build): close the maintained-model hold gaps and two rebuild residuals
- A model created during a build now runs the same earliest-hold check as acquire once its pause is recorded, so two builds creating the same model no longer both hold it. Our create made the model active, so it is suspended either way; the build holding it resumes it at its end, and a build that backed out resumes it itself at its end when no other build holds it, so a model never stays suspended after both back out. - When the losing build cannot close its own pause, the A718 error now names that failure too instead of being replaced by it. - Pauses are ordered by the warehouse clock when each insert starts, but an insert commits later. A build now also backs out for a rival that paused within the time its own insert took, since that rival may have read the table before our pause committed, and when a takeover closed its pause before it read the table back. Two builds that start at the same moment can both stop with A718; neither builds alongside the other. - A snapshot's first build and full rebuild turn change tracking on when a maintained model reads it, as table builds already do (new optional render_track_changes adapter method; Snowflake runs ALTER TABLE ... SET CHANGE_TRACKING = TRUE). - The planner's change check drops refresh_mode when it is the default, on both the recorded and the current side, so writing out or removing the default no longer recreates the model. Stored-status hashes are unchanged.
1 parent 8fbfce8 commit e6b04c2

28 files changed

Lines changed: 663 additions & 48 deletions

File tree

‎crates/sqlbuild-analysis/src/planner_changes/main/version_identity_payload.rs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,9 @@
33
use crate::planner_changes::models::IdentityPayload;
44
use sqlbuild_core::fingerprint_identity::main::decoded_object::decoded_object;
55
use sqlbuild_core::fingerprint_identity::main::normalize_version_identity_metadata::normalize_version_identity_metadata;
6+
use sqlbuild_core::fingerprint_identity::main::without_default_refresh_mode::without_default_refresh_mode;
67

7-
/// The metadata without audit-gate, migration and declared-column entries; with `without_functions`, also without function hashes.
8+
/// The metadata without audit-gate, migration, declared-column and default refresh-mode entries; with `without_functions`, also without function hashes.
89
pub fn version_identity_payload(
910
metadata_json: Option<&str>,
1011
without_functions: bool,
@@ -17,7 +18,7 @@ pub fn version_identity_payload(
1718
Ok(None) => IdentityPayload::Absent,
1819
Ok(Some(mut object)) => {
1920
normalize_version_identity_metadata(&mut object, without_functions);
20-
IdentityPayload::Object(object)
21+
IdentityPayload::Object(without_default_refresh_mode(object))
2122
}
2223
}
2324
}

‎crates/sqlbuild-core/src/fingerprint_identity/constants.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,12 @@ pub const AUDIT_GATE_METADATA_KEY: &str = "audit_gate";
44
pub const MIGRATION_FINGERPRINT_METADATA_KEY: &str = "migration_fingerprint";
55
pub const DECLARED_COLUMNS_METADATA_KEY: &str = "declared_columns_hash";
66
pub const LOCAL_FUNCTION_HASHES_METADATA_KEY: &str = "local_function_hashes";
7+
/// Metadata key of the model config entries in version identity.
8+
pub const CONFIG_METADATA_KEY: &str = "config";
9+
/// Config key of a maintained model's refresh mode.
10+
pub const REFRESH_MODE_CONFIG_KEY: &str = "refresh_mode";
11+
/// The refresh mode of a maintained model that declares none; writing it out changes nothing.
12+
pub const DEFAULT_REFRESH_MODE: &str = "incremental";
713
/// The fingerprint dialect when a project names none.
814
pub(crate) const GENERIC_FINGERPRINT_DIALECT: &str = "generic";
915
/// Python's `NATIVE_DIALECT_ALIASES`.

‎crates/sqlbuild-core/src/fingerprint_identity/main/mod.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,4 @@ pub mod normalize_version_identity_metadata;
66
pub mod python_equal;
77
pub mod recorded_model_local_hash;
88
pub mod stable_identity_hash;
9+
pub mod without_default_refresh_mode;
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
//! Drop a written-out default refresh mode, which plans the same as none.
2+
3+
use serde_json::{Map, Value};
4+
5+
use crate::fingerprint_identity::constants::{
6+
CONFIG_METADATA_KEY, DEFAULT_REFRESH_MODE, REFRESH_MODE_CONFIG_KEY,
7+
};
8+
9+
pub fn without_default_refresh_mode(mut object: Map<String, Value>) -> Map<String, Value> {
10+
if let Some(Value::Object(config)) = object.get_mut(CONFIG_METADATA_KEY)
11+
&& config.get(REFRESH_MODE_CONFIG_KEY) == Some(&Value::from(DEFAULT_REFRESH_MODE))
12+
{
13+
config.shift_remove(REFRESH_MODE_CONFIG_KEY);
14+
}
15+
object
16+
}

‎crates/sqlbuild-core/src/fingerprint_identity/mod.rs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,6 @@
22
33
pub mod constants;
44
pub mod main;
5+
6+
#[cfg(test)]
7+
mod tests;
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
#[path = "tests/test_fingerprint_identity.rs"]
2+
mod fingerprint_identity;
3+
#[path = "tests/test_types.rs"]
4+
mod test_types;
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
use crate::fingerprint_identity::main::canonical_text::canonical_text;
2+
use crate::fingerprint_identity::main::decoded_object::decoded_object;
3+
use crate::fingerprint_identity::main::model_local_identity_hash::model_local_identity_hash;
4+
use crate::fingerprint_identity::main::recorded_model_local_hash::recorded_model_local_hash;
5+
use crate::fingerprint_identity::tests::test_types::RecordedLocalHashTestCase;
6+
7+
#[test]
8+
fn given_recorded_metadata_when_hashing_then_it_matches_the_config_as_written() {
9+
let test_cases = [
10+
RecordedLocalHashTestCase {
11+
description: "a written-out default refresh mode stays in the hash",
12+
metadata_json: r#"{"config":{"materialized":"maintained","refresh_mode":"incremental"},"execution_signature":{},"local_function_hashes":{},"model_name":"daily_totals"}"#,
13+
expected_hashed_json: r#"{"config":{"materialized":"maintained","refresh_mode":"incremental"},"execution_signature":{},"local_function_hashes":{},"model_name":"daily_totals"}"#,
14+
},
15+
RecordedLocalHashTestCase {
16+
description: "a model that declares no refresh mode",
17+
metadata_json: r#"{"config":{"materialized":"maintained"},"execution_signature":{},"local_function_hashes":{},"model_name":"daily_totals"}"#,
18+
expected_hashed_json: r#"{"config":{"materialized":"maintained"},"execution_signature":{},"local_function_hashes":{},"model_name":"daily_totals"}"#,
19+
},
20+
];
21+
22+
for test_case in test_cases {
23+
assert_eq!(
24+
recorded_model_local_hash("query_hash", test_case.metadata_json),
25+
Some(model_local_identity_hash(
26+
"query_hash",
27+
&canonical_text(
28+
decoded_object(test_case.expected_hashed_json)
29+
.expect("metadata is JSON")
30+
.expect("metadata is an object")
31+
)
32+
)),
33+
"{}",
34+
test_case.description
35+
);
36+
}
37+
}
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pub(super) struct RecordedLocalHashTestCase {
2+
pub(super) description: &'static str,
3+
pub(super) metadata_json: &'static str,
4+
pub(super) expected_hashed_json: &'static str,
5+
}

‎crates/sqlbuild-python-planner/src/planner_phases/_helpers/outputs/maintained.rs‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ use sqlbuild_analysis::maintained_refresh::models::{
1616
LiveDynamicTable, MaintainedRead, MaintainedSettings, RefreshLag,
1717
};
1818
use sqlbuild_analysis::planner_scope::constants::MODEL;
19+
use sqlbuild_core::fingerprint_identity::constants::DEFAULT_REFRESH_MODE;
1920
use sqlbuild_model_config::model_validation::main::fixed_duration_seconds::fixed_duration_seconds;
2021
use sqlbuild_python_base::records::main::construct::enum_member::enum_member;
2122
use sqlbuild_python_base::records::main::fields::field::field as native_field;
@@ -35,8 +36,6 @@ const VIEW: &str = "view";
3536
const KEEP_MAINTAINED: &str = "keep_maintained";
3637
/// The scheduling state Snowflake lists for a dynamic table whose refreshes are paused.
3738
const SUSPENDED: &str = "SUSPENDED";
38-
/// The refresh mode of a maintained model that declares none, so Snowflake never picks one quietly.
39-
const DEFAULT_REFRESH_MODE: &str = "incremental";
4039
/// Actions that recreate the relation, which reinitialises every dynamic table reading it.
4140
const REPLACING_ACTIONS: [&str; 3] = ["create_table", "create_view", "create_maintained"];
4241

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -181,7 +181,8 @@ The build role needs these grants:
181181

182182
Dynamic tables refresh from changes, which needs change tracking on every table they read:
183183

184-
- A table model that a maintained model reads, directly or through views, is built with change tracking on.
184+
- A table or snapshot model that a maintained model reads, directly or through views, is built with
185+
change tracking on.
185186
- For a source table, Snowflake turns change tracking on when the build role owns the table.
186187
Otherwise its owner runs `ALTER TABLE <source> SET CHANGE_TRACKING = TRUE`.
187188

0 commit comments

Comments
 (0)