Skip to content

Commit b5bdb76

Browse files
authored
perf(compile): delta-compile dynamic columns, audit factories and model rules (#1201)
* fix(compile): name what changed when a delta compile falls back on identity Delta compile reported one message for any invocation, runtime, search path, environment, settings or module change. Name the first component that differs instead. SQLBUILD_VERIFY_DELTA_COMPILE was a tracked variable, so turning it on for one compile changed the reuse identity and made the very edit it was meant to verify fall back. It does not change compile output, so it no longer invalidates a stored compile. * perf(compile): delta-compile projects with dynamic columns, audit factories and model rules Three project shapes made every edit compile in full: - Any enforced model declaring dynamic_columns ruled out the project image. The image now keeps the analysis request's families and a delta injects them for models it does not analyse; an edit that changes a model's own families still falls back. - A delta saw only the rerun models' headers, so every audit factory attached elsewhere was reported as orphaned and the delta fell back. The image keeps each model's attached factories. - Any custom rule that had to run again ruled out the delta, so editing a model a model-subject rule reads always compiled in full. A delta now runs model-subject rules for its own models when their recorded reads are limited to those models' SQL, columns and contract, project files, and the source list (kept in full when a rule reads it). It checks the stored read sets before starting and the new ones after, and falls back otherwise. The image format moves to v9. * docs(compile): regenerate the docs skill compile page * fix(compile): fall back when a delta's custom rule lists models under a folder tree.resources_under is listed from the project's models, not the filesystem, and a delta compile holds only the models it reruns. A model-subject rule reading it for an edited model ran on the partial list and could report different findings than a full compile. Treat it as a read beyond the changed models so the delta falls back.
1 parent 1ecc1d0 commit b5bdb76

20 files changed

Lines changed: 789 additions & 53 deletions

File tree

‎crates/sqlbuild-analysis/src/assembly/analysis_session/_helpers/session.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,7 @@ impl AnalysisSession {
133133
dependency_ordered: self.dependency_ordered,
134134
cyclic: self.cyclic,
135135
legacy_models: self.legacy_models.clone(),
136+
families: self.request.dynamic_families_by_table.clone(),
136137
});
137138
let (schema_additions, analysis_names) = self.catalog.into_changes();
138139
let mut request: SessionRequest = self.request;

‎crates/sqlbuild-analysis/src/assembly/analysis_session/models.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,8 @@ pub struct RelationTables {
199199
pub cyclic: bool,
200200
/// Models whose analysis took the legacy path, which reads every relation's facts.
201201
pub legacy_models: Vec<String>,
202+
/// The request's dynamic column families by table, in request order.
203+
pub families: Vec<(String, Vec<DynamicFamily>)>,
202204
}
203205

204206
/// One inferred output column; nullability is Python's `InferredNullability` value.

‎crates/sqlbuild-cache/src/project_reuse/_helpers/attempt.rs‎

Lines changed: 41 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,13 @@ use crate::project_snapshot::models::PathStamp;
3030
const NO_STORED_COMPILE: &str = "no stored compile";
3131
const REUSE_BYPASSED: &str = "compile reuse is off";
3232
const OUTPUT_UNREADABLE: &str = "the stored output cannot be replayed";
33-
const IDENTITY_CHANGED: &str = "the invocation, runtime, environment or settings changed";
33+
const INVOCATION_CHANGED: &str = "the command invocation changed";
34+
const RUNTIME_CHANGED: &str = "the Python runtime changed";
35+
const SEARCH_PATH_CHANGED: &str = "a Python search path entry changed";
36+
const ENVIRONMENT_NAMES_CHANGED: &str = "the set of tracked environment variables changed";
37+
const ENVIRONMENT_CHANGED: &str = "a tracked environment variable changed";
38+
const SETTINGS_CHANGED: &str = "provider settings inputs changed";
39+
const MODULES_CHANGED: &str = "an imported Python module outside the project changed";
3440
const TARGET_CHANGED: &str = "the compiled target changed";
3541

3642
fn now_ns() -> i64 {
@@ -115,8 +121,8 @@ pub(crate) fn check(check: ReuseCheck<'_>) -> ReuseAttempt {
115121
attempt.restamped = restamped_paths(&inputs.project_files, &attempt.snapshot);
116122
if comparison.unchanged {
117123
attempt.delta = DeltaCandidate::Unavailable(OUTPUT_UNREADABLE);
118-
} else if !comparison.identity_unchanged {
119-
attempt.delta = DeltaCandidate::Unavailable(IDENTITY_CHANGED);
124+
} else if let Some(reason) = comparison.identity_change {
125+
attempt.delta = DeltaCandidate::Unavailable(reason);
120126
} else if !target_files_unchanged(
121127
&inputs.target_files,
122128
&attempt.project_dir,
@@ -146,21 +152,16 @@ pub(crate) fn check(check: ReuseCheck<'_>) -> ReuseAttempt {
146152
/// Whether a stored compile still matches, with the digests still valid either way.
147153
struct Comparison {
148154
unchanged: bool,
149-
identity_unchanged: bool,
155+
/// The first identity component that differs, if any.
156+
identity_change: Option<&'static str>,
150157
digests: HashMap<String, String>,
151158
digested_files: usize,
152159
}
153160

154161
/// Compare every stored identity, carrying forward digests still valid either way.
155162
fn checked(attempt: &ReuseAttempt, inputs: &StoredInputs, rules: &ReuseRules) -> Comparison {
156-
let identity_unchanged: bool = inputs.invocation_digest == attempt.identity.invocation_digest
157-
&& inputs.runtime == attempt.identity.runtime
158-
&& inputs.search_path == attempt.search_path
159-
&& tracked_environment_names(&inputs.environment_names, rules) == inputs.environment_names
160-
&& environment_digest(&inputs.environment_names, rules) == inputs.environment_digest
161-
&& settings_inputs_digest(&inputs.settings_inputs, rules) == inputs.settings_digest
162-
&& module_stamps_unchanged(&inputs.modules);
163-
let (files_unchanged, verified, digested_files) = if identity_unchanged {
163+
let identity_change: Option<&'static str> = identity_change(attempt, inputs, rules);
164+
let (files_unchanged, verified, digested_files) = if identity_change.is_none() {
164165
compare_project_files(
165166
&attempt.project_dir,
166167
&inputs.project_files,
@@ -170,7 +171,7 @@ fn checked(attempt: &ReuseAttempt, inputs: &StoredInputs, rules: &ReuseRules) ->
170171
(false, HashMap::new(), 0)
171172
};
172173
Comparison {
173-
identity_unchanged,
174+
identity_change,
174175
unchanged: files_unchanged
175176
&& target_files_unchanged(
176177
&inputs.target_files,
@@ -183,6 +184,33 @@ fn checked(attempt: &ReuseAttempt, inputs: &StoredInputs, rules: &ReuseRules) ->
183184
}
184185
}
185186

187+
/// The first stored identity component that no longer matches, in checking order.
188+
fn identity_change(
189+
attempt: &ReuseAttempt,
190+
inputs: &StoredInputs,
191+
rules: &ReuseRules,
192+
) -> Option<&'static str> {
193+
if inputs.invocation_digest != attempt.identity.invocation_digest {
194+
Some(INVOCATION_CHANGED)
195+
} else if inputs.runtime != attempt.identity.runtime {
196+
Some(RUNTIME_CHANGED)
197+
} else if inputs.search_path != attempt.search_path {
198+
Some(SEARCH_PATH_CHANGED)
199+
} else if tracked_environment_names(&inputs.environment_names, rules)
200+
!= inputs.environment_names
201+
{
202+
Some(ENVIRONMENT_NAMES_CHANGED)
203+
} else if environment_digest(&inputs.environment_names, rules) != inputs.environment_digest {
204+
Some(ENVIRONMENT_CHANGED)
205+
} else if settings_inputs_digest(&inputs.settings_inputs, rules) != inputs.settings_digest {
206+
Some(SETTINGS_CHANGED)
207+
} else if !module_stamps_unchanged(&inputs.modules) {
208+
Some(MODULES_CHANGED)
209+
} else {
210+
None
211+
}
212+
}
213+
186214
pub(crate) fn record(
187215
attempt: &ReuseAttempt,
188216
record: &CompileRecord,

‎crates/sqlbuild-python/src/bindings/_helpers/compile_command/constants.rs‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
33
use sqlbuild_core::constants::PROJECT_CONFIG_FILENAME;
44

5+
use crate::bindings::_helpers::compile_delta::driver::DELTA_VERIFY_ENV_VAR;
56
use crate::bindings::_helpers::frontier::stage_capture::CAPTURE_DIR_ENV_VAR;
67

78
/// The directory under the compiler cache directory holding the native reuse store.
@@ -16,8 +17,11 @@ pub(crate) const REUSE_DISABLE_VALUE: &str = "1";
1617
pub(crate) const REUSE_LOGGER_NAME: &str = "sqlbuild.compile.reuse";
1718
pub(crate) const TRACKED_ENVIRONMENT_PREFIXES: [&str; 2] = ["SQLBUILD_", "SQB_"];
1819
/// Sorted; tracked-prefix variables that never invalidate a stored compile.
19-
pub(crate) const UNTRACKED_ENVIRONMENT_NAMES: [&str; 2] =
20-
[CAPTURE_DIR_ENV_VAR, REUSE_DISABLE_ENV_VAR];
20+
pub(crate) const UNTRACKED_ENVIRONMENT_NAMES: [&str; 3] = [
21+
CAPTURE_DIR_ENV_VAR,
22+
REUSE_DISABLE_ENV_VAR,
23+
DELTA_VERIFY_ENV_VAR,
24+
];
2125
pub(crate) const MISSING_ENVIRONMENT_VALUE: &str = "\0missing";
2226
pub(crate) const MISSING_FILE_DIGEST: &str = "\0missing";
2327
pub(crate) const PROJECT_CONFIG_FILENAMES: [&str; 2] =

‎crates/sqlbuild-python/src/bindings/_helpers/compile_context/delta.rs‎

Lines changed: 43 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,9 @@ use std::collections::{BTreeMap, BTreeSet};
44
use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
55

66
use pyo3::prelude::PyResult;
7-
use sqlbuild_analysis::assembly::analysis_session::models::{RelationTables, SessionRequest};
7+
use sqlbuild_analysis::assembly::analysis_session::models::{
8+
DynamicFamily, RelationTables, SessionRequest,
9+
};
810
use sqlbuild_cache::project_reuse::types::ImageSections;
911
use sqlbuild_discovery::header_changes::types::HeaderKeys;
1012
use sqlbuild_rules::engine::main::retained_request::retained_request;
@@ -103,6 +105,8 @@ pub(crate) struct ImageCapture {
103105
pub(crate) environment_names: Vec<String>,
104106
/// Each discovered model's header keys, by model file path.
105107
pub(crate) header_keys: BTreeMap<String, HeaderKeys>,
108+
/// The audit factories each model file's header attaches, for files attaching any.
109+
pub(crate) audit_references: BTreeMap<String, Vec<String>>,
106110
/// Each model file's SQL lint cache key, in project model order.
107111
pub(crate) lint_model_keys: Option<Vec<(String, String)>>,
108112
/// Each other SQL file's lint cache key, in walk order.
@@ -193,6 +197,8 @@ pub(crate) struct DeltaInjection {
193197
pub(crate) coverage: Option<Vec<ModelCoverage>>,
194198
/// Every project model in project order, with the models it reads.
195199
pub(crate) coverage_order: Vec<(String, Vec<String>)>,
200+
/// The audit factories each model file's header attaches, for files attaching any.
201+
pub(crate) audit_references: BTreeMap<String, Vec<String>>,
196202
}
197203

198204
static CAPTURE: Mutex<Option<ImageCapture>> = Mutex::new(None);
@@ -328,17 +334,9 @@ pub(crate) fn injected_fact_digest(key: &[String]) -> Option<String> {
328334

329335
/// Give a delta's analysis session the whole project's relation facts and analysis order.
330336
pub(crate) fn inject_relations(mut request: SessionRequest) -> pyo3::PyResult<SessionRequest> {
331-
if !request.dynamic_families_by_table.is_empty() {
332-
capture_unsupported("a model declares dynamic column families");
333-
}
334337
let Some(injection) = injection() else {
335338
return Ok(request);
336339
};
337-
if !request.dynamic_families_by_table.is_empty() {
338-
return Err(delta_unsupported(
339-
"a model declares dynamic column families",
340-
));
341-
}
342340
let relations: &RelationTables = &injection.relations;
343341
request.forced_order = Some((relations.dependency_ordered, relations.cyclic));
344342
request.retain_relations = true;
@@ -347,6 +345,16 @@ pub(crate) fn inject_relations(mut request: SessionRequest) -> pyo3::PyResult<Se
347345
.iter()
348346
.map(|model| model.name.as_str())
349347
.collect();
348+
let rerun: BTreeSet<&str> = analysed
349+
.iter()
350+
.copied()
351+
.chain(injection.rerun_names.iter().map(String::as_str))
352+
.collect();
353+
request.dynamic_families_by_table = injected_families(
354+
&request.dynamic_families_by_table,
355+
&relations.families,
356+
&rerun,
357+
)?;
350358
for (table, injected) in [
351359
(&mut request.column_types, &relations.types),
352360
(&mut request.column_nullability, &relations.nullability),
@@ -363,6 +371,32 @@ pub(crate) fn inject_relations(mut request: SessionRequest) -> pyo3::PyResult<Se
363371
Ok(request)
364372
}
365373

374+
/// The stored compile's dynamic column families, when no rerun model changed its own.
375+
fn injected_families(
376+
current: &[(String, Vec<DynamicFamily>)],
377+
stored: &[(String, Vec<DynamicFamily>)],
378+
rerun: &BTreeSet<&str>,
379+
) -> pyo3::PyResult<Vec<(String, Vec<DynamicFamily>)>> {
380+
let families_of = |table: &[(String, Vec<DynamicFamily>)], name: &str| {
381+
table
382+
.iter()
383+
.find(|(table, _)| table == name)
384+
.map(|(_, families)| families.clone())
385+
};
386+
let changed: bool = current
387+
.iter()
388+
.any(|(name, _)| !rerun.contains(name.as_str()))
389+
|| rerun
390+
.iter()
391+
.any(|name| families_of(current, name) != families_of(stored, name));
392+
if changed {
393+
return Err(delta_unsupported(
394+
"a model's dynamic column families changed",
395+
));
396+
}
397+
Ok(stored.to_vec())
398+
}
399+
366400
/// What a delta compile's reuse record keeps of the stored compile it patched.
367401
#[derive(Debug, Default)]
368402
pub(crate) struct RecordMerge {

‎crates/sqlbuild-python/src/bindings/_helpers/compile_delta/driver.rs‎

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ use crate::bindings::_helpers::compile_driver::command::{
3434
};
3535
use crate::bindings::_helpers::compile_outputs::reuse::NativeReuseAttempt;
3636
use crate::bindings::_helpers::records::fields::field as native_field;
37+
use crate::bindings::_helpers::rules::custom_evaluation::stored_custom_rule_refusal;
3738
use sqlbuild_discovery::header_changes::main::custom_rule_reads::custom_rule_reads;
3839
use sqlbuild_discovery::header_changes::main::header_change::header_change;
3940
use sqlbuild_discovery::header_changes::models::HeaderChange;
@@ -57,6 +58,8 @@ const UNKNOWN_SOURCE: &str = "unknown source";
5758
const SOURCE_KIND: &str = "source";
5859
/// The upstream kind of a model a model reads.
5960
const MODEL_KIND: &str = "model";
61+
/// The custom-rule fact listing every project source.
62+
const PROJECT_SOURCES_FACT: &str = "project.sources";
6063
const TESTS_ROOT: &str = "tests/";
6164
const STDERR: &str = "stderr";
6265
const SQL_SUFFIX: &str = ".sql";
@@ -321,6 +324,7 @@ fn lent_injection(
321324
.collect(),
322325
lint_model_keys: image.lint_model_keys.take(),
323326
lint_file_keys: image.lint_file_keys.take(),
327+
audit_references: image.audit_references.clone(),
324328
..DeltaInjection::default()
325329
};
326330
(image, injection)
@@ -588,6 +592,17 @@ fn prepared(
588592
Ok(found) => found,
589593
Err(reason) => return Ok(Err(reason)),
590594
};
595+
let all_sources: bool = gate
596+
.fact_digests
597+
.iter()
598+
.any(|(key, _)| lists_all_sources(key));
599+
if let Some(reason) = stored_custom_rule_refusal(
600+
&project_dir,
601+
(&rerun.model_paths, &rerun.changed),
602+
all_sources,
603+
) {
604+
return Ok(Err(reason.to_owned()));
605+
}
591606
let limit: usize = MAX_RERUN_MODELS.max(gate.models.len() * MAX_RERUN_PERCENT / 100);
592607
if rerun.models.len() > limit {
593608
return Ok(Err(TOO_MANY_MODELS.to_owned()));
@@ -647,7 +662,11 @@ fn rounds<'py>(
647662
{
648663
return stopped("the stored compile reports problems in files the edit leaves alone");
649664
}
650-
rerun.sources = sources_read(&image, &rerun);
665+
rerun.sources = if reads_all_sources(&injection) {
666+
None
667+
} else {
668+
sources_read(&image, &rerun)
669+
};
651670
let injection_for_round: DeltaInjection =
652671
for_rerun(injection, &rerun, rerun.sources.clone());
653672
let (outcome, back) = round(request, attempt, (injection_for_round, &rerun))?;
@@ -705,6 +724,19 @@ fn rounds<'py>(
705724
stopped("changed shapes reach too far")
706725
}
707726

727+
/// Whether a custom rule read the project's whole source list, which a delta must then keep.
728+
fn reads_all_sources(injection: &DeltaInjection) -> bool {
729+
injection
730+
.fact_digests
731+
.keys()
732+
.any(|key| lists_all_sources(key))
733+
}
734+
735+
/// Whether a custom-rule fact key is the project's whole source list.
736+
fn lists_all_sources(key: &[String]) -> bool {
737+
matches!(key, [name] if name == PROJECT_SOURCES_FACT)
738+
}
739+
708740
/// The sources the rerun models, functions, audits and rerun SQL tests read; `None` for all.
709741
fn sources_read(image: &ProjectImage, rerun: &Rerun) -> Option<BTreeSet<String>> {
710742
if image.diagnostics.iter().any(outside_resources) {
@@ -1162,6 +1194,10 @@ fn patched_image(base: ProjectImage, rerun: &Rerun, capture: ImageCapture) -> Pr
11621194
image.coverage = capture.coverage;
11631195
}
11641196
image.header_keys.extend(capture.header_keys);
1197+
image
1198+
.audit_references
1199+
.retain(|path, _| !rerun.model_paths.contains(path));
1200+
image.audit_references.extend(capture.audit_references);
11651201
let mut pinned: BTreeSet<String> = std::mem::take(&mut image.pinned_sources)
11661202
.into_iter()
11671203
.collect();

0 commit comments

Comments
 (0)