Skip to content

Commit a411860

Browse files
authored
fix: scenario, SQL test and dialect fixes for fixtures, checks and truth tests (#1147)
* fix: fill each partial fixture column once across identifier case Required fixture columns were deduplicated by their authored spelling, so a column read as STATUS through a SELECT * import CTE and required again as the declared status was completed twice. Key required columns by their casefolded name, matching the metadata lookup, and complete each declared column at most once. Fixes CHI-630. * fix: reject IS [NOT] TRUE/FALSE on Snowflake and SQL Server at compile Both warehouses reject truth-value tests the SQL parser accepts, so models, audits, SQL tests and hooks compiled cleanly and failed at execution. Report a blocking B103 at the IS keyword of the authored resource with a null-safe rewrite. DuckDB, MotherDuck, PostgreSQL, BigQuery and Databricks accept them. Fixes CHI-631. * fix: create the SQL Server source freshness table with an existence check SQL Server has no CREATE TABLE IF NOT EXISTS. Guard the freshness table like the other state tables and narrow its indexed key columns from NVARCHAR(MAX), which SQL Server cannot index, to NVARCHAR(450). Fixes CHI-518. * fix: casefold names in explicit-reference relation matching Relation and temporary-table names were lowercased rather than casefolded, so names such as Straße and STRASSE did not match as they do in Python and the runtime relation guard. Fixes CHI-595. * fix: report built-in audits on missing columns once at the model header A built-in audit attached to a column the model does not output was bound as resource SQL and reported once per column reference at a made-up audits/generic/<name>.sql path. Report one B002 per audit at its declaration in MODEL() instead. Project audits that override a built-in keep their own file. Fixes CHI-476. * fix: diagnose scenario expected columns before running and report unrun comparisons A scenario expected CTE listing fewer columns than its model failed in the warehouse with a raw set-operation error, and the result still reported 0 mismatched with actual=0 expected=0. Reject an expected CTE that omits model output columns or lists columns the model does not output with S512 at planning time, and report a comparison that did not run with an ERROR status and no counts; --json adds comparison_status and reports null counts. Refs CHI-633. * feat: compare scenario expected output on its listed columns A scenario __expected__ CTE had to list every column its model outputs, so adding a column to a model broke every scenario covering it. Compare the listed columns by name in both directions, as SQL unit tests do, and report missing rows alongside mismatched ones. SCENARIO (expected_columns "all") restores the requirement that every model column be listed, diagnosed with S512 before the scenario runs. Fixes CHI-633. * fix: let scenario checks read helper CTEs and report unrun assertions Scenario assertions and expected output ran without the file's helper CTEs, so reading one failed in the warehouse with a missing-table error, and the summary still reported 0 failing rows. Place the helpers a query reads, with the helpers and fixtures they read, ahead of it in dependency order through the SQL-test helper scope, build models an assertion reads through a helper, and give fixtures only the helpers they read. An assertion that did not run reports ERROR without a count; --json adds query_status and a null count. Fixes CHI-634. * fix: grow the stack while keying deeply nested macro context values Keying a macro context value recurses once per nesting level up to the 900-level limit. In builds with large frames that recursion overflowed an 8 MB stack and crashed the process instead of reporting the value as unstorable. Each level now runs through stacker::maybe_grow, so the depth limit is reached on any stack size. * test: fail make test when a worker crashes instead of restarting it A crashed xdist worker was replaced and the run then never finished. With --max-worker-restart=0 the crash is reported as a failure and the run ends. * fix: reject scenario checks that redefine a helper CTE name at compile Scenario checks now receive the file's helper CTEs, so an expected or assertion CTE defining its own CTE with a helper's name failed in the warehouse with a duplicate CTE name. Pass the checks into the helper scope so the SQL-test shadowing check reports it at compile time. * test: expect scenario fixtures to carry only the helper CTEs they read CHI-634 gives each fixture only the helpers it reads, so the planner fixture test no longer expects every fixture to carry every helper.
1 parent f1a5b79 commit a411860

97 files changed

Lines changed: 3127 additions & 185 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Makefile‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ test:
5151
@mkdir -p /tmp/opencode
5252
@log="/tmp/opencode/test-$$(date +%Y%m%d-%H%M%S).log"; \
5353
echo "Logging to $$log"; \
54-
env PYTHONUNBUFFERED=1 SQLBUILD_CONCURRENCY=$(SQLBUILD_CONCURRENCY) uv run pytest tests/unit tests/integration -m "not real_warehouse and not dbt" -vv --color=yes -n auto --dist loadfile 2>&1 | tee "$$log"; \
54+
env PYTHONUNBUFFERED=1 SQLBUILD_CONCURRENCY=$(SQLBUILD_CONCURRENCY) uv run pytest tests/unit tests/integration -m "not real_warehouse and not dbt" -vv --color=yes -n auto --dist loadfile --max-worker-restart=0 2>&1 | tee "$$log"; \
5555
status=$${PIPESTATUS[0]}; \
5656
echo "TEST_EXIT=$$status (log: $$log)" | tee -a "$$log"; \
5757
exit $$status
@@ -139,7 +139,9 @@ E2E_DUCKDB_CLI_DATA_PATHS := \
139139
tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_promotion.py \
140140
tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_concurrency.py \
141141
tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_interrupts.py \
142-
tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_empty_fixture.py
142+
tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_empty_fixture.py \
143+
tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_expected_columns.py \
144+
tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_helper_ctes.py
143145

144146
E2E_DUCKDB_CLI_PATHS := \
145147
tests/e2e/scripts/cli_preview \

‎crates/sqlbuild-analysis/src/compiler/_helpers/sql_tests/helper_scope.rs‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ pub(crate) struct HelperScope {
3333
}
3434

3535
/// One mock CTE other test CTEs can read by its generated name.
36+
#[derive(Debug)]
3637
pub(crate) struct ScopeMock {
3738
pub(crate) generated_name: String,
3839
pub(crate) mock_name: String,
@@ -41,6 +42,7 @@ pub(crate) struct ScopeMock {
4142
}
4243

4344
/// One test-defined helper: its authored name and body, and its top-level CTE name and body.
45+
#[derive(Debug)]
4446
struct ScopeHelper {
4547
name: String,
4648
original: String,
@@ -51,6 +53,7 @@ struct ScopeHelper {
5153
}
5254

5355
/// A helper whose relation references were resolved like an assertion's, after the model chain.
56+
#[derive(Debug)]
5457
pub(crate) struct ResolvedHelper {
5558
/// The helper body with every reference replaced by the CTE that stands in for it.
5659
pub(crate) sql: String,
@@ -67,7 +70,7 @@ pub(crate) struct ScopeDependency<'a> {
6770
}
6871

6972
/// Where helper CTEs live in the rendered test query.
70-
#[derive(Clone, Copy, Default, PartialEq, Eq)]
73+
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
7174
enum HelperPlacement {
7275
/// Each top-level test CTE carries the helpers it reads in its own WITH.
7376
#[default]
@@ -95,7 +98,7 @@ enum ScopeNode {
9598
}
9699

97100
/// Name references between one test's helper and mock CTEs, tokenized once per test.
98-
#[derive(Default)]
101+
#[derive(Debug, Default)]
99102
pub(crate) struct ScopeGraph {
100103
placement: HelperPlacement,
101104
helpers: Vec<ScopeHelper>,
@@ -386,6 +389,17 @@ impl ScopeGraph {
386389
self.collect_scope(self.closure(&identifier_tokens(sql, patterns), true))
387390
}
388391

392+
/// Top-level CTEs `sql` reads, transitively and dependencies first; mocks only when asked.
393+
pub(crate) fn reader_ctes(
394+
&self,
395+
sql: &str,
396+
patterns: &SqlTestPatterns,
397+
through_mocks: bool,
398+
) -> Vec<(String, String)> {
399+
self.collect_scope(self.closure(&identifier_tokens(sql, patterns), through_mocks))
400+
.ctes
401+
}
402+
389403
fn collect_scope(&self, nodes: Vec<ScopeNode>) -> HelperScope {
390404
let mut scope = HelperScope {
391405
ctes: Vec::new(),

‎crates/sqlbuild-analysis/src/compiler/_helpers/sql_tests/mod.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,4 +21,5 @@ pub mod reference_calls;
2121
pub mod relation_markers;
2222
pub mod rendering;
2323
pub mod scenario_extraction;
24+
pub(crate) mod scenario_scope;
2425
pub(crate) mod simple_template;

‎crates/sqlbuild-analysis/src/compiler/_helpers/sql_tests/planning.rs‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -63,8 +63,8 @@ pub(crate) const MOCK_CTE_PREFIXES: [&str; 5] = [
6363
DBT_REF_PREFIX,
6464
TABLE_FUNCTION_PREFIX,
6565
];
66-
const EXPECTED_PREFIX: &str = "__expected__";
67-
const ASSERT_PREFIX: &str = "__assert__";
66+
pub(crate) const EXPECTED_PREFIX: &str = "__expected__";
67+
pub(crate) const ASSERT_PREFIX: &str = "__assert__";
6868
const REF_FUNCTION: &str = "__ref";
6969
const SOURCE_FUNCTION: &str = "__source";
7070
const SEED_FUNCTION: &str = "__seed";
@@ -102,7 +102,7 @@ struct ProjectContext {
102102
reserved_ctes: Arc<Mutex<HashMap<String, Option<String>>>>,
103103
}
104104

105-
#[derive(Clone)]
105+
#[derive(Clone, Debug)]
106106
pub(crate) struct SqlTestPatterns {
107107
pub(crate) reference: Regex,
108108
pub(crate) source: Regex,
@@ -116,7 +116,7 @@ pub(crate) struct SqlTestPatterns {
116116
}
117117

118118
impl SqlTestPatterns {
119-
fn new(lexical: LexicalSyntax) -> Result<Self, String> {
119+
pub(crate) fn new(lexical: LexicalSyntax) -> Result<Self, String> {
120120
Ok(Self {
121121
reference: compile_pattern(r#"(?i)__ref\(\"([^\"]+)\"\)"#)?,
122122
source: compile_pattern(r#"(?i)__source\(\"([^\"]+)\"\)"#)?,

‎crates/sqlbuild-analysis/src/compiler/_helpers/sql_tests/rendering.rs‎

Lines changed: 26 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -696,9 +696,26 @@ fn pruned_statement(statement: &str, dialect: SliceDialect) -> String {
696696

697697
/// Columns compared for an expected-output step: the listed expected columns, else every column.
698698
fn compared_projection(step: &SqlTestChainStep) -> String {
699-
step.expected_columns
700-
.as_ref()
701-
.map_or_else(|| "*".to_string(), |columns| columns.join(", "))
699+
compared_columns_projection(step.expected_columns.as_deref())
700+
}
701+
702+
/// The projection comparing `columns` by name, or every column positionally without a list.
703+
pub(crate) fn compared_columns_projection(columns: Option<&[String]>) -> String {
704+
columns.map_or_else(|| "*".to_string(), |columns| columns.join(", "))
705+
}
706+
707+
/// Row counts of `actual` and `expected` and of the rows each has that the other lacks.
708+
pub(crate) fn comparison_counts_sql(
709+
(actual, expected): (&str, &str),
710+
projection: &str,
711+
set_difference_operator: &str,
712+
) -> String {
713+
format!(
714+
"(SELECT COUNT(*) FROM {actual}) AS actual_count, \
715+
(SELECT COUNT(*) FROM {expected}) AS expected_count, \
716+
(SELECT COUNT(*) FROM (SELECT {projection} FROM {actual} {set_difference_operator} SELECT {projection} FROM {expected}) AS __sqlbuild_mismatch) AS unexpected_count, \
717+
(SELECT COUNT(*) FROM (SELECT {projection} FROM {expected} {set_difference_operator} SELECT {projection} FROM {actual}) AS __sqlbuild_missing) AS missing_count"
718+
)
702719
}
703720

704721
/// Render one planned test's comparison query, lifting authored CTE text verbatim.
@@ -753,16 +770,14 @@ pub(crate) fn render_comparison_sql(request: &RenderRequest) -> Result<String, S
753770
};
754771
let expected_sql = cte_state.expected_step_sql(step, expected_input, enabled)?;
755772
comparison_ctes.push((expected_cte.clone(), expected_sql));
756-
let projection = compared_projection(step);
757773
select_parts.push(format!(
758-
"SELECT {step_index} AS step_index, '{}' AS model_name, \
759-
(SELECT COUNT(*) FROM {actual_cte}) AS actual_count, \
760-
(SELECT COUNT(*) FROM {expected_cte}) AS expected_count, \
761-
(SELECT COUNT(*) FROM (SELECT {projection} FROM {actual_cte} {} SELECT {projection} FROM {expected_cte}) AS __sqlbuild_mismatch) AS unexpected_count, \
762-
(SELECT COUNT(*) FROM (SELECT {projection} FROM {expected_cte} {} SELECT {projection} FROM {actual_cte}) AS __sqlbuild_missing) AS missing_count",
774+
"SELECT {step_index} AS step_index, '{}' AS model_name, {}",
763775
escape_sql_string(&step.model_name),
764-
request.set_difference_operator,
765-
request.set_difference_operator,
776+
comparison_counts_sql(
777+
(&actual_cte, &expected_cte),
778+
&compared_projection(step),
779+
&request.set_difference_operator,
780+
),
766781
));
767782
}
768783

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
//! Scenario helper and fixture CTEs indexed through the SQL-test helper scope graph.
2+
3+
use std::collections::BTreeMap;
4+
5+
use crate::compiler::_helpers::sql_tests::cte_slices::SliceDialect;
6+
use crate::compiler::_helpers::sql_tests::helper_scope::{ScopeGraph, ScopeRequest};
7+
use crate::compiler::_helpers::sql_tests::planning::{
8+
ASSERT_PREFIX, DBT_REF_PREFIX, EXPECTED_PREFIX, REF_PREFIX, SEED_PREFIX, SOURCE_PREFIX,
9+
SqlTestPatterns, TestFixtures,
10+
};
11+
use crate::compiler::models::{ScenarioHelperScope, ScenarioScopeInput, SqlTestCte};
12+
13+
const ERROR_PREFIX: &str = "compile_input:";
14+
const FIXTURE_PREFIXES: [&str; 4] = [REF_PREFIX, SOURCE_PREFIX, SEED_PREFIX, DBT_REF_PREFIX];
15+
16+
/// Index every helper, including ones that call references, and every fixture as a mock.
17+
pub(crate) fn helper_scope(input: &ScenarioScopeInput<'_>) -> Result<ScenarioHelperScope, String> {
18+
let patterns = SqlTestPatterns::new(input.lexical.clone()).map_err(plain)?;
19+
let fixtures = scenario_fixtures(input.ctes, input.checks);
20+
let read_helper_names: Vec<String> = fixtures
21+
.helpers
22+
.iter()
23+
.map(|cte| cte.name.clone())
24+
.collect();
25+
let graph = ScopeGraph::new(&ScopeRequest {
26+
fixtures: &fixtures,
27+
patterns: &patterns,
28+
file_label: input.file_label,
29+
rename_dialect: None,
30+
flat_dialect: true,
31+
slice_dialect: SliceDialect::new(input.dialect),
32+
read_helper_names: &read_helper_names,
33+
})
34+
.map_err(plain)?;
35+
Ok(ScenarioHelperScope { graph, patterns })
36+
}
37+
38+
fn scenario_fixtures(ctes: &[(String, String)], checks: &[(String, String)]) -> TestFixtures {
39+
let mut fixtures = TestFixtures {
40+
mock_refs: BTreeMap::new(),
41+
mock_sources: BTreeMap::new(),
42+
mock_seeds: BTreeMap::new(),
43+
mock_dbt_refs: BTreeMap::new(),
44+
mock_table_functions: BTreeMap::new(),
45+
helpers: Vec::new(),
46+
scope: ScopeGraph::default(),
47+
expected: BTreeMap::new(),
48+
assertions: Vec::new(),
49+
};
50+
for (name, sql) in ctes {
51+
let fixture: Option<(&str, &str)> = fixture_key(name);
52+
let group = match fixture {
53+
Some((REF_PREFIX, _)) => &mut fixtures.mock_refs,
54+
Some((SOURCE_PREFIX, _)) => &mut fixtures.mock_sources,
55+
Some((SEED_PREFIX, _)) => &mut fixtures.mock_seeds,
56+
Some(_) => &mut fixtures.mock_dbt_refs,
57+
None => {
58+
fixtures.helpers.push(SqlTestCte {
59+
name: name.clone(),
60+
sql_body: sql.clone(),
61+
});
62+
continue;
63+
}
64+
};
65+
if let Some((_, key)) = fixture {
66+
group.insert(key.to_owned(), sql.clone());
67+
}
68+
}
69+
for (name, sql) in checks {
70+
if let Some(model) = name.strip_prefix(EXPECTED_PREFIX) {
71+
fixtures.expected.insert(model.to_owned(), sql.clone());
72+
} else if let Some(assertion) = name.strip_prefix(ASSERT_PREFIX) {
73+
fixtures
74+
.assertions
75+
.push((assertion.to_owned(), sql.clone()));
76+
}
77+
}
78+
fixtures
79+
}
80+
81+
/// The fixture prefix and name of a fixture CTE, or `None` for a helper.
82+
fn fixture_key(name: &str) -> Option<(&'static str, &str)> {
83+
for prefix in FIXTURE_PREFIXES {
84+
if let Some(key) = name.strip_prefix(prefix) {
85+
return Some((prefix, key));
86+
}
87+
}
88+
None
89+
}
90+
91+
fn plain(message: String) -> String {
92+
message
93+
.strip_prefix(ERROR_PREFIX)
94+
.map_or_else(|| message.clone(), str::to_owned)
95+
}

‎crates/sqlbuild-analysis/src/compiler/main/mod.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,11 @@
11
pub mod function_type_validation;
22
pub mod missing_expected_columns;
3+
pub mod scenario_expected_column_message;
4+
pub mod scenario_expected_columns;
35
pub mod scenario_expected_comparison_sql;
6+
pub mod scenario_helper_scope;
7+
pub mod scenario_reader_ctes;
8+
pub mod scenario_reader_sql;
49
pub mod sql_scenario_classification;
510
pub mod sql_test_assembly;
611
pub mod sql_test_chains;
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
//! The pre-execution error for a scenario expected CTE whose columns do not fit its model.
2+
3+
use crate::compiler::models::ScenarioExpectedColumns;
4+
5+
/// Why the expected CTE cannot be compared with its model, or `None` when it can.
6+
#[must_use]
7+
pub fn scenario_expected_column_message(
8+
scenario_name: &str,
9+
model_name: &str,
10+
columns: &ScenarioExpectedColumns,
11+
all_columns: bool,
12+
) -> Option<String> {
13+
let mut problems: Vec<String> = Vec::new();
14+
if all_columns && !columns.unlisted.is_empty() {
15+
problems.push(format!(
16+
"does not list columns that model '{model_name}' outputs: {}",
17+
columns.unlisted.join(", ")
18+
));
19+
}
20+
if !columns.unknown.is_empty() {
21+
problems.push(format!(
22+
"lists columns that model '{model_name}' does not output: {}",
23+
columns.unknown.join(", ")
24+
));
25+
}
26+
if problems.is_empty() {
27+
return None;
28+
}
29+
Some(format!(
30+
"scenario '{scenario_name}' expected output __expected__{model_name} {}",
31+
problems.join("; and ")
32+
))
33+
}
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
//! Compare a scenario `__expected__` CTE's listed columns with its model's output columns.
2+
3+
use crate::compiler::_helpers::sql_tests::expected_columns::{
4+
expected_columns, normalized_identifier,
5+
};
6+
use crate::compiler::_helpers::sql_tests::rendering::render_dialect;
7+
use crate::compiler::models::ScenarioExpectedColumns;
8+
9+
/// The columns an expected CTE lists and how they differ from the known model output.
10+
#[must_use]
11+
pub fn scenario_expected_columns(
12+
expected_sql: &str,
13+
dialect: Option<&str>,
14+
model_columns: Option<&[String]>,
15+
) -> ScenarioExpectedColumns {
16+
let listed: Option<Vec<String>> = expected_columns(expected_sql, &render_dialect(dialect));
17+
let (Some(listed_columns), Some(model_columns)) = (listed.as_deref(), model_columns) else {
18+
return ScenarioExpectedColumns {
19+
listed,
20+
unlisted: Vec::new(),
21+
unknown: Vec::new(),
22+
};
23+
};
24+
let listed_names: Vec<String> = listed_columns
25+
.iter()
26+
.map(|column| normalized_identifier(column))
27+
.collect();
28+
let model_names: Vec<String> = model_columns
29+
.iter()
30+
.map(|column| normalized_identifier(column))
31+
.collect();
32+
let unlisted: Vec<String> = model_columns
33+
.iter()
34+
.zip(&model_names)
35+
.filter(|(_, name)| !listed_names.contains(name))
36+
.map(|(column, _)| column.clone())
37+
.collect();
38+
let unknown: Vec<String> = listed_columns
39+
.iter()
40+
.zip(&listed_names)
41+
.filter(|(_, name)| !model_names.contains(name))
42+
.map(|(column, _)| column.clone())
43+
.collect();
44+
ScenarioExpectedColumns {
45+
listed,
46+
unlisted,
47+
unknown,
48+
}
49+
}

0 commit comments

Comments
 (0)