Repository navigation
Commit b625100
authored
fix(scenario): honour promotion mode and run scenarios concurrently (#984)
* fix(scenario): honour table promotion mode and enforced contracts
Scenario models were always materialized with immediate table promotion, so
any table model with an enforced contract failed with K011 and declared type
enforcement failed with R004 even when the project configured staged
promotion. Scenarios now resolve the same promotion mode as build (project
table_promotion_mode, else the adapter default) and use staged promotion
whenever a model's contract is enforced, so the runtime contract check runs
and the scenario proceeds to its expectations. Cleanup also drops the staging
relation a failed staged promotion keeps for inspection.
* perf(scenario): run scenarios concurrently with fewer statements
Scenario runs issued a blind DROP IF EXISTS for every planned relation both
before and after each scenario, ensured the destination schema before every
fixture and model, and ran strictly one scenario at a time.
- Cleanup reads the catalog once per database and drops only the
scenario-owned relations that exist, using the catalogued relation type, so
a retained relation whose materialization changed is still replaced. When
the catalog cannot be read, cleanup falls back to the previous blind drops.
- Scenario destination schemas are ensured once per invocation.
- An interrupt or unexpected error inside a scenario still drops that
scenario's relations before propagating (unless --retain).
- Independent scenarios run concurrently up to --concurrency or the project
concurrency setting, each on its own connection; results are rendered in
selection order. Scenario artifacts are already namespaced per scenario.
- Text output reports each scenario's wall time and the total; --json adds
scenarios[].duration_ms and summary.duration_ms. Nested model progress rows
are labelled with their scenario so concurrent rows stay unambiguous.
* fix(scenario): allow __empty_fixture() in scenario fixtures
Scenario planning already expands SELECT * FROM __empty_fixture() into the
relation's declared, typed columns, as unit tests do, but the SQL Rules engine
only exempted the marker in TEST files, so scenarios using it failed
SQBRSQL021. Scenario files now receive the same exemption.
* fix(scenario): fit auxiliary relation names within the identifier limit
Staging, type-enforcement, delta and snapshot-delta relations appended a
suffix to the target name without fitting the result. On Postgres (63-char
identifiers) a long model name truncated the staging name to the target
name, so staged promotion failed with R007, and slightly shorter names
leaked a truncated staging table that cleanup could not find.
Every auxiliary name now goes through fit_auxiliary_relation_name, which
keeps the suffix and replaces the overflow with a stable hash. Build, load
and scenario cleanup all use the same fitted names.
* fix(scenario): drop targets whose catalog listing is not authoritative
Adapters now declare whether list_relations covers a scope. Cleanup lists
only authoritative scopes, matches schema and name case-insensitively, and
falls back to DROP IF EXISTS for every other target, including the
type-enforcement relation.
* fix(scenario): stop concurrent scenarios cleanly on interrupt and match build K011
The first Ctrl-C prints one notice, cancels queued scenarios and stops
running ones at their next step boundary so they clean up before the
connections close; repeat presses are absorbed. Enforced contracts with
immediate promotion now fail with K011 exactly as sqb build does.
* fix(contracts): report K011 promotion conflict before execution
An enforced contract on a table or non-microbatch incremental model under
immediate table promotion is now a compile diagnostic, so compile, plan,
build and scenario test (including --local) fail before any connection or
statement. The help names the file and current value, shows the exact
staged TOML, and offers removing the enforced contract. Build and plan
failures now carry the help when every error shares it. The run-time guard
stays as a defensive check for paths the compile check does not cover.
* fix(scenario): cancel in-flight statements on Ctrl-C and fit names by bytes
Running every scenario through a worker pool and absorbing repeat
interrupts made long statements uninterruptible, even at concurrency 1.
Concurrency 1, --local replay and capture now run on the main thread again,
so Ctrl-C reaches the driver as before. A SIGINT handler records the stop
and cancels in-flight statements through the new adapter
interrupt_connection (DuckDB interrupt, Postgres cancel_safe); without it a
DuckDB query interrupted only by the signal kept running in the background
and blocked cleanup. Concurrent runs cancel queued scenarios, cancel or
close in-flight worker connections, wait for teardown, and sweep started
scenarios on a valid or fresh connection. A second Ctrl-C prints one notice
and exits at once; daemon workers never block exit, and the next run of
the same scenarios drops what was left behind.
Identifier fitting now measures UTF-8 bytes and cuts on character
boundaries, so non-ASCII names fit Postgres's 63-byte limit.1 parent 06fa96d commit b625100
86 files changed
Lines changed: 3527 additions & 332 deletions
File tree
- src/sqlbuild
- .agents/skills/sqlbuild/references/docs
- cli
- concepts
- models
- adapters
- bigquery/classes
- databricks/classes
- postgres/classes
- snowflake/classes
- sqlserver/classes
- adapter
- contract/classes
- relations
- _helpers
- main
- cli
- commands
- _helpers
- entry
- scenario_execution
- scenario_output
- output
- _helpers
- main
- progress/classes
- compiler
- contracts
- _helpers
- main
- pipeline/main
- executor
- load
- _helpers
- main
- pipeline/_helpers
- run
- _helpers
- execution
- materializations
- validation
- main
- scenario
- _helpers
- execution
- lifecycle
- local
- classes
- main
- lint/_helpers
- tests
- e2e/src/sqlbuild/cli/commands/main
- postgres
- scenario
- integration/src/sqlbuild/executor
- build
- scenario
- unit/src/sqlbuild
- adapter/relations/main
- fit_artifact_logical_name
- fit_auxiliary_relation_name
- compiler/contracts
- executor
- pipeline/_helpers
- scenario/main
- website/src/content/docs/docs
- cli
- concepts
- models
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
121 | 121 | | |
122 | 122 | | |
123 | 123 | | |
124 | | - | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
125 | 129 | | |
126 | 130 | | |
127 | 131 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
102 | 108 | | |
103 | 109 | | |
104 | 110 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
| |||
93 | 94 | | |
94 | 95 | | |
95 | 96 | | |
96 | | - | |
| 97 | + | |
97 | 98 | | |
98 | 99 | | |
99 | 100 | | |
100 | | - | |
| 101 | + | |
101 | 102 | | |
102 | 103 | | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
103 | 107 | | |
104 | 108 | | |
105 | 109 | | |
106 | | - | |
| 110 | + | |
107 | 111 | | |
108 | 112 | | |
109 | 113 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
| 36 | + | |
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
106 | 106 | | |
107 | 107 | | |
108 | 108 | | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
109 | 113 | | |
110 | 114 | | |
111 | 115 | | |
| |||
145 | 149 | | |
146 | 150 | | |
147 | 151 | | |
148 | | - | |
| 152 | + | |
149 | 153 | | |
150 | 154 | | |
151 | 155 | | |
152 | 156 | | |
153 | 157 | | |
154 | | - | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
155 | 178 | | |
156 | 179 | | |
157 | 180 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
149 | 149 | | |
150 | 150 | | |
151 | 151 | | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
152 | 164 | | |
153 | 165 | | |
154 | 166 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
112 | 112 | | |
113 | 113 | | |
114 | 114 | | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
115 | 127 | | |
116 | 128 | | |
117 | 129 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
101 | 101 | | |
102 | 102 | | |
103 | 103 | | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
104 | 114 | | |
105 | 115 | | |
106 | 116 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | | - | |
| 16 | + | |
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
35 | | - | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
0 commit comments