Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion benchmarks/duckshop/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ function, a hook-created JavaScript function, 192 unit tests and 4 scenarios, in
| Ordering, numbers and text | NULL ordering under `DESC`; ties in ordered aggregates, `MAX_BY`, `MODE` and `ANY_VALUE`; default window frames; `UNION ALL BY NAME`; `ROUND` half-way values, decimal division, `MOD` and 38-digit amounts; integers past 2^53 against `FLOAT`; regular expressions and backslash escapes; `LIKE ... ESCAPE` and `ILIKE ANY`; `SPLIT_PART` and `SUBSTR` before the start; `NOT IN` with a NULL; names in many scripts; `JOIN ... USING` between `NUMBER(18,0)`, `NUMBER(38,0)` and text keys, with the types of the merged and qualified keys; NULL arithmetic and the types of NULL, decimal and exponent literals, `COUNT`, `ROW_NUMBER`, `%` and `NULLIF`; `MEDIAN` at scales 0 to 15; `DIV0` by literal zeros; `MODE`, `MEDIAN` and `COUNT_IF` over no rows; aggregates inside window functions and `QUALIFY`; `ROLLUP` totals; `TRUNC` of `FLOAT`; infinity words in other scripts; `CASE` without `ELSE` and `DECODE` over text; regular expressions with a `?` after a lazy quantifier, classes opening with `]`, empty matches and `.` against a carriage return; case-insensitive matches outside ASCII; `ILIKE` with a long s; `JAROWINKLER_SIMILARITY` of accented names in both cases; Base64 text with line feeds; `STARTSWITH`, `ENDSWITH`, `SUBSTRING`, `POSITION` with `IN` and with a comma, `CHARINDEX`, `CHR`, `REGEXP_SUBSTR_ALL`, `::STRING` and `CAST ... AS STRING`; `HEX_ENCODE`, `HEX_DECODE_STRING`, `TRY_HEX_DECODE_STRING` and `TRY_BASE64_DECODE_BINARY`; a comment attached to a text literal; `FLOOR` with and without a scale, `BITAND`, `ZEROIFNULL`, `::BIGINT` and `::DOUBLE`; `IS NOT DISTINCT FROM` and `EQUAL_NULL`; a comma join after an explicit join; `HASH_AGG` read in two row orders |
| Lineage and analysis | clauses whose columns decide rows without being output (`WHERE`, `QUALIFY`, `GROUP BY` by position and `ALL`, `ORDER BY` under `LIMIT`, `EXISTS`, a `FLATTEN` input, `ROLLUP` by position, `EXCEPT` under `LIMIT`); `RIGHT JOIN ... USING`, `NATURAL JOIN`, a `USING` key under a later outer join, and unmatched `ASOF JOIN` rows; `CONNECT BY` beside a column named `level`; a recursive CTE without `RECURSIVE` whose columns shift per iteration; `LAST_DAY` with a date part; positional columns and the names Snowflake gives unaliased expressions; `TEXT(5)` and widened `COUNT` contracts; `LAG` and `LEAD` with one, two and three arguments; a comma-joined `LATERAL` whose nested derived table reads the left item; relations aliased `first` and `last`, one without `AS`, the words of Snowflake's read-order clauses; untyped and typed `FILTER` lambdas over `ARRAY_AGG` in a grouped query; column expression ranges in compiled SQL that SQL and Python macros, a variable and cursor bounds rewrote, and columns qualified by their reference call, whose ranges are dropped |
| Python functions | volatile tokens and numbers from a column, a constant and no argument, and `DIV0` of a volatile call beside look-alike names, checked by assertions that hold for any values; a handler answering NULL input without `STRICT`; Python `None` as JSON null, and datetimes, Decimals and integers past 2^53 handed to `VARIANT` and `FLOAT` results |
| Scenarios and lifecycle | typed values across model boundaries; `VARIANT` expected output; a star under a lower-case contract and a quoted camelCase export; a snapshot keyed and compared on quoted camelCase columns; a cursor-window incremental model over far-apart fixture loads; a change log merged across two loads, checked against a full rebuild |
| Scenarios and lifecycle | typed values across model boundaries; `VARIANT` expected output; a star under a lower-case contract and a quoted camelCase export; a snapshot keyed and compared on quoted camelCase columns; a cursor-window incremental model over far-apart fixture loads; a change log merged across two loads, checked against a full rebuild, and a keyed cursor `delete_insert` copy of it whose rows move out of the window between loads |
| Translation refusals | one model per construct that translation tools refuse or guard, most putting a construct refused only under strict checking before one refused by default, so a model shows both: `FLOAT` values through sums, averages, running totals, arrays, objects, `HASH`, `TO_CHAR` and `::NUMBER(38, 10)`; `VARIANT` dates, decimals and XML; `ARRAY_AGG` without `WITHIN GROUP`, sparse arrays and `ARRAY(NUMBER)` casts; `REDUCE`, `FILTER` and `TRANSFORM` lambdas; patterns and `REGEXP_REPLACE` replacements built from columns, and lazy patterns; `TRY_TO_DATE` with a format from a column; zoned literals, `CONVERT_TIMEZONE` from `CURRENT_TIMESTAMP`, `SEQ8`, `GENERATOR` with an expression `ROWCOUNT`; `PIVOT`, `ANY_VALUE`, `IGNORE NULLS` around `IFF`, unaliased columns read by Snowflake's name, `UNION` branches whose column is always NULL; `TO_NUMBER`, `TO_DECIMAL` and `TRY_BASE64_DECODE_BINARY`; an open (uncontracted) source `edge__supplier_notes`, the table function, read for one basket and after `LATERAL` for every basket, and the hook-created function. `rpt_edge_case_model_keys` and `rpt_edge_case_array_model_keys` read every model in the group, so a refusal there also refuses a downstream model |

Unit-test expectations in the edge-case area are captured on Snowflake under the `edge` target's
Expand Down
14 changes: 10 additions & 4 deletions benchmarks/duckshop/RUNBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,21 +196,27 @@ every token being distinct.

`int_edge_delivery_status` merges a shipment delivery change log incrementally, and the
`edge_delivery_status_matches_rebuild` audit fails if it differs from a rebuild from the whole log.
To exercise the merge across loads, start without the table, load the first load only, build, then
load everything and build again:
`int_edge_delivery_status_replaced` copies it with a keyed cursor `delete_insert`, and its `unique`
audit fails if a shipment's earlier row survives. To exercise both across loads, start without the
tables, load the first load only, build, then load everything and build again:

```bash
sqb query --target edge "DROP TABLE IF EXISTS <database>.<schema>.int_edge_delivery_status"
sqb query --target edge "DROP TABLE IF EXISTS <database>.<schema>.int_edge_delivery_status_replaced"
sqb load --target edge --select edge__delivery_updates --vars '{"edge_delivery_batch": "1"}'
sqb build --target edge --no-load \
--select int_edge_delivery_update_versions int_edge_delivery_status int_edge_delivery_status_rebuilt
--select int_edge_delivery_update_versions int_edge_delivery_status int_edge_delivery_status_rebuilt \
int_edge_delivery_status_replaced
sqb load --target edge --select edge__delivery_updates
sqb build --target edge --no-load \
--select int_edge_delivery_update_versions int_edge_delivery_status int_edge_delivery_status_rebuilt
--select int_edge_delivery_update_versions int_edge_delivery_status int_edge_delivery_status_rebuilt \
int_edge_delivery_status_replaced
```

The second build plans `int_edge_delivery_status` as a `merge` over the second load, which carries
a late older version, a redelivered version, a delete and a new shipment, and the audit passes.
It plans `int_edge_delivery_status_replaced` as a `delete_insert` that replaces each recomputed
shipment by key, and its audits pass.
Load with `sqb load` and build with `--no-load`, so that each build reads exactly the load you
chose.

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
MODEL (
description """
One row per shipment, copied from the merged delivery status by delete and insert. A shipment
with a later version moves its source_loaded_at out of the window an earlier build wrote, so
only its unique_key stops the later build from adding a second row for it.
""",
tags [edge_case],
materialized incremental,
incremental_strategy delete_insert,
unique_key [shipment_id],
cursor source_loaded_at,
cursor_type timestamp,
cursor_grain second,
cursor_inputs (
int_edge_delivery_status source_loaded_at,
),
columns (
shipment_id (type VARCHAR, nullable false, audits [not_null, unique]),
order_ref (type VARCHAR),
delivery_status (type VARCHAR),
ordered_on (type DATE),
delivered_on (type DATE),
is_deleted (type BOOLEAN),
latest_seq (type NUMBER(38, 0)),
source_loaded_at (type TIMESTAMP_NTZ, nullable false, audits [not_null]),
),
);

WITH final AS (
SELECT
s.shipment_id,
s.order_ref,
s.delivery_status,
s.ordered_on,
s.delivered_on,
s.is_deleted,
s.latest_seq,
s.source_loaded_at
FROM __ref("int_edge_delivery_status") AS s
WHERE
s.source_loaded_at >= __cursor_start()
AND s.source_loaded_at < __cursor_end()
)

SELECT *
FROM final
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
TEST (
name "int_edge_delivery_status_replaced__copies_only_shipments_loaded_in_the_window",
cursor_start "2026-03-02 00:00:00",
cursor_end "2026-03-03 00:00:00",
);

WITH __ref__int_edge_delivery_status AS (
SELECT
v.shipment_id,
v.order_ref,
v.delivery_status,
v.ordered_on::DATE AS ordered_on,
v.delivered_on::DATE AS delivered_on,
v.is_deleted,
v.latest_seq::NUMBER(38, 0) AS latest_seq,
v.source_loaded_at::TIMESTAMP_NTZ AS source_loaded_at
FROM VALUES
('SH-1', 'DS-O-3001', 'in_transit', '2026-02-27', NULL, FALSE, 12, '2026-03-02 10:00:00'),
('SH-3', 'DS-O-3003', 'delivered', '2026-02-20', '2026-02-25', FALSE, 30, '2026-03-01 10:00:00'),
('SH-4', 'DS-O-3004', 'in_transit', '2026-02-26', NULL, TRUE, 41, '2026-03-02 23:59:59'),
('SH-6', 'DS-O-3006', 'label_created', '2026-03-02', NULL, FALSE, 60, '2026-03-03 00:00:00') AS v(
shipment_id,
order_ref,
delivery_status,
ordered_on,
delivered_on,
is_deleted,
latest_seq,
source_loaded_at
)
),

__expected__int_edge_delivery_status_replaced AS (
SELECT
'SH-1' AS shipment_id,
'in_transit' AS delivery_status,
FALSE AS is_deleted,
12 AS latest_seq,
'2026-03-02 10:00:00'::TIMESTAMP_NTZ(9) AS source_loaded_at
UNION ALL
SELECT
'SH-4' AS shipment_id,
'in_transit' AS delivery_status,
TRUE AS is_deleted,
41 AS latest_seq,
'2026-03-02 23:59:59'::TIMESTAMP_NTZ(9) AS source_loaded_at
)

SELECT 1
12 changes: 3 additions & 9 deletions crates/sqlbuild-core/src/janitor_text/_helpers/usage.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,7 @@

use crate::cli_text::main::format_count_noun::format_count_noun;
use crate::janitor_text::_helpers::phases::noun;
use crate::janitor_text::constants::{
KEEP_WITHOUT_USAGE_NOTICE, READER_LIMIT, USAGE_NOT_CHECKED_PREFIX, WINDOW_UNITS,
};
use crate::janitor_text::constants::{READER_LIMIT, USAGE_NOT_CHECKED_PREFIX, WINDOW_UNITS};
use crate::janitor_text::models::{ArchiveCandidateText, JanitorPlanText, UsageCheckText};

/// The usage check when it ran and could read usage.
Expand Down Expand Up @@ -81,14 +79,10 @@ pub(crate) fn usage_notices(plan: &JanitorPlanText) -> Vec<String> {
if check.checked {
return check.notices.clone();
}
let mut notices: Vec<String> = vec![format!(
vec![format!(
"{USAGE_NOT_CHECKED_PREFIX}{}",
check.reason.as_deref().unwrap_or("unknown reason")
)];
if check.keep_in_use {
notices.push(KEEP_WITHOUT_USAGE_NOTICE.to_owned());
}
notices
)]
}

/// The prompt line stating how many archived relations are in use, or which are kept.
Expand Down
2 changes: 0 additions & 2 deletions crates/sqlbuild-core/src/janitor_text/constants.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,6 @@ pub(crate) const SECONDS_PER_HOUR: u64 = 3600;
/// How many readers each in-use line names before `+N more`.
pub(crate) const READER_LIMIT: usize = 3;
pub(crate) const USAGE_NOT_CHECKED_PREFIX: &str = "usage not checked: ";
pub(crate) const KEEP_WITHOUT_USAGE_NOTICE: &str =
"nothing kept in use: no usage could be checked, so relations in use are archived too";
/// Single-unit usage windows and the noun of their unit.
pub(crate) const WINDOW_UNITS: [(&str, &str); 4] = [
("d", "day"),
Expand Down
3 changes: 1 addition & 2 deletions crates/sqlbuild-core/src/janitor_text/tests/helpers.rs
Original file line number Diff line number Diff line change
Expand Up @@ -202,13 +202,12 @@ pub(super) fn kept_usage_plan() -> JanitorPlanText {
}

/// A plan whose usage was not checked because the adapter has no usage source.
pub(super) fn unchecked_plan(keep_in_use: bool) -> JanitorPlanText {
pub(super) fn unchecked_plan() -> JanitorPlanText {
JanitorPlanText {
usage_check: Some(UsageCheckText {
checked: false,
reason: Some("unsupported adapter".to_owned()),
window: "7d".to_owned(),
keep_in_use,
..UsageCheckText::default()
}),
..direct_archive_plan()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,24 +66,14 @@ fn given_usage_check_when_rendering_preview_then_in_use_relations_show_readers()
"\n view chains not followed: grant SNOWFLAKE.OBJECT_VIEWER\n",
"\nIn use\n",
],
unexpected_fragments: &["usage not checked", "nothing kept in use"],
unexpected_fragments: &["usage not checked"],
},
PreviewTestCase {
description: "an unchecked usage shows one notice and no in-use rows",
plan: unchecked_plan(false),
plan: unchecked_plan(),
verbose: false,
expected_fragments: &["\n usage not checked: unsupported adapter\n"],
unexpected_fragments: &["in use ", "In use", "nothing kept in use"],
},
PreviewTestCase {
description: "keep_in_use without a usage source warns that nothing is kept",
plan: unchecked_plan(true),
verbose: false,
expected_fragments: &[
" usage not checked: unsupported adapter\n nothing kept in use: no usage \
could be checked, so relations in use are archived too\n",
],
unexpected_fragments: &["Kept (in use)"],
unexpected_fragments: &["in use ", "In use"],
},
PreviewTestCase {
description: "a plan without a usage check renders as before",
Expand Down Expand Up @@ -141,18 +131,11 @@ fn given_usage_check_when_prompting_then_prompt_states_in_use_count() {
},
UsagePromptTestCase {
description: "an unchecked usage repeats the notice",
plan: unchecked_plan(false),
plan: unchecked_plan(),
expected_fragment: "Janitor will archive 1 and delete 1 objects from dev.\n\
Usage not checked: unsupported adapter\nRetention: 14 days\n",
expected_phrase: "archive 1 and delete 1 objects from dev",
},
UsagePromptTestCase {
description: "keep_in_use without a usage source repeats the warning",
plan: unchecked_plan(true),
expected_fragment: "Usage not checked: unsupported adapter\nNothing kept in use: \
no usage could be checked, so relations in use are archived too\nRetention",
expected_phrase: "archive 1 and delete 1 objects from dev",
},
];

for test_case in test_cases {
Expand Down Expand Up @@ -214,7 +197,7 @@ fn given_usage_check_when_formatting_json_then_candidates_carry_in_use() {
},
JsonTestCase {
description: "an unchecked usage leaves in_use null and states the reason",
plan: Some(unchecked_plan(false)),
plan: Some(unchecked_plan()),
outcome: JanitorOutcome::Previewed,
run: None,
error: None,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -121,16 +121,31 @@ fn optional_string<'py>(
})
}

/// `first or second or third`.
fn first_truthy<'py>(values: [Bound<'py, PyAny>; 3]) -> PyResult<Bound<'py, PyAny>> {
let [first, second, third] = values;
if first.is_truthy()? {
return Ok(first);
/// A project-level location: the target's value, then `[defaults]`, then the first set fallback.
pub(crate) fn project_location<'py>(
(target, default): (Bound<'py, PyAny>, Bound<'py, PyAny>),
fallbacks: Vec<Bound<'py, PyAny>>,
) -> PyResult<Bound<'py, PyAny>> {
let py: Python<'py> = target.py();
for value in [target, default].into_iter().chain(fallbacks) {
if value.is_truthy()? {
return Ok(value);
}
}
if second.is_truthy()? {
return Ok(second);
Ok(py.None().into_bound(py))
}

/// Whether the top-level `[connection]` is the implicit default target: no named connections or targets.
pub(crate) fn connection_is_default_target(discovered: &Bound<'_, PyAny>) -> PyResult<bool> {
for config in ["project_config", "local_config"] {
let config: Bound<'_, PyAny> = native_field(discovered, config)?;
for field in ["connections", "targets"] {
if native_field(&config, field)?.is_truthy()? {
return Ok(false);
}
}
}
Ok(third)
Ok(true)
}

/// Resolve the target and its effective database/schema without compiling resources.
Expand Down Expand Up @@ -170,18 +185,37 @@ pub(crate) fn build_effective_target_namespace<'py>(
}
native_field(&target_config, key)
};
let project_config: Bound<'py, PyAny> = native_field(discovered_inputs, "project_config")?;
let defaults: Bound<'py, PyAny> = native_field(&project_config, "defaults")?;
let inline: bool = connection_is_default_target(discovered_inputs)?;
let values: Bound<'py, PyDict> = PyDict::new(py);
values.set_item(
DATABASE_KEY,
first_truthy([
target_value(DATABASE_KEY)?,
connection_database,
default_database,
])?,
project_location(
(
target_value(DATABASE_KEY)?,
native_field(&defaults, DATABASE_KEY)?,
),
[connection_database]
.into_iter()
.filter(|_| inline)
.chain([default_database])
.collect(),
)?,
)?;
values.set_item(
SCHEMA_KEY,
first_truthy([target_value(SCHEMA_KEY)?, connection_schema, default_schema])?,
project_location(
(
target_value(SCHEMA_KEY)?,
native_field(&defaults, SCHEMA_KEY)?,
),
[connection_schema]
.into_iter()
.filter(|_| inline)
.chain([default_schema])
.collect(),
)?,
)?;
let run_id: Bound<'py, PyAny> = run_id(&py.None().into_bound(py))?;
let resolved: Bound<'py, PyDict> =
Expand Down
Loading
Loading