Skip to content

Commit 16fc2bf

Browse files
committed
fix: lay out Snowflake expression macro bodies instead of dedenting them
On Snowflake, `sqb format` removed one indentation level from an expression macro or scalar function body on every run, so `--check` never settled. The body was formatted as an unaliased SELECT item, which the Snowflake formatter keeps exactly as written, and the dedent that removes the SELECT again then stripped an authored level from every line. Expression bodies are now formatted as an aliased item, which every dialect lays out, and the alias is removed again; a result without it is a fault that leaves the file unchanged. The dedent also leaves lines inside a $$ dollar-quoted string alone instead of changing the string. DuckShop gains a scoped Snowflake expression macro whose body holds a comment.
1 parent 0e98bab commit 16fc2bf

10 files changed

Lines changed: 271 additions & 21 deletions

File tree

‎benchmarks/duckshop/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -133,7 +133,7 @@ function, a hook-created JavaScript function, 192 unit tests and 4 scenarios, in
133133
| Ordering through arrays | an unordered `ARRAY_AGG` ranked under `LIMIT` by position, by name, through `IFF` and over `UNION ALL`, with fixtures whose pick does not depend on element order; order-free reads (`ARRAY_SIZE`, `ARRAY_CONTAINS`, `ARRAYS_OVERLAP`, `ARRAY_DISTINCT` of `ARRAY_CAT`, `FLATTEN` values through a CTE); `SELECT DISTINCT` over arrays repeated in different orders; `ARRAY_APPEND`, `ARRAY_INTERSECTION`, `ARRAY_SLICE` with negative bounds, `TO_ARRAY` and bracket array literals |
134134
| VARIANT comparisons | `ARRAY_MAX`, `ARRAY_MIN` and `ARRAY_REVERSE` over arrays mixing numbers, strings, booleans, JSON nulls and integers past 2^53; `MAX` across kinds and non-ASCII strings; `=` and `EQUAL_NULL` for `1`, `"1"` and `1.0`, reordered keys, JSON null and DATE or TIMESTAMP values held in a `VARIANT` against their text; `MIN_BY` and `MAX_BY` of `VARIANT` values; `PARTITION BY`, `SELECT DISTINCT` and `ARRAY_AGG(DISTINCT)` over `VARIANT`; JSON nulls in `ARRAY_REMOVE`, `ARRAY_DISTINCT` and `ARRAYS_OVERLAP`; decimals with trailing zeros in `ARRAY_CONTAINS` and `ARRAY_POSITION`; `OBJECT_CONSTRUCT(* EXCLUDE ("note"))` |
135135
| Dates and times | `TRY_TO_TIMESTAMP_NTZ` and its siblings over free text; epoch milliseconds as numbers and text; daylight-saving gaps and overlaps; fractional offsets; nanosecond truncation and epoch parts after 2242; pre-1970 text with sub-microsecond digits; date-part normalisation; month ends and leap days; week numbering across a year; sentinel timestamps; `TRY_TO_DATE` with numeric and month-name formats over unpadded fields, two-digit years, ISO `T` and `Z`, trailing letters, a long s, mixed separators and runs of spaces; text read as `TIMESTAMP_TZ` and `TIMESTAMP_LTZ` with `T`, `Z`, `+00:00`, `+0000`, `+09:00` and `-05:30`, back as the UTC instant, date, year and day start, and compared with text, `TIMESTAMP_NTZ` and `DATE`; `FIRST_VALUE` of a zoned value as a `DATE`; `DATEADD` and `TIMEADD` on `TIME(3)` and the type they return; `DATEDIFF` across units; `TIME(9)`, `FF3` and bare `FF` formats truncating; `DATE`, `TO_TIME`, `TRY_TO_TIMESTAMP` and `TO_TIMESTAMP_LTZ` of clock text, and `CURRENT_DATE` and `SYSDATE` read only for facts that hold on any day |
136-
| Ordering, numbers and text | NULL ordering under `DESC`; ties in ordered aggregates, `MAX_BY`, `MODE` and `ANY_VALUE`; default window frames; `UNION ALL BY NAME`; `ROUND` half-way values, decimal division, `MOD` and 38-digit amounts; integers past 2^53 against `FLOAT`; regular expressions and backslash escapes; `LIKE ... ESCAPE` and `ILIKE ANY`; `SPLIT_PART` and `SUBSTR` before the start; `NOT IN` with a NULL; names in many scripts; `JOIN ... USING` between `NUMBER(18,0)`, `NUMBER(38,0)` and text keys, with the types of the merged and qualified keys; NULL arithmetic and the types of NULL, decimal and exponent literals, `COUNT`, `ROW_NUMBER`, `%` and `NULLIF`; `MEDIAN` at scales 0 to 15; `DIV0` by literal zeros; `MODE`, `MEDIAN` and `COUNT_IF` over no rows; aggregates inside window functions and `QUALIFY`; `ROLLUP` totals; `TRUNC` of `FLOAT`; infinity words in other scripts; `CASE` without `ELSE` and `DECODE` over text; regular expressions with a `?` after a lazy quantifier, classes opening with `]`, empty matches and `.` against a carriage return; case-insensitive matches outside ASCII; `ILIKE` with a long s; `JAROWINKLER_SIMILARITY` of accented names in both cases; Base64 text with line feeds; `STARTSWITH`, `ENDSWITH`, `SUBSTRING`, `POSITION` with `IN` and with a comma, `CHARINDEX`, `CHR`, `REGEXP_SUBSTR_ALL`, `::STRING` and `CAST ... AS STRING`; `HEX_ENCODE`, `HEX_DECODE_STRING`, `TRY_HEX_DECODE_STRING` and `TRY_BASE64_DECODE_BINARY`; a comment attached to a text literal; `FLOOR` with and without a scale, `BITAND`, `ZEROIFNULL`, `::BIGINT` and `::DOUBLE`; `IS NOT DISTINCT FROM` and `EQUAL_NULL`; a comma join after an explicit join; `HASH_AGG` read in two row orders |
136+
| Ordering, numbers and text | NULL ordering under `DESC`; ties in ordered aggregates, `MAX_BY`, `MODE` and `ANY_VALUE`; default window frames; `UNION ALL BY NAME`; `ROUND` half-way values, decimal division, `MOD` and 38-digit amounts; integers past 2^53 against `FLOAT`; regular expressions and backslash escapes, and a scoped SQL macro collapsing whitespace whose body holds a comment; `LIKE ... ESCAPE` and `ILIKE ANY`; `SPLIT_PART` and `SUBSTR` before the start; `NOT IN` with a NULL; names in many scripts; `JOIN ... USING` between `NUMBER(18,0)`, `NUMBER(38,0)` and text keys, with the types of the merged and qualified keys; NULL arithmetic and the types of NULL, decimal and exponent literals, `COUNT`, `ROW_NUMBER`, `%` and `NULLIF`; `MEDIAN` at scales 0 to 15; `DIV0` by literal zeros; `MODE`, `MEDIAN` and `COUNT_IF` over no rows; aggregates inside window functions and `QUALIFY`; `ROLLUP` totals; `TRUNC` of `FLOAT`; infinity words in other scripts; `CASE` without `ELSE` and `DECODE` over text; regular expressions with a `?` after a lazy quantifier, classes opening with `]`, empty matches and `.` against a carriage return; case-insensitive matches outside ASCII; `ILIKE` with a long s; `JAROWINKLER_SIMILARITY` of accented names in both cases; Base64 text with line feeds; `STARTSWITH`, `ENDSWITH`, `SUBSTRING`, `POSITION` with `IN` and with a comma, `CHARINDEX`, `CHR`, `REGEXP_SUBSTR_ALL`, `::STRING` and `CAST ... AS STRING`; `HEX_ENCODE`, `HEX_DECODE_STRING`, `TRY_HEX_DECODE_STRING` and `TRY_BASE64_DECODE_BINARY`; a comment attached to a text literal; `FLOOR` with and without a scale, `BITAND`, `ZEROIFNULL`, `::BIGINT` and `::DOUBLE`; `IS NOT DISTINCT FROM` and `EQUAL_NULL`; a comma join after an explicit join; `HASH_AGG` read in two row orders |
137137
| Lineage and analysis | clauses whose columns decide rows without being output (`WHERE`, `QUALIFY`, `GROUP BY` by position and `ALL`, `ORDER BY` under `LIMIT`, `EXISTS`, a `FLATTEN` input, `ROLLUP` by position, `EXCEPT` under `LIMIT`); `RIGHT JOIN ... USING`, `NATURAL JOIN`, a `USING` key under a later outer join, and unmatched `ASOF JOIN` rows; `CONNECT BY` beside a column named `level`; a recursive CTE without `RECURSIVE` whose columns shift per iteration; `LAST_DAY` with a date part; positional columns and the names Snowflake gives unaliased expressions; `TEXT(5)` and widened `COUNT` contracts; `LAG` and `LEAD` with one, two and three arguments; a comma-joined `LATERAL` whose nested derived table reads the left item; relations aliased `first` and `last`, one without `AS`, the words of Snowflake's read-order clauses; untyped and typed `FILTER` lambdas over `ARRAY_AGG` in a grouped query; column expression ranges in compiled SQL that SQL and Python macros, a variable and cursor bounds rewrote, and columns qualified by their reference call, whose ranges are dropped |
138138
| Python functions | volatile tokens and numbers from a column, a constant and no argument, and `DIV0` of a volatile call beside look-alike names, checked by assertions that hold for any values; a handler answering NULL input without `STRICT`; Python `None` as JSON null, and datetimes, Decimals and integers past 2^53 handed to `VARIANT` and `FLOAT` results |
139139
| Scenarios and lifecycle | typed values across model boundaries; `VARIANT` expected output; a star under a lower-case contract and a quoted camelCase export; a snapshot keyed and compared on quoted camelCase columns; a cursor-window incremental model over far-apart fixture loads; a change log merged across two loads, checked against a full rebuild, and a keyed cursor `delete_insert` copy of it whose rows move out of the window between loads |
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
MACRO (
2+
description "Text with each run of whitespace collapsed to one space",
3+
arguments (text_value VARCHAR),
4+
returns VARCHAR,
5+
);
6+
7+
REGEXP_REPLACE(
8+
@text_value,
9+
-- Each run of spaces, tabs and line breaks becomes one space
10+
'\\s+',
11+
' '
12+
)::VARCHAR

‎benchmarks/duckshop/project/models/staging/edge_cases/stg_edge_ticket_notes.sql‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
MODEL (
22
description """
33
One row per support ticket note with order references and reasons found by regular expressions,
4-
written with escaped and dollar-quoted patterns, a SKU normalised by a macro after an escaped
5-
quote, campaign and discount codes matched by pattern lists and escapes, and SKUs split at edge
6-
positions; the note text and tags are passed on as written
4+
written with escaped and dollar-quoted patterns, whitespace collapsed by a macro whose body
5+
holds a comment, a SKU normalised by a macro after an escaped quote, campaign and discount
6+
codes matched by pattern lists and escapes, and SKUs split at edge positions; the note text and
7+
tags are passed on as written
78
""",
89
tags [edge_case],
910
columns (
@@ -51,7 +52,7 @@ final AS (
5152
REGEXP_LIKE(s.body, '.*ds-o-\\d+.*', 'is') AS mentions_order_regexp_like,
5253
REGEXP_COUNT(s.body, 'DS-O-\\d+')::INT AS order_ref_count,
5354
REGEXP_SUBSTR(s.body, 'refund(ed)?', 1, 1, 'i')::VARCHAR AS refund_word,
54-
REGEXP_REPLACE(s.body, '\\s+', ' ')::VARCHAR AS body_collapsed,
55+
@sql.collapse_whitespace(s.body) AS body_collapsed,
5556
// A backslash-escaped quote inside a single-quoted literal, then a macro call to expand
5657
CONTAINS(s.body, 'O\'Brien')::BOOLEAN AS quotes_customer,
5758
@sql.normalise_sku(s.sku) AS sku_canonical,

‎crates/sqlbuild-rules/src/lint_text/_helpers/format_bodies.rs‎

Lines changed: 47 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,11 @@ use crate::dsl_headers::main::sql_body_ranges::sql_body_ranges;
1717
use crate::dsl_headers::models::HeaderSpan;
1818
use crate::lint_text::_helpers::sites::compiled;
1919
use crate::lint_text::constants::{
20-
DECLARATION_GROUP_DIRECTORY, DECLINED_WITHOUT_REASON, EXPRESSION_SELECT_PREFIX,
20+
DECLARATION_GROUP_DIRECTORY, DECLINED_WITHOUT_REASON, DOLLAR_QUOTE, EXPRESSION_SELECT_PREFIX,
2121
FIXTURE_HEADER_KINDS, FORMAT_API_VERSION, FUNCTION_HEADER_KIND, GLOBAL_NAMED_ROOTS,
22-
GROUPED_NAMED_ROLES, LINE_COMMENT_PREFIX, MACRO_HEADER_KIND, MAX_FUNCTION_CALL_DEPTH,
23-
MOVED_OMITTED_SELECT, OMITTED_CEREMONIAL_SELECT_SQL, OMITTED_SELECT_SUFFIX_PATTERN,
24-
STRING_QUOTES,
22+
GROUPED_NAMED_ROLES, LINE_COMMENT_PREFIX, LINTED_EXPRESSION_ALIAS, MACRO_HEADER_KIND,
23+
MAX_FUNCTION_CALL_DEPTH, MOVED_EXPRESSION_ALIAS, MOVED_OMITTED_SELECT,
24+
OMITTED_CEREMONIAL_SELECT_SQL, OMITTED_SELECT_SUFFIX_PATTERN, STRING_QUOTES,
2525
};
2626
use crate::lint_text::errors::LintTextError;
2727
use crate::lint_text::main::neutralize_interpolation::neutralize_interpolation;
@@ -31,6 +31,9 @@ use sqlclean::sql_lint::models::{FormatRequest, FormatResponse};
3131

3232
static OMITTED_SELECT_SUFFIX: LazyLock<Result<Regex, regex::Error>> =
3333
LazyLock::new(|| Regex::new(OMITTED_SELECT_SUFFIX_PATTERN));
34+
/// The alias that closes a formatted expression body, with or without an `AS` before it.
35+
static EXPRESSION_ALIAS_SUFFIX: LazyLock<Result<Regex, regex::Error>> =
36+
LazyLock::new(|| Regex::new(&format!(r"(?i)\s+(?:AS\s+)?{LINTED_EXPRESSION_ALIAS}\s*\z")));
3437

3538
/// Deduplicated format requests, keyed by neutralized SQL and sentinel widths.
3639
#[derive(Default)]
@@ -157,16 +160,26 @@ pub(crate) fn prepare_file(
157160
trailing = characters[offset..].iter().collect::<String>() + &trailing;
158161
core = characters[..offset].iter().collect();
159162
}
163+
if wraps_expression && let Some(statement) = core.strip_suffix(';') {
164+
trailing = format!(";{trailing}");
165+
core = statement.trim_end().to_string();
166+
}
160167
let neutralized = neutralize_interpolation(&core, dialect, true)?;
161168
let mut sql: String = if wraps_expression {
162-
format!("{EXPRESSION_SELECT_PREFIX}{}", neutralized.text)
169+
format!(
170+
"{EXPRESSION_SELECT_PREFIX}{}\n{LINTED_EXPRESSION_ALIAS}",
171+
neutralized.text
172+
)
163173
} else {
164174
neutralized.text
165175
};
166176
if omitted.is_some() {
167177
sql.push_str(OMITTED_CEREMONIAL_SELECT_SQL);
168178
}
169-
let widths: BTreeMap<String, usize> = sentinel_widths(&neutralized.sites);
179+
let mut widths: BTreeMap<String, usize> = sentinel_widths(&neutralized.sites);
180+
if wraps_expression {
181+
let _ = widths.insert(LINTED_EXPRESSION_ALIAS.to_uppercase(), 0);
182+
}
170183
let body: PreparedBody = PreparedBody {
171184
start,
172185
end,
@@ -187,7 +200,7 @@ enum MacroBodyForm {
187200
CteList,
188201
/// A `SELECT` list, formatted as it stands.
189202
ColumnList,
190-
/// An expression, formatted as the one item of a `SELECT` that is removed again.
203+
/// An expression, formatted as the one aliased item of a `SELECT` that is removed again.
191204
Expression,
192205
}
193206

@@ -269,8 +282,15 @@ pub(crate) fn formatted_body(
269282
}
270283
let unwrapped: String;
271284
if prepared.wraps_expression {
272-
unwrapped = unwrapped_expression(sql).ok_or_else(|| MOVED_OMITTED_SELECT.to_string())?;
285+
let pattern: &Regex =
286+
compiled(&EXPRESSION_ALIAS_SUFFIX).map_err(|error| error.to_string())?;
287+
let Some(alias) = pattern.find(sql) else {
288+
return Err(MOVED_EXPRESSION_ALIAS.to_string());
289+
};
290+
unwrapped = unwrapped_expression(&sql[..alias.start()])
291+
.ok_or_else(|| MOVED_EXPRESSION_ALIAS.to_string())?;
273292
sql = &unwrapped;
293+
changed = true;
274294
}
275295
if !changed {
276296
return Ok(None);
@@ -349,11 +369,29 @@ pub(crate) fn unwrapped_expression(formatted: &str) -> Option<String> {
349369
Some(dedented.join("\n").trim_end().to_owned())
350370
}
351371

352-
/// The quote still open after `line`, given the one open before it; a doubled quote escapes.
372+
/// The quote still open after `line`, given the one before it; `$` stands for `$$`, which has no escapes.
353373
fn open_quote(mut quote: Option<char>, line: &str) -> Option<char> {
354374
let mut characters = line.chars().peekable();
375+
let mut previous: Option<char> = None;
355376
while let Some(character) = characters.next() {
377+
let after_word: bool = previous.is_some_and(|before| {
378+
before.is_alphanumeric() || before == '_' || before == DOLLAR_QUOTE
379+
});
380+
previous = Some(character);
381+
if character == DOLLAR_QUOTE
382+
&& (quote == Some(DOLLAR_QUOTE) || (quote.is_none() && !after_word))
383+
&& characters.peek() == Some(&DOLLAR_QUOTE)
384+
{
385+
let _ = characters.next();
386+
quote = if quote.is_some() {
387+
None
388+
} else {
389+
Some(DOLLAR_QUOTE)
390+
};
391+
continue;
392+
}
356393
match quote {
394+
Some(DOLLAR_QUOTE) => {}
357395
Some(open) if character == open => {
358396
if characters.peek() == Some(&open) {
359397
let _ = characters.next();

‎crates/sqlbuild-rules/src/lint_text/constants.rs‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,8 @@ pub(crate) const LINE_COMMENT_PREFIX: &str = "--";
7979
pub(crate) const DECLINED_WITHOUT_REASON: &str = "native formatter declined without a reason";
8080
pub(crate) const MOVED_OMITTED_SELECT: &str =
8181
"native formatter moved the omitted ceremonial SELECT 1";
82+
pub(crate) const MOVED_EXPRESSION_ALIAS: &str =
83+
"native formatter moved the alias it formats an expression body under";
8284
pub(crate) const DECLARATION_GROUP_DIRECTORY: &str = "_sqlbuild";
8385
/// `GLOBAL_NAMED_DECLARATION_ROOTS`, in Python's dict order, with whether each holds audits.
8486
pub(crate) const GLOBAL_NAMED_ROOTS: [(&[&str], bool); 5] = [
@@ -211,10 +213,12 @@ pub(crate) const GROUPED_NAMED_ROLE_KINDS: [(&[&str], &str); 9] = [
211213
pub(crate) const CROSS_KEYWORD: &[u8] = b"cross";
212214
/// What a SQL macro expression body is formatted after, and removed from again.
213215
pub(crate) const EXPRESSION_SELECT_PREFIX: &str = "SELECT ";
214-
/// The output name lint gives an expression macro body, so it is not an unnamed projection.
216+
/// The output name lint and format give an expression body; Snowflake keeps unnamed items as written.
215217
pub(crate) const LINTED_EXPRESSION_ALIAS: &str = "__sqlbuild_macro_expression__";
216218
/// The characters that open a quoted string or identifier in formatted SQL.
217219
pub(crate) const STRING_QUOTES: [char; 3] = ['\'', '"', '`'];
220+
/// Each half of the `$$` that opens and closes a dollar-quoted string in formatted SQL.
221+
pub(crate) const DOLLAR_QUOTE: char = '$';
218222

219223
/// A CTE definition, capturing its `name`.
220224
pub(crate) const CTE_DEFINITION_PATTERN: &str = r#"(?is)(?:\bWITH(?:\s+RECURSIVE)?|,)\s*["`\[]?(?P<name>[A-Za-z_][A-Za-z0-9_]*)["`\]]?(?:\s*\([^)]*\))?\s+AS\s*\("#;

‎crates/sqlbuild-rules/src/lint_text/tests/helpers.rs‎

Lines changed: 39 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
use crate::lint_text::constants::{DIRECTIVE_FIX_REASON, DIRECTIVE_MESSAGE};
2-
use crate::lint_text::models::SuppressionFinding;
2+
use crate::lint_text::errors::LintTextError;
3+
use crate::lint_text::main::format_sql_bodies::format_sql_bodies;
4+
use crate::lint_text::models::{FormatFile, SuppressionFinding};
35

46
const INDENT: &str = "\n ";
57

@@ -23,3 +25,39 @@ pub(crate) fn directive_finding(line: usize, rule: &str, rest: &[&str]) -> Suppr
2325
fix_unavailable_reason: Some(DIRECTIVE_FIX_REASON),
2426
}
2527
}
28+
29+
/// A macro nesting `depth` calls, each with a comment that keeps it on several lines, `step` per level.
30+
pub(crate) fn nested_macro(depth: usize, step: &str) -> String {
31+
let mut body: String = String::new();
32+
for level in 0..depth {
33+
body.push_str(&format!("{}COALESCE(\n", step.repeat(level)));
34+
}
35+
body.push_str(&format!("{}@value_text", step.repeat(depth)));
36+
for level in (0..depth).rev() {
37+
let inner: String = step.repeat(level + 1);
38+
body.push_str(&format!(
39+
",\n{inner}-- Level {level}\n{inner}'level {level}'\n{})",
40+
step.repeat(level)
41+
));
42+
}
43+
format!("MACRO (arguments (value_text VARCHAR), returns VARCHAR);\n\n{body}\n")
44+
}
45+
46+
/// The project macro file `contents` as one `sqb format` run on `dialect` leaves it, without faults.
47+
pub(crate) fn formatted_macro(contents: &str, dialect: &str) -> Result<String, LintTextError> {
48+
let files = [FormatFile {
49+
relative_parts: Some(vec![
50+
"macros".to_string(),
51+
"sql".to_string(),
52+
"m.sql".to_string(),
53+
]),
54+
contents: contents.to_string(),
55+
}];
56+
let result = format_sql_bodies(&files, dialect, 100)?;
57+
assert_eq!(result.faults, Vec::new(), "{dialect}");
58+
Ok(result
59+
.formatted
60+
.into_iter()
61+
.next()
62+
.map_or_else(|| contents.to_string(), |(_, text)| text))
63+
}

0 commit comments

Comments
 (0)