Skip to content

fix(databricks): keep parenthesized queries, clauses and spellings apart in trees - #249

Merged
kvlonge merged 5 commits into
mainfrom
fix/databricks-parser-batch-3
Nov 18, 2026
Merged

kvlonge merged 5 commits into
mainfrom
fix/databricks-parser-batch-3

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Why

The Databricks tree screen found spellings that parsed to the same tree as another one: backquoted names, tag key and credential value spellings, and extra pairs of parentheses around a relation or a query. A review of the parentheses fix also found that clauses after a parenthesized query were read in the wrong scope, which hid unknown-column findings in Databricks, Snowflake, DuckDB and PostgreSQL.

Changes

  • Databricks parser: backquoted resource kinds, STORED AS formats and EXTRACT fields, tag key and credential value spellings, and TIMESTAMP WITH LOCAL TIME ZONE keep their spelling.
  • Every extra pair of parentheses around a relation or query is a paren node. Clauses after a parenthesized query go on an unaliased subquery node, as in Snowflake; WITH before the parentheses sets a new ctes_outside flag, serialized only when set.
  • Shared semantics: an unaliased subquery node's ORDER BY resolves in the scope of the query it holds, so (SELECT id FROM orders) ORDER BY nope is reported in every dialect.
  • Lint: the unused-CTE-output and key rules read through parentheses and count clauses after them as reads.
  • Coverage: object-type and alter-table records re-read at the new trees.

Verification

  • make check-ci on the final head.
  • Rust tests for the Databricks parser, semantics, lint, format, blocks, convert and the CLI; clippy; fensu at 0 faults.
  • New tests fail on the previous head: written forms, semantics readings and ranges, Snowflake findings, and lint cases.
  • A grid of 7,164 parenthesized texts: no pair distinct before collapses, and every formatted text re-parses to the same tree.
  • The other seven parsers and five formatters byte-identical over 109,883 corpus texts; every Databricks case re-dumped with no change.

…ial value types in trees

Each of these spellings parsed to the same tree as another one, so a case
could not tell them apart:

- a backquoted resource kind (USING `JAR`), STORED AS format and EXTRACT
  field now set kind_quoted, stored_as_quoted and field_quoted;
- SET TAGS and UNSET TAGS keys record which were backquoted and which
  were strings, as COPY credential keys already did, through one shared
  key_spellings helper;
- a COPY credential value that is not a string (k = 1, k = true) is
  listed in bare_values;
- TIMESTAMP WITH LOCAL TIME ZONE keeps its spelling.

The new shared fields serialise only when set, and the Snowflake parser
fills their defaults (through set_tag and unset_tag for the tag actions),
so other dialects' trees are unchanged.
A subquery node holding an ORDER BY or LIMIT written after a
parenthesized query was read through its inner query only, so its
ORDER BY was never checked: ( SELECT id FROM orders ) ORDER BY nope
gave no finding in Snowflake, DuckDB and PostgreSQL, though the bare
query does.

An unaliased subquery node's ORDER BY now resolves in the scope of the
query it holds, as that query's own ORDER BY does, with the same node
checks, and a set operation's outputs are checked as for its own ORDER
BY. Its ORDER BY keys decide rows when a LIMIT or OFFSET follows them,
and a nested query in an expression stops at such a node, so its
clauses are not read at the outer level.

Snowflake cases cover the query with and without a LIMIT inside the
parentheses.
…a query

Extra parentheses parsed to the same tree as fewer ones:

- FROM: a table's first pair still sets its parenthesized flag; every
  further pair, and a pair around any other relation (a table function,
  a join, an inline table), is now a paren node, so ( ( t1 ) ) and
  ( range ( 1 ) ) differ from ( t1 ) and range ( 1 );
- queryPrimary keeps ( query ) as a paren node, so TABLE ( ( SELECT 1 ) ),
  USING INVENTORY ( ( SELECT 1 ) ), set operands and IN or EXISTS
  subqueries in a second pair differ from one pair;
- CREATE TABLE AS and CREATE VIEW move one outer pair back into the flag
  they already set, so a single pair keeps its existing tree and a
  second pair shows as a paren node.

Clauses written after a parenthesized query, and WITH before it, keep
their place without a synthesized SELECT * FROM ( ... ):

- ORDER BY, SORT BY, CLUSTER BY, DISTRIBUTE BY, LIMIT and OFFSET after
  the parentheses go on an unaliased subquery node holding the query,
  the tree Snowflake gives, so they stay apart from the same clauses
  inside the parentheses; WINDOW moves onto a query that can take it;
- a pipe step after the parentheses records its mark on the select
  inside instead of wrapping it;
- WITH before the parentheses sets a new ctes_outside flag on that
  node, skipped when false, so other dialects' trees are unchanged.

Lineage and findings read through the paren and subquery nodes: semantics
tests check that each extra pair and each clause after the parentheses
read as the bare query, including an unknown ORDER BY column after a
LIMIT inside, and that a pipe step keeps its outputs' ranges.
Written-forms cases cover the parentheses, the clauses at one to three
pairs, in CTAS, IN and scalar subqueries, and both WITH positions.
The unused-CTE-output and key rules matched only a SELECT root or CTE
body, so a whole query in parentheses, WITH before a parenthesized
query, a clause after one, or a CTE body in a second pair of
parentheses lost their diagnostics. Both rules now read the query
within any parentheses or unaliased subquery node, and count that
node's clause expressions as reads, so a CTE column read only by an
ORDER BY after the parentheses is kept. A CTE body's SELECT is found
after any number of opening parentheses, so its fix still applies.

A Databricks test covers each form beside its bare control, and the
kept column.
…urrent parser tree

The spelling and parentheses fixes changed the parser tree and two of
alter-table's trees, so two records went stale:

- object-type's exclusion is repeated unchanged: the catalogue and
  obligations are the same, and the parser still refuses OBJECT as input
  in all three cases;
- alter-table is re-read: its two changed trees only add that a tag key
  was a string, two of its earlier findings now have trees of their own,
  and SET TAGS without = still waits on a receipt, so it stays held.
@kvlonge
kvlonge enabled auto-merge November 18, 2026 09:30
@kvlonge
kvlonge added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit d577724 Nov 18, 2026
9 checks passed
@kvlonge
kvlonge deleted the fix/databricks-parser-batch-3 branch November 18, 2026 10:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant