Skip to content

fix(databricks)!: keep raw strings, insert parentheses and WINDOW placement apart in trees - #254

Merged
kvlonge merged 14 commits into
mainfrom
fix/databricks-parser-batch-4
Nov 18, 2026
Merged

kvlonge merged 14 commits into
mainfrom
fix/databricks-parser-batch-4

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Why

The Databricks tree screen still found spellings that parsed to the same tree as another: raw and adjacent strings, an insert's parenthesised query, a written SELECT * FROM ( query ) against a pipe step's input, and a WINDOW after a parenthesized query. Name checks also skipped joins written in parentheses, in every dialect.

Changes

  • Databricks parser: r'c' stays a raw string, and adjacent strings become a new Literal::JoinedString that keeps each piece. This is breaking: a new public variant, and the tree JSON of these spellings changes. An insert's parenthesised query, a written SELECT * FROM ( query ) and a WINDOW after the parentheses each keep their own tree. A TIMESTAMP with both time zone spellings is refused.
  • Shared semantics, intended in every dialect: ON, USING, derived-table and LATERAL checks read joins in parentheses, and USING compares each join's own sides. In DuckDB, whose tree keeps the comma's lower precedence, t1 a, t2 b JOIN t3 USING (c1) JOIN t1 USING (c2) now reports E201, a true positive under DuckDB's rules.
  • A window named after a parenthesized query resolves inside it, and a raw string is typed STRING.
  • Coverage: string-type approved, object-type re-bound at the new parser tree, and letter case named as an allowed respelling.

Verification

  • make check-ci on the final head.
  • New tests fail on the previous head: written-forms pairs, Databricks and PostgreSQL findings, and lineage readings.
  • Review grid: 0 collapsed pairs over 7,737 texts, and all 226 changed Databricks trees round-trip through the formatter.
  • The other seven parsers and formatters are byte-identical, and no bare USING-chain finding changed in any of the 8 profiles.
  • SQLClean pins a release without JoinedString and never references Literal or Subquery.

…re one

INSERT INTO t1 ( SELECT 1 ) dropped its parentheses when no alias
followed, so it parsed to the same tree as INSERT INTO t1 SELECT 1, and
INSERT INTO t1 ( VALUES ( 1 ) ) read as bare VALUES rows. The pair is
now a paren node, as queryPrimary keeps elsewhere; an aliased query and
REPLACE ON's query keep their subquery node.

A written-forms case covers no, one and two pairs, VALUES and INSERT
OVERWRITE.
Name and scope checks walked only a SELECT's top-level joins, so a join
inside parentheses, as in FROM ( t1 JOIN t2 ON ... ), skipped its ON and
USING checks and its nested derived tables, in every dialect. Lineage
already flattened such joins.

One walk in the source tracing now lists every FROM part and join,
through any number of parentheses, and the ON, USING, scalar-subquery,
duplicate-alias, derived-table and LATERAL checks read it. Each part
records where its sources start, so a LATERAL item inside parentheses
reads the tables before it there.

Databricks and PostgreSQL tests cover ON, USING, a nested derived table,
a join of a parenthesised join, a scalar subquery and a repeated alias,
with LATERAL and non-LATERAL controls. A Databricks case also checks
that an insert's parenthesised query reads as the bare one.
…TIME ZONE

TIMESTAMP WITHOUT TIME ZONE WITH LOCAL TIME ZONE parsed as a WITHOUT
TIME ZONE type with a time zone. A TIMESTAMP now takes at most one of
the two clauses, so the text is refused, as Spark's grammar does.
…pipe step's input

A pipe operator wraps a query it cannot extend as SELECT * FROM ( query ),
and extended a written SELECT * FROM ( query ) in place, so
( q ) |> WHERE c read as SELECT * FROM ( ( q ) ) |> WHERE c, and
q |> WHERE c as SELECT * FROM ( q ) |> WHERE c. A written select of that
shape, one unaliased subquery and no pipe step before it, is now wrapped
like any other input; chained pipe steps and SELECT * FROM t still
extend in place.

A written-forms case covers the pipe forms with no, one and two pairs
of parentheses, written star selects, an alias and chained steps; the
insert case adds a column list before a parenthesized query.
A string literal and a property key lost how they were written:
r'k' parsed to the same tree as 'k' wherever a string literal or a
tag or credential key was read, and adjacent strings such as 'a' 'b'
to the same tree as 'ab'.

- A single raw string stays a raw string literal at every Databricks
  string-literal site, through one shared one_string helper.
- Adjacent strings at those sites are a new Literal::JoinedString
  holding the joined value and each piece as written. value_str gives
  the joined value and is_string holds, and the shared semantics read
  it as the string it joins to.
- Tag and COPY credential keys record those written as a raw string
  or adjacent strings in a new written_keys field, serialized only
  when set.

Adjacent strings stored in a plain text field (comments, LOCATION,
formats, function bodies and the like) are still joined; BigQuery
also still joins them.

A written-forms case covers raw and adjacent strings and tag keys, and
a semantics case checks that adjacent strings read as the joined one.

BREAKING CHANGE: sqlquack-syntax adds the Literal::JoinedString variant;
code that matches Literal exhaustively must handle it.
…INDOW over a star of it

A WINDOW clause after a parenthesized query whose query could not take
it, such as one with a LIMIT inside, wrapped the query in a synthesized
SELECT * FROM ( ... ), so ( SELECT 1 LIMIT 1 ) WINDOW w AS ( ) parsed
like SELECT * FROM ( ( SELECT 1 LIMIT 1 ) ) WINDOW w AS ( ). The window
now goes on the unaliased subquery node with the other clauses written
after the parentheses, in a new windows field serialized only when set.
A query that can take the window still takes it.

A written-forms case covers WINDOW after the parentheses with and
without a LIMIT inside, against WINDOW over a star of the same query.
… at the current parser tree

Keeping adjacent and raw string spellings changed the parser tree and one
tree of string-type.subscript-dot, so two records went stale:

- string-type.subscript-dot is re-read and approved: 'c' 'c' now keeps
  both pieces and no longer shares the tree of 'cc', which was the only
  reason it was held. All four cases are accepted on both sides, and the
  tree screen finds no same-tree variant in 128;
- object-type's exclusion is repeated unchanged: the catalogue and
  obligations are the same, and the parser still refuses OBJECT as input
  in all three cases.
…e screen

The screen makes no case variants, because the lexer reads a keyword and
the r or R prefix of a raw string case-insensitively. The allowlist now
says so, so r'x' and R'x' giving one tree reads as a deliberate allowance,
and a receipt that shows Databricks reading them differently would make
it a finding.
…INDOW inside it

A WINDOW clause after a parenthesized query still moved into the query
when that query could take it, so ( SELECT 1 ) ORDER BY 1 WINDOW w AS ( )
read as ( SELECT 1 ORDER BY 1 ) WINDOW w AS ( ), and a WITH before the
parentheses read as if the WINDOW were inside them. The clauses after
the parentheses now always stay on the subquery node that holds them,
WINDOW included.

Name checks now read those windows: a window function inside the
parentheses resolves a window named after them, and the window's
partition and order keys are checked in the inner query's scope, so
( SELECT sum ( c1 ) OVER w FROM t1 LIMIT 1 ) WINDOW w AS ( ORDER BY c1 )
no longer reports an unknown named window.

A Databricks case shows the six spellings give six trees, and lineage
cases compare the windowed query with its unparenthesized form.
…wn sides

A USING check took every earlier FROM part as its left side and only
the next part as its right, so a join written in parentheses was
checked against the wrong tables, in every dialect. A key missing from
the table before USING inside the parentheses went unreported when a
table outside had it, and USING after the parentheses reported a key
that a later table inside them had.

Each FROM part now records the parts on either side of the join that
adds it: inside parentheses the left side is the earlier parts there,
and the join that adds a parenthesised join has all of it on its right.

Databricks and PostgreSQL cases cover USING inside and after the
parentheses.
A raw string such as r'a' had no type in Databricks lineage, though a
plain string had STRING. It is now STRING too, as typeof reports.
Keeping a WINDOW written after a parenthesized query on the node after
the parentheses changed the parser tree, so object-type's exclusion went
stale. It is repeated unchanged: the catalogue and obligations are the
same, and the parser still refuses OBJECT as input in all three cases.
…its findings

The case for an unknown column in such a window compared two empty
finding lists, because named window specs are not checked for unknown
columns in any form. The test now shows what the change does: the
window function inside the parentheses finds the window named after
them, and a name no WINDOW clause defines is still unknown.
@kvlonge
kvlonge enabled auto-merge November 18, 2026 14:58
@kvlonge
kvlonge disabled auto-merge November 18, 2026 15:04
@kvlonge
kvlonge added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 7b0c982 Nov 18, 2026
9 checks passed
@kvlonge
kvlonge deleted the fix/databricks-parser-batch-4 branch November 18, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant