Skip to content

fix(bigquery)!: keep adjacent and raw string spellings apart in BigQuery and Databricks trees - #257

Merged
kvlonge merged 13 commits into
mainfrom
fix/databricks-string-spellings
Nov 18, 2026
Merged

kvlonge merged 13 commits into
mainfrom
fix/databricks-string-spellings

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Why

The Databricks and BigQuery tree screens still found string spellings that parsed to the same tree as another: adjacent strings in BigQuery, raw and adjacent strings in Databricks plain-text fields, credential values and typed literals, and a function parameter's comment that was dropped. Grouping checks also reported a false E230 when one string was spelled two ways.

Changes

  • BigQuery parser: adjacent strings become Literal::JoinedString and keep each piece. This is breaking: the tree JSON of these spellings changes. A GRANT or REVOKE grantee written as adjacent strings is still accepted.
  • Databricks parser: raw and adjacent strings stay as written in plain-text string fields and in COPY credential values, which record only raw and adjacent values in written_values. DATE r'...' and the other raw typed literals stay a cast of the raw string. A function parameter keeps its comment.
  • Shared semantics: a projection matches a GROUP BY expression whose strings are spelled differently, raw, adjacent or as a raw typed literal against its plain form, in every dialect. A different value is still refused.
  • Coverage: object-type re-bound at the new parser tree, and function-name case named as an allowed respelling.

Verification

  • make check-ci on the final head.
  • Each fix has tests that fail on the previous head: written-forms pairs, BigQuery grants, Databricks credentials, and Databricks and BigQuery grouping findings.
  • Tree screen: collapse groups went from 141 to 11, none of them at the changed sites, with 0 verdict changes.
  • The other dialects' parse and format output is byte-identical over 82,915 texts, apart from the BigQuery grantee texts.
  • Grouping grid of 25 templates and 8 by 8 spellings: every change is a removed E230.

… fields

Comments, paths, classes, URIs, SQLSTATEs and string names were read
into plain text fields, so 'a' 'b' gave the same tree as 'ab', and
r'x' the same tree as 'x', at 24 sites: column, view, struct and ALTER
COLUMN comments, SET LOCATION, INSERT OVERWRITE DIRECTORY with its
INPUTFORMAT and OUTPUTFORMAT, LOAD DATA INPATH, a function's string
body, resource URI and handler, SQLSTATE in conditions and SIGNAL, USE
and SET CATALOG with a string name, SHOW TBLPROPERTIES keys and OPTIONS
keys.

- Each affected shared node gets an additive <field>_written field
  that holds the field's Literal (a raw string, or a JoinedString with
  its pieces), serialized only when set. OPTIONS keys use written_keys,
  as tag keys do. ALTER COLUMN ... COMMENT keeps it on the AlterColumn
  action, since its Comment action is a tuple variant.
- The parser reads every such site through one written_string helper;
  SHOW TBLPROPERTIES keys become string literals, an expression site.
- Four values the parser reads without a field of its own are left to
  the next commit.

Other dialects fill the new fields with None, so their trees are
unchanged.

A written-forms case covers every site with adjacent strings against
the joined string, and raw against plain where the grammar allows it.
The BigQuery parser joined adjacent string literals into one string, so
SELECT 'a' 'b' gave the same tree as SELECT 'ab', and a raw piece lost
its prefix.

- A string literal written as several chunks is now a JoinedString
  holding the joined value and each chunk as written, raw chunks as raw
  strings. A single chunk keeps the tree it had: a string, or a raw
  string when raw.
- Bytes literals and the text read by string_lit (typed literals such as
  DATE '2014' "-01-01", and other plain-text fields) are unchanged.
- BigQuery typing reads a JoinedString as STRING.

Over the 109,883-text corpus, 13 BigQuery trees change, each a select of
adjacent strings; every other parser and every formatter is
byte-identical.

BREAKING CHANGE: the BigQuery parser now gives Literal::JoinedString,
serialized as literal_type "joined_string" with the joined value and its
pieces, for adjacent string literals, where it gave one Literal::String
of the joined text.
A routine parameter's COMMENT was read and dropped, so
CREATE FUNCTION f1 ( p1 INT COMMENT 'a' ) ..., the same with 'b', and
the same with no comment gave one tree.

FunctionParameter gets additive comment and comment_written fields,
serialized only when set; the Databricks parser fills them, and other
dialects leave them unset. The written-forms comment case now covers a
parameter with adjacent, raw, different and no comment.

The three other values the parser reads without a field of its own keep
their text another way: a temporary view's USING column comments and
UNDROP ... WITH ID are rebuilt from their tokens, and ALTER VIEW ... ADD
PARTITION ... LOCATION is an opaque gap that keeps its text, so two
different values give two trees there.
- Snowflake's USE statement parser moves out of ddl/alter.rs into its
  own statements module, so alter.rs stays under the file-length limit.
- Databricks table options return a named OptionSpelling instead of a
  wide tuple.
- CREATE VIEW keeps a comment and its spelling as one pair, set through
  a small helper, so the parser stays within its statement budget.

No behaviour change.
Keeping the written spelling of comments, locations, paths, formats,
function bodies, handlers, resource URIs and SQLSTATE values, and a
function parameter's comment, changed the parser tree, so object-type's
exclusion went stale. It is repeated unchanged: the catalogue and
obligations are the same, and the parser still refuses OBJECT as input
in all three cases.
…d literals

A COPY credential value kept only its text, so k = '1x', k = r'1x' and
k = '1' 'x' gave one tree, and a raw string after a literal type read
as the plain string, so DATE r'2020-01-01' and DATE '2020-01-01' gave
one tree, as did TIME, TIMESTAMP, TIMESTAMP_NTZ, X and INTERVAL.

A credential value written as a raw string or as adjacent strings is
now kept by position in a new written_values field, serialized only
when set, as the keys already were. A raw string after a literal type
is a cast of the raw string to the type word as written, the shape a
TIMESTAMP_NTZ literal already has, so it stays apart from both the
plain literal and a written CAST. An INTERVAL value keeps a raw string.

Written-forms cases show the 3 credential spellings and 17 typed
literal spellings give distinct trees, and lineage cases show each raw
typed literal reads with the type and findings of the plain one.
An unquoted function name keeps its written case in the tree, so count
and COUNT give two trees, and the formatter writes built-in names in
upper case. Databricks reads unquoted function names case-insensitively,
so the screen's allowance and the coverage README now name this case,
next to keyword case and the raw-string prefix, and say that a formatter
round trip compares unquoted function names without their case. A
backquoted name keeps its case.
… fixes

Keeping the written spelling of COPY credential values and of raw
strings in typed literals changed the parser tree again, so
object-type's exclusion went stale. It is repeated unchanged: the
catalogue and obligations are the same, and the parser still refuses
OBJECT as input in all three cases.
Keeping the pieces of adjacent string literals made a grantee such as
'user:a' '@x.com' a joined string, which the grantee rule did not take,
so GRANT and REVOKE refused it with a syntax error at the end of the
input. A joined string is now a grantee by its joined text, as a plain
or raw string is.

Verdict cases cover adjacent and raw-plus-plain grantees, a grantee
list and REVOKE. The GRANT and REVOKE verdict cases move to their own
test file, which keeps the parser's test file within the line limit.
…differently

A projection matched a GROUP BY expression only when the trees were
equal, so the same string written as adjacent pieces or as a raw string
in one place and as a plain string in the other made the projection
ungrouped and reported a false E230. BigQuery has had it since adjacent
pieces were kept, and Databricks since its raw and adjacent strings
were.

When the trees differ and either holds a raw or adjacent string, the
grouping check now compares them without source ranges and with each
such string as a plain string of the same value. Other expressions
compare as before.

BigQuery and Databricks cases cover both directions, a concatenation,
raw strings and a different value that is still refused.
A COPY credential's written_values recorded every value that was not a
plain string, so a number such as k = 1 gained a written spelling that
duplicated bare_values and moved the tree for a spelling that had not
changed. It now records only raw strings and adjacent strings.

A tree case shows a numeric credential value serializes as before.
Recording only raw and adjacent COPY credential values as written
changed the parser tree, so object-type's exclusion went stale. It is
repeated unchanged: the catalogue and obligations are the same, and the
parser still refuses OBJECT as input in all three cases.
… grouping

Databricks keeps DATE, TIME, TIMESTAMP and X written with a raw string
as a cast of the raw string to the word, while the plain form is a typed
literal. A projection using one spelling therefore did not match a GROUP
BY expression using the other, and reported a false E230.

The grouping comparison now reads a raw string cast to one of those
words as the typed literal its plain string makes. TIMESTAMP_NTZ and
TIMESTAMP_LTZ already matched, since both spellings are casts.

Databricks cases cover DATE and X in one direction, TIMESTAMP in the
other, and a different date that is still refused.
@kvlonge
kvlonge enabled auto-merge November 18, 2026 18:04
@kvlonge
kvlonge added this pull request to the merge queue Nov 18, 2026
@kvlonge
kvlonge removed this pull request from the merge queue due to a manual request Nov 18, 2026
@kvlonge
kvlonge added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 3c700d5 Nov 18, 2026
9 checks passed
@kvlonge
kvlonge deleted the fix/databricks-string-spellings branch November 18, 2026 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant