Repository navigation
fix(bigquery)!: keep adjacent and raw string spellings apart in BigQuery and Databricks trees - #257
Merged
Merged
Conversation
… fields Comments, paths, classes, URIs, SQLSTATEs and string names were read into plain text fields, so 'a' 'b' gave the same tree as 'ab', and r'x' the same tree as 'x', at 24 sites: column, view, struct and ALTER COLUMN comments, SET LOCATION, INSERT OVERWRITE DIRECTORY with its INPUTFORMAT and OUTPUTFORMAT, LOAD DATA INPATH, a function's string body, resource URI and handler, SQLSTATE in conditions and SIGNAL, USE and SET CATALOG with a string name, SHOW TBLPROPERTIES keys and OPTIONS keys. - Each affected shared node gets an additive <field>_written field that holds the field's Literal (a raw string, or a JoinedString with its pieces), serialized only when set. OPTIONS keys use written_keys, as tag keys do. ALTER COLUMN ... COMMENT keeps it on the AlterColumn action, since its Comment action is a tuple variant. - The parser reads every such site through one written_string helper; SHOW TBLPROPERTIES keys become string literals, an expression site. - Four values the parser reads without a field of its own are left to the next commit. Other dialects fill the new fields with None, so their trees are unchanged. A written-forms case covers every site with adjacent strings against the joined string, and raw against plain where the grammar allows it.
The BigQuery parser joined adjacent string literals into one string, so SELECT 'a' 'b' gave the same tree as SELECT 'ab', and a raw piece lost its prefix. - A string literal written as several chunks is now a JoinedString holding the joined value and each chunk as written, raw chunks as raw strings. A single chunk keeps the tree it had: a string, or a raw string when raw. - Bytes literals and the text read by string_lit (typed literals such as DATE '2014' "-01-01", and other plain-text fields) are unchanged. - BigQuery typing reads a JoinedString as STRING. Over the 109,883-text corpus, 13 BigQuery trees change, each a select of adjacent strings; every other parser and every formatter is byte-identical. BREAKING CHANGE: the BigQuery parser now gives Literal::JoinedString, serialized as literal_type "joined_string" with the joined value and its pieces, for adjacent string literals, where it gave one Literal::String of the joined text.
A routine parameter's COMMENT was read and dropped, so CREATE FUNCTION f1 ( p1 INT COMMENT 'a' ) ..., the same with 'b', and the same with no comment gave one tree. FunctionParameter gets additive comment and comment_written fields, serialized only when set; the Databricks parser fills them, and other dialects leave them unset. The written-forms comment case now covers a parameter with adjacent, raw, different and no comment. The three other values the parser reads without a field of its own keep their text another way: a temporary view's USING column comments and UNDROP ... WITH ID are rebuilt from their tokens, and ALTER VIEW ... ADD PARTITION ... LOCATION is an opaque gap that keeps its text, so two different values give two trees there.
- Snowflake's USE statement parser moves out of ddl/alter.rs into its own statements module, so alter.rs stays under the file-length limit. - Databricks table options return a named OptionSpelling instead of a wide tuple. - CREATE VIEW keeps a comment and its spelling as one pair, set through a small helper, so the parser stays within its statement budget. No behaviour change.
Keeping the written spelling of comments, locations, paths, formats, function bodies, handlers, resource URIs and SQLSTATE values, and a function parameter's comment, changed the parser tree, so object-type's exclusion went stale. It is repeated unchanged: the catalogue and obligations are the same, and the parser still refuses OBJECT as input in all three cases.
…d literals A COPY credential value kept only its text, so k = '1x', k = r'1x' and k = '1' 'x' gave one tree, and a raw string after a literal type read as the plain string, so DATE r'2020-01-01' and DATE '2020-01-01' gave one tree, as did TIME, TIMESTAMP, TIMESTAMP_NTZ, X and INTERVAL. A credential value written as a raw string or as adjacent strings is now kept by position in a new written_values field, serialized only when set, as the keys already were. A raw string after a literal type is a cast of the raw string to the type word as written, the shape a TIMESTAMP_NTZ literal already has, so it stays apart from both the plain literal and a written CAST. An INTERVAL value keeps a raw string. Written-forms cases show the 3 credential spellings and 17 typed literal spellings give distinct trees, and lineage cases show each raw typed literal reads with the type and findings of the plain one.
An unquoted function name keeps its written case in the tree, so count and COUNT give two trees, and the formatter writes built-in names in upper case. Databricks reads unquoted function names case-insensitively, so the screen's allowance and the coverage README now name this case, next to keyword case and the raw-string prefix, and say that a formatter round trip compares unquoted function names without their case. A backquoted name keeps its case.
… fixes Keeping the written spelling of COPY credential values and of raw strings in typed literals changed the parser tree again, so object-type's exclusion went stale. It is repeated unchanged: the catalogue and obligations are the same, and the parser still refuses OBJECT as input in all three cases.
Keeping the pieces of adjacent string literals made a grantee such as 'user:a' '@x.com' a joined string, which the grantee rule did not take, so GRANT and REVOKE refused it with a syntax error at the end of the input. A joined string is now a grantee by its joined text, as a plain or raw string is. Verdict cases cover adjacent and raw-plus-plain grantees, a grantee list and REVOKE. The GRANT and REVOKE verdict cases move to their own test file, which keeps the parser's test file within the line limit.
…differently A projection matched a GROUP BY expression only when the trees were equal, so the same string written as adjacent pieces or as a raw string in one place and as a plain string in the other made the projection ungrouped and reported a false E230. BigQuery has had it since adjacent pieces were kept, and Databricks since its raw and adjacent strings were. When the trees differ and either holds a raw or adjacent string, the grouping check now compares them without source ranges and with each such string as a plain string of the same value. Other expressions compare as before. BigQuery and Databricks cases cover both directions, a concatenation, raw strings and a different value that is still refused.
A COPY credential's written_values recorded every value that was not a plain string, so a number such as k = 1 gained a written spelling that duplicated bare_values and moved the tree for a spelling that had not changed. It now records only raw strings and adjacent strings. A tree case shows a numeric credential value serializes as before.
Recording only raw and adjacent COPY credential values as written changed the parser tree, so object-type's exclusion went stale. It is repeated unchanged: the catalogue and obligations are the same, and the parser still refuses OBJECT as input in all three cases.
… grouping Databricks keeps DATE, TIME, TIMESTAMP and X written with a raw string as a cast of the raw string to the word, while the plain form is a typed literal. A projection using one spelling therefore did not match a GROUP BY expression using the other, and reported a false E230. The grouping comparison now reads a raw string cast to one of those words as the typed literal its plain string makes. TIMESTAMP_NTZ and TIMESTAMP_LTZ already matched, since both spellings are casts. Databricks cases cover DATE and X in one direction, TIMESTAMP in the other, and a different date that is still refused.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Databricks and BigQuery tree screens still found string spellings that parsed to the same tree as another: adjacent strings in BigQuery, raw and adjacent strings in Databricks plain-text fields, credential values and typed literals, and a function parameter's comment that was dropped. Grouping checks also reported a false E230 when one string was spelled two ways.
Changes
Literal::JoinedStringand keep each piece. This is breaking: the tree JSON of these spellings changes. A GRANT or REVOKE grantee written as adjacent strings is still accepted.written_values.DATE r'...'and the other raw typed literals stay a cast of the raw string. A function parameter keeps its comment.Verification
make check-cion the final head.