Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
9 changes: 7 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ PYTHON_TYPED := scripts tests/unit/scripts python/src $(PYTHON_API_TESTS)
PYTHON_TESTS := tests/unit/scripts $(PYTHON_API_TESTS)
endif

.PHONY: distribution-notices format rust-lint python-check fensu syntax-coverage website-figures website-figures-update depth-probe depth-probe-debug instruction-counts instruction-baseline gate-build engine-free bundled-extensions dialect-builds generate-cases generated-cases generate-syntax-cases syntax-obligations syntax-cases dialect-cases corpus-rerecord corpus-verdicts corpus-verdicts-baseline generate-referee-cases referee-cases referee-pending translation-files translation-results help pending check-syntax record-syntax check-convert record-convert sql-sizes sql-size-baseline sql-benchmark check-ci check-merge test test-rust test-python python-build python-module python-api python-wheel verify-recorded verify-changed replay-scope test-crate check-pr-metadata evidence-clone evidence-refresh evidence-pin evidence-pin-check clutter-guard census-duckshop fuzz-smoke gate gate-parallel census duckshop-compiled duckshop-measure
.PHONY: distribution-notices format rust-lint python-check fensu syntax-coverage website-figures website-figures-update depth-probe depth-probe-debug instruction-counts instruction-baseline gate-build engine-free bundled-extensions dialect-builds generate-cases generated-cases generate-syntax-cases syntax-obligations syntax-cases dialect-cases dialect-validate corpus-rerecord corpus-verdicts corpus-verdicts-baseline generate-referee-cases referee-cases referee-pending translation-files translation-results help pending check-syntax record-syntax check-convert record-convert sql-sizes sql-size-baseline sql-benchmark check-ci check-merge test test-rust test-python python-build python-module python-api python-wheel verify-recorded verify-changed replay-scope test-crate check-pr-metadata evidence-clone evidence-refresh evidence-pin evidence-pin-check clutter-guard census-duckshop fuzz-smoke gate gate-parallel census duckshop-compiled duckshop-measure

format:
cargo fmt --all
Expand Down Expand Up @@ -225,6 +225,11 @@ DIALECT_CASE_DIALECTS ?= databricks
dialect-cases:
@set -e; $(foreach d,$(DIALECT_CASE_DIALECTS),uv run python -m scripts.build_syntax_obligations_$(d) --check --evidence $(call syntax_evidence_dir,$(d));)

# Fails on a broken reference in a dialect root, or when the tree screen finds a one-token variant of
# an approved form's case that parses to the same tree and screen.toml does not allow.
dialect-validate: gate-build
@set -e; $(foreach d,$(DIALECT_CASE_DIALECTS),$(GATE_DIR)/syntax-coverage validate --root $(call suite_dir,syntax,$(d)) --evidence $(call syntax_evidence_dir,$(d));)

# Where the generated referee cases live: the evidence once it is pinned, referee/ until then. The
# environments stay in referee/environments/.
REFEREE_CASE_ARGS = $(if $(EVIDENCE_PINNED),--cases $(EVIDENCE_ROOT)/referee/snowflake/cases)
Expand All @@ -244,7 +249,7 @@ referee-pending:
uv run python -m scripts.write_referee_pending $(REFEREE_ARGS)

# The static gate, the first step of `gate`, and the Python API's tests.
check-ci: rust-lint python-check fensu syntax-coverage website-figures depth-probe instruction-counts corpus-verdicts evidence-pin-check clutter-guard engine-free bundled-extensions dialect-builds generated-cases syntax-obligations syntax-cases dialect-cases referee-cases translation-files python-api distribution-notices replay-scope ## The static gate CI runs.
check-ci: rust-lint python-check fensu syntax-coverage website-figures depth-probe instruction-counts corpus-verdicts evidence-pin-check clutter-guard engine-free bundled-extensions dialect-builds generated-cases syntax-obligations syntax-cases dialect-cases dialect-validate referee-cases translation-files python-api distribution-notices replay-scope ## The static gate CI runs.

# Each distribution ships the licence and third-party notices; its copies must match the root files.
distribution-notices:
Expand Down
69 changes: 64 additions & 5 deletions coverage/syntax-databricks/README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# Databricks SQL syntax catalogue

The documented SQL syntax of Databricks SQL 2026.44 as stable form IDs `db.<family>.<slug>`,
The documented SQL syntax of Databricks SQL 2026.45 as stable form IDs `db.<family>.<slug>`,
with the same layout and checker as the Snowflake root in `coverage/syntax`, the PostgreSQL
root in `coverage/syntax-postgres` and the DuckDB root in `coverage/syntax-duckdb`.
`dialect.toml` names the dialect for `syntax-coverage`.

## Sources

- Documentation: the Databricks SQL language reference
(`https://docs.databricks.com/aws/en/sql/language-manual/`), retrieved 2026-11-06. The page
(`https://docs.databricks.com/aws/en/sql/language-manual/`), retrieved 2026-11-16. The page
set is every page of the docs sitemap under the language manual, plus three function pages
that the function indexes link but the sitemap leaves out (`index_linked` in
`scripts/syntax_catalogue_databricks`): 1,191 pages. `sources.toml` lists each page with the
Expand All @@ -21,7 +21,7 @@ root in `coverage/syntax-postgres` and the DuckDB root in `coverage/syntax-duckd
Databricks SQL's parser is not published; the Spark grammar it extends is the independent
check the omission audit reads.
- Engine: a Databricks SQL serverless warehouse whose `current_version()` reports
`dbsql_version` 2026.44. Every receipt records that value.
`dbsql_version` 2026.45. Every receipt records that value.

## Regenerating

Expand Down Expand Up @@ -113,7 +113,7 @@ listed, and when the lexer has a token that is neither compared nor listed, so a
or new token is caught. `tests/unit/scripts/syntax_catalogue_databricks/test_review.py` checks
this guard.

At the pin: 279 rules (3 unwalked), 10,135 keyword sequences, 3,259 gaps under 146 reviewed
At the pin: 279 rules (3 unwalked), 10,135 keyword sequences, 3,258 gaps under 145 reviewed
`[[gap]]` rows, 21 reviewed `[[unseen]]` tokens, 427 keywords (47 not catalogued) and 51
helper option words (6 not catalogued), 0 problems.

Expand Down Expand Up @@ -222,12 +222,18 @@ class rules. The reasons for each choice follow.
in an expression is refused (`UNSUPPORTED_GENERATOR.NESTED_IN_EXPRESSIONS`).
- The vector-search page's Syntax is a whole query, so it has no context.
- A clause page's form runs in a statement that the page names as its host.
- A column alias list labels the columns of a table-valued generator in the select list
(sql-ref-names, Column alias), so it follows one.
- LIKE in a partition spec is allowed only in ALTER SHARE ADD TABLE (sql-ref-partition).
- A securable object and a principal are the operands of GRANT (sql-ref-privileges,
sql-ref-principal).
- A control statement runs in a compound statement (control-flow pages): a cursor statement
after the cursor's declaration, FETCH and CLOSE after OPEN, LEAVE and ITERATE inside the loop
they name, and GET DIAGNOSTICS and RESIGNAL inside a condition handler.
- Form qualifiers (`FORM_QUALIFY`): a nested field is a path below its column (the manage-column
page: "The fully qualified name of the field"; the update page: "A reference to field within a
column of type STRUCT"), so `field_name` is written behind `c1.`, and a quoted field is quoted
after the column.
- Family contexts (`FAMILY_CONTEXTS`): a function or expression is written as a call in
parentheses, so that a word left at the end of a negative is not read as a column alias.
- Class rules (`DB_CLASS_RULES`): a clause, a column list or ALTER VIEW's rename is an operand
Expand Down Expand Up @@ -539,9 +545,62 @@ obligations 11666/14572. The 11 forms without obligations are not counted.
deadline, both answered after about 610 s with an internal engine error (`Cannot invoke
"java.lang.CharSequence.length()" because "sequence" is null`, SQLSTATE 42000, with no parser
excerpt). That error proves no phase, so no reviewed row maps it, and the form stays
unverified.
unverified. The referee leaves a case that answers exactly this message undecided as "no phase
proven", like a deadline cancel, so the rest of its run keeps its receipts; any other unmapped
error still refuses the run.
Another is `db.scripting.iterate-stmt` (o001, o003): its case `k1d3a5a4a6874`, a LOOP ITERATE
with no LEAVE, runs to the 300 s deadline, so no phase can be observed, and it is left out of
its run.

A form counts once even when several reasons apply, so the reasons overlap.

### Trees of written forms

Where a page's Syntax block offers different spellings of one construct, each spelling parses to
its own tree, so a review of the tree can see which one the SQL wrote. The parser keeps, among
others:

- `DESC` against `DESCRIBE`, `DATABASE`, `SCHEMA` and `NAMESPACE` as written (in `DESCRIBE`,
`CREATE`, `ALTER`, `DROP` and `USE`), `FROM` against `IN`, and whether `LIKE` was written before
a pattern;
- `SET CATALOG` against `USE CATALOG`, which both choose the catalog (the use node's `keyword`);
- `MINUS` against `EXCEPT`, `TEMP` against `TEMPORARY`, and the type synonyms `BYTE`, `SHORT`,
`LONG`, `DEC`, `NUMERIC` and `TIMESTAMP_LTZ`;
- `ADD COLUMN` against `ADD COLUMNS` and `DROP COLUMN` against `DROP COLUMNS`, because the ALTER
TABLE page writes `{ COLUMN | COLUMNS }` as alternatives of one clause;
- the sign written before an interval value, including `+`;
- the operator written for a construct another documented form shares: `==` against `=`, `!=`
against `<>`, `IS [NOT] DISTINCT FROM` against `<=>`, `REGEXP` and `RLIKE` against the `rlike` call, and `>>>`
against the `shiftrightunsigned` call, in a `written_as` field that analysis ignores;
- the parentheses of a `VALUES` row and of a `STREAM` source, which the Syntax blocks write as
alternatives (`expression | ( expression [, ...] )` and `STREAM ( name ) | STREAM name`).
- a trailing `WITH ROLLUP` or `WITH CUBE` against `ROLLUP ( ... )` and `CUBE ( ... )`, and the
Hive `GROUP BY c1 GROUPING SETS ( ... )` against a comma before the sets, in the `GROUP BY`
clause's `with_modifier`;
- `FROM` against a comma after the first argument of `SUBSTR` and `SUBSTRING`, in the call's
`from_separator`, as `TRIM` keeps its SQL-standard spelling;
- a `CACHE TABLE` option's value written without `=`, which the page marks optional, in the cache
node's `options_without_equals`;
- `INSERT INTO TABLE` against `INSERT INTO` (the insert's `table_keyword`), and a `REPLACE ON`
query in parentheses against one without, as the page writes `( query ) [ alias ] | query`;
- an `INSERT OVERWRITE DIRECTORY` option's value written without `=`, in the directory's
`options_without_equals`, and a path written after `DIRECTORY` against one given only as the
`'path'` option;
- `DROP CONSTRAINT` with neither, `RESTRICT` or `CASCADE`; `DROP PARTITION` with or without
`PURGE`; `UNSET TBLPROPERTIES` with or without `IF EXISTS`; a view's relationship constraint
named or not, by cardinality, with `ON` or `USING` and with or without an alias; and `CACHE
SELECT`'s `FROM` as a clause, as an alias and as a column;
- a join's `NEAREST BY`: `APPROX` against `EXACT`, `DISTANCE` against `SIMILARITY`, and the number
of results written or not; a `WATERMARK` expression bare, with an alias, with `AS` and with
`AS ( aliases )`;
- each pipe operator with the pipe written before it (`|>` or `|`), in the `pipe_operators` of
the select it amounts to, so `FROM t1 |> AGGREGATE sum ( c1 )` keeps `|> AGGREGATE` apart from
`SELECT sum ( c1 ) FROM t1` and from `FROM t1 | AGGREGATE sum ( c1 )`; a trailing `|>` with
no operation after it is kept as the token alone.

`MATCH_RECOGNIZE`'s `PATTERN` parses into the shared row-pattern tree (symbols, sequences,
alternations, groups, `PERMUTE` and quantifiers) beside the pattern's text, because the page's
Syntax block defines the pattern as a grammar of its own rather than as a string.

Spellings the language treats as one token stay one tree: the letter case of keywords and of a
raw-string prefix, the quote character around a string or binary literal, and comments.
68 changes: 68 additions & 0 deletions coverage/syntax-databricks/aliases.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Retired form ids of the Databricks catalogue. Each old id stays in the catalogue with status
# absent_from_source, is never reused, and counts in no total; `to` lists the forms that now carry
# its grammar, and `reason` cites the pinned page revision that shows it.

[[alias]]
from = "db.functions.ilike.any-some-all"
to = ["db.functions.ilike.ilike-any-some-all"]
reason = "the pinned functions/ilike page (page_sha256 a17d734d8e8b3864..., retrieved 2026-11-16) still shows str [ NOT ] ilike { ANY | SOME | ALL } ( [ pattern [, ...] ] ); this id read the lower-case ilike as a placeholder, and the line is now read with ILIKE as the operator, its pattern list required as hand.toml records (EMPTY_QUANTIFIED_PATTERN)"

[[alias]]
from = "db.functions.ilike.subscript"
to = ["db.functions.ilike.ilike"]
reason = "the pinned functions/ilike page (page_sha256 a17d734d8e8b3864..., retrieved 2026-11-16) still shows str [ NOT ] ilike ( pattern [ ESCAPE escape ] ); this id read the lower-case ilike as a placeholder, and the line is now read with ILIKE as the operator"

[[alias]]
from = "db.functions.like.any-some-all"
to = ["db.functions.like.like-any-some-all"]
reason = "the pinned functions/like page (page_sha256 cc5cbda129a27a23..., retrieved 2026-11-16) still shows str [ NOT ] like { ANY | SOME | ALL } ( [ pattern [, ...] ] ); this id read the lower-case like as a placeholder, and the line is now read with LIKE as the operator, its pattern list required as hand.toml records (EMPTY_QUANTIFIED_PATTERN)"

[[alias]]
from = "db.functions.like.subscript"
to = ["db.functions.like.like"]
reason = "the pinned functions/like page (page_sha256 cc5cbda129a27a23..., retrieved 2026-11-16) still shows str [ NOT ] like ( pattern [ ESCAPE escape ] ); this id read the lower-case like as a placeholder, and the line is now read with LIKE as the operator"

[[alias]]
from = "db.functions.in.subscript-dot"
to = ["db.functions.in.in.syntax"]
reason = "the pinned functions/in page (page_sha256 265a150b2ef73d13..., retrieved 2026-11-16) still shows elem in ( expr1 [, ...] ); this id read the lower-case in as a placeholder, and the line is now read with IN as the predicate"

[[alias]]
from = "db.functions.in.syntax"
to = ["db.functions.in.in-2"]
reason = "the pinned functions/in page (page_sha256 265a150b2ef73d13..., retrieved 2026-11-16) still shows elem in ( query ); this id read the lower-case in as a placeholder, and the line is now read with IN as the predicate"

[[alias]]
from = "db.functions.trim.trim-from-both-from"
to = ["db.functions.trim.trim-from", "db.functions.trim.trim-both-from.syntax"]
reason = "the pinned functions/trim page (page_sha256 164f61a9896e0850..., retrieved 2026-11-16) still shows trim(trimStr FROM str) and trim(BOTH trimStr FROM str) as separate lines; this id joined the two into one form and read the second trim as a placeholder, and each line is now its own form"

[[alias]]
from = "db.functions.vector-search.from"
to = ["db.functions.vector-search.query-text-query-vector"]
reason = "the pinned functions/vector_search page (page_sha256 8bd15371d35fac48..., retrieved 2026-11-16) still shows SELECT * FROM vector_search( index, { query_text | query_vector }, [ num_results ]); this id read the lower-case vector_search as a placeholder and its arguments as positional, while the page's Arguments say all arguments must be passed by name, so the line is now read with named arguments (hand.toml)"

[[alias]]
from = "db.functions.vector-search.from.syntax"
to = ["db.functions.vector-search.query-num-results"]
reason = "the pinned functions/vector_search page (page_sha256 8bd15371d35fac48..., retrieved 2026-11-16) still shows SELECT * FROM vector_search( index, query, num_results ) for Databricks Runtime 15.2 and below; this id read the lower-case vector_search as a placeholder and its arguments as positional, while the page's Arguments say all arguments must be passed by name, so the line is now read with named arguments (hand.toml)"

[[alias]]
from = "db.functions.vector-search.vector-search"
to = ["db.functions.vector-search.query-text-query-vector"]
reason = "the pinned functions/vector_search page (page_sha256 8bd15371d35fac48..., retrieved 2026-11-16) still shows SELECT * FROM vector_search( index, { query_text | query_vector }, [ num_results ]); this id read its arguments as positional, while the page's Arguments say all arguments must be passed by name, so the line is now read with named arguments (hand.toml)"

[[alias]]
from = "db.functions.vector-search.vector-search-2"
to = ["db.functions.vector-search.query-num-results"]
reason = "the pinned functions/vector_search page (page_sha256 8bd15371d35fac48..., retrieved 2026-11-16) still shows SELECT * FROM vector_search( index, query, num_results ); this id read its arguments as positional, while the page's Arguments say all arguments must be passed by name, so the line is now read with named arguments (hand.toml)"

[[alias]]
from = "db.functions.vector-search.vector-search.syntax"
to = ["db.functions.vector-search.query-num-results"]
reason = "the pinned functions/vector_search page (page_sha256 8bd15371d35fac48..., retrieved 2026-11-16) still shows SELECT * FROM vector_search( index, query, num_results ); this id read its arguments as positional and query as a query, while the page's Arguments say all arguments must be passed by name and query is a STRING, so the line is now read with named arguments (hand.toml)"

[[alias]]
from = "db.utility.call"
to = ["db.utility.call.call", "db.utility.call.call.syntax"]
reason = "the pinned sql-ref-syntax-aux-call page (page_sha256 7c95f94809b7ff0d..., retrieved 2026-11-16) still shows CALL procedure_name( [ argument [, ...] ] [ namedParameter => argument ] [, ...] ); this id read namedParameter as a keyword and left out the comma between positional and named arguments. hand.toml reads the line as two forms: call.call (positional arguments only, none included) and call.call.syntax (optional positional arguments, a comma, then one or more named ones). Between them they carry every argument list this id read"
Loading
Loading