Skip to content

feat(databricks): verify Databricks SQL syntax coverage against 2026.45 receipts - #248

Merged
kvlonge merged 10 commits into
mainfrom
feat/databricks-syntax-coverage
Nov 18, 2026
Merged

kvlonge merged 10 commits into
mainfrom
feat/databricks-syntax-coverage

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Why

SQLQuack claims Databricks SQL syntax coverage only where a form's cases parse as the Databricks 2026.45 warehouse does and each accepted case keeps what was written in its tree. This brings Databricks to that standard: documented forms, obligations, cases and receipts, parser fixes for what the receipts and the tree screen found, and per-form reviews.

Changes

  • Pin Databricks SQL 2026.45, its documentation pages and the published receipts; derive catalogue forms, obligations and cases from the pages.
  • Databricks parser: documented statements as trees of clauses, compound statements as the shared scripting tree, and the spellings the screen found (optional keywords, =, commas, parentheses, quoting) kept in the tree.
  • sqlquack-syntax: a call's set quantifier replaces its distinct flag (breaking for tree JSON readers); optional fields that serialize only when set; scripting types in their own domain.
  • Coverage check: a tree screen of one-token and paren-pair variants as a fail-closed gate on approved forms, retired forms in the summary, suppressed obligations reported, receipts from any engine release of the pinned year.
  • Review records for every form; 1062 of 1219 Databricks forms verified, 1 excluded, 13 retired.

Verification

  • make check-ci on the final head.
  • Rust tests for sqlquack-syntax, -format, -semantics, the coverage check and all eight parsers; pytest for the touched script packages.
  • Every Databricks case re-dumped at each parser change; the cross-dialect corpus byte-identical for the other seven dialects in parse and format.
  • Independent reviews of each range, with one follow-up each.

Function and AggregateFunction replace their distinct flag with set_quantifier: none, all or
distinct. The Databricks parser and Snowflake's generic aggregates record ALL, which was dropped,
so a call written with ALL no longer has the same tree as one without it. Translation, lint and
semantics read only whether the quantifier is DISTINCT, so ALL behaves exactly as no quantifier;
a conversion test holds that.

BREAKING CHANGE: serialised call trees carry set_quantifier in place of distinct.
The set quantifier rename serialises a call's quantifier as "none" or
"all" rather than a false distinct flag, and the check for COUNT(*) alone
read any non-empty field as something it does not understand. COUNT(*)
therefore looked like it read the source's columns, and converting it over
a table with ARRAY or VARIANT columns checked their JSON for nothing. The
check now reads the quantifier the way the rest of conversion does: only
DISTINCT counts.

The explicit-ALL conversion test moves to its own table-driven file,
since test_convert.rs passed 2,000 lines.
…quantifier

Snowflake and SQLite reviews whose forms' trees differ from the reviewed
ones only by the call flag now read as set_quantifier are re-bound, each
noted as diff-checked. A re-bound review check refuses every re-bind whose
form changed by more than its class allows: a set_quantifier re-bind only
by that flag in the trees, and a stage re-bind only by the stage of cases
expected to be accepted, each the stage of its receipt phase.
…p what was written

The Databricks parser reads each documented statement into a tree instead
of opaque text, and keeps in that tree every spelling the grammar tells
apart, so a form's evidence can be told from its neighbours':

- documented statements checked against their syntax pattern become trees
  of clauses, with a paren node only on a ( whose content a sibling
  choice spells without it; compound statements become the shared
  scripting tree;
- DDL, DML, COPY INTO, directory inserts, pipe operators, hints, STREAM,
  TABLESAMPLE, NEAREST BY and WATERMARK keep their clauses;
- optional keywords (VARIABLE, AS, DATABASE or NAMESPACE, TEMP), omitted
  = and commas, parenthesised lists and relations, quoting of names and
  keys, and written clause order stay in the tree.

The shared tree gains optional fields that serialise only when set, so
the other dialects' trees and formatting are unchanged; their parsers and
the semantics and diagnostics readers fill or ignore the defaults.
Scripting types move to their own expressions/scripting domain, whose
entry is the scripting_sql wrapper SQL Server and Databricks shared.
…es are not reviewed

The Databricks parser now reads as trees many forms that used to be gaps.
The formatter and lint read Databricks SQL through parse_strict_reviewed,
which keeps forms whose layout is not reviewed yet as gaps, so they still
refuse such a statement with the same opaque-statement message as before,
while the coverage check sees the trees. A form leaves that list only with
reviewed layout and formatter checks.
…e on it

The coverage check gains a tree screen: for each accepted case, variants
that change one lexer token (drop, synonym swap, backquote, string,
double quote, requote, unquote, insert AS, insert =, insert , and a
matching paren pair) are parsed, and any that gives the same tree marks a
spelling the tree does not keep. validate fails closed when an approved
form has a same-tree variant that screen.toml does not allow; an entry is
pinned to a token, the last token, or text no lexer token covers, and an
entry for an unknown or unscreened form fails.

The checker also:

- lets an approval resolve the changes_requested reviews it answers, and
  lets a changes_requested review block a current approval;
- shows retired forms in the summary, apart from excluded ones, and
  reports suppressed obligations;
- counts a receipt from any engine release of the pinned year;
- treats a native parser's resource guard as a guard failure, in the
  depth probe's analysis too;
- refuses a bad resolves entry and a family review that excludes unlisted
  forms.
…ling

Scripts that turn the pinned Databricks pages into catalogue forms,
derive each form's obligations and cases, and record and compare the
warehouse's receipts:

- the catalogue builder reads Databricks SQL 2026.45's documentation;
- obligations pair each clause-order interaction with the two clauses it
  reorders, and cases settle by obligation kind, never against a kind
  that refutes the receipt's phase;
- the referee maps warehouse error classes to the phase that raised them,
  records runs sequentially and compares two runs batch by batch.
…nd cases

The Databricks coverage root gains its forms from the pinned
documentation pages, each form's obligations and hand-written limits,
retirements for forms the pages no longer document, aliases, and the
screen allowlist, whose entries cover only respellings of one token and
comment text.
The pinned sqlquack-evidence commit now holds the Databricks 2026.45
cases and receipts the coverage root reads.
… screen

Adds a review record for every Databricks form read so far, each with its
own note: approvals bound to the form's fingerprint and definition, and
changes_requested records that withdraw a form where the screen or a
reader found a collapsed spelling, each naming the pair that showed it.
One exclusion keeps object-type, an output-only type, out of the count.

check-ci runs dialect-validate, so a change that makes an approved form's
tree lose a spelling fails the gate.

1062 of 1219 Databricks forms are verified, 1 is excluded and 13 are
retired.
@kvlonge
kvlonge enabled auto-merge November 18, 2026 04:22
@kvlonge
kvlonge added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit bfd239e Nov 18, 2026
9 checks passed
@kvlonge
kvlonge deleted the feat/databricks-syntax-coverage branch November 18, 2026 05:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant