Skip to content

feat(syntax): read more Snowflake statements into typed trees and re-baseline form coverage - #224

Merged
kvlonge merged 21 commits into
mainfrom
feat/syntax-pool-24
Nov 14, 2026
Merged

kvlonge merged 21 commits into
mainfrom
feat/syntax-pool-24

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Why

Many Snowflake forms parsed into opaque strings or lost keywords, so reviewers could not verify them against the frozen form review brief (version 1).

Changes

  • Snowflake statements read into typed trees: CREATE TASK, function and procedure properties, ALTER WAREHOUSE, CREATE VARIABLE, TAG, CONNECTION, DCM PROJECT, and grant/revoke targets. WITH, FORCE, quoted UNSET names and string contact purposes are kept as fields.
  • Name, word, numeric and placeholder operands are classified, with wrong-kind cases. The parser now agrees with the new receipts.
  • Form reviews under brief v1 are recorded, evidence is pinned, and the baseline is regenerated.

Verification

  • Verified forms: 2,730 to 2,826 of 3,479. Gross +139, drops 43, net +96.
  • Drops: 22 failed the final review, 19 have new cases still without receipts, and 2 (MERGE, ALTER SERVICE) have changed cases awaiting review. The list is in a comment.
  • Final review round: 73 approved, 323 rejected.
  • Known remaining items:
    • A part omitted from a..b is still serialized as "", which conflicts with brief v1 (d). 297 rejected forms cite it; 256 of them stay verified on earlier reviews of the same tree. A follow-up settles the representation with a brief clarification.
    • MERGE actions, the SERVICE ROLE svc!role raw string and the GRANT OWNERSHIP class kind string.
    • 26 forms rejected on other causes (including the two SERVICE ROLE forms), listed in the comment.
  • Caveats:
    • 461 verified and 203 unverified forms repeat a list item with no second-item case.
    • Several docs pages changed after the pin (GRAPH_RUN_GROUP_ID, CREATE SECRET WITH TAG, NETWORK RULE MODE, CONVERT TO MANAGED). The pinned grammar stays; a re-pin is separate.
  • make check-syntax and make check-ci pass on the final head.

…le parsed negatives

Work in progress, not yet reviewed or gated.

- Operands the pages name as objects (role, schema, stage, ...) get the
  identifier class; contact purposes, privileges and similar bare words
  get a word class with a string-literal wrong-kind case; more numeric
  names (days, precision, seconds) get the integer class.
- Suppress notes that still said 'a valid operand' follow the new class
  notes, so no hand row is withdrawn.
- Derived negatives Snowflake parses and fails later are observed under
  the settle rule instead of expected rejections.

Verified: syntax_obligations_snowflake unit tests; obligation ids unchanged
for every surviving obligation.
…, CONNECTION and DCM PROJECT

Work in progress, not yet reviewed or gated.

- AlterObject.name is optional; ALTER USER without a name records none
  instead of an empty name.
- CREATE TAG (with ALLOWED_VALUES as one list), CREATE CONNECTION and
  CREATE DCM PROJECT build create_object trees instead of raw text.

Verified: cargo test -p sqlquack-syntax-snowflake; cargo check --workspace --tests.
…gatives

Work in progress, not yet reviewed or gated.
Work in progress, not yet reviewed or gated.
…nce names, LIKE patterns and tags as operands
…AG, and refuse reserved words as INSERT, MERGE, LIKE, SET and UNSET names

Work in progress, not yet reviewed or gated.
…, and pin their evidence

Work in progress, not yet reviewed or gated.
…view brief version 1

Work in progress, not yet gated.
…e-only re-reviews under review brief version 1

Work in progress, not yet gated.
…poses and grant targets as tree fields

Keeps an optional WITH before TAG or CONTACT in property kinds, records
FORCE as a property instead of an action suffix, keeps the quoting of
UNSET parameter names, marks string contact purposes, and gives grant
and revoke separate caller, bulk and container fields instead of packing
them into privilege names and the object kind.

Work in progress: cargo check passes for the Snowflake, Postgres,
BigQuery and Databricks crates; the crate test suites have not been run
and expectations are not yet updated.
…ction strings, and test the kept tree fields

An identifier left out between two dots, as the schema of d1..t1 or of
SNOWFLAKE..<class>, is now marked omitted, so it differs from an absent
part and still prints back as a..b. FORCE, CASCADE and RECOVERY after
policy and Openflow actions become properties. INHERITED and CALLER come
once, before the privilege list. A quoted stage name with blanks after
FROM @ in ALTER SERVICE settles from its receipt as an undocumented
outcome.

Work in progress: the Snowflake, Postgres, BigQuery, Databricks, DuckDB,
SQLite, SQL Server and ClickHouse syntax crate tests pass; coverage has
not been rebuilt.
…w brief version 1 and pin the ALTER SERVICE settle

Work in progress, not yet gated.
… was

The omitted marker on identifiers is withdrawn; the representation is
settled in a later change.

Work in progress: the syntax crate tests pass.
Resource monitor trigger percentages and CODE BUNDLE / NOTEBOOK PROJECT
FROM sources must be literals of the recorded kind, a COPY transform
column takes no string after a dot, and RECLUSTER MAX_SIZE parses a
string that Snowflake rejects only semantically.

Regenerate the ALTER SERVICE obligations, drop unobservable records whose
obligations no longer exist, and refresh the syntax baseline.
…ules

Move CREATE FUNCTION parsing out of the commands module, return a user
credential action's existence flags instead of mutating a parameter,
read a procedure's EXECUTE AS in its own helper, keep the template
operand switch with the other grammar constants, and parametrize the
derived limit test.

kvlonge commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Drops, rejection causes and known remaining items

1. Drops (43)

Failed the final review under brief v1 (22):

  • sf.classes.anomaly-insights.methods.set-account-notification-emails (cites the a..b empty part)
  • sf.classes.anomaly-insights.methods.set-monitor-notification-emails (cites the a..b empty part)
  • sf.classes.anomaly-insights.methods.set-org-notification-emails (cites the a..b empty part)
  • sf.classes.classification-profile.commands.show-classification-profile (cites the a..b empty part)
  • sf.ddl.create-iceberg-table-snowflake.table-masking-policy-row
  • sf.ddl.create-streamlit.create-streamlit (cites the a..b empty part)
  • sf.ddl.create-table.create-table-masking-policy
  • sf.dml.insert-multi-table.first-all-when-then (cites the a..b empty part)
  • sf.expressions.functions-table
  • sf.functions.data-agent-run-snowflake-cortex (cites the a..b empty part)
  • sf.functions.get-ai-evaluation-data-snowflake-local (cites the a..b empty part)
  • sf.functions.get-ai-observability-events-snowflake-local (cites the a..b empty part)
  • sf.functions.get-ai-observability-logs-snowflake-local (cites the a..b empty part)
  • sf.functions.get-ai-record-trace-snowflake-local (cites the a..b empty part)
  • sf.functions.parse-document-snowflake-cortex (cites the a..b empty part)
  • sf.query.asof-join (cites the a..b empty part)
  • sf.query.join.join
  • sf.query.join.join-2
  • sf.query.join.join.syntax
  • sf.security_admin.alter-account.set-feature-policy-for
  • sf.security_admin.alter-account.set-feature-policy-for-2
  • sf.security_admin.alter-organization-user-group.set-visibility-all-accounts

New cases still without receipts (19):

  • sf.classes.anomaly-insights.methods.adhoc-calculate-anomalies-from-config
  • sf.classes.anomaly-insights.methods.create-monitor
  • sf.classes.anomaly-insights.methods.update-monitor-config
  • sf.ddl.alter-application-package-release-channel.modify-release-channel-add
  • sf.ddl.alter-application-package-release-channel.modify-release-channel-drop
  • sf.ddl.alter-application-set-app-spec.connection-label-description-server-application
  • sf.ddl.create-database.create-database-from-backup
  • sf.ddl.create-dynamic-table.create-dynamic-table-from
  • sf.ddl.create-postgres-instance.fork
  • sf.ddl.create-schema.create-schema-from-backup
  • sf.ddl.create-table.create-table-from-backup
  • sf.functions.dmf-source-target-lag.alter-table-add-data
  • sf.security_admin.create-catalog-integration-delta-sharing
  • sf.security_admin.create-catalog-integration-open-catalog.catalog-name-rest-authentication-type-oauth
  • sf.security_admin.create-catalog-integration-rest
  • sf.security_admin.create-maintenance-policy
  • sf.security_admin.create-snapshot-policy
  • sf.security_admin.revoke-caller.inherited-caller-on-all
  • sf.utility.execute-job-service

Changed cases awaiting review (2):

  • sf.ddl.alter-service.fromspecification-fromspecificationtemplate
  • sf.dml.merge

2. Rejection causes other than the a..b empty part (26 forms)

  • sf.functions.task-history: (a) Docs Syntax block now ends with [, GRAPH_RUN_GROUP_ID => '' ] after GRAPH_ID; the catalogue grammar stops at [, GRAPH_ID => ''] ), so that option is missing (and has no obligations).
  • sf.query.join.join: (d) o052/o053 SELECT 1 FROM 'v1' JOIN t2 and FROM t1 JOIN 'v1' (Snowflake accepted) give table {"name":"v1","quoted":true}, identical to "v1"; the single-quoted string kind is lost. Stricter reading taken.
  • sf.query.join.join-2: (d) o033/o034 SELECT 1 FROM 'v1' JOIN t2 and FROM t1 JOIN 'v1' (Snowflake accepted) give table {"name":"v1","quoted":true}, identical to "v1"; the single-quoted string kind is lost. Stricter reading taken.
  • sf.query.join.join.syntax: (d) o060/o061 SELECT * FROM 'v1' JOIN t2 and FROM t1 JOIN 'v1' (Snowflake accepted) give table {"name":"v1","quoted":true}, identical to "v1"; the single-quoted string kind is lost. Stricter reading taken.
  • sf.security_admin.alter-account.set-feature-policy-for: (d) FOR ALL APPLICATIONS has no field of its own: it is folded into action "SET FEATURE POLICY FOR ALL APPLICATIONS", reordered ahead of the policy operand (o001). Stricter reading of a judgement call.
  • sf.security_admin.alter-account.set-feature-policy-for-2: (d) FOR ALL PERSONAL DATABASES has no field of its own: it is folded into action "SET FEATURE POLICY FOR ALL PERSONAL DATABASES", reordered ahead of the policy operand (o001). Stricter reading of a judgement call.
  • sf.security_admin.alter-organization-user-group.set-visibility-all-accounts: (d) The VISIBILITY property and its chosen branch land in the action string: o001/o007 action "SET VISIBILITY = ALL" with no property or value field, while o029 gives the same SQL tail as properties[{kind:VISIBILITY,value:ALL}].
  • sf.security_admin.alter-user.set: b: identifier_class o308 ALTER USER "v1 Q" SET ; and o309 ALTER USER "SELECT" SET ; are syntax_error (empty SET), so quoted/reserved acceptance is never shown and no tree keeps quoting. Stricter d: WORKLOAD_IDENTITY=(ANOMALY_DETECTION=TRUE) -> tuple[tuple[table,bool]], '=' lost
  • sf.security_admin.create-authentication-policy.define-authentication-policy: d (stricter reading): DEFINE AUTHENTICATION POLICY d1.s1.v1 ; gives create_object{or_alter:true,or_replace:false} with no DEFINE marker, so DEFINE is lost and the tree equals CREATE OR ALTER, a different statement per the docs prose. a (prose), b, c, outcomes pass.
  • sf.security_admin.grant-service-role: (d) stricter reading: o058 GRANT SERVICE ROLE "v1 Q"!v1 TO ROLE v1 gives securable {"name":""v1 Q"!v1","quoted":false}; same o060/o059/o061. Service and service-role names stay in one raw string, and quoted=false contradicts the quoted part. (a)-(c) and outcomes pass.
  • sf.security_admin.revoke-service-role: d (stricter reading): <service_name> and <service_role_name> collapse into one raw string with quoted:false, e.g. o058 REVOKE SERVICE ROLE "v1 Q"!v1 FROM ROLE v1 -> securable {name:'"v1 Q"!v1',quoted:false}; quoted part has no flag. a, b, c and outcome hold.
  • sf.transaction_session.set.set: d: quoting lost on : o016 SET "v1 Q" = 1 -> identifier {name:'v1 Q',quoted:false}; o017 SET "SELECT" = 1 -> {name:'SELECT',quoted:false}, so it reads the same as unquoted. a (first statement of a two-statement Syntax block), b, c and outcome hold.
  • sf.ddl.alter-external-volume.add-storage-location-name-cloudproviderparams: (b) o036/o037/o038 (S3) and o057/o058 (GCS) alternative cases are all ENCRYPTION = ( ), none takes its TYPE branch (strict reading). (d) also: options become generic tuples with keywords as tables, e.g. {"table":{"name":"STORAGE_PROVIDER"}}.
  • sf.ddl.alter-iceberg-table-convert-to-managed: (a) catalogue has [ BASE_LOCATION = '<directory_for_table_files>' ] but docs Syntax reads [ { FILE_PATH | BASE_LOCATION } = '<directory_for_table_files>' ] [ DROP_FROM_SOURCE_CATALOG = { TRUE | FALSE } ]; FILE_PATH and DROP_FROM_SOURCE_CATALOG are missing.
  • sf.ddl.create-function.create-function-returns-table: d: RETURNS TABLE kept as raw string. o032 'RETURNS TABLE ( v1 INT )' -> return_type null, returns_table_body:"TABLE (v1 INT)"; o142 '( "v1 Q" INT )' -> "TABLE (v1 Q INT)" and o143 '"SELECT"' -> "TABLE (SELECT INT)" lose quoting; o041 -> return_type custom name "TABLE(v1 INT, v1 INT)".
  • sf.ddl.create-function.create-function-returns-table.javascript-handler: d: RETURNS TABLE kept as raw string. o030 'RETURNS TABLE ( v1 INT )' -> returns_table_body:"TABLE (v1 INT)", return_type null; o086 '"v1 Q"' -> "TABLE (v1 Q INT)" and o087 '"SELECT"' -> "TABLE (SELECT INT)" lose quoting; o039 -> return_type custom "TABLE(v1 INT, v1 INT)".
  • sf.ddl.create-function.create-function-returns-table.sql-handler: d: RETURNS TABLE kept as raw string. o030 'RETURNS TABLE ( v1 INT ) AS' -> returns_table_body:"TABLE (v1 INT)", return_type null; o077 '"v1 Q"' -> "TABLE (v1 Q INT)" and o078 '"SELECT"' -> "TABLE (SELECT INT)" lose quoting; o039 -> return_type custom "TABLE(v1 INT, v1 INT)".
  • sf.ddl.create-iceberg-table-snowflake.table-masking-policy-row: (d) o015 [ WITH ]: 'v1 INT DEFAULT 1 WITH MASKING POLICY v1' and o016 'v1 INT MASKING POLICY v1' give the same options:[masking_policy_column_constraint{this:v1}] with no WITH flag, though table-level 'WITH ROW ACCESS POLICY' keeps WITH. Strict reading.
  • sf.ddl.create-table.create-table-masking-policy: (d) o011 [ WITH ]: 'v1 INT WITH MASKING POLICY v1' and o012 'v1 INT MASKING POLICY v1' give the same column options:[masking_policy_column_constraint{this:v1}]; WITH is lost (kept only for ROW ACCESS POLICY). Strict reading.
  • sf.ddl.create-tag.define-tag: (d) o001 'DEFINE TAG d1.s1.v1 ;' gives create_object{kind:TAG, or_alter:true, source:""}: the DEFINE verb is not recorded anywhere, so the tree cannot be told apart from CREATE OR ALTER TAG. Prose grammar and cases are otherwise fine. Strict reading.
  • sf.expressions.data-types-datetime.timestamp-variations: (a) page prose lists DATETIME as a TIMESTAMP_NTZ synonym; grammar omits it. (d) o011 TIMESTAMP_LTZ ( 9 ) -> {"data_type":"custom","name":"TIMESTAMPLTZ(9)"}: precision kept in a raw string, not a field.
  • sf.expressions.data-types-numeric.number: (d) o002 NUMBER ( 9 , 2 ) -> {"data_type":"custom","name":"NUMBER(9, 2)"}; o007 -> "NUMBER(9)": precision and scale kept in a raw type-name string, not their own fields.
  • sf.expressions.data-types-text.string: (d), stricter reading: o008 NCHAR VARYING and o007 CHAR VARYING give the same tree {"data_type":"var_char","length":null}, so the NCHAR keyword the SQL wrote is lost (NVARCHAR keeps its name). Same-type synonym, so this could also be approved.
  • sf.expressions.functions-table: (d) o040 SELECT * FROM 'v1' , TABLE ( FLATTEN ( ) ) -> table name {"name":"v1","quoted":true}: the string literal becomes a double-quoted identifier, the same tree as "v1". Stricter reading taken.
  • sf.functions.complete-task-graphs: (a) the docs Syntax block ends with [, GRAPH_RUN_GROUP_ID => '' ] after ERROR_ONLY; the catalogue grammar omits that argument, so there are no obligations for it.
  • sf.functions.current-task-graphs: (a) the docs Syntax block ends with [, GRAPH_RUN_GROUP_ID => '' ] after GRAPH_NAME; the catalogue grammar omits that argument, so there are no obligations for it.

3. Known remaining items

  • a..b omitted part: serialized as the plain "", as on main. Brief v1 (d) asks for null, so 297 rejected forms cite it. 256 of them stay verified on earlier reviews whose tree is unchanged by fingerprint. Brief v1 stays frozen for this change; a follow-up settles the representation together with a brief clarification.
  • MERGE actions (sf.dml.merge): the THEN actions are raw strings in positional tuples (for example UPDATE ALL BY NAME), and VALUES is dropped from an INSERT action. It was rejected in an earlier round and is not fixed here.
  • SERVICE ROLE (grant-/revoke-service-role): <service>!<role> stays one raw string with quoted: false.
  • GRANT OWNERSHIP (to-role-database-role): <class_name> lands in the object-kind string, case-folded and without a quoted flag (ON "v1 Q" v1 gives kind "\"v1 Q\"").

4. Docs drift after the pin

The catalogue grammar follows the pinned page. Each live page below now has a different syntax-block hash than its pin in sources.toml. This change keeps the pinned grammar; a re-pin is a separate change. Until then, the new options have no obligations. The forms the reviewers rejected on (a) for this reason stay unverified.

  • TASK_HISTORY, COMPLETE_TASK_GRAPHS, CURRENT_TASK_GRAPHS: the live Syntax blocks add [, GRAPH_RUN_GROUP_ID => '<string>' ].
  • CREATE SECRET (all seven type forms, and CREATE OR ALTER SECRET by reference): every type block now ends with [ WITH TAG ( <tag_name> = '<tag_value>' [ , ... ] ) ].
  • CREATE NETWORK RULE: the live CREATE block prints [ MODE = { INGRESS | INTERNAL_STAGE | SNOWFLAKE_MANAGED_STORAGE_VOLUME | EGRESS } ] as optional; the pinned grammar has it as required.
  • ALTER ICEBERG TABLE ... CONVERT TO MANAGED: the live block has [ { FILE_PATH | BASE_LOCATION } = '...' ] and a new [ DROP_FROM_SOURCE_CATALOG = { TRUE | FALSE } ].

Not drift: the CREATE SECURITY INTEGRATION (Snowflake OAuth) page is unchanged since its pin. Its -- Required when OAUTH_CLIENT=LOOKER line is a docs comment, which the catalogue leaves out by design.

@kvlonge
kvlonge enabled auto-merge November 14, 2026 02:01
@kvlonge
kvlonge added this pull request to the merge queue Nov 14, 2026
Merged via the queue into main with commit f24e77d Nov 14, 2026
9 checks passed
@kvlonge
kvlonge deleted the feat/syntax-pool-24 branch November 14, 2026 03:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant