Repository navigation
Commit 6c0d395
authored
feat: format and lint with SQLQuack instead of Polyglot (#38)
* wip(format): lay out Snowflake and DuckDB with SQLQuack's formatter
Snowflake and DuckDB requests now go to sqlquack-format, pinned by git
revision, which lays out SQL on SQLQuack's own tokenizer and parser and
checks token identity and idempotency itself. Other dialects still use
the Polyglot layout oracle.
Work in progress: 6 formatter tests still expect Polyglot's layout.
* wip(format): gate Snowflake and DuckDB on SQLQuack's layout, and drop the Polyglot AST API
SQLClean's corpus check now expects SQLQuack's named departures, listed
with their reasons in sqlquack_departures.json: 15 DuckDB and 13
Snowflake queries that SQLQuack lays out or refuses differently. Called
names fold to upper case as SQLBuild's query fingerprint folds them,
except in unaliased Snowflake select items, whose text names their
column. The re-print count test runs on PostgreSQL, which still uses
the Polyglot layout.
RuleContext.polyglot_ast() and the ast extra are removed.
Work in progress: CTE macro placeholders are not laid out yet.
* wip(lint): lint and collect statement facts with SQLQuack instead of Polyglot
The lint entry points (lint, lint_json, lint_batch_json, parse_sql,
catalogue, default_rules, cross_join_subject, fill_finding_path and
validate_dialect) now call sqlquack-lint, the port of this lint engine
and its quality Rules onto SQLQuack's tokens and trees. The lint request,
response and parse types are SQLQuack's, named in sql_lint::types; the
format types stay in sql_lint::models. The placeholder constants take
sqlquack-lint's values. sql_quality is removed: its Rules run inside
sqlquack-lint.
sql_facts reads SQLQuack's tokens and trees through parse_sql, with the
same collection code.
Lint for a dialect SQLQuack does not parse yet, such as T-SQL or MySQL,
answers that lint is unavailable rather than reporting findings, so the
T-SQL bare UNION test is ignored until that parser lands.
The formatter is unchanged and still reads Polyglot; the one lint helper
it used, token_depths, stays in sql_lint/_helpers/engine.rs.
SQLQuack is pinned at wip/lint-port 4f9ac36d8.
* wip(format): lay out around CTE macro placeholders, and fingerprint with SQLQuack's tokens
* wip(facts): keep Snowflake date parts out of the column facts
SQLQuack's Snowflake parse keeps DATEPART, LAST_DAY and DATE_DIFF as the
calls written, so their bare date part, such as day in DATEPART(day, ts),
parsed as a column, where Polyglot lowered the call and read it as a unit.
Statement facts now leave that argument out of the columns: any bare name
for DATEPART and LAST_DAY, and only a known unit name for DATE_DIFF, as
Polyglot did.
Against SQLClean on Polyglot, the Snowflake facts of 12,313 inputs both
parse now differ on 6, all JSON_OBJECTAGG(t.x), whose argument SQLQuack
reports as a column and Polyglot dropped; DuckDB's 5,352 are identical.
* wip(facts): report facts as unavailable for a dialect SQLQuack does not parse
file_facts now answers a dialect SQLQuack does not parse yet, such as
MySQL, with no statements and an unavailable reason, the way lint's
response does, instead of an error; an unknown dialect is still an error.
statement_ends answers None for such a dialect. Both ask lint whether the
dialect is unavailable, so facts and lint always agree.
The CLI keeps failing those files for now: the custom Rule host turns an
unavailable reason into its error, and format_source turns None into one,
until the per-file unavailable report replaces them.
* wip(facts): read Postgres facts as written, and pin SQLQuack with the Postgres lint slice
SQLQuack is now pinned at the lint slice that parses PostgreSQL and
Generic natively (wip/lint-postgres bd308904a), so their lint and facts
are available.
The native PostgreSQL parser folds unquoted names to lower case, so the
facts read table, column, CTE and alias names from the source as written.
A column's span now starts at its qualifier when the parser located the
name alone, as for a subscripted column. The CREATE and ALTER statements
the native tree keeps as a generic object take the kind Polyglot's tree
gave them, such as create_index or alter_table.
The Rule test harness fails a case whose dialect has no facts, rather
than evaluating the Rule over an empty file.
* wip(tokens): fingerprint with sqlquack-fingerprint, as SQLBuild does
Dialects SQLQuack fingerprints use its shared function; the others keep the
Polyglot token stream SQLBuild also uses for them. A comma-less run of CTE
macro placeholders is now separated by blank lines like any other CTE.
* wip(format): report formatting unavailable where SQLQuack has no formatter, and drop the Polyglot oracle
Dialects without a native SQLQuack formatter now return their SQL unchanged
with an unavailable reason instead of a Polyglot layout, so the oracle
formatter and its token layout, printed layout and syntax protection are
removed. Fingerprints use SQLQuack's generic tokens for dialects without
their own tokenizer, as SQLBuild does; the SQL Server golden changes to the
generic digest. The recording demo's custom Rule reads statement facts
instead of the removed AST.
* wip(tokens): fold SQL tokens with SQLQuack's rules instead of Polyglot's catalogues
* wip(cli): report files in a dialect SQLQuack does not parse instead of failing
check lists each such file once as a warning, and as unavailable in JSON and
SARIF, without counting it as a problem or running custom Rules on it. format
and fix leave the file unchanged and do not fail.
* wip(format): name every fingerprint and unavailable-corpus test case, and move the format cases beside the layout rules
* wip(facts): name utility and DROP statement kinds from the native tree
Maps the native utility_statement and drop_object kinds back to the
statement kinds SQLClean reported before (describe, set_statement,
transaction, drop_table and so on). Pins SQLQuack to the lint-postgres
landing head provisionally; the shared rev is still to be agreed.
* wip(cli): read SQL without a leading byte order mark and write it back
Editors such as SSMS save SQL with a UTF-8 byte order mark. check, format and
fix now read the SQL after it, so every dialect lints and formats the file,
and format and fix write the mark back unchanged. Not yet built or tested.
* wip(facts): leave the SQLQuack pin to the release head
The statement-kind mapping builds on the shared SQLQuack rev already
pinned here, so the provisional pin bump is dropped.
* wip(deps): trial pin of SQLQuack at the train tip 19c51fa2
Not releasable: 19c51fa2 is not on SQLQuack main yet. Two test updates follow
SQLQuack's Snowflake grammar checks: the format corpus expects UNNEST refused
(Snowflake has no UNNEST), and the no-harness empty-fixture case calls the
relation through TABLE( ) as Snowflake requires.
* wip(format): pass SQLQuack its formatter dialect
SQLQuack's FormatOptions now takes a FormatDialect; SQLClean keeps its syntax dialect for the CTE macro pass and maps it for the layout.
* wip(format): lay out T-SQL with SQLQuack's SQL Server formatter
T-SQL requests go to SQLQuack's SQL Server formatter; the CTE macro pass, which needs the shared tokenizer, is skipped, so a placeholder SQLQuack cannot parse is refused. The corpus test gates T-SQL at 52 formatted and 23 refused, with SQLQuack's named departures.
* fix(facts): report comma and ON-nested joins as Polyglot did
The native tree keeps a comma relation that follows an explicit join in
the FROM list, and a join nested by its ON clauses inside the outer join's
relation, so facts lost both. A comma relation after the first explicit
join is now an IMPLICIT join, and the joins of an unparenthesized nested
join are reported, with the outer join named by its first table. A
parenthesized join group still reports only the join to it, as before.
* fix(facts): collect joins in plain loops and assert the expected joins directly
* build(deps): pin SQLQuack at main edfa30b2
Every SQLQuack crate resolves to the one main commit edfa30b26a0bab86661f96490b86a4f24b92db0a.1 parent c7f96f9 commit 6c0d395
133 files changed
Lines changed: 2262 additions & 13658 deletions
File tree
- crates
- sqlclean-cli
- skill/references/docs
- custom-rules
- src
- checking
- _helpers
- command/_helpers
- discovery
- _helpers
- fixing/_helpers
- formatting
- main
- reporting
- _helpers
- main
- sqlclean
- src
- sql_facts
- _helpers
- main
- tests
- sql_lint
- _helpers
- main
- tests
- fixtures
- sql_quality
- _helpers
- main
- tests
- sql_tokens
- _helpers
- main
- reserved_keywords
- tests
- src/sqlclean
- rule_testing
- _helpers
- rules
- tests/e2e/src/sqlclean/hosting
- website
- recordings
- src/content/docs/docs
- custom-rules
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | 24 | | |
36 | 25 | | |
37 | 26 | | |
| |||
40 | 29 | | |
41 | 30 | | |
42 | 31 | | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
43 | 37 | | |
44 | 38 | | |
45 | 39 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
123 | 123 | | |
124 | 124 | | |
125 | 125 | | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
126 | 135 | | |
127 | 136 | | |
128 | 137 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | 82 | | |
87 | 83 | | |
88 | 84 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
| 8 | + | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
| 11 | + | |
| 12 | + | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
19 | | - | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
| |||
33 | 34 | | |
34 | 35 | | |
35 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
36 | 47 | | |
37 | 48 | | |
38 | 49 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
189 | 189 | | |
190 | 190 | | |
191 | 191 | | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
192 | 196 | | |
193 | 197 | | |
194 | 198 | | |
195 | 199 | | |
196 | | - | |
| 200 | + | |
197 | 201 | | |
198 | 202 | | |
199 | 203 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
14 | | - | |
15 | | - | |
| 14 | + | |
| 15 | + | |
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| |||
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
52 | | - | |
| 52 | + | |
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
62 | 72 | | |
63 | 73 | | |
64 | 74 | | |
65 | 75 | | |
| 76 | + | |
66 | 77 | | |
67 | 78 | | |
68 | 79 | | |
| |||
97 | 108 | | |
98 | 109 | | |
99 | 110 | | |
| 111 | + | |
100 | 112 | | |
101 | 113 | | |
102 | 114 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
| 7 | + | |
6 | 8 | | |
7 | 9 | | |
8 | 10 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
62 | 69 | | |
63 | 70 | | |
64 | 71 | | |
65 | 72 | | |
66 | 73 | | |
| 74 | + | |
| 75 | + | |
67 | 76 | | |
68 | 77 | | |
69 | 78 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
| |||
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| 23 | + | |
22 | 24 | | |
23 | | - | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
24 | 30 | | |
25 | 31 | | |
0 commit comments