Skip to content

Commit 6c0d395

Browse files
authored
feat: format and lint with SQLQuack instead of Polyglot (#38)
* wip(format): lay out Snowflake and DuckDB with SQLQuack's formatter Snowflake and DuckDB requests now go to sqlquack-format, pinned by git revision, which lays out SQL on SQLQuack's own tokenizer and parser and checks token identity and idempotency itself. Other dialects still use the Polyglot layout oracle. Work in progress: 6 formatter tests still expect Polyglot's layout. * wip(format): gate Snowflake and DuckDB on SQLQuack's layout, and drop the Polyglot AST API SQLClean's corpus check now expects SQLQuack's named departures, listed with their reasons in sqlquack_departures.json: 15 DuckDB and 13 Snowflake queries that SQLQuack lays out or refuses differently. Called names fold to upper case as SQLBuild's query fingerprint folds them, except in unaliased Snowflake select items, whose text names their column. The re-print count test runs on PostgreSQL, which still uses the Polyglot layout. RuleContext.polyglot_ast() and the ast extra are removed. Work in progress: CTE macro placeholders are not laid out yet. * wip(lint): lint and collect statement facts with SQLQuack instead of Polyglot The lint entry points (lint, lint_json, lint_batch_json, parse_sql, catalogue, default_rules, cross_join_subject, fill_finding_path and validate_dialect) now call sqlquack-lint, the port of this lint engine and its quality Rules onto SQLQuack's tokens and trees. The lint request, response and parse types are SQLQuack's, named in sql_lint::types; the format types stay in sql_lint::models. The placeholder constants take sqlquack-lint's values. sql_quality is removed: its Rules run inside sqlquack-lint. sql_facts reads SQLQuack's tokens and trees through parse_sql, with the same collection code. Lint for a dialect SQLQuack does not parse yet, such as T-SQL or MySQL, answers that lint is unavailable rather than reporting findings, so the T-SQL bare UNION test is ignored until that parser lands. The formatter is unchanged and still reads Polyglot; the one lint helper it used, token_depths, stays in sql_lint/_helpers/engine.rs. SQLQuack is pinned at wip/lint-port 4f9ac36d8. * wip(format): lay out around CTE macro placeholders, and fingerprint with SQLQuack's tokens * wip(facts): keep Snowflake date parts out of the column facts SQLQuack's Snowflake parse keeps DATEPART, LAST_DAY and DATE_DIFF as the calls written, so their bare date part, such as day in DATEPART(day, ts), parsed as a column, where Polyglot lowered the call and read it as a unit. Statement facts now leave that argument out of the columns: any bare name for DATEPART and LAST_DAY, and only a known unit name for DATE_DIFF, as Polyglot did. Against SQLClean on Polyglot, the Snowflake facts of 12,313 inputs both parse now differ on 6, all JSON_OBJECTAGG(t.x), whose argument SQLQuack reports as a column and Polyglot dropped; DuckDB's 5,352 are identical. * wip(facts): report facts as unavailable for a dialect SQLQuack does not parse file_facts now answers a dialect SQLQuack does not parse yet, such as MySQL, with no statements and an unavailable reason, the way lint's response does, instead of an error; an unknown dialect is still an error. statement_ends answers None for such a dialect. Both ask lint whether the dialect is unavailable, so facts and lint always agree. The CLI keeps failing those files for now: the custom Rule host turns an unavailable reason into its error, and format_source turns None into one, until the per-file unavailable report replaces them. * wip(facts): read Postgres facts as written, and pin SQLQuack with the Postgres lint slice SQLQuack is now pinned at the lint slice that parses PostgreSQL and Generic natively (wip/lint-postgres bd308904a), so their lint and facts are available. The native PostgreSQL parser folds unquoted names to lower case, so the facts read table, column, CTE and alias names from the source as written. A column's span now starts at its qualifier when the parser located the name alone, as for a subscripted column. The CREATE and ALTER statements the native tree keeps as a generic object take the kind Polyglot's tree gave them, such as create_index or alter_table. The Rule test harness fails a case whose dialect has no facts, rather than evaluating the Rule over an empty file. * wip(tokens): fingerprint with sqlquack-fingerprint, as SQLBuild does Dialects SQLQuack fingerprints use its shared function; the others keep the Polyglot token stream SQLBuild also uses for them. A comma-less run of CTE macro placeholders is now separated by blank lines like any other CTE. * wip(format): report formatting unavailable where SQLQuack has no formatter, and drop the Polyglot oracle Dialects without a native SQLQuack formatter now return their SQL unchanged with an unavailable reason instead of a Polyglot layout, so the oracle formatter and its token layout, printed layout and syntax protection are removed. Fingerprints use SQLQuack's generic tokens for dialects without their own tokenizer, as SQLBuild does; the SQL Server golden changes to the generic digest. The recording demo's custom Rule reads statement facts instead of the removed AST. * wip(tokens): fold SQL tokens with SQLQuack's rules instead of Polyglot's catalogues * wip(cli): report files in a dialect SQLQuack does not parse instead of failing check lists each such file once as a warning, and as unavailable in JSON and SARIF, without counting it as a problem or running custom Rules on it. format and fix leave the file unchanged and do not fail. * wip(format): name every fingerprint and unavailable-corpus test case, and move the format cases beside the layout rules * wip(facts): name utility and DROP statement kinds from the native tree Maps the native utility_statement and drop_object kinds back to the statement kinds SQLClean reported before (describe, set_statement, transaction, drop_table and so on). Pins SQLQuack to the lint-postgres landing head provisionally; the shared rev is still to be agreed. * wip(cli): read SQL without a leading byte order mark and write it back Editors such as SSMS save SQL with a UTF-8 byte order mark. check, format and fix now read the SQL after it, so every dialect lints and formats the file, and format and fix write the mark back unchanged. Not yet built or tested. * wip(facts): leave the SQLQuack pin to the release head The statement-kind mapping builds on the shared SQLQuack rev already pinned here, so the provisional pin bump is dropped. * wip(deps): trial pin of SQLQuack at the train tip 19c51fa2 Not releasable: 19c51fa2 is not on SQLQuack main yet. Two test updates follow SQLQuack's Snowflake grammar checks: the format corpus expects UNNEST refused (Snowflake has no UNNEST), and the no-harness empty-fixture case calls the relation through TABLE( ) as Snowflake requires. * wip(format): pass SQLQuack its formatter dialect SQLQuack's FormatOptions now takes a FormatDialect; SQLClean keeps its syntax dialect for the CTE macro pass and maps it for the layout. * wip(format): lay out T-SQL with SQLQuack's SQL Server formatter T-SQL requests go to SQLQuack's SQL Server formatter; the CTE macro pass, which needs the shared tokenizer, is skipped, so a placeholder SQLQuack cannot parse is refused. The corpus test gates T-SQL at 52 formatted and 23 refused, with SQLQuack's named departures. * fix(facts): report comma and ON-nested joins as Polyglot did The native tree keeps a comma relation that follows an explicit join in the FROM list, and a join nested by its ON clauses inside the outer join's relation, so facts lost both. A comma relation after the first explicit join is now an IMPLICIT join, and the joins of an unparenthesized nested join are reported, with the outer join named by its first table. A parenthesized join group still reports only the join to it, as before. * fix(facts): collect joins in plain loops and assert the expected joins directly * build(deps): pin SQLQuack at main edfa30b2 Every SQLQuack crate resolves to the one main commit edfa30b26a0bab86661f96490b86a4f24b92db0a.
1 parent c7f96f9 commit 6c0d395

133 files changed

Lines changed: 2262 additions & 13658 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 78 additions & 44 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 5 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -21,17 +21,6 @@ include_dir = "0.7"
2121
ignore = "0.4"
2222
tempfile = "3"
2323
toml = "0.9"
24-
# The same Polyglot source and revision as SQLBuild, so a build that contains both compiles it once.
25-
polyglot-sql = { git = "https://github.com/sqlstack-io/polyglot.git", rev = "8767d688c5b504ad609645e27ab77a260d91c5a1", default-features = false, features = [
26-
"ast-tools",
27-
"builder",
28-
"all-dialects",
29-
"semantic",
30-
"stacker",
31-
] }
32-
polyglot-sql-function-catalogs = { git = "https://github.com/sqlstack-io/polyglot.git", rev = "8767d688c5b504ad609645e27ab77a260d91c5a1", default-features = false, features = [
33-
"dialect-duckdb",
34-
] }
3524
rayon = "1"
3625
regex = "1"
3726
rustc-hash = "2"
@@ -40,6 +29,11 @@ serde_json = { version = "1", features = ["preserve_order"] }
4029
sha2 = "0.10"
4130
similar = "2"
4231
sqlclean = { path = "crates/sqlclean" }
32+
# SQLQuack's formatter and lint engine, and the tokens and trees they read, which replace Polyglot.
33+
sqlquack-fingerprint = { git = "https://github.com/sqlstack-io/sqlquack.git", rev = "edfa30b26a0bab86661f96490b86a4f24b92db0a" }
34+
sqlquack-format = { git = "https://github.com/sqlstack-io/sqlquack.git", rev = "edfa30b26a0bab86661f96490b86a4f24b92db0a" }
35+
sqlquack-lint = { git = "https://github.com/sqlstack-io/sqlquack.git", rev = "edfa30b26a0bab86661f96490b86a4f24b92db0a" }
36+
sqlquack-syntax = { git = "https://github.com/sqlstack-io/sqlquack.git", rev = "edfa30b26a0bab86661f96490b86a4f24b92db0a" }
4337

4438
[workspace.lints.rust]
4539
unsafe_code = "forbid"

‎crates/sqlclean-cli/skill/references/docs/configuration.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,15 @@ SELECT [order id] FROM orders
123123
error[SQLCLEAN999]: SQL could not be parsed as duckdb: Parse error at line 1, column 17: Expected RBracket, got Var ('id') near [SELECT [order id] FROM orders]
124124
```
125125

126+
SQLClean reads SQL with SQLQuack's parsers. A file in a dialect SQLQuack does not parse yet, such
127+
as `mysql`, is not checked, fixed or formatted: `check` reports it once as a warning (in JSON, under
128+
`unavailable`), and it never fails a command or counts as a problem.
129+
130+
```text
131+
warning: SQL lint is unavailable for the mysql dialect: SQLQuack does not parse it yet
132+
--> reports/q.sql
133+
```
134+
126135
Supported dialects, case-insensitive:
127136

128137
| Dialect | Also accepted |

‎crates/sqlclean-cli/skill/references/docs/custom-rules/statement-facts.md‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -79,10 +79,6 @@ One statement of a file, without its terminating semicolon.
7979

8080
Each tuple is in source order and includes nodes inside CTEs and subqueries.
8181

82-
`statement.polyglot_ast()` returns the full Polyglot syntax tree of the statement, parsed on first
83-
use. It needs the `ast` extra: `pip install "sqlclean[ast]"`. Without it, the call fails with
84-
`polyglot_ast() needs the Polyglot SQL package: install the ast extra: pip install 'sqlclean[ast]'`.
85-
8682
## Cte
8783

8884
A common table expression, located at its name.

‎crates/sqlclean-cli/src/checking/_helpers/builtin.rs‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,18 +5,19 @@ use std::collections::BTreeSet;
55
use sqlclean::sql_lint::constants::LINT_API_VERSION;
66
use sqlclean::sql_lint::main::fill_finding_path::fill_finding_path;
77
use sqlclean::sql_lint::main::lint::lint;
8-
use sqlclean::sql_lint::models::{LintDiagnostic, LintRequest};
8+
use sqlclean::sql_lint::types::{LintDiagnostic, LintRequest};
99

1010
use crate::checking::constants::{
11-
PARSE_ERROR_CODE, PARSE_ERROR_MESSAGE, PARSE_ERROR_REMEDIATION, UNSAFE_FIX_BUILTIN_RULES,
11+
PARSE_ERROR_CODE, PARSE_ERROR_MESSAGE, PARSE_ERROR_REMEDIATION, UNAVAILABLE_CODE,
12+
UNSAFE_FIX_BUILTIN_RULES,
1213
};
1314
use crate::checking::models::{Finding, FixSafety};
1415
use crate::config::constants::CONFIG_FILE_NAME;
1516
use crate::config::models::Thresholds;
1617
use crate::discovery::models::SourceFile;
1718
use crate::positions::models::Edit;
1819

19-
/// The built-in findings of `file`, or one parse-error finding when the engine rejects it.
20+
/// The built-in findings of `file`, or its one parse-error finding or unavailable-dialect marker.
2021
pub(crate) fn builtin_findings(
2122
file: &SourceFile,
2223
codes: &BTreeSet<String>,
@@ -33,6 +34,16 @@ pub(crate) fn builtin_findings(
3334
..LintRequest::default()
3435
};
3536
match lint(request) {
37+
Ok(response) if response.unavailable.is_some() => vec![Finding {
38+
code: UNAVAILABLE_CODE.to_owned(),
39+
path: file.path.clone(),
40+
start: 0,
41+
end: 0,
42+
message: response.unavailable.unwrap_or_default(),
43+
remediation: String::new(),
44+
edits: Vec::new(),
45+
fix: None,
46+
}],
3647
Ok(response) => response
3748
.diagnostics
3849
.into_iter()

‎crates/sqlclean-cli/src/checking/_helpers/custom.rs‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -189,11 +189,15 @@ fn file_key(
189189
}
190190

191191
fn host_file(file: &SourceFile, rules: Vec<String>) -> Result<HostFile, String> {
192+
let facts = file_facts(&file.text, &file.dialect)?;
193+
if let Some(reason) = facts.unavailable {
194+
return Err(reason);
195+
}
192196
Ok(HostFile {
193197
path: file.path.clone(),
194198
source: file.text.clone(),
195199
dialect: file.dialect.clone(),
196-
facts: file_facts(&file.text, &file.dialect)?,
200+
facts,
197201
rules,
198202
})
199203
}

‎crates/sqlclean-cli/src/checking/_helpers/orchestration.rs‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,8 @@ use crate::checking::_helpers::catalogue::{builtin_rules, custom_rules, custom_s
1111
use crate::checking::_helpers::custom::{CustomEvaluation, custom_findings};
1212
use crate::checking::_helpers::exemptions::apply_exemptions;
1313
use crate::checking::_helpers::selection::selected_codes;
14-
use crate::checking::constants::{CUSTOM_RULE_PREFIX, PARSE_ERROR_CODE};
15-
use crate::checking::models::{CheckOptions, CheckOutcome, Finding, RuleInfo};
14+
use crate::checking::constants::{CUSTOM_RULE_PREFIX, PARSE_ERROR_CODE, UNAVAILABLE_CODE};
15+
use crate::checking::models::{CheckOptions, CheckOutcome, Finding, RuleInfo, UnavailableFile};
1616
use crate::config::models::Config;
1717
use crate::discovery::models::SourceFile;
1818

@@ -49,7 +49,7 @@ pub(crate) fn check_sources(
4949
.cloned()
5050
.collect();
5151
let selected: BTreeSet<String> = selected_codes(&all, select, &ignore)?;
52-
let (mut findings, updated) = run_builtins(
52+
let (builtin, updated) = run_builtins(
5353
&BuiltinRun {
5454
config,
5555
files,
@@ -59,10 +59,21 @@ pub(crate) fn check_sources(
5959
cache,
6060
);
6161
let mut cache: ResultCache = updated;
62+
let (markers, mut findings): (Vec<Finding>, Vec<Finding>) = builtin
63+
.into_iter()
64+
.partition(|finding| finding.code == UNAVAILABLE_CODE);
65+
let unavailable: Vec<UnavailableFile> = markers
66+
.into_iter()
67+
.map(|marker| UnavailableFile {
68+
path: marker.path,
69+
reason: marker.message,
70+
})
71+
.collect();
6272
let unparsed: HashSet<&str> = findings
6373
.iter()
6474
.filter(|finding| finding.code == PARSE_ERROR_CODE)
6575
.map(|finding| finding.path.as_str())
76+
.chain(unavailable.iter().map(|file| file.path.as_str()))
6677
.collect();
6778
let parsed: Vec<&SourceFile> = files
6879
.iter()
@@ -97,6 +108,7 @@ pub(crate) fn check_sources(
97108
Ok(CheckOutcome {
98109
findings: kept,
99110
stale_exceptions,
111+
unavailable,
100112
})
101113
}
102114

‎crates/sqlclean-cli/src/checking/constants.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ pub const PARSE_ERROR_CODE: &str = "SQLCLEAN999";
33
pub(crate) const PARSE_ERROR_MESSAGE: &str = "SQL could not be parsed";
44
pub(crate) const PARSE_ERROR_REMEDIATION: &str =
55
"Fix the syntax error, or set this file's dialect in sqlclean.toml.";
6+
/// Marks, in the built-in results and their cache, a file whose dialect SQLQuack does not parse.
7+
pub(crate) const UNAVAILABLE_CODE: &str = "SQLCLEAN998";
68
pub(crate) const CROSS_JOIN_CODE: &str = "SQLCLEAN038";
79
/// Custom Rule codes start with this prefix.
810
pub const CUSTOM_RULE_PREFIX: &str = "XSQLCLEAN";

‎crates/sqlclean-cli/src/checking/models.rs‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,11 +59,20 @@ pub struct StaleException {
5959
pub relation: Option<String>,
6060
}
6161

62+
/// A file not checked because SQLQuack does not parse its dialect yet; a notice, not a problem.
63+
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
64+
pub struct UnavailableFile {
65+
pub path: String,
66+
pub reason: String,
67+
}
68+
6269
/// Every finding of one check, sorted by path, position and code.
6370
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize)]
6471
pub struct CheckOutcome {
6572
pub findings: Vec<Finding>,
6673
pub stale_exceptions: Vec<StaleException>,
74+
/// Files whose dialect has no SQLQuack parser, by path; they never fail the check.
75+
pub unavailable: Vec<UnavailableFile>,
6776
}
6877

6978
/// What a check evaluates beyond the configuration.

‎crates/sqlclean-cli/src/command/_helpers/diffs.rs‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ use std::io::Write;
55

66
use similar::TextDiff;
77

8+
use crate::discovery::constants::UTF8_BOM;
89
use crate::discovery::models::SourceFile;
910

1011
const CONTEXT_LINES: usize = 3;
@@ -19,7 +20,12 @@ pub(crate) fn print_diff(file: &SourceFile, changed: &str) {
1920
let _ = std::io::stdout().write_all(rendered.as_bytes());
2021
}
2122

23+
/// Write `changed` over `file`, with the byte order mark it was read with.
2224
pub(crate) fn write_file(file: &SourceFile, changed: &str) -> Result<(), String> {
23-
fs::write(&file.absolute, changed)
25+
let text: String = match file.bom {
26+
true => format!("{UTF8_BOM}{changed}"),
27+
false => changed.to_string(),
28+
};
29+
fs::write(&file.absolute, text)
2430
.map_err(|error| format!("could not write {}: {error}", file.path))
2531
}

0 commit comments

Comments
 (0)