Skip to content

Commit 48e4132

Browse files
authored
fix(rules): report custom rule node findings at model file positions (#1546)
* feat(rules): give syntax tree nodes names, literal values and source extents Rules can read a node's name, quoting and qualifier, a literal's value and type, and where a node ends (end_line, end_column and a character span into the document source). A relation marker reads as the call it stands for, and offsets map back through marker names and Snowflake TABLE( ) wrappers to the text as written. Nodes the parser records no span for have no end. * docs(rules): report tree-reading findings at the model, not the node ctx.finding(node=...) takes a node's query-relative position, so the docs example and the DuckShop rule report at the model and quote the node's authored text in the message instead. * chore(fensu): record the source map's start and end mapping as an intentional mirror * style: format the DuckShop quoted alias rule tests * fix(rules): report custom rule findings at a node's place in the model file ctx.finding(subject=model, node=...) took the node's line and column in the model's expanded query and reported them against the model file, so a multi-line MODEL header or a multi-line macro expansion moved the finding. The expanded SQL document now carries where the query sits in the file, read the way built-in model rules map positions, and the finding maps the node through it; a node inside macro output points at the macro call. The query place travels in the custom host payload and in the replayed sql.for_model fact, so cached findings follow it. * fix(rules): place a document's node findings in one pass Each node finding rebuilt the model query's character and line tables, so reporting every column of a wide model took time proportional to findings times model size. A document now maps all its syntax tree node starts into the model file once, on the first node finding, and later findings look their node up. A node whose line or column is not an integer gets no file position again and falls back to its own line and column. * perf(rules): keep the custom-rule query place to a byte range and expansion passes The place of a model's expanded query repeated the authored query text in every model's custom-rule payload, about a sixth of the payload on a dense 3,000-model project. It now names the query by its byte range in the model file, which the rule's SQL document already holds, and offsets are derived only when a rule reports a node.
1 parent 7b79e4b commit 48e4132

20 files changed

Lines changed: 458 additions & 21 deletions

File tree

‎benchmarks/duckshop/project/rules/quoted_aliases.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ def mixed_case_names_stay_in_exports(*, model: Model, ctx: RuleContext) -> list[
2929
findings.append(
3030
ctx.finding(
3131
subject=model,
32+
node=node,
3233
message=f"Quoted mixed-case name {node.name!r} in `{text}`",
3334
)
3435
)

‎benchmarks/duckshop/project/tests/test_rule_quoted_aliases.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ def test_given_quoted_mixed_case_alias_outside_an_export_when_evaluating_then_re
3030
)
3131

3232
assert result.finding_count == 1
33+
assert (result.findings[0].line, result.findings[0].column) == (3, 3)
3334
assert result.findings[0].message == (
3435
"Quoted mixed-case name 'listingId' in `l.listing_id AS \"listingId\"`"
3536
)

‎crates/sqlbuild-python-compile/src/rules/_helpers/custom/custom_host_payload.rs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -212,6 +212,8 @@ impl<W: Write> PayloadWriter<W> {
212212
self.part(TEXT_TAG, authored.as_bytes())?,
213213
);
214214
model.insert("query".to_owned(), self.part(TEXT_TAG, query.as_bytes())?);
215+
let place: String = view.query_place(py)?;
216+
model.insert("place".to_owned(), self.part(TEXT_TAG, place.as_bytes())?);
215217
let inferred: Value = match view.inferred_rows(py)? {
216218
Some(rows) => self.part(INFERRED_TAG, &json_bytes(&rows)?)?,
217219
None => Value::Null,
@@ -242,6 +244,7 @@ impl<W: Write> PayloadWriter<W> {
242244
model.insert("authored".to_owned(), authored);
243245
let query: Value = self.text(&native_field(record, "query_sql")?)?;
244246
model.insert("query".to_owned(), query);
247+
model.insert("place".to_owned(), Value::Null);
245248
let inferred: Bound<'_, PyAny> = native_field(record, "inferred_columns")?;
246249
let inferred: Value = if inferred.is_none() {
247250
Value::Null

‎crates/sqlbuild-python-compile/src/rules/_helpers/fact/fact_replay.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -431,6 +431,7 @@ impl<'py> FactReplay<'py> {
431431
JsonValue::Array(vec![
432432
JsonValue::String(authored),
433433
JsonValue::String(expanded),
434+
JsonValue::String(view.query_place(py)?),
434435
])
435436
}
436437
None => {

‎crates/sqlbuild-python-compile/src/rules/_helpers/sql/sql_document.rs‎

Lines changed: 115 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
//! Custom rules' SQL documents: the SQLQuack syntax tree of one SQL text and its common structures.
22
3+
use std::collections::HashMap;
34
use std::sync::Arc;
45

5-
use pyo3::prelude::{Bound, Py, PyAny, PyErr, PyResult, Python};
6+
use pyo3::exceptions::{PyAttributeError, PyValueError};
7+
use pyo3::prelude::{Bound, Py, PyAny, PyAnyMethods, PyErr, PyResult, Python};
68
use pyo3::sync::PyOnceLock;
79
use pyo3::types::PyTuple;
810
use pyo3::{IntoPyObject, pyclass, pymethods};
@@ -11,9 +13,12 @@ use sqlbuild_analysis::sql_tree::main::sql_tree_nodes::sql_tree_nodes;
1113
use sqlbuild_analysis::sql_tree::models::{
1214
NodeExtent, NodeLiteral, NodeName, SqlTreeNode as TreeNode,
1315
};
16+
use sqlbuild_rules::engine::main::query_file_positions::query_file_positions;
17+
use sqlbuild_rules::models::ModelQueryPlace;
1418

1519
use sqlbuild_python_base::records::main::construct::new_annotated_error::new_annotated_error;
1620
use sqlbuild_python_base::records::main::construct::new_record_with::new_record_with;
21+
use sqlbuild_python_base::records::main::fields::field::field as native_field;
1722
use sqlbuild_python_base::records::types::ErrorId;
1823
use sqlbuild_python_base::records::types::RecordId;
1924

@@ -29,6 +34,14 @@ struct Tree {
2934
children: Vec<Vec<usize>>,
3035
}
3136

37+
/// A model query's character offsets and the file positions of its tree's node starts, built once.
38+
struct FilePositions {
39+
/// The character each line starts at, then the source's length.
40+
line_starts: Vec<usize>,
41+
/// The file `(line, column)` of every tree node's first character, by that character.
42+
node_starts: HashMap<usize, (u64, u64)>,
43+
}
44+
3245
/// One node of a rule's SQL syntax tree: an index into its shared tree, so freeing never recurses.
3346
#[pyclass(module = "sqlbuild._native", frozen)]
3447
pub(crate) struct SqlTreeNode {
@@ -165,19 +178,36 @@ pub(crate) struct SqlDocument {
165178
#[pyo3(get)]
166179
source: String,
167180
dialect: String,
181+
/// Where `source`, a model's expanded query, sits in the model file; `None` for other SQL.
182+
place: Option<ModelQueryPlace>,
183+
/// The model file `place` points into.
184+
file: Option<String>,
168185
tree: PyOnceLock<Arc<Tree>>,
186+
positions: PyOnceLock<FilePositions>,
169187
}
170188

171189
#[pymethods]
172190
impl SqlDocument {
173191
#[new]
174-
#[pyo3(signature = (*, source, dialect))]
175-
fn new(source: String, dialect: String) -> Self {
176-
Self {
192+
#[pyo3(signature = (*, source, dialect, place=None, file=None))]
193+
fn new(
194+
source: String,
195+
dialect: String,
196+
place: Option<&str>,
197+
file: Option<String>,
198+
) -> PyResult<Self> {
199+
let place: Option<ModelQueryPlace> = place
200+
.map(serde_json::from_str)
201+
.transpose()
202+
.map_err(|error| PyValueError::new_err(format!("invalid query place: {error}")))?;
203+
Ok(Self {
177204
source,
178205
dialect,
206+
place,
207+
file,
179208
tree: PyOnceLock::new(),
180-
}
209+
positions: PyOnceLock::new(),
210+
})
181211
}
182212

183213
/// The syntax tree: a `document` node whose children are the statements.
@@ -207,6 +237,31 @@ impl SqlDocument {
207237
self.nodes(py, "Subquery")
208238
}
209239

240+
/// Where `node`, read from this model query, is in the model file; `None` for other SQL.
241+
#[pyo3(signature = (*, node))]
242+
fn file_position(&self, node: &Bound<'_, PyAny>) -> PyResult<Option<(u64, u64)>> {
243+
let (Some(place), Some(file)) = (self.place.as_ref(), self.file.as_deref()) else {
244+
return Ok(None);
245+
};
246+
let (Some(line), Some(column)) = (node_index(node, "line")?, node_index(node, "column")?)
247+
else {
248+
return Ok(None);
249+
};
250+
let positions: &FilePositions = self
251+
.positions
252+
.get_or_init(node.py(), || self.file_positions(node.py(), (file, place)));
253+
let character: usize = positions.character(line, column);
254+
Ok(Some(
255+
positions
256+
.node_starts
257+
.get(&character)
258+
.copied()
259+
.unwrap_or_else(|| {
260+
query_file_positions(&self.source, (file, place), &[character])[0]
261+
}),
262+
))
263+
}
264+
210265
/// Every projected star, bare or qualified; a star a function call takes, such as `COUNT(*)`, is no projection.
211266
fn star_projections<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyTuple>> {
212267
self.source_nodes(py, "Star", |nodes, node| {
@@ -244,6 +299,37 @@ impl SqlDocument {
244299
PyTuple::new(py, found)
245300
}
246301

302+
/// Line starts and the file positions of every tree node's start, mapped through `place` at once.
303+
fn file_positions(&self, py: Python<'_>, located: (&str, &ModelQueryPlace)) -> FilePositions {
304+
let mut line_starts: Vec<usize> = vec![0];
305+
let mut length: usize = 0;
306+
for character in self.source.chars() {
307+
length += 1;
308+
if character == '\n' {
309+
line_starts.push(length);
310+
}
311+
}
312+
line_starts.push(length);
313+
let mut positions: FilePositions = FilePositions {
314+
line_starts,
315+
node_starts: HashMap::new(),
316+
};
317+
let mut starts: Vec<usize> = self.parsed(py).map_or_else(
318+
|_| Vec::new(),
319+
|tree| {
320+
tree.nodes
321+
.iter()
322+
.map(|node| positions.character(node.line, node.column))
323+
.collect()
324+
},
325+
);
326+
starts.sort_unstable();
327+
starts.dedup();
328+
let mapped: Vec<(u64, u64)> = query_file_positions(&self.source, located, &starts);
329+
positions.node_starts = starts.into_iter().zip(mapped).collect();
330+
positions
331+
}
332+
247333
/// The parsed tree; a failure raises `RuleUsageError` each time it is read.
248334
fn parsed(&self, py: Python<'_>) -> PyResult<&Arc<Tree>> {
249335
self.tree.get_or_try_init(py, || {
@@ -253,6 +339,30 @@ impl SqlDocument {
253339
}
254340
}
255341

342+
impl FilePositions {
343+
/// The source character at 1-based `line` and `column`; a line past the end starts at the end.
344+
fn character(&self, line: usize, column: usize) -> usize {
345+
let last: usize = self.line_starts.len() - 1;
346+
self.line_starts[line.saturating_sub(1).min(last)] + column.saturating_sub(1)
347+
}
348+
}
349+
350+
/// A node's 1-based `line` or `column` attribute; `None` when it has none or it is not a positive integer.
351+
fn node_index(node: &Bound<'_, PyAny>, name: &str) -> PyResult<Option<usize>> {
352+
let value: i64 = match native_field(node, name) {
353+
Ok(value) => match value.extract() {
354+
Ok(value) => value,
355+
Err(_) => return Ok(None),
356+
},
357+
Err(error) if error.is_instance_of::<PyAttributeError>(node.py()) => return Ok(None),
358+
Err(error) => return Err(error),
359+
};
360+
Ok(match usize::try_from(value) {
361+
Ok(index) if index > 0 => Some(index),
362+
_ => None,
363+
})
364+
}
365+
256366
/// `nodes` with each node's children listed, in ascending index order, under its parent.
257367
fn tree(nodes: Vec<TreeNode>) -> Arc<Tree> {
258368
let mut children: Vec<Vec<usize>> = vec![Vec::new(); nodes.len() + 1];

‎crates/sqlbuild-python-project/src/compiled_views/models.rs‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
//! Public records of compiled views.
22
3+
use sqlbuild_rules::engine::main::model_query_place::model_query_place;
4+
use sqlbuild_rules::models::ModelQueryPlace;
5+
36
use crate::compiled_project::main::reference_row_tuple::reference_row_tuple;
47
use crate::compiled_project::models::NativeCompiledProject;
58
use crate::compiled_project::models::ReferenceTuple;
@@ -420,12 +423,23 @@ impl NativeCompiledModel {
420423

421424
/// The model's authored file text and expanded query, from its recorded facts.
422425
pub fn sql_texts(&self, py: Python<'_>) -> PyResult<(String, String)> {
426+
self.with_facts(py, |facts| {
427+
(facts.authored_sql.to_string(), facts.query_sql.to_string())
428+
})
429+
}
430+
431+
/// `read` applied to the model's recorded facts.
432+
pub fn with_facts<T>(
433+
&self,
434+
py: Python<'_>,
435+
read: impl FnOnce(&CompiledModelFacts) -> T,
436+
) -> PyResult<T> {
423437
let name: String = self.name(py)?;
424438
let owner = self.owner.bind(py).borrow();
425439
let facts: &CompiledModelFacts = model_facts(owner.facts(), &name).ok_or_else(|| {
426440
pyo3::exceptions::PyRuntimeError::new_err("the model was not assembled")
427441
})?;
428-
Ok((facts.authored_sql.to_string(), facts.query_sql.to_string()))
442+
Ok(read(facts))
429443
}
430444

431445
pub(crate) fn model_file<T>(
@@ -685,6 +699,14 @@ impl NativeCompiledModel {
685699
self.model_file(py, |file| Ok(file.contents(py)))
686700
}
687701

702+
/// Where the expanded query sits in the model file, as JSON a rule's SQL document maps positions through.
703+
#[getter]
704+
pub fn query_place(&self, py: Python<'_>) -> PyResult<String> {
705+
let place: ModelQueryPlace = self.with_facts(py, model_query_place)?;
706+
serde_json::to_string(&place)
707+
.map_err(|error| pyo3::exceptions::PyRuntimeError::new_err(error.to_string()))
708+
}
709+
688710
#[getter]
689711
pub fn authored_query_sql<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyString>> {
690712
self.model_file(py, |file| Ok(file.query_sql(py)))

‎crates/sqlbuild-rules/src/engine/_helpers/project_rows.rs‎

Lines changed: 30 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -119,23 +119,43 @@ pub(crate) fn declaration(fact: &DeclarationFact) -> Declaration {
119119

120120
/// The expansion passes back to the authored query and where that query starts in the file.
121121
fn authored_query_map(facts: &CompiledModelFacts) -> Option<AuthoredQueryMap> {
122+
if facts.authored_query_sql.is_empty() || facts.expansion_passes.iter().all(Vec::is_empty) {
123+
return None;
124+
}
125+
located_query_map(
126+
&facts.authored_sql,
127+
&facts.authored_query_sql,
128+
expansion_spans(&facts.expansion_passes),
129+
)
130+
}
131+
132+
/// `passes` in characters, as the rules request carries them.
133+
pub(crate) fn expansion_spans(passes: &[Vec<CharSpan>]) -> Vec<Vec<ExpansionSpan>> {
122134
let span = |span: &CharSpan| ExpansionSpan {
123135
source_start: character_offset(span.source_start),
124136
source_end: character_offset(span.source_end),
125137
output_start: character_offset(span.output_start),
126138
output_end: character_offset(span.output_end),
127139
};
128-
if facts.authored_query_sql.is_empty() || facts.expansion_passes.iter().all(Vec::is_empty) {
140+
passes
141+
.iter()
142+
.map(|pass| pass.iter().map(span).collect())
143+
.collect()
144+
}
145+
146+
/// `passes` back to `authored_query` and where that query starts in `file`.
147+
pub(crate) fn located_query_map(
148+
file: &str,
149+
authored_query: &str,
150+
passes: Vec<Vec<ExpansionSpan>>,
151+
) -> Option<AuthoredQueryMap> {
152+
if authored_query.is_empty() || passes.iter().all(Vec::is_empty) {
129153
return None;
130154
}
131-
let start: usize = facts.authored_sql.rfind(&*facts.authored_query_sql)?;
132-
let above: &str = &facts.authored_sql[..start];
155+
let start: usize = file.rfind(authored_query)?;
156+
let above: &str = &file[..start];
133157
Some(AuthoredQueryMap {
134-
passes: facts
135-
.expansion_passes
136-
.iter()
137-
.map(|pass| pass.iter().map(span).collect())
138-
.collect(),
158+
passes,
139159
line_offset: above.matches('\n').count() as u64,
140160
column_offset: above
141161
.rsplit('\n')
@@ -151,7 +171,7 @@ fn character_offset(offset: usize) -> i64 {
151171
}
152172

153173
/// The file-line characters before the query's first content that the query itself lacks.
154-
fn query_column_offset(file: &str, authored_query: &str, query: &str) -> u64 {
174+
pub(crate) fn query_column_offset(file: &str, authored_query: &str, query: &str) -> u64 {
155175
let line_prefix = |text: &str, at: usize| {
156176
text[..at]
157177
.rsplit('\n')
@@ -172,7 +192,7 @@ fn query_column_offset(file: &str, authored_query: &str, query: &str) -> u64 {
172192
}
173193

174194
/// The lines of `file` above `query`, the rendered form of its `authored_query`; 0 if unknown.
175-
fn query_line_offset(file: &str, authored_query: &str, query: &str) -> u64 {
195+
pub(crate) fn query_line_offset(file: &str, authored_query: &str, query: &str) -> u64 {
176196
let leading_lines = |text: &str| {
177197
let body: &str = text.trim_start();
178198
text[..text.len() - body.len()].matches('\n').count()

‎crates/sqlbuild-rules/src/engine/main/mod.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,9 @@ pub mod evaluate_rows;
66
pub mod finalize_rows;
77
pub mod json_grain_skipped_models;
88
pub mod json_grains;
9+
pub mod model_query_place;
910
pub mod needs_json_grains;
1011
pub mod project_model_rows;
12+
pub mod query_file_positions;
1113
pub mod retained_request;
1214
pub mod start_speculative_rows;
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
//! What places a compiled model's expanded query in its file.
2+
3+
use sqlbuild_analysis::compiled_project::models::CompiledModelFacts;
4+
5+
use crate::engine::_helpers::project_rows::expansion_spans;
6+
use crate::models::{AuthoredQuery, ModelQueryPlace};
7+
8+
/// Where `facts`' authored query is and its expansion passes; finding the query is a suffix check.
9+
#[must_use]
10+
pub fn model_query_place(facts: &CompiledModelFacts) -> ModelQueryPlace {
11+
let file: &str = &facts.authored_sql;
12+
let query: &str = &facts.authored_query_sql;
13+
let start: Option<usize> = if file.ends_with(query) {
14+
Some(file.len() - query.len())
15+
} else {
16+
file.rfind(query)
17+
};
18+
ModelQueryPlace {
19+
authored_query: (!query.is_empty()).then(|| {
20+
start.map_or_else(
21+
|| AuthoredQuery::Text(query.to_owned()),
22+
|start| AuthoredQuery::Range(start, start + query.len()),
23+
)
24+
}),
25+
passes: if facts.expansion_passes.iter().all(Vec::is_empty) {
26+
Vec::new()
27+
} else {
28+
expansion_spans(&facts.expansion_passes)
29+
},
30+
}
31+
}

0 commit comments

Comments
 (0)