Skip to content

Commit 7f9ba65

Browse files
committed
refactor: move the compile layer of the Python bindings into sqlbuild-python-compile
sqlbuild-python is still over the crate size limit. The compile layer (compile command and driver, input loops, compile-input attachment, project assembly, analysis and the analysis session, Rules, lineage, semantic checks, compile deltas and outputs, the DAG artifact, manifest, frontier, offline scope facts, SQL test glue and auditing) moves into a new crate above sqlbuild-python-project and beside sqlbuild-python-planner. Neither depends on the other. Command handlers that also reach the planner stay in sqlbuild-python. Old paths are deleted and callers updated; there are no re-exports. main/ entries hold the bodies they used to forward to, except 13 whose bodies are over the entry size budgets. The sqlbuild-python cap drops to its new size, XSB072/XSB073 budgets are re-keyed with unchanged totals, and the native field fallback list and compiler-differential scan paths follow the moved files.
1 parent e350e72 commit 7f9ba65

684 files changed

Lines changed: 14883 additions & 11653 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 31 additions & 5 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ members = [
2727
"crates/sqlbuild-python-front",
2828
"crates/sqlbuild-python-project",
2929
"crates/sqlbuild-python-planner",
30+
"crates/sqlbuild-python-compile",
3031
"crates/sqlbuild-python",
3132
]
3233
resolver = "2"
@@ -63,9 +64,10 @@ order = [
6364
"sqlbuild-python-front",
6465
"sqlbuild-python-project",
6566
"sqlbuild-python-planner",
67+
"sqlbuild-python-compile",
6668
"sqlbuild-python",
6769
]
68-
python-boundary = ["sqlbuild-python-base", "sqlbuild-python-front", "sqlbuild-python-project", "sqlbuild-python-planner", "sqlbuild-python"]
70+
python-boundary = ["sqlbuild-python-base", "sqlbuild-python-front", "sqlbuild-python-project", "sqlbuild-python-planner", "sqlbuild-python-compile", "sqlbuild-python"]
6971
polyglot-floor = "sqlbuild-analysis"
7072

7173
[workspace.package]
@@ -189,6 +191,7 @@ sqlbuild-python-base = { path = "crates/sqlbuild-python-base" }
189191
sqlbuild-python-front = { path = "crates/sqlbuild-python-front" }
190192
sqlbuild-python-project = { path = "crates/sqlbuild-python-project" }
191193
sqlbuild-python-planner = { path = "crates/sqlbuild-python-planner" }
194+
sqlbuild-python-compile = { path = "crates/sqlbuild-python-compile" }
192195
sqlbuild-pysource = { path = "crates/sqlbuild-pysource" }
193196
sqlbuild-recall = { path = "crates/sqlbuild-recall" }
194197
sqlbuild-scopes = { path = "crates/sqlbuild-scopes" }

‎crates/README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,14 +32,15 @@ tested and benchmarked with `cargo test` alone.
3232
| `sqlbuild-python-front` | The front of the compile pipeline as PyO3 bindings, on top of `sqlbuild-python-base`: project discovery, model configuration and headers, scopes, macro rendering and authored values, SQL text, model attachment and the compile driver, cursors, contract and refactor facts, and the warehouse state tables of consumption, observations and audit results. |
3333
| `sqlbuild-python-project` | The compiled project as PyO3 bindings, on top of `sqlbuild-python-front`: the project host and the Python views of compiled models, inputs and resources, the compiled-project snapshot, analysis classes and the analysis session, the project graph, references, lineage, fingerprints, contracts and semantic shapes, audit rendering, and the planner's identities, scope, selection, phases, fixture strategy and SQL test planning. |
3434
| `sqlbuild-python-planner` | The plan layer as PyO3 bindings, on top of `sqlbuild-python-project`: execution plans and their entries, planner phases, change detection, migrations, reference and cursor resolution, scenarios, strategy and warehouse inspection, plus freshness, consumption, microbatches, observations, lifecycle events and plan output. |
35-
| `sqlbuild-python` | The `_native` module: its Python classes and functions and the conversions from Python objects, on top of `sqlbuild-python-planner`, `sqlbuild-python-project`, `sqlbuild-python-front` and `sqlbuild-python-base`. |
35+
| `sqlbuild-python-compile` | The compile layer as PyO3 bindings, on top of `sqlbuild-python-project` and beside `sqlbuild-python-planner`: the compile command and driver, input loops and compile-input attachment, project assembly, analysis and the analysis session, Rules, lineage, semantic checks, compile deltas and outputs, the DAG artifact, manifest, frontier, offline scope facts, SQL test glue and auditing. |
36+
| `sqlbuild-python` | The `_native` module: its Python classes and functions and the conversions from Python objects, on top of `sqlbuild-python-compile`, `sqlbuild-python-planner`, `sqlbuild-python-project`, `sqlbuild-python-front` and `sqlbuild-python-base`. |
3637

3738
## Layering
3839

3940
Dependencies point one way, from the top of this graph to the bottom:
4041

4142
```text
42-
sqlbuild-python -> sqlbuild-python-planner -> sqlbuild-python-project -> sqlbuild-python-front -> sqlbuild-python-base
43+
sqlbuild-python -> sqlbuild-python-compile -> sqlbuild-python-planner -> sqlbuild-python-project -> sqlbuild-python-front -> sqlbuild-python-base
4344
-> sqlbuild-project -> sqlbuild-refactor -> sqlbuild-rules -> sqlbuild-analysis -> sqlbuild-attachments
4445
-> sqlbuild-model-config -> sqlbuild-scopes -> sqlbuild-render -> sqlbuild-discovery -> sqlbuild-host
4546
-> sqlbuild-config -> sqlbuild-sqltext -> sqlbuild-cache -> sqlbuild-pysource -> sqlbuild-state -> sqlbuild-cursor -> sqlbuild-core
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
[package]
2+
name = "sqlbuild-python-compile"
3+
description = "The compile layer of the sqlbuild._native bindings: the compile driver and command, input loops, project assembly, analysis and the analysis session, rules, lineage, semantic checks, compile deltas and outputs, the DAG artifact, manifest, frontier and auditing."
4+
version.workspace = true
5+
edition.workspace = true
6+
license.workspace = true
7+
publish.workspace = true
8+
rust-version.workspace = true
9+
10+
[dependencies]
11+
fensu-policy.workspace = true
12+
pyo3.workspace = true
13+
rayon.workspace = true
14+
serde.workspace = true
15+
serde_json.workspace = true
16+
sqlbuild-analysis.workspace = true
17+
sqlbuild-attachments.workspace = true
18+
sqlbuild-cache.workspace = true
19+
sqlbuild-cli.workspace = true
20+
sqlbuild-config.workspace = true
21+
sqlbuild-core.workspace = true
22+
sqlbuild-discovery.workspace = true
23+
sqlbuild-host.workspace = true
24+
sqlbuild-model-config.workspace = true
25+
sqlbuild-project.workspace = true
26+
sqlbuild-pysource.workspace = true
27+
sqlbuild-python-base.workspace = true
28+
sqlbuild-python-front.workspace = true
29+
sqlbuild-python-project.workspace = true
30+
sqlbuild-render.workspace = true
31+
sqlbuild-rules.workspace = true
32+
sqlbuild-scopes.workspace = true
33+
sqlbuild-sqltext.workspace = true
34+
sqlclean.workspace = true
35+
36+
[lints]
37+
workspace = true
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
//! A hand-built model's binding diagnostics at their authored locations.
2+
3+
use crate::analysis::main::locate_model_binding_diagnostics::locate_model_binding_diagnostics;
4+
use crate::analysis::models::LocatedRow;
5+
use pyo3::prelude::{Bound, PyModule, PyModuleMethods, PyResult};
6+
use pyo3::wrap_pyfunction;
7+
use sqlbuild_analysis::binding_locations::models::LocatedDiagnostic;
8+
9+
use sqlbuild_python_project::analysis::models::Positional;
10+
11+
pub(crate) fn located_row(item: LocatedDiagnostic) -> LocatedRow {
12+
LocatedRow {
13+
code: item.code,
14+
message: item.message,
15+
severity: item.severity,
16+
line: item.line,
17+
column: item.column,
18+
span: item.location.map(Positional),
19+
}
20+
}
21+
22+
pub(crate) fn register(module: &Bound<'_, PyModule>) -> PyResult<()> {
23+
module.add_function(wrap_pyfunction!(locate_model_binding_diagnostics, module)?)?;
24+
Ok(())
25+
}

crates/sqlbuild-python/src/bindings/_helpers/analysis/getters.rs renamed to crates/sqlbuild-python-compile/src/analysis/_helpers/getters.rs

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -3,17 +3,9 @@
33
use pyo3::IntoPyObjectExt;
44
use pyo3::prelude::{Bound, PyAny, PyResult, Python};
55

6-
use sqlbuild_python_base::records::main::registry::class_of::class_of;
7-
use sqlbuild_python_base::records::models::GetterEntry;
86
use sqlbuild_python_project::analysis::models::BindingCatalog;
97

10-
/// The getter readers of this module's native pyclasses.
11-
pub(crate) const GETTERS: &[GetterEntry] = &[GetterEntry {
12-
class: class_of::<BindingCatalog>,
13-
read: read_binding_catalog,
14-
}];
15-
16-
fn read_binding_catalog<'py>(
8+
pub(crate) fn read_binding_catalog<'py>(
179
value: &Bound<'py, PyAny>,
1810
name: &str,
1911
) -> Option<PyResult<Bound<'py, PyAny>>> {

crates/sqlbuild-python/src/bindings/_helpers/analysis/mod.rs renamed to crates/sqlbuild-python-compile/src/analysis/_helpers/mod.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
pub(crate) mod binding_locations;
22
pub(crate) mod getters;
33
pub(crate) mod queries;
4-
mod required_columns;
4+
pub(crate) mod required_columns;
55
pub(crate) mod sql_test_comparisons;
66
pub(crate) mod sql_tests;
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
//! Validate, fingerprint and analyze SQL queries for the Python compiler.
2+
3+
use pyo3::prelude::{PyResult, Python};
4+
use pyo3::pyfunction;
5+
6+
use sqlbuild_python_base::boundary::main::value_error::value_error;
7+
use sqlbuild_python_base::types::CompilerDetach;
8+
9+
#[pyfunction]
10+
pub(crate) fn query_fingerprint(py: Python<'_>, sql: &str, dialect: &str) -> PyResult<String> {
11+
py.compiler_detach(|| {
12+
sqlbuild_analysis::sql_tokens::main::query_fingerprint::query_fingerprint(sql, dialect)
13+
})
14+
.map_err(value_error)
15+
}
16+
17+
/// A warehouse type name's JSON type tree, None without a parser, `ValueError` on a panic.
18+
#[pyfunction]
19+
pub(crate) fn parse_warehouse_type(
20+
py: Python<'_>,
21+
spelling: &str,
22+
dialect: &str,
23+
) -> PyResult<Option<String>> {
24+
py.compiler_detach(|| {
25+
sqlbuild_analysis::warehouse_types::main::parse_warehouse_type::parse_warehouse_type(
26+
spelling, dialect,
27+
)
28+
.map(|tree| tree.as_ref().map(serde_json::Value::to_string))
29+
})
30+
.map_err(value_error)
31+
}

crates/sqlbuild-python/src/bindings/_helpers/analysis/required_columns.rs renamed to crates/sqlbuild-python-compile/src/analysis/_helpers/required_columns.rs

File renamed without changes.

crates/sqlbuild-python/src/bindings/_helpers/analysis/sql_test_comparisons.rs renamed to crates/sqlbuild-python-compile/src/analysis/_helpers/sql_test_comparisons.rs

Lines changed: 10 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
33
use std::collections::HashMap;
44

5-
use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyModule, PyModuleMethods, PyResult, Python};
5+
use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyResult, Python};
66
use pyo3::types::{PyDict, PyDictMethods};
7-
use pyo3::{FromPyObject, pyfunction, wrap_pyfunction};
7+
use pyo3::{FromPyObject, pyfunction};
88
use serde_json::{Value, json};
99
use sqlbuild_analysis::compiler::main::missing_expected_columns::missing_expected_columns;
1010
use sqlbuild_analysis::compiler::main::scenario_expected_comparison_sql::scenario_expected_comparison_sql;
@@ -22,11 +22,11 @@ const TOP_CLAUSE_DIALECT: &str = "tsql";
2222

2323
/// Operator, dialect and optional probed chain step of one comparison render.
2424
pub(crate) struct RenderSettings {
25-
set_difference_operator: String,
26-
sql_analysis_dialect: Option<String>,
27-
probe_step_index: Option<usize>,
28-
output_step_index: Option<usize>,
29-
actual_relations: HashMap<usize, String>,
25+
pub(crate) set_difference_operator: String,
26+
pub(crate) sql_analysis_dialect: Option<String>,
27+
pub(crate) probe_step_index: Option<usize>,
28+
pub(crate) output_step_index: Option<usize>,
29+
pub(crate) actual_relations: HashMap<usize, String>,
3030
}
3131

3232
impl RenderSettings {
@@ -59,15 +59,15 @@ fn optional_value<T: for<'py> FromPyObject<'py>>(
5959
/// Comparison SQL of each planned SQL test, or of one chain step's actual-output probe.
6060
#[pyfunction]
6161
#[pyo3(signature = (test_entries, **options))]
62-
fn render_sql_test_comparison_sql(
62+
pub(crate) fn render_sql_test_comparison_sql(
6363
py: Python<'_>,
6464
test_entries: Vec<Bound<'_, PyAny>>,
6565
options: Option<&Bound<'_, PyDict>>,
6666
) -> PyResult<Vec<String>> {
6767
rendered_comparisons(py, &test_entries, &RenderSettings::read(options)?)
6868
}
6969

70-
fn rendered_comparisons(
70+
pub(crate) fn rendered_comparisons(
7171
py: Python<'_>,
7272
test_entries: &[Bound<'_, PyAny>],
7373
settings: &RenderSettings,
@@ -116,26 +116,6 @@ fn rendered_comparisons(
116116
.collect()
117117
}
118118

119-
/// The comparison SQL of one planned SQL test under `adapter`'s operator and dialect.
120-
pub(crate) fn single_comparison_sql(
121-
entry: &Bound<'_, PyAny>,
122-
adapter: &Bound<'_, PyAny>,
123-
) -> PyResult<String> {
124-
let settings = RenderSettings {
125-
set_difference_operator: adapter
126-
.call_method0("render_set_difference_operator")?
127-
.extract()?,
128-
sql_analysis_dialect: adapter.call_method0("sql_analysis_dialect")?.extract()?,
129-
probe_step_index: None,
130-
output_step_index: None,
131-
actual_relations: HashMap::new(),
132-
};
133-
rendered_comparisons(entry.py(), std::slice::from_ref(entry), &settings)?
134-
.into_iter()
135-
.next()
136-
.ok_or_else(|| value_error("native SQL-test rendering returned an invalid batch response"))
137-
}
138-
139119
/// The native comparison render request of one planned SQL test.
140120
pub(crate) fn comparison_request(
141121
entry: &Bound<'_, PyAny>,
@@ -183,7 +163,7 @@ pub(crate) fn chain_step_request(step: &Bound<'_, PyAny>) -> PyResult<Value> {
183163
/// A bounded query for one direction of an expected-output difference of one chain step.
184164
#[pyfunction]
185165
#[pyo3(signature = (**options))]
186-
fn render_sql_test_difference_sample_sql(
166+
pub(crate) fn render_sql_test_difference_sample_sql(
187167
py: Python<'_>,
188168
options: Option<&Bound<'_, PyDict>>,
189169
) -> PyResult<String> {
@@ -251,17 +231,3 @@ pub(crate) fn build_scenario_expected_comparison_sql(
251231
set_difference_operator,
252232
)
253233
}
254-
255-
pub(crate) fn register(module: &Bound<'_, PyModule>) -> PyResult<()> {
256-
module.add_function(wrap_pyfunction!(render_sql_test_comparison_sql, module)?)?;
257-
module.add_function(wrap_pyfunction!(
258-
render_sql_test_difference_sample_sql,
259-
module
260-
)?)?;
261-
module.add_function(wrap_pyfunction!(describe_missing_expected_columns, module)?)?;
262-
module.add_function(wrap_pyfunction!(
263-
build_scenario_expected_comparison_sql,
264-
module
265-
)?)?;
266-
Ok(())
267-
}

0 commit comments

Comments
 (0)