Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
34 changes: 31 additions & 3 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 4 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ members = [
"crates/sqlbuild-cli",
"crates/sqlbuild-python-base",
"crates/sqlbuild-python-front",
"crates/sqlbuild-python-project",
"crates/sqlbuild-python",
]
resolver = "2"
Expand Down Expand Up @@ -59,9 +60,10 @@ order = [
"sqlbuild-cli",
"sqlbuild-python-base",
"sqlbuild-python-front",
"sqlbuild-python-project",
"sqlbuild-python",
]
python-boundary = ["sqlbuild-python-base", "sqlbuild-python-front", "sqlbuild-python"]
python-boundary = ["sqlbuild-python-base", "sqlbuild-python-front", "sqlbuild-python-project", "sqlbuild-python"]
polyglot-floor = "sqlbuild-analysis"

[workspace.package]
Expand Down Expand Up @@ -183,6 +185,7 @@ sqlbuild-process-ownership = { path = "crates/sqlbuild-process-ownership" }
sqlbuild-project = { path = "crates/sqlbuild-project" }
sqlbuild-python-base = { path = "crates/sqlbuild-python-base" }
sqlbuild-python-front = { path = "crates/sqlbuild-python-front" }
sqlbuild-python-project = { path = "crates/sqlbuild-python-project" }
sqlbuild-pysource = { path = "crates/sqlbuild-pysource" }
sqlbuild-recall = { path = "crates/sqlbuild-recall" }
sqlbuild-scopes = { path = "crates/sqlbuild-scopes" }
Expand Down
7 changes: 4 additions & 3 deletions crates/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,18 +26,19 @@ tested and benchmarked with `cargo test` alone.
| `sqlbuild-analysis` | SQL analysis over polyglot: query analysis, binding, semantic validation and usage, column references, type normalization, FAST/RICH lineage, SQL-test/scenario extraction and compiler-side assembly, and Snowflake cost statement classification and query-tag parsing. |
| `sqlbuild-rules` | Built-in rules and the rules engine, the custom-rule host, SQL lint, quality checks and formatting, rules configuration and the request models. Its caches and rule fingerprints key on the build identity the binary installs at startup. |
| `sqlbuild-refactor` | Native rename/move plans, contract-adoption edits, MODEL/SCHEMA/YAML text edits and atomic staging/commit with rollback. Shared scope placement delegates to the native scope implementation; user Python source inspection remains at the Python AST boundary. |
| `sqlbuild-project` | The compiled project as plain Rust: `CompiledProject` with the producers' facts and each resource's records (model inputs and schemas, compiled models and their analysis, sources, functions, audits and seeds), and what those records answer without Python, such as lineage rows, resolved namespaces and dependency keys. Values the host owns, such as user config and discovered files, and the views it builds lazily live in one slot per record, whose types `sqlbuild-python` supplies through the `ProjectHost` trait; reads that need Python stay in the bindings as extension traits. |
| `sqlbuild-project` | The compiled project as plain Rust: `CompiledProject` with the producers' facts and each resource's records (model inputs and schemas, compiled models and their analysis, sources, functions, audits and seeds), and what those records answer without Python, such as lineage rows, resolved namespaces and dependency keys. Values the host owns, such as user config and discovered files, and the views it builds lazily live in one slot per record, whose types `sqlbuild-python-project` supplies through the `ProjectHost` trait; reads that need Python stay in the bindings as extension traits. |
| `sqlbuild-cli` | The `sqb` and `sqlbuild` executables: `--version`, unchanged compiles replayed from the shared reuse store with Python's exact streams, exit code and compute log capture, and a handoff of every other invocation to the installed `sqlbuild-python` script. Built as its own wheel (`crates/sqlbuild-cli/pyproject.toml`). |
| `sqlbuild-python-base` | The base of the PyO3 bindings: native record, enum and error classes with their specs, registration, construction and field reads; keyword calls, imports, conversions and panics at the Python boundary; host entry calls; the process allocator; and the shared runtime the binding modules build on: the native store, compile context, diagnostics, progress and terminal text, warehouse access and orchestration reads. |
| `sqlbuild-python-front` | The front of the compile pipeline as PyO3 bindings, on top of `sqlbuild-python-base`: project discovery, model configuration and headers, scopes, macro rendering and authored values, SQL text, model attachment and the compile driver, cursors, contract and refactor facts, and the warehouse state tables of consumption, observations and audit results. |
| `sqlbuild-python` | The `_native` module: its Python classes and functions and the conversions from Python objects, on top of `sqlbuild-python-front` and `sqlbuild-python-base`. |
| `sqlbuild-python-project` | The compiled project as PyO3 bindings, on top of `sqlbuild-python-front`: the project host and the Python views of compiled models, inputs and resources, the compiled-project snapshot, analysis classes and the analysis session, the project graph, references, lineage, fingerprints, contracts and semantic shapes, audit rendering, and the planner's identities, scope, selection, phases, fixture strategy and SQL test planning. |
| `sqlbuild-python` | The `_native` module: its Python classes and functions and the conversions from Python objects, on top of `sqlbuild-python-project`, `sqlbuild-python-front` and `sqlbuild-python-base`. |

## Layering

Dependencies point one way, from the top of this graph to the bottom:

```text
sqlbuild-python -> sqlbuild-python-front -> sqlbuild-python-base
sqlbuild-python -> sqlbuild-python-project -> sqlbuild-python-front -> sqlbuild-python-base
-> sqlbuild-project -> sqlbuild-refactor -> sqlbuild-rules -> sqlbuild-analysis -> sqlbuild-attachments
-> sqlbuild-model-config -> sqlbuild-scopes -> sqlbuild-render -> sqlbuild-discovery -> sqlbuild-host
-> sqlbuild-config -> sqlbuild-sqltext -> sqlbuild-cache -> sqlbuild-pysource -> sqlbuild-state -> sqlbuild-cursor -> sqlbuild-core
Expand Down
Original file line number Diff line number Diff line change
@@ -1,59 +1,14 @@
//! Python audit argument values read as plain argument values.

use crate::model_config::main::config::config_object::config_object;
use crate::model_config::types::PyConfigNode;
use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyResult, Python};
use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyResult};
use pyo3::types::{
PyBool, PyDict, PyDictMethods, PyFloat, PyInt, PyList, PyListMethods, PyString,
PyStringMethods, PyTuple, PyTupleMethods,
PyBool, PyFloat, PyInt, PyList, PyListMethods, PyString, PyStringMethods, PyTuple,
PyTupleMethods,
};
use sqlbuild_attachments::audits::models::ArgumentValue;
use sqlbuild_model_config::config_values::models::ConfigNode;

/// Native-built argument containers need no intermediate Python list or tuple.
pub(crate) fn config_argument_value(
py: Python<'_>,
value: &PyConfigNode,
) -> PyResult<ArgumentValue> {
Ok(match value {
ConfigNode::Null => ArgumentValue::Null,
ConfigNode::Bool(value) => ArgumentValue::Boolean(*value),
ConfigNode::Int(value) => ArgumentValue::Number(value.to_string()),
ConfigNode::Str(value) => ArgumentValue::Text(value.clone()),
ConfigNode::List(items) => ArgumentValue::List(
items
.iter()
.map(|item| config_argument_value(py, item))
.collect::<PyResult<_>>()?,
),
ConfigNode::Tuple(items) => ArgumentValue::Tuple(
items
.iter()
.map(|item| config_argument_value(py, item))
.collect::<PyResult<_>>()?,
),
ConfigNode::Map(_) => ArgumentValue::Opaque,
ConfigNode::Float(_) | ConfigNode::Host(_) => {
return argument_value(&config_object(py, value)?);
}
})
}

/// Read a `dict[str, object]` in insertion order, leaving out keys no parameter can name.
pub(crate) fn argument_pairs(
arguments: &Bound<'_, PyDict>,
) -> PyResult<Vec<(String, ArgumentValue)>> {
let mut pairs: Vec<(String, ArgumentValue)> = Vec::with_capacity(arguments.len());
for (key, value) in arguments.iter() {
if let Ok(key) = key.downcast::<PyString>() {
pairs.push((key.to_string_lossy().into_owned(), argument_value(&value)?));
}
}
Ok(pairs)
}

/// One plain value; text with lone surrogates, rejected where it enters a compile, reads lossily.
fn argument_value(value: &Bound<'_, PyAny>) -> PyResult<ArgumentValue> {
pub(crate) fn argument_value(value: &Bound<'_, PyAny>) -> PyResult<ArgumentValue> {
if value.is_none() {
return Ok(ArgumentValue::Null);
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
//! Where a SQL test or scenario body omits its trailing ceremonial `SELECT 1`.

use pyo3::prelude::{Bound, PyModule, PyModuleMethods, PyResult};
use pyo3::{pyfunction, wrap_pyfunction};
use pyo3::prelude::PyResult;
use pyo3::pyfunction;
use sqlbuild_attachments::ceremonial_select::main::omitted_select_offset::omitted_select_offset;
use sqlbuild_attachments::ceremonial_select::models::OmittedSelect;
use sqlbuild_sqltext::sql_scan::models::LexicalSyntax;
Expand All @@ -11,7 +11,10 @@ use sqlbuild_python_base::boundary::main::compiler_guard::compiler_guard;

/// The code-point offset to insert `SELECT 1` at, or `None`.
#[pyfunction]
fn omitted_ceremonial_select(sql: &str, syntax: LexicalSyntaxInput) -> PyResult<Option<usize>> {
pub(crate) fn omitted_ceremonial_select(
sql: &str,
syntax: LexicalSyntaxInput,
) -> PyResult<Option<usize>> {
let syntax: LexicalSyntax = syntax.into();
compiler_guard(|| {
Ok(match omitted_select_offset(sql, &syntax) {
Expand All @@ -20,8 +23,3 @@ fn omitted_ceremonial_select(sql: &str, syntax: LexicalSyntaxInput) -> PyResult<
})
})
}

pub(crate) fn register(module: &Bound<'_, PyModule>) -> PyResult<()> {
module.add_function(wrap_pyfunction!(omitted_ceremonial_select, module)?)?;
Ok(())
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,49 +2,31 @@

use crate::attachments::constants::MICROBATCH_END_SENTINEL;
use crate::attachments::constants::MICROBATCH_START_SENTINEL;
use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyErr, PyModule, PyModuleMethods, PyResult};
use crate::attachments::main::analysis_cursor_sql::analysis_cursor_sql;
use crate::attachments::main::compile_input_error::compile_input_error;
use crate::attachments::main::validated_model_cursor_sql::validated_model_cursor_sql;
use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyResult};
use pyo3::types::{PyString, PyStringMethods};
use pyo3::{Python, pyfunction, wrap_pyfunction};
use sqlbuild_analysis::sql_macros::models::TextSplice;
use pyo3::{Python, pyfunction};
use sqlbuild_attachments::cursor_intrinsics::main::replace_intrinsics::replace_intrinsics;
use sqlbuild_attachments::cursor_intrinsics::main::replace_intrinsics_with_spans::replace_intrinsics_with_spans;
use sqlbuild_attachments::cursor_intrinsics::main::validated_model_intrinsics::validated_model_intrinsics;
use sqlbuild_attachments::cursor_intrinsics::models::IntrinsicModel;

use sqlbuild_python_base::boundary::main::compiler_guard::compiler_guard;
use sqlbuild_python_base::records::main::construct::error_class::error_class;
use sqlbuild_python_base::records::main::fields::mapping_get::mapping_get;
use sqlbuild_python_base::records::types::ErrorId;

const INCREMENTAL_MATERIALIZATION: &str = "incremental";
pub(crate) const INCREMENTAL_MATERIALIZATION: &str = "incremental";
const CURSOR_START: &str = "__cursor_start";
const CURSOR_START_CALL: &str = "__cursor_start()";
const CURSOR_END_CALL: &str = "__cursor_end()";
pub(crate) const CURSOR_START_CALL: &str = "__cursor_start()";
pub(crate) const CURSOR_END_CALL: &str = "__cursor_end()";
const INTEGER_CURSOR_TYPE: &str = "integer";
const INTEGER_ANALYSIS_BOUND: &str = "CAST(0 AS BIGINT)";
const TIMESTAMP_ANALYSIS_BOUND: &str = "CAST('2000-01-01 00:00:00' AS TIMESTAMP)";
const MODEL_SQL_CONTEXT: &str = "Model SQL";
const SQL_CONTEXT: &str = "SQL";
pub(crate) const MODEL_SQL_CONTEXT: &str = "Model SQL";
pub(crate) const SQL_CONTEXT: &str = "SQL";

const RESERVED_CURSOR_MARKERS: [&str; 2] = [MICROBATCH_START_SENTINEL, MICROBATCH_END_SENTINEL];

/// The reserved executor bound markers no authored SQL may contain.
pub(crate) fn reserved_cursor_markers() -> Vec<String> {
RESERVED_CURSOR_MARKERS
.iter()
.map(|marker| (*marker).to_owned())
.collect()
}

/// A `CompileInputError` carrying `message`.
pub(crate) fn compile_input_error(py: Python<'_>, message: String) -> PyResult<PyErr> {
Ok(PyErr::from_value(
error_class(py, ErrorId::CompileInput)?.call1((message,))?,
))
}
pub(crate) const RESERVED_CURSOR_MARKERS: [&str; 2] =
[MICROBATCH_START_SENTINEL, MICROBATCH_END_SENTINEL];

/// `text` when it is a string, else `None`.
fn text_value(value: &Bound<'_, PyAny>) -> Option<String> {
pub(crate) fn text_value(value: &Bound<'_, PyAny>) -> Option<String> {
match value.downcast::<PyString>() {
Ok(text) => Some(text.to_string_lossy().into_owned()),
Err(_) => None,
Expand All @@ -70,60 +52,8 @@ pub(crate) fn replaced(
outcome.or_else(|message| Err(compile_input_error(py, message)?))
}

/// One model query with canonical intrinsic calls, raising when the model may not use them.
pub(crate) fn validated_model_cursor_sql(
py: Python<'_>,
sql: &str,
(model_name, config_values): (&str, &Bound<'_, PyAny>),
) -> PyResult<String> {
let incremental: bool = text_value(&mapping_get(config_values, "materialized")?)
.is_some_and(|materialized| materialized == INCREMENTAL_MATERIALIZATION);
let cursor: Option<String> = text_value(&mapping_get(config_values, "cursor")?);
let reserved_markers: Vec<String> = reserved_cursor_markers();
let outcome: Result<String, String> = compiler_guard(|| {
Ok(validated_model_intrinsics(
sql,
&reserved_markers,
&IntrinsicModel {
name: model_name,
incremental,
cursor: cursor.as_deref(),
},
))
})?;
outcome.or_else(|message| Err(compile_input_error(py, message)?))
}

/// `sql` with intrinsics replaced by stable typed literals, only for static SQL analysis.
pub(crate) fn analysis_cursor_sql(
py: Python<'_>,
sql: &str,
cursor_type: &Bound<'_, PyAny>,
) -> PyResult<String> {
let literal: &str = analysis_bound(text_value(cursor_type).as_deref());
Ok(replaced(py, sql, MODEL_SQL_CONTEXT, (literal, literal))?.0)
}

/// `analysis_cursor_sql` with each replaced call as a splice of `sql`.
pub(crate) fn analysis_cursor_splices(
sql: &str,
cursor_type: Option<&str>,
) -> Result<(String, Vec<TextSplice>), String> {
let literal: &str = analysis_bound(cursor_type);
let (replaced, spans) =
replace_intrinsics_with_spans(sql, MODEL_SQL_CONTEXT, |_| literal.to_owned())?;
let splices: Vec<TextSplice> = spans
.iter()
.map(|span| TextSplice {
source: (span.source_start, span.source_end),
output: (span.output_start, span.output_end),
})
.collect();
Ok((replaced, splices))
}

/// The literal static analysis reads for every cursor intrinsic of a model with `cursor_type`.
fn analysis_bound(cursor_type: Option<&str>) -> &'static str {
pub(crate) fn analysis_bound(cursor_type: Option<&str>) -> &'static str {
if cursor_type == Some(INTEGER_CURSOR_TYPE) {
INTEGER_ANALYSIS_BOUND
} else {
Expand All @@ -134,7 +64,7 @@ fn analysis_bound(cursor_type: Option<&str>) -> &'static str {
/// One model query with canonical intrinsic calls, raising `CompileInputError` when invalid.
#[pyfunction]
#[pyo3(signature = (*, sql, config_values, model_name))]
fn validated_model_cursor_intrinsics(
pub(crate) fn validated_model_cursor_intrinsics(
py: Python<'_>,
sql: &str,
config_values: &Bound<'_, PyAny>,
Expand All @@ -146,41 +76,10 @@ fn validated_model_cursor_intrinsics(
/// `sql` with intrinsics replaced by typed literals of `cursor_type` for static analysis.
#[pyfunction]
#[pyo3(signature = (*, sql, cursor_type))]
fn cursor_intrinsics_analysis_sql(
pub(crate) fn cursor_intrinsics_analysis_sql(
py: Python<'_>,
sql: &str,
cursor_type: &Bound<'_, PyAny>,
) -> PyResult<String> {
analysis_cursor_sql(py, sql, cursor_type)
}

/// Whether `sql` uses cursor intrinsics, and `sql` with both bounds rendered when given.
#[pyfunction]
#[pyo3(signature = (*, sql, start_sql=None, end_sql=None))]
pub(crate) fn resolve_cursor_intrinsics(
py: Python<'_>,
sql: &str,
start_sql: Option<&str>,
end_sql: Option<&str>,
) -> PyResult<(String, bool)> {
let (rendered, found) = replaced(
py,
sql,
SQL_CONTEXT,
(
start_sql.unwrap_or(CURSOR_START_CALL),
end_sql.unwrap_or(CURSOR_END_CALL),
),
)?;
if !found || start_sql.is_none() || end_sql.is_none() {
return Ok((sql.to_owned(), found));
}
Ok((rendered, true))
}

pub(crate) fn register(module: &Bound<'_, PyModule>) -> PyResult<()> {
module.add_function(wrap_pyfunction!(validated_model_cursor_intrinsics, module)?)?;
module.add_function(wrap_pyfunction!(cursor_intrinsics_analysis_sql, module)?)?;
module.add_function(wrap_pyfunction!(resolve_cursor_intrinsics, module)?)?;
Ok(())
}
Loading
Loading