Skip to content

Commit e350e72

Browse files
authored
refactor: move the plan layer of the Python bindings into sqlbuild-python-planner (#1523)
* refactor: move planner scope resolution and the cost context below the plan layer The plan layer moves into its own crate next. Two pieces it uses sat in sqlbuild-python although their layer is lower: the planner scope resolution (project layer) and the cost attribution context (front layer). They now live in sqlbuild-python-project and sqlbuild-python-front, and their callers import them from there. * refactor: move the plan layer of the Python bindings into sqlbuild-python-planner Work in progress: the move itself. main/ forwarders are not folded yet, and the field-fallback list, duplicate allowlist and crate README are not updated. * refactor: fold planner forwarders and regroup imports * refactor: fit the planner crate to the lint, duplicate and fallback checks * fix: key the dynamic planner field reads to their call site and correct the register allowlist reason
1 parent fe97014 commit e350e72

587 files changed

Lines changed: 9718 additions & 6647 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 24 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ members = [
2626
"crates/sqlbuild-python-base",
2727
"crates/sqlbuild-python-front",
2828
"crates/sqlbuild-python-project",
29+
"crates/sqlbuild-python-planner",
2930
"crates/sqlbuild-python",
3031
]
3132
resolver = "2"
@@ -61,9 +62,10 @@ order = [
6162
"sqlbuild-python-base",
6263
"sqlbuild-python-front",
6364
"sqlbuild-python-project",
65+
"sqlbuild-python-planner",
6466
"sqlbuild-python",
6567
]
66-
python-boundary = ["sqlbuild-python-base", "sqlbuild-python-front", "sqlbuild-python-project", "sqlbuild-python"]
68+
python-boundary = ["sqlbuild-python-base", "sqlbuild-python-front", "sqlbuild-python-project", "sqlbuild-python-planner", "sqlbuild-python"]
6769
polyglot-floor = "sqlbuild-analysis"
6870

6971
[workspace.package]
@@ -186,6 +188,7 @@ sqlbuild-project = { path = "crates/sqlbuild-project" }
186188
sqlbuild-python-base = { path = "crates/sqlbuild-python-base" }
187189
sqlbuild-python-front = { path = "crates/sqlbuild-python-front" }
188190
sqlbuild-python-project = { path = "crates/sqlbuild-python-project" }
191+
sqlbuild-python-planner = { path = "crates/sqlbuild-python-planner" }
189192
sqlbuild-pysource = { path = "crates/sqlbuild-pysource" }
190193
sqlbuild-recall = { path = "crates/sqlbuild-recall" }
191194
sqlbuild-scopes = { path = "crates/sqlbuild-scopes" }

‎crates/README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,14 +31,15 @@ tested and benchmarked with `cargo test` alone.
3131
| `sqlbuild-python-base` | The base of the PyO3 bindings: native record, enum and error classes with their specs, registration, construction and field reads; keyword calls, imports, conversions and panics at the Python boundary; host entry calls; the process allocator; and the shared runtime the binding modules build on: the native store, compile context, diagnostics, progress and terminal text, warehouse access and orchestration reads. |
3232
| `sqlbuild-python-front` | The front of the compile pipeline as PyO3 bindings, on top of `sqlbuild-python-base`: project discovery, model configuration and headers, scopes, macro rendering and authored values, SQL text, model attachment and the compile driver, cursors, contract and refactor facts, and the warehouse state tables of consumption, observations and audit results. |
3333
| `sqlbuild-python-project` | The compiled project as PyO3 bindings, on top of `sqlbuild-python-front`: the project host and the Python views of compiled models, inputs and resources, the compiled-project snapshot, analysis classes and the analysis session, the project graph, references, lineage, fingerprints, contracts and semantic shapes, audit rendering, and the planner's identities, scope, selection, phases, fixture strategy and SQL test planning. |
34-
| `sqlbuild-python` | The `_native` module: its Python classes and functions and the conversions from Python objects, on top of `sqlbuild-python-project`, `sqlbuild-python-front` and `sqlbuild-python-base`. |
34+
| `sqlbuild-python-planner` | The plan layer as PyO3 bindings, on top of `sqlbuild-python-project`: execution plans and their entries, planner phases, change detection, migrations, reference and cursor resolution, scenarios, strategy and warehouse inspection, plus freshness, consumption, microbatches, observations, lifecycle events and plan output. |
35+
| `sqlbuild-python` | The `_native` module: its Python classes and functions and the conversions from Python objects, on top of `sqlbuild-python-planner`, `sqlbuild-python-project`, `sqlbuild-python-front` and `sqlbuild-python-base`. |
3536

3637
## Layering
3738

3839
Dependencies point one way, from the top of this graph to the bottom:
3940

4041
```text
41-
sqlbuild-python -> sqlbuild-python-project -> sqlbuild-python-front -> sqlbuild-python-base
42+
sqlbuild-python -> sqlbuild-python-planner -> sqlbuild-python-project -> sqlbuild-python-front -> sqlbuild-python-base
4243
-> sqlbuild-project -> sqlbuild-refactor -> sqlbuild-rules -> sqlbuild-analysis -> sqlbuild-attachments
4344
-> sqlbuild-model-config -> sqlbuild-scopes -> sqlbuild-render -> sqlbuild-discovery -> sqlbuild-host
4445
-> sqlbuild-config -> sqlbuild-sqltext -> sqlbuild-cache -> sqlbuild-pysource -> sqlbuild-state -> sqlbuild-cursor -> sqlbuild-core
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
//! The `ContextVar` that holds the current cost resource context, and the scopes built over it.
2+
3+
use crate::cost_context::main::process_object::process_object;
4+
use crate::cost_context::models::CostScope;
5+
use pyo3::prelude::{Bound, Py, PyAny, PyAnyMethods, PyResult, Python};
6+
use pyo3::sync::PyOnceLock;
7+
use pyo3::types::PyDict;
8+
9+
use sqlbuild_python_base::boundary::main::imported::imported;
10+
use sqlbuild_python_base::records::main::fields::field::field as native_field;
11+
12+
const CONTEXT_VARIABLE_NAME: &str = "sqlbuild_cost_resource_context";
13+
pub(crate) const DEFAULT_PHASE: &str = "execute";
14+
pub(crate) const DEFAULT_ATTEMPT: i64 = 1;
15+
16+
static CURRENT: PyOnceLock<Py<PyAny>> = PyOnceLock::new();
17+
18+
/// The `ContextVar` holding the current cost resource context.
19+
pub(crate) fn current_variable(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>> {
20+
process_object(&CURRENT, py, || {
21+
let options: Bound<'_, PyDict> = PyDict::new(py);
22+
options.set_item("default", py.None())?;
23+
imported(py, "contextvars", "ContextVar")?.call((CONTEXT_VARIABLE_NAME,), Some(&options))
24+
})
25+
}
26+
27+
pub(crate) fn or_current<'py>(
28+
value: Bound<'py, PyAny>,
29+
current: &Bound<'py, PyAny>,
30+
field: &str,
31+
) -> PyResult<Bound<'py, PyAny>> {
32+
if !value.is_none() || current.is_none() {
33+
return Ok(value);
34+
}
35+
native_field(current, field)
36+
}
37+
38+
pub(crate) fn scope_of(context: Bound<'_, PyAny>) -> CostScope {
39+
CostScope {
40+
context: Some(context.unbind()),
41+
token: None,
42+
}
43+
}
44+
45+
/// The active `CostResourceContext`, or `None`.
46+
pub(crate) fn current_context(py: Python<'_>) -> PyResult<Bound<'_, PyAny>> {
47+
current_variable(py)?.call_method0("get")
48+
}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
pub(crate) mod context;
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
//! Constants of cost context.
2+
3+
/// The methods an adapter with warehouse cost telemetry implements.
4+
pub(crate) const COST_AWARE_METHODS: [&str; 2] = ["cost_capability", "collect_run_cost"];
5+
6+
pub(crate) const DECIMAL_MODULE: &str = "decimal";
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
//! Whether an adapter implements the cost telemetry methods.
2+
3+
use crate::cost_context::constants::COST_AWARE_METHODS;
4+
use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyResult};
5+
use sqlbuild_python_base::records::main::fields::field::field as native_field;
6+
7+
/// Whether `adapter` implements the cost telemetry methods.
8+
pub fn cost_aware(adapter: &Bound<'_, PyAny>) -> PyResult<bool> {
9+
for name in COST_AWARE_METHODS {
10+
if !adapter.hasattr(name)? || native_field(adapter, name)?.is_none() {
11+
return Ok(false);
12+
}
13+
}
14+
Ok(true)
15+
}
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
//! Microseconds since the epoch: UTC for aware values, the wall clock for naive ones.
2+
3+
use crate::cursor::constants::DATETIME_MODULE;
4+
use crate::state_rows::main::event_rows::offset_aware::offset_aware;
5+
use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyResult, Python};
6+
use pyo3::types::PyDict;
7+
use sqlbuild_python_base::boundary::main::imported::imported;
8+
9+
/// Microseconds since the epoch: UTC for aware values, the wall clock for naive ones.
10+
pub fn instant_micros(value: &Bound<'_, PyAny>) -> PyResult<i128> {
11+
let py: Python<'_> = value.py();
12+
let datetime: Bound<'_, PyAny> = imported(py, DATETIME_MODULE, "datetime")?;
13+
let epoch: Bound<'_, PyAny> = if offset_aware(value)? {
14+
let options: Bound<'_, PyDict> = PyDict::new(py);
15+
options.set_item("tzinfo", imported(py, DATETIME_MODULE, "UTC")?)?;
16+
datetime.call((1970, 1, 1), Some(&options))?
17+
} else {
18+
datetime.call1((1970, 1, 1))?
19+
};
20+
let options: Bound<'_, PyDict> = PyDict::new(py);
21+
options.set_item("microseconds", 1)?;
22+
let micro: Bound<'_, PyAny> =
23+
imported(py, DATETIME_MODULE, "timedelta")?.call((), Some(&options))?;
24+
value.sub(epoch)?.floor_div(micro)?.extract()
25+
}
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
pub mod cost_aware;
2+
pub mod instant_micros;
3+
pub mod native_decimal;
4+
pub mod process_object;
5+
pub mod python_decimal;
6+
pub mod register_cost_context;
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
//! A finite non-negative `Decimal` as its exact digits and exponent.
2+
3+
use pyo3::exceptions::PyValueError;
4+
use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyResult};
5+
use sqlbuild_core::cost_allocation::models::PyDecimal;
6+
7+
/// A finite non-negative `Decimal` as its exact digits and exponent.
8+
pub fn native_decimal(value: &Bound<'_, PyAny>) -> PyResult<PyDecimal> {
9+
let (sign, digits, exponent): (u8, Vec<u8>, Bound<'_, PyAny>) =
10+
value.call_method0("as_tuple")?.extract()?;
11+
let Ok(exponent) = exponent.extract::<i64>() else {
12+
return Err(PyValueError::new_err("cost decimals must be finite"));
13+
};
14+
if sign != 0 && digits.iter().any(|digit| *digit != 0) {
15+
return Err(PyValueError::new_err("cost decimals must not be negative"));
16+
}
17+
let leading: usize = digits.iter().take_while(|digit| **digit == 0).count();
18+
let digits: Vec<u8> = if leading == digits.len() {
19+
vec![0]
20+
} else {
21+
digits[leading..].to_vec()
22+
};
23+
Ok(PyDecimal { digits, exponent })
24+
}

0 commit comments

Comments
 (0)