Skip to content

Commit 6877100

Browse files
authored
refactor: move the shared core of the Python bindings into sqlbuild-python-base (#1505)
The native record, enum and error classes, the Python call plumbing, the host entry calls and the process allocator move into a new PyO3 crate that every later binding crate can depend on. sqlbuild-python imports them directly; the extension module surface is unchanged.
1 parent d94e5b7 commit 6877100

892 files changed

Lines changed: 7063 additions & 4585 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 16 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ members = [
2222
"crates/sqlbuild-refactor",
2323
"crates/sqlbuild-project",
2424
"crates/sqlbuild-cli",
25+
"crates/sqlbuild-python-base",
2526
"crates/sqlbuild-python",
2627
]
2728
resolver = "2"
@@ -53,9 +54,10 @@ order = [
5354
"sqlbuild-refactor",
5455
"sqlbuild-project",
5556
"sqlbuild-cli",
57+
"sqlbuild-python-base",
5658
"sqlbuild-python",
5759
]
58-
python-boundary = "sqlbuild-python"
60+
python-boundary = ["sqlbuild-python-base", "sqlbuild-python"]
5961
polyglot-floor = "sqlbuild-analysis"
6062

6163
[workspace.package]
@@ -174,6 +176,7 @@ sqlbuild-model-config = { path = "crates/sqlbuild-model-config" }
174176
sqlbuild-orchestration = { path = "crates/sqlbuild-orchestration" }
175177
sqlbuild-process-ownership = { path = "crates/sqlbuild-process-ownership" }
176178
sqlbuild-project = { path = "crates/sqlbuild-project" }
179+
sqlbuild-python-base = { path = "crates/sqlbuild-python-base" }
177180
sqlbuild-pysource = { path = "crates/sqlbuild-pysource" }
178181
sqlbuild-recall = { path = "crates/sqlbuild-recall" }
179182
sqlbuild-scopes = { path = "crates/sqlbuild-scopes" }

‎crates/README.md‎

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
# Native crates
22

33
SQLBuild's native code is a Cargo workspace that builds one Python extension module,
4-
`sqlbuild._native`, and the `sqb`/`sqlbuild` executables in the `sqlbuild-cli` wheel. Only `sqlbuild-python` knows about Python; every other crate is plain Rust
5-
that can be tested and benchmarked with `cargo test` alone.
4+
`sqlbuild._native`, and the `sqb`/`sqlbuild` executables in the `sqlbuild-cli` wheel. Only the
5+
`sqlbuild-python*` binding crates know about Python; every other crate is plain Rust that can be
6+
tested and benchmarked with `cargo test` alone.
67

78
| Crate | Responsibility |
89
|---|---|
@@ -26,14 +27,15 @@ that can be tested and benchmarked with `cargo test` alone.
2627
| `sqlbuild-refactor` | Native rename/move plans, contract-adoption edits, MODEL/SCHEMA/YAML text edits and atomic staging/commit with rollback. Shared scope placement delegates to the native scope implementation; user Python source inspection remains at the Python AST boundary. |
2728
| `sqlbuild-project` | The compiled project as plain Rust: `CompiledProject` with the producers' facts and each resource's records (model inputs and schemas, compiled models and their analysis, sources, functions, audits and seeds), and what those records answer without Python, such as lineage rows, resolved namespaces and dependency keys. Values the host owns, such as user config and discovered files, and the views it builds lazily live in one slot per record, whose types `sqlbuild-python` supplies through the `ProjectHost` trait; reads that need Python stay in the bindings as extension traits. |
2829
| `sqlbuild-cli` | The `sqb` and `sqlbuild` executables: `--version`, unchanged compiles replayed from the shared reuse store with Python's exact streams, exit code and compute log capture, and a handoff of every other invocation to the installed `sqlbuild-python` script. Built as its own wheel (`crates/sqlbuild-cli/pyproject.toml`). |
29-
| `sqlbuild-python` | The only PyO3 crate: the `_native` module, its Python classes and functions, conversions from Python objects, and the process allocator. |
30+
| `sqlbuild-python-base` | The base of the PyO3 bindings: native record, enum and error classes with their specs, registration, construction and field reads; keyword calls, imports, conversions and panics at the Python boundary; host entry calls; and the process allocator. |
31+
| `sqlbuild-python` | The `_native` module: its Python classes and functions and the conversions from Python objects, on top of `sqlbuild-python-base`. |
3032

3133
## Layering
3234

3335
Dependencies point one way, from the top of this graph to the bottom:
3436

3537
```text
36-
sqlbuild-python
38+
sqlbuild-python -> sqlbuild-python-base
3739
-> sqlbuild-project -> sqlbuild-refactor -> sqlbuild-rules -> sqlbuild-analysis -> sqlbuild-attachments
3840
-> sqlbuild-model-config -> sqlbuild-scopes -> sqlbuild-render -> sqlbuild-discovery -> sqlbuild-host
3941
-> sqlbuild-config -> sqlbuild-sqltext -> sqlbuild-cache -> sqlbuild-pysource -> sqlbuild-state -> sqlbuild-cursor -> sqlbuild-core
@@ -52,7 +54,7 @@ in `[workspace.metadata.native-layers]` in the workspace `Cargo.toml`, and
5254

5355
- a crate depends on a crate that is not in an earlier layer, or a workspace crate is missing
5456
from the order;
55-
- any crate other than `sqlbuild-python` reaches `pyo3`, directly or through another package;
57+
- a crate not listed in `python-boundary` reaches `pyo3`, directly or through another package;
5658
- a crate below `sqlbuild-analysis` reaches polyglot.
5759

5860
The same check fails when the `sqlbuild-cli` and `sqlbuild-python` versions, the `pyproject.toml`
@@ -63,7 +65,8 @@ entry carries a version other than its manifest's.
6365
`main/` entry points, `models.rs`, `types.rs` and `constants.rs`, not through `_helpers/`.
6466

6567
Inside `sqlbuild-python`, the custom rule XSB075 orders the `bindings/_helpers` modules in layers
66-
that follow the planned crate split: shared Python plumbing (`infra`), then the glue of the lower
68+
that follow the planned crate split: shared Python plumbing (`infra`, part of which is already
69+
`sqlbuild-python-base`, which Cargo keeps below every other layer), then the glue of the lower
6770
crates (`front`), the compiled project (`project`), planning and compilation as siblings (`plan`,
6871
`compile`), command output (`output`), command handlers (`commands`) and registration
6972
(`registry`). A module may import only from lower layers; `plan` and `compile` may not import each
@@ -88,10 +91,10 @@ empty: every import points down, and a new upward import fails the check.
8891
`sqlbuild._native_host`.
8992

9093
Staying-Python entry points (adapter, user code, executor and runtime, third-party and public
91-
facade) are reached through the host table and `native_host::host::entry`, which imports only
92-
`sqlbuild._native_host`. Classes, enums, errors and functions `sqlbuild._native` registers are
93-
reached through the typed registry (`records::registry` with `RecordId`, `ErrorId` and
94-
`NativeAttr`), never by module path.
94+
facade) are reached through the host table and `host_entry` in `sqlbuild-python-base`, which
95+
imports only `sqlbuild._native_host`. Classes, enums, errors and functions `sqlbuild._native` registers are
96+
reached through the typed registry of `sqlbuild-python-base` (`records` with `RecordId`,
97+
`ErrorId` and `NativeAttr`), never by module path.
9598

9699
## Compiler
97100

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
[package]
2+
name = "sqlbuild-python-base"
3+
description = "The shared base of the sqlbuild._native bindings: native records, Python calls and host entries."
4+
version.workspace = true
5+
edition.workspace = true
6+
license.workspace = true
7+
publish.workspace = true
8+
rust-version.workspace = true
9+
10+
[dependencies]
11+
mimalloc.workspace = true
12+
pyo3.workspace = true
13+
serde_json.workspace = true
14+
sqlbuild-analysis.workspace = true
15+
sqlbuild-config.workspace = true
16+
sqlbuild-core.workspace = true
17+
sqlbuild-discovery.workspace = true
18+
sqlbuild-host.workspace = true
19+
sqlbuild-rules.workspace = true
20+
21+
[lints]
22+
workspace = true

crates/sqlbuild-python/src/bindings/_helpers/boundary/allocator.rs renamed to crates/sqlbuild-python-base/src/boundary/_helpers/allocator.rs

File renamed without changes.

crates/sqlbuild-python/src/bindings/_helpers/boundary/call_arguments.rs renamed to crates/sqlbuild-python-base/src/boundary/_helpers/call_arguments.rs

File renamed without changes.

crates/sqlbuild-python/src/bindings/_helpers/boundary/config_values.rs renamed to crates/sqlbuild-python-base/src/boundary/_helpers/config_values.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
//! Loaded configuration values as the Python objects `tomllib` builds.
22
3-
use crate::bindings::_helpers::records::fields::field as native_field;
3+
use crate::records::main::fields::field::field as native_field;
44
use pyo3::IntoPyObjectExt;
55
use pyo3::prelude::{Bound, IntoPyObject, Py, PyAny, PyAnyMethods, PyModule, PyResult, Python};
66
use pyo3::types::{IntoPyDict, PyDict, PyDictMethods, PyList, PyListMethods};

crates/sqlbuild-python/src/bindings/_helpers/boundary/expected_errors.rs renamed to crates/sqlbuild-python-base/src/boundary/_helpers/expected_errors.rs

File renamed without changes.

crates/sqlbuild-python/src/bindings/_helpers/boundary/mod.rs renamed to crates/sqlbuild-python-base/src/boundary/_helpers/mod.rs

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@ mod allocator;
55
pub(crate) mod call_arguments;
66
pub(crate) mod config_values;
77
pub(crate) mod expected_errors;
8-
pub(crate) mod keyword_request;
98
pub(crate) mod oracles;
109
pub(crate) mod panics;
1110
mod positional_records;

crates/sqlbuild-python/src/bindings/_helpers/boundary/oracles.rs renamed to crates/sqlbuild-python-base/src/boundary/_helpers/oracles.rs

File renamed without changes.

0 commit comments

Comments
 (0)