Skip to content

Commit 799db35

Browse files
authored
feat(native): resolve declaration scopes natively behind a native-preview engine tier (#1006)
* feat(native): add the native declaration scope crate The new sqlbuild-scopes crate projects the scope builder's walked resources and declarations into the static scope index with the Python builder's record orders, duplicate and naming diagnostics and macro dependency usages, the lookup's canonical orders and repr-ordered groups, declaration visibility and the expected-model and tested-macro relationship grants. Any path Python would reject defers the stage. * feat(native): resolve declaration scopes natively behind the compiler engine switch With the native engine, the scope index, its validation, relationship grants and the scope lookup are built by sqlbuild-scopes and materialised as the same Python records, orders and mappings. Python keeps the relationship-name extraction and every error message: an invalid index or a relationship fault raises exactly as before, and a project the native index cannot reproduce runs the Python stage. sqb scope uses the same path. The Python engine is unchanged. * feat(native): compare declaration scopes in the compiler differential The failure corpus gains scope cases: a duplicate scoped declaration, inaccessible local and scoped declarations, the first of two visibility errors, a test outside its relationship grants and a malformed expected model. A harness test compares a scoped project with expected-model and macro-test grants under both engines. * refactor(native): group the Python bindings by domain Move the PyO3 binding helpers into per-domain folders (analysis, boundary, discovery, rules, scopes, sqltext) and split the flat function module along the same lines. Function bodies are unchanged; only module paths and registration move. * feat(native): add a native-preview compiler engine tier Each native stage declares a shipped or preview tier in one registry. The native engine runs shipped stages only (discovery); native-preview adds stages still in development (declaration scopes). Preview keeps its own compiler, Rules and reuse stores. The per-PR differential compares python with native-preview on the full corpus and python with native on the seeds and failure corpus. * feat(native): scan SQL tests natively for expected-model relationships Under native-preview the declaration scope reads each model-mode SQL test's and scenario's __expected__<model> names with a native port of the Python top-level CTE scan, using the adapter's lexical rules. Any text Python would reject, or a non-ASCII character at a code position, defers that body to the Python extraction, so names and fault messages stay byte-identical. * ci: cover the shipped engine in the dense differential and gate on the Python version job
1 parent 1175b83 commit 799db35

112 files changed

Lines changed: 6331 additions & 694 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/compiler-differential.yml‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ jobs:
1616
dense:
1717
name: Dense compiler differential (${{ matrix.models }} models)
1818
runs-on: ubuntu-24.04
19-
timeout-minutes: 90
19+
timeout-minutes: 150
2020
strategy:
2121
fail-fast: false
2222
matrix:
@@ -37,8 +37,14 @@ jobs:
3737
run: uv python install 3.12
3838
- name: Install dependencies
3939
run: uv sync --locked --all-groups --all-extras
40-
- name: Compare dense projects with and without custom rules
40+
- name: Compare dense projects with and without custom rules on the preview engine
4141
env:
4242
MODELS: ${{ matrix.models }}
4343
run: make compiler-differential-dense COMPILER_DIFFERENTIAL_DENSE_MODELS="${MODELS}"
44+
- name: Compare dense projects with and without custom rules on the shipped engine
45+
env:
46+
MODELS: ${{ matrix.models }}
47+
run: >-
48+
make compiler-differential-dense COMPILER_DIFFERENTIAL_DENSE_MODELS="${MODELS}"
49+
COMPILER_DIFFERENTIAL_ARGS="--engines python native"
4450

‎.github/workflows/verify.yml‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -290,11 +290,13 @@ jobs:
290290
run: uv python install 3.12
291291
- name: Install dependencies
292292
run: uv sync --locked --all-groups --all-extras
293-
- name: Compare fixtures, examples, generated seeds and the failure corpus
293+
- name: Compare fixtures, examples, generated seeds and the failure corpus (python vs native-preview)
294294
run: >-
295295
make compiler-differential
296296
COMPILER_DIFFERENTIAL_ARGS="--stage-captures --require-discovery-coverage
297297
--require-render-coverage"
298+
- name: Compare generated seeds and the failure corpus (python vs shipped native)
299+
run: make compiler-differential-shipped
298300

299301
compiler_differential_python_versions:
300302
name: Native discovery on Python ${{ matrix.python }}
@@ -333,11 +335,12 @@ jobs:
333335
uv run --no-sync pytest
334336
tests/integration/src/sqlbuild/compiler/test_native_python_compat_oracle.py
335337
-n auto --dist loadfile
336-
- name: Compare engines on fixtures, examples, the failure corpus and a few seeds
338+
- name: Compare python with the shipped native default on fixtures, examples, the failure corpus and a few seeds
337339
env:
338340
PYTHONUNBUFFERED: "1"
339341
run: >-
340342
uv run --no-sync python -m scripts.run_compiler_differential
343+
--engines python native
341344
--corpus fixtures examples failures seeds --seeds 4 --jobs 4
342345
343346
format_performance:
@@ -758,6 +761,7 @@ jobs:
758761
checks,
759762
e2e_shards,
760763
compiler_differential,
764+
compiler_differential_python_versions,
761765
format_performance,
762766
performance,
763767
cold_compile_performance,
@@ -778,6 +782,7 @@ jobs:
778782
CHECKS_RESULT: ${{ needs.checks.result }}
779783
E2E_SHARDS_RESULT: ${{ needs.e2e_shards.result }}
780784
COMPILER_DIFFERENTIAL_RESULT: ${{ needs.compiler_differential.result }}
785+
COMPILER_DIFFERENTIAL_PYTHON_VERSIONS_RESULT: ${{ needs.compiler_differential_python_versions.result }}
781786
FORMAT_PERFORMANCE_RESULT: ${{ needs.format_performance.result }}
782787
PERFORMANCE_RESULT: ${{ needs.performance.result }}
783788
COLD_COMPILE_PERFORMANCE_RESULT: ${{ needs.cold_compile_performance.result }}
@@ -800,6 +805,7 @@ jobs:
800805
for result in \
801806
"${E2E_SHARDS_RESULT}" \
802807
"${COMPILER_DIFFERENTIAL_RESULT}" \
808+
"${COMPILER_DIFFERENTIAL_PYTHON_VERSIONS_RESULT}" \
803809
"${FORMAT_PERFORMANCE_RESULT}" \
804810
"${PERFORMANCE_RESULT}" \
805811
"${COLD_COMPILE_PERFORMANCE_RESULT}" \

‎Cargo.lock‎

Lines changed: 8 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ members = [
44
"crates/sqlbuild-sqltext",
55
"crates/sqlbuild-config",
66
"crates/sqlbuild-discovery",
7+
"crates/sqlbuild-scopes",
78
"crates/sqlbuild-analysis",
89
"crates/sqlbuild-rules",
910
"crates/sqlbuild-python",
@@ -19,6 +20,7 @@ order = [
1920
"sqlbuild-sqltext",
2021
"sqlbuild-config",
2122
"sqlbuild-discovery",
23+
"sqlbuild-scopes",
2224
"sqlbuild-analysis",
2325
"sqlbuild-rules",
2426
"sqlbuild-python",
@@ -68,6 +70,7 @@ sqlbuild-config = { path = "crates/sqlbuild-config" }
6870
sqlbuild-core = { path = "crates/sqlbuild-core" }
6971
sqlbuild-discovery = { path = "crates/sqlbuild-discovery" }
7072
sqlbuild-rules = { path = "crates/sqlbuild-rules" }
73+
sqlbuild-scopes = { path = "crates/sqlbuild-scopes" }
7174
sqlbuild-sqltext = { path = "crates/sqlbuild-sqltext" }
7275
stacker = "0.1"
7376
sqlparser = { version = "0.58", features = ["visitor"] }

‎Makefile‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ SHELL := /bin/bash
77
test-e2e-duckdb-integrations test-e2e-performance \
88
test-e2e-cold-compile-performance test-e2e-cache-compile-performance \
99
test-e2e-dense-compile-performance test-e2e-varied-cache-performance \
10-
compiler-differential compiler-differential-dense
10+
compiler-differential compiler-differential-shipped compiler-differential-dense
1111

1212
format:
1313
uv run ruff format .
@@ -289,6 +289,7 @@ test-e2e-varied-cache-performance:
289289

290290
COMPILER_DIFFERENTIAL_JOBS ?= 4
291291
COMPILER_DIFFERENTIAL_SEEDS ?= 12
292+
COMPILER_DIFFERENTIAL_SHIPPED_SEEDS ?= 12
292293
COMPILER_DIFFERENTIAL_DENSE_MODELS ?= 3000
293294
COMPILER_DIFFERENTIAL_ARGS ?=
294295

@@ -297,6 +298,14 @@ compiler-differential:
297298
--jobs $(COMPILER_DIFFERENTIAL_JOBS) --seeds $(COMPILER_DIFFERENTIAL_SEEDS) \
298299
$(COMPILER_DIFFERENTIAL_ARGS)
299300

301+
# python vs native-preview above covers every native stage; this keeps the shipped `native` default
302+
# covered on a bounded corpus.
303+
compiler-differential-shipped:
304+
env PYTHONUNBUFFERED=1 uv run python -m scripts.run_compiler_differential \
305+
--engines python native --corpus seeds failures \
306+
--jobs $(COMPILER_DIFFERENTIAL_JOBS) --seeds $(COMPILER_DIFFERENTIAL_SHIPPED_SEEDS) \
307+
$(COMPILER_DIFFERENTIAL_ARGS)
308+
300309
compiler-differential-dense:
301310
env PYTHONUNBUFFERED=1 uv run python -m scripts.run_compiler_differential \
302311
--corpus dense --dense-models $(COMPILER_DIFFERENTIAL_DENSE_MODELS) \

‎crates/README.md‎

Lines changed: 28 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ that can be tested and benchmarked with `cargo test` alone.
1010
| `sqlbuild-sqltext` | Lexical SQL text without polyglot: comment, quote and parenthesis scanning, the rules quote policy, model header tokenization and matching, static variable substitution and static reference extraction. |
1111
| `sqlbuild-config` | Configuration file reading without Python: `tomllib`-compatible TOML, PyYAML `safe_load`-compatible YAML 1.1, and typed project and local config readers for the fields discovery needs. Any error means "defer to Python", which re-parses for exact messages. |
1212
| `sqlbuild-discovery` | Native project discovery: the shared directory walk with Python's glob and sort semantics, file reading with Python's newline handling, and parsing of authored files with Python's exact discovery messages. Results are plain data the Python discovery facade materialises; anything it cannot reproduce exactly defers to Python. |
13+
| `sqlbuild-scopes` | Declaration scopes: the scope index with Python's record orders and diagnostics, declaration visibility, relationship grants, the scope lookup groups, and the dialect-aware scan of SQL tests and scenarios for their expected models. Anything it cannot reproduce exactly defers to Python. |
1314
| `sqlbuild-analysis` | SQL analysis over polyglot: SQL tokens, query analysis, the binding catalog, semantic validation and usage, column references, and SQL-test extraction, planning and rendering. |
1415
| `sqlbuild-rules` | Built-in rules and the rules engine, the custom-rule host, SQL lint, quality checks and formatting, rules configuration and the request models. It also owns the build identity script. |
1516
| `sqlbuild-python` | The only PyO3 crate: the `_native` module, its Python classes and functions, conversions from Python objects, and the process allocator. |
@@ -20,13 +21,14 @@ Dependencies point one way, from the top of this graph to the bottom:
2021

2122
```text
2223
sqlbuild-python
23-
-> sqlbuild-rules -> sqlbuild-analysis -> sqlbuild-discovery -> sqlbuild-config
24+
-> sqlbuild-rules -> sqlbuild-analysis -> sqlbuild-scopes -> sqlbuild-discovery -> sqlbuild-config
2425
-> sqlbuild-sqltext -> sqlbuild-core
2526
```
2627

27-
`sqlbuild-config` does not depend on the crates below it today; its place in the order only
28-
fixes which crates may use it. The JSON emitter lives in `sqlbuild-core` so that any layer can
29-
produce text that must equal Python's `json.dumps` output.
28+
`sqlbuild-config` does not depend on the crates below it today, and `sqlbuild-scopes` uses only
29+
`sqlbuild-sqltext`; their place in the order fixes which crates may use them. The JSON emitter
30+
lives in `sqlbuild-core` so that any layer can produce text that must equal Python's
31+
`json.dumps` output.
3032

3133
Each crate may depend only on crates below it, and may also skip layers. The order is declared
3234
in `[workspace.metadata.native-layers]` in the workspace `Cargo.toml`, and
@@ -43,6 +45,28 @@ The same check fails when the workspace version, the `pyproject.toml` version an
4345
`fensu check` applies the Rust structure rules inside each crate: other crates use items through
4446
`main/` entry points, `models.rs`, `types.rs` and `constants.rs`, not through `_helpers/`.
4547

48+
## Compiler engines
49+
50+
`SQLBUILD_COMPILER_ENGINE` (or the hidden `--compiler-engine` flag) selects which compiler
51+
stages run natively:
52+
53+
| Engine | Runs |
54+
|---|---|
55+
| `python` | The Python compiler only. It is the oracle every native stage must match byte for byte. |
56+
| `native` | The default: native stages that passed their flip gate (`shipped` tier). |
57+
| `native-preview` | Opt-in: shipped stages plus stages still in development (`preview` tier). |
58+
59+
Each native stage declares its tier once, in `NATIVE_STAGE_TIERS` in
60+
`src/sqlbuild/compiler/frontier/constants.py`. A stage moves from `preview` to `shipped` by
61+
changing that line, after its flip gate passes: a byte-identical real project, a green
62+
differential, and no measured slowdown on cold, edit and no-change compiles. Every engine keeps
63+
its own compiler, Rules and compile-reuse stores, so preview output is never reused by `native`.
64+
65+
`make compiler-differential` compares `python` with `native-preview` on the full per-PR corpus.
66+
`make compiler-differential-shipped` compares `python` with `native` on the generated seeds and
67+
the failure corpus, so the shipped default stays covered on its own. CI runs both, and compares
68+
`python` with `native` on Python 3.13 and 3.14 as well.
69+
4670
## Working on the crates
4771

4872
- Tests live next to the module they cover; run them for one crate with

‎crates/sqlbuild-python/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ sqlbuild-config.workspace = true
2525
sqlbuild-core.workspace = true
2626
sqlbuild-discovery.workspace = true
2727
sqlbuild-rules.workspace = true
28+
sqlbuild-scopes.workspace = true
2829
sqlbuild-sqltext.workspace = true
2930

3031
[lints]

crates/sqlbuild-python/src/bindings/_helpers/catalog.rs renamed to crates/sqlbuild-python/src/bindings/_helpers/analysis/catalog.rs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
//! Python methods of the compile-owned binding catalog and its compact analysis jobs.
22
3-
use crate::bindings::_helpers::conversions;
4-
use crate::bindings::_helpers::normalization_results::normalization_results;
5-
use crate::bindings::_helpers::panics::compiler_error;
3+
use crate::bindings::_helpers::analysis::conversions;
4+
use crate::bindings::_helpers::analysis::normalization_results::normalization_results;
5+
use crate::bindings::_helpers::boundary::panics::compiler_error;
66
use crate::bindings::models::{CatalogInput, CompactAnalysisJob, ProjectCatalog};
77
use crate::bindings::types::CompilerDetach;
88
use pyo3::exceptions::PyValueError;

crates/sqlbuild-python/src/bindings/_helpers/conversions.rs renamed to crates/sqlbuild-python/src/bindings/_helpers/analysis/conversions.rs

File renamed without changes.
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
mod catalog;
2+
pub(crate) mod conversions;
3+
pub(crate) mod normalization;
4+
pub(crate) mod normalization_results;
5+
mod positions;
6+
pub(crate) mod queries;
7+
pub(crate) mod sql_tests;

0 commit comments

Comments
 (0)