Skip to content

Commit f08ff72

Browse files
committed
perf: keep the build identity out of sqlbuild-rules so edits stop rebuilding it
The source hash that keys local caches was a compile-time constant in sqlbuild-rules, whose build script hashes every workspace crate. Any source edit therefore rebuilt sqlbuild-rules and every crate above it, and the constant changed the same codegen units of the extension on every build. The hash now lives in a leaf crate, sqlbuild-build-identity, behind one non-inlined accessor. sqb and the extension install it into sqlbuild-rules at startup through sqlbuild-cli; reading it before then fails instead of falling back. Cache keys and BUILD_IDENTITY keep the same value.
1 parent 899efe2 commit f08ff72

72 files changed

Lines changed: 520 additions & 110 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 8 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
[workspace]
22
members = [
33
"crates/sqlbuild-build-features",
4+
"crates/sqlbuild-build-identity",
45
"crates/sqlbuild-core",
56
"crates/sqlbuild-cursor",
67
"crates/sqlbuild-process-ownership",
@@ -33,6 +34,7 @@ resolver = "2"
3334
[workspace.metadata.native-layers]
3435
order = [
3536
"sqlbuild-build-features",
37+
"sqlbuild-build-identity",
3638
"sqlbuild-core",
3739
"sqlbuild-cursor",
3840
"sqlbuild-process-ownership",
@@ -162,6 +164,7 @@ utoipa = { version = "5", features = ["chrono", "uuid"] }
162164
sqlbuild-analysis = { path = "crates/sqlbuild-analysis" }
163165
sqlbuild-attachments = { path = "crates/sqlbuild-attachments" }
164166
sqlbuild-build-features = { path = "crates/sqlbuild-build-features" }
167+
sqlbuild-build-identity = { path = "crates/sqlbuild-build-identity" }
165168
sqlbuild-cache = { path = "crates/sqlbuild-cache" }
166169
sqlbuild-cli = { path = "crates/sqlbuild-cli" }
167170
sqlbuild-config = { path = "crates/sqlbuild-config" }

‎crates/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ tested and benchmarked with `cargo test` alone.
88
| Crate | Responsibility |
99
|---|---|
1010
| `sqlbuild-build-features` | No code. Enables, in the `sqb` build, the dependency features the extension build resolves, so Cargo compiles the crates both builds share (polyglot-sql, `sqlbuild-analysis`, `sqlbuild-rules` and the rest) once. `make check-build-features` (part of `make check-ci`) fails when the two builds resolve a shared crate differently and names the features to add. |
11+
| `sqlbuild-build-identity` | The content identity of the native build: a hash of every workspace crate's sources, manifest and lockfile, behind one accessor. It rebuilds after any source edit, so only the top-level crates (`sqlbuild-python` and `sqlbuild-cli`) depend on it; they install the identity into `sqlbuild-rules` at startup, so the crates in between keep their incremental builds. |
1112
| `sqlbuild-core` | Constants shared by several crates, the panic boundary that turns an unwinding native failure into a named compiler error, passive identity, diagnostic and SQL value types, text positions, a JSON emitter whose output matches Python's `json.dumps` and `orjson` byte for byte, and run-cost allocation with exact Python `decimal` default-context arithmetic and Snowflake query-history SQL. |
1213
| `sqlbuild-cursor` | Microbatch cursor algebra: typed cursor values, grain flooring and stepping, half-open interval clamping, merging and batch splitting, timestamp text normalization and calendar durations, with Python `datetime` results and errors. Zones other than fixed offsets are resolved through a caller-supplied lookup. |
1314
| `sqlbuild-process-ownership` | Proof that a live process owns a scheduler worker claim, and the only way to stop one: an inherited claim lock on Linux and macOS, checked through `/proc` and a pidfd or through libproc with a start-time recheck, and a private named Job object on Windows. It is the only crate allowed unsafe code, for those platform calls. |
@@ -23,7 +24,7 @@ tested and benchmarked with `cargo test` alone.
2324
| `sqlbuild-render` | Macro call scanning, argument evaluation, expansion, splicing, memoisation and persistent call-store keys. Python executes user macro bodies at the extension boundary. |
2425
| `sqlbuild-attachments` | Native attachment of tests, audits, sources, functions, scenarios and seeds, including argument merging, validation and audit policy resolution. Adapter value rendering remains an extension boundary. |
2526
| `sqlbuild-analysis` | SQL analysis over polyglot: query analysis, binding, semantic validation and usage, column references, type normalization, FAST/RICH lineage, SQL-test/scenario extraction and compiler-side assembly, and Snowflake cost statement classification and query-tag parsing. |
26-
| `sqlbuild-rules` | Built-in rules and the rules engine, the custom-rule host, SQL lint, quality checks and formatting, rules configuration and the request models. It also owns the build identity script. |
27+
| `sqlbuild-rules` | Built-in rules and the rules engine, the custom-rule host, SQL lint, quality checks and formatting, rules configuration and the request models. Its caches and rule fingerprints key on the build identity the binary installs at startup. |
2728
| `sqlbuild-refactor` | Native rename/move plans, contract-adoption edits, MODEL/SCHEMA/YAML text edits and atomic staging/commit with rollback. Shared scope placement delegates to the native scope implementation; user Python source inspection remains at the Python AST boundary. |
2829
| `sqlbuild-project` | The compiled project as plain Rust: `CompiledProject` with the producers' facts and each resource's records (model inputs and schemas, compiled models and their analysis, sources, functions, audits and seeds), and what those records answer without Python, such as lineage rows, resolved namespaces and dependency keys. Values the host owns, such as user config and discovered files, and the views it builds lazily live in one slot per record, whose types `sqlbuild-python` supplies through the `ProjectHost` trait; reads that need Python stay in the bindings as extension traits. |
2930
| `sqlbuild-cli` | The `sqb` and `sqlbuild` executables: `--version`, unchanged compiles replayed from the shared reuse store with Python's exact streams, exit code and compute log capture, and a handoff of every other invocation to the installed `sqlbuild-python` script. Built as its own wheel (`crates/sqlbuild-cli/pyproject.toml`). |
@@ -40,6 +41,7 @@ sqlbuild-python -> sqlbuild-python-base
4041
-> sqlbuild-model-config -> sqlbuild-scopes -> sqlbuild-render -> sqlbuild-discovery -> sqlbuild-host
4142
-> sqlbuild-config -> sqlbuild-sqltext -> sqlbuild-cache -> sqlbuild-pysource -> sqlbuild-state -> sqlbuild-cursor -> sqlbuild-core
4243
sqlbuild-cli -> sqlbuild-cache, sqlbuild-core, sqlbuild-build-features, sqlbuild-process-ownership
44+
sqlbuild-python, sqlbuild-cli -> sqlbuild-build-identity
4345
```
4446

4547
`sqlbuild-scopes` reuses the macro scanner in `sqlbuild-render`; render itself depends only on
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
[package]
2+
name = "sqlbuild-build-identity"
3+
version.workspace = true
4+
description = "The content identity of this native build, which local caches key on so they notice source changes."
5+
edition.workspace = true
6+
license.workspace = true
7+
publish.workspace = true
8+
rust-version.workspace = true
9+
10+
# The build script hashes the sources of every workspace crate, so this crate rebuilds after any
11+
# source edit, and so does every crate that depends on it. Only the top-level crates that are
12+
# rebuilt anyway (sqlbuild-python and sqlbuild-cli) depend on it; they install the identity into
13+
# sqlbuild-rules at startup.
14+
[build-dependencies]
15+
sha2.workspace = true
16+
17+
[lints]
18+
workspace = true
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
//! The build identity accessor.
2+
3+
pub mod native_build_identity;
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
//! A content hash of every native crate, including the release version in their manifests.
2+
3+
/// The source content hash, never inlined so callers do not change, and lose incremental reuse, after every edit.
4+
#[inline(never)]
5+
#[must_use]
6+
pub fn native_build_identity() -> &'static str {
7+
env!("SQLBUILD_NATIVE_SOURCE_HASH")
8+
}
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
//! The source content hash the build script derives, behind one accessor.
2+
3+
pub mod main;
4+
5+
#[cfg(test)]
6+
mod tests;
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
#[path = "tests/test_identity.rs"]
2+
mod identity;
3+
#[path = "tests/test_types.rs"]
4+
mod test_types;
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
use crate::identity::main::native_build_identity::native_build_identity;
2+
use crate::identity::tests::test_types;
3+
4+
#[test]
5+
fn given_native_build_when_reading_cache_identity_then_it_is_a_source_hash() {
6+
let test_cases = [test_types::BuildIdentityTestCase {
7+
description: "cache identity is a source content hash",
8+
expected_hash_length: 16,
9+
}];
10+
for test_case in &test_cases {
11+
let hash = native_build_identity();
12+
13+
assert!(
14+
hash.len() == test_case.expected_hash_length
15+
&& hash.bytes().all(|byte| byte.is_ascii_hexdigit()),
16+
"{}",
17+
test_case.description
18+
);
19+
}
20+
}
21+
22+
#[test]
23+
fn given_workspace_checkout_when_deriving_build_identity_then_workspace_files_are_hashed() {
24+
let test_cases = [
25+
test_types::HashedWorkspaceFileTestCase {
26+
description: "dependency features and build profiles change the identity",
27+
workspace_file: "../../Cargo.toml",
28+
expected_hashed: true,
29+
},
30+
test_types::HashedWorkspaceFileTestCase {
31+
description: "resolved dependency versions change the identity",
32+
workspace_file: "../../Cargo.lock",
33+
expected_hashed: true,
34+
},
35+
];
36+
for test_case in &test_cases {
37+
let hashed = env!("SQLBUILD_NATIVE_HASHED_WORKSPACE_FILES")
38+
.split(',')
39+
.any(|file| file == test_case.workspace_file);
40+
assert_eq!(
41+
hashed, test_case.expected_hashed,
42+
"{}",
43+
test_case.description
44+
);
45+
}
46+
}

0 commit comments

Comments
 (0)