Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[workspace]
members = [
"crates/sqlbuild-build-features",
"crates/sqlbuild-build-identity",
"crates/sqlbuild-core",
"crates/sqlbuild-cursor",
"crates/sqlbuild-process-ownership",
Expand Down Expand Up @@ -33,6 +34,7 @@ resolver = "2"
[workspace.metadata.native-layers]
order = [
"sqlbuild-build-features",
"sqlbuild-build-identity",
"sqlbuild-core",
"sqlbuild-cursor",
"sqlbuild-process-ownership",
Expand Down Expand Up @@ -162,6 +164,7 @@ utoipa = { version = "5", features = ["chrono", "uuid"] }
sqlbuild-analysis = { path = "crates/sqlbuild-analysis" }
sqlbuild-attachments = { path = "crates/sqlbuild-attachments" }
sqlbuild-build-features = { path = "crates/sqlbuild-build-features" }
sqlbuild-build-identity = { path = "crates/sqlbuild-build-identity" }
sqlbuild-cache = { path = "crates/sqlbuild-cache" }
sqlbuild-cli = { path = "crates/sqlbuild-cli" }
sqlbuild-config = { path = "crates/sqlbuild-config" }
Expand Down
4 changes: 3 additions & 1 deletion crates/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ tested and benchmarked with `cargo test` alone.
| Crate | Responsibility |
|---|---|
| `sqlbuild-build-features` | No code. Enables, in the `sqb` build, the dependency features the extension build resolves, so Cargo compiles the crates both builds share (polyglot-sql, `sqlbuild-analysis`, `sqlbuild-rules` and the rest) once. `make check-build-features` (part of `make check-ci`) fails when the two builds resolve a shared crate differently and names the features to add. |
| `sqlbuild-build-identity` | The content identity of the native build: a hash of every workspace crate's sources, manifest and lockfile, behind one accessor. It rebuilds after any source edit, so only the top-level crates (`sqlbuild-python` and `sqlbuild-cli`) depend on it; they install the identity into `sqlbuild-rules` at startup, so the crates in between keep their incremental builds. |
| `sqlbuild-core` | Constants shared by several crates, the panic boundary that turns an unwinding native failure into a named compiler error, passive identity, diagnostic and SQL value types, text positions, a JSON emitter whose output matches Python's `json.dumps` and `orjson` byte for byte, and run-cost allocation with exact Python `decimal` default-context arithmetic and Snowflake query-history SQL. |
| `sqlbuild-cursor` | Microbatch cursor algebra: typed cursor values, grain flooring and stepping, half-open interval clamping, merging and batch splitting, timestamp text normalization and calendar durations, with Python `datetime` results and errors. Zones other than fixed offsets are resolved through a caller-supplied lookup. |
| `sqlbuild-process-ownership` | Proof that a live process owns a scheduler worker claim, and the only way to stop one: an inherited claim lock on Linux and macOS, checked through `/proc` and a pidfd or through libproc with a start-time recheck, and a private named Job object on Windows. It is the only crate allowed unsafe code, for those platform calls. |
Expand All @@ -23,7 +24,7 @@ tested and benchmarked with `cargo test` alone.
| `sqlbuild-render` | Macro call scanning, argument evaluation, expansion, splicing, memoisation and persistent call-store keys. Python executes user macro bodies at the extension boundary. |
| `sqlbuild-attachments` | Native attachment of tests, audits, sources, functions, scenarios and seeds, including argument merging, validation and audit policy resolution. Adapter value rendering remains an extension boundary. |
| `sqlbuild-analysis` | SQL analysis over polyglot: query analysis, binding, semantic validation and usage, column references, type normalization, FAST/RICH lineage, SQL-test/scenario extraction and compiler-side assembly, and Snowflake cost statement classification and query-tag parsing. |
| `sqlbuild-rules` | Built-in rules and the rules engine, the custom-rule host, SQL lint, quality checks and formatting, rules configuration and the request models. It also owns the build identity script. |
| `sqlbuild-rules` | Built-in rules and the rules engine, the custom-rule host, SQL lint, quality checks and formatting, rules configuration and the request models. Its caches and rule fingerprints key on the build identity the binary installs at startup. |
| `sqlbuild-refactor` | Native rename/move plans, contract-adoption edits, MODEL/SCHEMA/YAML text edits and atomic staging/commit with rollback. Shared scope placement delegates to the native scope implementation; user Python source inspection remains at the Python AST boundary. |
| `sqlbuild-project` | The compiled project as plain Rust: `CompiledProject` with the producers' facts and each resource's records (model inputs and schemas, compiled models and their analysis, sources, functions, audits and seeds), and what those records answer without Python, such as lineage rows, resolved namespaces and dependency keys. Values the host owns, such as user config and discovered files, and the views it builds lazily live in one slot per record, whose types `sqlbuild-python` supplies through the `ProjectHost` trait; reads that need Python stay in the bindings as extension traits. |
| `sqlbuild-cli` | The `sqb` and `sqlbuild` executables: `--version`, unchanged compiles replayed from the shared reuse store with Python's exact streams, exit code and compute log capture, and a handoff of every other invocation to the installed `sqlbuild-python` script. Built as its own wheel (`crates/sqlbuild-cli/pyproject.toml`). |
Expand All @@ -40,6 +41,7 @@ sqlbuild-python -> sqlbuild-python-base
-> sqlbuild-model-config -> sqlbuild-scopes -> sqlbuild-render -> sqlbuild-discovery -> sqlbuild-host
-> sqlbuild-config -> sqlbuild-sqltext -> sqlbuild-cache -> sqlbuild-pysource -> sqlbuild-state -> sqlbuild-cursor -> sqlbuild-core
sqlbuild-cli -> sqlbuild-cache, sqlbuild-core, sqlbuild-build-features, sqlbuild-process-ownership
sqlbuild-python, sqlbuild-cli -> sqlbuild-build-identity
```

`sqlbuild-scopes` reuses the macro scanner in `sqlbuild-render`; render itself depends only on
Expand Down
18 changes: 18 additions & 0 deletions crates/sqlbuild-build-identity/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
[package]
name = "sqlbuild-build-identity"
version.workspace = true
description = "The content identity of this native build, which local caches key on so they notice source changes."
edition.workspace = true
license.workspace = true
publish.workspace = true
rust-version.workspace = true

# The build script hashes the sources of every workspace crate, so this crate rebuilds after any
# source edit, and so does every crate that depends on it. Only the top-level crates that are
# rebuilt anyway (sqlbuild-python and sqlbuild-cli) depend on it; they install the identity into
# sqlbuild-rules at startup.
[build-dependencies]
sha2.workspace = true

[lints]
workspace = true
File renamed without changes.
3 changes: 3 additions & 0 deletions crates/sqlbuild-build-identity/src/identity/main/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
//! The build identity accessor.

pub mod native_build_identity;
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
//! A content hash of every native crate, including the release version in their manifests.

/// The source content hash, never inlined so callers do not change, and lose incremental reuse, after every edit.
#[inline(never)]
#[must_use]
pub fn native_build_identity() -> &'static str {
env!("SQLBUILD_NATIVE_SOURCE_HASH")
}
6 changes: 6 additions & 0 deletions crates/sqlbuild-build-identity/src/identity/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
//! The source content hash the build script derives, behind one accessor.

pub mod main;

#[cfg(test)]
mod tests;
4 changes: 4 additions & 0 deletions crates/sqlbuild-build-identity/src/identity/tests.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
#[path = "tests/test_identity.rs"]
mod identity;
#[path = "tests/test_types.rs"]
mod test_types;
46 changes: 46 additions & 0 deletions crates/sqlbuild-build-identity/src/identity/tests/test_identity.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
use crate::identity::main::native_build_identity::native_build_identity;
use crate::identity::tests::test_types;

#[test]
fn given_native_build_when_reading_cache_identity_then_it_is_a_source_hash() {
let test_cases = [test_types::BuildIdentityTestCase {
description: "cache identity is a source content hash",
expected_hash_length: 16,
}];
for test_case in &test_cases {
let hash = native_build_identity();

assert!(
hash.len() == test_case.expected_hash_length
&& hash.bytes().all(|byte| byte.is_ascii_hexdigit()),
"{}",
test_case.description
);
}
}

#[test]
fn given_workspace_checkout_when_deriving_build_identity_then_workspace_files_are_hashed() {
let test_cases = [
test_types::HashedWorkspaceFileTestCase {
description: "dependency features and build profiles change the identity",
workspace_file: "../../Cargo.toml",
expected_hashed: true,
},
test_types::HashedWorkspaceFileTestCase {
description: "resolved dependency versions change the identity",
workspace_file: "../../Cargo.lock",
expected_hashed: true,
},
];
for test_case in &test_cases {
let hashed = env!("SQLBUILD_NATIVE_HASHED_WORKSPACE_FILES")
.split(',')
.any(|file| file == test_case.workspace_file);
assert_eq!(
hashed, test_case.expected_hashed,
"{}",
test_case.description
);
}
}
10 changes: 10 additions & 0 deletions crates/sqlbuild-build-identity/src/identity/tests/test_types.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
pub(crate) struct BuildIdentityTestCase {
pub(crate) description: &'static str,
pub(crate) expected_hash_length: usize,
}

pub(crate) struct HashedWorkspaceFileTestCase {
pub(crate) description: &'static str,
pub(crate) workspace_file: &'static str,
pub(crate) expected_hashed: bool,
}
5 changes: 5 additions & 0 deletions crates/sqlbuild-build-identity/src/lib.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
//! The content identity of this native build, which local caches key on so they notice source changes.

#![forbid(unsafe_code)]

pub mod identity;
1 change: 1 addition & 0 deletions crates/sqlbuild-cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ serde.workspace = true
serde_json = { workspace = true, features = ["arbitrary_precision"] }
sqlbuild-analysis.workspace = true
sqlbuild-build-features.workspace = true
sqlbuild-build-identity.workspace = true
sqlbuild-cache.workspace = true
sqlbuild-config.workspace = true
sqlbuild-core.workspace = true
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
//! Install this build's identity into the rules engine, first thing in every native entry point.

use sqlbuild_build_identity::identity::main::native_build_identity::native_build_identity;
use sqlbuild_rules::build_identity::errors::BuildIdentityError;
use sqlbuild_rules::build_identity::main::install_build_identity::install_build_identity;

/// Install this build's identity into the rules engine and return it, before anything reads a cache key.
pub fn install_native_build_identity() -> Result<&'static str, BuildIdentityError> {
let identity: &'static str = native_build_identity();
install_build_identity(identity)?;
Ok(identity)
}
1 change: 1 addition & 0 deletions crates/sqlbuild-cli/src/launcher/main/mod.rs
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
pub mod install_native_build_identity;
pub mod run_launcher;
6 changes: 6 additions & 0 deletions crates/sqlbuild-cli/src/launcher/main/run_launcher.rs
Original file line number Diff line number Diff line change
@@ -1,14 +1,20 @@
//! The `sqb` executable's whole process.

use std::ffi::OsString;
use std::io::Write;
use std::process::ExitCode;
use std::time::{Instant, SystemTime};

use crate::launcher::_helpers::dispatch::launch;
use crate::launcher::_helpers::handoff::process_exit;
use crate::launcher::main::install_native_build_identity::install_native_build_identity;

/// Run this process's invocation; an exit code beyond one byte ends the process directly.
pub fn run_launcher() -> ExitCode {
if let Err(error) = install_native_build_identity() {
let _ = writeln!(std::io::stderr(), "sqb: {error}");
return ExitCode::FAILURE;
}
let started = (Instant::now(), SystemTime::now());
let arguments: Vec<OsString> = std::env::args_os().skip(1).collect();
process_exit(launch(arguments, started))
Expand Down
2 changes: 2 additions & 0 deletions crates/sqlbuild-cli/src/native_commands/tests/test_rules.rs
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
use crate::launcher::main::install_native_build_identity::install_native_build_identity;
use crate::native_commands::_helpers::rules::run_rules;
use crate::native_commands::errors::PythonNeeded;
use crate::native_commands::models::{RulesAction, RulesOptions};
Expand All @@ -9,6 +10,7 @@ const OPTIONS_PROJECT: &str = "name = \"shop\"\nadapter = \"duckdb\"\n\n[rules.r

#[test]
fn given_project_when_inspecting_rules_then_builtin_catalogue_prints_unless_python_owns_it() {
install_native_build_identity().expect("the build identity installs once");
let test_cases = [
RulesTestCase {
description: "list prints code and message lines",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use std::path::{Path, PathBuf};
use std::time::{Duration, SystemTime, UNIX_EPOCH};

use sqlbuild_analysis::definition_changes::main::definition_digest::definition_digest;
use sqlbuild_rules::constants::NATIVE_BUILD_IDENTITY;
use sqlbuild_rules::build_identity::main::native_build_identity::native_build_identity;

use crate::bindings::_helpers::change_selection::base_compile::NOTES_EXTENSION;
use crate::bindings::_helpers::change_selection::fingerprints::{
Expand Down Expand Up @@ -82,7 +82,7 @@ pub(crate) fn cache_key_with(
lookup: &dyn Fn(&str) -> Option<String>,
) -> std::io::Result<String> {
Ok(definition_digest(&[
NATIVE_BUILD_IDENTITY,
native_build_identity(),
target.unwrap_or_default(),
vars,
local_config,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//! Register the exported native constants and the cursor grain order on `sqlbuild._native`.
//! Register the exported native constants, the installed build identity and the cursor grain order on `sqlbuild._native`.

use pyo3::prelude::{Bound, PyAnyMethods, PyModule, PyModuleMethods, PyResult};
use pyo3::types::{PyDict, PyDictMethods, PyFrozenSet, PyTuple};
Expand All @@ -7,11 +7,15 @@ use crate::bindings::_helpers::cursor::conversion::GRAIN_NAMES;
use crate::bindings::_helpers::exported_constants::constants::{
EXPORTED_INTEGERS, EXPORTED_TEXT_SETS, EXPORTED_TEXT_TUPLES, EXPORTED_TEXTS,
};
use sqlbuild_cli::launcher::main::install_native_build_identity::install_native_build_identity;
use sqlbuild_python_base::boundary::main::compiler_error::compiler_error;
use sqlbuild_python_base::records::main::fields::field::field as native_field;

/// Registers after the native enums, since `GRAIN_ORDER` is keyed by `CursorGrain` members.
pub(crate) fn register(module: &Bound<'_, PyModule>) -> PyResult<()> {
let py = module.py();
let identity: &str = install_native_build_identity().map_err(compiler_error)?;
module.add("BUILD_IDENTITY", identity)?;
for (name, value) in EXPORTED_TEXTS {
module.add(*name, *value)?;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ use pyo3::prelude::{
};
use pyo3::types::{PyDict, PyDictMethods};
use pyo3::{pyfunction, wrap_pyfunction};
use sqlbuild_rules::constants::NATIVE_BUILD_IDENTITY;
use sqlbuild_rules::build_identity::main::native_build_identity::native_build_identity;
use sqlbuild_rules::errors::{SkillInstallError, SkillInstallErrorKind};
use sqlbuild_rules::models::RulesConfig;
use sqlbuild_rules::rule_catalogue::main::format_rule_text::format_rule_text;
Expand Down Expand Up @@ -118,7 +118,7 @@ fn fingerprint_rule(
implementation_hash: if custom {
custom_rule_implementation_fingerprint(rule, project_dir, None)?
} else {
NATIVE_BUILD_IDENTITY.to_owned()
native_build_identity().to_owned()
},
})
}
Expand Down
3 changes: 1 addition & 2 deletions crates/sqlbuild-python/src/bindings/main/register.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
use pyo3::prelude::{Bound, PyModule, PyModuleMethods, PyResult};
use pyo3::pymodule;
use sqlbuild_rules::constants::{API_VERSION, NATIVE_BUILD_IDENTITY};
use sqlbuild_rules::constants::API_VERSION;

use crate::bindings::_helpers::{
analysis, analysis_session, attachments, auditing, authored_values, cache, cli,
Expand Down Expand Up @@ -45,7 +45,6 @@ pub fn _native(module: &Bound<'_, PyModule>) -> PyResult<()> {
register_project_stages(module)?;
register_analysis_stages(module)?;
module.add("API_VERSION", API_VERSION)?;
module.add("BUILD_IDENTITY", NATIVE_BUILD_IDENTITY)?;
verify_attributes(module)
}

Expand Down
3 changes: 3 additions & 0 deletions crates/sqlbuild-python/src/bindings/tests/test_since_cache.rs
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
use std::path::PathBuf;

use sqlbuild_cli::launcher::main::install_native_build_identity::install_native_build_identity;

use crate::bindings::tests::helpers::{cache_key_under, project_reading_environment};
use crate::bindings::tests::test_types::CacheKeyTestCase;

const BASE_ENVIRONMENT: [(&str, &str); 2] = [("ORDERS_REGION", "eu"), ("UNREAD_SETTING", "a")];

#[test]
fn given_referenced_environment_variable_when_keying_base_cache_then_its_value_counts() {
install_native_build_identity().expect("the build identity installs once");
let project: PathBuf = project_reading_environment();
let base_key: String = cache_key_under(&project, &BASE_ENVIRONMENT);
let test_cases = [
Expand Down
3 changes: 0 additions & 3 deletions crates/sqlbuild-rules/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,6 @@ sqlclean.workspace = true
stacker.workspace = true
toml.workspace = true

[build-dependencies]
sha2.workspace = true

[dev-dependencies]
tempfile.workspace = true

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
//! The identity the binary installed, once per process.

use std::sync::OnceLock;

/// The build identity the binary installed at startup; empty until then.
pub(crate) static INSTALLED_BUILD_IDENTITY: OnceLock<String> = OnceLock::new();
3 changes: 3 additions & 0 deletions crates/sqlbuild-rules/src/build_identity/_helpers/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
//! The installed identity cell.

pub(crate) mod installed;
4 changes: 4 additions & 0 deletions crates/sqlbuild-rules/src/build_identity/constants.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
//! Messages of the build identity.

/// Why reading the identity before the binary installed it fails.
pub(crate) const MISSING_BUILD_IDENTITY: &str = "the native build identity was not installed: the binary must call install_build_identity at startup, before any cache key or rule fingerprint is read";
26 changes: 26 additions & 0 deletions crates/sqlbuild-rules/src/build_identity/errors.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
//! Why a build identity cannot be installed.

use std::fmt;

/// The reason `install_build_identity` refused an identity.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum BuildIdentityError {
/// The offered identity is empty, so it would match builds of any sources.
Empty,
/// A different identity is already installed in this process.
Conflicting { installed: String, offered: String },
}

impl fmt::Display for BuildIdentityError {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::Empty => formatter.write_str("the native build identity is empty"),
Self::Conflicting { installed, offered } => write!(
formatter,
"the native build identity {installed} is already installed; refusing {offered}"
),
}
}
}

impl std::error::Error for BuildIdentityError {}
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
//! BLAKE3 over length-prefixed fields keyed on a cache version and this build, so no two field lists share a key.

use crate::build_identity::main::native_build_identity::native_build_identity;

/// The hex digest of `version`, the native build, `kind` and each field in order.
#[must_use]
pub fn build_field_digest(version: &str, kind: &str, fields: &[&str]) -> String {
let mut digest = blake3::Hasher::new();
for field in [version, native_build_identity(), kind]
.iter()
.chain(fields.iter())
{
digest.update(&(field.len() as u64).to_le_bytes());
digest.update(field.as_bytes());
}
digest.finalize().to_hex().to_string()
}
Loading
Loading