Skip to content

Commit 0fc54fa

Browse files
authored
fix!: keep Python node results in one location per target (#1551)
* fix: keep Python node results in one location per target Loaders, tasks, assets and checks now write their results and loader fingerprints to one location for every target: the target's schema, the target's loader_schema when the schema is preserve, or the adapter's default schema when the target schema is unset or resolved per model. Scheduled loaders and build-time checks used the adapter default before, so adapters without one, such as Snowflake, skipped those writes. Compile stops with S105 when a preserve target without a loader_schema has Python nodes. For one release, a node whose result is missing from the new location is read from the adapter's default location instead. * fix: read results left in a preserve schema and cover every runtime Under a preserve target, also read a node's result from the schema named preserve that earlier releases wrote to, after the adapter default. The location for `sqb load`, build-time checks and `sqb check` now comes from the target settings, so each runtime reads and writes the same place. * fix: resolve sqb load's result location with CLI vars and share loader result writes sqb load now takes its node-result location from the CLI-var-aware loader defaults, as build does, so a schema set with --vars is read from the same place both commands write. Build and ingress loaders write their results through one load operation. * test: add the preserved-schema refusal to the compiler failure corpus * test: give the preserved-target load cases a loader schema A preserve target with a loader now needs loader_schema (S105), so the managed-source load cases set one. The source still lands in its own schema. * fix!: require a loader schema for preserve targets with Python nodes BREAKING CHANGE: a preserve target with Python loaders, tasks, assets or checks now needs loader_schema, which is where their results are kept. Without it, compile fails with S105, which says to set targets.<name>.loader_schema.
1 parent f5d8664 commit 0fc54fa

49 files changed

Lines changed: 936 additions & 167 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,7 @@ E2E_DUCKDB_BUILD_CORE_PATHS := \
151151
tests/e2e/src/sqlbuild/cli/commands/main/build/test_defer_migration_origin.py \
152152
tests/e2e/src/sqlbuild/cli/commands/main/build/test_sql_hook_reads.py \
153153
tests/e2e/src/sqlbuild/cli/commands/main/build/test_python_node_identity.py \
154+
tests/e2e/src/sqlbuild/cli/commands/main/build/test_node_result_location.py \
154155
tests/e2e/src/sqlbuild/cli/commands/main/build/test_sql_hook_identity.py \
155156
tests/e2e/src/sqlbuild/cli/commands/main/build/test_query_change_tracking.py \
156157
tests/e2e/src/sqlbuild/cli/commands/main/build/test_query_fingerprint_changes.py \
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
"""Two loaders and a task that reads both of their results.
2+
3+
A build that selects only one loader still reads the other loader's latest result, which may come
4+
from an earlier build. On Snowflake the adapter has no default schema, so the results must go to
5+
the target's schema for the task to find them.
6+
"""
7+
8+
from sqlbuild.executor.load.models import LoaderContext
9+
from sqlbuild.loaders import loader
10+
from sqlbuild.refs import source
11+
from sqlbuild.tasks import task
12+
13+
14+
@loader
15+
def raw_order_events(ctx: LoaderContext) -> list[dict[str, object]]:
16+
"""Order events."""
17+
return [{"event_id": 1}]
18+
19+
20+
@loader
21+
def raw_customer_events(ctx: LoaderContext) -> list[dict[str, object]]:
22+
"""Customer events."""
23+
return [{"event_id": 1}]
24+
25+
26+
@task(depends_on=[source("raw_order_events"), source("raw_customer_events")])
27+
def announce_events(ctx):
28+
"""Record the run each loader result came from."""
29+
return ctx.result(
30+
payload={
31+
"orders": ctx.result_of(node_function=raw_order_events).run_id,
32+
"customers": ctx.result_of(node_function=raw_customer_events).run_id,
33+
}
34+
)
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
sources:
2+
- name: raw_order_events
3+
description: Order events, read by announce_events.
4+
managed: true
5+
write_strategy: table
6+
- name: raw_customer_events
7+
description: Customer events, read by announce_events.
8+
managed: true
9+
write_strategy: table

‎crates/sqlbuild-analysis/src/assembly/target_checks/constants.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,5 +10,7 @@ pub const LOADER_COLLISION_CODE: &str = "S102";
1010
pub const MANAGED_WRITE_SCHEMA_CODE: &str = "S103";
1111
/// A named target chooses no schema strategy.
1212
pub const NAMED_TARGET_SCHEMA_CODE: &str = "S104";
13+
/// A target that keeps each model's schema has no schema for Python node results.
14+
pub const PRESERVED_NODE_RESULTS_CODE: &str = "S105";
1315
pub(crate) const DATABASE_PART: &str = "database";
1416
pub(crate) const SCHEMA_PART: &str = "schema";

‎crates/sqlbuild-analysis/src/assembly/target_checks/errors.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,6 @@ pub enum TargetError {
1515
NamedTargetSchema(String),
1616
/// An intermediate loader, by name, whose schema resolves to the target's `preserve`.
1717
PreservedLoaderSchema(String),
18+
/// A target, by name, whose schema is `preserve` and that names no loader schema.
19+
PreservedNodeResults(String),
1820
}

‎crates/sqlbuild-analysis/src/assembly/target_checks/main/target_error_message.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,5 +48,10 @@ pub fn target_error_message(error: &TargetError) -> String {
4848
'{PRESERVE_TARGET_VALUE}'. The target's schema '{PRESERVE_TARGET_VALUE}' keeps each \
4949
model's own schema, and a loader has none, so its destination must name a schema."
5050
),
51+
TargetError::PreservedNodeResults(target) => format!(
52+
"Target '{target}' keeps each model's own schema, so it has no schema for the \
53+
results of loaders, tasks, assets and checks. Set targets.{target}.loader_schema; \
54+
Python node results are kept there."
55+
),
5156
}
5257
}
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
//! The adapter's own default database or schema.
2+
3+
use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyResult};
4+
5+
/// The adapter's `default_database` or `default_schema`, named by `method`.
6+
pub fn adapter_default_location<'py>(
7+
adapter: &Bound<'py, PyAny>,
8+
method: &str,
9+
) -> PyResult<Bound<'py, PyAny>> {
10+
adapter.call_method0(method)
11+
}

‎crates/sqlbuild-python-base/src/warehouse_access/main/mod.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
pub mod adapter_default_location;
12
pub mod ensure_event_id_column;
23
pub mod is_warehouse_access_refused;
34
pub mod read_catalog;

‎crates/sqlbuild-python-compile/src/compile_driver/_helpers/setup/mod.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
pub(crate) mod inputs;
2+
pub(crate) mod model_free_locations;
3+
pub(crate) mod node_results;
24
pub(crate) mod pipeline;
35
pub(crate) mod pipeline_entries;
46
pub(crate) mod project;
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
//! The target database and schema that Python nodes use outside any model.
2+
3+
use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyResult};
4+
use pyo3::types::PyString;
5+
use sqlbuild_analysis::assembly::target_checks::constants::PRESERVE_TARGET_VALUE;
6+
use sqlbuild_model_config::templates::main::holds_context_placeholder::holds_context_placeholder;
7+
use sqlbuild_python_base::warehouse_access::main::adapter_default_location::adapter_default_location;
8+
9+
/// Python nodes' `(database, schema)`: a `preserve` schema reads `loader_schema`; unusable values the adapter's.
10+
pub(crate) fn model_free_locations<'py>(
11+
(database, schema): (Bound<'py, PyAny>, Bound<'py, PyAny>),
12+
loader_schema: Option<&str>,
13+
adapter: &Bound<'py, PyAny>,
14+
) -> PyResult<(Bound<'py, PyAny>, Bound<'py, PyAny>)> {
15+
let kept_schema: Option<&str> =
16+
loader_schema.filter(|name| !name.is_empty() && *name != PRESERVE_TARGET_VALUE);
17+
let location = |value: Bound<'py, PyAny>, method: &str, preserved: Option<&str>| match (
18+
value.extract::<Option<String>>()?.as_deref(),
19+
preserved,
20+
) {
21+
(Some(PRESERVE_TARGET_VALUE), Some(name)) => Ok(PyString::new(value.py(), name).into_any()),
22+
(Some(text), _)
23+
if !text.is_empty()
24+
&& text != PRESERVE_TARGET_VALUE
25+
&& !holds_context_placeholder(text) =>
26+
{
27+
Ok(value)
28+
}
29+
_ => adapter_default_location(adapter, method),
30+
};
31+
Ok((
32+
location(database, "default_database", None)?,
33+
location(schema, "default_schema", kept_schema)?,
34+
))
35+
}

0 commit comments

Comments
 (0)