Skip to content

Commit 2909eda

Browse files
authored
feat: publish Python nodes for jobs, roles and server runs (#1472)
* test: reproduce Python selectors in published job selections * wip: resolve published Python selectors with the CLI's unified resolver Work in progress, not ready for review. - Publish-time job, sensor-condition and role selections resolve with the same unified SQL/Python resolver `sqb build --select` uses. - Python snapshot nodes publish deps (Python and SQL) and tags; an intermediate loader publishes the source node it writes, and loader chains list it in `loads`. - Server-side ad-hoc selection rebuilds the Python selection graph from the snapshot and resolves with the same resolver. - The worker selects Python nodes by typed key. Validated so far: test_orchestration_snapshot.py passes (99 tests, including the Python selector reproduction). The Rust unit tests for the ad-hoc planner and planner scope have not finished yet; no review and no full gate. * feat: let a selected Python node read an unselected upstream's result A Python node can now be selected without its Python upstream in every path that uses the shared resolver: sqb build --select, published jobs, sensors, roles and ad-hoc runs. It reads the upstream's latest persisted result, and fails at run time with an error naming the upstream and the command to build it when the upstream has never succeeded. An intermediate loader without a write_strategy that returns rows now writes them to __loader__<name> as a table, as the loaders docs describe. A self-managed intermediate that returns nothing keeps its behaviour. The published snapshot rebuilds the Python selection graph from node deps, and job groups order jobs across Python and Python-to-SQL edges. Validated locally: python_selection and adhoc planner Rust tests, clippy on both crates, fensu check, the worker claim tests, orchestration snapshot, Python node selection E2E, load E2E (sqb load and sqb build on the docs example) and the python_nodes and executor/load unit tests. Not yet run: make check-ci, the full suite, review. * feat: select Python-only jobs and SQL tests across CLI and server runs - `job:` and bare job names resolve jobs through the Python-aware resolver publication uses, so a job that selects only Python nodes builds them. - Server ad-hoc runs accept `test:` selectors; a selected SQL test reports as a run node rather than as an attached check. - A provider function whose context is positional-only (`ctx, /`) receives it by position instead of failing at run time. - A role must grant the intermediate loader a job loads to run that job. * fix: keep Python selections runnable on workers and require checked nodes Selections now publish a managed source instead of its loader, ad-hoc test, audit, seed and freshness runs resolve SQL only, a selected SQL test reports its own outcome, and a selected model folds in its failing test. A selector that matches a Python check must also select the nodes the check depends on, in jobs, roles and the CLI alike. * fix: refuse selecting a Python check on a source no loader writes A build never loads such a source, so a matched check on it would be accepted and then never run. The resolver now names the check and the source and points to sqb check. * refactor: share upstream result reads between loader and node contexts Loader and Python node contexts read upstream results through one mixin, and the selection entry points build their context in one place. Regenerates the packaged docs skill for the selector and check pages.
1 parent 4502c30 commit 2909eda

77 files changed

Lines changed: 2152 additions & 361 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
"""The loader-dependencies example from the loaders docs, as written.
2+
3+
raw_accounts is an intermediate loader without a write_strategy that returns rows, so it writes
4+
them to __loader__raw_accounts as a table.
5+
"""
6+
7+
from sqlbuild.executor.load.models import LoaderContext
8+
from sqlbuild.loaders import loader
9+
10+
11+
@loader
12+
def raw_accounts(ctx: LoaderContext) -> list[dict[str, object]]:
13+
"""Load customer accounts."""
14+
return [
15+
{"account_id": 1, "account_name": "Pawnee Parks"},
16+
{"account_id": 2, "account_name": "Eagleton"},
17+
]
18+
19+
20+
@loader(depends_on=[raw_accounts])
21+
def raw_account_metrics(ctx: LoaderContext) -> list[dict[str, object]]:
22+
"""Account metrics."""
23+
accounts = ctx.loader(raw_accounts)
24+
rows = ctx.query(f"SELECT account_id FROM {accounts.destination}")
25+
return [{"account_id": row[0], "metric": "active"} for row in rows.fetchall()]
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
sources:
2+
- name: raw_account_metrics
3+
description: Account metrics read from the intermediate raw_accounts loader.
4+
managed: true
5+
write_strategy: table
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Roles over a loader chain whose middle loader no source declares: running the job needs a grant
2+
# on the intermediate loader's node as well as the model and the source it feeds.
3+
[roles.shipments_runner]
4+
description = "Run the shipments build, including its staging loader"
5+
6+
[[roles.shipments_runner.grants]]
7+
nodes = ["orders", "source:raw_shipments", "loader:stage_shipments"]
8+
permissions = ["job.run"]
9+
10+
[roles.orders_runner]
11+
description = "Run the orders build without the staging loader"
12+
13+
[[roles.orders_runner.grants]]
14+
nodes = ["orders", "source:raw_shipments"]
15+
permissions = ["job.run"]

‎benchmarks/duckshop/edge_cases/orchestration/intermediate_loader_sources.py‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,12 @@
1-
"""A source loader that depends on an intermediate loader no source declares."""
1+
"""A source loader that depends on an intermediate loader no source declares.
2+
3+
A job that selects the model must claim the intermediate loader's relation as well as the source.
4+
"""
25

36
from sqlbuild.loaders import loader
47

58

6-
@loader
9+
@loader(write_strategy="table")
710
def stage_shipments(ctx):
811
"""Stage synthetic shipments in the intermediate loader's own relation."""
912
return [{"order_id": 9}]
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
"""Jobs whose selections are Python nodes, which `--select job:<name>` must select as published."""
2+
3+
from sqlbuild.orchestration import job
4+
5+
customer_counts = job(
6+
"customer_counts",
7+
description="Count customers.",
8+
select="task:count_customers asset:customers_extract",
9+
)
10+
order_counts = job("order_counts", description="Count orders.", select="+task:count_orders")
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
"""A task whose Python upstream another job builds, so a job group must order the two jobs."""
2+
3+
from python.orders.partner_export.node import partner_orders_file
4+
5+
from sqlbuild.tasks import task
6+
7+
8+
@task(depends_on=partner_orders_file, tags=["orders"], description="Book collection.")
9+
def request_carrier_collection(ctx):
10+
return None
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
"""A Python check on the orders model that a job's tag selector matches.
2+
3+
A job matching the check and the model it checks publishes and runs the check with the model. A job
4+
matching the check alone fails publication, naming the check and the model it leaves out.
5+
"""
6+
7+
from sqlbuild.checks import check
8+
from sqlbuild.refs import model
9+
10+
11+
@check(depends_on=model("orders"), tags=["order_checks"])
12+
def orders_present(ctx):
13+
"""Orders are present."""
14+
orders = ctx.relation(model("orders"))
15+
count = ctx.query(f"SELECT count(*) FROM {orders}").fetchone()[0]
16+
return ctx.pass_() if count else ctx.fail(message="no orders")
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
"""An asset one job builds that a task in another job of the same group depends on."""
2+
3+
from sqlbuild.assets import asset
4+
from sqlbuild.refs import model
5+
6+
7+
@asset(depends_on=model("orders"), tags=["orders"], description="Partner order file.")
8+
def partner_orders_file(ctx):
9+
return None
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
"""A job that reaches a managed source through `+`, so its run selects the source's loader chain.
2+
3+
`+orders` selects the model, its managed source raw_shipments, the raw_shipments loader and the
4+
intermediate stage_shipments loader it depends on. The run records each loader as the source it
5+
writes, which is the node its load reports under.
6+
"""
7+
8+
from sqlbuild.orchestration import job, schedule
9+
10+
annual_orders = job(
11+
"annual_orders",
12+
description="Build orders and everything upstream of it, loaders included.",
13+
select="+orders",
14+
schedule=schedule(cron="0 0 * * *", timezone="UTC", starts_at="2000-01-01T00:00:00Z"),
15+
)

‎crates/sqlbuild-analysis/src/graph/_helpers/jobs.rs‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,17 +8,23 @@ use crate::graph::errors::SelectorError;
88
use crate::graph::models::{JobSelection, ProjectGraph};
99

1010
/// Resolve every job on the graph without jobs attached, as publication does, then attach them.
11-
pub(crate) fn attach(
11+
pub(crate) fn attach(graph: ProjectGraph, catalog: BTreeMap<String, JobSelection>) -> ProjectGraph {
12+
attach_with(graph, catalog, |graph, select, exclude| {
13+
resolve(graph, select, exclude).map(|keys| keys.into_iter().collect())
14+
})
15+
}
16+
17+
/// [`attach`] with each job's selectors resolved by `resolve_job` on the graph without jobs.
18+
pub(crate) fn attach_with(
1219
mut graph: ProjectGraph,
1320
catalog: BTreeMap<String, JobSelection>,
21+
resolve_job: impl Fn(&ProjectGraph, &[String], &[String]) -> Result<Keys, SelectorError>,
1422
) -> ProjectGraph {
1523
graph.jobs = None;
1624
let mut resolved: BTreeMap<String, Result<Keys, SelectorError>> = BTreeMap::new();
1725
for (name, selection) in &catalog {
1826
let keys: Result<Keys, SelectorError> = match selection {
19-
JobSelection::Job { select, exclude } => {
20-
resolve(&graph, select, exclude).map(|keys| keys.into_iter().collect())
21-
}
27+
JobSelection::Job { select, exclude } => resolve_job(&graph, select, exclude),
2228
JobSelection::Resolved { keys } => Ok(keys.iter().cloned().collect()),
2329
JobSelection::Group { .. } => continue,
2430
};

0 commit comments

Comments
 (0)