Skip to content

Commit f5bab28

Browse files
authored
feat: add SQBRPYTHON orchestration, test and purity rules, Python tests and secret-free commands (#1497)
* feat: add SQBRPYTHON rules for orchestration sensors and calendars - Compile: a sensor or calendar declared inside a `_`-prefixed folder under orchestration/sensors/ or orchestration/calendars/ now fails with an error naming the file, since such folders are private support. - SQBRPYTHON016-020: module paths, private support, module surface, import direction and no docstring on sensor and calendar functions. - Provider and orchestration rules share parsing and import resolution. - Evict nested namespace packages before their parents between projects in one interpreter, so a project importing orchestration.* no longer breaks the next compile. - Docs and DuckShop fixtures for each rule. * feat: run commands without provider secrets they don't need A provider whose required secret settings are unset no longer stops every command. Static commands (compile, dag, format, scope, lineage, rules run) still fail with D012 naming the variable, unless --allow-missing-secrets, SQLBUILD_ALLOW_MISSING_SECRETS or the local allow_missing_secrets setting allows it. The project file rejects that setting. build, load and check read secrets only for the providers the selected nodes and sinks use, and fail before anything runs when one is missing. Providers are instantiated lazily in the runtime session. sqb debug reports how many providers are ready and lists missing secrets (--verbose lists every one). Location health reports missing_secrets and exits 1, and the Providers page marks unset secret settings. * feat: run project Python tests with sqb test and check them with SQBRPYTHON rules sqb test now runs the project's pytest modules under tests/python/ and tests/providers/ after its SQL tests, from the project directory with --import-mode=importlib. --sql and --python pick one kind. Selecting a Python node runs the tests in its mirrored folder; selecting only models or unit tests runs none. The command fails when either kind fails, and fails early when Python tests exist but pytest is not installed. sqb test --since, --inspect, review and ad-hoc runs stay SQL-only, and sqb build never runs Python tests. With --json, pytest writes to stderr. - SQBRPYTHON021: a test sits in the folder mirroring the code it tests, and test_<name>.py has a <name>.py or <name>/ beside that code. - SQBRPYTHON701-724: the Fensu test-shape rules (table-driven dataclass cases, naming, parametrize ids), applied only to Python tests. - Docs for sqb test, sqb build, testing, project layout, custom rule tests and the Python Rules reference, plus DuckShop fixtures. * feat: add SQBRPYTHON022 compile-time purity rule sqb compile imports project Python with no warehouse connection, so the code that runs then must not reach outside the machine. SQBRPYTHON022 checks calls at module scope, in class bodies and directly in @factory bodies, follows the project functions they call through their imports, and reports a call that reaches a subprocess, an HTTP request, a socket, a database or warehouse driver, or a cloud or SSH client. It reads source only and follows only calls it can resolve, so a call through a variable or an object is never reported. - approved_loader_boundaries lists exact project functions the walk does not enter; a glob fails configuration and a module name fails the run. - Findings are reported at the root call and name the external call and where it is. Results per function are memoized, with a scaling test over deep shared call chains. - Docs for the rule, its option and its reference row, and a DuckShop violation behind a branch compile never takes. * test: list allow_missing_secrets in the unknown local key golden * fix: send pytest output to stderr portably for sqb test --json
1 parent 755b468 commit f5bab28

188 files changed

Lines changed: 6252 additions & 388 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
name = "inventory_job"
2+
description = "Destination inventory."
3+
select = "stg_orders"
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
"""A sensor declared inside a private support folder fails compile instead of never running."""
2+
3+
from sqlbuild.orchestration import NodeEventKind, node_event, sensor
4+
5+
on_orders = sensor(
6+
"on_orders_sensor",
7+
description="Build inventory for every genuine orders rebuild.",
8+
when=node_event(select="orders orders*", kinds=[NodeEventKind.SUCCESS]),
9+
run="inventory_job",
10+
)
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
name = "inventory_job"
2+
description = "Destination inventory."
3+
select = "stg_orders"
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
"""Private support: modules in a _-prefixed folder declare nothing."""
2+
3+
ORDERS_SELECTION: str = "orders orders*"
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
"""A sensor in its domain folder reads its selection from a private support folder."""
2+
3+
from orchestration.sensors.inventory._helpers.selection import ORDERS_SELECTION
4+
5+
from sqlbuild.orchestration import NodeEventKind, node_event, sensor
6+
7+
on_orders = sensor(
8+
"on_orders_sensor",
9+
description="Build inventory for every genuine orders rebuild.",
10+
when=node_event(select=ORDERS_SELECTION, kinds=[NodeEventKind.SUCCESS]),
11+
run="inventory_job",
12+
)
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
name = "restock_shipments_job"
2+
description = "Rebuild shipments."
3+
select = "shipments"
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
"""Shipment backlog settings."""
2+
3+
BACKLOG_REASON: str = "No shipment backlog"
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
"""Skip reasons for the fulfillment sensors."""
2+
3+
4+
def prefixed(reason: str) -> str:
5+
"""Prefix a skip reason with its domain."""
6+
return f"fulfillment: {reason}"
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
"""Skip restocking while the shipment backlog is empty."""
2+
3+
from orchestration.sensors.fulfillment._constants import BACKLOG_REASON
4+
from orchestration.sensors.fulfillment._helpers.reasons import prefixed
5+
6+
from sqlbuild.orchestration import SensorContext, SkipReason, sensor
7+
8+
_REASON_SUFFIX: str = "."
9+
10+
11+
@sensor(
12+
"shipment_backlog_sensor",
13+
description="Rebuild shipments when the backlog grows.",
14+
run="restock_shipments_job",
15+
minimum_interval_seconds=60,
16+
)
17+
def shipment_backlog(ctx: SensorContext) -> SkipReason:
18+
return SkipReason(prefixed(BACKLOG_REASON) + _REASON_SUFFIX)
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
"""Test cases for carrier money formatting."""
2+
3+
from dataclasses import dataclass
4+
5+
6+
@dataclass(frozen=True)
7+
class CentsTextTestCase:
8+
"""One amount in cents and its carrier text."""
9+
10+
description: str
11+
cents: int
12+
expected_text: str

0 commit comments

Comments
 (0)