Skip to content

Commit 7948c79

Browse files
committed
feat: add a named-selector limit Rule for orchestration jobs and sensors
SQBRORCHESTRATION101 reports a job select, or the select of a node_event, source_new_data or source_stale sensor condition, that names more than max_named_selectors nodes. Sensor conditions are followed through module-level names and imports from project modules; what cannot be resolved is reported and can be suppressed with an exact exception.
1 parent 0592ea7 commit 7948c79

70 files changed

Lines changed: 1903 additions & 85 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
MODEL (description "Synthetic orders edge case int_orders");
2+
SELECT 1 AS order_id
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
MODEL (description "Synthetic orders edge case order_totals");
2+
SELECT 1 AS order_id
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
MODEL (description "Synthetic orders edge case stg_orders");
2+
SELECT 1 AS order_id
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
MODEL (description "Synthetic orders edge case stg_payments");
2+
SELECT 1 AS order_id
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# A folder, an expansion, a glob, an intersection's name, a job name and a repeated name: one named node.
2+
name = "orders_folder_job"
3+
description = "Selects the orders pipeline without listing it."
4+
select = "models/orders +order_totals stg_* order_totals,tag:orders orders_listed_job order_totals"
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Four nodes listed one by one: over the default limit of three.
2+
name = "orders_listed_job"
3+
description = "Lists every orders model by name."
4+
select = "stg_orders stg_payments int_orders order_totals"
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# A deliberate list, kept by an exact rule exception in the test's [rules] table.
2+
name = "orders_repair_job"
3+
description = "Rebuilds the four orders models after a late payments load."
4+
select = "stg_orders stg_payments int_orders order_totals"
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
"""Conditions the orders sensors import: one over the limit, a platform event and a builder."""
2+
3+
from sqlbuild.orchestration import PlatformEventKind, node_event, platform_event
4+
5+
ORDERS_LANDED = node_event(
6+
select="stg_orders stg_payments int_orders order_totals", kinds=["success"]
7+
)
8+
TICK_OVERDUE = platform_event(kinds=[PlatformEventKind.TICK_OVERDUE])
9+
10+
11+
def landed_after(model: str) -> object:
12+
"""A success condition on one model, built when the sensor module loads."""
13+
14+
return node_event(select=model, kinds=["success"])
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
"""Refreshes totals after one model: a condition a function builds, which cannot be counted."""
2+
3+
from orchestration.sensors._orders_conditions import landed_after
4+
5+
from sqlbuild.orchestration import sensor
6+
7+
on_orders_built = sensor(
8+
"on_orders_built_sensor",
9+
description="Refresh order totals after the totals model",
10+
when=landed_after("order_totals"),
11+
run="orders_folder_job",
12+
)
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
"""A condition whose select is not a string literal cannot be counted and is reported."""
2+
3+
from sqlbuild.orchestration import node_event, sensor
4+
5+
ORDERS_SELECTION = "stg_orders"
6+
7+
on_orders_computed = sensor(
8+
"on_orders_computed_sensor",
9+
description="Refresh totals when staged orders change",
10+
when=node_event(select=ORDERS_SELECTION, kinds=["success"]),
11+
run="orders_folder_job",
12+
)

0 commit comments

Comments
 (0)