Repository navigation
Commit f5bab28
authored
feat: add SQBRPYTHON orchestration, test and purity rules, Python tests and secret-free commands (#1497)
* feat: add SQBRPYTHON rules for orchestration sensors and calendars
- Compile: a sensor or calendar declared inside a `_`-prefixed folder under
orchestration/sensors/ or orchestration/calendars/ now fails with an error
naming the file, since such folders are private support.
- SQBRPYTHON016-020: module paths, private support, module surface, import
direction and no docstring on sensor and calendar functions.
- Provider and orchestration rules share parsing and import resolution.
- Evict nested namespace packages before their parents between projects in one
interpreter, so a project importing orchestration.* no longer breaks the
next compile.
- Docs and DuckShop fixtures for each rule.
* feat: run commands without provider secrets they don't need
A provider whose required secret settings are unset no longer stops
every command. Static commands (compile, dag, format, scope, lineage,
rules run) still fail with D012 naming the variable, unless
--allow-missing-secrets, SQLBUILD_ALLOW_MISSING_SECRETS or the local
allow_missing_secrets setting allows it. The project file rejects that
setting.
build, load and check read secrets only for the providers the selected
nodes and sinks use, and fail before anything runs when one is missing.
Providers are instantiated lazily in the runtime session.
sqb debug reports how many providers are ready and lists missing
secrets (--verbose lists every one). Location health reports
missing_secrets and exits 1, and the Providers page marks unset
secret settings.
* feat: run project Python tests with sqb test and check them with SQBRPYTHON rules
sqb test now runs the project's pytest modules under tests/python/ and
tests/providers/ after its SQL tests, from the project directory with
--import-mode=importlib. --sql and --python pick one kind. Selecting a
Python node runs the tests in its mirrored folder; selecting only models
or unit tests runs none. The command fails when either kind fails, and
fails early when Python tests exist but pytest is not installed. sqb test
--since, --inspect, review and ad-hoc runs stay SQL-only, and sqb build
never runs Python tests. With --json, pytest writes to stderr.
- SQBRPYTHON021: a test sits in the folder mirroring the code it tests,
and test_<name>.py has a <name>.py or <name>/ beside that code.
- SQBRPYTHON701-724: the Fensu test-shape rules (table-driven dataclass
cases, naming, parametrize ids), applied only to Python tests.
- Docs for sqb test, sqb build, testing, project layout, custom rule
tests and the Python Rules reference, plus DuckShop fixtures.
* feat: add SQBRPYTHON022 compile-time purity rule
sqb compile imports project Python with no warehouse connection, so the
code that runs then must not reach outside the machine. SQBRPYTHON022
checks calls at module scope, in class bodies and directly in @factory
bodies, follows the project functions they call through their imports,
and reports a call that reaches a subprocess, an HTTP request, a socket,
a database or warehouse driver, or a cloud or SSH client. It reads
source only and follows only calls it can resolve, so a call through a
variable or an object is never reported.
- approved_loader_boundaries lists exact project functions the walk does
not enter; a glob fails configuration and a module name fails the run.
- Findings are reported at the root call and name the external call and
where it is. Results per function are memoized, with a scaling test
over deep shared call chains.
- Docs for the rule, its option and its reference row, and a DuckShop
violation behind a branch compile never takes.
* test: list allow_missing_secrets in the unknown local key golden
* fix: send pytest output to stderr portably for sqb test --json1 parent 755b468 commit f5bab28
188 files changed
Lines changed: 6252 additions & 388 deletions
File tree
- benchmarks/duckshop/edge_cases
- orchestration
- private_folder_sensor
- jobs
- sensors/inventory/_helpers
- private_support_orders
- jobs
- sensors/inventory
- _helpers
- python_rules
- conforming
- orchestration
- jobs
- sensors/fulfillment
- _helpers
- tests/python/utils/currency
- violations
- orchestration
- jobs
- sensors
- fulfillment/_helpers
- inventory
- _helpers
- _lib
- python/inventory
- restock
- _helpers
- stock_count
- tests/python/inventory
- clients/typescript/src
- crates
- sqlbuild-cli
- src
- argv
- dispatch
- native_commands
- _helpers
- main
- tests/native_commands
- sqlbuild-config/src/project
- _helpers
- sqlbuild-core/src/debug_report
- main
- tests
- sqlbuild-discovery/src
- orchestration_files
- _helpers
- tests
- python_paths
- _helpers
- main
- tests
- sqlbuild-orchestration
- src
- declarations
- platform
- main
- tests/api
- sqlbuild-pysource/src/node_units
- _helpers
- main
- tests
- sqlbuild-python/src/bindings/_helpers
- build_command
- check
- cli
- command_runtime
- compile_command
- compile_driver
- contract_records
- debug_command
- discovery
- event_exporting
- load
- location_command
- native_host
- query_command
- records
- review_command
- sql_test_command
- sqlbuild-rules/src
- configuration/_helpers
- engine
- _helpers
- tests
- python_rules
- _helpers
- engine
- units
- main
- tests
- rules/main
- src/sqlbuild
- .agents/skills/sqlbuild/references/docs
- cli
- concepts
- python-nodes
- rules
- custom-rules
- compiler/discovery
- _helpers/filesystem
- main
- provider
- _helpers
- classes
- tests
- e2e/src/sqlbuild/cli/commands/main
- debug
- providers
- test
- goldens/compiler/failure
- integration/src/sqlbuild
- cli/commands/main
- compiler/compiled_project
- unit/src/sqlbuild/cli/commands/main/entry
- ui/src
- lib
- api
- mock
- types
- platform
- routes/providers/[name]
- website/src/content/docs/docs
- cli
- concepts
- python-nodes
- rules
- custom-rules
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
0 commit comments