diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md index 468d2f250c..ade6e96003 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md @@ -8,90 +8,88 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in - [Introduction](index.md) (`index`) - The refactorable warehouse. Verify early, test properly, and refactor safely. - [Quickstart](quickstart.md) (`quickstart`) - Get a SQLBuild project running locally with DuckDB in under a minute. -- [Feature Comparison](feature-comparison.md) (`feature-comparison`) - Feature comparison between SQLBuild, dbt, and SQLMesh. -- [Roadmap](roadmap.md) (`roadmap`) - What's coming next in SQLBuild. -- [Benchmarks](benchmarks.md) (`benchmarks`) - SQLBuild and dbt v2 compiling the same 10,000-model project: cold, unchanged, one-model and 1% edits. - -## dbt Compatibility - +- [Set up a project](project-setup.md) (`project-setup`) - Connect a SQLBuild project to your warehouse with dev and prod targets and a personal local file. - [Using SQLBuild with dbt](concepts/dbt-compatibility/overview.md) (`concepts/dbt-compatibility/overview`) - Run dbt and SQLBuild side by side with coordinated selection and SQLBuild models downstream. - [Selection](concepts/dbt-compatibility/selection.md) (`concepts/dbt-compatibility/selection`) - How select and exclude flags route work across the dbt and SQLBuild graphs. - [Adding SQLBuild models](concepts/dbt-compatibility/adding-sqlbuild-models.md) (`concepts/dbt-compatibility/adding-sqlbuild-models`) - Grow into SQLBuild's own models, tests, audits, and scenarios downstream of your dbt project. +- [Feature Comparison](feature-comparison.md) (`feature-comparison`) - Feature comparison between SQLBuild, dbt, and SQLMesh. +- [Roadmap](roadmap.md) (`roadmap`) - What's coming next in SQLBuild. +- [Benchmarks](benchmarks.md) (`benchmarks`) - SQLBuild and dbt v2 compiling the same 10,000-model project: cold, unchanged, one-model and 1% edits. -## Concepts +## Build models -- [Project Configuration](concepts/project-configuration.md) (`concepts/project-configuration`) - Configure your SQLBuild project with sqlbuild_project.toml and sqlbuild_local.toml. -- [Resource Identities](concepts/resource-identities.md) (`concepts/resource-identities`) - Canonical names for SQLBuild resources, selectors, state, and integrations. -- [Semantic Compilation](concepts/semantic-compilation.md) (`concepts/semantic-compilation`) - Offline reference, semantic and type checks that sqb compile runs before any warehouse work. -- [Refactoring](concepts/refactoring.md) (`concepts/refactoring`) - Rename and move models and columns, keep their warehouse history, and keep old names working. -- [Model migrations](concepts/models/migrations.md) (`concepts/models/migrations`) - Keep a renamed model's history, and keep its old name working for a while. -- [Column migrations](concepts/models/column-migrations.md) (`concepts/models/column-migrations`) - Keep a column's history when you rename it in an incremental or snapshot model. -- [Overview](concepts/adapters.md) (`concepts/adapters`) - Supported database engines and their connection configuration. -- [Snowflake](concepts/adapters/snowflake.md) (`concepts/adapters/snowflake`) - Snowflake adapter configuration for SQLBuild. -- [DuckDB](concepts/adapters/duckdb.md) (`concepts/adapters/duckdb`) - DuckDB adapter configuration for SQLBuild. -- [MotherDuck](concepts/adapters/motherduck.md) (`concepts/adapters/motherduck`) - MotherDuck adapter configuration for SQLBuild. -- [PostgreSQL](concepts/adapters/postgres.md) (`concepts/adapters/postgres`) - PostgreSQL adapter configuration for SQLBuild. -- [SQLite](concepts/adapters/sqlite.md) (`concepts/adapters/sqlite`) - SQLite adapter configuration, capabilities and limits for SQLBuild. -- [BigQuery](concepts/adapters/bigquery.md) (`concepts/adapters/bigquery`) - BigQuery adapter configuration for SQLBuild. -- [Databricks](concepts/adapters/databricks.md) (`concepts/adapters/databricks`) - Databricks adapter configuration for SQLBuild. -- [SQL Server](concepts/adapters/sqlserver.md) (`concepts/adapters/sqlserver`) - Microsoft SQL Server adapter configuration for SQLBuild. -- [ClickHouse](concepts/adapters/clickhouse.md) (`concepts/adapters/clickhouse`) - ClickHouse adapter configuration for SQLBuild. -- [Sources](concepts/sources.md) (`concepts/sources`) - Declare external data inputs for your pipeline. -- [Seeds](concepts/seeds.md) (`concepts/seeds`) - Load static CSV data into your pipeline as tables. - [Overview](concepts/models.md) (`concepts/models`) - SQL model anatomy, references, dependencies, and the model documentation guide. - [Materializations](concepts/models/materializations.md) (`concepts/models/materializations`) - Choose how SQLBuild persists model output. -- [Schemas](concepts/models/schemas.md) (`concepts/models/schemas`) - Declare model columns inline or reuse canonical inherited schemas. -- [Type Enforcement](concepts/models/type-enforcement.md) (`concepts/models/type-enforcement`) - Understand declared model types, static checks, and runtime casting. -- [Contracts](concepts/models/contracts.md) (`concepts/models/contracts`) - Validate required or exact model output schemas. -- [Hooks](concepts/models/hooks.md) (`concepts/models/hooks`) - Run SQL or Python lifecycle hooks around model materialization. -- [SQL Hooks](concepts/models/hooks/sql.md) (`concepts/models/hooks/sql`) - Define, parameterize, compile, and invoke reusable or inline SQL lifecycle hooks. -- [Python Hooks](concepts/models/hooks/python.md) (`concepts/models/hooks/python`) - Define Python lifecycle hooks with runtime context, providers, SQL access, and skips. -- [Configuration](concepts/models/configuration.md) (`concepts/models/configuration`) - MODEL() header fields and SQL-analysis controls. -- [Enums](concepts/enums.md) (`concepts/enums`) - Define a fixed set of named string or integer values and use them safely in SQL. -- [Constants](concepts/constants.md) (`concepts/constants`) - Define reusable compiler-validated values and reference them safely from SQL. -- [Collections and Rendering](concepts/constants/collections-and-rendering.md) (`concepts/constants/collections-and-rendering`) - Define list, set, and object constants and control their adapter-specific SQL rendering. -- [Enum Model Contracts](concepts/enums/model-contracts.md) (`concepts/enums/model-contracts`) - Use an enum as a portable model-column domain with generated accepted-value validation. -- [Model-Private Values](concepts/model-private-values.md) (`concepts/model-private-values`) - Keep enums and constants inside one model when no other resource should use them. +- [Sources](concepts/sources.md) (`concepts/sources`) - Declare external data inputs for your pipeline. +- [Seeds](concepts/seeds.md) (`concepts/seeds`) - Load static CSV data into your pipeline as tables. +- [Incremental Models](concepts/incremental.md) (`concepts/incremental`) - Cursor-based incremental strategies, microbatch execution, and backfill policies. +- [Snapshots (SCD Type 2)](concepts/snapshots.md) (`concepts/snapshots`) - Preserve row history over time using SCD Type 2 semantics with timestamp or check-based change detection. +- [Functions](concepts/functions.md) (`concepts/functions`) - User-defined functions and table functions managed as part of your project. - [Macros](concepts/macros.md) (`concepts/macros`) - Reuse SQL with typed SQL macros, or generate it with Python macros when it needs logic. - [SQL Macros](concepts/macros/sql-macros.md) (`concepts/macros/sql-macros`) - Write typed, logic-free SQL fragments that the compiler checks and sees inside. +- [SQL macro reference](concepts/macros/sql-macro-reference.md) (`concepts/macros/sql-macro-reference`) - Literal positions, relation slot reads and mappings, call checks and result types for SQL macros. - [Python Macros](concepts/macros/python-macros.md) (`concepts/macros/python-macros`) - Write Python functions that generate SQL at compile time when the SQL needs logic. - [Python Macro Composition and Context](concepts/macros/composition-and-context.md) (`concepts/macros/composition-and-context`) - Compose Python macros through imports, and use adapter, target, constants and enums. -- [Interpolation](concepts/interpolation.md) (`concepts/interpolation`) - How SQLBuild processes variables, context, and dynamic content in SQL and config. -- [Functions](concepts/functions.md) (`concepts/functions`) - User-defined functions and table functions managed as part of your project. -- [Incremental Models](concepts/incremental.md) (`concepts/incremental`) - Cursor-based incremental strategies, microbatch execution, and backfill policies. -- [Planning and change detection](concepts/planning.md) (`concepts/planning`) - How SQLBuild explains build work, detects changes, and chooses safe model actions. -- [Replay decisions](concepts/planning/replay-decisions.md) (`concepts/planning/replay-decisions`) - How much history each model reprocesses: decided by its own change and its own replay_on_change, never inherited from upstream. -- [Source freshness](concepts/planning/source-freshness.md) (`concepts/planning/source-freshness`) - Observe external source changes and propagate them through planning. -- [Selection and staleness](concepts/planning/selection-and-staleness.md) (`concepts/planning/selection-and-staleness`) - How selection interacts with change detection, and the stale warnings that stop silent partial rebuilds. -- [Snapshots (SCD Type 2)](concepts/snapshots.md) (`concepts/snapshots`) - Preserve row history over time using SCD Type 2 semantics with timestamp or check-based change detection. +- [Selectors](concepts/selectors.md) (`concepts/selectors`) - Target specific models, paths, tags, or DAG subsets with select and exclude flags. + +## Testing + +- [Unit tests](concepts/testing.md) (`concepts/testing`) - Check a model's logic on rows you write, by mocking its inputs and comparing its output. +- [Scenarios](concepts/scenarios.md) (`concepts/scenarios`) - End-to-end tests that build real project graphs against coherent fixture data. - [Audits](concepts/audits.md) (`concepts/audits`) - Violation and measurement checks that gate data and record quality outcomes. +- [Data Diffs](concepts/diff.md) (`concepts/diff`) - Compare schemas and data between targets to validate changes before they reach production. + +## Correctness + +- [What sqb compile checks](concepts/semantic-compilation.md) (`concepts/semantic-compilation`) - Offline reference, semantic and type checks that sqb compile runs before any warehouse work. +- [Contracts](concepts/models/contracts.md) (`concepts/models/contracts`) - Say which columns a model promises to output, and when SQLBuild checks them. +- [Type Enforcement](concepts/models/type-enforcement.md) (`concepts/models/type-enforcement`) - Understand declared model types, static checks, and runtime casting. +- [Reusable column schemas](concepts/models/schemas.md) (`concepts/models/schemas`) - Declare a model's columns in its header, or once in a SCHEMA() that many models share. - [Overview](concepts/rules.md) (`concepts/rules`) - Turn repeated SQL and project review decisions into compiler-enforced requirements. - [Configuration and selection](concepts/rules/configuration-and-selection.md) (`concepts/rules/configuration-and-selection`) - Select built-in and custom Rules by exact code or family prefix. - [Findings and exceptions](concepts/rules/findings-and-exceptions.md) (`concepts/rules/findings-and-exceptions`) - Understand Rule diagnostics and record intentional exceptions safely. -- [Published-layer Rules](concepts/rules/published-layer-rules.md) (`concepts/rules/published-layer-rules`) - Keep each domain's published layer free of consumer-specific variants. - [Execution and caching](concepts/rules/execution-and-caching.md) (`concepts/rules/execution-and-caching`) - Understand Rule enforcement order, focused runs, and dependency-aware reuse. -- [Testing](concepts/testing.md) (`concepts/testing`) - SQL unit tests and multi-model tests with macro support, assertions, and model chaining. -- [Scenarios](concepts/scenarios.md) (`concepts/scenarios`) - End-to-end tests that build real project graphs against coherent fixture data. -- [Selectors](concepts/selectors.md) (`concepts/selectors`) - Target specific models, paths, tags, or DAG subsets with select and exclude flags. + +## Ship + +- [Planning and change detection](concepts/planning.md) (`concepts/planning`) - How SQLBuild explains build work, detects changes, and chooses safe model actions. +- [Replay decisions](concepts/planning/replay-decisions.md) (`concepts/planning/replay-decisions`) - How much history each model reprocesses: decided by its own change and its own replay_on_change, never inherited from upstream. +- [Source freshness](concepts/planning/source-freshness.md) (`concepts/planning/source-freshness`) - Observe external source changes and propagate them through planning. +- [Selection and staleness](concepts/planning/selection-and-staleness.md) (`concepts/planning/selection-and-staleness`) - How selection interacts with change detection, and the stale warnings that stop silent partial rebuilds. - [Column Lineage](concepts/column-lineage.md) (`concepts/column-lineage`) - Trace individual columns through your SQL pipeline - understand where data comes from and where it goes. -- [Data Diffs](concepts/diff.md) (`concepts/diff`) - Compare schemas and data between targets to validate changes before they reach production. -## Advanced Concepts +## Refactor + +- [Refactoring](concepts/refactoring.md) (`concepts/refactoring`) - Rename and move models and columns, keep their warehouse history, and keep old names working. +- [Model migrations](concepts/models/migrations.md) (`concepts/models/migrations`) - Keep a renamed model's history, and keep its old name working for a while. +- [Column migrations](concepts/models/column-migrations.md) (`concepts/models/column-migrations`) - Keep a column's history when you rename it in an incremental or snapshot model. + +## Investigate -- [Execution Observability](concepts/observability.md) (`concepts/observability`) - Choose authoritative lifecycle facts, readable logs, or command-output records. - [Recall](concepts/observability/recall.md) (`concepts/observability/recall`) - The local history of every command: what ran, what it executed, and what it printed. +- [Sharing Recall in ClickHouse](concepts/observability/recall-clickhouse.md) (`concepts/observability/recall-clickhouse`) - Keep one Recall history for a team or CI in ClickHouse: users, grants, setup, retention and reading. +- [Observability](concepts/observability.md) (`concepts/observability`) - Find out what a run did, or send its events to your own systems. - [Typed Sinks](concepts/observability/sinks.md) (`concepts/observability/sinks`) - Export lifecycle facts and command output through project-owned providers. -- [Execution history](concepts/observability/execution-history.md) (`concepts/observability/execution-history`) - Store and query lifecycle events and run history from Python, with SQLite locally or PostgreSQL when deployed. + +## Advanced + - [Overview](concepts/declaration-scopes.md) (`concepts/declaration-scopes`) - Limit enums, constants, macros, audits, schemas, and named hooks to the parts of a project that use them. - [How Visibility Works](concepts/declaration-scopes/visibility.md) (`concepts/declaration-scopes/visibility`) - See which enums, constants, and macros are available to each SQL file. - [Where to Put Declarations](concepts/declaration-scopes/placement.md) (`concepts/declaration-scopes/placement`) - Choose the narrowest folder that contains every real use. - [Scope Explorer](concepts/declaration-scopes/explorer.md) (`concepts/declaration-scopes/explorer`) - Inspect visibility, explain resolution, browse declarations, and preview moves offline. +- [Enums](concepts/enums.md) (`concepts/enums`) - Define a fixed set of named string or integer values and use them safely in SQL. +- [Constants](concepts/constants.md) (`concepts/constants`) - Define reusable compiler-validated values and reference them safely from SQL. +- [Collections and Rendering](concepts/constants/collections-and-rendering.md) (`concepts/constants/collections-and-rendering`) - Define list, set, and object constants and control their adapter-specific SQL rendering. +- [Enum Model Contracts](concepts/enums/model-contracts.md) (`concepts/enums/model-contracts`) - Use an enum as a portable model-column domain with generated accepted-value validation. +- [Model-Private Values](concepts/model-private-values.md) (`concepts/model-private-values`) - Keep enums and constants inside one model when no other resource should use them. +- [Variables and placeholders](concepts/interpolation.md) (`concepts/interpolation`) - How SQLBuild processes variables, context, and dynamic content in SQL and config. +- [Hooks](concepts/models/hooks.md) (`concepts/models/hooks`) - Run SQL or Python lifecycle hooks around model materialization. +- [SQL Hooks](concepts/models/hooks/sql.md) (`concepts/models/hooks/sql`) - Define, parameterize, compile, and invoke reusable or inline SQL lifecycle hooks. +- [Python Hooks](concepts/models/hooks/python.md) (`concepts/models/hooks/python`) - Define Python lifecycle hooks with runtime context, providers, SQL access, and skips. - [Overview](concepts/rules/custom-rules/overview.md) (`concepts/rules/custom-rules/overview`) - Define repository-owned checks with one typed Python API. - [RuleContext and compiler facts](concepts/rules/custom-rules/rule-context.md) (`concepts/rules/custom-rules/rule-context`) - Inspect stable compiler-owned facts and opt into lazy SQL AST access. - [Testing and deterministic Rules](concepts/rules/custom-rules/testing-and-determinism.md) (`concepts/rules/custom-rules/testing-and-determinism`) - Test custom findings and keep implementations safe for dependency-aware caching. - -## Python Nodes - +- [Published-layer Rules](concepts/rules/published-layer-rules.md) (`concepts/rules/published-layer-rules`) - Keep each domain's published layer free of consumer-specific variants. - [Overview](concepts/python-nodes/overview.md) (`concepts/python-nodes/overview`) - Tasks, assets, loaders, and checks as first-class nodes in the SQLBuild graph. - [Loaders](concepts/python-nodes/loaders.md) (`concepts/python-nodes/loaders`) - Load external data into source tables with Python functions. - [Tasks](concepts/python-nodes/tasks.md) (`concepts/python-nodes/tasks`) - Run Python computation and side effects as nodes in the SQLBuild graph. @@ -100,6 +98,21 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in - [Factories](concepts/python-nodes/factories.md) (`concepts/python-nodes/factories`) - Generate Python nodes programmatically with @factory. - [Providers](concepts/python-nodes/providers.md) (`concepts/python-nodes/providers`) - Shared runtime services for Python nodes and hooks. - [SQL References](concepts/python-nodes/sql-references.md) (`concepts/python-nodes/sql-references`) - Read SQL models, sources, and seeds from Python nodes without creating SQL dependencies. +- [Custom adapters](concepts/adapters/custom.md) (`concepts/adapters/custom`) - Extend a built-in adapter or write one for another database engine in Python. +- [Execution history](concepts/observability/execution-history.md) (`concepts/observability/execution-history`) - Store and query lifecycle events and run history from Python, with SQLite locally or PostgreSQL when deployed. + +## Adapters + +- [Overview](concepts/adapters.md) (`concepts/adapters`) - Supported database engines and their connection configuration. +- [Snowflake](concepts/adapters/snowflake.md) (`concepts/adapters/snowflake`) - Snowflake adapter configuration for SQLBuild. +- [DuckDB](concepts/adapters/duckdb.md) (`concepts/adapters/duckdb`) - DuckDB adapter configuration for SQLBuild. +- [MotherDuck](concepts/adapters/motherduck.md) (`concepts/adapters/motherduck`) - MotherDuck adapter configuration for SQLBuild. +- [PostgreSQL](concepts/adapters/postgres.md) (`concepts/adapters/postgres`) - PostgreSQL adapter configuration for SQLBuild. +- [SQLite](concepts/adapters/sqlite.md) (`concepts/adapters/sqlite`) - SQLite adapter configuration, capabilities and limits for SQLBuild. +- [BigQuery](concepts/adapters/bigquery.md) (`concepts/adapters/bigquery`) - BigQuery adapter configuration for SQLBuild. +- [Databricks](concepts/adapters/databricks.md) (`concepts/adapters/databricks`) - Databricks adapter configuration for SQLBuild. +- [SQL Server](concepts/adapters/sqlserver.md) (`concepts/adapters/sqlserver`) - Microsoft SQL Server adapter configuration for SQLBuild. +- [ClickHouse](concepts/adapters/clickhouse.md) (`concepts/adapters/clickhouse`) - ClickHouse adapter configuration for SQLBuild. ## Integrations @@ -109,40 +122,43 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in - [dlt](integrations/dlt.md) (`integrations/dlt`) - Declarative dlt sources in YAML, or full dlt pipelines inside Python source loaders. - [ingestr](integrations/ingestr.md) (`integrations/ingestr`) - Declarative data ingestion from 50+ sources using ingestr. +## Reference + +- [Configuration reference](concepts/project-configuration.md) (`concepts/project-configuration`) - Every key in sqlbuild_project.toml and sqlbuild_local.toml. +- [Configuration](concepts/models/configuration.md) (`concepts/models/configuration`) - MODEL() header fields and SQL-analysis controls. +- [Testing reference](concepts/testing-reference.md) (`concepts/testing-reference`) - Test modes, parameterized cases, cursor windows, cost guards and time limits for SQL unit tests. +- [Naming rules](concepts/resource-identities.md) (`concepts/resource-identities`) - Names of models, tests, macros and other SQLBuild resources are lowercase snake_case. + ## CLI Reference - [sqb init](cli/init.md) (`cli/init`) - Scaffold a new SQLBuild project. - [sqb playground](cli/playground.md) (`cli/playground`) - Create a self-contained SQLBuild project to explore locally. - [sqb skills](cli/skills.md) (`cli/skills`) - Install SQLBuild skill files for AI coding agents. -- [sqb server](cli/server.md) (`cli/server`) - Run the control service and scheduler. -- [sqb login and logout](cli/login.md) (`cli/login`) - Authorize a bounded CLI session through your server's browser sign-in. +- [sqb debug](cli/debug.md) (`cli/debug`) - Validate project configuration and test the warehouse connection. - [sqb compile](cli/compile.md) (`cli/compile`) - Compile models into resolved SQL, validate contracts, and write target artifacts - fully offline. - [sqb format](cli/format.md) (`cli/format`) - Apply canonical SQLBuild SQL formatting. -- [sqb contract](cli/contract.md) (`cli/contract`) - Compare or generate repository contracts from physical warehouse schemas. -- [scope](cli/scope.md) (`cli/scope`) - Inspect declaration visibility, usage, placement, and move impact offline. -- [rename and mv](cli/rename.md) (`cli/rename`) - Rename or move a model, or rename a column, and update every reference in one verified step. +- [sqb test](cli/test.md) (`cli/test`) - Run SQL unit tests and multi-model tests in isolation. +- [sqb scenario](cli/scenario.md) (`cli/scenario`) - Run end-to-end scenario tests against the warehouse or locally with DuckDB. +- [sqb lineage](cli/lineage.md) (`cli/lineage`) - Explore model and column-level dependency graphs from the command line. +- [sqb scope](cli/scope.md) (`cli/scope`) - Inspect declaration visibility, usage, placement, and move impact offline. +- [sqb rename and sqb mv](cli/rename.md) (`cli/rename`) - Rename or move a model, or rename a column, and update every reference in one verified step. - [sqb rules](cli/rules.md) (`cli/rules`) - List, inspect, or run compiler-integrated Rules and generate project guidance. +- [sqb contract](cli/contract.md) (`cli/contract`) - Compare or generate repository contracts from physical warehouse schemas. - [sqb plan](cli/plan.md) (`cli/plan`) - Preview what SQLBuild will do before executing. - [sqb build](cli/build.md) (`cli/build`) - Compile, plan, and execute the selected build lifecycle. - [sqb load](cli/load.md) (`cli/load`) - Load managed sources into the warehouse. - [sqb seed](cli/seed.md) (`cli/seed`) - Load seed CSV files into the warehouse. -- [sqb test](cli/test.md) (`cli/test`) - Run SQL unit tests and multi-model tests in isolation. -- [sqb scenario](cli/scenario.md) (`cli/scenario`) - Run end-to-end scenario tests against the warehouse or locally with DuckDB. - [sqb audit](cli/audit.md) (`cli/audit`) - Run data quality audits in isolation. -- [sqb freshness](cli/freshness.md) (`cli/freshness`) - Observe source freshness without writing state. - [sqb check](cli/check.md) (`cli/check`) - Run Python checks against tasks, assets, and loaders. +- [sqb freshness](cli/freshness.md) (`cli/freshness`) - Observe source freshness without writing state. +- [sqb diff](cli/diff.md) (`cli/diff`) - Compare schemas and data between targets. +- [sqb review](cli/review.md) (`cli/review`) - Build what a branch changed in a temporary schema, test it, compare it with a target and report. - [sqb clone](cli/clone.md) (`cli/clone`) - Copy model relations between configured targets. -- [diff](cli/diff.md) (`cli/diff`) - Compare schemas and data between targets. -- [lineage](cli/lineage.md) (`cli/lineage`) - Explore model and column-level dependency graphs from the command line. -- [sqb dag](cli/dag.md) (`cli/dag`) - Generate the static DAG artifact for Dagster and other integrations. -- [sqb query](cli/query.md) (`cli/query`) - Run ad hoc SQL queries against the project database. -- [sqb recall](cli/recall.md) (`cli/recall`) - Query the recorded history of past invocations. -- [debug](cli/debug.md) (`cli/debug`) - Validate project configuration and test the warehouse connection. - [sqb janitor](cli/janitor.md) (`cli/janitor`) - Archive and then delete stale warehouse relations. +- [sqb recall](cli/recall.md) (`cli/recall`) - Query the recorded history of past invocations. +- [sqb query](cli/query.md) (`cli/query`) - Run ad hoc SQL queries against the project database. - [sqb clean](cli/clean.md) (`cli/clean`) - Remove compiled artifacts from the target directory. +- [sqb dag](cli/dag.md) (`cli/dag`) - Generate the static DAG artifact for Dagster and other integrations. - [sqb dbt](cli/dbt.md) (`cli/dbt`) - Coordinate dbt and SQLBuild projects. - -## Other pages - -- [sqb review](cli/review.md) (`cli/review`) - Build what a branch changed in a temporary schema, test it, compare it with a target and report. -- [Enums and Constants Have Moved](concepts/enums-and-constants.md) (`concepts/enums-and-constants`) - Find the new focused guides for enums, constants, collections, contracts, and private values. +- [sqb server](cli/server.md) (`cli/server`) - Run the control service and scheduler. +- [sqb login and logout](cli/login.md) (`cli/login`) - Authorize a bounded CLI session through your server's browser sign-in. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/audit.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/audit.md index d859ce83aa..e13a437384 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/audit.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/audit.md @@ -29,7 +29,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb | `--concurrency` | Maximum number of audits to run concurrently (default: `1`) | | `--json` | Print structured audit results | | `--json-output` | Write structured audit results to a file | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | ## Concurrency diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md index 5bd84c5745..843fa32d69 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md @@ -17,6 +17,7 @@ Online: https://sqlbuild.com/docs/cli/build/ - Output - Deferred builds - Failure behavior +- Limits - Fingerprints - Runtime artifacts @@ -61,7 +62,7 @@ own option. `--version` works only directly after `sqb`, as in `sqb --version`: | `--defer-sources-to` | Read managed source data from another target | | `--fail-fast` | Stop on first failure and skip remaining nodes | | `--concurrency` | Number of worker connections (default: 1) | -| `--test-timeout ` | Time limit for each unit test the build runs, such as `30s`, or `disabled`; see [Time limits](../concepts/testing.md#time-limits) | +| `--test-timeout ` | Time limit for each unit test the build runs, such as `30s`, or `disabled`; see [Time limits](../concepts/testing-reference.md#time-limits) | | `--verbose`, `-v` | Show lifecycle SQL inline after each model | | `--start-cursor` | Override the start cursor, typed from the selected incremental models' cursor type; fails with `S303` if the selection mixes timestamp and integer cursors. With no cursor model selected, an integer value is read as an integer cursor and anything else as an ISO timestamp | | `--end-cursor` | Override the end cursor, typed the same way as `--start-cursor` | @@ -77,7 +78,7 @@ own option. `--version` works only directly after `sqb`, as in `sqb --version`: | `SELECTOR ...` | Positional selectors, the same as `--select` | | `--select`, `-s` | Select specific models, or unit tests by name (`test:`); selecting only unit tests runs them without building models | | `--exclude` | Exclude specific models | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | ## Fast iteration @@ -219,6 +220,58 @@ skipped as for any other failure. Its row shows `TIMEOUT`. In the summary line, failures were timeouts. A timeout is a failure for the exit code: the build exits non-zero and its JSON `status` is `failed`. +## Limits + +A target's [`execution_limits`](../concepts/project-configuration.md#execution-limits) cap a whole +build, and a model's [`max_duration`](../concepts/models/configuration.md#execution-time-limit) caps +one model. + +### Model limit + +When the build would run more models than `max_models`, the build stops before any clone, load or +model runs, and says that nothing in the warehouse changed. `sqb plan` has no limit, so use it to +check a broad selection. + +### Duration limit + +`max_duration` covers planning and the build. At the deadline SQLBuild starts no more work and +cancels the statements still running. Where the warehouse has a statement time limit, each +statement also gets only the time that remains. + +| Adapter | Warehouse statement limit | Cancellation at the deadline | Longest accepted duration | +|---------|---------------------------|------------------------------|---------------------------| +| Snowflake | `STATEMENT_TIMEOUT_IN_SECONDS` per statement | Driver cancel request for the running query | 7 days | +| DuckDB, MotherDuck | None | Connection interrupt | About 24 days | +| PostgreSQL | `SET LOCAL statement_timeout` inside transactions; statements outside a transaction rely on cancellation | Driver cancel (`cancel_safe`) | About 24 days | +| BigQuery | Job timeout (`job_timeout_ms`) on each query job | `jobs.cancel`; reported as requested, not confirmed | About 24 days | +| Databricks | `STATEMENT_TIMEOUT` session setting, capped at 48 hours | Cursor cancel; reported as requested, not confirmed | About 24 days | +| SQL Server | None; the driver's query timeout is process-wide | Driver cancel | About 24 days | +| SQLite | None | Connection interrupt | About 24 days | + +A duration timeout can come after earlier models finished, so its error says that warehouse changes +may already have been made. A custom adapter that can't cancel a running statement can't be used +with `max_duration`. + +### Per-item time limits + +Unit test, scenario and model limits (`test_timeout`, `scenario_timeout` and a model's +`max_duration`) count running time only, from the item's first warehouse statement. Rollback and +cleanup always run and don't count. Where a warehouse reports queueing separately, queued +time doesn't count: + +| Adapter | How the item limit is enforced | Queued time | +|---------|--------------------------------|-------------| +| Snowflake | `STATEMENT_TIMEOUT_IN_SECONDS` per statement, with a client-side backstop after a grace margin of half the limit (at least 30 seconds) | Excluded by the warehouse limit; the backstop counts it | +| DuckDB, MotherDuck | Connection interrupt at the limit | Not applicable | +| PostgreSQL | `SET LOCAL statement_timeout` inside transactions; cancellation at the limit | Lock waits count | +| BigQuery | `jobs.cancel` at the limit; no job timeout, because a job timeout counts from job creation | Excluded: the clock stops while a job is pending | +| Databricks | `STATEMENT_TIMEOUT` session setting and cursor cancel | May count | +| SQL Server | Driver cancel at the limit | Lock waits count | +| SQLite | Connection interrupt at the limit | Waits for the write lock count | + +When limits nest, such as a model's unit tests inside its own limit, the earliest deadline applies. +After a timeout, SQLBuild checks the connection and replaces it if it is no longer usable. + ## Fingerprints After a successful build, SQLBuild writes version identities to `_sqlbuild_fingerprints` in the target schema. These are used on subsequent runs to detect changes and skip unchanged work. A build writes a row only for a resource whose identity differs from its latest row, so a build that changes no definitions writes none. In a project with model migration history, every build writes a row for each resource it builds. See [Planning and Change Detection](../concepts/planning.md) for details. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/check.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/check.md index 967cadc701..c35049ecd0 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/check.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/check.md @@ -27,7 +27,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb | `--json` | Print check results as JSON | | `--json-output` | Write check results JSON to a file path | | `--vars` | Override project variables | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | Selecting a non-check node is rejected; use [`sqb build`](build.md) to run tasks and assets. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clone.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clone.md index dd95ba1219..fd7e912523 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clone.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clone.md @@ -44,7 +44,7 @@ sqb --project-dir clone --from [--to ] [flags] | `--hard-copy` | Force physical table copies instead of zero-copy cloning | | `--select`, `-s` | Select specific models to clone | | `--exclude` | Exclude specific models from cloning | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | ## Examples @@ -95,4 +95,4 @@ allow_as_clone_destination = true The origin target must already contain the built relations being cloned. The destination connection's credentials must be able to read the origin namespace and create relations in the destination. Managed physical sources may bootstrap a destination that has not been built. See -[Project Configuration](../concepts/project-configuration.md) for details. +[Configuration reference](../concepts/project-configuration.md) for details. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md index 598f2d3363..48b1f1d846 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md @@ -78,7 +78,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb 2. **Graph.** Resolves `__ref()`, `__source()` and `__seed()`, expands macros, and orders models by dependency. 3. **SQL checks.** Parses and checks SQL, when SQL analysis is on. See - [Semantic Compilation](../concepts/semantic-compilation.md). + [What `sqb compile` checks](../concepts/semantic-compilation.md). 4. **Column lineage.** Traces column-level dependencies across models. 5. **Contracts.** Checks declared columns against the inferred query output. 6. **Rules.** Runs the selected built-in Rules, then the selected custom Rules. @@ -89,8 +89,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb Rules that need inferred types, such as `SQBRCONTRACT105`, are skipped when SQL analysis is off, and compile prints one note naming them. -`sqb compile` is authoritative for correctness, contracts, lineage and configured Rules. Rule -findings block the artifacts. To check one Rule code or family without changing project +Rule findings block the artifacts. To check one Rule code or family without changing project configuration, use [`sqb rules run`](rules.md). ### When Rules fail @@ -170,19 +169,14 @@ error[K001]: declared column 'total_cents' was not found in statically inferred | `K001` | A declared column is missing from the query output | | `K002` | A column's inferred type doesn't match the declared type | | `K003` | A column's type can't be proven (with `type_enforcement` on) | +| `K004` | A `nullable false` column can be null | +| `K005` | A model with `contract enforced` outputs a column it doesn't list | +| `K006` | A model with `contract enforced` lists no columns | | `K013` | A string literal fills an enum-typed column of an enforced contract but isn't a member of the enum (not checked on SQL Server) | -Which models are checked depends on `[settings] check_declared_columns`: - -- `true` (default): a model with declared columns and no `contract` setting must output every - declared column. -- `false`: only models with `contract enforced` are shape-checked; declared columns elsewhere - are metadata and audit targets. - -Explicit type enforcement works with either value. Setting `false` is a project policy, not a -way to silence an error: if a model is meant to have a contract, fix the query or the -declaration. Errors on `contract enforced` models don't suggest changing the setting. See -[Contracts](../concepts/models/contracts.md). +Models without a `contract` setting are checked unless the project sets +`[settings] check_declared_columns = false`; see [Contracts](../concepts/models/contracts.md). Don't +turn the check off to silence one error: fix the query or the column list. ## Output diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/contract.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/contract.md index ecb0141d99..06382ab2e9 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/contract.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/contract.md @@ -6,7 +6,9 @@ Online: https://sqlbuild.com/docs/cli/contract/ -Existing-schema comparison is explicitly online and read-only with respect to the warehouse. +`sqb contract` compares a model's or source's declared columns with the table that already exists in +the warehouse, or writes declarations from it. It reads the warehouse and never changes it. The +target named by `--from` supplies the database and schema to read, using the project's connection. ## diff @@ -22,8 +24,22 @@ Reports declaration differences. Exit code `0` means agreement and `1` means dif sqb contract generate --from [flags] ``` -Prints proposed declaration updates. Add `--write` to change repository files and `--overwrite` to -replace conflicts rather than preserving authored declarations. +Prints proposed declaration updates without changing files. + +```bash +# Fill in missing types and add warehouse columns missing from code +sqb contract generate --from prod --select tag:commerce --write + +# Also replace conflicting types and names, and remove columns the warehouse doesn't have +sqb contract generate --from prod --select tag:commerce --write --overwrite +``` + +Without `--overwrite`, generation keeps everything you wrote: existing types where they conflict, +columns only in code, comments, descriptions, audits, nullability, enums, tags, freshness and loader +settings. It never adds `contract enforced`; that stays your decision. + +A shared `SCHEMA()` isn't changed from one model's columns; the conflict is reported instead. Writes +are atomic, and SQLBuild recompiles the project afterwards and restores the old files if it fails. ## Flags @@ -37,7 +53,7 @@ replace conflicts rather than preserving authored declarations. | `--vars` | both | Override project variables | | `--write` | generate | Apply generated repository changes | | `--overwrite` | generate | Replace conflicting declarations; requires `--write` | -| `--warehouse ` | both | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | both | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | -See [Contracts](../concepts/models/contracts.md#adopting-an-existing-schema) for ownership and safety -guarantees. +Sources can be selected like models, for example `--select source:raw_orders`. See +[Contracts](../concepts/models/contracts.md) for what a contract checks. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dbt.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dbt.md index 0dfdb25eaa..6ea4c53a3c 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dbt.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dbt.md @@ -6,6 +6,16 @@ Online: https://sqlbuild.com/docs/cli/dbt/ +## Contents + +- sqb dbt plan +- sqb dbt run +- sqb dbt build +- Flags +- sqb dbt debug +- Selectors +- Configuration + Orchestrate dbt and SQLBuild together. Each subcommand runs dbt first, then SQLBuild, with selection logic across both project graphs. dbt remains responsible for dbt-owned models; SQLBuild validates and executes SQLBuild-owned models downstream. See [Using SQLBuild with dbt](../concepts/dbt-compatibility/overview.md) for scope and selection behavior. ## sqb dbt plan @@ -48,6 +58,40 @@ confirmation asks for the typed confirmation on an interactive terminal and refu build the affected models with `sqb build` and its allow flag instead, or relax the policy. A refusal at this point leaves completed dbt work in place and builds no SQLBuild models. +## Flags + +`sqb dbt plan/run/build` declare the common flags directly. Anything declared goes **before** a `--` separator; any other raw dbt flag goes **after** it and is forwarded verbatim. A flag placed on the wrong side errors rather than silently reaching dbt. + +```bash +sqb dbt build --select path:models/marts --full-refresh +``` + +Declared flags are routed to the right place automatically: + +| Flag | Routed to | Notes | +| ---- | --------- | ----- | +| `--select` / `--exclude` | dbt | dbt resolves selection (see [Selection](../concepts/dbt-compatibility/selection.md)). | +| `--vars` | dbt **and** SQLBuild | The same vars feed dbt's compile and SQLBuild's own variable resolution, so both sides see identical values. | +| `--full-refresh` | dbt **and** SQLBuild | Requests a full rebuild on both sides. dbt and native SQLBuild models independently apply their model-level `full_refresh` setting, including `false` opt-outs. | +| `--threads` | dbt (`--threads`) and SQLBuild (`--concurrency`) | | +| `--target` / `--project-dir` / `--profiles-dir` / `--profile` / `--target-path` | dbt | Standard dbt locators. | + +### `--vars` + +`--vars` accepts the same JSON object dbt accepts, and SQLBuild passes it to **both** the underlying `dbt` invocation and its own variable resolution. That means a value referenced as `@@my_var` in SQLBuild model SQL (see [Variables and placeholders](../concepts/interpolation.md)) and as `{{ var('my_var') }}` in a dbt model both resolve to the value you passed. CLI vars take precedence over project and local config vars. You do not need to declare vars twice. + +```bash +sqb dbt build --select path:models/marts --vars '{"my_var": 1}' +``` + +### Forwarding other dbt flags + +For any native dbt flag SQLBuild does not declare, put it after `--` and it is passed straight to the `dbt` invocation untouched: + +```bash +sqb dbt build -- --log-level debug +``` + ## sqb dbt debug Run dbt diagnostics followed by SQLBuild diagnostics. @@ -93,9 +137,9 @@ profiles_dir = "../profiles" target_path = "../dbt_project/target" ``` -See [Project Configuration](../concepts/project-configuration.md#dbt) for all fields. +See [Using SQLBuild with dbt](../concepts/dbt-compatibility/overview.md#configuration) for all fields. SQLBuild-owned work in `sqb dbt plan`, `run` and `build` connects with the target's `build` -[warehouse group](../concepts/project-configuration.md#command-group-warehouses). dbt itself +[warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses). dbt itself keeps using the warehouse from its profile. These commands do not accept `--warehouse`, because their remaining arguments pass through to dbt. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/debug.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/debug.md index 6360945e36..4f53bb48a9 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/debug.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/debug.md @@ -1,6 +1,6 @@ -# debug +# sqb debug > Validate project configuration and test the warehouse connection. @@ -22,7 +22,7 @@ The command reports four groups of checks: **Runtime** - SQLBuild version, Python version, Python path, OS info. -**Configuration** - Finds and validates `sqlbuild_project.toml`, loads `sqlbuild_local.toml` if present, resolves the adapter, active target, connection, database, and schema. For Snowflake it also shows the warehouse each [command group](../concepts/project-configuration.md#command-group-warehouses) connects with and where that value came from. +**Configuration** - Finds and validates `sqlbuild_project.toml`, loads `sqlbuild_local.toml` if present, resolves the adapter, active target, connection, database, and schema. For Snowflake it also shows the warehouse each [command group](../concepts/adapters/snowflake.md#command-group-warehouses) connects with and where that value came from. **Providers** - Lists the [providers](../concepts/python-nodes/providers.md) the project defines. @@ -34,7 +34,7 @@ The command reports four groups of checks: |------|-------------| | `--no-connection` | Skip the connection and query tests. Useful for validating config without warehouse access. | | `--json` | Output results as JSON instead of formatted text. | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | ## Example diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/diff.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/diff.md index 3a4ecc75a4..79bb52fae3 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/diff.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/diff.md @@ -1,6 +1,6 @@ -# diff +# sqb diff > Compare schemas and data between targets. @@ -70,7 +70,7 @@ In query mode, omit `FROM:TO` and model selectors; each query identifies its own | `--json` | Print one JSON document to stdout | | `--select`, `-s` | Select specific models to diff (required in model mode) | | `--exclude` | Exclude specific models from diffing | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | ## Size guard diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/format.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/format.md index 5cf1bcd4ec..01cf27cd3e 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/format.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/format.md @@ -45,14 +45,11 @@ one-file-at-a-time run. ## Skipping unchanged files -`sqb format` remembers every file it found already formatted and free of faults and warnings, in -`target/cache/compiler-native-v1/format-results.bin`. A later run skips such a file while its text, -its path, the `[format]` and rule threshold settings, the adapter dialect and the SQLBuild build are -all unchanged, so `sqb format --check` on an unchanged project costs little more than reading the -files. A file that needed changes or reported anything is always formatted again, and a damaged or -unreadable cache is ignored. `--fix` never uses the cache, and SQL test files are not skipped -while typed-null fixture formatting reads their models' contracts. Pass `--no-cache` to format -every file. +`sqb format` skips files it already found formatted and clean, as long as the file and the format +settings haven't changed, so `sqb format --check` on an unchanged project is fast. A file that +needed changes or reported anything is always checked again, and SQL test files are always +formatted. `--fix` never skips files. Pass +`--no-cache` to format every file. ## Line width and descriptions @@ -63,16 +60,11 @@ Configure the maximum physical line width in `sqlbuild_project.toml` (the defaul line_width = 100 ``` -`MODEL`, `SCENARIO`, `FUNCTION` and `HOOK` headers must carry a non-empty `description`; -`sqb format --check` fails with `description-present` when one is missing, so pre-commit catches it -before compile does (see [Descriptions](../concepts/models/configuration.md#descriptions)). A model -described by its `model_schema`, or by the one `[path_defaults]` entry that applies to it, passes, -exactly as it compiles. `[defaults]` does not accept `description` (`D001`), and only the nearest -matching path default applies (see -[Which path default applies](../concepts/project-configuration.md#which-path-default-applies)). -Format does not render templates, so a templated description that renders empty passes format and -is reported by `sqb compile`. -Plain `sqb format` still formats the file and exits non-zero, like any other remaining fault. +`MODEL`, `SCENARIO`, `FUNCTION` and `HOOK` headers need a non-empty `description`. When one is +missing, `sqb format --check` fails with `description-present`, so pre-commit catches it before +compile does. A model described by its `model_schema` or its `[path_defaults]` entry passes, as it +does in compile; see [Descriptions](../concepts/models/configuration.md#descriptions). Plain +`sqb format` still formats the file and exits non-zero. Descriptions of `MODEL`, `SCENARIO`, `SCHEMA`, `FUNCTION`, `HOOK` and `MACRO` headers have one canonical layout, which never changes their words, their quotes or the text they store (see @@ -111,15 +103,11 @@ Clause layout follows a few fixed rules: - A broken `DECODE` puts its first argument on its own line, then one search/result pair per line, then any default. -SQL lines longer than the width, counting any trailing alias and comma, are broken at the outermost -structure first: before top-level `AND` and `OR` (never inside `BETWEEN ... AND`), then before -arithmetic operators, then by putting the arguments of the widest outermost function call, `IN` -list, or array one per line, or the branches of a `CASE` on their own lines. Inside `OVER (...)`, -`PARTITION BY` and `ORDER BY` go on separate lines. Type parameters such as `NUMBER(38, 2)` or -`VARCHAR(MAX)` are never split; the enclosing call breaks instead. Header `audits [...]` lists put -each audit, and then each audit argument, on its own line. String literals, comments, and lines that -carry a comment are never broken, so a line can stay longer than the width when nothing on it can be -split. `sqb format --check` fails while a long line can still be shortened. +A line longer than the width is broken at its outermost structure first: before top-level `AND` +and `OR`, then before arithmetic operators, then one argument per line in the widest function call, +`IN` list or array, or one `CASE` branch per line. String literals, comments, type parameters such as +`NUMBER(38, 2)`, and lines that carry a comment are never broken, so a line can stay longer than the +width. `sqb format --check` fails while a long line can still be shortened. ## What formatting preserves diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md index b8a2788104..84eaa4d94b 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md @@ -40,7 +40,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb | `SELECTOR ...` | Positional selectors, the same as `--select` | | `--select`, `-s` | Select specific sources or models (sources upstream of selected models are included) | | `--exclude` | Exclude specific sources or models | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | ## Source selection diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md index 998306878c..fffdcaa3b3 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md @@ -17,6 +17,7 @@ Online: https://sqlbuild.com/docs/cli/janitor/ - Configuration - Output - Examples +- Details `sqb janitor` cleans up relations your project no longer builds. It never drops a live relation directly: it first renames a stale relation to an archive name, and drops the archive only once it @@ -60,7 +61,7 @@ sqb --project-dir janitor [flags] | `--drop-old-name-view ` | Drop the [compatibility view](../concepts/models/migrations.md#old-names) at a renamed model's old name before it expires. Repeat for several views. Fails with `C503` if no compatibility view is recorded at the name | | `--verbose`, `-v` | List every entry of each preview section and every skipped object with its reason | | `--json` | Print the plan and the result as one JSON document on stdout; the preview, prompt and progress go to stderr | -| `--warehouse ` | Snowflake warehouse for this run; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this run; overrides the target's `query` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | ## Lifecycle @@ -245,62 +246,7 @@ preview and the prompt show one notice, such as `usage not checked: unsupported janitor archives as usual. With `keep_in_use` on, a second notice, `nothing kept in use`, says that in-use relations are archived too. -#### What counts as a read - -**Snowflake.** A relation read through the objects a query named shows `via` those objects. -Everything a directly read object is built on, through any number of views, is in use, so a -candidate view between a view someone reads and a candidate table is kept too. Links through views -in other databases aren't followed, and dependencies created by `CREATE TABLE ... AS SELECT` and -similar statements aren't recorded. These don't count as reads: - -- queries SQLBuild tags as its own (`QUERY_TAG` with `"app":"sqlbuild"`, set by builds, audits, - loads and hooks); -- `CREATE ... CLONE` of tables, dynamic tables, schemas and databases. A statement counts as a clone - only when it starts with `CREATE`, after any comments, and names its source after `CLONE`. - -Without `OBJECT_VIEWER`, reads are still checked but view chains aren't followed, and the preview -says `view chains not followed: grant SNOWFLAKE.OBJECT_VIEWER`. - -**BigQuery.** Successful query jobs that reference the relation count, by `user_email`. A job lists -the views it reads and the tables behind them, so both count. These are missed or don't count: - -- a candidate view read only through another view may not show, because BigQuery doesn't document - whether nested views are listed; -- jobs answered from cached results record no tables, so with an `in_use_window` under about a day - a reader whose queries were all cached can be missed; -- jobs that run in another project aren't seen; -- script parent jobs, `CREATE SNAPSHOT TABLE` and `CREATE TABLE ... CLONE` don't count; -- SQLBuild's own jobs count like any other. - -**Databricks.** Read events in Unity Catalog's table lineage count, by `created_by`. A view a query -reads is a direct read, and the tables behind it are reads through the view. Lineage covers only -reads Databricks can infer lineage for, and can arrive late, so "not in use" is a weaker signal than -on Snowflake. Clones and SQLBuild's own reads count as reads. - -**ClickHouse.** Finished `SELECT` and `INSERT` queries count, by the user who started them, once per -initial query. A query lists every table and view it touched, so a whole view chain shows as in -use, without `via`. Writes count, since renaming the relation breaks the writer too. These are -missed or don't count: - -- queries on other replicas or shards, which matters on ClickHouse Cloud and behind load-balanced - replicas. When `system.clusters` shows other replicas of the connected server's shard, the preview - says `usage read from this server only; other replicas not checked`; -- queries ClickHouse answers from metadata without logging the table, such as a plain `count()`; -- readers whose settings profile turns `log_queries` off or samples it with - `log_queries_probability`. When `log_queries` is off for the janitor's own connection, usage isn't - checked; -- queries in the last few seconds: the log is flushed about every 7.5 seconds; -- queries older than the server's `query_log` TTL; -- SQLBuild's own reads count. - -### Audit record - -Every archive and delete is recorded in an append-only `_sqlbuild_janitor_events` table in the -affected schema: the original name, the archive name, the time and the invocation. The janitor -never reads it to decide what to do; it is for auditing, and for restoring archives whose names -were shortened, which are recorded right after their rename. Other events are written in batches -of up to 100 per schema, so if the process is killed, up to 100 actions in a schema may have no -event. Those archives still expire by the timestamp in their names. +See [What counts as a read](#what-counts-as-a-read) for each warehouse's limits. ## Targets @@ -342,7 +288,7 @@ in_use_window = "7d" keep_in_use = false ``` -See [Project Configuration](../concepts/project-configuration.md#janitor) for every key and default. +See [Configuration reference](../concepts/project-configuration.md#janitor) for every key and default. On Snowflake, BigQuery and Databricks, the actions of each phase run on up to the target's [`concurrency`](../concepts/project-configuration.md#target-concurrency) connections at once. @@ -406,3 +352,61 @@ sqb janitor --target prod # Keep relations that are still read instead of archiving them sqb janitor --keep-in-use ``` + +## Details + +### What counts as a read + +**Snowflake.** A relation read through the objects a query named shows `via` those objects. +Everything a directly read object is built on, through any number of views, is in use, so a +candidate view between a view someone reads and a candidate table is kept too. Links through views +in other databases aren't followed, and dependencies created by `CREATE TABLE ... AS SELECT` and +similar statements aren't recorded. These don't count as reads: + +- queries SQLBuild tags as its own (`QUERY_TAG` with `"app":"sqlbuild"`, set by builds, audits, + loads and hooks); +- `CREATE ... CLONE` of tables, dynamic tables, schemas and databases. A statement counts as a clone + only when it starts with `CREATE`, after any comments, and names its source after `CLONE`. + +Without `OBJECT_VIEWER`, reads are still checked but view chains aren't followed, and the preview +says `view chains not followed: grant SNOWFLAKE.OBJECT_VIEWER`. + +**BigQuery.** Successful query jobs that reference the relation count, by `user_email`. A job lists +the views it reads and the tables behind them, so both count. These are missed or don't count: + +- a candidate view read only through another view may not show, because BigQuery doesn't document + whether nested views are listed; +- jobs answered from cached results record no tables, so with an `in_use_window` under about a day + a reader whose queries were all cached can be missed; +- jobs that run in another project aren't seen; +- script parent jobs, `CREATE SNAPSHOT TABLE` and `CREATE TABLE ... CLONE` don't count; +- SQLBuild's own jobs count like any other. + +**Databricks.** Read events in Unity Catalog's table lineage count, by `created_by`. A view a query +reads is a direct read, and the tables behind it are reads through the view. Lineage covers only +reads Databricks can infer lineage for, and can arrive late, so "not in use" is a weaker signal than +on Snowflake. Clones and SQLBuild's own reads count as reads. + +**ClickHouse.** Finished `SELECT` and `INSERT` queries count, by the user who started them, once per +initial query. A query lists every table and view it touched, so a whole view chain shows as in +use, without `via`. Writes count, since renaming the relation breaks the writer too. These are +missed or don't count: + +- queries on other replicas or shards, which matters on ClickHouse Cloud and behind load-balanced + replicas. When `system.clusters` shows other replicas of the connected server's shard, the preview + says `usage read from this server only; other replicas not checked`; +- queries ClickHouse answers from metadata without logging the table, such as a plain `count()`; +- readers whose settings profile turns `log_queries` off or samples it with + `log_queries_probability`. When `log_queries` is off for the janitor's own connection, usage isn't + checked; +- queries in the last few seconds: the log is flushed about every 7.5 seconds; +- queries older than the server's `query_log` TTL; +- SQLBuild's own reads count. + +### Audit record + +Every archive and delete is recorded in an append-only `_sqlbuild_janitor_events` table in the +affected schema: the original name, the archive name, the time and the invocation. The janitor +never reads it to decide what to do; it is for auditing, and for restoring archives whose names +were shortened, which are recorded right after their rename. If the process is killed, its last +actions may have no record; those archives still expire by the timestamp in their names. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/lineage.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/lineage.md index a36f972e1b..710838a916 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/lineage.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/lineage.md @@ -1,6 +1,6 @@ -# lineage +# sqb lineage > Explore model and column-level dependency graphs from the command line. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md index c4229a95fe..227a3b9818 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md @@ -31,7 +31,7 @@ sqb --project-dir load [flags] | `--json` | Output results as JSON | | `--json-output` | Write JSON results to a file path | | `--vars` | Override project variables with a JSON object (`--vars '{"key": "value"}'`) | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | ## Examples diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md index fc8cb69d9c..5524f18908 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md @@ -46,7 +46,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb | `SELECTOR ...` | Positional selectors, the same as `--select` | | `--select`, `-s` | Select specific models | | `--exclude` | Exclude specific models | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | ## Output diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/query.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/query.md index 994aa9bbff..be4ee837a2 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/query.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/query.md @@ -42,7 +42,7 @@ File paths are resolved from the current working directory. | `--json` | Shorthand for `--format json` | | `--limit` | Maximum rows to return (default: 20) | | `--no-limit` | Disable the row limit | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | ## Output diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/recall.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/recall.md index ee013c108d..3068676d84 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/recall.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/recall.md @@ -29,7 +29,7 @@ sqb recall output last ## sqb recall query Run one read-only `SELECT` against the Recall views, in the store's dialect: SQLite for the local -store, ClickHouse for a [ClickHouse store](../concepts/observability/recall.md#sharing-history-in-clickhouse). +store, ClickHouse for a [ClickHouse store](../concepts/observability/recall-clickhouse.md). Statements that write, and multiple statements, are rejected. | Flag | Description | diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/rename.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/rename.md index e578057321..c80138c627 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/rename.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/rename.md @@ -1,6 +1,6 @@ -# rename and mv +# sqb rename and sqb mv > Rename or move a model, or rename a column, and update every reference in one verified step. @@ -75,7 +75,7 @@ stg_order_cents models/finance/ stg_order_refunds models/finance/ ``` -The moves are planned and verified together: the project compiles once before planning and the edited project once after, whatever the number of moves. As with a single move, adding `migrate_from` to models whose relation moves takes one more compile of the edited project. Declarations and unit tests are placed for the state after every move, so a macro whose users all move goes straight to their new folder instead of up to a shared folder and back. States between the moves are never checked, which lets a batch make a change that no order of single moves could make. As with one move, nothing is written unless the whole batch compiles. +The moves are checked together, as one change: nothing is written unless the project compiles after all of them. Declarations and unit tests go where the final state needs them, so a macro whose users all move goes straight to their new folder. A batch is refused when it moves the same model twice, moves two models to the same file, gives two models the same new name, or gives a model the name another model of the batch gives up: the old name keeps working through a compatibility view after the rename, so it must stay free. With `--dry-run`, the output lists the combined edits of every move. @@ -88,7 +88,7 @@ sqb rename stg_orders stg_order_lines fact_orders fact_order_lines sqb rename --from-file renames.tsv --dry-run ``` -The renames are planned and verified together, with one compile before and one of the edited project after, and each renamed model gets its own `migrate_from`, so every old name keeps working through its compatibility view after the next build. The same refusals as for batch moves apply: a model renamed twice, two models given one name, or a model taking a name another model of the batch gives up. Column renames and `--cascade` take one `. ` per command; a batch that names a column is refused. +The renames are checked together, and each renamed model gets its own `migrate_from`. The same refusals as for batch moves apply: a model renamed twice, two models given one name, or a model taking a name another model of the batch gives up. Column renames and `--cascade` take one `. ` per command; a batch that names a column is refused. ## What changes @@ -141,7 +141,7 @@ By default a column rename is one step: downstream models keep their output name A renamed model keeps its warehouse history through a [model migration](../concepts/models/migrations.md). Whenever the model's relation moves and its materialization keeps warehouse data (table, view, incremental, or snapshot), the command adds `migrate_from ` to the model header, so the move does not depend on automatic discovery finding exactly one match. The next `sqb build` migrates the relation and leaves a compatibility view at the old name. A move that keeps the relation, such as a folder change in the same schema, adds nothing. -On a target that never built the old name, the origin is missing. By default the plan warns and the model builds fresh; set [`missing_migration_origin`](../concepts/project-configuration.md#missing-migration-origins) on a target to require confirmation or stop the build instead. +On a target that never built the old name, the origin is missing. By default the plan warns and the model builds fresh; set [`missing_migration_origin`](../concepts/models/migrations.md#missing-migration-origins) on a target to require confirmation or stop the build instead. A renamed column of an incremental or snapshot model gets ` (migrate_from )` in its `columns` block, so the next build renames it in place as a [column migration](../concepts/models/column-migrations.md). Table and view models are rebuilt with the new column anyway; they get the declaration only when the model itself declares `migrate_from`. @@ -151,9 +151,9 @@ The command refuses a rename while the model, or the column, still declares `mig A project that already fails some project rules, such as a folder holding more models than a layout rule allows, can still be refactored, so that a move can repair it. Only rule failures are tolerated: a project with syntax, binding, or reference errors is refused before planning, as is any project `sqb compile` cannot load. -The command compiles the project as `sqb compile` does, including SQL test planning, records its rule failures before planning, and compares the edited project with them. A rule failure of the edited project is existing when the project already had one with the same code, at the same place (a moved file keeps its identity), and with the same message. It is also existing when only its numbers changed and none grew, such as a folder that now holds 41 models instead of 44. Any other change, new numbers, larger numbers, or different words, makes it new or worse, and it blocks the write, as do all errors that are not rule failures. Rules stay enabled throughout; nothing is ignored. +A rule failure that the project already had, at the same place and with the same message, doesn't block the change. Neither does one whose numbers only went down, such as a folder that now holds 41 models instead of 44. A new or worse failure blocks the write. Rules stay enabled throughout. -The output, including `--dry-run`, lists the existing failures under `Existing rule failures`, with how many the change resolved and how many remain. Some rules report a failure at one sample file of a group, so moving part of the group can change the reported place or message and block the move even when it helps. Move the whole group in one batch instead: moving the direct models of a folder into a sub-folder at once, so that it turns from a leaf into a branch, is `sqb mv a models/entry/records/ b models/entry/records/ c models/entry/records/`. +The output, including `--dry-run`, lists them under `Existing rule failures`, with how many the change resolved. Some rules report a failure at one sample file of a group, so moving part of the group can block the move even when it helps. Move the whole group in one batch instead: moving the direct models of a folder into a sub-folder at once, so that it turns from a leaf into a branch, is `sqb mv a models/entry/records/ b models/entry/records/ c models/entry/records/`. ## Refusals diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scenario.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scenario.md index 687fa8b274..9d449d32ef 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scenario.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scenario.md @@ -41,7 +41,7 @@ sqb scenario test [flags] | `--max-snapshot-total-rows` | Override total row limit for capture | | `--max-snapshot-bytes` | Override per-relation byte limit for capture | | `--max-snapshot-total-bytes` | Override total byte limit for capture | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | | `--since ` | Run only the scenarios affected by changes since the merge base of `` and `HEAD`; `@base` reads `[git] base_ref` | | `--explain` | With `--since`, print why each selected scenario was selected | @@ -159,7 +159,7 @@ sqb scenario capture [flags] | `--max-snapshot-total-rows` | Override total row limit | | `--max-snapshot-bytes` | Override per-relation byte limit | | `--max-snapshot-total-bytes` | Override total byte limit | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | ### Examples diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scope.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scope.md index 6e87762f4b..7106ff1ba7 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scope.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scope.md @@ -1,6 +1,6 @@ -# scope +# sqb scope > Inspect declaration visibility, usage, placement, and move impact offline. @@ -20,7 +20,7 @@ Online: https://sqlbuild.com/docs/cli/scope/ - Paths And JSON - Partial Projects - Output Safety -- Native SQLBuild Only +- SQLBuild declarations only `sqb scope` explains the lexical environment around native SQLBuild macros, enums, and constants. Use it to see what a resource can access, what it actually uses, why a declaration is visible or inaccessible, and whether a move would cross a scope boundary. @@ -417,9 +417,9 @@ Scope reports describe declarations without exposing authored values or runtime There is no `--show-values` option. -## Native SQLBuild Only +## SQLBuild declarations only -`sqb scope` inspects native SQLBuild authored resources and declarations. It does not discover or emulate dbt models, dbt or Jinja macros, package dispatch, dbt manifests, dbt selectors, dbt tests, or dbt schema YAML visibility. An external dbt graph dependency does not contribute declarations or lexical scope. +`sqb scope` covers SQLBuild declarations only. Models and macros in a [dbt project](../concepts/dbt-compatibility/overview.md) used through `sqb dbt` are not part of any scope. For declaration directory rules, placement checks, test access through expected output, and scoped macro imports, see [Declarations and Scopes](../concepts/declaration-scopes.md). diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/seed.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/seed.md index 9ba3c4fae3..afe2c2a0ce 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/seed.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/seed.md @@ -23,7 +23,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb | `SELECTOR ...` | Positional selectors, the same as `--select` | | `--select`, `-s` | Select specific seeds by name | | `--exclude` | Exclude specific seeds | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | ## Examples diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md index 83e33999ed..70ab7391c9 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md @@ -32,7 +32,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb | `SELECTOR ...` | Positional selectors, the same as `--select` | | `--select`, `-s` | Select tests by name (`order_status_rules` or `test:order_status_rules`), or the tests targeting specific models | | `--exclude` | Exclude tests by name, or the tests targeting specific models | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse | | `--test-timeout ` | Time limit for each test in this run, such as `30s` or `2m`, or `disabled`; overrides `TEST(timeout ...)`, the target and `[settings] test_timeout` | | `--since ` | Run only the tests affected by changes since the merge base of `` and `HEAD` (see [Run only affected tests](#run-only-affected-tests)); `@base` reads `[git] base_ref` | | `--explain` | With `--since`, print why each selected test was selected | @@ -54,7 +54,7 @@ are strings in JSON so their scale is preserved. Each test check in JSON output `duration_ms`. Each test row shows how many real (unmocked) models the test executes, such as `3 real models`. A -test that runs past its [time limit](../concepts/testing.md#time-limits) is reported as `TIMEOUT`, +test that runs past its [time limit](../concepts/testing-reference.md#time-limits) is reported as `TIMEOUT`, counted under `TIMEOUT=` and excluded from `FAIL=` in the summary line (`PASS=4 FAIL=0 TIMEOUT=1 TOTAL=5`), and the run continues with the remaining tests. In `--json` output, `summary.fail_count` still includes timed-out tests, and `summary.timeout_count` reports how @@ -182,14 +182,13 @@ of guessing; compare with a later ref whose project compiles, or run without `-- with its default target instead; the run says so, and because the target changed, every test is selected. -SQL comparisons ignore layout and comments using a generic SQL tokenizer. In dialects where `--` -starts a comment only when followed by whitespace, such as MySQL, `1--1` is read as `1` followed by -a comment, so an edit inside such an expression can go unnoticed. The base tree is written from git -objects with the repository's filters. Git LFS content is restored for files that an unanchored -`.gitattributes` pattern marks, such as `*.csv filter=lfs`, but a pattern anchored to a path, such -as `/data/*.csv` at the repository root for a project in a subdirectory, does not match the -exported paths. Such a file is an LFS pointer at the base and real content in the checkout, so it -always counts as changed: at most extra tests run. +Details: + +- SQL comparisons ignore layout and comments. In dialects where `--` starts a comment only before + whitespace, such as MySQL, an edit inside an expression like `1--1` can go unnoticed. +- A file stored in Git LFS through a pattern anchored to a path, such as `/data/*.csv`, always + counts as changed, so extra tests may run. Unanchored patterns such as `*.csv filter=lfs` work as + expected. `` is any git ref or commit, such as the merge request's target branch from a CI variable. `--since` always takes a value. A ref that does not exist, or a shallow clone that does not reach diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md index 6c562417b0..d430bc210f 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md @@ -6,10 +6,6 @@ Online: https://sqlbuild.com/docs/concepts/adapters/ -## Contents - -- Custom adapters - SQLBuild uses adapters to connect to different database engines. Snowflake is the main target. - **Supported:** ready for production use. @@ -46,89 +42,5 @@ adapter = "duckdb" ## Custom adapters -You can write your own adapter for any database engine by placing Python files under `adapters/` in your project directory. - -### Extending a built-in adapter - -The most common case is extending an existing adapter with custom behavior. Subclass the built-in adapter, set a new `adapter_name`, and override what you need: - -```python -# adapters/duckdb_plus.py -from sqlbuild.adapters.duckdb.classes.duckdb_adapter import DuckDbAdapter - -class DuckDbPlusAdapter(DuckDbAdapter): - adapter_name = "duckdb_plus" - - def connect(self, config): - connection = super().connect(config) - # custom setup - load extensions, set pragmas, etc. - self.execute(connection=connection, sql="SET enable_progress_bar = true") - return connection -``` - -```toml -adapter = "duckdb_plus" -``` - -A subclass inherits the parent's capabilities. For example, an adapter extending the Snowflake -adapter keeps `supports_session_warehouse = True`, so it can use -[command-group warehouses](project-configuration.md#command-group-warehouses) and -`--warehouse`. An adapter written from scratch that sets this attribute receives the selected -warehouse as the connection's `warehouse` key. - -### Writing an adapter from scratch - -For a database engine with no built-in support, subclass `BaseAdapter`. It provides ANSI SQL defaults for most methods - you only need to implement `connect`, `execute`, `close`, and any methods where your engine differs from standard SQL: - -```python -# adapters/my_database.py -from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter - -class MyDatabaseAdapter(BaseAdapter): - adapter_name = "my_database" - sql_analysis_dialect_name = "postgres" # SQL dialect for validation and lineage - - def connect(self, config): - ... - - def execute(self, *, connection, sql): - ... - - def close(self, connection): - ... -``` - -Set `sql_analysis_dialect_name` to the SQL dialect name that matches your engine's SQL syntax. This enables compile-time SQL validation, column inference, column lineage, and local scenario replay for your adapter. If omitted, SQLBuild uses generic SQL parsing. SQL analysis is powered by [Polyglot](https://github.com/tobilg/polyglot), a Rust reimplementation of SQLGlot supporting 32+ dialects. - -For full control with no inherited defaults, subclass `StrictAdapter` instead. Every method is abstract and must be implemented explicitly. SQLBuild raises a clear error listing any unimplemented methods. - -A few `StrictAdapter` methods are optional and have a documented default. `create_sql_test_output` and `drop_sql_test_output` let an adapter store a unit test's actual output once, in a temporary relation, so every comparison of that test reads it instead of recomputing the model chain. By default `create_sql_test_output` returns `None`, and SQLBuild runs the test's inline comparison query instead, so adapters that do not implement them keep working unchanged. `render_create_temporary_function` returns the call target and DDL for a session-scoped copy of a project function, which `sqb test` calls instead of the built function (see [Functions in `sqb test`](testing.md#functions-in-sqb-test)). By default it returns `None`, and tests call the built function. `render_drop_schema` returns the statements that drop a schema and everything in it, which [`sqb review`](../cli/review.md) uses to remove the schema it built into. By default it raises an error, and `sqb review` is unavailable for that adapter. - -### Discovery rules - -- SQLBuild discovers all `.py` files under `adapters/` recursively (excluding `__init__.py` and files starting with `_`) -- Each file is scanned for classes that define a string `adapter_name` and subclass `StrictAdapter` (or any of its subclasses like `BaseAdapter` or a built-in adapter) -- Adapter names must be unique across all adapter files - duplicates raise an error -- Custom adapter names cannot shadow built-in names (`duckdb`, `motherduck`, `snowflake`, `bigquery`, `databricks`, `postgres`, `sqlite`, `sqlserver`, `clickhouse`) - -### Adapter class hierarchy - -``` -StrictAdapter (fully abstract - all methods must be implemented) - └── BaseAdapter (ANSI SQL defaults - override only what differs) - ├── DuckDbAdapter - ├── MotherDuckAdapter - ├── SnowflakeAdapter - ├── BigQueryAdapter - ├── DatabricksAdapter - ├── PostgresAdapter - ├── SqlServerAdapter - └── ClickHouseAdapter -``` - -`StrictAdapter` composes four mixins that define the full adapter contract: - -- **ConnectionMixin** - `connect`, `close`, `begin`, `commit`, `rollback` -- **SchemaMixin** - `relation_exists`, `list_relations`, `get_columns`, `ensure_schema` -- **MaterializationMixin** - `create_table_as`, `create_view_as`, `drop`, `rename`, `swap`, `load_seed` -- **DiffMixin** - `diff_schema`, `diff_rows`, `sample_unequal_rows`, `sample_side_only_rows`, `diff_unkeyed_rows` +To extend a built-in adapter or add another database engine, see +[Custom adapters](adapters/custom.md). diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/bigquery.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/bigquery.md index 6d7d3aa55f..5a964a8e5e 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/bigquery.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/bigquery.md @@ -69,4 +69,4 @@ See [Collections and Rendering](../constants/collections-and-rendering.md#native ## Functions in unit tests BigQuery temporary functions exist only inside one script, so `sqb test` calls the built function. -Run `sqb build` before testing a new or changed function. See [Functions in `sqb test`](../testing.md#functions-in-sqb-test). +Run `sqb build` before testing a new or changed function. See [Functions in `sqb test`](../testing-reference.md#functions-in-sqb-test). diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/clickhouse.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/clickhouse.md index e042fd3154..de0bc48827 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/clickhouse.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/clickhouse.md @@ -201,4 +201,4 @@ history for a whole team, whichever adapter the project builds with. Set reader users. Recall creates its database, tables and views on first use, keeps every output byte by default, and needs only `SELECT` and `INSERT` once the tables exist; readers need only `SELECT` and may be `readonly = 1` users. See -[Sharing history in ClickHouse](../observability/recall.md#sharing-history-in-clickhouse). +[Sharing Recall in ClickHouse](../observability/recall-clickhouse.md). diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/custom.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/custom.md new file mode 100644 index 0000000000..733cd175a7 --- /dev/null +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/custom.md @@ -0,0 +1,94 @@ + + +# Custom adapters + +> Extend a built-in adapter or write one for another database engine in Python. + +Online: https://sqlbuild.com/docs/concepts/adapters/custom/ + +You can write your own adapter for any database engine by placing Python files under `adapters/` in your project directory. + +## Extending a built-in adapter + +The most common case is extending an existing adapter with custom behavior. Subclass the built-in adapter, set a new `adapter_name`, and override what you need: + +```python +# adapters/duckdb_plus.py +from sqlbuild.adapters.duckdb.classes.duckdb_adapter import DuckDbAdapter + +class DuckDbPlusAdapter(DuckDbAdapter): + adapter_name = "duckdb_plus" + + def connect(self, config): + connection = super().connect(config) + # custom setup - load extensions, set pragmas, etc. + self.execute(connection=connection, sql="SET enable_progress_bar = true") + return connection +``` + +```toml +adapter = "duckdb_plus" +``` + +A subclass inherits the parent's capabilities. For example, an adapter extending the Snowflake +adapter keeps `supports_session_warehouse = True`, so it can use +[command-group warehouses](snowflake.md#command-group-warehouses) and +`--warehouse`. An adapter written from scratch that sets this attribute receives the selected +warehouse as the connection's `warehouse` key. + +## Writing an adapter from scratch + +For a database engine with no built-in support, subclass `BaseAdapter`. It provides ANSI SQL defaults for most methods - you only need to implement `connect`, `execute`, `close`, and any methods where your engine differs from standard SQL: + +```python +# adapters/my_database.py +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter + +class MyDatabaseAdapter(BaseAdapter): + adapter_name = "my_database" + sql_analysis_dialect_name = "postgres" # SQL dialect for validation and lineage + + def connect(self, config): + ... + + def execute(self, *, connection, sql): + ... + + def close(self, connection): + ... +``` + +Set `sql_analysis_dialect_name` to the SQL dialect name that matches your engine's SQL syntax. This enables compile-time SQL validation, column inference, column lineage, and local scenario replay for your adapter. If omitted, SQLBuild uses generic SQL parsing. + +For full control with no inherited defaults, subclass `StrictAdapter` instead. Every method is abstract and must be implemented explicitly. SQLBuild raises a clear error listing any unimplemented methods. + +A few `StrictAdapter` methods are optional and have a documented default. `create_sql_test_output` and `drop_sql_test_output` let an adapter store a unit test's actual output once, in a temporary relation, so every comparison of that test reads it instead of recomputing the model chain. By default `create_sql_test_output` returns `None`, and SQLBuild runs the test's inline comparison query instead, so adapters that do not implement them keep working unchanged. `render_create_temporary_function` returns the call target and DDL for a session-scoped copy of a project function, which `sqb test` calls instead of the built function (see [Functions in `sqb test`](../testing-reference.md#functions-in-sqb-test)). By default it returns `None`, and tests call the built function. `render_drop_schema` returns the statements that drop a schema and everything in it, which [`sqb review`](../../cli/review.md) uses to remove the schema it built into. By default it raises an error, and `sqb review` is unavailable for that adapter. + +## Discovery rules + +- SQLBuild discovers all `.py` files under `adapters/` recursively (excluding `__init__.py` and files starting with `_`) +- Each file is scanned for classes that define a string `adapter_name` and subclass `StrictAdapter` (or any of its subclasses like `BaseAdapter` or a built-in adapter) +- Adapter names must be unique across all adapter files - duplicates raise an error +- Custom adapter names cannot shadow built-in names (`duckdb`, `motherduck`, `snowflake`, `bigquery`, `databricks`, `postgres`, `sqlite`, `sqlserver`, `clickhouse`) + +## Adapter class hierarchy + +``` +StrictAdapter (fully abstract - all methods must be implemented) + └── BaseAdapter (ANSI SQL defaults - override only what differs) + ├── DuckDbAdapter + ├── MotherDuckAdapter + ├── SnowflakeAdapter + ├── BigQueryAdapter + ├── DatabricksAdapter + ├── PostgresAdapter + ├── SqlServerAdapter + └── ClickHouseAdapter +``` + +`StrictAdapter` composes four mixins that define the full adapter contract: + +- **ConnectionMixin** - `connect`, `close`, `begin`, `commit`, `rollback` +- **SchemaMixin** - `relation_exists`, `list_relations`, `get_columns`, `ensure_schema` +- **MaterializationMixin** - `create_table_as`, `create_view_as`, `drop`, `rename`, `swap`, `load_seed` +- **DiffMixin** - `diff_schema`, `diff_rows`, `sample_unequal_rows`, `sample_side_only_rows`, `diff_unkeyed_rows` diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/databricks.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/databricks.md index 094ba0a9f7..11b62a2f09 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/databricks.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/databricks.md @@ -50,4 +50,4 @@ the authoritative session namespace. `sqb test` creates each SQL UDF and table function the selected tests call as a session `TEMPORARY FUNCTION __sqb_test__`. Python UDFs cannot be temporary, so tests call the built -Python function. See [Functions in `sqb test`](../testing.md#functions-in-sqb-test). +Python function. See [Functions in `sqb test`](../testing-reference.md#functions-in-sqb-test). diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/duckdb.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/duckdb.md index d8d1dc708f..e2315e19fc 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/duckdb.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/duckdb.md @@ -6,33 +6,8 @@ Online: https://sqlbuild.com/docs/concepts/adapters/duckdb/ -DuckDB is included as a core dependency. No extra installation needed. - -SQLBuild requires DuckDB 2.0 or later. DuckDB 2.0 writes new database files in a storage format -that DuckDB 1.x cannot open, so a database file SQLBuild creates can only be read by DuckDB 2.0 -clients. A file that already exists keeps its storage version, so a file created by DuckDB 1.x -stays readable by 1.x. To create a new file that 1.x can read, create it before the first build -with an explicit storage version, for example -`duckdb my_project.duckdb -storage-version v1.5.0 -c "SELECT 1"`. - -## SQL that behaves differently on DuckDB 2.0 - -DuckDB 2.0 changes how some SQL runs. SQLBuild runs model SQL as written, so check models for -these before upgrading: - -- Arrow lambdas such as `list_filter(xs, x -> x > 0)` are rejected by default. Write - `list_filter(xs, lambda x: x > 0)` instead. DuckDB 2.1 removes the arrow syntax. -- `~` and `!~` match a regular expression anywhere in the string, as in PostgreSQL, so - `'abc' ~ 'b'` is now true. Use `regexp_full_match` for a match of the whole string. -- `MERGE` fails when one target row matches several source rows, instead of picking one. Keep - an incremental model's new rows unique on its `unique_key`. -- An untyped `NULL`, as in `SELECT NULL AS discount`, creates a column of type `"NULL"` that - rejects later non-null values. Cast it, for example `CAST(NULL AS DECIMAL(18, 2)) AS discount`. -- Integer division `//` and modulo `%` by zero raise an error instead of returning `NULL`. Wrap - the expression in `TRY(...)` to get `NULL`. - -dlt's dataset read API calls a DuckDB method that DuckDB 2.0 removed when it reads a loaded table -back as one whole Arrow table, so that read fails. Dataframe and chunked Arrow reads still work. Loading data with dlt is not affected. +DuckDB is included as a core dependency. No extra installation needed. SQLBuild requires DuckDB 2.0 +or later; see [Upgrading from DuckDB 1.x](#upgrading-from-duckdb-1x). ## Connection config @@ -85,17 +60,34 @@ the dbt project or the working directory. Every connection of a command attaches it once: connections to the same project database share the attachment. -## Table promotion mode - -DuckDB defaults to `staged` promotion: tables are materialized into a staging table, audited, then swapped into the target. This is configurable in `settings`: - -```toml -[settings] -table_promotion_mode = "staged" -``` - ## Functions in unit tests `sqb test` copies each SQL UDF and table function the selected tests call into a `CREATE TEMP MACRO` on every test connection, called as `temp.main.`, so tests use the current project -definition without a build. Python UDFs are registered on each test connection. See [Functions in `sqb test`](../testing.md#functions-in-sqb-test). +definition without a build. Python UDFs are registered on each test connection. See [Functions in `sqb test`](../testing-reference.md#functions-in-sqb-test). + +## Upgrading from DuckDB 1.x + +DuckDB 2.0 writes new database files in a storage format +that DuckDB 1.x cannot open, so a database file SQLBuild creates can only be read by DuckDB 2.0 +clients. A file that already exists keeps its storage version, so a file created by DuckDB 1.x +stays readable by 1.x. To create a new file that 1.x can read, create it before the first build +with an explicit storage version, for example +`duckdb my_project.duckdb -storage-version v1.5.0 -c "SELECT 1"`. + +When you upgrade from DuckDB 1.x, check models for SQL that runs differently on 2.0. SQLBuild runs +model SQL as written: + +- Arrow lambdas such as `list_filter(xs, x -> x > 0)` are rejected by default. Write + `list_filter(xs, lambda x: x > 0)` instead. DuckDB 2.1 removes the arrow syntax. +- `~` and `!~` match a regular expression anywhere in the string, as in PostgreSQL, so + `'abc' ~ 'b'` is true. Use `regexp_full_match` for a match of the whole string. +- `MERGE` fails when one target row matches several source rows, instead of picking one. Keep + an incremental model's new rows unique on its `unique_key`. +- An untyped `NULL`, as in `SELECT NULL AS discount`, creates a column of type `"NULL"` that + rejects later non-null values. Cast it, for example `CAST(NULL AS DECIMAL(18, 2)) AS discount`. +- Integer division `//` and modulo `%` by zero raise an error instead of returning `NULL`. Wrap + the expression in `TRY(...)` to get `NULL`. + +dlt's dataset read API calls a DuckDB method that DuckDB 2.0 removed when it reads a loaded table +back as one whole Arrow table, so that read fails. Dataframe and chunked Arrow reads still work. Loading data with dlt is not affected. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/motherduck.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/motherduck.md index 0d7d82f76a..8863efc57a 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/motherduck.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/motherduck.md @@ -89,4 +89,4 @@ This lets you develop and test locally with zero MotherDuck compute cost, then d `sqb test` copies each SQL UDF and table function the selected tests call into a session `TEMP MACRO`, the same as [DuckDB](duckdb.md#functions-in-unit-tests), so the -functions in your MotherDuck database are not replaced. See [Functions in `sqb test`](../testing.md#functions-in-sqb-test). +functions in your MotherDuck database are not replaced. See [Functions in `sqb test`](../testing-reference.md#functions-in-sqb-test). diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/postgres.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/postgres.md index 3517b77ccf..34e2d1ac71 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/postgres.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/postgres.md @@ -72,4 +72,4 @@ schema = "dev" calls it as `pg_temp.`, so tests use the current project definition without a build. Temporary functions belong to one server session: behind a transaction-mode pooler, run `sqb test` through a direct connection or a session-mode pool, or set `temporary_test_functions = false` so tests call the -built functions. See [Functions in `sqb test`](../testing.md#functions-in-sqb-test). +built functions. See [Functions in `sqb test`](../testing-reference.md#functions-in-sqb-test). diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md index fbd6184f4b..7646d3ce01 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md @@ -10,9 +10,9 @@ Online: https://sqlbuild.com/docs/concepts/adapters/snowflake/ - Connection config - Session initialization +- Command-group warehouses - Case-sensitive column names -- Planning metadata -- Build metadata +- Metadata reads - State tables - Shared connections across targets - Cost estimates @@ -59,10 +59,59 @@ On connect, SQLBuild applies the connection's role and warehouse, then applies t target's authoritative database and schema. These ensure the session context is set correctly regardless of the user's default settings. -The warehouse can differ by command. A target's `[targets..warehouses]` section selects -one warehouse for build-style commands and another for ad-hoc commands such as `sqb query` and -`sqb diff`, and `--warehouse` overrides both for one invocation. See -[command-group warehouses](../project-configuration.md#command-group-warehouses). +## Command-group warehouses + +Builds and ad-hoc queries usually need different compute. A target can set a warehouse for each +group of commands, so `sqb build` runs on a large warehouse while `sqb query` and `sqb diff` use a +small one: + +```toml +[connections.warehouse] +account = "my_org-my_account" +warehouse = "ANALYTICS_WH" # fallback for every command + +[targets.dev] +connection = "warehouse" +database = "ANALYTICS" +schema = "DEV" + +[targets.dev.warehouses] +build = "BUILD_WH" +query = "ADHOC_WH" +``` + +| Group | Commands | +|-------|----------| +| `build` | `audit`, `build`, `check`, `clone`, `dbt` (SQLBuild's own work), `load`, `plan`, `seed` | +| `query` | `contract`, `cost`, `debug`, `diff`, `freshness`, `janitor`, `query`, `scenario`, `test` | + +Each command uses the first of these that is set: + +1. `--warehouse `, for that run only +2. The target's warehouse for the command's group +3. The connection's `warehouse` +4. The Snowflake user's default warehouse + +Every connection a command opens uses the same warehouse. Tests and audits that run inside +`sqb build` use the `build` warehouse. Under `sqb dbt`, dbt's own nodes use the warehouse in their +dbt profile. + +A group warehouse wins over a connection `warehouse` from either file. To change one for yourself, +set the group in `sqlbuild_local.toml`; the other group keeps its project value: + +```toml +# sqlbuild_local.toml +[targets.dev.warehouses] +query = "ALICE_XS_WH" +``` + +Values accept `${ENV:...}` and `${my_var}` templates and must be Snowflake identifiers, unquoted +(`ADHOC_WH`) or double-quoted (`"Adhoc WH"`). `sqb debug` shows each group's warehouse and where it +came from. + +Groups need the Snowflake adapter or a custom adapter built on it; on other adapters they are an +error. If you switch `adapter` locally, for example to DuckDB, the shared groups are ignored and +`sqb debug` reports them as `not used`. ## Case-sensitive column names @@ -87,46 +136,19 @@ A snapshot or microbatch build stops when an unquoted setting names a case-sensi an error such as `unique_key 'orderId' names the case-sensitive column "orderId"; write it with its quotes, as '"orderId"'`. -## Planning metadata - -Planning reads relation and column metadata with `SHOW TABLES`, `SHOW VIEWS`, and `SHOW COLUMNS` -once per schema, several schemas at a time. `SHOW` commands are served by Snowflake's metadata -service, so they don't wait for, or resume, a warehouse. Two reads still use `INFORMATION_SCHEMA` -and need a running warehouse: - -- `LAST_ALTERED` for source freshness, read only for the sources the plan observes. -- Columns for a schema whose `SHOW COLUMNS` output is exactly 10,000 rows, the documented - limit at which the output may be cut off. +## Metadata reads -Run `sqb plan -v` to see each metadata query with its elapsed time and row count. - -## Build metadata - -Builds and scenario runs read relation metadata with `SHOW` commands too, so these reads don't -need a running warehouse either: - -| Read | Statement | -| -- | -- | -| Whether one relation exists | `SHOW TERSE TABLES` and `SHOW TERSE VIEWS` | -| A second relation in the same schema | `SHOW TABLES` and `SHOW VIEWS` for the whole schema, once | -| A table's time travel retention | `SHOW TABLES LIKE`, or the schema listing | -| One relation's columns | `SHOW COLUMNS` | -| A scenario's leftover relations | `SHOW TERSE OBJECTS` for the scenario's name prefix | - -Each run reuses these answers until it changes that relation, for example by replacing a delta -table or adding a column, and then reads it again. +Planning and builds read table, view and column metadata with `SHOW` commands, which don't need a +running warehouse. Source freshness, which reads `LAST_ALTERED` from `INFORMATION_SCHEMA`, and +columns of a schema with 10,000 or more columns need one. Probes for sources the listing didn't +find may need one too. Run `sqb plan -v` to see each metadata query with its time +and row count. ## State tables -SQLBuild records its state in `_sqlbuild_*` tables, such as `_sqlbuild_fingerprints`, -`_sqlbuild_source_freshness`, and `_sqlbuild_microbatches`. On Snowflake, SQLBuild creates every -state table as a permanent table with `DATA_RETENTION_TIME_IN_DAYS = 90`, the maximum time travel, -so the table also has Fail-safe. On editions that allow only 1 day of time travel, Snowflake -rejects 90 and SQLBuild creates the table with 1 day instead. - -The retention is set only in the statement that creates the table. SQLBuild never alters an -existing state table, so a state table that already exists keeps its current table type and -retention. Models are not affected: their table type follows the model `table_type` setting. +SQLBuild records its state in `_sqlbuild_*` tables. On Snowflake it creates them as permanent tables +with 90 days of time travel, or 1 day on editions that allow only that. It never alters an existing +state table. Models are not affected: their table type follows the model's `table_type` setting. ## Shared connections across targets @@ -171,26 +193,22 @@ contract rate. Use `sqb cost`, `sqb cost latest`, `sqb cost `, or provide a versioned, decimal-safe output contract. Pending detail records are refreshed from Snowflake when inspected again. -SQLBuild reads `INFORMATION_SCHEMA.QUERY_HISTORY` for the whole run in one query when the run -finishes. Each read has a 10-second statement timeout; a window that reaches it, or that returns -more rows than one read allows, is split in half and read again. Collection at the end of a command -has a 30-second budget so the command exits promptly. If history cannot be read within it, the -build is unaffected and its cost is recorded as `pending`, with a message naming the timeout. -`sqb cost ` collects it later with a 5-minute budget, which completes long runs. Any other -failure is recorded as `collection_failed` with the Snowflake error that caused it. - -These values are attributed compute credits and estimated cost, not Snowflake-billed credits or -invoice reconciliation. The estimate uses only query history visible to the executing role and does -not reconstruct invisible concurrent work, warehouse resume or idle tail, the 60-second minimum, -cloud-services credits, contract adjustments, or multi-cluster billing. A statement run with a `QUERY_TAG` -of your own is not recognised as a metadata-only `CREATE`, so it may report missing warehouse -metadata. Run metadata and query IDs -are stored under `target/executions//`; that statement ledger stores only an SQL digest, -not SQL text. Executed SQL artifacts are stored separately under the sensitive `target/run/` tree. +The estimate is read from `INFORMATION_SCHEMA.QUERY_HISTORY` when the run finishes. If it can't be +read within 30 seconds, the build is unaffected and its cost is recorded as `pending`; +`sqb cost ` collects it later. Any other failure is recorded as `collection_failed` with the +Snowflake error. + +The estimate is attributed compute, not your Snowflake bill. It sees only query history visible to +the executing role, and leaves out warehouse resume and idle time, the 60-second minimum, +cloud-services credits, contract adjustments and multi-cluster billing. + +A statement run with your own `QUERY_TAG` may report missing warehouse metadata. Executed SQL is +kept under `target/run/`, which can hold sensitive data; the cost records under +`target/executions/` store only a digest of each statement. ## Functions in unit tests `sqb test` creates each UDF and table function the selected tests call, SQL or Python, as a `TEMPORARY FUNCTION __SQB_TEST__` in the session's current schema. The session needs a current schema and permission to create functions there; otherwise tests call the built function. See -[Functions in `sqb test`](../testing.md#functions-in-sqb-test). +[Functions in `sqb test`](../testing-reference.md#functions-in-sqb-test). diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/sqlserver.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/sqlserver.md index 3ba3f8f00c..8c213358ca 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/sqlserver.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/sqlserver.md @@ -84,4 +84,4 @@ schema = "dev" ## Functions in unit tests SQL Server has no session-scoped functions, so `sqb test` calls the built function. Run `sqb build` -before testing a new or changed function. See [Functions in `sqb test`](../testing.md#functions-in-sqb-test). +before testing a new or changed function. See [Functions in `sqb test`](../testing-reference.md#functions-in-sqb-test). diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/audits.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/audits.md index a1495fdc8e..7d39789499 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/audits.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/audits.md @@ -301,7 +301,7 @@ runs the audit exactly where any other audit on that target runs: promotion, a `final` audit checks the staged table before promotion; with `immediate` promotion, or for final-phase audits on incremental and snapshot models, it runs after the write, like any other audit there. See [Run scope](#run-scope) and - [Table promotion mode](project-configuration.md#table-promotion-mode). + [Table promotion mode](models/materializations.md#table). - On a source or seed, it runs before any model that depends on it is built. An `error` failure blocks everything downstream of the target, and where the target's other audits diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/column-lineage.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/column-lineage.md index 592d0cb948..f2f73c41bc 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/column-lineage.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/column-lineage.md @@ -26,7 +26,7 @@ Online: https://sqlbuild.com/docs/concepts/column-lineage/ ## How it works -SQLBuild analyzes column lineage statically at compile time. No warehouse connection is needed. The analyzer parses each model's SQL, resolves `ref()` and `source()` calls, and traces columns through `SELECT` lists, CTEs, JOINs, subqueries, and expressions. +SQLBuild analyzes column lineage statically at compile time. No warehouse connection is needed. The analyzer parses each model's SQL, resolves `__ref()` and `__source()` calls, and traces columns through `SELECT` lists, CTEs, JOINs, subqueries, and expressions. Column lineage requires SQL analysis to be enabled in project settings (it is by default). diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/overview.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/overview.md index e0ff7236bb..6c2095f878 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/overview.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/overview.md @@ -82,37 +82,8 @@ Each step calls the `dbt` CLI directly: `dbt compile` for the manifest, `dbt ls` ### Flags -`sqb dbt plan/run/build` declare the common flags directly. Anything declared goes **before** a `--` separator; any other raw dbt flag goes **after** it and is forwarded verbatim. A flag placed on the wrong side errors rather than silently reaching dbt. - -```bash -sqb dbt build --select path:models/marts --full-refresh -``` - -Declared flags are routed to the right place automatically: - -| Flag | Routed to | Notes | -| ---- | --------- | ----- | -| `--select` / `--exclude` | dbt | dbt resolves selection (see [Selection](selection.md)). | -| `--vars` | dbt **and** SQLBuild | The same vars feed dbt's compile and SQLBuild's own variable resolution, so both sides see identical values. | -| `--full-refresh` | dbt **and** SQLBuild | Requests a full rebuild on both sides. dbt and native SQLBuild models independently apply their model-level `full_refresh` setting, including `false` opt-outs. | -| `--threads` | dbt (`--threads`) and SQLBuild (`--concurrency`) | | -| `--target` / `--project-dir` / `--profiles-dir` / `--profile` / `--target-path` | dbt | Standard dbt locators. | - -#### `--vars` - -`--vars` accepts the same JSON object dbt accepts, and SQLBuild passes it to **both** the underlying `dbt` invocation and its own variable resolution. That means a value referenced as `@@my_var` in SQLBuild model SQL (see [Interpolation](../interpolation.md)) and as `{{ var('my_var') }}` in a dbt model both resolve to the value you passed. CLI vars take precedence over project and local config vars. You do not need to declare vars twice. - -```bash -sqb dbt build --select path:models/marts --vars '{"my_var": 1}' -``` - -#### Forwarding other dbt flags - -For any native dbt flag SQLBuild does not declare, put it after `--` and it is passed straight to the `dbt` invocation untouched: - -```bash -sqb dbt build -- --log-level debug -``` +`sqb dbt` accepts the common dbt flags, such as `--select`, `--vars` and `--full-refresh`, and passes +any other dbt flag after `--`. See [`sqb dbt`](../../cli/dbt.md#flags) for how each flag is routed. ## Configuration diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes.md index ff91087e21..7f3596a780 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes.md @@ -9,7 +9,7 @@ Online: https://sqlbuild.com/docs/concepts/declaration-scopes/ ## Contents - Example -- Terms used in text output +- Labels in `sqb scope` output - Choose a location - Explore the feature @@ -96,11 +96,11 @@ the owner folder, or compilation fails; they never change where the declaration Declarations do not flow upward or sideways into sibling directories. -## Terms used in text output +## Labels in `sqb scope` output The CLI's text output uses short labels for the same four rules: -| Canonical label | Plain-language meaning | +| Label | Meaning | |-----------------|------------------------| | `project` | Available throughout the project | | `descendant-public` | Available in this folder and folders below it | @@ -111,11 +111,6 @@ Here, an **owner folder** is simply the folder directly containing a model, test other authored resource. The [Scope Explorer](declaration-scopes/explorer.md) shows the owner folder, its parents, and project-wide declarations separately. -JSON keeps the underlying enum values for integrations. Its `scope` field uses `global`, -`inherited`, `local`, or `private`, while derived `visibility` values use underscores, such as -`descendant_public` and `exact_owner_private`. Consumers should use the fields documented in the -versioned JSON schema rather than parsing text labels. - ## Choose a location | Where the value is needed | Placement | @@ -141,5 +136,5 @@ top-level roles as a genuine project API instead of a neutral dumping ground. Learn the features themselves in [Enums](enums.md), [Constants](constants.md), [Macros](macros.md), [Audits](audits.md), -[Reusable Schemas](models/schemas.md), and [Hooks](models/hooks.md). See [Interpolation](interpolation.md) for project, +[Reusable column schemas](models/schemas.md), and [Hooks](models/hooks.md). See [Variables and placeholders](interpolation.md) for project, environment, and runtime context values, which are separate from declarations. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/explorer.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/explorer.md index 6a8484b376..1fc3f76411 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/explorer.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/explorer.md @@ -335,5 +335,10 @@ Use `--json` for editor integrations and repository tooling. Output has a versio ordering, filters, pagination, and move-preview results. Constant values, credentials, and secret connection settings are not included. +In JSON, the `scope` field uses `global`, +`inherited`, `local`, or `private`, while derived `visibility` values use underscores, such as +`descendant_public` and `exact_owner_private`. Consumers should use the fields documented in the +versioned JSON schema rather than parsing text labels. + Review the directory rules behind the report. See all selectors, filters, pagination options, output sections, and JSON behavior. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/diff.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/diff.md index 3f2f8c2245..36f118de52 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/diff.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/diff.md @@ -400,4 +400,5 @@ reconciliation, inspection, materialization, comparison, cleanup, and total dura ## Exit codes -In model mode, `sqb diff` returns exit code `0` when all selected models have no differences, `1` when any model has schema or row differences, and `2` when the size guard stopped a default full diff before reading data. Raw-query mode uses the exit codes in [Query output and exit codes](#query-output-and-exit-codes). This makes it usable in CI pipelines as a validation gate. +`sqb diff` exits `1` when it finds differences, so it works as a CI gate. See +[`sqb diff`](../cli/diff.md#exit-codes) for every exit code. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/enums-and-constants.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/enums-and-constants.md deleted file mode 100644 index ba4b4207a6..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/enums-and-constants.md +++ /dev/null @@ -1,52 +0,0 @@ - - -# Enums and Constants Have Moved - -> Find the new focused guides for enums, constants, collections, contracts, and private values. - -Online: https://sqlbuild.com/docs/concepts/enums-and-constants/ - -Enums and constants now have separate guides so each feature is easier to find and learn. This page -remains available for existing links and bookmarks but is not part of the main navigation. - - Define fixed string or integer domains and reference validated members. - Define reusable scalar values and reference them safely from SQL. - Work with lists, sets, objects, value lists, and native arrays. - Keep enums and constants inside one model. - -## Public declarations - -See [Enums](enums.md) and [Constants](constants.md) for project-wide declarations. - -## Constant values - -See [Constants](constants.md#scalar-values) for scalars and -[Collections and Rendering](constants/collections-and-rendering.md) for collections. - -## References and rendering - -See [Collections and Rendering](constants/collections-and-rendering.md). - -## Adapter matrix - -See the [native-array rendering matrix](constants/collections-and-rendering.md#native-array-rendering). - -## Rendering configuration - -See [Project default](constants/collections-and-rendering.md#project-default). - -## Model-local declarations - -See [Model-Private Values](model-private-values.md). - -## Model-private declarations - -See [Model-Private Values](model-private-values.md). - -## Enum validation - -See [Enum validation rules](enums.md#validation-rules). - -## Enum-typed contracts - -See [Enum Model Contracts](enums/model-contracts.md). diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/incremental.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/incremental.md index ac86af1841..0846575b22 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/incremental.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/incremental.md @@ -14,11 +14,24 @@ Online: https://sqlbuild.com/docs/concepts/incremental/ - Microbatch execution - Replay on change -Incremental models process only new or changed data instead of rebuilding the entire table. In the -default sequential path, SQLBuild works out where to resume from the current target and input -relations with no separate checkpoint store. A retry recomputes its interval from current warehouse -state rather than reusing the exact interval of a failed attempt. Opt-in concurrent microbatching -adds immutable coordination facts as described below. +An incremental model processes only new or changed rows on each build instead of rebuilding the +whole table. Pick a strategy and a cursor column, and SQLBuild works out where to resume from the +table itself, with no separate checkpoint store: + +```sql +MODEL ( + description "Order activity per hour", + materialized incremental, + incremental_strategy delete_insert, + cursor activity_hour, + cursor_type timestamp, + cursor_grain hour, + unique_key [activity_hour], +); +``` + +A retry works out its range again from the warehouse. [Concurrent microbatches](#concurrent-batches) are the exception: +they record each batch to coordinate. ## Strategies diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/interpolation.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/interpolation.md index b5e882f520..6b54363cfe 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/interpolation.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/interpolation.md @@ -1,6 +1,6 @@ -# Interpolation +# Variables and placeholders > How SQLBuild processes variables, context, and dynamic content in SQL and config. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/sql-macro-reference.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/sql-macro-reference.md new file mode 100644 index 0000000000..0eb7afd699 --- /dev/null +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/sql-macro-reference.md @@ -0,0 +1,221 @@ + + +# SQL macro reference + +> Literal positions, relation slot reads and mappings, call checks and result types for SQL macros. + +Online: https://sqlbuild.com/docs/concepts/macros/sql-macro-reference/ + +## Contents + +- Positions that need a literal +- Reading slots +- Mapping slots at the call +- Rules for groups +- Slot errors +- Checks on every call +- Result types + +The rules behind [SQL macros](sql-macros.md): which positions need a literal, how +relation slots are read and mapped, what is checked at each call, and the result type of each form. + +## Positions that need a literal + +SQL needs a literal in some positions. A parameter without `CONST` there fails with `P023` when the +macro is checked: + +``` +error[P023]: places is used where a literal is required; declare it places INT CONST +``` + +| Position | Adapters | +|---|---| +| `LIMIT`, `OFFSET`, `TOP` and `FETCH FIRST` counts | all | +| `INTERVAL` and `EXTRACT` units | all | +| `PIVOT ... IN (...)` values | all | +| `DATE_TRUNC` unit (the first argument, or the last on BigQuery) | all | +| `ROUND` precision | DuckDB and MotherDuck (for `DECIMAL` values), Databricks | + +The positions every adapter shares are checked for every macro file. `DATE_TRUNC` and `ROUND` are +checked for the project's adapter at each call. + +## Reading slots + +The examples use the `weekly_rates` macro from +[Relation slots](sql-macros.md#relation-slots). + +- `alias.@weekly.order_count` reads the slot from a table alias or CTE, such as `s` in that macro. For a + group, the prefix applies to every column: `s.@weekly.keys` becomes `s.product_id, s.region`. +- A plain read of a group, `@weekly.keys`, always stands as the bare column names, + `product_id, region`. So later CTEs, such as `rated`, read the group from whatever + relation they select from, without an alias. A bare name that two joined relations share fails + as hand-written SQL does, with `B003`; read it as `alias.@weekly.keys` there. +- A plain read of a single-column slot always reads through the passed relation, mapped or not: + `@weekly.refund_count` becomes `weekly.refund_count`, or `weekly.refunds_n` with + `weekly.refund_count := refunds_n`. So the same body expands the same way for every mapping. +- Every read inside `USING (...)` stands unqualified, as `USING` needs, including + `USING (@weekly.week)` for a column the type does not declare. +- In a column-list macro, `o.@weekly.keys` outputs the group's columns under their own names. +- Rules judge a read as you wrote it. `SQLCLEAN028`, which rejects a query mixing qualified and + unqualified columns, judges a query the macro writes whole, such as a CTE of a CTE-list macro, + once, in the macro file. There every plain read counts as unqualified and every + `alias.@weekly...` read as qualified. A step written only in plain reads, such as + `SELECT @weekly.keys, @weekly.refund_count AS refunds FROM @weekly`, passes; to qualify, read + every slot through an alias. +- Where a call's output joins the caller's query, as with expression and column-list macros, the + query is judged at the call. A read through the relation the call passes, such as `o.created_at` in + the `audit_columns` example in [SQL macros](sql-macros.md), fits either style; any other column counts as the macro writes it, so a bare `status` + beside a hand-qualified `o.order_id` fails. + +A plain group read inside a subquery, such as `EXISTS (SELECT @weekly.keys FROM refunds AS r ...)`, +could bind its bare columns to the subquery's relations instead of the one the slot reads. The +call fails with `P017` when a relation of an enclosing subquery, other than the relation passed +for the slot, has one of the group's columns, or when its columns are not known. `LATERAL` +subqueries in `FROM` or `JOIN` are checked the same way: + +``` +error[P017]: slot weekly.keys column product_id would bind to refunds in this subquery; write alias.@weekly.keys +``` + +Read the group through its alias there. A group read in a CTE, or in a derived table in `FROM`, +reads that query's own relations and is not checked. + +`alias.@weekly.name` naming no slot, or `alias.@weekly` naming none at all, fails with `P022`. +`@weekly.name` without an alias may still read a column the type does not declare, through the +passed relation, as before. + +## Mapping slots at the call + +A call maps slots with `relation.slot := ...`, next to its other named arguments: + +```sql +WITH weekly AS (SELECT * FROM __ref("weekly_orders")), +@sql.weekly_rates(weekly, weekly.keys := (product_id), weekly.order_count := orders_placed, __cte := by_product) +@sql.weekly_rates(weekly, weekly.keys := (product_id, region), weekly.order_count := orders_placed, __cte := by_product_region) +final AS (SELECT * FROM by_product_region) +SELECT * FROM final +``` + +The first call compiles to the same SQL as a macro written for `product_id`, the second groups by +product and region. The left of `:=` is always a parameter path, and the right side is column +names of the passed relation: + +- A single-column slot takes one column name. An unmapped single-column slot reads the column of + its own name, so `refund_count` above reads `refund_count`. +- A group takes a parenthesised list of column names, such as `(product_id, region)`, and every + call must map it. +- A slot reads a column as it is: compute an expression in the relation you pass. +- A macro that passes its own relation on maps the inner slots to its slot names, as in + `@sql.weekly_rates(weekly := @src, weekly.keys := (dims), weekly.order_count := placed)`, or to + columns of its relation. Each inner slot keeps its kind: a group mapped to columns is still read + bare, or after its alias, and mapping a group to a single-column slot of the outer relation, or + the reverse, fails with `P017`. + +## Rules for groups + +1. A group is pasted verbatim, comma-separated. It takes no per-item expression, indexing, + counting or renaming, and groups are never combined. +2. A group stands only as a whole item of a select list, `PARTITION BY`, `GROUP BY`, `ORDER BY` + (without `ASC`, `DESC` or `NULLS`) or `USING (...)`. Anywhere else fails with `P015` at the + macro file. +3. Grouped columns keep their real names in the output. +4. A group is never empty: `weekly.keys := ()` fails. +5. A relation may have several groups, such as `keys COLUMNS, carried COLUMNS`. Groups of one + relation may not share a column, and a column that fills a single-column slot, mapped or by its + own name, may not also be in a group. + +## Slot errors + +Slot errors fail with `P017` and name the slot and the caller's relation: + +``` +error[P017]: slot weekly.order_count needs a BIGINT column; weekly has no column orders_total +``` + +- Every mapped and grouped column must exist in the passed relation, and a single-column slot needs + a compatible type: `slot weekly.order_count needs a BIGINT column; weekly.region is VARCHAR`, + or `slot weekly.keys groups column channel; weekly has no column channel`. Like other argument + checks, these run when the relation is a CTE or table alias; the model's own column checks still + run on the expanded SQL. +- Overlapping groups, or a grouped column that also fills a single-column slot, fail at the call. +- A grouped column named like a column the macro defines, or like a bare column such as `s.week` + or a column read through the relation such as `@weekly.region` beside the group in the same + select list, fails with `P018`, naming both: + `column refunds of slot weekly.keys clashes with column refunds that macro weekly_rates() defines`. +- A mapped single-column slot read in a select list needs an `AS` in the macro, so output names + don't depend on the caller's mapping. Groups are exempt, because they keep their real names, and + so is an unmapped slot, which keeps its own name. So are reads in `IN`, `EXISTS` and scalar + subqueries, whose column names never reach the macro's output. A `LATERAL` subquery in `FROM` or + `JOIN` is checked like a derived table, because its columns reach the output, as through `l.*`. + +## Checks on every call + +Each macro is checked on its own when the project compiles, and each call is checked against the +caller's column types, inferred like the rest of the model: + +- a `RELATION` argument must have every declared column, each of a compatible type, and every + column its [slots](#slot-errors) are mapped to; +- a typed argument must have a compatible type, so `@sql.cents_to_dollars(o.status)` with a + `VARCHAR` status fails. + +These fail with `P017`, naming the argument, the expected type or column and what the caller has: + +``` +error[P017]: argument src of macro clean_orders() needs column is_deleted BOOLEAN; orders_raw has no column is_deleted + --> models/marts/orders_clean.sql:2:1 + parameter declared here: + --> macros/sql/clean_orders.sql:3:14 + = help: pass a relation with these columns: order_id BIGINT, updated_at TIMESTAMP, is_deleted BOOLEAN +``` + +An argument whose type the compiler cannot infer passes. So do calls inside a subquery or in a +query part with `UNION`, `INTERSECT` or `EXCEPT`, arguments built from `__ref()`, macro calls or +parameters of a calling macro, and calls in models with runtime placeholders; the model's own +column checks still run on the expanded SQL. All other errors also point at the call and at the +macro file. + +## Result types + +An expression macro call is typed wherever it stands, from its expanded SQL, as if the SQL were +written there. `returns` declares the type for when that SQL's type is unknown: + +```sql +MACRO (description "Status bar", arguments (value VARCHAR), returns VARCHAR); + +bar(LENGTH(@value), 0, 10) +``` + +- The declared type applies only where the expanded SQL's type is unknown, including a list + literal such as `[]`, whose type analysis doesn't infer. A known type stands, even a close one such as `VARCHAR(100)` for + `returns VARCHAR(5)`, and contract checks report a mismatch with `K002`. +- The type flows on as an inferred type would: through CTEs, `SELECT *`, subqueries, joins and each + branch of a set operation, through operators and functions around the call, to contract checks + (`K002`, `K003`), `SQBRCONTRACT105` and models that read the column. +- `returns` is used only for analysis. It doesn't appear in the generated SQL, so on its own it doesn't + change the column's physical type. A body of `'ACTIVE'` still creates `VARCHAR(6)` on Snowflake. + For `SQBRCONTRACT105`, a select-list item that is one call needs no `CAST` at the call only when + the macro body's outermost expression is a `CAST(... AS type)` or `...::type` to the column's + contract type, compared as the dialect does, such as `CAST(LEFT(@value, 3) AS VARCHAR)` for a + `VARCHAR` column. Otherwise the rule asks for a cast in the macro or at the call, whether the + macro declares a fixed `returns` or `returns TYPE OF`. A call inside a larger expression always + needs an outer cast. +- Without `returns`, the call is typed by inference alone. When an output is one call whose type is + unknown, `K003` and `SQBRCONTRACT105` suggest declaring `returns` on the macro. +- `returns TYPE OF ` declares the type of that argument at each call, under the same + rule: `arguments (value ANY NUMERIC, places INT CONST DEFAULT 2), returns TYPE OF value`. If the + argument's own type is unknown, so is the call's. The argument must be a scalar argument of the + macro (`P015`). +- A fixed `returns` is checked against the body at the macro file. The body is typed from the + declared argument types; a known type of another family, such as `UPPER(@value)` with + `returns BIGINT`, fails with `P024`. A body whose type cannot be inferred, for example because it + reads `ANY` arguments, is a list literal, or calls a function whose type is unknown or only + guessed, trusts `returns`. +- `returns` and every argument type must be a type the adapter's SQL dialect accepts, including a + decimal precision and scale it can hold, such as at most `DECIMAL(38, s)` on DuckDB (`P015`). +- A call where only a keyword can stand, such as the date part of `EXTRACT`, is typed by inference + alone. +- A call inside a Python macro argument, as in `@py.bracket(@sql.status_bar(o.status))`, is typed + by inference alone. +- `returns` never changes the compiled SQL. + +A `* REPLACE` item takes its source column's type rather than the call's. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/sql-macros.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/sql-macros.md index 276c80230f..db5b45f30c 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/sql-macros.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/sql-macros.md @@ -14,9 +14,8 @@ Online: https://sqlbuild.com/docs/concepts/macros/sql-macros/ - Parameters in the body - Relation slots - Compose macros -- Checks -- Result types - Tools +- Reference A SQL macro is a reusable piece of SQL with typed parameters. Unlike a [Python macro](python-macros.md), it holds no logic: its body is SQL, so the compiler checks @@ -76,7 +75,7 @@ CAST(ROUND(@amount_cents / 100, @places) AS DECIMAL(18, 2)) ``` `returns` is optional for expression macros and not allowed for the other forms. See -[Result types](#result-types) for what it does. +[Result types](sql-macro-reference.md#result-types) for what it does. The description follows the same rules as every header string: write text that contains double quotes as `"""..."""`, and a wrapped description is stored folded into one flowing text (see @@ -254,26 +253,6 @@ These fail with `P022`: Parameters may not be named `sql`, `py`, `const`, `enum`, `param` or `__cte`. -### Positions that need a literal - -SQL needs a literal in some positions. A parameter without `CONST` there fails with `P023` when the -macro is checked: - -``` -error[P023]: places is used where a literal is required; declare it places INT CONST -``` - -| Position | Adapters | -|---|---| -| `LIMIT`, `OFFSET`, `TOP` and `FETCH FIRST` counts | all | -| `INTERVAL` and `EXTRACT` units | all | -| `PIVOT ... IN (...)` values | all | -| `DATE_TRUNC` unit (the first argument, or the last on BigQuery) | all | -| `ROUND` precision | DuckDB and MotherDuck (for `DECIMAL` values), Databricks | - -The positions every adapter shares are checked for every macro file. `DATE_TRUNC` and `ROUND` are -checked for the project's adapter at each call. - ## Relation slots Everything a macro reads from a relation is declared in its `RELATION` type as a slot. A slot is @@ -307,112 +286,6 @@ rated AS ( A `COLUMNS` slot is untyped: each column passed to it keeps its own type, so the expanded SQL is fully typed. -### Reading slots - -- `alias.@weekly.order_count` reads the slot from a table alias or CTE, such as `s` above. For a - group, the prefix applies to every column: `s.@weekly.keys` becomes `s.product_id, s.region`. -- A plain read of a group, `@weekly.keys`, always stands as the bare column names, - `product_id, region`. So later CTEs, such as `rated` above, read the group from whatever - relation they select from, without an alias. A bare name that two joined relations share fails - as hand-written SQL does, with `B003`; read it as `alias.@weekly.keys` there. -- A plain read of a single-column slot always reads through the passed relation, mapped or not: - `@weekly.refund_count` becomes `weekly.refund_count`, or `weekly.refunds_n` with - `weekly.refund_count := refunds_n`. So the same body expands the same way for every mapping. -- Every read inside `USING (...)` stands unqualified, as `USING` needs, including - `USING (@weekly.week)` for a column the type does not declare. -- In a column-list macro, `o.@weekly.keys` outputs the group's columns under their own names. -- Rules judge a read as you wrote it. `SQLCLEAN028`, which rejects a query mixing qualified and - unqualified columns, judges a query the macro writes whole, such as a CTE of a CTE-list macro, - once, in the macro file. There every plain read counts as unqualified and every - `alias.@weekly...` read as qualified. A step written only in plain reads, such as - `SELECT @weekly.keys, @weekly.refund_count AS refunds FROM @weekly`, passes; to qualify, read - every slot through an alias. -- Where a call's output joins the caller's query, as with expression and column-list macros, the - query is judged at the call. A read through the relation the call passes, such as `o.created_at` - above, fits either style; any other column counts as the macro writes it, so a bare `status` - beside a hand-qualified `o.order_id` fails. - -A plain group read inside a subquery, such as `EXISTS (SELECT @weekly.keys FROM refunds AS r ...)`, -could bind its bare columns to the subquery's relations instead of the one the slot reads. The -call fails with `P017` when a relation of an enclosing subquery, other than the relation passed -for the slot, has one of the group's columns, or when its columns are not known. `LATERAL` -subqueries in `FROM` or `JOIN` are checked the same way: - -``` -error[P017]: slot weekly.keys column product_id would bind to refunds in this subquery; write alias.@weekly.keys -``` - -Read the group through its alias there. A group read in a CTE, or in a derived table in `FROM`, -reads that query's own relations and is not checked. - -`alias.@weekly.name` naming no slot, or `alias.@weekly` naming none at all, fails with `P022`. -`@weekly.name` without an alias may still read a column the type does not declare, through the -passed relation, as before. - -### Mapping slots at the call - -A call maps slots with `relation.slot := ...`, next to its other named arguments: - -```sql -WITH weekly AS (SELECT * FROM __ref("weekly_orders")), -@sql.weekly_rates(weekly, weekly.keys := (product_id), weekly.order_count := orders_placed, __cte := by_product) -@sql.weekly_rates(weekly, weekly.keys := (product_id, region), weekly.order_count := orders_placed, __cte := by_product_region) -final AS (SELECT * FROM by_product_region) -SELECT * FROM final -``` - -The first call compiles to the same SQL as a macro written for `product_id`, the second groups by -product and region. The left of `:=` is always a parameter path, and the right side is column -names of the passed relation: - -- A single-column slot takes one column name. An unmapped single-column slot reads the column of - its own name, so `refund_count` above reads `refund_count`. -- A group takes a parenthesised list of column names, such as `(product_id, region)`, and every - call must map it. -- A slot reads a column as it is: compute an expression in the relation you pass. -- A macro that passes its own relation on maps the inner slots to its slot names, as in - `@sql.weekly_rates(weekly := @src, weekly.keys := (dims), weekly.order_count := placed)`, or to - columns of its relation. Each inner slot keeps its kind: a group mapped to columns is still read - bare, or after its alias, and mapping a group to a single-column slot of the outer relation, or - the reverse, fails with `P017`. - -### Rules for groups - -1. A group is pasted verbatim, comma-separated. It takes no per-item expression, indexing, - counting or renaming, and groups are never combined. -2. A group stands only as a whole item of a select list, `PARTITION BY`, `GROUP BY`, `ORDER BY` - (without `ASC`, `DESC` or `NULLS`) or `USING (...)`. Anywhere else fails with `P015` at the - macro file. -3. Grouped columns keep their real names in the output. -4. A group is never empty: `weekly.keys := ()` fails. -5. A relation may have several groups, such as `keys COLUMNS, carried COLUMNS`. Groups of one - relation may not share a column, and a column that fills a single-column slot, mapped or by its - own name, may not also be in a group. - -### Checks at the call - -Slot errors fail with `P017` and name the slot and the caller's relation: - -``` -error[P017]: slot weekly.order_count needs a BIGINT column; weekly has no column orders_total -``` - -- Every mapped and grouped column must exist in the passed relation, and a single-column slot needs - a compatible type: `slot weekly.order_count needs a BIGINT column; weekly.region is VARCHAR`, - or `slot weekly.keys groups column channel; weekly has no column channel`. Like other argument - checks, these run when the relation is a CTE or table alias; the model's own column checks still - run on the expanded SQL. -- Overlapping groups, or a grouped column that also fills a single-column slot, fail at the call. -- A grouped column named like a column the macro defines, or like a bare column such as `s.week` - or a column read through the relation such as `@weekly.region` beside the group in the same - select list, fails with `P018`, naming both: - `column refunds of slot weekly.keys clashes with column refunds that macro weekly_rates() defines`. -- A mapped single-column slot read in a select list needs an `AS` in the macro, so output names - don't depend on the caller's mapping. Groups are exempt, because they keep their real names, and - so is an unmapped slot, which keeps its own name. So are reads in `IN`, `EXISTS` and scalar - subqueries, whose column names never reach the macro's output. A `LATERAL` subquery in `FROM` or - `JOIN` is checked like a derived table, because its columns reach the output, as through `l.*`. - ## Compose macros A SQL macro may call other SQL macros, and pass its own parameters on as `@name`: @@ -440,78 +313,6 @@ isn't visible from there fails with `S026`; to use a caller's value, make it an `@const(...)` at the call site. SQL macros a body calls resolve the same way, so a macro can call a folder-local macro beside it that its callers cannot see. -## Checks - -Each macro is checked on its own when the project compiles, and each call is checked against the -caller's column types, inferred like the rest of the model: - -- a `RELATION` argument must have every declared column, each of a compatible type, and every - column its [slots](#checks-at-the-call) are mapped to; -- a typed argument must have a compatible type, so `@sql.cents_to_dollars(o.status)` with a - `VARCHAR` status fails. - -These fail with `P017`, naming the argument, the expected type or column and what the caller has: - -``` -error[P017]: argument src of macro clean_orders() needs column is_deleted BOOLEAN; orders_raw has no column is_deleted - --> models/marts/orders_clean.sql:2:1 - parameter declared here: - --> macros/sql/clean_orders.sql:3:14 - = help: pass a relation with these columns: order_id BIGINT, updated_at TIMESTAMP, is_deleted BOOLEAN -``` - -An argument whose type the compiler cannot infer passes. So do calls inside a subquery or in a -query part with `UNION`, `INTERSECT` or `EXCEPT`, arguments built from `__ref()`, macro calls or -parameters of a calling macro, and calls in models with runtime placeholders; the model's own -column checks still run on the expanded SQL. All other errors also point at the call and at the -macro file. - -## Result types - -An expression macro call is typed wherever it stands, from its expanded SQL, as if the SQL were -written there. `returns` declares the type for when that SQL's type is unknown: - -```sql -MACRO (description "Status bar", arguments (value VARCHAR), returns VARCHAR); - -bar(LENGTH(@value), 0, 10) -``` - -- The declared type applies only where the expanded SQL's type is unknown, including a list - literal such as `[]`, whose type analysis doesn't infer. A known type stands, even a close one such as `VARCHAR(100)` for - `returns VARCHAR(5)`, and contract checks report a mismatch with `K002`. -- The type flows on as an inferred type would: through CTEs, `SELECT *`, subqueries, joins and each - branch of a set operation, through operators and functions around the call, to contract checks - (`K002`, `K003`), `SQBRCONTRACT105` and models that read the column. -- `returns` is used only for analysis. It doesn't appear in the generated SQL, so on its own it doesn't - change the column's physical type. A body of `'ACTIVE'` still creates `VARCHAR(6)` on Snowflake. - For `SQBRCONTRACT105`, a select-list item that is one call needs no `CAST` at the call only when - the macro body's outermost expression is a `CAST(... AS type)` or `...::type` to the column's - contract type, compared as the dialect does, such as `CAST(LEFT(@value, 3) AS VARCHAR)` for a - `VARCHAR` column. Otherwise the rule asks for a cast in the macro or at the call, whether the - macro declares a fixed `returns` or `returns TYPE OF`. A call inside a larger expression always - needs an outer cast. -- Without `returns`, the call is typed by inference alone. When an output is one call whose type is - unknown, `K003` and `SQBRCONTRACT105` suggest declaring `returns` on the macro. -- `returns TYPE OF ` declares the type of that argument at each call, under the same - rule: `arguments (value ANY NUMERIC, places INT CONST DEFAULT 2), returns TYPE OF value`. If the - argument's own type is unknown, so is the call's. The argument must be a scalar argument of the - macro (`P015`). -- A fixed `returns` is checked against the body at the macro file. The body is typed from the - declared argument types; a known type of another family, such as `UPPER(@value)` with - `returns BIGINT`, fails with `P024`. A body whose type cannot be inferred, for example because it - reads `ANY` arguments, is a list literal, or calls a function whose type is unknown or only - guessed, trusts `returns`. -- `returns` and every argument type must be a type the adapter's SQL dialect accepts, including a - decimal precision and scale it can hold, such as at most `DECIMAL(38, s)` on DuckDB (`P015`). -- A call where only a keyword can stand, such as the date part of `EXTRACT`, is typed by inference - alone. -- A call inside a Python macro argument, as in `@py.bracket(@sql.status_bar(o.status))`, is typed - by inference alone. -- `returns` never changes the compiled SQL. - -A `* REPLACE` item takes its source column's type rather than the call's. - ## Tools - `sqb format` formats SQL macro files, bodies of every form and the `MACRO` header, and @@ -523,8 +324,13 @@ A `* REPLACE` item takes its source column's type rather than the call's. [When Rules fail](../../cli/compile.md#when-rules-fail). - Editing a macro recompiles its callers. -Test SQL macros with [macro-mode unit tests](../testing.md#macro-tests), and replace one +Test SQL macros with [macro-mode unit tests](../testing-reference.md#macro-tests), and replace one in a model test with a [`__macro__` mock](../testing.md#macro-mocking). See [Macros](../macros.md#choose-the-right-tool) for when to use a SQL macro, a Python macro, a SQL function or a generic audit. + +## Reference + +[SQL macro reference](sql-macro-reference.md) covers positions that need a +literal, reading and mapping relation slots, the checks at each call and result types. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models.md index f8cfb36fc1..9a19b1ad84 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models.md @@ -62,7 +62,7 @@ See [Functions](functions.md) for scalar UDF and table-function references. ## Model guide - [Materializations](models/materializations.md): views, tables, incrementals, snapshots, and custom materializations. -- [Schemas](models/schemas.md): inline columns, reusable schemas, inheritance, audits, and model-local extensions. +- [Reusable column schemas](models/schemas.md): inline columns, reusable schemas, inheritance, audits, and model-local extensions. - [Type Enforcement](models/type-enforcement.md): static type checks and runtime cast behavior by materialization. - [Contracts](models/contracts.md): exact and open output validation, runtime guarantees, nullability, and enum-backed columns. - [Hooks](models/hooks.md): SQL and Python lifecycle hooks. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/column-migrations.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/column-migrations.md index dc077b8caa..abb2054443 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/column-migrations.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/column-migrations.md @@ -11,19 +11,37 @@ Online: https://sqlbuild.com/docs/concepts/models/column-migrations/ - Declaring a rename - Automatic detection - Decisions -- How a rename runs -- State - Adapter support +- Details -`on_schema_change` compares columns by name, so renaming a column looks like one column removed and another added. With the default `append_new_columns`, the new column starts empty and older rows keep their values under the old name; with `sync_all_columns`, the old column and its data are dropped. A column migration renames the column in the warehouse instead, so every row keeps its value under the new name, with no leftover column and no replay. +Rename a column with `sqb rename fact_orders.amount revenue`. On the next build, SQLBuild renames +the column in the warehouse table, so every row keeps its value under the new name and nothing is +replayed. -Column migrations apply to incremental models, including microbatch models, and snapshot models. Tables and views are rebuilt on every change and don't need them. +```bash +sqb rename fact_orders.amount revenue +sqb plan +``` -## Declaring a rename +```text +Column migrations (1) +└── fact_orders migrate columns + └── amount -> revenue rename in place +``` -[`sqb rename .`](../../cli/rename.md) renames the column in the model and every query that reads it, and adds the `migrate_from` declaration below for you. +Column migrations apply to incremental models, including microbatch models, and snapshot models. +Tables and views are rebuilt on every build, so they don't need them. -Add `migrate_from` to the renamed column in the model header, naming the old column: +Without a column migration, `on_schema_change` compares columns by name and sees one column removed +and another added. With the default `append_new_columns`, the new column starts empty and older rows +keep their values under the old name. With `sync_all_columns`, the old column and its data are +dropped. + +## Declaring a rename + +[`sqb rename .`](../../cli/rename.md) renames the column in the model and in every query +that reads it, and writes a `migrate_from` declaration on the renamed column. You can also write the +declaration yourself: ```sql MODEL ( @@ -41,30 +59,45 @@ SELECT order_id, order_date, amount AS revenue FROM __source("raw_orders") ``` -On the next build, SQLBuild renames `amount` to `revenue` in the existing table, records the rename, and then builds the model incrementally, so the new delta lands in `revenue`. +On the next build, SQLBuild renames `amount` to `revenue` in the existing table, records the rename, +and then builds the model incrementally, so the new delta lands in `revenue`. -A column entry may contain only `migrate_from`, or combine it with `type`, `nullable`, `description`, and `audits`. When the model binds a reusable schema with `model_schema`, an inherited column may add `migrate_from` without redeclaring its metadata. +Once the rename is recorded, `migrate_from` has no further effect, and SQLBuild reports (`M108`) +that it can be removed. Keep it until every target that needs the rename has built. On a target +where the table doesn't exist yet, the declaration does nothing and the table is created with the +new name. -After the rename is recorded, `migrate_from` has no further effect, and SQLBuild reports (`M108`) that it can be removed. Keep it until every target that needs the rename has built. On a target where the table doesn't exist yet, the declaration does nothing: the table is created with the new column name. +A column entry can hold `migrate_from` on its own or together with `type`, `nullable`, +`description` and `audits`. A column inherited from a [reusable schema](schemas.md) +can add `migrate_from` without repeating its metadata. -A declaration is rejected when compiling if: +Compilation rejects a declaration when: -- the model is not incremental or snapshot, unless it is a table or view model that declares its own `migrate_from`; there the column's `migrate_from` only names the old column in the [compatibility view](migrations.md#old-names) at the model's old name; -- `migrate_from` names the column itself, or a qualified name instead of one column; -- two columns declare the same `migrate_from`; -- a declared column migrates from another column that itself declares `migrate_from`. Chains and swaps can't be renamed in place safely; rename through a name no other column uses, or rebuild with `--full-refresh`. +- `migrate_from` names the column itself, or a qualified name instead of one column. +- Two columns declare the same `migrate_from`. +- A column migrates from another column that declares its own `migrate_from`. Chains and swaps + can't be renamed in place safely. Rename through a name no other column uses, or rebuild with + `--full-refresh`. +- The model isn't incremental or snapshot, unless the model itself declares `migrate_from` (see + below). -## Automatic detection +On a table or view model that declares its own model-level `migrate_from`, a column's `migrate_from` +only names the old column in the [old-name view](migrations.md#old-names). -A pure rename is detected without a declaration: exactly one column that disappeared from the model has exactly the same expression as exactly one new column, for example `amount` becoming `amount AS revenue`, or `ROUND(amount, 2) AS revenue` becoming `ROUND(amount, 2) AS net_revenue`. Expressions are compared ignoring formatting, comments, and keyword and unquoted identifier case. Detection compares the query the table was last built from with the current query, so it applies to the build that makes the change. +## Automatic detection -A rename is only automatic when it is the whole change. Apart from the renamed output names, the query must be the same: the same CTEs, `FROM`, joins, filters, grouping, and other columns. References to the new name in `ORDER BY` or `QUALIFY` count as references to the old name only when SQLBuild can prove they bind to the output column: the columns of every relation the query reads must be known, from the inspected warehouse tables of its sources and upstream models or from CTEs and subqueries with named columns, the new name must not be an input column, and the old name must either not be an input column or be passed through unchanged. Otherwise the change is not treated as a rename, because engines differ on whether such a name refers to the output alias or the input column. The output columns may be reordered unless the query groups, orders, or uses `DISTINCT ON` by position. If anything else changed, an unchanged expression such as `amount` may now read different data, for example from a CTE that redefines `amount`, so nothing is renamed automatically and the change follows `on_schema_change`. Declare `migrate_from` to rename the column anyway. +SQLBuild detects a rename by itself when renames are the only change to the query: each removed +column has exactly one new column with the same expression. For example, `amount` becomes +`amount AS revenue`. If you changed anything else in the query, declare `migrate_from`, or use +`sqb rename`. -Detection never guesses: +Detection never guesses. If more than one removed or new column has the same expression, nothing is +renamed automatically. Two columns that swap names keep their names. Automatically detected renames +are marked `(automatic)` in the plan, and they never stop a build: when one can't be applied, the +columns follow `on_schema_change`. -- Columns are matched by expression, not by position. Two columns renamed at once each pair with the column whose expression they kept; two columns that swap names keep their names and are not renamed. -- If more than one removed column has the same expression, or more than one new column does, nothing is migrated automatically. -- A near match, where the expressions reference the same columns but differ or the rest of the query changed, for example `amount` becoming `ROUND(amount, 2) AS revenue`, is not a rename. It follows the model's `on_schema_change` policy, and `sqb plan` shows a hint next to the added column until the change has been built: +A near match, where the expression changed, is not a rename. It follows `on_schema_change`, and +`sqb plan` suggests the declaration until the change has been built: ```text ├── schema diff: @@ -73,59 +106,37 @@ Detection never guesses: - amount (removed) ``` -Add the suggested `migrate_from` to keep the history. A declared rename with a changed expression or other query changes is still renamed in place, but it is a query change, so `replay_on_change` decides how much history is rebuilt, exactly as it would without the rename. +A declared rename with other query changes still renames the column in place. The other changes are +a query change, so the model's `replay_on_change` decides how much history is rebuilt, as it would +without the rename. ## Decisions -`sqb plan` and `sqb build` list column migrations before the models they change: - -```text -Column migrations (1) -└── fact_orders migrate columns - └── amount -> revenue rename in place -``` - -Automatically detected renames are marked `(automatic)`. The decision is based on the table's current columns first, then on recorded renames: +`sqb plan` and `sqb build` list column migrations before the models they change. The decision is +based on the table's current columns first, then on recorded renames: | Decision | Meaning | |----------|---------| | `rename in place` | The old column exists and the new one doesn't. Rename it. | -| `already renamed, record` | The new column exists and the old one doesn't, but no rename is recorded, which is what an interrupted rename leaves behind. Record it without renaming again. | +| `already renamed, record` | The new column exists, the old one doesn't, and no rename is recorded. An interrupted rename leaves this. Record it without renaming again. | | `done` | The rename is recorded and the new column exists. Nothing to do. | -| `conflict` | Both columns exist. The build stops (`M110`); drop one of them, or remove `migrate_from` and let `on_schema_change` handle the columns. | -| `source missing` | Neither column exists and no rename is recorded. The plan warns (`M109`) and the columns follow `on_schema_change` as if `migrate_from` were absent. A target can require confirmation or stop the build instead with [`missing_migration_origin`](../project-configuration.md#missing-migration-origins). | +| `conflict` | Both columns exist. The build stops (`M110`). Drop one of them, or remove `migrate_from` and let `on_schema_change` handle the columns. | +| `source missing` | Neither column exists and no rename is recorded. The plan warns (`M109`) and the columns follow `on_schema_change`. A target can require confirmation or stop instead with [`missing_migration_origin`](migrations.md#missing-migration-origins). | | `source still produced` | The query still selects the old column. The build stops (`M111`). | -| `unsupported` | The warehouse can't rename this table's columns in place (see [Adapter support](#adapter-support)). The build stops (`M112`). | - -Automatically detected renames never stop a build: when they can't be applied, the columns follow `on_schema_change` as before. - -## How a rename runs - -Renames run after [model migrations](migrations.md) and before any model builds, one model at a time: - -1. The table's current columns are read again. A column that is already renamed is not renamed twice. -2. The column is renamed with `ALTER TABLE ... RENAME COLUMN` or the adapter's equivalent. -3. The rename is recorded. - -On adapters with transactional DDL (PostgreSQL, SQL Server, DuckDB, MotherDuck), each model's renames and their records commit together. On other adapters, a build interrupted between the rename and the record is recovered by the `already renamed, record` decision on the next build. A build that fails after the rename, for example while applying the delta, resumes on the next build without renaming again. - -After the rename, the remaining differences follow `on_schema_change`: `append_new_columns` adds any other new columns, `sync_all_columns` also drops removed ones and changes types, and `fail` only fails when a difference other than the rename remains. - -A pure rename is not a query change. This applies to declared renames too, as long as the rename is the whole change by the rule above. It doesn't trigger `replay_on_change` for the model, and it doesn't cascade a replay to downstream models. Downstream models that you edit to use the new name have query changes of their own and follow their own `replay_on_change`. If the renamed column is the model's cursor, the next delta starts from the watermark stored under the old name. - -## State - -Each rename is recorded in an append-only `_sqlbuild_column_migrations` table in the model's schema, with the model, relation, old and new column, discovery (`manual` or `automatic`), and decision (`rename` or `record`). Janitor never archives it. +| `unsupported` | The warehouse can't rename this table's columns (see [Adapter support](#adapter-support)). The build stops (`M112`). | ## Adapter support | Adapter | Statement | |---------|-----------| | Snowflake, PostgreSQL, DuckDB, MotherDuck, BigQuery | `ALTER TABLE ... RENAME COLUMN ... TO ...` | -| Databricks | `ALTER TABLE ... RENAME COLUMN ... TO ...`, which requires Delta column mapping | +| Databricks | `ALTER TABLE ... RENAME COLUMN ... TO ...`, which needs Delta column mapping | | SQL Server | `EXEC sp_rename '..', '', 'COLUMN'` | -Databricks renames a Delta table column only when the table uses name-based column mapping (`delta.columnMapping.mode = 'name'`). SQLBuild checks this during planning and doesn't change the table protocol for you. A declared rename on a table without column mapping stops the build (`M112`) with the statement that enables it: +On Databricks, a Delta table column can be renamed only when the table uses name-based column +mapping. SQLBuild checks this during planning and doesn't change the table for you. A declared +rename on a table without column mapping stops the build (`M112`) with the statement that enables +it: ```sql ALTER TABLE
SET TBLPROPERTIES ( @@ -135,6 +146,33 @@ ALTER TABLE
SET TBLPROPERTIES ( ); ``` -Enabling column mapping upgrades the table's Delta protocol, which older readers may not support. An automatically detected rename on such a table is skipped with a warning (`M112`) and follows `on_schema_change`. - -On PostgreSQL, views that select the renamed column keep working and keep their own output column names. On SQL Server, `sp_rename` warns that renaming can break scripts and stored procedures that reference the old name. On PostgreSQL, the new name is created in lower case, as an unquoted identifier would be. +Enabling column mapping upgrades the table's Delta protocol, which older readers may not support. +An automatically detected rename on such a table is skipped with a warning (`M112`). + +## Details + +- Expressions are compared ignoring formatting, comments, and the case of keywords and unquoted + identifiers. Detection compares the query the table was last built from with the current query, + so it applies to the build that makes the change. +- "The only change" means the same CTEs, `FROM`, joins, filters, grouping and other columns. Output + columns may be reordered, unless the query groups, orders or uses `DISTINCT ON` by position. +- A new name used in `ORDER BY` or `QUALIFY` counts as the old name only when SQLBuild can prove it + refers to the output column: every input relation's columns are known, the new name isn't an + input column, and the old name either isn't one or passes through unchanged. +- A rename that is the whole change is not a query change, whether declared or detected. It doesn't + trigger `replay_on_change` and doesn't replay downstream models. Downstream models you edit to use + the new name follow their own `replay_on_change`. +- If the renamed column is the model's cursor, the next delta starts from the watermark stored under + the old name. +- Renames run after [model migrations](migrations.md) and before any model + builds, one model at a time. SQLBuild reads the table's columns again first, so a column is never + renamed twice. On PostgreSQL, SQL Server, DuckDB and MotherDuck, a model's renames and their + records commit together; elsewhere an interrupted run recovers through `already renamed, record`. +- After the rename, other differences follow `on_schema_change`: `append_new_columns` adds new + columns, `sync_all_columns` also drops removed ones and changes types, and `fail` fails only when + a difference other than the rename remains. +- Renames are recorded in a `_sqlbuild_column_migrations` table in the model's schema. Janitor + never archives it. +- On PostgreSQL, views that select the renamed column keep working with their own output names, and + the new name is created in lower case. On SQL Server, `sp_rename` warns that renaming can break + scripts and stored procedures that use the old name. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md index bbf4c9afad..f1eaafc8d9 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md @@ -108,22 +108,21 @@ MODEL ( ); ``` -A `"` inside a `"..."` string ends it, so it is an error with its line and column rather than the -start of an escape. A description has no `\"` escape and no `""` doubling: in a `"..."` -description a backslash is an ordinary character, so `"Files under C:\"` ends with a backslash and -`"Path C:\\temp"` stores both backslashes. In a `'...'` description, `\'` and `''` both stand for -one `'`, as in `'it\'s'` and `'it''s'`. `sqb format --fix` rewrites a description that escapes its -quotes as `\"` or doubles them as `""` into a `"""..."""` string holding real quotes: +Quotes inside strings: + +- A `"` inside a `"..."` string ends it, so it is an error. There is no `\"` escape and no `""` + doubling; use `"""..."""` instead. `sqb format --fix` rewrites such a description for you. +- In a `"..."` string, a backslash is an ordinary character: `"Path C:\\temp"` stores both + backslashes. +- In a `'...'` string, `\'` and `''` both stand for one `'`, as in `'it\'s'` and `'it''s'`. ```text error[D002]: MODEL(...) in 'models/customers.sql:2:34' contains invalid SQLBuild header syntax: a double quote ends this "..." string early; use """…""" for text containing double quotes = help: write the text between triple quotes, where quotes need no escaping: description """Matches "Miss"."""; sqb format --fix rewrites a description that escapes its quotes as \" or doubles them as "" ``` -A `"""..."""` string ends at the first three double quotes in a row. Up to two quotes just before -them belong to the text, so the text may end with a quote, but it cannot contain three in a row. -A `"""` inside the text and a `"""` that is never closed fail with the position of the offending -quote or of the opening `"""`. +- A `"""..."""` string ends at the first three double quotes in a row. The text may end with a + quote but can't contain three in a row. ### Folded descriptions @@ -329,7 +328,7 @@ whole attempt, starting when the model begins executing: its hooks, statements, every microbatch. Time spent waiting for upstream models or for a free worker does not count. At the deadline SQLBuild cancels the model's running statements through the adapter (see the -[adapter table](../project-configuration.md#duration-limit)), runs no further statements +[adapter table](../../cli/build.md#duration-limit)), runs no further statements for the attempt, and reports the model as timed out with the elapsed time, the limit, where it was set, the cancellation outcome and the query IDs it saw. Statements that already completed are not undone, but the attempt stops before any later promotion or swap, so a staged table is not @@ -337,5 +336,5 @@ published. Models that depend on it are skipped as for any failure, and independ A model that finishes successfully just after its limit keeps its successful result. The per-model limit is separate from the target's whole-build -[`max_duration`](../project-configuration.md#duration-limit). When both apply, each +[`max_duration`](../../cli/build.md#duration-limit). When both apply, each statement is bounded by whichever deadline comes first. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/contracts.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/contracts.md index efb708a7c5..1ad5602cc1 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/contracts.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/contracts.md @@ -2,35 +2,19 @@ # Contracts -> Validate required or exact model output schemas. +> Say which columns a model promises to output, and when SQLBuild checks them. Online: https://sqlbuild.com/docs/concepts/models/contracts/ ## Contents -- Validation -- Type enforcement -- Reusable schemas -- Enum columns -- Adopting an existing schema -- Related policies +- What fails and when +- Starting from an existing warehouse +- Models that read an enforced contract +- Details -A model's declared columns provide output metadata, type enforcement, and column audit attachment. The project `check_declared_columns` setting and model `contract` policy determine whether those declarations also define a statically validated output shape. - -| Model declaration | Behavior | -|-------------------|----------| -| Unspecified | Follow `settings.check_declared_columns`: `true` validates declared columns as an open shape; `false` treats them as metadata and audit attachment only. | -| `none` | Do not activate declared-shape or nullability contract validation for this model. Explicit type enforcement remains active. | -| `enforced` | Treat declared columns as the complete authoritative output shape and enable runtime exact-schema checks on supported materializations. | - -The project default is `check_declared_columns = true`. Projects that use column declarations primarily for metadata and audits, including migrations from dbt, can turn the check off: - -```toml -[settings] -check_declared_columns = false -``` - -With `check_declared_columns = false`, a model must declare `contract enforced` to activate shape validation, the same as every other model declaring `contract none`. A model-level `contract enforced` or `contract none` always overrides the project setting. +A contract is the list of columns a model promises to output. You write it in the model's +`columns (...)`, and SQLBuild checks the model's SQL against it. ```sql MODEL ( @@ -46,128 +30,87 @@ MODEL ( ); ``` -## Validation - -### Compile time - -When declared-shape validation is active and SQL analysis can infer the model's output: - -- A missing declared column fails. -- A proven type mismatch fails when type enforcement or an exact contract is active. -- A declared non-null column fails when its expression is proven nullable. -- Additional inferred columns fail only for `contract enforced`. - -If SQLBuild cannot infer the output shape, these static shape checks cannot run. An enforced contract additionally requires a non-empty column declaration and adds runtime requirements on supported materializations. - -Type enforcement remains independent. Typed columns continue to receive static type checks and supported runtime casts with `check_declared_columns = false` and `contract none`. - -Configuration fields that reference output columns, including `unique_key`, `cursor`, `updated_at`, and `check_columns`, are checked against an enforced declaration. +How strictly the list is checked depends on the model's `contract` setting, which can also come +from `[defaults]` or `[path_defaults]`: -### Runtime - -For supported materialization paths, `contract enforced` inspects the staged relation and rejects: - -- Missing declared columns. -- Additional undeclared columns. -- Warehouse types that differ from declared types. - -Runtime contract validation does not scan data for nulls and does not inspect warehouse nullability metadata. `nullable false` participates in static nullability analysis; add a `not_null` audit when null values must be checked at runtime. - -Runtime exact-schema validation currently runs for: - -- Staged full-table builds. -- Non-microbatch incremental deltas. -- Snapshot deltas. +| `contract` setting | Listed columns | Extra columns | Checked | +|--------------------|----------------|---------------|---------| +| not set (default) | must be in the output | allowed | at compile time | +| `enforced` | must be in the output, with their declared types | not allowed | at compile time, and at runtime where the [table below](#runtime) says so | +| `none` | documentation and audit targets only | allowed | declared types only | -Views and custom materializations rely on compile-time contract analysis. Microbatch incrementals currently apply type enforcement and audits but do not perform the framework runtime exact-schema validation step. +To make models without a `contract` setting treat their columns as documentation and audit targets +only, set `[settings] check_declared_columns = false`. -Staged table validation happens before promotion, so a failure leaves the existing destination untouched. Immediate table promotion is incompatible with `contract enforced` because SQLBuild cannot validate output before replacing the destination. +Do this when column lists may not match the SQL, such as lists carried over from dbt YAML during a +migration. Models that need a contract then opt in with `contract enforced`. -## Type enforcement +Declared types are checked and cast whatever the contract; see +[Type Enforcement](type-enforcement.md). -Declaring a model column type enables type enforcement automatically, independently of the contract policy. Type enforcement controls static type compatibility and runtime casts on supported table and incremental paths; contracts control output shape. See [Type Enforcement](type-enforcement.md) for the materialization matrix. +## What fails and when -## Reusable schemas - -Contracts apply to the effective declaration, not only the reusable base: +### Compile time -```sql -MODEL ( - model_schema order, - columns ( - ingestion_batch_id (type VARCHAR, nullable false), - ), - contract enforced, -); -``` +When SQLBuild can work out the model's output columns, these fail unless the model's columns are +documentation only: -With `contract enforced`, this requires exactly the resolved `order` columns plus `ingestion_batch_id`. With `contract none`, the resolved columns remain metadata, audit attachment points, and inputs to explicit type enforcement without activating shape validation. See [Schemas](schemas.md). +- A listed column is missing from the output (`K001`). +- A `nullable false` column can be null (`K004`). +- With `contract enforced` only: an extra column (`K005`), or a `unique_key`, `cursor`, `updated_at` + or `check_columns` that names an unlisted column. -## Enum columns +A column whose type doesn't match its declared type fails too (`K002`), whatever the contract. If +SQLBuild can't work out the output, these checks are skipped. `contract enforced` needs at least one +listed column (`K006`). -A column may use a declared enum as a portable logical domain type: +### Runtime -```sql -MODEL ( - description "Example model", - contract enforced, - columns ( - fulfillment_method (type fulfillment_method), - ), -); -``` +With `contract enforced`, SQLBuild also checks the built output's column names and types before it +replaces anything: -This does not require, create, or reference a warehouse-native enum type. SQLBuild resolves string-valued enums to `VARCHAR` and integer-valued enums to `INTEGER`. Under `contract enforced`, it also generates an `accepted_values` audit for the declared members. Audit severity and timing follow normal audit configuration and materialization behavior. With the default error severity, it gates staged-table promotion and pre-DML delta paths; views and custom materializations may already have changed their relation when the audit runs. +| Materialization | Runtime check | +|-----------------|---------------| +| Table (staged, the default) | Before the new table replaces the old one | +| Table or non-microbatch incremental with `table_promotion_mode = "immediate"` | Not allowed: a `K011` compile error. Use staged promotion | +| Incremental | On each delta, before it is written | +| Microbatch incremental | None; compile time only | +| Snapshot | On each delta | +| View, custom | None; compile time only | -Enum member references such as `@enum("fulfillment_method").DELIVERY` are a separate feature that render one validated SQL literal. See [Enum Model Contracts](../enums/model-contracts.md) for the complete distinction and lowering behavior. +Runtime checks don't look at nulls. Add a `not_null` audit to check for nulls at runtime. -## Adopting an existing schema +## Starting from an existing warehouse -Ordinary `sqb compile` validates authored contracts offline. To compare declarations with existing -warehouse relations, use the explicitly online, read-only contract commands: +To compare your declarations with tables that already exist, or write declarations from them, use +[`sqb contract`](../../cli/contract.md). It reads the warehouse and never changes it: ```bash sqb contract diff --from prod --select tag:commerce -sqb contract generate --from prod --select tag:commerce -``` - -The target named by `--from` supplies the database and schema namespace to inspect. SQLBuild uses -the active project connection and never resolves a second set of origin credentials. - -### Preview and write - -`contract diff` returns exit code `1` when declarations and physical relations disagree. -`contract generate` prints proposed declaration changes without changing files: - -```bash -# Fill missing types and append physical columns missing from code sqb contract generate --from prod --select tag:commerce --write - -# Replace conflicting types/names and remove declarations absent physically -sqb contract generate --from prod --select tag:commerce --write --overwrite ``` -Contract commands never mutate warehouse relations, lifecycle state, or fingerprints. - -### Safety rules - -- Additive generation preserves existing types on conflict, repository-only columns, comments, - descriptions, audits, nullability, enums, tags, freshness, and loader configuration. -- `--overwrite` is an explicit repository replacement policy, not a warehouse write. -- Generation does not add `contract enforced`; activation remains an authored decision. -- A shared `SCHEMA()` is not changed from one model's evidence. Ownership conflicts are reported. -- Writes are atomic and SQLBuild recompiles the project, restoring prior contents if validation - fails. -- Sources are first-class selectable resources, for example `--select source:raw_orders`. - -Enforced upstream contracts are authoritative compile-time interfaces. When input evidence is -complete, missing or ambiguous columns fail in projections, joins, filters, grouping, windows, and -ordering. Partial or opaque schemas remain open; SQLBuild does not turn missing metadata into a -false error. Use `sqb lineage --include-uses --format json` to inspect direct -non-projection uses such as `join_on`, `where`, `group_by`, and `window_order_by`. - -## Related policies - -The project default checks declared columns as an open shape for models that omit `contract`. A repository can require an explicit `contract enforced` instead with `settings.check_declared_columns = false` or configure a custom [Rule](../rules/custom-rules/overview.md) that requires enforced contracts for selected model families. - -Contracts also constrain schema-change behavior. For example, `snapshot_schema_change append_new_columns` is incompatible with `contract enforced` because an unannounced appended column would violate the exact declaration. +`generate` fills in missing types and columns but never adds `contract enforced`; that stays your +decision. See [`sqb contract`](../../cli/contract.md) for `--overwrite` and what generation keeps. + +## Models that read an enforced contract + +When a model reads a model or source with `contract enforced`, SQLBuild knows its exact columns. A +reference to a column it doesn't have, or to an ambiguous one, fails at compile time, wherever it +appears: the select list, a join, a filter, grouping, a window or `ORDER BY`. When an upstream's columns aren't fully known, +nothing is reported. To see where a model uses each upstream column, run +`sqb lineage --include-uses --format json`. + +## Details + +- **Reusable schemas.** A contract covers the model's full column list: the columns of its + [reusable schema](schemas.md) plus any it adds. With `contract enforced`, the + output must have exactly those. +- **Column order.** Contracts match columns by name; output order isn't checked. +- **Enum columns.** A column typed with an enum, such as `fulfillment_method (type fulfillment_method)`, + is a `VARCHAR` or `INTEGER`. With `contract enforced`, SQLBuild adds an `accepted_values` audit for + its members. See [Enum Model Contracts](../enums/model-contracts.md). +- **Schema changes.** `snapshot_schema_change append_new_columns` can't be combined with + `contract enforced`, because an appended column would break the exact list. +- **Requiring contracts.** A custom [Rule](../rules/custom-rules/overview.md) can require + `contract enforced` for chosen models. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks/sql.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks/sql.md index 2c76f6e667..6b8e75009c 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks/sql.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks/sql.md @@ -212,7 +212,7 @@ The model must still see a named hook under the usual SQLBuild strictly validates hook resource syntax, arguments, interpolation, macros, metadata, and non-empty payloads. It does not classify executable statement kinds or implement vendor SQL grammar. Each rendered hook payload is passed to the adapter in one `execute` call; whether a driver accepts multiple statements or client-side batch separators is adapter and warehouse behavior. Use separate hook entries when portable ordering between statements matters. -After expansion, Polyglot validates the complete payload using the active adapter's analysis dialect when the model's effective `sql_analysis` setting is enabled, which it is by default. The `--no-sql-analysis` flag disables this analysis for the invocation. Polyglot does not understand every administrative or procedural command supported by every warehouse. If it cannot parse valid vendor-specific hook SQL, disable SQL analysis for that model or invocation and let the adapter and warehouse provide the authoritative result. SQLBuild does not compensate with keyword allowlists or handwritten parser fallbacks. +After expansion, SQLBuild's SQL analysis validates the complete payload using the active adapter's analysis dialect when the model's effective `sql_analysis` setting is enabled, which it is by default. The `--no-sql-analysis` flag disables this analysis for the invocation. SQL analysis does not understand every administrative or procedural command supported by every warehouse. If it cannot parse valid vendor-specific hook SQL, disable SQL analysis for that model or invocation and let the adapter and warehouse provide the authoritative result. SQLBuild does not compensate with keyword allowlists or handwritten parser fallbacks. Common errors include: @@ -220,6 +220,6 @@ Common errors include: - Unsupported header keys, empty descriptions, and missing SQL bodies - Missing or unused arguments and unsupported argument values - Unknown named hooks, unquoted hook names, and bare hook strings -- SQL rejected by Polyglot while optional SQL validation is enabled +- SQL rejected by SQL analysis while optional SQL validation is enabled Definition errors point to the hook file. Invocation and argument errors also identify the consuming model entry, such as `post_hooks[1] sql("grant_access")`. Runtime output preserves the authored hook index and identifies named and inline SQL entries. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/materializations.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/materializations.md index a9caf5a4f7..3baa7ed8c0 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/materializations.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/materializations.md @@ -6,6 +6,15 @@ Online: https://sqlbuild.com/docs/concepts/models/materializations/ +## Contents + +- View +- Table +- Incremental +- Snapshot +- Custom +- When a build fails + The `materialized` field selects how a model becomes a warehouse relation. ## View @@ -19,11 +28,11 @@ SELECT id AS order_id, customer_id, status FROM __source("raw__orders") ``` -View audits run after the view has been replaced. A failing audit marks the build as failed but cannot preserve the previous view. Views use compile-time contract and type analysis; SQLBuild does not rewrite the view with runtime casts or run the staged runtime-contract step. +Views are checked at compile time only; SQLBuild doesn't add runtime casts or contract checks to a view. ## Table -By default, creates a staging table, applies supported type and contract enforcement, runs blocking audits, and only then promotes it to the destination. A pre-promotion failure leaves the previous destination unchanged. +Builds the new table in a staging table, checks its types and contract, runs blocking audits, and only then replaces the old table. ```sql MODEL (description "One row per order", materialized table); @@ -33,7 +42,7 @@ FROM __ref("stg_orders") GROUP BY customer_id ``` -Projects may opt into `settings.table_promotion_mode = "immediate"`. Immediate promotion replaces the destination before audits, so a failed audit does not restore the old table. Immediate promotion rejects models that require declared-type enforcement or `contract enforced`; use staged promotion for those guarantees. A table or incremental model with `contract enforced` under immediate promotion is a `K011` compile error, so `sqb compile`, `sqb plan`, `sqb build`, and `sqb scenario test` report it before any warehouse work. +Set `[settings] table_promotion_mode = "immediate"` to build straight into the destination and audit afterwards. Immediate promotion can't enforce declared types or `contract enforced`. A model with `contract enforced` is a `K011` compile error; a model with declared types fails when it builds. ## Incremental @@ -51,7 +60,7 @@ MODEL ( ); ``` -For non-microbatch models, the first run, `--full-refresh`, and a full replay-on-change rebuild use the full-table path rather than incremental DML. Microbatch models retain batched execution: a full rebuild drops the existing target, creates its replacement from the first batch, and applies later batches with incremental DML. A failed full-refresh microbatch does not preserve the previous target. +The first run, `--full-refresh` and a full `replay_on_change` rebuild build the whole table, as a table model does. A microbatch model instead drops the table and rebuilds it batch by batch. See [Incremental](../incremental.md) for cursor semantics, replay, schema changes, and microbatch execution. @@ -101,3 +110,14 @@ WHERE ordered_at >= @@@partition_start The `config` block is passed to the Python function through `ctx.config`. Runtime-owned `@@@placeholder` values remain unresolved until materialization execution. The custom function owns staging, runtime type or contract enforcement, audit timing, promotion, and rollback. Framework final audits run after the function returns unless `MaterializationResult.audit_results` is populated. A custom materialization can call `ctx.run_audits` against a staging relation before applying changes and return those results through `MaterializationResult.audit_results`. + +## When a build fails + +| Materialization | What a failed build leaves | +|-----------------|----------------------------| +| View | The new view. Audits run after it is replaced. | +| Table (staged, the default) | The old table, untouched. Checks and audits run before it is replaced. | +| Table (`immediate`) | The new table. Audits run after it is replaced. | +| Incremental | The old table if a delta audit fails. Rows already written stay if a final audit or a later microbatch fails. | +| Microbatch full rebuild | A partly rebuilt table; the old one was dropped first. | +| Custom | Whatever the custom materialization does. | diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md index 3576e4d1c2..8ac221da44 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md @@ -11,21 +11,40 @@ Online: https://sqlbuild.com/docs/concepts/models/migrations/ - Declaring a migration - Decisions - Automatic discovery -- How a move runs - Old names -- State - Adapter support - Previewing another target +- Details -Renaming an incremental or snapshot model normally means rebuilding it from scratch under the new name, which loses any history the source data can no longer reproduce. A model migration moves the existing relation's data to the new name instead, then continues building incrementally. +Rename a model with `sqb rename stg_orders orders`. On the next build, SQLBuild moves the existing +table to the new name and carries on building it incrementally, so no history is lost and nothing is +backfilled. The old name keeps working as a view for 30 days. -Table and view models can declare a migration too. They hold no history, so they are rebuilt under the new name, but they keep their identity for change detection. For every materialization, the old name keeps working for a while through a [compatibility view](#old-names). To keep a column's history when you rename the column, see [Column migrations](column-migrations.md). +```bash +sqb rename hourly_order_activity order_activity_by_hour +sqb plan +``` -## Declaring a migration +```text +Migrations (1) +└── order_activity_by_hour migrate dev.hourly_order_activity -> dev.order_activity_by_hour + ├── compatibility compatible + ├── transfer physical copy, promote by transactional rename + └── old name dev.hourly_order_activity + ├── view until 2026-11-06 + └── old table archived +``` + +This matters most for incremental and snapshot models, whose history the source data may no longer +reproduce. Table and view models are rebuilt under the new name, but they keep their identity for +change detection, and their old name keeps working too. To rename a column instead, see +[Column migrations](column-migrations.md). -[`sqb rename`](../../cli/rename.md) and `sqb mv` rename or move a model, update every reference to it, and add `migrate_from` whenever the model's relation moves and its materialization keeps warehouse data (table, view, incremental, or snapshot). +## Declaring a migration -To declare a migration by hand, add `migrate_from` to the renamed model's header, naming the old model: +[`sqb rename`](../../cli/rename.md) and `sqb mv` rename or move a model, update every reference to it, +and write `migrate_from` into the model's header. You can also write it yourself, naming the old +model: ```sql MODEL ( @@ -37,13 +56,20 @@ MODEL ( ); ``` -Use `schema.name` when the old relation is in a different schema of the same database. +Use `schema.name` when the old relation is in another schema of the same database. -On the next build, SQLBuild copies the old relation's data to the new name through a staging table (see [How a move runs](#how-a-move-runs)), records the move, and builds the model incrementally from the migrated state. There is no first-run rebuild, and no `replay_on_change` replay caused by the rename. After the model builds, the old relation is archived and a compatibility view takes its name (see [Old names](#old-names)). +On the next build, SQLBuild copies the old relation to the new name, records the move, and builds +the model incrementally from there. There is no first-run rebuild, and the rename alone causes no +`replay_on_change` replay. After the model builds, the old relation is archived and a view takes its +name (see [Old names](#old-names)). -A rename is usually made together with other edits, for example a changed filter or a renamed upstream reference. If the renamed model's query also differs from the old model's, the plan still recognises the rename: the model is listed as `Renamed` with its query diff, never as a first run. The query change follows the model's own `replay_on_change`, exactly as it would without the rename (forward-only by default). Models downstream of the renamed model are not replayed because of the rename; see [Replay decisions](../planning/replay-decisions.md). +A rename usually comes with other edits, such as a changed filter. The plan still recognises the +rename and lists the model as `Renamed` with its query diff. The query change follows the model's +own `replay_on_change`, and downstream models are not replayed because of the rename; see +[Replay decisions](../planning/replay-decisions.md). -After the move is recorded, `migrate_from` has no further effect, and SQLBuild tells you it can be removed. Keep it until every target that needs the move has built. +Once the move is recorded, `migrate_from` has no further effect, and SQLBuild tells you it can be +removed. Keep it until every target that needs the move has built. ## Decisions @@ -53,78 +79,67 @@ After the move is recorded, `migrate_from` has no further effect, and SQLBuild t |----------|---------| | `migrate` | The new relation doesn't exist. Move the old one's data to it. | | `done` | The move is already recorded. Nothing to do. | -| `redo` | The new relation exists but has no build history and no recorded move, which is what an interrupted migration leaves behind. Move it again. | -| `superseded replace` | The new relation was itself the origin of the latest recorded move, for example after renaming a model back. Replace it with the newer data. | +| `redo` | The new relation exists with no build history and no recorded move. An interrupted migration leaves this. Move it again. | +| `superseded replace` | The new relation was the origin of the latest recorded move, for example after renaming a model back. Replace it with the newer data. | | `forced replace` | The new relation has its own build history and `migrate_force true` is set. Replace it. | | `conflict` | The new relation has its own build history. The build stops (`M103`) until you set `migrate_force true` or remove `migrate_from`. | -| `origin missing` | The old relation doesn't exist and no move was recorded, for example on a target that never built the old name. The plan warns (`M102`) and the model builds fresh. A target can require confirmation or stop the build instead with [`missing_migration_origin`](../project-configuration.md#missing-migration-origins). If SQLBuild has build history for the old relation in this target, it should exist, so the build stops (`M102`) whatever the policy: the connection may lack permission to see it, or it was dropped outside SQLBuild. Restore access, or remove `migrate_from` to build fresh deliberately. | +| `origin missing` | The old relation doesn't exist and no move was recorded, for example on a fresh target. The plan warns (`M102`) and the model builds fresh. | -The origin and destination must be compatible under the model's normal `on_schema_change` rules; otherwise the build stops (`M104`). +On a target where the old name was never built, `origin missing` is normal. A target can ask for +confirmation or stop the build instead; see [Missing migration origins](#missing-migration-origins). -`migrate_force true` only affects the `conflict` case and is safe to leave in the header. +If SQLBuild built the old relation in this target but can't see it now, the build always stops with +`M102`: the connection may lack permission, or the relation was dropped outside SQLBuild. Restore +access, or remove `migrate_from` to build fresh on purpose. -Table and view models only ever see `migrate`, `done`, and `origin missing`: nothing is replaced, so there is no conflict. `migrate_force` on a table or view model is a compile error. In `sqb plan --json`, their decision is `renamed`. +The old and new relations must be compatible under the model's `on_schema_change` rules; otherwise +the build stops (`M104`). `migrate_force true` only affects the `conflict` case and is safe to leave +in the header. -## Automatic discovery +Table and view models only see `migrate`, `done` and `origin missing`, because nothing is replaced. +`migrate_force` on a table or view model is a compile error. In `sqb plan --json`, their decision is +`renamed`. -If a selected model has never been built, SQLBuild compares it with models that were removed from the project but whose relations still exist. When exactly one removed model has equivalent logic, it is migrated automatically. Equivalent means the same query and configuration, ignoring the model's own name, CTE and table alias names, comments, formatting, and storage-only settings. Renamed upstream models are matched first, so a renamed chain of models is migrated together. Removed models whose data was already moved on by a recorded migration are not candidates, so a model renamed several times matches its latest table. +### Missing migration origins -Renamed tables and views are matched the same way. They are rebuilt under the new name, since they hold no history, but they keep their identity for change detection. These renames are recorded as `renamed` events. +On a target that never built the old name, such as a fresh development schema, there is nothing to +move. The target's `missing_migration_origin` decides what the build does: -Automatic discovery never guesses. If a match is ambiguous, SQLBuild warns (`M107`) and builds from scratch; declare `migrate_from` to choose. An explicit `migrate_from` always wins. Automatic discovery covers renames within the project's schemas; use `migrate_from` to move a model to another schema. Relations last built before this feature carry no stored fingerprint and aren't matched automatically. +| Value | Behavior | +|-------|----------| +| `allow` | Default. The plan warns (`M102` for models, `M109` for columns) and the model builds as if `migrate_from` were absent. | +| `require_confirmation` | The build asks on a terminal and stops otherwise. Pass `--allow-missing-migration-origin` to confirm in a non-interactive run. | +| `deny` | The build stops with `M102` or `M109`. Use it where a missing origin means the wrong target or a lost table. | -## How a move runs +```toml +[targets.prod] +missing_migration_origin = "deny" +``` -A move never overwrites the destination directly: +`sqlbuild_local.toml` can set it for a target too. `sqb dbt run` and `sqb dbt build` follow it but +have no confirmation flag: they ask on a terminal and stop otherwise. In a non-interactive dbt run, +build the affected models with `sqb build --allow-missing-migration-origin` and leave them out of +the dbt selection, or set the target to `allow`. -1. The old relation is cloned or copied into a new staging table named `_sqb_archive____migration_stage__`, with the destination's configured table type. -2. The staging table is checked: it must exist, have the expected columns and table type, and, for a physical copy, the same row count as the old relation. -3. The staging table is swapped in as the destination. If a destination already existed, for example after `superseded replace` or `forced replace`, it is kept as `_sqb_archive____migration_previous__` rather than dropped. -4. The move is recorded, then the model builds as usual. -5. After the model builds successfully, the old relation is archived and a compatibility view is created at its name (see [Old names](#old-names)). +## Automatic discovery -Until step 5, the old relation is not modified. Staging and replaced tables use the [janitor](../../cli/janitor.md) archive format, so an abandoned staging table or a replaced destination is deleted by janitor after `archive_retention_days`. A staging table left by an interrupted run is never reused; the next run stages again. Don't run janitor at the same time as a build in the same schema. +A model you rename by hand, without `migrate_from`, is still found when its logic is unchanged. If a +selected model has never been built and exactly one removed model has the same query and +configuration, SQLBuild migrates it automatically. Renamed upstream models are matched first, so a +renamed chain of models moves together. -`sqb plan` shows how each move will run, for example `physical copy, transient -> permanent, promote by swap`. +Discovery never guesses. If more than one removed model matches, SQLBuild warns (`M107`) and builds +from scratch; declare `migrate_from` to choose. An explicit `migrate_from` always wins. Discovery +covers renames within the project's schemas; use `migrate_from` to move a model to another schema. ## Old names -A rename breaks every query outside the project that still reads the old name: dashboards, notebooks, other teams' jobs. After the renamed model builds successfully, SQLBuild keeps the old name working for a while: - -1. The old relation is renamed to `_sqb_archive____migration_origin__`. -2. A view is created at the old name that selects from the new relation. -3. The view gets the privileges that were granted on the old relation, read from the archive. Only existing grants are copied; nothing new is granted. - -On DuckDB, MotherDuck, PostgreSQL, and SQL Server these steps and their records commit in one transaction. On Snowflake, BigQuery, and Databricks each step is recorded after it runs, and a re-run resumes from the first unrecorded step without repeating one. A failure is reported as `M117`; re-run the build. - -| Adapter | Privileges copied to the view | -|---------|-------------------------------| -| Snowflake | `SHOW GRANTS` on the archive, replayed to roles and database roles, keeping `WITH GRANT OPTION`. `OWNERSHIP`, grants to shares, and privileges that don't apply to views are skipped | -| PostgreSQL | Table and column privileges from the catalog, including grants to `PUBLIC` and `WITH GRANT OPTION`. The owner's own privileges are not copied | -| BigQuery | Table IAM role bindings from `INFORMATION_SCHEMA.OBJECT_PRIVILEGES`, replayed with `GRANT`. See the note below | -| Databricks | Unity Catalog grants on the table itself that apply to views, such as `SELECT`. `OWN`, `MODIFY`, and inherited catalog or schema grants are skipped | -| SQL Server | Object and column permissions, including `DENY` and `WITH GRANT OPTION`, that apply to views | -| DuckDB, MotherDuck | No object privileges exist | - -Column privileges are copied to the view's column of the same name, which is the old name when the view aliases a renamed column. A column privilege on a column the view no longer exposes is not copied; the view shows nothing of that column. - -The copied grants are recorded on the view's creation. After that, the view's own privileges are what counts: when SQLBuild redefines the view after a rebuild, the view ends up with exactly the privileges it had just before, so a privilege you revoke from the view stays revoked and one you grant on the view is kept. SQLBuild uses `CREATE OR REPLACE VIEW` on PostgreSQL, `COPY GRANTS` on Snowflake, and `ALTER VIEW` on SQL Server and Databricks. - -Creating or replacing a view can also apply privileges you did not give it: PostgreSQL applies `ALTER DEFAULT PRIVILEGES` when it re-creates a view, and Snowflake applies the schema's future grants for views on `CREATE OR REPLACE VIEW … COPY GRANTS`. SQLBuild reads the view's privileges before every create or redefinition and reconciles afterwards: it grants what went missing and revokes what was added. A new compatibility view likewise ends up with only the privileges copied from the old relation. Ownership and the owner's own privileges are never changed. - -A recorded view that no longer exists at the old name is not re-created by a build; [janitor](../../cli/janitor.md) records it as dropped. +A rename breaks every query outside the project that reads the old name: dashboards, notebooks, +other teams' jobs. So after the renamed model builds, SQLBuild archives the old relation and creates +a view at the old name that reads the new one. The view gets the privileges that were granted on the +old relation; nothing new is granted. -On Snowflake, BigQuery, Databricks, DuckDB, and MotherDuck, views read their tables by name when queried, so a build redefines a compatibility view only when its SQL would change, for example when the view lists columns and the new relation's columns changed. PostgreSQL views are bound to the table itself, so they are redefined whenever the table is replaced. - -BigQuery cannot rename views, so an old view model is re-created under the archive name with its IAM bindings, and then dropped. - -On BigQuery, a view reads its tables with the permissions of whoever queries it. A principal that could read the old table keeps access to the old name through the copied bindings, but also needs read access on the new table. SQLBuild does not grant that and does not create authorized views; the build warns with `M118` and names the principals. - -If columns were renamed, the view presents them under their old names, for example `SELECT order_id, revenue AS amount FROM analytics.daily_revenue`. Incremental and snapshot models alias every recorded [column migration](column-migrations.md). A table or view model that declares `migrate_from` can declare old column names the same way, as `revenue (migrate_from amount)`; this only aliases the column in the compatibility view, since the table is rebuilt with the new name anyway. The view follows the new relation's current columns after every build of the model. - -`sqb plan` shows what happens at the old name: - -``` +```text Migrations (1) └── daily_revenue migrate analytics.revenue -> analytics.daily_revenue ├── transfer rebuild (table) @@ -135,61 +150,109 @@ Migrations (1) └── old table archived ``` -A move resumed after the archive shows `old table already archived`. Once the view exists it shows `view live until 2026-10-28` and how many grants were copied. When no view is kept, the block shows `left for janitor` and the reason, such as `old_name_view false` or `name reused by model:revenue`. +If columns were renamed, the view shows them under their old names, for example +`SELECT order_id, revenue AS amount FROM analytics.daily_revenue`. Incremental and snapshot models +alias every recorded [column migration](column-migrations.md). A table or view +model that declares `migrate_from` can name old columns the same way, as +`revenue (migrate_from amount)`; this only aliases the column in the view. -In `sqb plan --json`, each migration has an `old_name` object with `action` (`archive_and_view`, `view_only`, `live`, or `none`), `view`, `reads`, `expires_at`, `column_aliases`, `grants_copied` (the number of grants copied once the view exists, otherwise `null`), and `reason`. The top-level `old_names` list also includes steps that resume an earlier move. - -The view expires after 30 days by default. After that, [janitor](../../cli/janitor.md) drops it; `sqb janitor --drop-old-name-view analytics.revenue` drops it earlier. Change the default for the project, or turn the views off with `false`: +The view expires after 30 days. Then [janitor](../../cli/janitor.md) drops it; +`sqb janitor --drop-old-name-view analytics.revenue` drops it earlier. Change the default for the +project, or turn views off with `false`: ```toml [migrations] old_name_views = "7d" ``` -A model header overrides the project setting with `old_name_view 7d` or `old_name_view false`. The retention of a move is fixed when the move is recorded. - -With views off, or when a view can't be kept, the old relation is left in place and janitor archives it once it is no longer part of the project, as before. No view is created when: - -- the old relation was not built by SQLBuild; -- another model in the project builds at the old name; -- the old name is itself a compatibility view of an earlier rename. - -Moves recorded before a SQLBuild version that supports old-name views never gain a view later. - -### Code inside the project - -The compatibility view is for consumers outside the project. Project code must use the new model: - -- Python nodes, hooks, and loaders whose literal SQL names the old name fail with `P008`, at compile time for a declared `migrate_from` and at plan time for an automatically discovered rename. SQL models reading it through `__ref` are unaffected, since `__ref` resolves the new name. -- A new model whose name is a live compatibility view stops the build with `M114`, which names the view and its expiry. Drop the view early with janitor, or choose another name. +A model header overrides this with `old_name_view 7d` or `old_name_view false`. A move keeps the +retention it had when it was recorded. -### PostgreSQL views +No view is created when views are off, when the old relation wasn't built by SQLBuild, when another +model builds at the old name, or when the old name is itself a view from an earlier rename. The plan +then shows `left for janitor` and the reason, and janitor archives the old relation once it is no +longer part of the project. -PostgreSQL views are bound to the table they read, not its name. When a build replaces the renamed model's table, or drops or retypes its columns, SQLBuild points its compatibility views at the new table in the same transaction, so they are never missing, even if the build is interrupted. A view is replaced in place when its columns allow it, so views you build on top of the compatibility view keep working; otherwise it is dropped and re-created with the privileges it had, which fails if another view depends on it. +### Privileges by adapter -Views outside SQLBuild that read the old relation follow it into the archive and keep reading its old data; the build warns with `M115` and names them. Re-create them against the old name or the new relation. On the other adapters, views read relations by name, so no rebinding is needed. - -### Upgrading +| Adapter | Privileges copied to the view | +|---------|-------------------------------| +| Snowflake | `SHOW GRANTS` on the archive, replayed to roles and database roles, keeping `WITH GRANT OPTION`. `OWNERSHIP`, grants to shares, and privileges that don't apply to views are skipped | +| PostgreSQL | Table and column privileges, including grants to `PUBLIC` and `WITH GRANT OPTION`. The owner's own privileges are not copied | +| BigQuery | Table IAM role bindings from `INFORMATION_SCHEMA.OBJECT_PRIVILEGES`, replayed with `GRANT` | +| Databricks | Unity Catalog grants on the table that apply to views, such as `SELECT`. `OWN`, `MODIFY`, and inherited catalog or schema grants are skipped | +| SQL Server | Object and column permissions, including `DENY` and `WITH GRANT OPTION`, that apply to views | +| DuckDB, MotherDuck | No object privileges exist | -Upgrade the SQLBuild version that runs `sqb janitor` together with the one that runs builds. An older janitor does not know compatibility views and archives them as stale relations. +On BigQuery, a view reads its tables with the permissions of whoever queries it. A principal that +could read the old table also needs read access on the new one. SQLBuild doesn't grant it and +doesn't create authorized views; the build warns with `M118` and names the principals. -## State +### Code inside the project -Each move is recorded in an append-only `_sqlbuild_migrations` table in the destination schema. A missing record means completion is unrecorded; staging or promotion may already have happened. Re-running after an interruption recovers through the migration decisions above. +The view is for consumers outside the project. Project code must use the new name: -What happens at the old name is recorded in an append-only `_sqlbuild_old_name_views` table next to it, one row per step: `required` (with the move), `origin_archived`, `view_created`, and `view_dropped`. Rows are never updated. +- Python nodes, hooks and loaders whose literal SQL names the old name fail with `P008`. SQL models + that read it through `__ref` are unaffected, since `__ref` resolves the new name. +- A new model named like a live view stops the build with `M114`, which names the view and its + expiry. Drop the view early with janitor, or choose another name. ## Adapter support -| Adapter | Staging | Promotion | -|---------|---------|-----------| -| Snowflake | Zero-copy clone with `COPY GRANTS`. Transient → permanent is a physical copy, because Snowflake can't clone a transient table into a permanent one. | `ALTER TABLE … SWAP WITH` for an existing destination; rename for a missing destination | +| Adapter | Copy | Promotion | +|---------|------|-----------| +| Snowflake | Zero-copy clone with `COPY GRANTS`. Transient → permanent is a physical copy, because Snowflake can't clone a transient table into a permanent one. | `ALTER TABLE … SWAP WITH` for an existing destination; rename for a missing one | | BigQuery | `CREATE TABLE … CLONE`, or `CREATE TABLE … COPY` when BigQuery refuses the clone | Two renames; the destination is briefly missing between them, and an interrupted run recovers on retry | | Databricks | `DEEP CLONE` | Two renames, as for BigQuery | | PostgreSQL, SQL Server, DuckDB, MotherDuck | Physical copy | Renames and the migration record in one transaction | -On PostgreSQL, views that depend on the destination are re-pointed to the new table in the same transaction, keeping their options, owner and grants. Materialized views are not re-pointed. - ## Previewing another target -`sqb plan --as ` compiles the project for another target, for example production, and shows its migration decisions using your current connection. It only inspects; nothing is written. +`sqb plan --as ` compiles the project for another target, such as production, and shows its +migration decisions using your current connection. It only inspects; nothing is written. + +## Details + +- **How a move runs.** The old relation is cloned or copied into a staging table named + `_sqb_archive____migration_stage__`. The staging table is checked for its + columns, table type and, for a physical copy, row count. It is then swapped in as the destination. + A destination that already existed is kept as `_sqb_archive____migration_previous__`. + The move is recorded, the model builds, and only then is the old relation archived as + `_sqb_archive____migration_origin__`. +- Staging and replaced tables use the [janitor](../../cli/janitor.md) archive format, so janitor + deletes an abandoned one after `archive_retention_days`. A staging table left by an interrupted + run is never reused. Don't run janitor at the same time as a build in the same schema. +- Archiving the old relation, creating the view and copying grants commit in one transaction on + DuckDB, MotherDuck, PostgreSQL and SQL Server. On Snowflake, BigQuery and Databricks each step is + recorded after it runs, and a re-run resumes from the first unrecorded step. A failure is reported + as `M117`; re-run the build. +- Column privileges are copied to the view's column of the same name, which is the old name when the + view aliases a renamed column. A privilege on a column the view doesn't expose is not copied. +- After it is created, the view's own privileges are what counts. When SQLBuild redefines the view, + it reads its privileges first and restores them afterwards, so a privilege you revoke stays revoked + and one you grant is kept. This also undoes privileges that PostgreSQL `ALTER DEFAULT PRIVILEGES` + or Snowflake future grants would add. Ownership is never changed. +- On Snowflake, BigQuery, Databricks, DuckDB and MotherDuck, views read tables by name, so a build + redefines the view only when its SQL would change. PostgreSQL views are bound to the table, so + SQLBuild re-points them in the same transaction whenever the table is replaced, and other views + that depend on the destination are re-pointed too, keeping their options, owner and grants. + A view whose columns no longer fit is dropped and re-created with its privileges, which fails if + another view depends on it. Materialized views are not re-pointed. +- On PostgreSQL, views outside SQLBuild that read the old relation follow it into the archive and + keep reading its old data; the build warns with `M115` and names them. Re-create them against the + old name or the new relation. +- BigQuery can't rename views, so an old view model is re-created under the archive name with its + IAM bindings, then dropped. +- A recorded view that no longer exists is not re-created by a build; janitor records it as dropped. +- Run the same SQLBuild version for `sqb janitor` as for builds. An older janitor doesn't recognise + compatibility views and archives them as stale relations. +- In `sqb plan --json`, each migration has an `old_name` object with `action` (`archive_and_view`, + `view_only`, `live` or `none`), `view`, `reads`, `expires_at`, `column_aliases`, `grants_copied` + and `reason`. The top-level `old_names` list also includes steps that resume an earlier move. +- Discovery ignores the model's own name, CTE and table alias names, comments, formatting and + storage-only settings. A model already moved on by a recorded migration is not a candidate, so a + model renamed several times matches its latest table. Relations without a stored fingerprint are + not matched. Renamed tables and views are recorded as `renamed` events. +- Moves are recorded in an append-only `_sqlbuild_migrations` table in the destination schema, and + old-name steps in `_sqlbuild_old_name_views`. A missing record means the move may be partly done; + re-running recovers through the decisions above. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/schemas.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/schemas.md index 99d7255c66..b2ade1f3ef 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/schemas.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/schemas.md @@ -1,8 +1,8 @@ -# Schemas +# Reusable column schemas -> Declare model columns inline or reuse canonical inherited schemas. +> Declare a model's columns in its header, or once in a SCHEMA() that many models share. Online: https://sqlbuild.com/docs/concepts/models/schemas/ @@ -13,10 +13,12 @@ Online: https://sqlbuild.com/docs/concepts/models/schemas/ - Model-local columns - Model-specific column audits - Inheritance -- Contracts and planning +- Contracts and rebuilds - Limitations -Schema metadata defines column names, types, nullability, descriptions, and column audits. It can live inline in one `MODEL()` header or in a reusable `SCHEMA()` declaration. +A model's columns, with their types, nullability, descriptions and audits, are declared in its +`MODEL()` header. When several models output the same columns, declare them once in a `SCHEMA()` and +bind each model to it. ## Inline columns @@ -135,13 +137,16 @@ binding a schema. Enum column types must likewise be visible from the schema's o Inheritance may be transitive. Parent columns resolve before child columns. An inherited column cannot be redeclared or overridden in a child schema. SQLBuild rejects unknown parents, cycles, case-insensitive duplicates, and multiple parents. -Physical SQL output order is not currently enforced. Static contract analysis matches names and checks declared types and proven non-nullability. Runtime exact-contract validation matches names and types but does not validate nullability. +## Contracts and rebuilds -## Contracts and planning +A model's [contract](contracts.md) covers its full column list: the schema's +columns plus the ones the model adds. -`contract enforced` treats the complete named-plus-local declaration as the exact output shape. An unspecified model follows the project's `check_declared_columns` setting; `contract none` explicitly keeps the resolved columns as metadata, audit attachment, and type-enforcement inputs without activating shape validation. See [Contracts](contracts.md). - -For models bound to a reusable schema, effective column names, types, nullability, and enum members participate in model version identity. Changing those fields on a parent therefore affects models bound through descendants. Descriptions do not change model identity. Audits have their own audit-gate identities, so audit changes invalidate reusable audit results without changing the model version itself. +Changing a column's name or type in a schema changes every model bound to it, or to a schema that +extends it. The plan shows those models as `Schema changed` and replays them as described in +[Replay decisions](../planning/replay-decisions.md). Changing nullability or a description +doesn't change the plan. +Changing an audit reruns that audit without changing the model. ## Limitations diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/type-enforcement.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/type-enforcement.md index 02ac567e6a..c5db622751 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/type-enforcement.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/type-enforcement.md @@ -6,7 +6,7 @@ Online: https://sqlbuild.com/docs/concepts/models/type-enforcement/ -Type enforcement makes authored column types operational. It is a model feature as well as a related source feature, and it is separate from whether a model uses an exact contract. +Declare a column's type and SQLBuild checks it at compile time and, on staged tables and incremental models, casts the column to it at runtime. You don't switch it on: any declared type enables it. ## Enabling it @@ -58,17 +58,10 @@ Runtime cast support currently depends on materialization: Staged promotion is the default table mode. If a project selects immediate table promotion, a typed model fails with guidance to use staged promotion because SQLBuild cannot inspect and reconstruct output before mutating the destination. -## Type enforcement versus contracts +## Type enforcement and contracts -These controls answer different questions: - -| Feature | Question | -|---------|----------| -| Type enforcement | Should typed columns be checked and, on supported paths, cast to their declared physical types? | -| `contract none` | Which declared columns can SQLBuild verify statically while still allowing additional output columns? | -| `contract enforced` | Must the output have exactly the complete declared shape? | - -Type enforcement can be active with `contract none`. Conversely, an enforced contract may contain untyped columns: exact shape validation still checks their names, but there is no declared physical type to cast or compare. +Type enforcement works whatever the model's `contract` setting, including `contract none`. A +contract decides which columns must exist; see [Contracts](contracts.md). ## Schema changes diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability.md index 04e47f9354..bd446baa2c 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability.md @@ -1,27 +1,13 @@ -# Execution Observability +# Observability -> Choose authoritative lifecycle facts, readable logs, or command-output records. +> Find out what a run did, or send its events to your own systems. Online: https://sqlbuild.com/docs/concepts/observability/ -SQLBuild keeps execution facts separate from human output. Do not parse terminal text to determine -whether work started, completed, failed, skipped, or retried. - -| Record | Purpose | Authority | -|--------|---------|-----------| -| `LifecycleEvent` | Immutable invocation, run, resource-attempt, operation, statement, retry, and audit facts | Canonical lifecycle evidence once durably stored | -| `DiagnosticLog` | Structured framework diagnostics | Explanatory only | -| [Recall](observability/recall.md) | Queryable history of invocations, statements, SQL, audit results, and output, locally or in ClickHouse | Troubleshooting history; best effort | -| `stdout.log` / `stderr.log` | Exact host-local process output (compute log files) | Troubleshooting transcript; also stored by Recall | -| `CommandOutputRecord` | Bounded, ANSI-free remote stdout/stderr chunks | Potentially sensitive and lossy troubleshooting data | -| Final JSON output | End-of-command aggregate | Result projection, not event history | - -Human-readable CLI output remains the normal operator interface. Structured records are a separate -contract for integrations and durable consumers. - -## Which to use +SQLBuild records what each command did, separately from what it prints. Use these records instead +of parsing terminal output. | You want to | Use | |-------------|-----| @@ -32,12 +18,23 @@ contract for integrations and durable consumers. Recall is on by default and covers investigation for people and agents. Sinks and execution history are for building on SQLBuild's events elsewhere. The `stdout.log`, `stderr.log` and -`diagnostics.jsonl` compute log files under `logs//` are still written alongside Recall and -hold the same bytes; prefer `sqb recall output`, which works the same with a shared store. +`diagnostics.jsonl` files under `logs//` hold the same output as Recall; prefer +`sqb recall output`, which also works with a shared store. + +## What SQLBuild records + +| Record | What it holds | How far to rely on it | +|--------|---------------|-----------------------| +| `LifecycleEvent` | Each invocation, run, resource attempt, operation, statement, retry and audit | Complete once stored; events are never changed | +| `DiagnosticLog` | SQLBuild's structured diagnostics | Explanation only | +| [Recall](observability/recall.md) | Invocations, statements, SQL, audit results and output, locally or in ClickHouse | Best effort, for troubleshooting | +| `stdout.log` / `stderr.log` | The exact output of each process on this machine | For troubleshooting; Recall also stores it | +| `CommandOutputRecord` | Command output in bounded chunks, without colour codes, for remote systems | May be incomplete and may hold sensitive data | +| Final JSON output | The command's result | The result only, not what happened on the way | ## Lifecycle envelope -New events use lifecycle schema version 2: +Events use lifecycle schema version 2: ```json { diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/execution-history.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/execution-history.md index bc1da46322..c50d0acf9d 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/execution-history.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/execution-history.md @@ -12,14 +12,16 @@ Online: https://sqlbuild.com/docs/concepts/observability/execution-history/ - Local SQLite - Deployed PostgreSQL -`LifecycleEventLogStorage` stores canonical immutable lifecycle facts. `RunStorage` serves an atomically -published, rebuildable run projection. Diagnostic and compute logs are never accepted by either -contract. The public types, filters, pages, records, errors, and helper functions are exported from -`sqlbuild.execution_history`. +Execution history is a Python API for building your own run history: store SQLBuild's lifecycle +events in SQLite or PostgreSQL and read runs back, for example to show run status in an internal +tool. Everything is exported from `sqlbuild.execution_history`. -This is a Python API for applications that store and serve SQLBuild events themselves. To look into -what past runs did, use [Recall](recall.md), which records every command -automatically and is queried with [`sqb recall`](../../cli/recall.md). +To look into what past runs did, you don't need this API: use +[Recall](recall.md), which records every command automatically and is +queried with [`sqb recall`](../../cli/recall.md). + +`LifecycleEventLogStorage` stores events as they arrive and never changes them. `RunStorage` serves +the current state of each run, rebuilt from those events. Neither stores logs or command output. ## Public contracts @@ -29,10 +31,8 @@ automatically and is queried with [`sqb recall`](../../cli/recall.md). The concrete SQLite and PostgreSQL classes additionally provide transactional `append_and_project`, `reconcile`, and `check_health` operations. -The deprecated `EventLogStorage` alias has been removed; use `LifecycleEventLogStorage`. Storage -tables use the lifecycle-specific names `lifecycle_event_log` in SQLite and -`sqlbuild_lifecycle_event_log` in PostgreSQL; the former `event_log` and `sqlbuild_event_log` names -are retired. +Events are stored in the `lifecycle_event_log` table in SQLite and `sqlbuild_lifecycle_event_log` +in PostgreSQL. Each newly stored event receives a positive, increasing `storage_order`, opaque stable `cursor`, and UTC `received_at`. Pages ascend in storage order. `after_cursor` is exclusive, diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/recall-clickhouse.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/recall-clickhouse.md new file mode 100644 index 0000000000..4cccd56d69 --- /dev/null +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/recall-clickhouse.md @@ -0,0 +1,124 @@ + + +# Sharing Recall in ClickHouse + +> Keep one Recall history for a team or CI in ClickHouse: users, grants, setup, retention and reading. + +Online: https://sqlbuild.com/docs/concepts/observability/recall-clickhouse/ + +## Contents + +- Users and grants +- Setting up and upgrading the store +- Retention in ClickHouse +- Reading a ClickHouse store + +Point `write_connection` and `read_connection` at ClickHouse connections to keep one history for a +whole team. Every machine +that writes to it, including CI and scheduled production runs, becomes queryable from anywhere: +see what last night's production build ran and printed without access to the machine, find the +first run where an audit started failing across everyone's runs, or compare a production run with +your own by `target`, `git_revision` or `sqlbuild_version`. + +The connection needs `adapter = "clickhouse"` when the project uses another adapter. Recall creates +and upgrades its own store on the first write: the `database` when it does not exist yet, then its +tables, views and schema version. A writer that may run that DDL needs no other setup. + +```toml +[connections.history_writer] +adapter = "clickhouse" +host = "${env:CLICKHOUSE_HOST}" +user = "recall_writer" +password = "${env:RECALL_WRITER_PASSWORD}" +database = "observability" + +[connections.history_reader] +adapter = "clickhouse" +host = "${env:CLICKHOUSE_HOST}" +user = "recall_reader" +password = "${env:RECALL_READER_PASSWORD}" +database = "observability" + +[recall] +write_connection = "history_writer" # every command writes history; creates the store on first use +read_connection = "history_reader" # sqb recall query, schema and output (SELECT only) +retention = "30d" +``` + +`sqb recall query`, `schema` and `output` use only `read_connection` and never create or change +anything in the store, so someone investigating production runs needs no credentials that could +rewrite or delete its history. Both keys may name one connection when separate users are not +wanted. A machine that only reads shared history, such as a laptop investigating production runs, +sets only `read_connection`; its own commands keep recording to the local store. Each side +resolves only its own connection, so environment references in the other connection, such as the +writer's password on a reader's laptop, need not be set. + +## Users and grants + +SQLBuild doesn't enforce grants; it lets you separate the roles. Give the writer the rights to +create and upgrade the store in its database, so the first command sets it up and later SQLBuild +versions upgrade it without anyone running DDL by hand: + +```sql +-- once, as an administrator: the users +CREATE USER recall_writer IDENTIFIED BY '...'; +CREATE USER recall_reader IDENTIFIED BY '...'; +GRANT CREATE DATABASE, CREATE TABLE, CREATE VIEW, DROP TABLE, ALTER ADD COLUMN, SELECT, INSERT + ON observability.* TO recall_writer; +GRANT SELECT ON observability.* TO recall_reader; +``` + +`DROP TABLE` is needed because ClickHouse replaces a view through a temporary table; Recall never +drops a table itself. + +| User | Rights | What happens | +|------|--------|--------------| +| Writer with the grants above | Create and upgrade the store | The first command creates the store; later versions upgrade it | +| Writer with only `SELECT` and `INSERT` | Write history | Works once the store exists. A missing or older store makes Recall print the statements an administrator should run | +| Reader with `SELECT` | Read history | `sqb recall query`, `schema` and `output` work, and can't change the store | + +A reader may be a standard read-only user (`SETTINGS readonly = 1`, or a settings profile with +it). Its queries run with the server's own settings; every other reader gets the +[settings SQLBuild pins](../adapters/clickhouse.md#settings-sqlbuild-pins). The views +return the same rows either way. `sqb recall query` runs with `readonly = 2`, which refuses writes +but allows query settings, unless the user is already read-only. A user that may not set the row +limit gets the same truncated result, cut off while reading. A setting Recall needs that the user +may not change fails with an error naming it: set it in the user's profile, or give the user +`readonly = 2`, which keeps it read-only. + +## Setting up and upgrading the store + +- **Setup.** On first use, the writer creates the database if it's missing, printing + ``Recall: created ClickHouse database `observability` for the store.`` once, then the store's + tables and views. An existing database is used as it is, whatever its engine. +- **Clusters.** The database is created on the server the writer connects to, never `ON CLUSTER`. + On a self-managed cluster whose default database engine isn't `Replicated`, create the database + yourself first with `ENGINE = Replicated(...)`; Recall leaves it alone. ClickHouse Cloud needs + nothing. `ON CLUSTER` stores aren't supported: use a single ClickHouse server or ClickHouse Cloud. +- **Without DDL rights.** If the writer may not create or upgrade the store, the command still + runs and exits as it otherwise would. Recall prints the statements an administrator should run, at + most once a day, starting with `CREATE DATABASE IF NOT EXISTS` when the database is missing. + `sqb recall schema --ddl` prints them again. A store behind the current version still receives + history, in the columns that exist. +- **Mixed versions.** A writer never replaces the views of a store that a newer SQLBuild version + created. +- **Many writers.** Many concurrent commands can write one store. A repeated delivery is counted + once. A record written again with different content keeps the stored row, and the rejected copy + is listed in the `conflicts` view. +- **Writes.** History is sent about every five seconds and when the command ends, with the same shutdown + budget and exit-code guarantees as the local store. + +## Retention in ClickHouse + +Retention belongs to the store. Every table has a TTL of `retention`; SQL text is kept a week +longer, and for as long as a kept invocation references it. The views hide expired rows at once, +but ClickHouse drops expired parts in the background, so rows may stay on disk for a while. + +When a project's `retention` differs from the tables' TTL, Recall warns once a day with the +`ALTER TABLE … MODIFY TTL` and view statements that would apply it. It never changes the TTL +itself. + +## Reading a ClickHouse store + +`sqb recall query`, `schema` and `output` read the ClickHouse store, so queries use ClickHouse SQL. +`--json` output includes the server's `rows_read`, `bytes_read` and `query_id`. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/recall.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/recall.md index 4c7cd9a531..9a6586ede0 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/recall.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/recall.md @@ -25,7 +25,7 @@ Online: https://sqlbuild.com/docs/concepts/observability/recall/ - Relationship to compute logs Recall records each `sqb` invocation in a local SQLite store, or in a shared -[ClickHouse store](#sharing-history-in-clickhouse), so you can ask what happened after the fact: which command ran, how it ended, which statements it executed and how long they took, and +[ClickHouse store](recall-clickhouse.md), so you can ask what happened after the fact: which command ran, how it ended, which statements it executed and how long they took, and exactly what it printed. Recall is on by default and needs no setup. ## An investigation, start to finish @@ -304,115 +304,9 @@ prints a notice and records nothing. ## Sharing history in ClickHouse -Point `write_connection` and `read_connection` at ClickHouse connections to keep one history for a -whole team. Every machine -that writes to it, including CI and scheduled production runs, becomes queryable from anywhere: -see what last night's production build ran and printed without access to the machine, find the -first run where an audit started failing across everyone's runs, or compare a production run with -your own by `target`, `git_revision` or `sqlbuild_version`. - -The connection needs `adapter = "clickhouse"` when the project uses another adapter. Recall creates -and upgrades its own store on the first write: the `database` when it does not exist yet, then its -tables, views and schema version. A writer that may run that DDL needs no other setup. - -```toml -[connections.history_writer] -adapter = "clickhouse" -host = "${env:CLICKHOUSE_HOST}" -user = "recall_writer" -password = "${env:RECALL_WRITER_PASSWORD}" -database = "observability" - -[connections.history_reader] -adapter = "clickhouse" -host = "${env:CLICKHOUSE_HOST}" -user = "recall_reader" -password = "${env:RECALL_READER_PASSWORD}" -database = "observability" - -[recall] -write_connection = "history_writer" # every command writes history; creates the store on first use -read_connection = "history_reader" # sqb recall query, schema and output (SELECT only) -retention = "30d" -``` - -`sqb recall query`, `schema` and `output` use only `read_connection` and never create or change -anything in the store, so someone investigating production runs needs no credentials that could -rewrite or delete its history. Both keys may name one connection when separate users are not -wanted. A machine that only reads shared history, such as a laptop investigating production runs, -sets only `read_connection`; its own commands keep recording to the local store. Each side -resolves only its own connection, so environment references in the other connection, such as the -writer's password on a reader's laptop, need not be set. - -### Users and grants - -SQLBuild doesn't enforce grants; it lets you separate the roles. Give the writer the rights to -create and upgrade the store in its database, so the first command sets it up and later SQLBuild -versions upgrade it without anyone running DDL by hand: - -```sql --- once, as an administrator: the users -CREATE USER recall_writer IDENTIFIED BY '...'; -CREATE USER recall_reader IDENTIFIED BY '...'; -GRANT CREATE DATABASE, CREATE TABLE, CREATE VIEW, DROP TABLE, ALTER ADD COLUMN, SELECT, INSERT - ON observability.* TO recall_writer; -GRANT SELECT ON observability.* TO recall_reader; -``` - -`DROP TABLE` is needed because ClickHouse replaces a view through a temporary table; Recall never -drops a table itself. - -| User | Rights | What happens | -|------|--------|--------------| -| Writer with the grants above | Create and upgrade the store | The first command creates the store; later versions upgrade it | -| Writer with only `SELECT` and `INSERT` | Write history | Works once the store exists. A missing or older store makes Recall print the statements an administrator should run | -| Reader with `SELECT` | Read history | `sqb recall query`, `schema` and `output` work, and can't change the store | - -A reader may be a standard read-only user (`SETTINGS readonly = 1`, or a settings profile with -it). Its queries run with the server's own settings; every other reader gets the -[settings SQLBuild pins](../adapters/clickhouse.md#settings-sqlbuild-pins). The views -return the same rows either way. `sqb recall query` runs with `readonly = 2`, which refuses writes -but allows query settings, unless the user is already read-only. A user that may not set the row -limit gets the same truncated result, cut off while reading. A setting Recall needs that the user -may not change fails with an error naming it: set it in the user's profile, or give the user -`readonly = 2`, which keeps it read-only. - -### Setting up and upgrading the store - -- **Setup.** On first use, the writer creates the database if it's missing, printing - ``Recall: created ClickHouse database `observability` for the store.`` once, then the store's - tables and views. An existing database is used as it is, whatever its engine. -- **Clusters.** The database is created on the server the writer connects to, never `ON CLUSTER`. - On a self-managed cluster whose default database engine isn't `Replicated`, create the database - yourself first with `ENGINE = Replicated(...)`; Recall leaves it alone. ClickHouse Cloud needs - nothing. `ON CLUSTER` stores aren't supported: use a single ClickHouse server or ClickHouse Cloud. -- **Without DDL rights.** If the writer may not create or upgrade the store, the command still - runs and exits as it otherwise would. Recall prints the statements an administrator should run, at - most once a day, starting with `CREATE DATABASE IF NOT EXISTS` when the database is missing. - `sqb recall schema --ddl` prints them again. A store behind the current version still receives - history, in the columns that exist. -- **Mixed versions.** A writer never replaces the views of a store that a newer SQLBuild version - created. -- **Many writers.** Many concurrent commands can write one store. A repeated delivery is counted - once. A record written again with different content keeps the stored row, and the rejected copy - is listed in the `conflicts` view. -- **Writes.** History is sent about every five seconds and when the command ends, with the same shutdown - budget and exit-code guarantees as the local store. - -### Retention in ClickHouse - -Retention belongs to the store. Every table has a TTL of `retention`; SQL text is kept a week -longer, and for as long as a kept invocation references it. The views hide expired rows at once, -but ClickHouse drops expired parts in the background, so rows may stay on disk for a while. - -When a project's `retention` differs from the tables' TTL, Recall warns once a day with the -`ALTER TABLE … MODIFY TTL` and view statements that would apply it. It never changes the TTL -itself. - -### Reading a ClickHouse store - -`sqb recall query`, `schema` and `output` read the ClickHouse store, so queries use ClickHouse SQL. -`--json` output includes the server's `rows_read`, `bytes_read` and `query_id`. +Recall writes to a local SQLite store by default. To keep one history for a team, CI and production +runs, point it at ClickHouse; see +[Sharing Recall in ClickHouse](recall-clickhouse.md). ## Filtering what Recall stores diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/selection-and-staleness.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/selection-and-staleness.md index 309f5c2af5..b291fc5e07 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/selection-and-staleness.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/selection-and-staleness.md @@ -6,19 +6,39 @@ Online: https://sqlbuild.com/docs/concepts/planning/selection-and-staleness/ -Change detection is selection-aware. When you scope a run with `--select`, SQLBuild only runs (and only displays) the resources you selected, but it still reasons about the *whole* graph to keep the result honest. The key case it handles: a selected model whose upstream changed but is **not** in the selection. +When you select part of the project with `--select`, SQLBuild runs only what you selected, but it +still checks the whole graph. The case it watches for: a selected model whose upstream changed but +isn't selected, so the model would build on stale input. ## Only selected resources appear in the plan -For a scoped run like `sqb dbt build --select agg_daily_revenue`, the plan shows only the selected resources, not their passive upstream closure. SQLBuild still walks upstreams internally for change propagation (a selected model flips to run when an upstream changed), but those passive upstreams are not listed as plan entries or counted in the header. +The plan lists only the selected resources, not the upstreams they read. SQLBuild still looks at +those upstreams: a selected model runs when an upstream changed. `--full-refresh` also applies only +to the selected models. -`--full-refresh` is likewise scoped to the selected models, not the expanded upstream closure. +## Stale upstreams -## Stale warnings instead of silent partial rebuilds +`sqb plan` and `sqb build` run every selected model, even when an unselected upstream changed. Pass +`--selection-diagnostics` to list those stale upstreams. After changing `stg_orders`, +`sqb plan --select fact_orders --selection-diagnostics` shows: -If a selected model has an upstream that changed but is **outside the selection**, rebuilding the selected model alone would run it on top of a stale upstream and silently produce a result that looks current but isn't. SQLBuild does not do that. It leaves the selected model current (it is not rebuilt) and emits a stale warning naming the changed upstreams, with the selector you can use to incorporate them. +``` +Warnings (1) +└── fact_orders + ├── selected model 'fact_orders' will build on 1 stale upstream(s) not selected for rebuild: + ├── - stg_orders + └── rebuild the closure to refresh them: --select +fact_orders +``` -Suppose `stg_orders` (upstream) changed, and you select only the downstream `agg_daily_revenue`: +To include the changed upstreams, select the closure: `--select +fact_orders` builds `stg_orders` +and everything between it and `fact_orders`. + +A changed seed outside the selection is treated the same way as a changed model. + +## With dbt + +`sqb dbt plan`, `run` and `build` are stricter. A selected model whose upstream changed outside the +selection is not rebuilt: it stays current, and the plan warns, naming the changed upstreams. ```bash sqb dbt plan --select agg_daily_revenue @@ -40,23 +60,7 @@ Warnings (1) (e.g. +model) to incorporate it ``` -The model stays current rather than being rebuilt on a stale input, and the warning tells you exactly which upstreams changed and how to pull them in. - -`sqb plan` and `sqb build` work differently: they run every selected model, even when an -unselected upstream changed. Pass `--selection-diagnostics` to list those stale upstreams. After -changing `stg_orders`, `sqb plan --select fact_orders --selection-diagnostics` shows: - -``` -Warnings (1) -└── fact_orders - ├── selected model 'fact_orders' will build on 1 stale upstream(s) not selected for rebuild: - ├── - stg_orders - └── rebuild the closure to refresh them: --select +fact_orders -``` - -## Incorporating the changed upstreams - -Use a closure selector to include the changed upstreams in the run. With `+agg_daily_revenue`, SQLBuild pulls in the changed `stg_orders`, cascades the change through the intermediate models, and rebuilds: +Select the closure to pull in the changed upstreams and rebuild: ```bash sqb dbt plan --select +agg_daily_revenue @@ -75,11 +79,3 @@ Plan ready (8 selected resources) model.jaffle_analytics.int_order_payments upstream changed model.jaffle_analytics.stg_order_statuses upstream changed ``` - -## Seeds follow the same rule - -Seed changes are tracked the same selection-aware way. A changed seed that is outside the current selection does not trigger a partial rebuild of its dependents; instead the dependents are left current and a stale warning is emitted, exactly as for models. This keeps a scoped run from quietly building on a seed that the run didn't reload. - -## Why this matters - -A run that reports success but silently built a model on stale inputs is a correctness hazard - the data looks fresh but isn't. Surfacing staleness as an explicit warning (and refusing the misleading partial rebuild) keeps scoped runs honest: you either get a result built on current inputs, or a clear warning telling you it would not be, with the fix. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/source-freshness.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/source-freshness.md index 8c6d8a9161..d6d95a21f6 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/source-freshness.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/source-freshness.md @@ -12,7 +12,9 @@ are affected and choose the correct incremental action. ## Configuration -Source freshness is configured per source in `sources/*.yml` with a `freshness:` block. See [Sources: Source freshness](../sources.md#source-freshness) for the full configuration reference. +Configure freshness per source with a `freshness:` block, including `lag_tolerance` for timestamps +that move without the data changing. See [Sources](../sources.md#source-freshness) for the +fields. ## How observations work @@ -24,15 +26,11 @@ During planning, SQLBuild observes the current data version of each source that Sources without explicit `freshness:` config are auto-observed using the `adapter` strategy if the adapter supports table metadata and the source has a physical table (not an expression source, not a managed source). -Adapter metadata for all sources is fetched in one batched lookup, but each source gets its own -result. When one source can't be observed, only that source is unknown; the others keep their -freshness. The `Freshness metadata` progress line reports the outcome, for example -`Freshness metadata OK (257 sources, 3 unknown)`. The plan adds one warning per reason, listing up -to 10 sources, such as `source freshness unknown (unavailable) for raw_events, raw_sessions: ...`. -Sources without freshness configuration that the adapter can't observe are not warned about. -Every unknown source is listed with its reason and message under `unknown_source_details` in the -`direct_source_freshness` metadata of `sqb plan --json`; run `sqb freshness` for the same detail -in text form. +When one source can't be observed, only that source is unknown; the others keep their freshness. +The plan warns once per reason, for example +`source freshness unknown (unavailable) for raw_events, raw_sessions: ...`. Sources without +freshness configuration that the adapter can't observe are not warned about. `sqb freshness` shows +each unknown source with its reason, as does `unknown_source_details` in `sqb plan --json`. ## Missing source tables @@ -51,15 +49,10 @@ sources and sources with a loader are exempt, because SQLBuild creates their tab them. Expression sources, sources without a schema, and wildcard tables are also exempt. A missing table that only unselected models read does not fail the plan; its freshness is reported as unknown. -The check first looks for each source in the warehouse listing that planning already reads for -source columns, so it normally issues no extra queries. A source the listing does not find (for -example because of identifier case, a stream, a materialized view, or a table in another database) -is then probed directly with `SELECT 1 FROM WHERE 1=0`, using the same relation name the -model SQL reads. Probes for several such sources run concurrently, within the same limit as other -warehouse inspection reads. Only a not-found error from that probe counts as missing, including a -missing schema, catalog, or attached database; any other error (for example a connection error) -fails planning with that error. On Snowflake the probe may need a running warehouse, but it only -runs for sources the listing did not find. +SQLBuild finds missing tables in the warehouse listing it already reads, then probes the sources it +didn't find there, such as streams or tables in another database, with +`SELECT 1 FROM WHERE 1=0`. On Snowflake the probe may need a running warehouse. Only a not-found error counts as missing; any other error, +such as a lost connection, fails planning with that error. Snowflake reports a missing table and a table the current role can't read with the same error, and its listing hides tables the role can't access. For those sources `S405` says the table "does not @@ -70,10 +63,6 @@ A table that a model's own `pre_hook` creates does not exist yet when the plan i fails with `S405`. Declare such a table as a managed source or a source with a loader, or create it before planning. -## Lag tolerance - -For timestamp-based freshness, `lag_tolerance` controls how much the observed value can drift before being considered a real change. If the current timestamp is within the tolerance of the previous observation, the source is treated as unchanged. This is useful for sources where the freshness timestamp moves by seconds or minutes on every query but the underlying data hasn't meaningfully changed. - ## State storage Source freshness observations are appended to `_sqlbuild_source_freshness` in each target schema. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md index a3bea98e9c..1d1b1add4a 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md @@ -1,8 +1,8 @@ -# Project Configuration +# Configuration reference -> Configure your SQLBuild project with sqlbuild_project.toml and sqlbuild_local.toml. +> Every key in sqlbuild_project.toml and sqlbuild_local.toml. Online: https://sqlbuild.com/docs/concepts/project-configuration/ @@ -28,32 +28,11 @@ Online: https://sqlbuild.com/docs/concepts/project-configuration/ - Project Python - sqlbuild_local.toml -SQLBuild projects are configured with two files in the project root: +Every key in `sqlbuild_project.toml`, which you commit, and `sqlbuild_local.toml`, which each +developer keeps out of Git. For a walkthrough, see [Set up a project](../project-setup.md). -- **`sqlbuild_project.toml`** - shared project configuration, committed to version control -- **`sqlbuild_local.toml`** - local developer overrides, gitignored - -Macro, constant, and enum visibility is not configured in either file. Declaration scopes use filesystem conventions; see [Declaration Scopes](declaration-scopes.md). - -Most projects need only one committed `sqlbuild_project.toml`. Define shared targets such -as `dev` and `prod` there, including clone policies and team-wide defaults. Do not maintain -separate complete project files for each environment. - -Create `sqlbuild_local.toml` only when a developer or execution environment needs different -target selection, credentials, schemas, adapter settings, or variables. SQLBuild loads it -automatically and merges its explicitly configured values over the shared project config. - -```text -project/ - sqlbuild_project.toml # committed: shared targets and behavior - sqlbuild_local.toml # gitignored: this developer's overrides -``` - -Add the local file to `.gitignore`: - -```gitignore -sqlbuild_local.toml -``` +Where macros, constants and enums are visible is set by folders, not in these files; see +[Declaration Scopes](declaration-scopes.md). ## sqlbuild_project.toml @@ -92,7 +71,7 @@ materialized = "view" | Field | Description | |-------|-------------| -| `name` | Project name. Used in fingerprint tracking and manifest generation. | +| `name` | Project name | | `adapter` | Database adapter: `duckdb`, `motherduck`, `snowflake`, `bigquery`, `databricks`, `postgres`, `sqlite`, `sqlserver`, or `clickhouse`. See [Adapters](adapters.md). | ### Optional fields @@ -161,11 +140,9 @@ explicitly: database = ":memory:" ``` -A `database` or `schema` present in a named connection is connection/session metadata only; -it does not satisfy the mandatory namespace strategy for a named target. Put the target's -authoritative database and schema on `[targets.]`. SQLBuild validates connection -references while loading configuration, without opening a warehouse connection, and reports -an unknown `targets..connection` name as an offline configuration error. +A `database` or `schema` on a connection doesn't decide where models are built; set them on the +target. A target that names an unknown connection fails when the configuration loads, before +anything connects. ## Targets @@ -180,8 +157,8 @@ A target is a named build context - the database and schema you build into, plus | `vars` | Target-specific project variables | | `defer_sources_to` | Target name to read managed source data from (see [Loaders](python-nodes/loaders.md#source-deferral)) | | `clone` | Clone policy (see below) | -| `missing_migration_origin` | What a build does when a declared `migrate_from` origin does not exist in this target: `allow` (default), `require_confirmation`, or `deny` (see below) | -| `warehouses` | Default Snowflake warehouse per command group, `build` and `query` (see below) | +| `missing_migration_origin` | What a build does when a declared `migrate_from` origin does not exist in this target: `allow` (default), `require_confirmation`, or `deny`. See [Missing migration origins](models/migrations.md#missing-migration-origins) | +| `warehouses` | Snowflake warehouse per command group, `build` and `query`. See [Command-group warehouses](adapters/snowflake.md#command-group-warehouses) | | `concurrency` | Worker limit for commands run against this target, replacing `[settings] concurrency` (see below) | ```toml @@ -276,97 +253,6 @@ The value must be an integer of at least `1`; anything else fails with `D001`. ` `8 configured limit (targets.prod.concurrency)`. `sqb scenario test` has no `--target` option and uses the active target's value. -### Command-group warehouses - -Builds and ad-hoc inspection usually need different compute. A Snowflake target can choose a -default warehouse for each command group, so `sqb build` runs on a large warehouse while -`sqb query` and `sqb diff` stay on a small one: - -```toml -[connections.warehouse] -account = "my_org-my_account" -warehouse = "ANALYTICS_WH" # fallback for every command - -[targets.dev] -connection = "warehouse" -database = "ANALYTICS" -schema = "DEV" - -[targets.dev.warehouses] -build = "BUILD_WH" # commands in the build group -query = "ADHOC_WH" # commands in the query group -``` - -Each command connects with the first warehouse set in this order: - -1. `--warehouse ` on the command line, for that invocation only -2. The target's group warehouse, `[targets..warehouses] build` or `query` -3. The connection's `warehouse` -4. The Snowflake user's default warehouse - -Both groups are optional; a target without `warehouses` uses the connection's `warehouse`. A group -warehouse takes precedence over the connection `warehouse` in every file. In particular, a -shared group overrides a connection `warehouse` you set in `sqlbuild_local.toml`; to change a -group locally, set the group itself, or pass `--warehouse` for one invocation. Group values -accept the same `${ENV:...}` and project-variable (`${my_var}`) templates as connection values. -Group values and `--warehouse` must be Snowflake identifiers: unquoted (`ADHOC_WH`) or -double-quoted (`"Adhoc WH"`). - -Every connection a command opens uses its group, so one invocation never mixes warehouses: -`sqb build` plans, inspects, and executes on the `build` warehouse, and `sqb diff` creates its -temporary comparison tables on the `query` warehouse. The exception is `sqb dbt`: it applies the -`build` group to SQLBuild-owned work, while dbt nodes keep the warehouse from their dbt profile. - -| Group | Commands | -|-------|----------| -| `build` | `audit`, `build`, `check`, `clone`, `dbt` (SQLBuild-owned work), `load`, `plan`, `seed` | -| `query` | `contract`, `cost`, `debug`, `diff`, `freshness`, `janitor`, `query`, `scenario`, `test` | - -Other commands, such as `compile`, `lineage`, and `format`, never connect to a warehouse. -Integrations that run builds, such as Dagster, invoke `sqb build` and use the `build` group. -Standalone `sqb test`, `sqb scenario` and `sqb janitor` use the `query` group because they work on -fixture data or catalogue metadata; tests and audits that run inside `sqb build` use the `build` group. - -Each developer can point ad-hoc commands at their own warehouse without editing shared -configuration: - -```toml -# sqlbuild_local.toml -[targets.dev.warehouses] -query = "ALICE_XS_WH" -``` - -Local group values override the project value for the same group only; the other group keeps -its project value. `sqb debug` shows the resolved warehouse for each group and where it came -from, and `sqb build --verbose` shows it in the execution header. - -Groups need an adapter with a session warehouse: the Snowflake adapter, or a custom adapter -that extends it. `warehouses` in `sqlbuild_project.toml` is checked against the project's -adapter, and `warehouses` in `sqlbuild_local.toml` against the adapter you actually run. Setting -them for an adapter without a session warehouse, or passing `--warehouse`, is an error rather -than a silently ignored setting. The one exception is a local `adapter` override, for example a -developer running a Snowflake project on DuckDB: shared groups then don't apply, and -`sqb debug` reports them as `not used (adapter duckdb)`. - -### Missing migration origins - -A `migrate_from` declaration, on a model or a column, names a relation or column that exists in the targets where the rename is pending. A target that never built the old name, such as a fresh development schema, has nothing to migrate. `missing_migration_origin` decides what happens there: - -| Value | Behavior | -|-------|----------| -| `allow` | Default. The plan warns (`M102` for models, `M109` for columns), nothing is migrated, and the model builds as if `migrate_from` were absent. | -| `require_confirmation` | The build asks for confirmation on a terminal, and stops otherwise; pass `--allow-missing-migration-origin` to confirm in non-interactive runs. | -| `deny` | The build stops with `M102` or `M109`. Use it for targets where a missing origin means the wrong target or a lost relation. | - -```toml -[targets.prod] -missing_migration_origin = "deny" -``` - -`sqlbuild_local.toml` can override the value for a target. - -The policy covers only origins with no build history in the target. A model origin that SQLBuild built in this target but can no longer see, because of permissions or because it was dropped outside SQLBuild, always stops the build with `M102`. Column origins have no such history, so `M109` always follows the policy. `sqb dbt run` and `sqb dbt build` apply the policy too, but have no confirmation flag: they ask on a terminal and stop otherwise. In non-interactive dbt runs, build the affected models with `sqb build --allow-missing-migration-origin` and leave them out of the dbt selection, or set the target's `missing_migration_origin` to `allow`. - ### Clone policies Targets can declare whether they allow cloning to or from: @@ -391,9 +277,8 @@ Both policies default to `false`. `sqb clone --from prod --to dev` requires `all ### Execution limits -Named targets can set optional limits for `sqb build`, and per-item time limits for unit tests and -scenarios. Limits are useful for development targets where an unexpectedly broad selector or -long-running command should stop instead of consuming an unbounded amount of warehouse compute. +A target can cap what one `sqb build` runs, so a broad selector on a development target stops +instead of running up warehouse cost. A target without this section has no limits. ```toml [targets.dev] @@ -412,93 +297,16 @@ Automated tools must request approval before changing this policy. """ ``` -All fields are optional. A target without an `execution_limits` section has no limits, so -production or orchestrated targets need no placeholder values. - -#### Model limit - -`max_models` is a positive integer. SQLBuild expands selectors and counts model entries that would -execute. When the count exceeds the target limit, the build exits before clone, load, or model -execution begins. The error reports the target, selected count, configured maximum, configuration -key, and confirms that no warehouse changes were made. - -`sqb plan` remains unrestricted and can be used to inspect a broad selection safely. - -#### Duration limit - -`max_duration` is a positive fixed duration such as `30m`, `2h`, or `90s`. It covers planning and -warehouse-mutating build phases after target resolution. SQLBuild stops dispatching work at the -deadline, bounds each warehouse statement by the remaining time where the warehouse supports a -statement limit, and cancels statements still running at the deadline. - -Every built-in adapter enforces the limit: - -| Adapter | Warehouse statement limit | Cancellation at the deadline | Longest accepted duration | -|---------|---------------------------|------------------------------|---------------------------| -| Snowflake | `STATEMENT_TIMEOUT_IN_SECONDS` per statement | Driver cancel request for the running query | 7 days | -| DuckDB, MotherDuck | None | Connection interrupt | About 24 days | -| PostgreSQL | `SET LOCAL statement_timeout` inside transactions; statements outside a transaction rely on cancellation | Driver cancel (`cancel_safe`) | About 24 days | -| BigQuery | Job timeout (`job_timeout_ms`) on each query job | `jobs.cancel`; reported as requested, not confirmed | About 24 days | -| Databricks | `STATEMENT_TIMEOUT` session setting, capped at 48 hours | Cursor cancel; reported as requested, not confirmed | About 24 days | -| SQL Server | None; the driver's query timeout is process-wide | Driver cancel | About 24 days | -| SQLite | None | Connection interrupt | About 24 days | - -A project adapter that can't cancel a running statement is rejected for `max_duration`, rather than -showing a limit that may leave warehouse work running. Under a per-item timeout, its items stop -before their next statement and report the cancellation as not supported. - -A duration failure may come after earlier models completed. Its error therefore says it is a -timeout, unlike a model-limit failure before execution, and that warehouse changes may already -have happened. - -#### Per-item time limits - -Unit test, scenario and model limits (`test_timeout`, `scenario_timeout` and a model's -`max_duration`) count running time only. The clock starts at the item's first warehouse statement, -so rendering and setup before it do not count, and it stops during rollback and cleanup, which always -run even after the limit. A timeout is reported only when the limit passed before the item's work -finished. Where a warehouse reports queueing separately, queued time is excluded: - -| Adapter | How the item limit is enforced | Queued time | -|---------|--------------------------------|-------------| -| Snowflake | `STATEMENT_TIMEOUT_IN_SECONDS` per statement, with a client-side backstop after a grace margin of half the limit (at least 30 seconds) | Excluded by the warehouse limit; the backstop counts it | -| DuckDB, MotherDuck | Connection interrupt at the limit | Not applicable | -| PostgreSQL | `SET LOCAL statement_timeout` inside transactions; cancellation at the limit | Lock waits count | -| BigQuery | `jobs.cancel` at the limit; no job timeout, because a job timeout counts from job creation | Excluded: the clock stops while a job is pending | -| Databricks | `STATEMENT_TIMEOUT` session setting and cursor cancel | May count | -| SQL Server | Driver cancel at the limit | Lock waits count | -| SQLite | Connection interrupt at the limit | Waits for the write lock count | - -When a model contains unit tests or nested work with its own limit, the earliest limit applies and -cancels whatever is running under it. After a timeout interrupted a statement, SQLBuild checks the -connection before giving it to the next item and replaces it if it is no longer usable. - -#### Unit test and scenario timeouts - -`test_timeout` and `scenario_timeout` set the per-test and per-scenario time limit on this target, -replacing `[settings] test_timeout` and `[settings] scenario_timeout`. Each takes a positive fixed -duration or `"disabled"`. See [Unit test and scenario limits](#unit-test-and-scenario-limits). - -#### Custom remediation - -`remediation` is optional project-authored text. SQLBuild appends it to model-count, duration, and -unsupported-adapter errors after the factual error details. This lets each project define its own -approval or recovery process without SQLBuild recommending that users weaken local policy. - -#### Local overrides - -The same section is accepted in `sqlbuild_local.toml` and follows normal field-by-field local target -precedence: - -```toml -[targets.dev.execution_limits] -max_models = 200 -max_duration = "90m" -``` +| Field | Description | +|-------|-------------| +| `max_models` | Most models one build may run. A larger selection stops before anything runs; `sqb plan` has no limit | +| `max_duration` | Longest one build may run, such as `"45m"`. At the deadline SQLBuild cancels the statements still running | +| `test_timeout` | Time limit for each unit test on this target, or `"disabled"`; replaces `[settings] test_timeout` | +| `scenario_timeout` | Time limit for each scenario on this target, or `"disabled"`; replaces `[settings] scenario_timeout` | +| `remediation` | Text added to limit errors, such as how your team approves a larger build | -This permits an intentional developer override while keeping the shared target policy visible in -`sqlbuild_project.toml`. Other target fields and any execution-limit fields omitted locally continue -to inherit their project values. +See [Limits](../cli/build.md#limits) for how each warehouse enforces them. `sqlbuild_local.toml` +accepts the same section: the fields it sets replace the project's, and the rest are inherited. ### Diff limits @@ -670,12 +478,12 @@ delta_compile = true |-------|---------|-------------| | `sql_analysis` | `true` | Enable SQL syntax analysis, column binding, type inference, output inference, and semantic validation at compile time | | `require_sql_analysis` | `false` | Allow `sql_analysis false` (in a MODEL header or a `[path_defaults]` entry) only on models whose SQL the parser cannot read, and reject `[settings] sql_analysis = false` in `sqlbuild_local.toml`. Project file only. See [SQL analysis](models/configuration.md#sql-analysis). | -| `query_change_tracking` | `true` | Track query fingerprints for change detection. A fingerprint covers the SQL tokens of the compiled query and ignores whitespace, comments, and the letter case of reserved keywords and built-in function names, so formatting-only edits do not rebuild. Every other word, including non-reserved keywords, identifiers, aliases, JSON path keys and user-defined function names, counts exactly, as do string literals, quoted identifiers, numbers, and optimizer hints. | -| `check_declared_columns` | `true` | Controls whether column declarations on models without a `contract` declaration activate static shape/nullability validation. `true` checks that every declared column is in the SELECT and checks its nullability; `false` treats columns as metadata and audit attachment unless the model declares `contract enforced`. Model-level `contract enforced` and `contract none` override this setting. Explicit type enforcement remains independent. See [Contracts](models/contracts.md). | +| `query_change_tracking` | `true` | Rebuild a model when its compiled query changes. Whitespace, comments, and the case of reserved keywords and built-in function names don't count as changes | +| `check_declared_columns` | `true` | Check that models without a `contract` setting output their declared columns, and that `nullable false` columns can't be null. `false` makes those columns documentation and audit targets only. See [Contracts](models/contracts.md) | | `concurrency` | `1` | Default number of worker connections for commands that run resources in parallel, such as `sqb build`, `sqb test`, `sqb audit`, and `sqb seed`, and for `sqb janitor` actions on warehouses that run DDL concurrently; a target's `concurrency` and `--concurrency` override it (see [Target concurrency](#target-concurrency)) | -| `microbatch_concurrency` | `false` | Explicitly permit models with `batch_concurrency > 1`; concurrent batches use immutable coordination facts | +| `microbatch_concurrency` | `false` | Allow models with `batch_concurrency > 1` | | `auto_load_sources` | `true` | Automatically run source loaders before building dependent models during `sqb build`. See [Loaders](python-nodes/loaders.md). | -| `table_promotion_mode` | adapter default | `staged` (CTAS to staging, audit, then promote) or `immediate` (CTAS directly to target, then audit) | +| `table_promotion_mode` | `staged` | `staged` builds a table under a staging name and replaces the old one only after audits pass; `immediate` builds in place and audits afterwards. See [Table](models/materializations.md#table) | | `default_audit_severity` | `error` | Default severity for audits: `warn` or `error` | | `default_audit_run_scope` | `delta_and_final` | Default run scope for audits: `final` or `delta_and_final` | | `unit_test_max_real_models` | `10` | Most real (unmocked) models one SQL unit test may execute; a larger test fails compilation. See [Unit test and scenario limits](#unit-test-and-scenario-limits). | @@ -687,22 +495,17 @@ delta_compile = true | Setting | Limits | Details | |---------|--------|---------| -| `unit_test_max_real_models` | Real (unmocked) models one SQL unit test may run; a larger test fails compilation and names each model, the path that pulls it in, and the inputs to mock | [Real-model limit](testing.md#real-model-limit) | -| `test_timeout` | Each unit test's warehouse time, from its first statement | [Time limits](testing.md#time-limits), including precedence | +| `unit_test_max_real_models` | Real (unmocked) models one SQL unit test may run; a larger test fails compilation and names each model, the path that pulls it in, and the inputs to mock | [Real-model limit](testing-reference.md#real-model-limit) | +| `test_timeout` | Each unit test's warehouse time, from its first statement | [Time limits](testing-reference.md#time-limits), including precedence | | `scenario_timeout` | Each scenario's warehouse time; no default | Below | The scenario limit that applies is the first one set of `--scenario-timeout`, `SCENARIO(timeout "...")`, `[targets..execution_limits] scenario_timeout`, and `[settings] scenario_timeout`. At the limit, SQLBuild cancels the scenario's running statement, reports it as `TIMEOUT`, and continues with independent items, as for -[unit tests](testing.md#time-limits). Every value is a positive fixed duration such as +[unit tests](testing-reference.md#time-limits). Every value is a positive fixed duration such as `"45s"` or `"10m"`, or `"disabled"`. `sqlbuild_local.toml` accepts all three settings. -### Table promotion mode - -- **`staged`** (default for most adapters): Materializes into a staging table, runs audits, then swaps into the target. If audits fail, the production table is untouched. -- **`immediate`**: Creates the table directly at the target location. Audits run after materialization. Simpler but no pre-promotion safety net. - ## Rules Rules configuration belongs in the shared `sqlbuild_project.toml` so local and CI compilation use @@ -895,24 +698,8 @@ critical_tags = ["critical"] ## dbt -Configuration for running SQLBuild alongside an existing dbt project: - -```toml -[dbt] -project_dir = "../dbt_project" -profiles_dir = "../profiles" -target_path = "../dbt_project/target" -target = "dev" -``` - -| Field | Description | -|-------|-------------| -| `project_dir` | Path to the dbt project root (where `dbt_project.yml` lives) | -| `profiles_dir` | Path to the directory containing `profiles.yml` | -| `target_path` | Path to dbt's `target/` directory (where `manifest.json` is written) | -| `target` | dbt target name override (optional) | - -Paths can be absolute or relative to the SQLBuild project root. See [Using SQLBuild with dbt](dbt-compatibility/overview.md) for setup and usage details. +`[dbt]` points SQLBuild at an existing dbt project. See +[Using SQLBuild with dbt](dbt-compatibility/overview.md#configuration) for its fields. ## Skills @@ -996,7 +783,3 @@ configured fields. Target blocks merge the same way and may override the connect `database`, `schema`, `loader_schema`, variables, source deferral, and policy fields. Unspecified values continue to come from `sqlbuild_project.toml`; a local reference to an unknown merged connection still fails offline during configuration loading. - -This replaces the common dbt pattern of switching profiles or setting environment variables -to change targets. Each developer sets their target, named connection, and preferences once in -`sqlbuild_local.toml` and it persists across sessions. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/loaders.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/loaders.md index 8d2415eb3c..46c9fdb96e 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/loaders.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/loaders.md @@ -175,10 +175,9 @@ Existing rows matching the unique key are updated; new rows are inserted. Snowflake reads a backslash inside a string literal as an escape, so SQLBuild doubles every backslash in the values a loader returns. A value such as `C:\orders\2026` is stored exactly as -returned. Earlier releases wrote those backslashes unescaped, so Snowflake dropped them or read them -as escapes. If a `merge` or `delete_insert` unique key, or a value an `append` load is -deduplicated by, contains a backslash, rows loaded by an earlier release will not match newly -loaded rows. Rewrite or delete those rows before relying on the key. +returned. If a `merge` or `delete_insert` unique key, or a value an `append` load is deduplicated +by, contains a backslash, rows stored with single backslashes won't match newly loaded rows. +Rewrite or delete those rows before relying on the key. ## Self-managed loaders diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/resource-identities.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/resource-identities.md index 8e15f0e76f..6df3063a67 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/resource-identities.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/resource-identities.md @@ -1,64 +1,47 @@ -# Resource Identities +# Naming rules -> Canonical names for SQLBuild resources, selectors, state, and integrations. +> Names of models, tests, macros and other SQLBuild resources are lowercase snake_case. Online: https://sqlbuild.com/docs/concepts/resource-identities/ -SQLBuild resource identities use lowercase ASCII snake_case. A name must start with a lowercase -letter, end with a lowercase letter or digit, and contain only lowercase letters, digits, and -underscores. Consecutive underscores are valid, so established names such as -`commerce__mart_v_order` remain canonical. - -This contract applies to models, seeds, sources, SQL and Python functions, generic and singular -audits, attached audit definitions and instance names, SQL tests and parameterized cases, -scenarios, SQL and Python hooks, macros, model schemas, enums, constants, materializations, -providers, event exporters, loaders, tasks, assets, and checks. Private scoped declarations use -exactly one leading underscore, such as `_country_codes`; public declarations must not use a -leading underscore. - -Provider classes are the one conventional derivation: when `provider_name` is omitted, -`AnalyticsApiProvider` resolves to `analytics_api_provider`. An explicit `provider_name` is an -authored identity and must already be canonical snake_case; SQLBuild does not normalize it. - -Names derived from files use the filename stem. For example, `models/daily_orders.sql` defines -the model identity `daily_orders`, while `audits/generic/expression_is_true.sql` defines the -generic audit identity `expression_is_true`. Directories organize resources but do not change -their names. - -Physical warehouse identifiers are separate from SQLBuild resource identities. Database, schema, -table, model alias, column, tag, group, and directory names retain their existing adapter-specific -contracts and do not need to follow this profile. - -## Invalid names - -Compilation fails during discovery with `D016` when an authored identity is not canonical: +Names of models, tests, macros and every other SQLBuild resource are lowercase snake_case, such as +`daily_orders`. SQLBuild never renames anything for you; it suggests the right spelling. Renaming a +file by hand creates a new resource with no history, so use [`sqb rename`](../cli/rename.md) to +keep it. ```text error[D016]: Invalid model identity 'DailyOrders' in models/DailyOrders.sql; use snake_case 'daily_orders' ``` -Discovery also fails with `D016` when the project directory, or a model, unit test, scenario, source -or schema YAML file it reads, has a path that is not valid UTF-8, including a directory above the -file. The error shows each invalid byte as a `\xNN` escape; rename the path to valid UTF-8. Other -files with such names, such as notes or data files, and directories that hold none of those files -are ignored. Commands that tolerate unreadable files, such as `sqb scope`, report the file and -continue. +A name starts with a lowercase letter, ends with a lowercase letter or digit, and contains only +lowercase letters, digits and underscores. Double underscores are fine, as in +`commerce__mart_v_order`. + +A resource named after its file takes the file name without the extension: +`models/daily_orders.sql` is the model `daily_orders`. Folders organise files but are not part of +the name. + +## Renaming a resource -SQLBuild suggests a corrected spelling but never silently normalizes an identity. Silent -normalization would make selectors, manifests, persisted execution state, and integration keys -disagree about which resource ran. +[`sqb rename`](../cli/rename.md) and `sqb mv` rename a model or column, update every reference, and +keep the warehouse table's history. See [Refactoring](refactoring.md). -## Migrating existing projects +A rename you make by hand creates a new resource. Its build history, fingerprints and audit results +start fresh, and selectors and integrations that use the old name, such as Dagster asset keys, need +updating too. Compile before you build. -1. Rename file-derived resources and explicit `name` values to snake_case. -2. Update `__ref`, `__seed`, `__source`, function, hook, macro, audit, test, and dependency - references. -3. Update selectors and external integrations, including Dagster asset/check keys, that use the - old identity. -4. Compile before building. +## Details -A rename intentionally creates a new resource identity. Existing fingerprints, audit history, -and other persisted state under the old name are not silently reassigned to the new resource. +- A private scoped declaration starts with exactly one underscore, such as `_country_codes`. Public + declarations don't start with an underscore. See [Declaration scopes](declaration-scopes.md). +- A provider class without a `provider_name` gets a snake_case name from its class name: + `AnalyticsApiProvider` is `analytics_api_provider`. An explicit `provider_name` must already be + snake_case. +- Warehouse names are not affected: databases, schemas, tables, aliases, columns, tags and groups + follow the warehouse's own rules. +- Paths SQLBuild reads, such as models, tests, scenarios and source or schema YAML files, must be + valid UTF-8, including the folders above them. Otherwise compilation fails with `D016`, showing + each invalid byte as `\xNN`. `sqb scope` reports such a file and continues. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules.md index 2f7792a7a3..67b8419a87 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules.md @@ -13,10 +13,10 @@ Online: https://sqlbuild.com/docs/concepts/rules/ - SQL Rules - Built-in and custom Rules - Dialect support -- One definition of project validity +- When Rules run - Rules compared with tests, audits, and contracts - Findings and intentional exceptions -- Performance is part of the design +- Performance - Continue Code review is an expensive place to keep enforcing a decision the team has already made. @@ -199,20 +199,11 @@ built-in Rules that did not run. `L007` never changes an exit code, including fo `sqb build`: it reports a gap in SQLBuild's dialect support, not a problem in your SQL. A syntax error in a supported dialect still fails the Rules that read it. -## One definition of project validity +## When Rules run -`sqb compile` is authoritative. SQLBuild evaluates a project in this order: - -```text -mandatory compiler correctness -→ selected native built-in Rules -→ selected custom Python Rules -→ artifact completion -``` - -Build and planning commands use the same compiler path. Configured findings stop execution before a -warehouse connection is opened. Compile artifacts are not completed from a project that failed its -Rules. +Rules run in every `sqb compile`, `sqb plan` and `sqb build`, after SQLBuild's own checks pass. If a +Rule fails, nothing runs in the warehouse. See +[Execution and caching](rules/execution-and-caching.md). ## Rules compared with tests, audits, and contracts @@ -242,24 +233,12 @@ comments, and mandatory compiler correctness cannot be excepted at all. See [Findings and exceptions](rules/findings-and-exceptions.md) for configuration examples. -## Performance is part of the design - -Model Rules are evaluated and cached per model. Editing one model invalidates affected subjects -rather than every invocation. Project Rules run once over the complete project view and are the -explicit choice for genuinely project-wide requirements, with broader invalidation. SQLBuild tracks -the inputs each Rule depends on so cached findings are reused only while those inputs remain valid. - -The default SQL Rules lint every compiled model, so a cold compile without a cache costs more than -the compiler alone: about 14% more wall time and 28% more CPU on a generated 3,000-model project. -Warm and one-model edit compiles reuse cached findings and cost the same. Set `select = []` to -measure or run the compiler without them. - -Required CI exercises custom Rule packs over generated projects, checking compilation time, -memory, and cache invalidation. Separate 20- and 100-Rule profiles measure the cost of custom -validation at 5,000-model scale. +## Performance -See [Benchmarks](../benchmarks.md#compiler-guards-in-ci) for the benchmark profiles and reproduction commands, -and [Execution and caching](rules/execution-and-caching.md) for the Rules lifecycle. +Rules are cached per model, so editing one model re-checks only what it affects. The default SQL +Rules add about 14% to a cold compile and nothing to a warm one. See +[Benchmarks](../benchmarks.md#compiler-guards-in-ci) and +[Execution and caching](rules/execution-and-caching.md). ## Continue diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/configuration-and-selection.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/configuration-and-selection.md index 7c746213ae..0304325230 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/configuration-and-selection.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/configuration-and-selection.md @@ -76,7 +76,7 @@ allowed_tests = ["customers__reviewed_empty_input"] `SQBRTEST203` (`empty-input-only-test`) accepts `allowed_tests`, a list of SQL test names that are reviewed exceptions. Listed tests are not flagged and still count toward `SQBRTEST202`'s `min_tests_per_model`. Entries that name no existing test are reported as stale. See -[Empty-input tests](../testing.md#empty-input-tests). +[Empty-input tests](../testing-reference.md#empty-input-tests). `SQBRGRAPH103` and `SQBRPROJECT107` accept `published_layers`, and `SQBRPROJECT107` accepts `allowed_qualifiers`, a list of `{ qualifier, reason }` or `{ paths, reason }` tables with an @@ -145,19 +145,19 @@ final AS ( SELECT event_id, order_id, status FROM final ``` -Each `SELECT` is checked on its own: its select list, `FROM`, `JOIN ... ON`, `WHERE`, `GROUP BY`, -`HAVING`, `QUALIFY` and `ORDER BY`. A nested `SELECT`, subquery, CTE body or set-operation branch is -a separate scope. Calls are grouped by function and by the column reference as written, with names -compared the way the warehouse resolves them: on Snowflake unquoted names fold to upper case and -quoted names compare exactly, so `"VALUE"` matches `value` but `"Value"` does not; on DuckDB, -BigQuery and Databricks names compare case-insensitively whether or not they are quoted. References -are not resolved, so `value` and `e.value`, or `events.value` and -`e.value`, are different groups. The argument may be wrapped in parentheses, `LOWER`, `UPPER`, -`TRIM` without trim characters, `CAST`, `TRY_CAST` or a single-argument `COALESCE`; a wrapper with -more arguments, such as `COALESCE(e.value, e.backup)`, is not a plain column and is not counted. One -finding is reported at the second call and names the count with the column as written, for example -`TRY_PARSE_JSON(e.value) is parsed 3 times in one SELECT`. Different parse functions, such as -`PARSE_JSON` and `TRY_PARSE_JSON`, are counted separately. There is no automatic fix. +The finding is reported at the second call, for example +`TRY_PARSE_JSON(e.value) is parsed 3 times in one SELECT`. There is no automatic fix. + +Details: + +- Each `SELECT` is checked on its own. A subquery, CTE body or set-operation branch is a separate + `SELECT`. +- Calls count as the same when they use the same function on the same column as written: `value` + and `e.value` are different. Names compare as the warehouse compares them: on Snowflake, + unquoted names fold to upper case; on DuckDB, BigQuery and Databricks, case never matters. +- The column may be wrapped in parentheses, `LOWER`, `UPPER`, `TRIM`, `CAST`, `TRY_CAST` or a + one-argument `COALESCE`. `COALESCE(e.value, e.backup)` is not a plain column and isn't counted. +- `PARSE_JSON` and `TRY_PARSE_JSON` are counted separately. Known limitations: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/execution-and-caching.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/execution-and-caching.md index 3fcbc90a80..1b6a0d345a 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/execution-and-caching.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/execution-and-caching.md @@ -16,10 +16,7 @@ SQLBuild evaluates a project in this order: Native built-ins always run before custom Rules. Structured compile output reports built-in and custom durations separately, together with cache hits and misses. -## Authoritative enforcement - -`sqb compile` runs the complete configured Rules set. Build and execution commands use the same -compiler path and reject Rule findings before opening a warehouse connection. +## What a selection checks `sqb rules run` is intentionally focused. Use it to inspect one exact code or family, but run the normal compiler before treating the complete project as valid. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/findings-and-exceptions.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/findings-and-exceptions.md index e9598a47a9..833de2dc37 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/findings-and-exceptions.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/findings-and-exceptions.md @@ -12,7 +12,6 @@ Online: https://sqlbuild.com/docs/concepts/rules/findings-and-exceptions/ - Relation-level exceptions - Path- and resource-scoped ignores - No inline suppressions -- Removed settings - Mandatory correctness - Machine-readable findings @@ -133,14 +132,6 @@ error[SQBRSUPPRESSION000]: Inline Rule suppressions are not supported; declare t Copy the entry into `sqlbuild_project.toml`, adjust the reason for a reviewer, and delete the comment. -## Removed settings - -`[rules] allow_exceptions` was removed. Configured exceptions are always allowed because the project -file is the review gate, and inline suppressions no longer exist. A project that still sets -`allow_exceptions` fails configuration loading with an error that explains the removal: delete the -line, keep reviewed deviations as `[[rules.rule_exceptions]]` entries, and use `[rules] ignore` to -turn a Rule off for the whole project. - ## Mandatory correctness Mandatory compiler correctness is not configurable and cannot be suppressed. A project must first diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/scenarios.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/scenarios.md index 091f62fa3c..720a178609 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/scenarios.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/scenarios.md @@ -34,7 +34,7 @@ Cost is a second reason to move a check into a scenario. A unit test inlines eve chain into one SQL statement, so the warehouse compiles and optimises the whole chain at once, and that cost can grow much faster than the query's size. A scenario materialises each model step by step, so each statement is compiled separately, as in a real build. SQLBuild limits how many real -models one unit test may execute (see [Cost guards](testing.md#cost-guards)); when a test +models one unit test may execute (see [Cost guards](testing-reference.md#cost-guards)); when a test reaches that limit because it checks behaviour across several models, it may belong here. ## Authoring scenarios diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/semantic-compilation.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/semantic-compilation.md index a9519eafde..810becb3f3 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/semantic-compilation.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/semantic-compilation.md @@ -1,6 +1,6 @@ -# Semantic Compilation +# What sqb compile checks > Offline reference, semantic and type checks that sqb compile runs before any warehouse work. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md index 97d0318f70..a3b3bba6e0 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md @@ -255,15 +255,10 @@ Accepts positive durations: `15m` (minutes), `2h` (hours), `1d` (days). If the c ### Auto-observation -Sources without an explicit `freshness:` block are auto-observed using the `adapter` strategy if: - -- The source has a physical table (not an expression source) -- The source is not managed -- The adapter supports table freshness metadata - -This means most unmanaged table sources get freshness tracking automatically on adapters that support it, with no configuration needed. - -Use [`sqb freshness`](../cli/freshness.md) to observe source freshness on demand without triggering a build. See [Source freshness](planning/source-freshness.md) for how freshness feeds into change-aware builds. +A table source without a `freshness:` block is observed with the `adapter` strategy when the adapter +supports it, unless it is managed. See [Source freshness](planning/source-freshness.md) +for how observations feed into plans, and [`sqb freshness`](../cli/freshness.md) to observe sources +without building. ### Freshness config reference diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing-reference.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing-reference.md new file mode 100644 index 0000000000..a5748e44ff --- /dev/null +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing-reference.md @@ -0,0 +1,447 @@ + + +# Testing reference + +> Test modes, parameterized cases, cursor windows, cost guards and time limits for SQL unit tests. + +Online: https://sqlbuild.com/docs/concepts/testing-reference/ + +## Contents + +- Test modes +- Repeating test logic +- Cursor windows in tests +- Empty-input tests +- Cost guards +- Skipping SQL analysis +- Details + +This page lists the less common unit test features. Start with [Unit tests](testing.md) +for writing and running a test. + +## Test modes + +Model tests mock inputs and compare model output. Three more modes test reusable logic directly, +without a model chain: + +| Mode | Actual CTE | Expected CTE | Also allowed | +|------|-----------|--------------|--------------| +| `model` | none: the model chain | `__expected__` | `__source__`, `__ref__`, `__seed__`, `__assert__`, `__macro__` | +| `macro` | `__macro_actual__` | `__macro_expected__` | Helper CTEs, `@py.name()` and `@sql.name()` calls in actual, CTE-list `@sql.name()` calls in the `WITH` list | +| `udf` | `__udf_actual__` | `__udf_expected__` | Helper CTEs, `__udf()` calls in actual | +| `table_fn` | `__table_fn_actual__` | `__table_fn_expected__` | Helper CTEs, `__table_fn()` calls in actual | + +SQLBuild infers the mode from the actual CTE, so `TEST (mode ...)` is optional. An explicit mode +that contradicts the CTEs is a compile error, and so is a prefix from another mode, such as +`__source__` in a macro test. + +Only the actual CTE calls the macro, UDF or table function under test. Helpers don't call it +either, so a helper of a macro test must not call `__udf()` or `__table_fn()`. Expected CTEs must +not call macros, UDFs or table functions, so the expected data stays independent and easy to read. + +### Macro tests + +Call the macro in `__macro_actual__` and give the result in `__macro_expected__`: + +```sql +TEST (mode macro, name "calculates_line_total_cents"); + +WITH +input_values AS ( + SELECT 950 AS price_cents, 3 AS quantity +), +__macro_actual__ AS ( + SELECT @py.line_total_cents("price_cents", "quantity") AS line_total_cents + FROM input_values +), +__macro_expected__ AS ( + SELECT 2850 AS line_total_cents +) +``` + +Macros expand at compile time. During `sqb build`, a macro test runs before any model that uses the +macro. + +[SQL macros](macros/sql-macros.md) are tested the same way. Call an expression macro in +`__macro_actual__`: + +```sql +TEST (mode macro, name "cents_to_dollars_rounds"); + +WITH +input_values AS (SELECT CAST(1250 AS BIGINT) AS amount_cents), +__macro_actual__ AS ( + SELECT @sql.cents_to_dollars(amount_cents) AS dollars FROM input_values +), +__macro_expected__ AS (SELECT CAST(12.5 AS DECIMAL(18, 2)) AS dollars) +``` + +Call a column-list macro as an item of the `__macro_actual__` select list, and pass a helper CTE as +each `RELATION` argument: + +```sql +TEST (mode macro, name "audit_columns_flag_updates"); + +WITH +input_rows AS ( + SELECT TIMESTAMP '2026-01-01' AS created_at, TIMESTAMP '2026-01-02' AS updated_at +), +__macro_actual__ AS (SELECT @sql.audit_columns(input_rows) FROM input_rows), +__macro_expected__ AS ( + SELECT TIMESTAMP '2026-01-01' AS created_at, TIMESTAMP '2026-01-02' AS updated_at, + TRUE AS was_updated +) +``` + +Call a CTE-list macro in the test's `WITH` list, and select from one of its CTEs in +`__macro_actual__`: + +```sql +TEST (mode macro, name "clean_orders_keeps_latest_live"); + +WITH +input_orders AS ( + SELECT 1 AS order_id, TIMESTAMP '2026-01-01' AS updated_at, FALSE AS is_deleted + UNION ALL SELECT 1, TIMESTAMP '2026-01-02', FALSE + UNION ALL SELECT 2, TIMESTAMP '2026-01-01', TRUE +), +@sql.clean_orders(input_orders) +__macro_actual__ AS (SELECT order_id, updated_at FROM latest), +__macro_expected__ AS (SELECT 1 AS order_id, TIMESTAMP '2026-01-02' AS updated_at) +``` + +This is the only place outside `__macro_actual__` where a macro test may call a macro. The call may +pass `__cte := name`, as in a model, to call the macro more than once or to rename its CTEs. Other +modes can't call a CTE-list macro in their `WITH` list. + +### UDF tests + +Call the UDF in `__udf_actual__` and give the result in `__udf_expected__`: + +```sql +TEST (name "detects_completed_orders"); + +WITH +input_values AS ( + SELECT 'completed' AS order_status + UNION ALL + SELECT 'pending' AS order_status +), +__udf_actual__ AS ( + SELECT + order_status, + __udf("udf__is_completed_order")(order_status) AS is_completed_order + FROM input_values +), +__udf_expected__ AS ( + SELECT 'completed' AS order_status, TRUE AS is_completed_order + UNION ALL + SELECT 'pending' AS order_status, FALSE AS is_completed_order +) +``` + +During `sqb build`, a UDF test runs after the function is created and before any model that uses +it. + +### Functions in `sqb test` + +`sqb test` tests the current version of your functions without a build. Before the tests run, +SQLBuild creates a temporary copy of every project UDF and table function the selected tests call, +and of the functions those call. The tests call the copies, so an edited function is tested without +`sqb build`, and the built function in the target schema is never created, replaced or dropped. The +copies disappear when the test connections close. + +| Adapter | Temporary copy | +|---|---| +| DuckDB, MotherDuck | `CREATE TEMP MACRO`, called as `temp.main.`. Python UDFs are registered on each test connection | +| PostgreSQL | SQL function in the session's `pg_temp` schema, called as `pg_temp.` | +| Snowflake | `CREATE TEMPORARY FUNCTION __sqb_test__` in the session's current schema, for SQL and Python UDFs and table functions | +| Databricks | `CREATE TEMPORARY FUNCTION __sqb_test__` for SQL functions | +| BigQuery, SQL Server, ClickHouse | No copy; tests call the built function (ClickHouse functions are server-wide) | +| SQLite | Not applicable; SQLite has no SQL functions | + +When a copy can't be made, because the adapter doesn't support it (a Databricks Python UDF, for +example) or the warehouse refuses (the role lacks permission, or the Snowflake session has no +current schema), the test calls the built function. If that function isn't built either, the test +fails with an error naming the function and the reason, and asks you to run `sqb build` first. A +function whose callee has no copy calls the built functions too, so a test never mixes copies and +built functions. + +Every test connection gets the same copies as the first one. If a later connection can't create a +copy, `sqb test` stops with `T006`, naming the function and the connection. + +On PostgreSQL, the copies live in one server session. Run `sqb test` for projects with functions +through a direct connection or a session-mode pool, not a transaction-mode connection pooler. + +### Table function tests + +Call the table function in `__table_fn_actual__` and give its rows in `__table_fn_expected__`: + +```sql +TEST (name "returns_customer_orders"); + +WITH +__table_fn_actual__ AS ( + SELECT order_id, order_status, is_completed_order + FROM __table_fn("table_fn__customer_orders")(1) +), +__table_fn_expected__ AS ( + SELECT 1 AS order_id, 'completed' AS order_status, TRUE AS is_completed_order + UNION ALL + SELECT 2 AS order_id, 'completed' AS order_status, TRUE AS is_completed_order +) +``` + +Table function tests run after the function is created. Models can't depend on table functions, so +these tests check the function on its own. + +## Repeating test logic + +| Pattern | Use when | Results | +|---------|----------|---------| +| Several `TEST` blocks | Cases need different SQL | One per block | +| A `VALUES` case table or macro | One comparison over all cases is enough | One for the table | +| Parameters and cases | Cases share a template but each needs its own result | One per named case | + +### Native independent cases + +Declare typed parameters and ordered, named cases in the header: + +```sql +TEST ( + name "order_status_maps_source_states", + parameters ( + source_status string, + expected_status string, + ), + cases ( + completed (source_status "completed", expected_status "completed"), + cancelled (source_status "cancelled", expected_status "excluded"), + pending (source_status "pending", expected_status "open"), + ), +); + +WITH +__source__raw__orders AS ( + SELECT 1 AS id, @param("source_status") AS status +), +__expected__stg_orders AS ( + SELECT 1 AS id, @param("expected_status") AS status +) +``` + +Each case reports on its own, in the order written: `order status: maps source states [completed]`, +`[cancelled]` and `[pending]`. Selecting `stg_orders` runs every case. Run one case with `--case`: + +```bash +sqb test --select stg_orders --case cancelled +``` + +An unknown case name lists the available ones. + +Parameter types are `string`, `integer`, `boolean`, `float` and exact `decimal`. Quote decimal +values, such as `tax_rate "0.2000"`, so they never pass through binary floating point. Booleans and +integers stay distinct. Raw SQL and collection parameters are rejected. + +A nullable parameter uses the expanded form, and works where SQL can infer the null's type: + +```sql +parameters ( + cancellation_reason (type string, nullable true), +), +cases ( + completed (cancellation_reason null), +) +``` + +Every case gives exactly the declared parameters, and every parameter appears as `@param("name")` +in the test. Missing, extra, incompatible, undeclared and unused values are rejected before +anything runs. + +Parameters are rendered first, as typed literals for the active adapter, then `@const` and `@enum` +references, then macros. A parameter can therefore be a macro argument, as in +`@py.order_fixture(@param("source_status"))`. Parameters work in every mode and keep every expected +model of a multi-model test. + +Text and JSON output show the test, the case and its typed values. Compile JSON, manifests, DAG +checks and compiled SQL record each case's source, block and fingerprint. Changing one case changes +only that case's fingerprint. + +Scenarios can't be parameterized: keep one scenario per coherent business world. + +### Aggregate SQL case tables + +When cases don't need their own results, a `VALUES` table is shorter: + +```sql +TEST(); + +WITH cases(case_name, source_status, expected_status) AS ( + VALUES + ('completed', 'completed', 'completed'), + ('cancelled', 'cancelled', 'excluded') +), +__expected__status_mapping AS ( + SELECT case_name, expected_status FROM cases +) +``` + +This gives one test result. Quoted strings in `VALUES` rows may contain commas, brackets and +parentheses; escape embedded quotes as usual in SQL. + +## Cursor windows in tests + +Models that call `__cursor_start()` and `__cursor_end()` run in a test over one wide window, +never the model's own `cursor_start` and `cursor_end`: + +| Cursor type | Window | +|-------------|--------| +| Timestamp | `1900-01-01 00:00:00` to `2999-12-31 00:00:00` | +| Integer | `-1000000000000000` to `1000000000000000` | + +The bounds render as the same typed literals as in a real run, so date arithmetic such as +`DATEADD('day', -7, __cursor_start()::DATE)` works and mocked rows aren't filtered out. +Microbatch models run once over the whole window. + +To test windowing itself, set the window in the header. Either key is optional; `cursor_start` is +inclusive and `cursor_end` exclusive: + +```sql +TEST (name "daily_orders_skips_rows_outside_window", cursor_start "2026-02-01", cursor_end "2026-02-03"); +``` + +The window applies to every model in the test that uses the cursor functions, and is checked +against each model's `cursor_type` and `cursor_grain`. Invalid, misaligned or inverted bounds, and a +window on a test whose models don't use the cursor functions, are compile errors. The compiled test +SQL shows the rendered window. + +## Empty-input tests + +A test whose mocks are all empty (`WHERE FALSE`, `WHERE 1 = 0`, `LIMIT 0` or `__EMPTY_FIXTURE()`) +and whose only checks are an empty `__expected__` CTE or a bare `SELECT ... FROM __ref("")` +assertion can't fail for a model whose rows come from its inputs. Rule `SQBRTEST203` rejects it, +and `SQBRTEST202` doesn't count it toward `min_tests_per_model`. Mock representative rows and +assert concrete output instead. + +An empty-input test is fine when it asserts concrete output, such as a global aggregate that must +return one zero row: + +```sql +TEST (name "order_summary__empty_inputs_return_zero_row"); + +WITH +__source__raw__orders AS ( + SELECT CAST(NULL AS INTEGER) AS id, CAST(NULL AS INTEGER) AS amount WHERE FALSE +), +__expected__order_summary AS ( + SELECT 0 AS order_count, 0 AS total_amount +) +``` + +Assertions that filter for invalid rows, compare with expected rows, aggregate or limit their result +are not bare existence checks. List reviewed exceptions by test name in the project file; see +[Rule options](rules/configuration-and-selection.md#rule-options). + +## Cost guards + +A unit test runs the real SQL of every unmocked model between its mocks and the model under test as +one warehouse statement. The warehouse compiles and optimises that whole chain at once, and the cost +can grow much faster than the query's size. SQLBuild bounds each test in two ways. + +### Real-model limit + +`[settings] unit_test_max_real_models` (default `10`) caps how many real models one test may run, +counting the model under test. It can also be set in `sqlbuild_local.toml`. A test over the limit +fails compilation, in `sqb test` and in `sqb build` when it runs tests, before any warehouse +connection opens: + +```text +error[S000]: Invalid SQL test fixtures: +- SQL unit test 'customer_totals_rollup' executes 3 real models (limit 2). + Test file: tests/unit/customer_totals_rollup.sql + Model under test: customer_totals + +Real models executed, and the dependency path that pulls each one in: + customer_totals model under test + order_totals customer_totals -> order_totals + stg_orders customer_totals -> order_totals -> stg_orders + +Options: + 1. Mock the direct inputs of the model under test to cut the chain: + __ref__order_totals AS (SELECT ...) + 2. If this test checks behaviour across several models, consider a scenario + (tests/scenarios/). A unit test inlines every real model into one statement, + so the warehouse compiles and optimises the whole chain at once; a scenario + builds each model step by step, compiling each statement separately. + 3. Raise the limit in sqlbuild_project.toml (or sqlbuild_local.toml): + [settings] + unit_test_max_real_models = 3 +``` + +`sqb test` shows each test's count on its row, such as `3 real models`, so a growing chain is +visible early. Macro, UDF and table function tests run no models and aren't counted. + +### Time limits + +Each unit test has a time limit, `30s` by default. The first of these that is set applies: + +1. `--test-timeout` on `sqb test` or `sqb build` +2. `TEST(timeout "45s")` in the test file +3. `[targets..execution_limits] test_timeout` +4. `[settings] test_timeout`, in `sqlbuild_local.toml` or `sqlbuild_project.toml` +5. The `30s` default + +Each takes a positive fixed duration, such as `"45s"` or `"10m"`, or `"disabled"`. + +The clock starts at the test's first warehouse statement. Rendering, warehouse queueing where the +warehouse reports it, and cleanup after the comparison don't count. See +[Per-item time limits](../cli/build.md#per-item-time-limits) for how each +adapter enforces the limit, and [Duration limit](../cli/build.md#duration-limit) +for how each adapter cancels a running statement. + +At the limit, SQLBuild cancels the running statement, reports the test as `TIMEOUT`, and continues +with the other tests. The report gives the elapsed time, the limit, where it was set, the +cancellation outcome and the query IDs it saw. When the test runs real models beyond the model +under test, it also lists them, with the same options as the real-model limit: + +```text +SQL unit test 'order_scans_kept' timed out after 1.0s (limit 1s). + Limit set by: TEST() timeout in tests/unit/order_scans_kept.sql + Cancellation: confirmed; the running statement stopped + +This test inlines 1 real model beyond the model under test into one statement: + stg_orders + +Options: + 1. Mock the inlined models closest to the model under test to cut the chain, + for example __ref__stg_orders AS (SELECT ...). + 2. If this test checks behaviour across several models, consider a scenario + (tests/scenarios/). A unit test inlines every real model into one statement, + so the warehouse compiles and optimises the whole chain at once; a scenario + builds each model step by step, compiling each statement separately. + 3. To allow more time, set TEST(timeout "3s") on this test, [settings] test_timeout = "3s" + in sqlbuild_project.toml (or sqlbuild_local.toml), or --test-timeout for one run. +``` + +A test that runs only the model under test reports the timeout, the limit, where it was set and how +to change it, without the scenario advice. + +Scenarios have their own limit, `scenario_timeout`; see +[Scenario timeouts](project-configuration.md#unit-test-and-scenario-limits). + +## Skipping SQL analysis + +Compile checks test fixtures and assertions like model SQL. For SQL the parser can't read, add +`sql_analysis false` to the `TEST` header; the test still runs. Tests of a model with +`sql_analysis false`, and every test in a `--no-sql-analysis` run, aren't analysed. With +`[settings] require_sql_analysis = true`, the opt-out is accepted only on test SQL that can't be +parsed; see [Requiring SQL analysis](models/configuration.md#requiring-sql-analysis). + +## Details + +- SQL tests need SQLBuild's SQL analysis for the adapter's dialect. On an adapter without it, such + as a custom adapter without `sql_analysis_dialect_name`, planning SQL tests fails with a message + that says SQL tests are unavailable for that dialect. +- On DuckDB, MotherDuck, PostgreSQL and Snowflake, a test evaluates each compared model's full + output once. If that output fails, for example because a fixture value can't be converted in a + column the expected rows leave out, the test errors with the warehouse's message. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md index 39133b20a6..067ba1cb30 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md @@ -1,8 +1,8 @@ -# Testing +# Unit tests -> SQL unit tests and multi-model tests with macro support, assertions, and model chaining. +> Check a model's logic on rows you write, by mocking its inputs and comparing its output. Online: https://sqlbuild.com/docs/concepts/testing/ @@ -19,14 +19,8 @@ Online: https://sqlbuild.com/docs/concepts/testing/ - Macros, enums and constants in tests - Macro mocking - Assertions -- Cursor windows in tests -- Empty-input tests -- Test modes -- Repeating test logic - Running tests -- Cost guards -- Skipping SQL analysis -- Details +- More test features A unit test checks a model's logic on rows you write. You mock the model's inputs with CTEs, SQLBuild runs the model's real SQL over them, and the test passes when the output matches the rows @@ -61,11 +55,6 @@ Tests can chain several models, mock models, seeds, macros and table functions, that must hold for every row, and test macros, UDFs and table functions directly. For end-to-end tests over physical warehouse relations, see [Scenarios](scenarios.md). -SQL tests are planned with SQLBuild's native parser for the adapter's SQL analysis dialect. On an -adapter whose dialect has no native parser, including a custom adapter without -`sql_analysis_dialect_name`, a project with SQL tests fails to plan them, with a message that says -SQL tests are unavailable for that dialect. - ## How tests work The test above: @@ -79,9 +68,9 @@ The test above: Every test starts with a `TEST()` header. All of its fields are optional: `TEST();` is a complete header, and a test without a `name` is named after its file. Fields name the test, set its -[mode](#test-modes) or [cursor window](#cursor-windows-in-tests), declare -[parameters](#native-independent-cases), set a [time limit](#time-limits), or -[skip SQL analysis](#skipping-sql-analysis). +[mode](testing-reference.md#test-modes) or [cursor window](testing-reference.md#cursor-windows-in-tests), declare +[parameters](testing-reference.md#native-independent-cases), set a [time limit](testing-reference.md#time-limits), or +[skip SQL analysis](testing-reference.md#skipping-sql-analysis). A test ends after its last CTE. A trailing `SELECT 1` is accepted; nothing else may follow the last CTE. @@ -158,7 +147,7 @@ __expected__stg_orders AS ( Test names are unique across the project: a test can't share a name with another test, a scenario, a model, a source, a seed, a function or a Python node. Name a test after the behaviour it checks, such as `orders_status_rules`, rather than after the model. The cases of one -[parameterized test](#native-independent-cases) share its name. +[parameterized test](testing-reference.md#native-independent-cases) share its name. ## CTE conventions @@ -310,7 +299,7 @@ The plan lists the mocked sources, refs and seeds, the real models in execution models, and any unsatisfied leaf dependencies. A mocked ref is called out, because its real SQL won't run. Inspection exits non-zero when the plan has unresolved dependencies. -Each real model adds to the cost of the test; see [Cost guards](#cost-guards). +Each real model adds to the cost of the test; see [Cost guards](testing-reference.md#cost-guards). ## Mocking refs and seeds @@ -410,7 +399,7 @@ __expected__customer_order_totals AS ( The fixture name must match a declared table function. Fixtures can read helper CTEs, and an assertion can read a table-function fixture directly. To test the table function itself, use a -[table function test](#table-function-tests). +[table function test](testing-reference.md#table-function-tests). ## Multiple expected models @@ -537,329 +526,6 @@ An assertion must read a model the test runs. A test whose checks read only mock `__ref("stg_orders")` in a test that also defines `__ref__stg_orders`, has nothing to run, and compile rejects it with `P013`. Mock the model's inputs instead, as above, or remove the test. -## Cursor windows in tests - -Models that call `__cursor_start()` and `__cursor_end()` run in a test over one wide window, -never the model's own `cursor_start` and `cursor_end`: - -| Cursor type | Window | -|-------------|--------| -| Timestamp | `1900-01-01 00:00:00` to `2999-12-31 00:00:00` | -| Integer | `-1000000000000000` to `1000000000000000` | - -The bounds render as the same typed literals as in a real run, so date arithmetic such as -`DATEADD('day', -7, __cursor_start()::DATE)` works and mocked rows aren't filtered out. -Microbatch models run once over the whole window. - -To test windowing itself, set the window in the header. Either key is optional; `cursor_start` is -inclusive and `cursor_end` exclusive: - -```sql -TEST (name "daily_orders_skips_rows_outside_window", cursor_start "2026-02-01", cursor_end "2026-02-03"); -``` - -The window applies to every model in the test that uses the cursor functions, and is checked -against each model's `cursor_type` and `cursor_grain`. Invalid, misaligned or inverted bounds, and a -window on a test whose models don't use the cursor functions, are compile errors. The compiled test -SQL shows the rendered window. - -## Empty-input tests - -A test whose mocks are all empty (`WHERE FALSE`, `WHERE 1 = 0`, `LIMIT 0` or `__EMPTY_FIXTURE()`) -and whose only checks are an empty `__expected__` CTE or a bare `SELECT ... FROM __ref("")` -assertion can't fail for a model whose rows come from its inputs. Rule `SQBRTEST203` rejects it, -and `SQBRTEST202` doesn't count it toward `min_tests_per_model`. Mock representative rows and -assert concrete output instead. - -An empty-input test is fine when it asserts concrete output, such as a global aggregate that must -return one zero row: - -```sql -TEST (name "order_summary__empty_inputs_return_zero_row"); - -WITH -__source__raw__orders AS ( - SELECT CAST(NULL AS INTEGER) AS id, CAST(NULL AS INTEGER) AS amount WHERE FALSE -), -__expected__order_summary AS ( - SELECT 0 AS order_count, 0 AS total_amount -) -``` - -Assertions that filter for invalid rows, compare with expected rows, aggregate or limit their result -are not bare existence checks. List reviewed exceptions by test name in the project file; see -[Rule options](rules/configuration-and-selection.md#rule-options). - -## Test modes - -Model tests mock inputs and compare model output. Three more modes test reusable logic directly, -without a model chain: - -| Mode | Actual CTE | Expected CTE | Also allowed | -|------|-----------|--------------|--------------| -| `model` | none: the model chain | `__expected__` | `__source__`, `__ref__`, `__seed__`, `__assert__`, `__macro__` | -| `macro` | `__macro_actual__` | `__macro_expected__` | Helper CTEs, `@py.name()` and `@sql.name()` calls in actual, CTE-list `@sql.name()` calls in the `WITH` list | -| `udf` | `__udf_actual__` | `__udf_expected__` | Helper CTEs, `__udf()` calls in actual | -| `table_fn` | `__table_fn_actual__` | `__table_fn_expected__` | Helper CTEs, `__table_fn()` calls in actual | - -SQLBuild infers the mode from the actual CTE, so `TEST (mode ...)` is optional. An explicit mode -that contradicts the CTEs is a compile error, and so is a prefix from another mode, such as -`__source__` in a macro test. - -Only the actual CTE calls the macro, UDF or table function under test. Helpers don't call it -either, so a helper of a macro test must not call `__udf()` or `__table_fn()`. Expected CTEs must -not call macros, UDFs or table functions, so the expected data stays independent and easy to read. - -### Macro tests - -Call the macro in `__macro_actual__` and give the result in `__macro_expected__`: - -```sql -TEST (mode macro, name "calculates_line_total_cents"); - -WITH -input_values AS ( - SELECT 950 AS price_cents, 3 AS quantity -), -__macro_actual__ AS ( - SELECT @py.line_total_cents("price_cents", "quantity") AS line_total_cents - FROM input_values -), -__macro_expected__ AS ( - SELECT 2850 AS line_total_cents -) -``` - -Macros expand at compile time. During `sqb build`, a macro test runs before any model that uses the -macro. - -[SQL macros](macros/sql-macros.md) are tested the same way. Call an expression macro in -`__macro_actual__`: - -```sql -TEST (mode macro, name "cents_to_dollars_rounds"); - -WITH -input_values AS (SELECT CAST(1250 AS BIGINT) AS amount_cents), -__macro_actual__ AS ( - SELECT @sql.cents_to_dollars(amount_cents) AS dollars FROM input_values -), -__macro_expected__ AS (SELECT CAST(12.5 AS DECIMAL(18, 2)) AS dollars) -``` - -Call a column-list macro as an item of the `__macro_actual__` select list, and pass a helper CTE as -each `RELATION` argument: - -```sql -TEST (mode macro, name "audit_columns_flag_updates"); - -WITH -input_rows AS ( - SELECT TIMESTAMP '2026-01-01' AS created_at, TIMESTAMP '2026-01-02' AS updated_at -), -__macro_actual__ AS (SELECT @sql.audit_columns(input_rows) FROM input_rows), -__macro_expected__ AS ( - SELECT TIMESTAMP '2026-01-01' AS created_at, TIMESTAMP '2026-01-02' AS updated_at, - TRUE AS was_updated -) -``` - -Call a CTE-list macro in the test's `WITH` list, and select from one of its CTEs in -`__macro_actual__`: - -```sql -TEST (mode macro, name "clean_orders_keeps_latest_live"); - -WITH -input_orders AS ( - SELECT 1 AS order_id, TIMESTAMP '2026-01-01' AS updated_at, FALSE AS is_deleted - UNION ALL SELECT 1, TIMESTAMP '2026-01-02', FALSE - UNION ALL SELECT 2, TIMESTAMP '2026-01-01', TRUE -), -@sql.clean_orders(input_orders) -__macro_actual__ AS (SELECT order_id, updated_at FROM latest), -__macro_expected__ AS (SELECT 1 AS order_id, TIMESTAMP '2026-01-02' AS updated_at) -``` - -This is the only place outside `__macro_actual__` where a macro test may call a macro. The call may -pass `__cte := name`, as in a model, to call the macro more than once or to rename its CTEs. Other -modes can't call a CTE-list macro in their `WITH` list. - -### UDF tests - -Call the UDF in `__udf_actual__` and give the result in `__udf_expected__`: - -```sql -TEST (name "detects_completed_orders"); - -WITH -input_values AS ( - SELECT 'completed' AS order_status - UNION ALL - SELECT 'pending' AS order_status -), -__udf_actual__ AS ( - SELECT - order_status, - __udf("udf__is_completed_order")(order_status) AS is_completed_order - FROM input_values -), -__udf_expected__ AS ( - SELECT 'completed' AS order_status, TRUE AS is_completed_order - UNION ALL - SELECT 'pending' AS order_status, FALSE AS is_completed_order -) -``` - -During `sqb build`, a UDF test runs after the function is created and before any model that uses -it. - -### Functions in `sqb test` - -`sqb test` tests the current version of your functions without a build. Before the tests run, -SQLBuild creates a temporary copy of every project UDF and table function the selected tests call, -and of the functions those call. The tests call the copies, so an edited function is tested without -`sqb build`, and the built function in the target schema is never created, replaced or dropped. The -copies disappear when the test connections close. - -| Adapter | Temporary copy | -|---|---| -| DuckDB, MotherDuck | `CREATE TEMP MACRO`, called as `temp.main.`. Python UDFs are registered on each test connection | -| PostgreSQL | SQL function in the session's `pg_temp` schema, called as `pg_temp.` | -| Snowflake | `CREATE TEMPORARY FUNCTION __sqb_test__` in the session's current schema, for SQL and Python UDFs and table functions | -| Databricks | `CREATE TEMPORARY FUNCTION __sqb_test__` for SQL functions | -| BigQuery, SQL Server, ClickHouse | No copy; tests call the built function (ClickHouse functions are server-wide) | -| SQLite | Not applicable; SQLite has no SQL functions | - -When a copy can't be made, because the adapter doesn't support it (a Databricks Python UDF, for -example) or the warehouse refuses (the role lacks permission, or the Snowflake session has no -current schema), the test calls the built function. If that function isn't built either, the test -fails with an error naming the function and the reason, and asks you to run `sqb build` first. A -function whose callee has no copy calls the built functions too, so a test never mixes copies and -built functions. - -Every test connection gets the same copies as the first one. If a later connection can't create a -copy, `sqb test` stops with `T006`, naming the function and the connection. - -On PostgreSQL, the copies live in one server session. Run `sqb test` for projects with functions -through a direct connection or a session-mode pool, not a transaction-mode connection pooler. - -### Table function tests - -Call the table function in `__table_fn_actual__` and give its rows in `__table_fn_expected__`: - -```sql -TEST (name "returns_customer_orders"); - -WITH -__table_fn_actual__ AS ( - SELECT order_id, order_status, is_completed_order - FROM __table_fn("table_fn__customer_orders")(1) -), -__table_fn_expected__ AS ( - SELECT 1 AS order_id, 'completed' AS order_status, TRUE AS is_completed_order - UNION ALL - SELECT 2 AS order_id, 'completed' AS order_status, TRUE AS is_completed_order -) -``` - -Table function tests run after the function is created. Models can't depend on table functions, so -these tests check the function on its own. - -## Repeating test logic - -| Pattern | Use when | Results | -|---------|----------|---------| -| Several `TEST` blocks | Cases need different SQL | One per block | -| A `VALUES` case table or macro | One comparison over all cases is enough | One for the table | -| Parameters and cases | Cases share a template but each needs its own result | One per named case | - -### Native independent cases - -Declare typed parameters and ordered, named cases in the header: - -```sql -TEST ( - name "order_status_maps_source_states", - parameters ( - source_status string, - expected_status string, - ), - cases ( - completed (source_status "completed", expected_status "completed"), - cancelled (source_status "cancelled", expected_status "excluded"), - pending (source_status "pending", expected_status "open"), - ), -); - -WITH -__source__raw__orders AS ( - SELECT 1 AS id, @param("source_status") AS status -), -__expected__stg_orders AS ( - SELECT 1 AS id, @param("expected_status") AS status -) -``` - -Each case reports on its own, in the order written: `order status: maps source states [completed]`, -`[cancelled]` and `[pending]`. Selecting `stg_orders` runs every case. Run one case with `--case`: - -```bash -sqb test --select stg_orders --case cancelled -``` - -An unknown case name lists the available ones. - -Parameter types are `string`, `integer`, `boolean`, `float` and exact `decimal`. Quote decimal -values, such as `tax_rate "0.2000"`, so they never pass through binary floating point. Booleans and -integers stay distinct. Raw SQL and collection parameters are rejected. - -A nullable parameter uses the expanded form, and works where SQL can infer the null's type: - -```sql -parameters ( - cancellation_reason (type string, nullable true), -), -cases ( - completed (cancellation_reason null), -) -``` - -Every case gives exactly the declared parameters, and every parameter appears as `@param("name")` -in the test. Missing, extra, incompatible, undeclared and unused values are rejected before -anything runs. - -Parameters are rendered first, as typed literals for the active adapter, then `@const` and `@enum` -references, then macros. A parameter can therefore be a macro argument, as in -`@py.order_fixture(@param("source_status"))`. Parameters work in every mode and keep every expected -model of a multi-model test. - -Text and JSON output show the test, the case and its typed values. Compile JSON, manifests, DAG -checks and compiled SQL record each case's source, block and fingerprint. Changing one case changes -only that case's fingerprint. - -Scenarios can't be parameterized: keep one scenario per coherent business world. - -### Aggregate SQL case tables - -When cases don't need their own results, a `VALUES` table is shorter: - -```sql -TEST(); - -WITH cases(case_name, source_status, expected_status) AS ( - VALUES - ('completed', 'completed', 'completed'), - ('cancelled', 'cancelled', 'excluded') -), -__expected__status_mapping AS ( - SELECT case_name, expected_status FROM cases -) -``` - -This gives one test result. Quoted strings in `VALUES` rows may contain commas, brackets and -parentheses; escape embedded quotes as usual in SQL. - ## Running tests Tests run during `sqb build` in DAG order, before the model they test is built. Run them on their @@ -870,103 +536,7 @@ sqb test sqb test --select stg_orders ``` -## Cost guards - -A unit test runs the real SQL of every unmocked model between its mocks and the model under test as -one warehouse statement. The warehouse compiles and optimises that whole chain at once, and the cost -can grow much faster than the query's size. SQLBuild bounds each test in two ways. - -### Real-model limit - -`[settings] unit_test_max_real_models` (default `10`) caps how many real models one test may run, -counting the model under test. It can also be set in `sqlbuild_local.toml`. A test over the limit -fails compilation, in `sqb test` and in `sqb build` when it runs tests, before any warehouse -connection opens: - -```text -error[S000]: Invalid SQL test fixtures: -- SQL unit test 'customer_totals_rollup' executes 3 real models (limit 2). - Test file: tests/unit/customer_totals_rollup.sql - Model under test: customer_totals - -Real models executed, and the dependency path that pulls each one in: - customer_totals model under test - order_totals customer_totals -> order_totals - stg_orders customer_totals -> order_totals -> stg_orders - -Options: - 1. Mock the direct inputs of the model under test to cut the chain: - __ref__order_totals AS (SELECT ...) - 2. If this test checks behaviour across several models, consider a scenario - (tests/scenarios/). A unit test inlines every real model into one statement, - so the warehouse compiles and optimises the whole chain at once; a scenario - builds each model step by step, compiling each statement separately. - 3. Raise the limit in sqlbuild_project.toml (or sqlbuild_local.toml): - [settings] - unit_test_max_real_models = 3 -``` - -`sqb test` shows each test's count on its row, such as `3 real models`, so a growing chain is -visible early. Macro, UDF and table function tests run no models and aren't counted. - -### Time limits - -Each unit test has a time limit, `30s` by default. The first of these that is set applies: - -1. `--test-timeout` on `sqb test` or `sqb build` -2. `TEST(timeout "45s")` in the test file -3. `[targets..execution_limits] test_timeout` -4. `[settings] test_timeout`, in `sqlbuild_local.toml` or `sqlbuild_project.toml` -5. The `30s` default - -Each takes a positive fixed duration, such as `"45s"` or `"10m"`, or `"disabled"`. - -The clock starts at the test's first warehouse statement. Rendering, warehouse queueing where the -warehouse reports it, and cleanup after the comparison don't count. See -[Per-item time limits](project-configuration.md#per-item-time-limits) for how each -adapter enforces the limit, and [Duration limit](project-configuration.md#duration-limit) -for how each adapter cancels a running statement. - -At the limit, SQLBuild cancels the running statement, reports the test as `TIMEOUT`, and continues -with the other tests. The report gives the elapsed time, the limit, where it was set, the -cancellation outcome and the query IDs it saw. When the test runs real models beyond the model -under test, it also lists them, with the same options as the real-model limit: - -```text -SQL unit test 'order_scans_kept' timed out after 1.0s (limit 1s). - Limit set by: TEST() timeout in tests/unit/order_scans_kept.sql - Cancellation: confirmed; the running statement stopped - -This test inlines 1 real model beyond the model under test into one statement: - stg_orders - -Options: - 1. Mock the inlined models closest to the model under test to cut the chain, - for example __ref__stg_orders AS (SELECT ...). - 2. If this test checks behaviour across several models, consider a scenario - (tests/scenarios/). A unit test inlines every real model into one statement, - so the warehouse compiles and optimises the whole chain at once; a scenario - builds each model step by step, compiling each statement separately. - 3. To allow more time, set TEST(timeout "3s") on this test, [settings] test_timeout = "3s" - in sqlbuild_project.toml (or sqlbuild_local.toml), or --test-timeout for one run. -``` - -A test that runs only the model under test reports the timeout, the limit, where it was set and how -to change it, without the scenario advice. - -Scenarios have their own limit, `scenario_timeout`; see -[Scenario timeouts](project-configuration.md#unit-test-and-scenario-limits). - -## Skipping SQL analysis - -Compile checks test fixtures and assertions like model SQL. For SQL the parser can't read, add -`sql_analysis false` to the `TEST` header; the test still runs. Tests of a model with -`sql_analysis false`, and every test in a `--no-sql-analysis` run, aren't analysed. With -`[settings] require_sql_analysis = true`, the opt-out is accepted only on test SQL that can't be -parsed; see [Requiring SQL analysis](models/configuration.md#requiring-sql-analysis). - -## Details +## More test features -- On DuckDB, MotherDuck, PostgreSQL and Snowflake, a test evaluates each compared model's full - output once. If that output fails, for example because a fixture value can't be converted in a - column the expected rows leave out, the test errors with the warehouse's message. +[Testing reference](testing-reference.md) covers macro, UDF and table function tests, +parameterized cases, cursor windows, empty-input tests, cost guards and time limits. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/feature-comparison.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/feature-comparison.md index 575b324148..c1564311b6 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/feature-comparison.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/feature-comparison.md @@ -10,7 +10,6 @@ Online: https://sqlbuild.com/docs/feature-comparison/ - Feature comparison - Where each tool fits -- Not yet in SQLBuild SQLBuild, dbt, and SQLMesh are all SQL pipeline frameworks. They share common ground but differ in design philosophy and feature focus. @@ -44,9 +43,9 @@ SQLBuild, dbt, and SQLMesh are all SQL pipeline frameworks. They share common gr | Feature | SQLBuild | dbt | SQLMesh | |---------|----------|-----|---------| -| SQL analysis | Offline syntax, binding, type inference, semantic validation, and lineage (Polyglot) | dbt Core: none; dbt Fusion engine: compile-time (proprietary; built on Apache-2.0 dbt Core v2) | Compile-time (SQLGlot) | +| SQL analysis | Offline syntax, binding, type inference, semantic validation, and lineage | dbt Core: none; dbt Fusion: compile-time | Compile-time (SQLGlot) | | Focused compilation | Full graph integrity with deep analysis limited to selection and required upstream closure | Selected compilation | Selected planning | -| Column-level lineage | Compile-time, fast and rich modes | dbt Core: post-hoc via docs; dbt Fusion engine: compile-time | Compile-time | +| Column-level lineage | Compile-time, fast and rich modes | dbt Core: post-hoc via docs; dbt Fusion: compile-time | Compile-time | | Column contract validation | Compile-time inference plus runtime enforcement with `contract enforced` | YAML schema contracts at runtime | Schema contracts via plan | | Existing-schema contracts | Online read-only diff and safe repository generation | Codegen/packages | External schema tooling | | Rules and formatting | Compiler-integrated native and custom diagnostics, reasoned suppressions, and separate canonical formatting | External tools | Built-in formatter plus external linting | @@ -88,7 +87,7 @@ SQLBuild, dbt, and SQLMesh are all SQL pipeline frameworks. They share common gr | Feature | SQLBuild | dbt | SQLMesh | |---------|----------|-----|---------| | SQL models | `MODEL()` header with inline config | Jinja-templated SQL + YAML sidecar | `MODEL` DDL | -| Python models | Coming soon | Pandas, PySpark, Snowpark, BigFrames | Pandas, PySpark, Snowpark, BigFrames | +| Python models | Python nodes (tasks, assets, loaders, checks); no dataframe models | Pandas, PySpark, Snowpark, BigFrames | Pandas, PySpark, Snowpark, BigFrames | | Custom materializations | Python with full framework hooks | Jinja-based | Python-based custom model kinds | | Lifecycle hooks | Typed inline SQL, reusable parameterized SQL resources, and Python hooks with compile-time validation and `HookContext` | Jinja pre/post hooks | Python pre/post hooks | @@ -136,6 +135,4 @@ SQLBuild, dbt, and SQLMesh are all SQL pipeline frameworks. They share common gr | **dbt** | The most widely adopted SQL transformation framework with the largest adapter and community ecosystem. | | **SQLMesh** | State-managed pipelines with interval tracking and cross-dialect transpilation. | -## Not yet in SQLBuild - -- **Broader adapter support** - Redshift, Trino, Spark, Athena +See the [Roadmap](roadmap.md) for what comes next. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/index.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/index.md index 86455e87ef..578b988562 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/index.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/index.md @@ -60,9 +60,8 @@ audit results and everything it printed. Agents and people query that history wi | Review | [`sqb review`](cli/review.md) | Builds a branch's changes in a temporary schema, tests them and compares them with a target | | Investigate | [`sqb recall`](cli/recall.md) | Queries the history of past runs, then live warehouse state | -SQLBuild works with DuckDB, MotherDuck, Snowflake, BigQuery, Databricks, PostgreSQL, SQL Server, -ClickHouse and SQLite. -DuckDB runs locally, so you can try everything without warehouse credentials. See +SQLBuild works with Snowflake, DuckDB, MotherDuck and PostgreSQL. BigQuery, Databricks, SQL Server, +ClickHouse and SQLite are in beta. DuckDB runs locally, so you can try everything without warehouse credentials. See [Adapters](concepts/adapters.md) for connection setup. ## Where to start diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dagster.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dagster.md index c1503e8ff4..f043f730d8 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dagster.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dagster.md @@ -121,7 +121,7 @@ metadata to command-output and lifecycle records: Lifecycle events remain SQLBuild's immutable execution facts. Human stdout/stderr remains a correlated troubleshooting transcript; the integration does not parse terminal text to establish -resource status. See [Execution Observability](../concepts/observability.md). +resource status. See [Observability](../concepts/observability.md). ## Scenarios as checks diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/project-setup.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/project-setup.md new file mode 100644 index 0000000000..768ce204a4 --- /dev/null +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/project-setup.md @@ -0,0 +1,89 @@ + + +# Set up a project + +> Connect a SQLBuild project to your warehouse with dev and prod targets and a personal local file. + +Online: https://sqlbuild.com/docs/project-setup/ + +A project is configured by `sqlbuild_project.toml`, which you commit, and an optional +`sqlbuild_local.toml`, which each developer keeps to themselves. Start a project with +[`sqb init`](cli/init.md), or explore one with [`sqb playground`](cli/playground.md). + +```text +project/ + sqlbuild_project.toml # committed: shared targets and settings + sqlbuild_local.toml # gitignored: your own target, schema and credentials + models/ + sources/ + tests/ +``` + +## Connect to the warehouse + +A connection holds how to reach the warehouse. A target names where models are built: a database +and schema, such as `dev` or `prod`. Several targets can share one connection. + +```toml +name = "shop" +adapter = "snowflake" +default_target = "dev" + +[connections.warehouse] +account = "my_org-my_account" +user = "${ENV:SNOWFLAKE_USER}" +password = "${ENV:SNOWFLAKE_PASSWORD}" +warehouse = "TRANSFORM_WH" + +[targets.dev] +connection = "warehouse" +database = "ANALYTICS" +schema = "DEV" + +[targets.prod] +connection = "warehouse" +database = "ANALYTICS" +schema = "PROD" +``` + +`${ENV:NAME}` reads an environment variable, so passwords and tokens stay out of the file. Each +adapter page lists its connection fields; see [Adapters](concepts/adapters.md). + +Check the setup with [`sqb debug`](cli/debug.md), which validates the config and opens a +connection. Commands that don't need the warehouse, such as `sqb compile`, `sqb test --inspect` and +`sqb lineage`, work without one. + +## Choose a target + +Commands build into the target given by `--target`, then the `target` in `sqlbuild_local.toml`, then +`default_target`: + +```bash +sqb build # default_target, or your local target +sqb build --target prod +``` + +## Keep your own settings local + +Put what differs for you in `sqlbuild_local.toml` and add it to `.gitignore`. It merges over the +shared file, key by key, so it holds only what you change: + +```toml +# sqlbuild_local.toml +target = "dev" + +[targets.dev] +schema = "DEV_ALICE" + +[connections.warehouse] +user = "alice" +``` + +The local file can also switch `adapter`, for example to build on DuckDB while the team uses +Snowflake. + +## Next steps + +- [Models](concepts/models.md) - write your first model. +- [Configuration reference](concepts/project-configuration.md) - every project setting, including + defaults, path defaults, clone policies and execution limits. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/quickstart.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/quickstart.md index b1ffe7fe0d..77f2596196 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/quickstart.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/quickstart.md @@ -13,7 +13,8 @@ Online: https://sqlbuild.com/docs/quickstart/ - 2. Plan - 3. Build - 4. Verify -- 5. Common operations +- 5. Change a model +- 6. Common operations - What you just built - Next steps @@ -35,11 +36,7 @@ cd waffle-shop This creates a self-contained waffle shop project with everything you need to explore SQLBuild. -The playground creates the shared `sqlbuild_project.toml`; that is sufficient for this -quickstart. In a team project, you can additionally create a gitignored -`sqlbuild_local.toml` to select your usual target and override your personal development -schema or credentials without changing the committed project config. See -[Project Configuration](concepts/project-configuration.md#sqlbuild_localtoml). +To connect your own warehouse later, see [Set up a project](project-setup.md). ## 2. Plan @@ -87,7 +84,7 @@ Seeds (1) Execute the full build: ```bash -sqb --no-color build +sqb build ``` SQLBuild materializes all models in DAG order. SQL unit tests run before their target models. Error-severity audits run against a staging table before promotion - if any fail, the production table is untouched: @@ -103,50 +100,14 @@ Execution sqb build (concurrency: 1) audit not_null (customer_id) PASS audit unique (customer_id) PASS audit not_null (email) PASS - 7/17 view stg_orders OK 0.04s - test test_daily_revenue_chain PASS - expect expected stg_orders PASS - expect expected stg_payments PASS - expect expected daily_revenue PASS - expect assertion daily_revenue_is_non_negative PASS - test test_stg_orders PASS - expect expected stg_orders PASS - expect assertion order_ids_are_not_null PASS - audit not_null (order_id) PASS - audit unique (order_id) PASS - audit not_null (customer_id) PASS - audit relationships (waffle_type_id) PASS - audit accepted_values (status) PASS - 8/17 custom daily_order_partitioned OK 0.21s - audit waffles_ordered_is_positive PASS 4/4 - audit not_null (order_date) PASS 4/4 - 9/17 table daily_revenue OK 0.07s - test test_daily_revenue_chain PASS - expect expected stg_orders PASS - expect expected stg_payments PASS - expect expected daily_revenue PASS - expect assertion daily_revenue_is_non_negative PASS - audit revenue_is_non_negative PASS - 11/17 table fact_orders OK 0.05s - test test_fact_orders PASS - expect expected fact_orders PASS - expect assertion line_totals_are_non_negative PASS - audit not_null (order_id) PASS - 12/17 table customer_status_snapshot (merge) OK 0.05s - audit not_null (customer_id) PASS - audit unique (customer_id) PASS + ... 13/17 table hourly_order_activity (delete_insert) OK 0.24s 5 batches (1d) range 2026-03-31T09:00:00 → 2026-04-04T14:00:00 8 rows audit (d) orders_placed_is_non_negative PASS 5/5 audit (d) not_null (activity_hour) PASS 5/5 audit (f) orders_placed_is_non_negative PASS audit (f) not_null (activity_hour) PASS - 17/17 table hourly_activity_with_daily_context (delete_insert) OK 0.63s - 13 batches (6h) range 2026-03-31T18:00:00 → 2026-04-03T23:00:00 0 rows - audit (d) day_orders_cover_hourly_orders PASS 13/13 - audit (d) not_null (activity_hour) PASS 13/13 - audit (f) day_orders_cover_hourly_orders PASS - audit (f) not_null (activity_hour) PASS + ... ✓ Completed successfully PASS=99 WARN=0 FAIL=0 INSUFFICIENT=0 SKIP=0 TOTAL=99 (2.13s) ``` @@ -194,7 +155,40 @@ Seeds (1) The models and functions are no longer listed as first run, and the seed is current. The two loader-managed sources load on every build. -## 5. Common operations +## 5. Change a model + +Edit `models/staging/stg_orders.sql` to drop cancelled orders, by adding this line at the end: + +```sql +WHERE status <> 'cancelled' +``` + +Then plan again: + +```bash +sqb plan +``` + +``` +Query changed (1) +└── stg_orders recreate view + └── query diff: + --- previous + +++ current + @@ -5,4 +5,5 @@ + quantity, + ordered_at, + status + -FROM __source("raw__orders") + +FROM __source("raw__orders") + +WHERE status <> 'cancelled' +``` + +The plan shows which model changed and the exact diff, before anything runs. Downstream models +still build as usual; each incremental model's own `replay_on_change` decides whether it reprocesses +history. See [Planning](concepts/planning.md). + +## 6. Common operations ```bash # Rebuild a specific model diff --git a/website/STYLE.md b/website/STYLE.md index 105b407dc8..5401c6f7d7 100644 --- a/website/STYLE.md +++ b/website/STYLE.md @@ -2,7 +2,7 @@ These rules apply to every page under `website/src/content/docs/`. Pages that follow them well: `concepts/semantic-compilation.mdx`, `cli/review.mdx`, `concepts/audits.mdx`, -`concepts/models/column-migrations.mdx` and `benchmarks.mdx`. +`concepts/refactoring.mdx` and `benchmarks.mdx`. ## Write for users @@ -17,6 +17,10 @@ These rules apply to every page under `website/src/content/docs/`. Pages that fo - Lead with what the feature does and a short example: a command, a config block or the output a user sees. +- The first paragraph says what the user does and what they get, in two sentences. Conditions, + exceptions and proofs go in a "Details" list at the end. +- Show the normal path first. If a command writes something for the user, such as `sqb rename` + adding `migrate_from`, show the command before the hand-written form. - Organise sections around the user's tasks ("Declaring a rename", "Forcing a fresh compile"), not around the implementation. - One topic per section. Put flags, config keys and codes in reference tables near the top or @@ -34,6 +38,10 @@ These rules apply to every page under `website/src/content/docs/`. Pages that fo ## Sentences - Write short, plain sentences in the present tense. One idea per sentence. +- A paragraph states one rule. If a sentence starts with "unless", "only when" or "except", it is a + separate rule: give it its own bullet or move it to Details. +- Name the tool, not the engine behind it. Write "SQLBuild's SQL analysis", not the parser library. + Name other vendors only where the comparison is the point of the page. - Give every rule an example where it matters, such as the SQL that triggers an error and the SQL that fixes it. - Use neutral orders-style examples (orders, customers, products, payments). See `AGENTS.md`. @@ -51,6 +59,6 @@ These rules apply to every page under `website/src/content/docs/`. Pages that fo belongs to, update tables and examples, and remove what is no longer true. Don't append a paragraph to the end of a section or page. - Don't repeat a rule that another page owns. Link to it instead, for example to - `concepts/testing#time-limits` for unit test timeout precedence. + `concepts/testing-reference#time-limits` for unit test timeout precedence. - After editing a page, regenerate the skill reference copies with `scripts.generate_docs_skill` and commit them with the page. diff --git a/website/src/content/docs/docs/cli/audit.mdx b/website/src/content/docs/docs/cli/audit.mdx index b5ca48c715..afc681d050 100644 --- a/website/src/content/docs/docs/cli/audit.mdx +++ b/website/src/content/docs/docs/cli/audit.mdx @@ -28,7 +28,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb | `--concurrency` | Maximum number of audits to run concurrently (default: `1`) | | `--json` | Print structured audit results | | `--json-output` | Write structured audit results to a file | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | ## Concurrency diff --git a/website/src/content/docs/docs/cli/build.mdx b/website/src/content/docs/docs/cli/build.mdx index a9a901c12b..46a5523148 100644 --- a/website/src/content/docs/docs/cli/build.mdx +++ b/website/src/content/docs/docs/cli/build.mdx @@ -46,7 +46,7 @@ own option. `--version` works only directly after `sqb`, as in `sqb --version`: | `--defer-sources-to` | Read managed source data from another target | | `--fail-fast` | Stop on first failure and skip remaining nodes | | `--concurrency` | Number of worker connections (default: 1) | -| `--test-timeout ` | Time limit for each unit test the build runs, such as `30s`, or `disabled`; see [Time limits](/docs/concepts/testing#time-limits) | +| `--test-timeout ` | Time limit for each unit test the build runs, such as `30s`, or `disabled`; see [Time limits](/docs/concepts/testing-reference#time-limits) | | `--verbose`, `-v` | Show lifecycle SQL inline after each model | | `--start-cursor` | Override the start cursor, typed from the selected incremental models' cursor type; fails with `S303` if the selection mixes timestamp and integer cursors. With no cursor model selected, an integer value is read as an integer cursor and anything else as an ISO timestamp | | `--end-cursor` | Override the end cursor, typed the same way as `--start-cursor` | @@ -62,7 +62,7 @@ own option. `--version` works only directly after `sqb`, as in `sqb --version`: | `SELECTOR ...` | Positional selectors, the same as `--select` | | `--select`, `-s` | Select specific models, or unit tests by name (`test:`); selecting only unit tests runs them without building models | | `--exclude` | Exclude specific models | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | ## Fast iteration @@ -204,6 +204,58 @@ skipped as for any other failure. Its row shows `TIMEOUT`. In the summary line, failures were timeouts. A timeout is a failure for the exit code: the build exits non-zero and its JSON `status` is `failed`. +## Limits + +A target's [`execution_limits`](/docs/concepts/project-configuration#execution-limits) cap a whole +build, and a model's [`max_duration`](/docs/concepts/models/configuration#execution-time-limit) caps +one model. + +### Model limit + +When the build would run more models than `max_models`, the build stops before any clone, load or +model runs, and says that nothing in the warehouse changed. `sqb plan` has no limit, so use it to +check a broad selection. + +### Duration limit + +`max_duration` covers planning and the build. At the deadline SQLBuild starts no more work and +cancels the statements still running. Where the warehouse has a statement time limit, each +statement also gets only the time that remains. + +| Adapter | Warehouse statement limit | Cancellation at the deadline | Longest accepted duration | +|---------|---------------------------|------------------------------|---------------------------| +| Snowflake | `STATEMENT_TIMEOUT_IN_SECONDS` per statement | Driver cancel request for the running query | 7 days | +| DuckDB, MotherDuck | None | Connection interrupt | About 24 days | +| PostgreSQL | `SET LOCAL statement_timeout` inside transactions; statements outside a transaction rely on cancellation | Driver cancel (`cancel_safe`) | About 24 days | +| BigQuery | Job timeout (`job_timeout_ms`) on each query job | `jobs.cancel`; reported as requested, not confirmed | About 24 days | +| Databricks | `STATEMENT_TIMEOUT` session setting, capped at 48 hours | Cursor cancel; reported as requested, not confirmed | About 24 days | +| SQL Server | None; the driver's query timeout is process-wide | Driver cancel | About 24 days | +| SQLite | None | Connection interrupt | About 24 days | + +A duration timeout can come after earlier models finished, so its error says that warehouse changes +may already have been made. A custom adapter that can't cancel a running statement can't be used +with `max_duration`. + +### Per-item time limits + +Unit test, scenario and model limits (`test_timeout`, `scenario_timeout` and a model's +`max_duration`) count running time only, from the item's first warehouse statement. Rollback and +cleanup always run and don't count. Where a warehouse reports queueing separately, queued +time doesn't count: + +| Adapter | How the item limit is enforced | Queued time | +|---------|--------------------------------|-------------| +| Snowflake | `STATEMENT_TIMEOUT_IN_SECONDS` per statement, with a client-side backstop after a grace margin of half the limit (at least 30 seconds) | Excluded by the warehouse limit; the backstop counts it | +| DuckDB, MotherDuck | Connection interrupt at the limit | Not applicable | +| PostgreSQL | `SET LOCAL statement_timeout` inside transactions; cancellation at the limit | Lock waits count | +| BigQuery | `jobs.cancel` at the limit; no job timeout, because a job timeout counts from job creation | Excluded: the clock stops while a job is pending | +| Databricks | `STATEMENT_TIMEOUT` session setting and cursor cancel | May count | +| SQL Server | Driver cancel at the limit | Lock waits count | +| SQLite | Connection interrupt at the limit | Waits for the write lock count | + +When limits nest, such as a model's unit tests inside its own limit, the earliest deadline applies. +After a timeout, SQLBuild checks the connection and replaces it if it is no longer usable. + ## Fingerprints After a successful build, SQLBuild writes version identities to `_sqlbuild_fingerprints` in the target schema. These are used on subsequent runs to detect changes and skip unchanged work. A build writes a row only for a resource whose identity differs from its latest row, so a build that changes no definitions writes none. In a project with model migration history, every build writes a row for each resource it builds. See [Planning and Change Detection](/docs/concepts/planning) for details. diff --git a/website/src/content/docs/docs/cli/check.mdx b/website/src/content/docs/docs/cli/check.mdx index 431f4e9fc8..2fa1dff9d5 100644 --- a/website/src/content/docs/docs/cli/check.mdx +++ b/website/src/content/docs/docs/cli/check.mdx @@ -26,7 +26,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb | `--json` | Print check results as JSON | | `--json-output` | Write check results JSON to a file path | | `--vars` | Override project variables | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | Selecting a non-check node is rejected; use [`sqb build`](/docs/cli/build) to run tasks and assets. diff --git a/website/src/content/docs/docs/cli/clone.mdx b/website/src/content/docs/docs/cli/clone.mdx index dabc1b3103..f9d1663b38 100644 --- a/website/src/content/docs/docs/cli/clone.mdx +++ b/website/src/content/docs/docs/cli/clone.mdx @@ -43,7 +43,7 @@ sqb --project-dir clone --from [--to ] [flags] | `--hard-copy` | Force physical table copies instead of zero-copy cloning | | `--select`, `-s` | Select specific models to clone | | `--exclude` | Exclude specific models from cloning | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | ## Examples @@ -94,4 +94,4 @@ allow_as_clone_destination = true The origin target must already contain the built relations being cloned. The destination connection's credentials must be able to read the origin namespace and create relations in the destination. Managed physical sources may bootstrap a destination that has not been built. See -[Project Configuration](/docs/concepts/project-configuration) for details. +[Configuration reference](/docs/concepts/project-configuration) for details. diff --git a/website/src/content/docs/docs/cli/compile.mdx b/website/src/content/docs/docs/cli/compile.mdx index 818480b327..ed2f9227ac 100644 --- a/website/src/content/docs/docs/cli/compile.mdx +++ b/website/src/content/docs/docs/cli/compile.mdx @@ -62,7 +62,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb 2. **Graph.** Resolves `__ref()`, `__source()` and `__seed()`, expands macros, and orders models by dependency. 3. **SQL checks.** Parses and checks SQL, when SQL analysis is on. See - [Semantic Compilation](/docs/concepts/semantic-compilation). + [What `sqb compile` checks](/docs/concepts/semantic-compilation). 4. **Column lineage.** Traces column-level dependencies across models. 5. **Contracts.** Checks declared columns against the inferred query output. 6. **Rules.** Runs the selected built-in Rules, then the selected custom Rules. @@ -73,8 +73,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb Rules that need inferred types, such as `SQBRCONTRACT105`, are skipped when SQL analysis is off, and compile prints one note naming them. -`sqb compile` is authoritative for correctness, contracts, lineage and configured Rules. Rule -findings block the artifacts. To check one Rule code or family without changing project +Rule findings block the artifacts. To check one Rule code or family without changing project configuration, use [`sqb rules run`](/docs/cli/rules). ### When Rules fail @@ -154,19 +153,14 @@ error[K001]: declared column 'total_cents' was not found in statically inferred | `K001` | A declared column is missing from the query output | | `K002` | A column's inferred type doesn't match the declared type | | `K003` | A column's type can't be proven (with `type_enforcement` on) | +| `K004` | A `nullable false` column can be null | +| `K005` | A model with `contract enforced` outputs a column it doesn't list | +| `K006` | A model with `contract enforced` lists no columns | | `K013` | A string literal fills an enum-typed column of an enforced contract but isn't a member of the enum (not checked on SQL Server) | -Which models are checked depends on `[settings] check_declared_columns`: - -- `true` (default): a model with declared columns and no `contract` setting must output every - declared column. -- `false`: only models with `contract enforced` are shape-checked; declared columns elsewhere - are metadata and audit targets. - -Explicit type enforcement works with either value. Setting `false` is a project policy, not a -way to silence an error: if a model is meant to have a contract, fix the query or the -declaration. Errors on `contract enforced` models don't suggest changing the setting. See -[Contracts](/docs/concepts/models/contracts). +Models without a `contract` setting are checked unless the project sets +`[settings] check_declared_columns = false`; see [Contracts](/docs/concepts/models/contracts). Don't +turn the check off to silence one error: fix the query or the column list. ## Output diff --git a/website/src/content/docs/docs/cli/contract.mdx b/website/src/content/docs/docs/cli/contract.mdx index e1eee35090..472c34288c 100644 --- a/website/src/content/docs/docs/cli/contract.mdx +++ b/website/src/content/docs/docs/cli/contract.mdx @@ -5,7 +5,9 @@ sidebar: label: "contract" --- -Existing-schema comparison is explicitly online and read-only with respect to the warehouse. +`sqb contract` compares a model's or source's declared columns with the table that already exists in +the warehouse, or writes declarations from it. It reads the warehouse and never changes it. The +target named by `--from` supplies the database and schema to read, using the project's connection. ## diff @@ -21,8 +23,22 @@ Reports declaration differences. Exit code `0` means agreement and `1` means dif sqb contract generate --from [flags] ``` -Prints proposed declaration updates. Add `--write` to change repository files and `--overwrite` to -replace conflicts rather than preserving authored declarations. +Prints proposed declaration updates without changing files. + +```bash +# Fill in missing types and add warehouse columns missing from code +sqb contract generate --from prod --select tag:commerce --write + +# Also replace conflicting types and names, and remove columns the warehouse doesn't have +sqb contract generate --from prod --select tag:commerce --write --overwrite +``` + +Without `--overwrite`, generation keeps everything you wrote: existing types where they conflict, +columns only in code, comments, descriptions, audits, nullability, enums, tags, freshness and loader +settings. It never adds `contract enforced`; that stays your decision. + +A shared `SCHEMA()` isn't changed from one model's columns; the conflict is reported instead. Writes +are atomic, and SQLBuild recompiles the project afterwards and restores the old files if it fails. ## Flags @@ -36,7 +52,7 @@ replace conflicts rather than preserving authored declarations. | `--vars` | both | Override project variables | | `--write` | generate | Apply generated repository changes | | `--overwrite` | generate | Replace conflicting declarations; requires `--write` | -| `--warehouse ` | both | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | both | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | -See [Contracts](/docs/concepts/models/contracts#adopting-an-existing-schema) for ownership and safety -guarantees. +Sources can be selected like models, for example `--select source:raw_orders`. See +[Contracts](/docs/concepts/models/contracts) for what a contract checks. diff --git a/website/src/content/docs/docs/cli/dbt.mdx b/website/src/content/docs/docs/cli/dbt.mdx index 1238052e98..cb10ab81d4 100644 --- a/website/src/content/docs/docs/cli/dbt.mdx +++ b/website/src/content/docs/docs/cli/dbt.mdx @@ -47,6 +47,40 @@ confirmation asks for the typed confirmation on an interactive terminal and refu build the affected models with `sqb build` and its allow flag instead, or relax the policy. A refusal at this point leaves completed dbt work in place and builds no SQLBuild models. +## Flags + +`sqb dbt plan/run/build` declare the common flags directly. Anything declared goes **before** a `--` separator; any other raw dbt flag goes **after** it and is forwarded verbatim. A flag placed on the wrong side errors rather than silently reaching dbt. + +```bash +sqb dbt build --select path:models/marts --full-refresh +``` + +Declared flags are routed to the right place automatically: + +| Flag | Routed to | Notes | +| ---- | --------- | ----- | +| `--select` / `--exclude` | dbt | dbt resolves selection (see [Selection](/docs/concepts/dbt-compatibility/selection)). | +| `--vars` | dbt **and** SQLBuild | The same vars feed dbt's compile and SQLBuild's own variable resolution, so both sides see identical values. | +| `--full-refresh` | dbt **and** SQLBuild | Requests a full rebuild on both sides. dbt and native SQLBuild models independently apply their model-level `full_refresh` setting, including `false` opt-outs. | +| `--threads` | dbt (`--threads`) and SQLBuild (`--concurrency`) | | +| `--target` / `--project-dir` / `--profiles-dir` / `--profile` / `--target-path` | dbt | Standard dbt locators. | + +### `--vars` + +`--vars` accepts the same JSON object dbt accepts, and SQLBuild passes it to **both** the underlying `dbt` invocation and its own variable resolution. That means a value referenced as `@@my_var` in SQLBuild model SQL (see [Variables and placeholders](/docs/concepts/interpolation)) and as `{{ var('my_var') }}` in a dbt model both resolve to the value you passed. CLI vars take precedence over project and local config vars. You do not need to declare vars twice. + +```bash +sqb dbt build --select path:models/marts --vars '{"my_var": 1}' +``` + +### Forwarding other dbt flags + +For any native dbt flag SQLBuild does not declare, put it after `--` and it is passed straight to the `dbt` invocation untouched: + +```bash +sqb dbt build -- --log-level debug +``` + ## sqb dbt debug Run dbt diagnostics followed by SQLBuild diagnostics. @@ -92,9 +126,9 @@ profiles_dir = "../profiles" target_path = "../dbt_project/target" ``` -See [Project Configuration](/docs/concepts/project-configuration#dbt) for all fields. +See [Using SQLBuild with dbt](/docs/concepts/dbt-compatibility/overview#configuration) for all fields. SQLBuild-owned work in `sqb dbt plan`, `run` and `build` connects with the target's `build` -[warehouse group](/docs/concepts/project-configuration#command-group-warehouses). dbt itself +[warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses). dbt itself keeps using the warehouse from its profile. These commands do not accept `--warehouse`, because their remaining arguments pass through to dbt. diff --git a/website/src/content/docs/docs/cli/debug.mdx b/website/src/content/docs/docs/cli/debug.mdx index 3039b125df..488c1e6f4b 100644 --- a/website/src/content/docs/docs/cli/debug.mdx +++ b/website/src/content/docs/docs/cli/debug.mdx @@ -1,6 +1,8 @@ --- -title: "debug" +title: "sqb debug" description: "Validate project configuration and test the warehouse connection." +sidebar: + label: "debug" --- Checks that your project config is valid, the adapter is resolvable, and the warehouse connection works. Useful for diagnosing setup issues. @@ -19,7 +21,7 @@ The command reports four groups of checks: **Runtime** - SQLBuild version, Python version, Python path, OS info. -**Configuration** - Finds and validates `sqlbuild_project.toml`, loads `sqlbuild_local.toml` if present, resolves the adapter, active target, connection, database, and schema. For Snowflake it also shows the warehouse each [command group](/docs/concepts/project-configuration#command-group-warehouses) connects with and where that value came from. +**Configuration** - Finds and validates `sqlbuild_project.toml`, loads `sqlbuild_local.toml` if present, resolves the adapter, active target, connection, database, and schema. For Snowflake it also shows the warehouse each [command group](/docs/concepts/adapters/snowflake#command-group-warehouses) connects with and where that value came from. **Providers** - Lists the [providers](/docs/concepts/python-nodes/providers) the project defines. @@ -31,7 +33,7 @@ The command reports four groups of checks: |------|-------------| | `--no-connection` | Skip the connection and query tests. Useful for validating config without warehouse access. | | `--json` | Output results as JSON instead of formatted text. | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | ## Example diff --git a/website/src/content/docs/docs/cli/diff.mdx b/website/src/content/docs/docs/cli/diff.mdx index 12960b32e6..aff10a80bc 100644 --- a/website/src/content/docs/docs/cli/diff.mdx +++ b/website/src/content/docs/docs/cli/diff.mdx @@ -1,6 +1,8 @@ --- -title: "diff" +title: "sqb diff" description: "Compare schemas and data between targets." +sidebar: + label: "diff" --- Compares schemas and optionally row-level data between two targets (e.g. `prod:dev`), or between two read-only SQL queries. See [Data Diffs](/docs/concepts/diff) for detailed usage. @@ -59,7 +61,7 @@ In query mode, omit `FROM:TO` and model selectors; each query identifies its own | `--json` | Print one JSON document to stdout | | `--select`, `-s` | Select specific models to diff (required in model mode) | | `--exclude` | Exclude specific models from diffing | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | ## Size guard diff --git a/website/src/content/docs/docs/cli/format.mdx b/website/src/content/docs/docs/cli/format.mdx index 13963130ee..145dbab35c 100644 --- a/website/src/content/docs/docs/cli/format.mdx +++ b/website/src/content/docs/docs/cli/format.mdx @@ -36,14 +36,11 @@ one-file-at-a-time run. ## Skipping unchanged files -`sqb format` remembers every file it found already formatted and free of faults and warnings, in -`target/cache/compiler-native-v1/format-results.bin`. A later run skips such a file while its text, -its path, the `[format]` and rule threshold settings, the adapter dialect and the SQLBuild build are -all unchanged, so `sqb format --check` on an unchanged project costs little more than reading the -files. A file that needed changes or reported anything is always formatted again, and a damaged or -unreadable cache is ignored. `--fix` never uses the cache, and SQL test files are not skipped -while typed-null fixture formatting reads their models' contracts. Pass `--no-cache` to format -every file. +`sqb format` skips files it already found formatted and clean, as long as the file and the format +settings haven't changed, so `sqb format --check` on an unchanged project is fast. A file that +needed changes or reported anything is always checked again, and SQL test files are always +formatted. `--fix` never skips files. Pass +`--no-cache` to format every file. ## Line width and descriptions @@ -54,16 +51,11 @@ Configure the maximum physical line width in `sqlbuild_project.toml` (the defaul line_width = 100 ``` -`MODEL`, `SCENARIO`, `FUNCTION` and `HOOK` headers must carry a non-empty `description`; -`sqb format --check` fails with `description-present` when one is missing, so pre-commit catches it -before compile does (see [Descriptions](/docs/concepts/models/configuration#descriptions)). A model -described by its `model_schema`, or by the one `[path_defaults]` entry that applies to it, passes, -exactly as it compiles. `[defaults]` does not accept `description` (`D001`), and only the nearest -matching path default applies (see -[Which path default applies](/docs/concepts/project-configuration#which-path-default-applies)). -Format does not render templates, so a templated description that renders empty passes format and -is reported by `sqb compile`. -Plain `sqb format` still formats the file and exits non-zero, like any other remaining fault. +`MODEL`, `SCENARIO`, `FUNCTION` and `HOOK` headers need a non-empty `description`. When one is +missing, `sqb format --check` fails with `description-present`, so pre-commit catches it before +compile does. A model described by its `model_schema` or its `[path_defaults]` entry passes, as it +does in compile; see [Descriptions](/docs/concepts/models/configuration#descriptions). Plain +`sqb format` still formats the file and exits non-zero. Descriptions of `MODEL`, `SCENARIO`, `SCHEMA`, `FUNCTION`, `HOOK` and `MACRO` headers have one canonical layout, which never changes their words, their quotes or the text they store (see @@ -102,15 +94,11 @@ Clause layout follows a few fixed rules: - A broken `DECODE` puts its first argument on its own line, then one search/result pair per line, then any default. -SQL lines longer than the width, counting any trailing alias and comma, are broken at the outermost -structure first: before top-level `AND` and `OR` (never inside `BETWEEN ... AND`), then before -arithmetic operators, then by putting the arguments of the widest outermost function call, `IN` -list, or array one per line, or the branches of a `CASE` on their own lines. Inside `OVER (...)`, -`PARTITION BY` and `ORDER BY` go on separate lines. Type parameters such as `NUMBER(38, 2)` or -`VARCHAR(MAX)` are never split; the enclosing call breaks instead. Header `audits [...]` lists put -each audit, and then each audit argument, on its own line. String literals, comments, and lines that -carry a comment are never broken, so a line can stay longer than the width when nothing on it can be -split. `sqb format --check` fails while a long line can still be shortened. +A line longer than the width is broken at its outermost structure first: before top-level `AND` +and `OR`, then before arithmetic operators, then one argument per line in the widest function call, +`IN` list or array, or one `CASE` branch per line. String literals, comments, type parameters such as +`NUMBER(38, 2)`, and lines that carry a comment are never broken, so a line can stay longer than the +width. `sqb format --check` fails while a long line can still be shortened. ## What formatting preserves diff --git a/website/src/content/docs/docs/cli/freshness.mdx b/website/src/content/docs/docs/cli/freshness.mdx index 674f3f7232..b309ccba6d 100644 --- a/website/src/content/docs/docs/cli/freshness.mdx +++ b/website/src/content/docs/docs/cli/freshness.mdx @@ -28,7 +28,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb | `SELECTOR ...` | Positional selectors, the same as `--select` | | `--select`, `-s` | Select specific sources or models (sources upstream of selected models are included) | | `--exclude` | Exclude specific sources or models | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | ## Source selection diff --git a/website/src/content/docs/docs/cli/janitor.mdx b/website/src/content/docs/docs/cli/janitor.mdx index 5c5e080ab6..37ff868ee8 100644 --- a/website/src/content/docs/docs/cli/janitor.mdx +++ b/website/src/content/docs/docs/cli/janitor.mdx @@ -47,7 +47,7 @@ sqb --project-dir janitor [flags] | `--drop-old-name-view ` | Drop the [compatibility view](/docs/concepts/models/migrations#old-names) at a renamed model's old name before it expires. Repeat for several views. Fails with `C503` if no compatibility view is recorded at the name | | `--verbose`, `-v` | List every entry of each preview section and every skipped object with its reason | | `--json` | Print the plan and the result as one JSON document on stdout; the preview, prompt and progress go to stderr | -| `--warehouse ` | Snowflake warehouse for this run; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this run; overrides the target's `query` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | ## Lifecycle @@ -232,62 +232,7 @@ preview and the prompt show one notice, such as `usage not checked: unsupported janitor archives as usual. With `keep_in_use` on, a second notice, `nothing kept in use`, says that in-use relations are archived too. -#### What counts as a read - -**Snowflake.** A relation read through the objects a query named shows `via` those objects. -Everything a directly read object is built on, through any number of views, is in use, so a -candidate view between a view someone reads and a candidate table is kept too. Links through views -in other databases aren't followed, and dependencies created by `CREATE TABLE ... AS SELECT` and -similar statements aren't recorded. These don't count as reads: - -- queries SQLBuild tags as its own (`QUERY_TAG` with `"app":"sqlbuild"`, set by builds, audits, - loads and hooks); -- `CREATE ... CLONE` of tables, dynamic tables, schemas and databases. A statement counts as a clone - only when it starts with `CREATE`, after any comments, and names its source after `CLONE`. - -Without `OBJECT_VIEWER`, reads are still checked but view chains aren't followed, and the preview -says `view chains not followed: grant SNOWFLAKE.OBJECT_VIEWER`. - -**BigQuery.** Successful query jobs that reference the relation count, by `user_email`. A job lists -the views it reads and the tables behind them, so both count. These are missed or don't count: - -- a candidate view read only through another view may not show, because BigQuery doesn't document - whether nested views are listed; -- jobs answered from cached results record no tables, so with an `in_use_window` under about a day - a reader whose queries were all cached can be missed; -- jobs that run in another project aren't seen; -- script parent jobs, `CREATE SNAPSHOT TABLE` and `CREATE TABLE ... CLONE` don't count; -- SQLBuild's own jobs count like any other. - -**Databricks.** Read events in Unity Catalog's table lineage count, by `created_by`. A view a query -reads is a direct read, and the tables behind it are reads through the view. Lineage covers only -reads Databricks can infer lineage for, and can arrive late, so "not in use" is a weaker signal than -on Snowflake. Clones and SQLBuild's own reads count as reads. - -**ClickHouse.** Finished `SELECT` and `INSERT` queries count, by the user who started them, once per -initial query. A query lists every table and view it touched, so a whole view chain shows as in -use, without `via`. Writes count, since renaming the relation breaks the writer too. These are -missed or don't count: - -- queries on other replicas or shards, which matters on ClickHouse Cloud and behind load-balanced - replicas. When `system.clusters` shows other replicas of the connected server's shard, the preview - says `usage read from this server only; other replicas not checked`; -- queries ClickHouse answers from metadata without logging the table, such as a plain `count()`; -- readers whose settings profile turns `log_queries` off or samples it with - `log_queries_probability`. When `log_queries` is off for the janitor's own connection, usage isn't - checked; -- queries in the last few seconds: the log is flushed about every 7.5 seconds; -- queries older than the server's `query_log` TTL; -- SQLBuild's own reads count. - -### Audit record - -Every archive and delete is recorded in an append-only `_sqlbuild_janitor_events` table in the -affected schema: the original name, the archive name, the time and the invocation. The janitor -never reads it to decide what to do; it is for auditing, and for restoring archives whose names -were shortened, which are recorded right after their rename. Other events are written in batches -of up to 100 per schema, so if the process is killed, up to 100 actions in a schema may have no -event. Those archives still expire by the timestamp in their names. +See [What counts as a read](#what-counts-as-a-read) for each warehouse's limits. ## Targets @@ -329,7 +274,7 @@ in_use_window = "7d" keep_in_use = false ``` -See [Project Configuration](/docs/concepts/project-configuration#janitor) for every key and default. +See [Configuration reference](/docs/concepts/project-configuration#janitor) for every key and default. On Snowflake, BigQuery and Databricks, the actions of each phase run on up to the target's [`concurrency`](/docs/concepts/project-configuration#target-concurrency) connections at once. @@ -393,3 +338,61 @@ sqb janitor --target prod # Keep relations that are still read instead of archiving them sqb janitor --keep-in-use ``` + +## Details + +### What counts as a read + +**Snowflake.** A relation read through the objects a query named shows `via` those objects. +Everything a directly read object is built on, through any number of views, is in use, so a +candidate view between a view someone reads and a candidate table is kept too. Links through views +in other databases aren't followed, and dependencies created by `CREATE TABLE ... AS SELECT` and +similar statements aren't recorded. These don't count as reads: + +- queries SQLBuild tags as its own (`QUERY_TAG` with `"app":"sqlbuild"`, set by builds, audits, + loads and hooks); +- `CREATE ... CLONE` of tables, dynamic tables, schemas and databases. A statement counts as a clone + only when it starts with `CREATE`, after any comments, and names its source after `CLONE`. + +Without `OBJECT_VIEWER`, reads are still checked but view chains aren't followed, and the preview +says `view chains not followed: grant SNOWFLAKE.OBJECT_VIEWER`. + +**BigQuery.** Successful query jobs that reference the relation count, by `user_email`. A job lists +the views it reads and the tables behind them, so both count. These are missed or don't count: + +- a candidate view read only through another view may not show, because BigQuery doesn't document + whether nested views are listed; +- jobs answered from cached results record no tables, so with an `in_use_window` under about a day + a reader whose queries were all cached can be missed; +- jobs that run in another project aren't seen; +- script parent jobs, `CREATE SNAPSHOT TABLE` and `CREATE TABLE ... CLONE` don't count; +- SQLBuild's own jobs count like any other. + +**Databricks.** Read events in Unity Catalog's table lineage count, by `created_by`. A view a query +reads is a direct read, and the tables behind it are reads through the view. Lineage covers only +reads Databricks can infer lineage for, and can arrive late, so "not in use" is a weaker signal than +on Snowflake. Clones and SQLBuild's own reads count as reads. + +**ClickHouse.** Finished `SELECT` and `INSERT` queries count, by the user who started them, once per +initial query. A query lists every table and view it touched, so a whole view chain shows as in +use, without `via`. Writes count, since renaming the relation breaks the writer too. These are +missed or don't count: + +- queries on other replicas or shards, which matters on ClickHouse Cloud and behind load-balanced + replicas. When `system.clusters` shows other replicas of the connected server's shard, the preview + says `usage read from this server only; other replicas not checked`; +- queries ClickHouse answers from metadata without logging the table, such as a plain `count()`; +- readers whose settings profile turns `log_queries` off or samples it with + `log_queries_probability`. When `log_queries` is off for the janitor's own connection, usage isn't + checked; +- queries in the last few seconds: the log is flushed about every 7.5 seconds; +- queries older than the server's `query_log` TTL; +- SQLBuild's own reads count. + +### Audit record + +Every archive and delete is recorded in an append-only `_sqlbuild_janitor_events` table in the +affected schema: the original name, the archive name, the time and the invocation. The janitor +never reads it to decide what to do; it is for auditing, and for restoring archives whose names +were shortened, which are recorded right after their rename. If the process is killed, its last +actions may have no record; those archives still expire by the timestamp in their names. diff --git a/website/src/content/docs/docs/cli/lineage.mdx b/website/src/content/docs/docs/cli/lineage.mdx index 742caeccc2..e1fd442f08 100644 --- a/website/src/content/docs/docs/cli/lineage.mdx +++ b/website/src/content/docs/docs/cli/lineage.mdx @@ -1,6 +1,8 @@ --- -title: "lineage" +title: "sqb lineage" description: "Explore model and column-level dependency graphs from the command line." +sidebar: + label: "lineage" --- Inspect upstream and downstream dependencies for any model, source, seed, or function in your project. Supports both model-level lineage (dependency graph) and column-level lineage (tracing individual columns through transformations). Outputs as a tree, edge list, or structured JSON. diff --git a/website/src/content/docs/docs/cli/load.mdx b/website/src/content/docs/docs/cli/load.mdx index 46e2515106..997c0eee0e 100644 --- a/website/src/content/docs/docs/cli/load.mdx +++ b/website/src/content/docs/docs/cli/load.mdx @@ -30,7 +30,7 @@ sqb --project-dir load [flags] | `--json` | Output results as JSON | | `--json-output` | Write JSON results to a file path | | `--vars` | Override project variables with a JSON object (`--vars '{"key": "value"}'`) | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | ## Examples diff --git a/website/src/content/docs/docs/cli/plan.mdx b/website/src/content/docs/docs/cli/plan.mdx index 44514fdb97..31670de0e0 100644 --- a/website/src/content/docs/docs/cli/plan.mdx +++ b/website/src/content/docs/docs/cli/plan.mdx @@ -36,7 +36,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb | `SELECTOR ...` | Positional selectors, the same as `--select` | | `--select`, `-s` | Select specific models | | `--exclude` | Exclude specific models | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | ## Output diff --git a/website/src/content/docs/docs/cli/query.mdx b/website/src/content/docs/docs/cli/query.mdx index b5805ed023..6c31a18041 100644 --- a/website/src/content/docs/docs/cli/query.mdx +++ b/website/src/content/docs/docs/cli/query.mdx @@ -33,7 +33,7 @@ File paths are resolved from the current working directory. | `--json` | Shorthand for `--format json` | | `--limit` | Maximum rows to return (default: 20) | | `--no-limit` | Disable the row limit | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | ## Output diff --git a/website/src/content/docs/docs/cli/recall.mdx b/website/src/content/docs/docs/cli/recall.mdx index d219edfe46..b5883f38d5 100644 --- a/website/src/content/docs/docs/cli/recall.mdx +++ b/website/src/content/docs/docs/cli/recall.mdx @@ -20,7 +20,7 @@ sqb recall output last ## sqb recall query Run one read-only `SELECT` against the Recall views, in the store's dialect: SQLite for the local -store, ClickHouse for a [ClickHouse store](/docs/concepts/observability/recall#sharing-history-in-clickhouse). +store, ClickHouse for a [ClickHouse store](/docs/concepts/observability/recall-clickhouse). Statements that write, and multiple statements, are rejected. | Flag | Description | diff --git a/website/src/content/docs/docs/cli/rename.mdx b/website/src/content/docs/docs/cli/rename.mdx index 165673bb4e..a1bfdc8471 100644 --- a/website/src/content/docs/docs/cli/rename.mdx +++ b/website/src/content/docs/docs/cli/rename.mdx @@ -1,6 +1,8 @@ --- -title: "rename and mv" +title: "sqb rename and sqb mv" description: "Rename or move a model, or rename a column, and update every reference in one verified step." +sidebar: + label: "rename / mv" --- `sqb rename` and `sqb mv` change a name everywhere the compiler knows it is used, instead of leaving you to find every `__ref`, fixture, and header by hand. Every edit comes from compiled facts, not text search: a column rename follows the columns each query actually binds, so a same-named column of another table is left alone. @@ -57,7 +59,7 @@ stg_order_cents models/finance/ stg_order_refunds models/finance/ ``` -The moves are planned and verified together: the project compiles once before planning and the edited project once after, whatever the number of moves. As with a single move, adding `migrate_from` to models whose relation moves takes one more compile of the edited project. Declarations and unit tests are placed for the state after every move, so a macro whose users all move goes straight to their new folder instead of up to a shared folder and back. States between the moves are never checked, which lets a batch make a change that no order of single moves could make. As with one move, nothing is written unless the whole batch compiles. +The moves are checked together, as one change: nothing is written unless the project compiles after all of them. Declarations and unit tests go where the final state needs them, so a macro whose users all move goes straight to their new folder. A batch is refused when it moves the same model twice, moves two models to the same file, gives two models the same new name, or gives a model the name another model of the batch gives up: the old name keeps working through a compatibility view after the rename, so it must stay free. With `--dry-run`, the output lists the combined edits of every move. @@ -70,7 +72,7 @@ sqb rename stg_orders stg_order_lines fact_orders fact_order_lines sqb rename --from-file renames.tsv --dry-run ``` -The renames are planned and verified together, with one compile before and one of the edited project after, and each renamed model gets its own `migrate_from`, so every old name keeps working through its compatibility view after the next build. The same refusals as for batch moves apply: a model renamed twice, two models given one name, or a model taking a name another model of the batch gives up. Column renames and `--cascade` take one `. ` per command; a batch that names a column is refused. +The renames are checked together, and each renamed model gets its own `migrate_from`. The same refusals as for batch moves apply: a model renamed twice, two models given one name, or a model taking a name another model of the batch gives up. Column renames and `--cascade` take one `. ` per command; a batch that names a column is refused. ## What changes @@ -123,7 +125,7 @@ By default a column rename is one step: downstream models keep their output name A renamed model keeps its warehouse history through a [model migration](/docs/concepts/models/migrations). Whenever the model's relation moves and its materialization keeps warehouse data (table, view, incremental, or snapshot), the command adds `migrate_from ` to the model header, so the move does not depend on automatic discovery finding exactly one match. The next `sqb build` migrates the relation and leaves a compatibility view at the old name. A move that keeps the relation, such as a folder change in the same schema, adds nothing. -On a target that never built the old name, the origin is missing. By default the plan warns and the model builds fresh; set [`missing_migration_origin`](/docs/concepts/project-configuration#missing-migration-origins) on a target to require confirmation or stop the build instead. +On a target that never built the old name, the origin is missing. By default the plan warns and the model builds fresh; set [`missing_migration_origin`](/docs/concepts/models/migrations#missing-migration-origins) on a target to require confirmation or stop the build instead. A renamed column of an incremental or snapshot model gets ` (migrate_from )` in its `columns` block, so the next build renames it in place as a [column migration](/docs/concepts/models/column-migrations). Table and view models are rebuilt with the new column anyway; they get the declaration only when the model itself declares `migrate_from`. @@ -133,9 +135,9 @@ The command refuses a rename while the model, or the column, still declares `mig A project that already fails some project rules, such as a folder holding more models than a layout rule allows, can still be refactored, so that a move can repair it. Only rule failures are tolerated: a project with syntax, binding, or reference errors is refused before planning, as is any project `sqb compile` cannot load. -The command compiles the project as `sqb compile` does, including SQL test planning, records its rule failures before planning, and compares the edited project with them. A rule failure of the edited project is existing when the project already had one with the same code, at the same place (a moved file keeps its identity), and with the same message. It is also existing when only its numbers changed and none grew, such as a folder that now holds 41 models instead of 44. Any other change, new numbers, larger numbers, or different words, makes it new or worse, and it blocks the write, as do all errors that are not rule failures. Rules stay enabled throughout; nothing is ignored. +A rule failure that the project already had, at the same place and with the same message, doesn't block the change. Neither does one whose numbers only went down, such as a folder that now holds 41 models instead of 44. A new or worse failure blocks the write. Rules stay enabled throughout. -The output, including `--dry-run`, lists the existing failures under `Existing rule failures`, with how many the change resolved and how many remain. Some rules report a failure at one sample file of a group, so moving part of the group can change the reported place or message and block the move even when it helps. Move the whole group in one batch instead: moving the direct models of a folder into a sub-folder at once, so that it turns from a leaf into a branch, is `sqb mv a models/entry/records/ b models/entry/records/ c models/entry/records/`. +The output, including `--dry-run`, lists them under `Existing rule failures`, with how many the change resolved. Some rules report a failure at one sample file of a group, so moving part of the group can block the move even when it helps. Move the whole group in one batch instead: moving the direct models of a folder into a sub-folder at once, so that it turns from a leaf into a branch, is `sqb mv a models/entry/records/ b models/entry/records/ c models/entry/records/`. ## Refusals diff --git a/website/src/content/docs/docs/cli/scenario.mdx b/website/src/content/docs/docs/cli/scenario.mdx index 3b7feb3e67..43e2c46638 100644 --- a/website/src/content/docs/docs/cli/scenario.mdx +++ b/website/src/content/docs/docs/cli/scenario.mdx @@ -34,7 +34,7 @@ sqb scenario test [flags] | `--max-snapshot-total-rows` | Override total row limit for capture | | `--max-snapshot-bytes` | Override per-relation byte limit for capture | | `--max-snapshot-total-bytes` | Override total byte limit for capture | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | | `--since ` | Run only the scenarios affected by changes since the merge base of `` and `HEAD`; `@base` reads `[git] base_ref` | | `--explain` | With `--since`, print why each selected scenario was selected | @@ -152,7 +152,7 @@ sqb scenario capture [flags] | `--max-snapshot-total-rows` | Override total row limit | | `--max-snapshot-bytes` | Override per-relation byte limit | | `--max-snapshot-total-bytes` | Override total byte limit | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | ### Examples diff --git a/website/src/content/docs/docs/cli/scope.mdx b/website/src/content/docs/docs/cli/scope.mdx index 21c40b70bc..ad9fee78fe 100644 --- a/website/src/content/docs/docs/cli/scope.mdx +++ b/website/src/content/docs/docs/cli/scope.mdx @@ -1,6 +1,8 @@ --- -title: "scope" +title: "sqb scope" description: "Inspect declaration visibility, usage, placement, and move impact offline." +sidebar: + label: "scope" --- `sqb scope` explains the lexical environment around native SQLBuild macros, enums, and constants. Use it to see what a resource can access, what it actually uses, why a declaration is visible or inaccessible, and whether a move would cross a scope boundary. @@ -399,9 +401,9 @@ Scope reports describe declarations without exposing authored values or runtime There is no `--show-values` option. -## Native SQLBuild Only +## SQLBuild declarations only -`sqb scope` inspects native SQLBuild authored resources and declarations. It does not discover or emulate dbt models, dbt or Jinja macros, package dispatch, dbt manifests, dbt selectors, dbt tests, or dbt schema YAML visibility. An external dbt graph dependency does not contribute declarations or lexical scope. +`sqb scope` covers SQLBuild declarations only. Models and macros in a [dbt project](/docs/concepts/dbt-compatibility/overview) used through `sqb dbt` are not part of any scope. For declaration directory rules, placement checks, test access through expected output, and scoped macro imports, see [Declarations and Scopes](/docs/concepts/declaration-scopes). diff --git a/website/src/content/docs/docs/cli/seed.mdx b/website/src/content/docs/docs/cli/seed.mdx index ec3a253b85..977c2ba030 100644 --- a/website/src/content/docs/docs/cli/seed.mdx +++ b/website/src/content/docs/docs/cli/seed.mdx @@ -22,7 +22,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb | `SELECTOR ...` | Positional selectors, the same as `--select` | | `--select`, `-s` | Select specific seeds by name | | `--exclude` | Exclude specific seeds | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | ## Examples diff --git a/website/src/content/docs/docs/cli/test.mdx b/website/src/content/docs/docs/cli/test.mdx index 4f4048db4a..d810a8fec4 100644 --- a/website/src/content/docs/docs/cli/test.mdx +++ b/website/src/content/docs/docs/cli/test.mdx @@ -24,7 +24,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb | `SELECTOR ...` | Positional selectors, the same as `--select` | | `--select`, `-s` | Select tests by name (`order_status_rules` or `test:order_status_rules`), or the tests targeting specific models | | `--exclude` | Exclude tests by name, or the tests targeting specific models | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/adapters/snowflake#command-group-warehouses) and the connection warehouse | | `--test-timeout ` | Time limit for each test in this run, such as `30s` or `2m`, or `disabled`; overrides `TEST(timeout ...)`, the target and `[settings] test_timeout` | | `--since ` | Run only the tests affected by changes since the merge base of `` and `HEAD` (see [Run only affected tests](#run-only-affected-tests)); `@base` reads `[git] base_ref` | | `--explain` | With `--since`, print why each selected test was selected | @@ -46,7 +46,7 @@ are strings in JSON so their scale is preserved. Each test check in JSON output `duration_ms`. Each test row shows how many real (unmocked) models the test executes, such as `3 real models`. A -test that runs past its [time limit](/docs/concepts/testing#time-limits) is reported as `TIMEOUT`, +test that runs past its [time limit](/docs/concepts/testing-reference#time-limits) is reported as `TIMEOUT`, counted under `TIMEOUT=` and excluded from `FAIL=` in the summary line (`PASS=4 FAIL=0 TIMEOUT=1 TOTAL=5`), and the run continues with the remaining tests. In `--json` output, `summary.fail_count` still includes timed-out tests, and `summary.timeout_count` reports how @@ -174,14 +174,13 @@ of guessing; compare with a later ref whose project compiles, or run without `-- with its default target instead; the run says so, and because the target changed, every test is selected. -SQL comparisons ignore layout and comments using a generic SQL tokenizer. In dialects where `--` -starts a comment only when followed by whitespace, such as MySQL, `1--1` is read as `1` followed by -a comment, so an edit inside such an expression can go unnoticed. The base tree is written from git -objects with the repository's filters. Git LFS content is restored for files that an unanchored -`.gitattributes` pattern marks, such as `*.csv filter=lfs`, but a pattern anchored to a path, such -as `/data/*.csv` at the repository root for a project in a subdirectory, does not match the -exported paths. Such a file is an LFS pointer at the base and real content in the checkout, so it -always counts as changed: at most extra tests run. +Details: + +- SQL comparisons ignore layout and comments. In dialects where `--` starts a comment only before + whitespace, such as MySQL, an edit inside an expression like `1--1` can go unnoticed. +- A file stored in Git LFS through a pattern anchored to a path, such as `/data/*.csv`, always + counts as changed, so extra tests may run. Unanchored patterns such as `*.csv filter=lfs` work as + expected. `` is any git ref or commit, such as the merge request's target branch from a CI variable. `--since` always takes a value. A ref that does not exist, or a shallow clone that does not reach diff --git a/website/src/content/docs/docs/concepts/adapters.mdx b/website/src/content/docs/docs/concepts/adapters.mdx index 0bdbeff4a5..28adf79759 100644 --- a/website/src/content/docs/docs/concepts/adapters.mdx +++ b/website/src/content/docs/docs/concepts/adapters.mdx @@ -39,89 +39,5 @@ adapter = "duckdb" ## Custom adapters -You can write your own adapter for any database engine by placing Python files under `adapters/` in your project directory. - -### Extending a built-in adapter - -The most common case is extending an existing adapter with custom behavior. Subclass the built-in adapter, set a new `adapter_name`, and override what you need: - -```python -# adapters/duckdb_plus.py -from sqlbuild.adapters.duckdb.classes.duckdb_adapter import DuckDbAdapter - -class DuckDbPlusAdapter(DuckDbAdapter): - adapter_name = "duckdb_plus" - - def connect(self, config): - connection = super().connect(config) - # custom setup - load extensions, set pragmas, etc. - self.execute(connection=connection, sql="SET enable_progress_bar = true") - return connection -``` - -```toml -adapter = "duckdb_plus" -``` - -A subclass inherits the parent's capabilities. For example, an adapter extending the Snowflake -adapter keeps `supports_session_warehouse = True`, so it can use -[command-group warehouses](/docs/concepts/project-configuration#command-group-warehouses) and -`--warehouse`. An adapter written from scratch that sets this attribute receives the selected -warehouse as the connection's `warehouse` key. - -### Writing an adapter from scratch - -For a database engine with no built-in support, subclass `BaseAdapter`. It provides ANSI SQL defaults for most methods - you only need to implement `connect`, `execute`, `close`, and any methods where your engine differs from standard SQL: - -```python -# adapters/my_database.py -from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter - -class MyDatabaseAdapter(BaseAdapter): - adapter_name = "my_database" - sql_analysis_dialect_name = "postgres" # SQL dialect for validation and lineage - - def connect(self, config): - ... - - def execute(self, *, connection, sql): - ... - - def close(self, connection): - ... -``` - -Set `sql_analysis_dialect_name` to the SQL dialect name that matches your engine's SQL syntax. This enables compile-time SQL validation, column inference, column lineage, and local scenario replay for your adapter. If omitted, SQLBuild uses generic SQL parsing. SQL analysis is powered by [Polyglot](https://github.com/tobilg/polyglot), a Rust reimplementation of SQLGlot supporting 32+ dialects. - -For full control with no inherited defaults, subclass `StrictAdapter` instead. Every method is abstract and must be implemented explicitly. SQLBuild raises a clear error listing any unimplemented methods. - -A few `StrictAdapter` methods are optional and have a documented default. `create_sql_test_output` and `drop_sql_test_output` let an adapter store a unit test's actual output once, in a temporary relation, so every comparison of that test reads it instead of recomputing the model chain. By default `create_sql_test_output` returns `None`, and SQLBuild runs the test's inline comparison query instead, so adapters that do not implement them keep working unchanged. `render_create_temporary_function` returns the call target and DDL for a session-scoped copy of a project function, which `sqb test` calls instead of the built function (see [Functions in `sqb test`](/docs/concepts/testing#functions-in-sqb-test)). By default it returns `None`, and tests call the built function. `render_drop_schema` returns the statements that drop a schema and everything in it, which [`sqb review`](/docs/cli/review) uses to remove the schema it built into. By default it raises an error, and `sqb review` is unavailable for that adapter. - -### Discovery rules - -- SQLBuild discovers all `.py` files under `adapters/` recursively (excluding `__init__.py` and files starting with `_`) -- Each file is scanned for classes that define a string `adapter_name` and subclass `StrictAdapter` (or any of its subclasses like `BaseAdapter` or a built-in adapter) -- Adapter names must be unique across all adapter files - duplicates raise an error -- Custom adapter names cannot shadow built-in names (`duckdb`, `motherduck`, `snowflake`, `bigquery`, `databricks`, `postgres`, `sqlite`, `sqlserver`, `clickhouse`) - -### Adapter class hierarchy - -``` -StrictAdapter (fully abstract - all methods must be implemented) - └── BaseAdapter (ANSI SQL defaults - override only what differs) - ├── DuckDbAdapter - ├── MotherDuckAdapter - ├── SnowflakeAdapter - ├── BigQueryAdapter - ├── DatabricksAdapter - ├── PostgresAdapter - ├── SqlServerAdapter - └── ClickHouseAdapter -``` - -`StrictAdapter` composes four mixins that define the full adapter contract: - -- **ConnectionMixin** - `connect`, `close`, `begin`, `commit`, `rollback` -- **SchemaMixin** - `relation_exists`, `list_relations`, `get_columns`, `ensure_schema` -- **MaterializationMixin** - `create_table_as`, `create_view_as`, `drop`, `rename`, `swap`, `load_seed` -- **DiffMixin** - `diff_schema`, `diff_rows`, `sample_unequal_rows`, `sample_side_only_rows`, `diff_unkeyed_rows` +To extend a built-in adapter or add another database engine, see +[Custom adapters](/docs/concepts/adapters/custom). diff --git a/website/src/content/docs/docs/concepts/adapters/bigquery.mdx b/website/src/content/docs/docs/concepts/adapters/bigquery.mdx index 0613aa982a..aeb8be61b0 100644 --- a/website/src/content/docs/docs/concepts/adapters/bigquery.mdx +++ b/website/src/content/docs/docs/concepts/adapters/bigquery.mdx @@ -70,4 +70,4 @@ See [Collections and Rendering](/docs/concepts/constants/collections-and-renderi ## Functions in unit tests BigQuery temporary functions exist only inside one script, so `sqb test` calls the built function. -Run `sqb build` before testing a new or changed function. See [Functions in `sqb test`](/docs/concepts/testing#functions-in-sqb-test). +Run `sqb build` before testing a new or changed function. See [Functions in `sqb test`](/docs/concepts/testing-reference#functions-in-sqb-test). diff --git a/website/src/content/docs/docs/concepts/adapters/clickhouse.mdx b/website/src/content/docs/docs/concepts/adapters/clickhouse.mdx index 2444770b00..def10cd800 100644 --- a/website/src/content/docs/docs/concepts/adapters/clickhouse.mdx +++ b/website/src/content/docs/docs/concepts/adapters/clickhouse.mdx @@ -187,4 +187,4 @@ history for a whole team, whichever adapter the project builds with. Set reader users. Recall creates its database, tables and views on first use, keeps every output byte by default, and needs only `SELECT` and `INSERT` once the tables exist; readers need only `SELECT` and may be `readonly = 1` users. See -[Sharing history in ClickHouse](/docs/concepts/observability/recall#sharing-history-in-clickhouse). +[Sharing Recall in ClickHouse](/docs/concepts/observability/recall-clickhouse). diff --git a/website/src/content/docs/docs/concepts/adapters/custom.mdx b/website/src/content/docs/docs/concepts/adapters/custom.mdx new file mode 100644 index 0000000000..921e8d9b6a --- /dev/null +++ b/website/src/content/docs/docs/concepts/adapters/custom.mdx @@ -0,0 +1,91 @@ +--- +title: "Custom adapters" +description: "Extend a built-in adapter or write one for another database engine in Python." +--- + +You can write your own adapter for any database engine by placing Python files under `adapters/` in your project directory. + +## Extending a built-in adapter + +The most common case is extending an existing adapter with custom behavior. Subclass the built-in adapter, set a new `adapter_name`, and override what you need: + +```python +# adapters/duckdb_plus.py +from sqlbuild.adapters.duckdb.classes.duckdb_adapter import DuckDbAdapter + +class DuckDbPlusAdapter(DuckDbAdapter): + adapter_name = "duckdb_plus" + + def connect(self, config): + connection = super().connect(config) + # custom setup - load extensions, set pragmas, etc. + self.execute(connection=connection, sql="SET enable_progress_bar = true") + return connection +``` + +```toml +adapter = "duckdb_plus" +``` + +A subclass inherits the parent's capabilities. For example, an adapter extending the Snowflake +adapter keeps `supports_session_warehouse = True`, so it can use +[command-group warehouses](/docs/concepts/adapters/snowflake#command-group-warehouses) and +`--warehouse`. An adapter written from scratch that sets this attribute receives the selected +warehouse as the connection's `warehouse` key. + +## Writing an adapter from scratch + +For a database engine with no built-in support, subclass `BaseAdapter`. It provides ANSI SQL defaults for most methods - you only need to implement `connect`, `execute`, `close`, and any methods where your engine differs from standard SQL: + +```python +# adapters/my_database.py +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter + +class MyDatabaseAdapter(BaseAdapter): + adapter_name = "my_database" + sql_analysis_dialect_name = "postgres" # SQL dialect for validation and lineage + + def connect(self, config): + ... + + def execute(self, *, connection, sql): + ... + + def close(self, connection): + ... +``` + +Set `sql_analysis_dialect_name` to the SQL dialect name that matches your engine's SQL syntax. This enables compile-time SQL validation, column inference, column lineage, and local scenario replay for your adapter. If omitted, SQLBuild uses generic SQL parsing. + +For full control with no inherited defaults, subclass `StrictAdapter` instead. Every method is abstract and must be implemented explicitly. SQLBuild raises a clear error listing any unimplemented methods. + +A few `StrictAdapter` methods are optional and have a documented default. `create_sql_test_output` and `drop_sql_test_output` let an adapter store a unit test's actual output once, in a temporary relation, so every comparison of that test reads it instead of recomputing the model chain. By default `create_sql_test_output` returns `None`, and SQLBuild runs the test's inline comparison query instead, so adapters that do not implement them keep working unchanged. `render_create_temporary_function` returns the call target and DDL for a session-scoped copy of a project function, which `sqb test` calls instead of the built function (see [Functions in `sqb test`](/docs/concepts/testing-reference#functions-in-sqb-test)). By default it returns `None`, and tests call the built function. `render_drop_schema` returns the statements that drop a schema and everything in it, which [`sqb review`](/docs/cli/review) uses to remove the schema it built into. By default it raises an error, and `sqb review` is unavailable for that adapter. + +## Discovery rules + +- SQLBuild discovers all `.py` files under `adapters/` recursively (excluding `__init__.py` and files starting with `_`) +- Each file is scanned for classes that define a string `adapter_name` and subclass `StrictAdapter` (or any of its subclasses like `BaseAdapter` or a built-in adapter) +- Adapter names must be unique across all adapter files - duplicates raise an error +- Custom adapter names cannot shadow built-in names (`duckdb`, `motherduck`, `snowflake`, `bigquery`, `databricks`, `postgres`, `sqlite`, `sqlserver`, `clickhouse`) + +## Adapter class hierarchy + +``` +StrictAdapter (fully abstract - all methods must be implemented) + └── BaseAdapter (ANSI SQL defaults - override only what differs) + ├── DuckDbAdapter + ├── MotherDuckAdapter + ├── SnowflakeAdapter + ├── BigQueryAdapter + ├── DatabricksAdapter + ├── PostgresAdapter + ├── SqlServerAdapter + └── ClickHouseAdapter +``` + +`StrictAdapter` composes four mixins that define the full adapter contract: + +- **ConnectionMixin** - `connect`, `close`, `begin`, `commit`, `rollback` +- **SchemaMixin** - `relation_exists`, `list_relations`, `get_columns`, `ensure_schema` +- **MaterializationMixin** - `create_table_as`, `create_view_as`, `drop`, `rename`, `swap`, `load_seed` +- **DiffMixin** - `diff_schema`, `diff_rows`, `sample_unequal_rows`, `sample_side_only_rows`, `diff_unkeyed_rows` diff --git a/website/src/content/docs/docs/concepts/adapters/databricks.mdx b/website/src/content/docs/docs/concepts/adapters/databricks.mdx index 8dcc5e3e2c..6fc258b352 100644 --- a/website/src/content/docs/docs/concepts/adapters/databricks.mdx +++ b/website/src/content/docs/docs/concepts/adapters/databricks.mdx @@ -51,4 +51,4 @@ the authoritative session namespace. `sqb test` creates each SQL UDF and table function the selected tests call as a session `TEMPORARY FUNCTION __sqb_test__`. Python UDFs cannot be temporary, so tests call the built -Python function. See [Functions in `sqb test`](/docs/concepts/testing#functions-in-sqb-test). +Python function. See [Functions in `sqb test`](/docs/concepts/testing-reference#functions-in-sqb-test). diff --git a/website/src/content/docs/docs/concepts/adapters/duckdb.mdx b/website/src/content/docs/docs/concepts/adapters/duckdb.mdx index e526cf06d3..c8401bcb8c 100644 --- a/website/src/content/docs/docs/concepts/adapters/duckdb.mdx +++ b/website/src/content/docs/docs/concepts/adapters/duckdb.mdx @@ -3,33 +3,8 @@ title: "DuckDB" description: "DuckDB adapter configuration for SQLBuild." --- -DuckDB is included as a core dependency. No extra installation needed. - -SQLBuild requires DuckDB 2.0 or later. DuckDB 2.0 writes new database files in a storage format -that DuckDB 1.x cannot open, so a database file SQLBuild creates can only be read by DuckDB 2.0 -clients. A file that already exists keeps its storage version, so a file created by DuckDB 1.x -stays readable by 1.x. To create a new file that 1.x can read, create it before the first build -with an explicit storage version, for example -`duckdb my_project.duckdb -storage-version v1.5.0 -c "SELECT 1"`. - -## SQL that behaves differently on DuckDB 2.0 - -DuckDB 2.0 changes how some SQL runs. SQLBuild runs model SQL as written, so check models for -these before upgrading: - -- Arrow lambdas such as `list_filter(xs, x -> x > 0)` are rejected by default. Write - `list_filter(xs, lambda x: x > 0)` instead. DuckDB 2.1 removes the arrow syntax. -- `~` and `!~` match a regular expression anywhere in the string, as in PostgreSQL, so - `'abc' ~ 'b'` is now true. Use `regexp_full_match` for a match of the whole string. -- `MERGE` fails when one target row matches several source rows, instead of picking one. Keep - an incremental model's new rows unique on its `unique_key`. -- An untyped `NULL`, as in `SELECT NULL AS discount`, creates a column of type `"NULL"` that - rejects later non-null values. Cast it, for example `CAST(NULL AS DECIMAL(18, 2)) AS discount`. -- Integer division `//` and modulo `%` by zero raise an error instead of returning `NULL`. Wrap - the expression in `TRY(...)` to get `NULL`. - -dlt's dataset read API calls a DuckDB method that DuckDB 2.0 removed when it reads a loaded table -back as one whole Arrow table, so that read fails. Dataframe and chunked Arrow reads still work. Loading data with dlt is not affected. +DuckDB is included as a core dependency. No extra installation needed. SQLBuild requires DuckDB 2.0 +or later; see [Upgrading from DuckDB 1.x](#upgrading-from-duckdb-1x). ## Connection config @@ -82,17 +57,34 @@ the dbt project or the working directory. Every connection of a command attaches it once: connections to the same project database share the attachment. -## Table promotion mode - -DuckDB defaults to `staged` promotion: tables are materialized into a staging table, audited, then swapped into the target. This is configurable in `settings`: - -```toml -[settings] -table_promotion_mode = "staged" -``` - ## Functions in unit tests `sqb test` copies each SQL UDF and table function the selected tests call into a `CREATE TEMP MACRO` on every test connection, called as `temp.main.`, so tests use the current project -definition without a build. Python UDFs are registered on each test connection. See [Functions in `sqb test`](/docs/concepts/testing#functions-in-sqb-test). +definition without a build. Python UDFs are registered on each test connection. See [Functions in `sqb test`](/docs/concepts/testing-reference#functions-in-sqb-test). + +## Upgrading from DuckDB 1.x + +DuckDB 2.0 writes new database files in a storage format +that DuckDB 1.x cannot open, so a database file SQLBuild creates can only be read by DuckDB 2.0 +clients. A file that already exists keeps its storage version, so a file created by DuckDB 1.x +stays readable by 1.x. To create a new file that 1.x can read, create it before the first build +with an explicit storage version, for example +`duckdb my_project.duckdb -storage-version v1.5.0 -c "SELECT 1"`. + +When you upgrade from DuckDB 1.x, check models for SQL that runs differently on 2.0. SQLBuild runs +model SQL as written: + +- Arrow lambdas such as `list_filter(xs, x -> x > 0)` are rejected by default. Write + `list_filter(xs, lambda x: x > 0)` instead. DuckDB 2.1 removes the arrow syntax. +- `~` and `!~` match a regular expression anywhere in the string, as in PostgreSQL, so + `'abc' ~ 'b'` is true. Use `regexp_full_match` for a match of the whole string. +- `MERGE` fails when one target row matches several source rows, instead of picking one. Keep + an incremental model's new rows unique on its `unique_key`. +- An untyped `NULL`, as in `SELECT NULL AS discount`, creates a column of type `"NULL"` that + rejects later non-null values. Cast it, for example `CAST(NULL AS DECIMAL(18, 2)) AS discount`. +- Integer division `//` and modulo `%` by zero raise an error instead of returning `NULL`. Wrap + the expression in `TRY(...)` to get `NULL`. + +dlt's dataset read API calls a DuckDB method that DuckDB 2.0 removed when it reads a loaded table +back as one whole Arrow table, so that read fails. Dataframe and chunked Arrow reads still work. Loading data with dlt is not affected. diff --git a/website/src/content/docs/docs/concepts/adapters/motherduck.mdx b/website/src/content/docs/docs/concepts/adapters/motherduck.mdx index 31a55a561b..e3f6b89e89 100644 --- a/website/src/content/docs/docs/concepts/adapters/motherduck.mdx +++ b/website/src/content/docs/docs/concepts/adapters/motherduck.mdx @@ -86,4 +86,4 @@ This lets you develop and test locally with zero MotherDuck compute cost, then d `sqb test` copies each SQL UDF and table function the selected tests call into a session `TEMP MACRO`, the same as [DuckDB](/docs/concepts/adapters/duckdb#functions-in-unit-tests), so the -functions in your MotherDuck database are not replaced. See [Functions in `sqb test`](/docs/concepts/testing#functions-in-sqb-test). +functions in your MotherDuck database are not replaced. See [Functions in `sqb test`](/docs/concepts/testing-reference#functions-in-sqb-test). diff --git a/website/src/content/docs/docs/concepts/adapters/postgres.mdx b/website/src/content/docs/docs/concepts/adapters/postgres.mdx index c38c0a2714..7ee9b1c273 100644 --- a/website/src/content/docs/docs/concepts/adapters/postgres.mdx +++ b/website/src/content/docs/docs/concepts/adapters/postgres.mdx @@ -69,4 +69,4 @@ schema = "dev" calls it as `pg_temp.`, so tests use the current project definition without a build. Temporary functions belong to one server session: behind a transaction-mode pooler, run `sqb test` through a direct connection or a session-mode pool, or set `temporary_test_functions = false` so tests call the -built functions. See [Functions in `sqb test`](/docs/concepts/testing#functions-in-sqb-test). +built functions. See [Functions in `sqb test`](/docs/concepts/testing-reference#functions-in-sqb-test). diff --git a/website/src/content/docs/docs/concepts/adapters/snowflake.mdx b/website/src/content/docs/docs/concepts/adapters/snowflake.mdx index e04ec2e193..38c05437b6 100644 --- a/website/src/content/docs/docs/concepts/adapters/snowflake.mdx +++ b/website/src/content/docs/docs/concepts/adapters/snowflake.mdx @@ -44,10 +44,59 @@ On connect, SQLBuild applies the connection's role and warehouse, then applies t target's authoritative database and schema. These ensure the session context is set correctly regardless of the user's default settings. -The warehouse can differ by command. A target's `[targets..warehouses]` section selects -one warehouse for build-style commands and another for ad-hoc commands such as `sqb query` and -`sqb diff`, and `--warehouse` overrides both for one invocation. See -[command-group warehouses](/docs/concepts/project-configuration#command-group-warehouses). +## Command-group warehouses + +Builds and ad-hoc queries usually need different compute. A target can set a warehouse for each +group of commands, so `sqb build` runs on a large warehouse while `sqb query` and `sqb diff` use a +small one: + +```toml +[connections.warehouse] +account = "my_org-my_account" +warehouse = "ANALYTICS_WH" # fallback for every command + +[targets.dev] +connection = "warehouse" +database = "ANALYTICS" +schema = "DEV" + +[targets.dev.warehouses] +build = "BUILD_WH" +query = "ADHOC_WH" +``` + +| Group | Commands | +|-------|----------| +| `build` | `audit`, `build`, `check`, `clone`, `dbt` (SQLBuild's own work), `load`, `plan`, `seed` | +| `query` | `contract`, `cost`, `debug`, `diff`, `freshness`, `janitor`, `query`, `scenario`, `test` | + +Each command uses the first of these that is set: + +1. `--warehouse `, for that run only +2. The target's warehouse for the command's group +3. The connection's `warehouse` +4. The Snowflake user's default warehouse + +Every connection a command opens uses the same warehouse. Tests and audits that run inside +`sqb build` use the `build` warehouse. Under `sqb dbt`, dbt's own nodes use the warehouse in their +dbt profile. + +A group warehouse wins over a connection `warehouse` from either file. To change one for yourself, +set the group in `sqlbuild_local.toml`; the other group keeps its project value: + +```toml +# sqlbuild_local.toml +[targets.dev.warehouses] +query = "ALICE_XS_WH" +``` + +Values accept `${ENV:...}` and `${my_var}` templates and must be Snowflake identifiers, unquoted +(`ADHOC_WH`) or double-quoted (`"Adhoc WH"`). `sqb debug` shows each group's warehouse and where it +came from. + +Groups need the Snowflake adapter or a custom adapter built on it; on other adapters they are an +error. If you switch `adapter` locally, for example to DuckDB, the shared groups are ignored and +`sqb debug` reports them as `not used`. ## Case-sensitive column names @@ -72,46 +121,19 @@ A snapshot or microbatch build stops when an unquoted setting names a case-sensi an error such as `unique_key 'orderId' names the case-sensitive column "orderId"; write it with its quotes, as '"orderId"'`. -## Planning metadata - -Planning reads relation and column metadata with `SHOW TABLES`, `SHOW VIEWS`, and `SHOW COLUMNS` -once per schema, several schemas at a time. `SHOW` commands are served by Snowflake's metadata -service, so they don't wait for, or resume, a warehouse. Two reads still use `INFORMATION_SCHEMA` -and need a running warehouse: - -- `LAST_ALTERED` for source freshness, read only for the sources the plan observes. -- Columns for a schema whose `SHOW COLUMNS` output is exactly 10,000 rows, the documented - limit at which the output may be cut off. +## Metadata reads -Run `sqb plan -v` to see each metadata query with its elapsed time and row count. - -## Build metadata - -Builds and scenario runs read relation metadata with `SHOW` commands too, so these reads don't -need a running warehouse either: - -| Read | Statement | -| -- | -- | -| Whether one relation exists | `SHOW TERSE TABLES` and `SHOW TERSE VIEWS` | -| A second relation in the same schema | `SHOW TABLES` and `SHOW VIEWS` for the whole schema, once | -| A table's time travel retention | `SHOW TABLES LIKE`, or the schema listing | -| One relation's columns | `SHOW COLUMNS` | -| A scenario's leftover relations | `SHOW TERSE OBJECTS` for the scenario's name prefix | - -Each run reuses these answers until it changes that relation, for example by replacing a delta -table or adding a column, and then reads it again. +Planning and builds read table, view and column metadata with `SHOW` commands, which don't need a +running warehouse. Source freshness, which reads `LAST_ALTERED` from `INFORMATION_SCHEMA`, and +columns of a schema with 10,000 or more columns need one. Probes for sources the listing didn't +find may need one too. Run `sqb plan -v` to see each metadata query with its time +and row count. ## State tables -SQLBuild records its state in `_sqlbuild_*` tables, such as `_sqlbuild_fingerprints`, -`_sqlbuild_source_freshness`, and `_sqlbuild_microbatches`. On Snowflake, SQLBuild creates every -state table as a permanent table with `DATA_RETENTION_TIME_IN_DAYS = 90`, the maximum time travel, -so the table also has Fail-safe. On editions that allow only 1 day of time travel, Snowflake -rejects 90 and SQLBuild creates the table with 1 day instead. - -The retention is set only in the statement that creates the table. SQLBuild never alters an -existing state table, so a state table that already exists keeps its current table type and -retention. Models are not affected: their table type follows the model `table_type` setting. +SQLBuild records its state in `_sqlbuild_*` tables. On Snowflake it creates them as permanent tables +with 90 days of time travel, or 1 day on editions that allow only that. It never alters an existing +state table. Models are not affected: their table type follows the model's `table_type` setting. ## Shared connections across targets @@ -156,26 +178,22 @@ contract rate. Use `sqb cost`, `sqb cost latest`, `sqb cost `, or provide a versioned, decimal-safe output contract. Pending detail records are refreshed from Snowflake when inspected again. -SQLBuild reads `INFORMATION_SCHEMA.QUERY_HISTORY` for the whole run in one query when the run -finishes. Each read has a 10-second statement timeout; a window that reaches it, or that returns -more rows than one read allows, is split in half and read again. Collection at the end of a command -has a 30-second budget so the command exits promptly. If history cannot be read within it, the -build is unaffected and its cost is recorded as `pending`, with a message naming the timeout. -`sqb cost ` collects it later with a 5-minute budget, which completes long runs. Any other -failure is recorded as `collection_failed` with the Snowflake error that caused it. - -These values are attributed compute credits and estimated cost, not Snowflake-billed credits or -invoice reconciliation. The estimate uses only query history visible to the executing role and does -not reconstruct invisible concurrent work, warehouse resume or idle tail, the 60-second minimum, -cloud-services credits, contract adjustments, or multi-cluster billing. A statement run with a `QUERY_TAG` -of your own is not recognised as a metadata-only `CREATE`, so it may report missing warehouse -metadata. Run metadata and query IDs -are stored under `target/executions//`; that statement ledger stores only an SQL digest, -not SQL text. Executed SQL artifacts are stored separately under the sensitive `target/run/` tree. +The estimate is read from `INFORMATION_SCHEMA.QUERY_HISTORY` when the run finishes. If it can't be +read within 30 seconds, the build is unaffected and its cost is recorded as `pending`; +`sqb cost ` collects it later. Any other failure is recorded as `collection_failed` with the +Snowflake error. + +The estimate is attributed compute, not your Snowflake bill. It sees only query history visible to +the executing role, and leaves out warehouse resume and idle time, the 60-second minimum, +cloud-services credits, contract adjustments and multi-cluster billing. + +A statement run with your own `QUERY_TAG` may report missing warehouse metadata. Executed SQL is +kept under `target/run/`, which can hold sensitive data; the cost records under +`target/executions/` store only a digest of each statement. ## Functions in unit tests `sqb test` creates each UDF and table function the selected tests call, SQL or Python, as a `TEMPORARY FUNCTION __SQB_TEST__` in the session's current schema. The session needs a current schema and permission to create functions there; otherwise tests call the built function. See -[Functions in `sqb test`](/docs/concepts/testing#functions-in-sqb-test). +[Functions in `sqb test`](/docs/concepts/testing-reference#functions-in-sqb-test). diff --git a/website/src/content/docs/docs/concepts/adapters/sqlserver.mdx b/website/src/content/docs/docs/concepts/adapters/sqlserver.mdx index 991b99c390..59fa39854b 100644 --- a/website/src/content/docs/docs/concepts/adapters/sqlserver.mdx +++ b/website/src/content/docs/docs/concepts/adapters/sqlserver.mdx @@ -85,4 +85,4 @@ schema = "dev" ## Functions in unit tests SQL Server has no session-scoped functions, so `sqb test` calls the built function. Run `sqb build` -before testing a new or changed function. See [Functions in `sqb test`](/docs/concepts/testing#functions-in-sqb-test). +before testing a new or changed function. See [Functions in `sqb test`](/docs/concepts/testing-reference#functions-in-sqb-test). diff --git a/website/src/content/docs/docs/concepts/audits.mdx b/website/src/content/docs/docs/concepts/audits.mdx index 5b39a4fd3d..fd74d1c3c0 100644 --- a/website/src/content/docs/docs/concepts/audits.mdx +++ b/website/src/content/docs/docs/concepts/audits.mdx @@ -281,7 +281,7 @@ runs the audit exactly where any other audit on that target runs: promotion, a `final` audit checks the staged table before promotion; with `immediate` promotion, or for final-phase audits on incremental and snapshot models, it runs after the write, like any other audit there. See [Run scope](#run-scope) and - [Table promotion mode](/docs/concepts/project-configuration#table-promotion-mode). + [Table promotion mode](/docs/concepts/models/materializations#table). - On a source or seed, it runs before any model that depends on it is built. An `error` failure blocks everything downstream of the target, and where the target's other audits diff --git a/website/src/content/docs/docs/concepts/column-lineage.mdx b/website/src/content/docs/docs/concepts/column-lineage.mdx index f43b220f69..932b450845 100644 --- a/website/src/content/docs/docs/concepts/column-lineage.mdx +++ b/website/src/content/docs/docs/concepts/column-lineage.mdx @@ -13,7 +13,7 @@ description: "Trace individual columns through your SQL pipeline - understand wh ## How it works -SQLBuild analyzes column lineage statically at compile time. No warehouse connection is needed. The analyzer parses each model's SQL, resolves `ref()` and `source()` calls, and traces columns through `SELECT` lists, CTEs, JOINs, subqueries, and expressions. +SQLBuild analyzes column lineage statically at compile time. No warehouse connection is needed. The analyzer parses each model's SQL, resolves `__ref()` and `__source()` calls, and traces columns through `SELECT` lists, CTEs, JOINs, subqueries, and expressions. Column lineage requires SQL analysis to be enabled in project settings (it is by default). diff --git a/website/src/content/docs/docs/concepts/dbt-compatibility/overview.mdx b/website/src/content/docs/docs/concepts/dbt-compatibility/overview.mdx index dace6c4460..4f2b1ab155 100644 --- a/website/src/content/docs/docs/concepts/dbt-compatibility/overview.mdx +++ b/website/src/content/docs/docs/concepts/dbt-compatibility/overview.mdx @@ -74,37 +74,8 @@ Each step calls the `dbt` CLI directly: `dbt compile` for the manifest, `dbt ls` ### Flags -`sqb dbt plan/run/build` declare the common flags directly. Anything declared goes **before** a `--` separator; any other raw dbt flag goes **after** it and is forwarded verbatim. A flag placed on the wrong side errors rather than silently reaching dbt. - -```bash -sqb dbt build --select path:models/marts --full-refresh -``` - -Declared flags are routed to the right place automatically: - -| Flag | Routed to | Notes | -| ---- | --------- | ----- | -| `--select` / `--exclude` | dbt | dbt resolves selection (see [Selection](/docs/concepts/dbt-compatibility/selection)). | -| `--vars` | dbt **and** SQLBuild | The same vars feed dbt's compile and SQLBuild's own variable resolution, so both sides see identical values. | -| `--full-refresh` | dbt **and** SQLBuild | Requests a full rebuild on both sides. dbt and native SQLBuild models independently apply their model-level `full_refresh` setting, including `false` opt-outs. | -| `--threads` | dbt (`--threads`) and SQLBuild (`--concurrency`) | | -| `--target` / `--project-dir` / `--profiles-dir` / `--profile` / `--target-path` | dbt | Standard dbt locators. | - -#### `--vars` - -`--vars` accepts the same JSON object dbt accepts, and SQLBuild passes it to **both** the underlying `dbt` invocation and its own variable resolution. That means a value referenced as `@@my_var` in SQLBuild model SQL (see [Interpolation](/docs/concepts/interpolation)) and as `{{ var('my_var') }}` in a dbt model both resolve to the value you passed. CLI vars take precedence over project and local config vars. You do not need to declare vars twice. - -```bash -sqb dbt build --select path:models/marts --vars '{"my_var": 1}' -``` - -#### Forwarding other dbt flags - -For any native dbt flag SQLBuild does not declare, put it after `--` and it is passed straight to the `dbt` invocation untouched: - -```bash -sqb dbt build -- --log-level debug -``` +`sqb dbt` accepts the common dbt flags, such as `--select`, `--vars` and `--full-refresh`, and passes +any other dbt flag after `--`. See [`sqb dbt`](/docs/cli/dbt#flags) for how each flag is routed. ## Configuration diff --git a/website/src/content/docs/docs/concepts/declaration-scopes.mdx b/website/src/content/docs/docs/concepts/declaration-scopes.mdx index e4d905a62f..3911b59a29 100644 --- a/website/src/content/docs/docs/concepts/declaration-scopes.mdx +++ b/website/src/content/docs/docs/concepts/declaration-scopes.mdx @@ -89,11 +89,11 @@ the owner folder, or compilation fails; they never change where the declaration Declarations do not flow upward or sideways into sibling directories. -## Terms used in text output +## Labels in `sqb scope` output The CLI's text output uses short labels for the same four rules: -| Canonical label | Plain-language meaning | +| Label | Meaning | |-----------------|------------------------| | `project` | Available throughout the project | | `descendant-public` | Available in this folder and folders below it | @@ -104,11 +104,6 @@ Here, an **owner folder** is simply the folder directly containing a model, test other authored resource. The [Scope Explorer](/docs/concepts/declaration-scopes/explorer) shows the owner folder, its parents, and project-wide declarations separately. -JSON keeps the underlying enum values for integrations. Its `scope` field uses `global`, -`inherited`, `local`, or `private`, while derived `visibility` values use underscores, such as -`descendant_public` and `exact_owner_private`. Consumers should use the fields documented in the -versioned JSON schema rather than parsing text labels. - ## Choose a location | Where the value is needed | Placement | @@ -142,5 +137,5 @@ top-level roles as a genuine project API instead of a neutral dumping ground. Learn the features themselves in [Enums](/docs/concepts/enums), [Constants](/docs/concepts/constants), [Macros](/docs/concepts/macros), [Audits](/docs/concepts/audits), -[Reusable Schemas](/docs/concepts/models/schemas), and [Hooks](/docs/concepts/models/hooks). See [Interpolation](/docs/concepts/interpolation) for project, +[Reusable column schemas](/docs/concepts/models/schemas), and [Hooks](/docs/concepts/models/hooks). See [Variables and placeholders](/docs/concepts/interpolation) for project, environment, and runtime context values, which are separate from declarations. diff --git a/website/src/content/docs/docs/concepts/declaration-scopes/explorer.mdx b/website/src/content/docs/docs/concepts/declaration-scopes/explorer.mdx index b38a84d9d1..edd8e80838 100644 --- a/website/src/content/docs/docs/concepts/declaration-scopes/explorer.mdx +++ b/website/src/content/docs/docs/concepts/declaration-scopes/explorer.mdx @@ -324,6 +324,11 @@ Use `--json` for editor integrations and repository tooling. Output has a versio ordering, filters, pagination, and move-preview results. Constant values, credentials, and secret connection settings are not included. +In JSON, the `scope` field uses `global`, +`inherited`, `local`, or `private`, while derived `visibility` values use underscores, such as +`descendant_public` and `exact_owner_private`. Consumers should use the fields documented in the +versioned JSON schema rather than parsing text labels. + Review the directory rules behind the report. diff --git a/website/src/content/docs/docs/concepts/diff.mdx b/website/src/content/docs/docs/concepts/diff.mdx index 68b4b49225..93e4825fd3 100644 --- a/website/src/content/docs/docs/concepts/diff.mdx +++ b/website/src/content/docs/docs/concepts/diff.mdx @@ -388,4 +388,5 @@ reconciliation, inspection, materialization, comparison, cleanup, and total dura ## Exit codes -In model mode, `sqb diff` returns exit code `0` when all selected models have no differences, `1` when any model has schema or row differences, and `2` when the size guard stopped a default full diff before reading data. Raw-query mode uses the exit codes in [Query output and exit codes](#query-output-and-exit-codes). This makes it usable in CI pipelines as a validation gate. +`sqb diff` exits `1` when it finds differences, so it works as a CI gate. See +[`sqb diff`](/docs/cli/diff#exit-codes) for every exit code. diff --git a/website/src/content/docs/docs/concepts/enums-and-constants.mdx b/website/src/content/docs/docs/concepts/enums-and-constants.mdx deleted file mode 100644 index 75c26e5fd1..0000000000 --- a/website/src/content/docs/docs/concepts/enums-and-constants.mdx +++ /dev/null @@ -1,62 +0,0 @@ ---- -title: "Enums and Constants Have Moved" -description: "Find the new focused guides for enums, constants, collections, contracts, and private values." ---- - -import Card from '../../../../components/mintlify/Card.astro'; -import CardGroup from '../../../../components/mintlify/CardGroup.astro'; - -Enums and constants now have separate guides so each feature is easier to find and learn. This page -remains available for existing links and bookmarks but is not part of the main navigation. - - - - Define fixed string or integer domains and reference validated members. - - - Define reusable scalar values and reference them safely from SQL. - - - Work with lists, sets, objects, value lists, and native arrays. - - - Keep enums and constants inside one model. - - - -## Public declarations - -See [Enums](/docs/concepts/enums) and [Constants](/docs/concepts/constants) for project-wide declarations. - -## Constant values - -See [Constants](/docs/concepts/constants#scalar-values) for scalars and -[Collections and Rendering](/docs/concepts/constants/collections-and-rendering) for collections. - -## References and rendering - -See [Collections and Rendering](/docs/concepts/constants/collections-and-rendering). - -## Adapter matrix - -See the [native-array rendering matrix](/docs/concepts/constants/collections-and-rendering#native-array-rendering). - -## Rendering configuration - -See [Project default](/docs/concepts/constants/collections-and-rendering#project-default). - -## Model-local declarations - -See [Model-Private Values](/docs/concepts/model-private-values). - -## Model-private declarations - -See [Model-Private Values](/docs/concepts/model-private-values). - -## Enum validation - -See [Enum validation rules](/docs/concepts/enums#validation-rules). - -## Enum-typed contracts - -See [Enum Model Contracts](/docs/concepts/enums/model-contracts). diff --git a/website/src/content/docs/docs/concepts/incremental.mdx b/website/src/content/docs/docs/concepts/incremental.mdx index 83ea632849..50acb867cc 100644 --- a/website/src/content/docs/docs/concepts/incremental.mdx +++ b/website/src/content/docs/docs/concepts/incremental.mdx @@ -3,11 +3,24 @@ title: "Incremental Models" description: "Cursor-based incremental strategies, microbatch execution, and backfill policies." --- -Incremental models process only new or changed data instead of rebuilding the entire table. In the -default sequential path, SQLBuild works out where to resume from the current target and input -relations with no separate checkpoint store. A retry recomputes its interval from current warehouse -state rather than reusing the exact interval of a failed attempt. Opt-in concurrent microbatching -adds immutable coordination facts as described below. +An incremental model processes only new or changed rows on each build instead of rebuilding the +whole table. Pick a strategy and a cursor column, and SQLBuild works out where to resume from the +table itself, with no separate checkpoint store: + +```sql +MODEL ( + description "Order activity per hour", + materialized incremental, + incremental_strategy delete_insert, + cursor activity_hour, + cursor_type timestamp, + cursor_grain hour, + unique_key [activity_hour], +); +``` + +A retry works out its range again from the warehouse. [Concurrent microbatches](#concurrent-batches) are the exception: +they record each batch to coordinate. ## Strategies diff --git a/website/src/content/docs/docs/concepts/interpolation.mdx b/website/src/content/docs/docs/concepts/interpolation.mdx index 81b518280a..ab5bdde0ce 100644 --- a/website/src/content/docs/docs/concepts/interpolation.mdx +++ b/website/src/content/docs/docs/concepts/interpolation.mdx @@ -1,5 +1,5 @@ --- -title: "Interpolation" +title: "Variables and placeholders" description: "How SQLBuild processes variables, context, and dynamic content in SQL and config." --- diff --git a/website/src/content/docs/docs/concepts/macros/sql-macro-reference.mdx b/website/src/content/docs/docs/concepts/macros/sql-macro-reference.mdx new file mode 100644 index 0000000000..34576b7fe2 --- /dev/null +++ b/website/src/content/docs/docs/concepts/macros/sql-macro-reference.mdx @@ -0,0 +1,208 @@ +--- +title: "SQL macro reference" +description: "Literal positions, relation slot reads and mappings, call checks and result types for SQL macros." +--- + +The rules behind [SQL macros](/docs/concepts/macros/sql-macros): which positions need a literal, how +relation slots are read and mapped, what is checked at each call, and the result type of each form. + +## Positions that need a literal + +SQL needs a literal in some positions. A parameter without `CONST` there fails with `P023` when the +macro is checked: + +``` +error[P023]: places is used where a literal is required; declare it places INT CONST +``` + +| Position | Adapters | +|---|---| +| `LIMIT`, `OFFSET`, `TOP` and `FETCH FIRST` counts | all | +| `INTERVAL` and `EXTRACT` units | all | +| `PIVOT ... IN (...)` values | all | +| `DATE_TRUNC` unit (the first argument, or the last on BigQuery) | all | +| `ROUND` precision | DuckDB and MotherDuck (for `DECIMAL` values), Databricks | + +The positions every adapter shares are checked for every macro file. `DATE_TRUNC` and `ROUND` are +checked for the project's adapter at each call. + +## Reading slots + +The examples use the `weekly_rates` macro from +[Relation slots](/docs/concepts/macros/sql-macros#relation-slots). + +- `alias.@weekly.order_count` reads the slot from a table alias or CTE, such as `s` in that macro. For a + group, the prefix applies to every column: `s.@weekly.keys` becomes `s.product_id, s.region`. +- A plain read of a group, `@weekly.keys`, always stands as the bare column names, + `product_id, region`. So later CTEs, such as `rated`, read the group from whatever + relation they select from, without an alias. A bare name that two joined relations share fails + as hand-written SQL does, with `B003`; read it as `alias.@weekly.keys` there. +- A plain read of a single-column slot always reads through the passed relation, mapped or not: + `@weekly.refund_count` becomes `weekly.refund_count`, or `weekly.refunds_n` with + `weekly.refund_count := refunds_n`. So the same body expands the same way for every mapping. +- Every read inside `USING (...)` stands unqualified, as `USING` needs, including + `USING (@weekly.week)` for a column the type does not declare. +- In a column-list macro, `o.@weekly.keys` outputs the group's columns under their own names. +- Rules judge a read as you wrote it. `SQLCLEAN028`, which rejects a query mixing qualified and + unqualified columns, judges a query the macro writes whole, such as a CTE of a CTE-list macro, + once, in the macro file. There every plain read counts as unqualified and every + `alias.@weekly...` read as qualified. A step written only in plain reads, such as + `SELECT @weekly.keys, @weekly.refund_count AS refunds FROM @weekly`, passes; to qualify, read + every slot through an alias. +- Where a call's output joins the caller's query, as with expression and column-list macros, the + query is judged at the call. A read through the relation the call passes, such as `o.created_at` in + the `audit_columns` example in [SQL macros](/docs/concepts/macros/sql-macros), fits either style; any other column counts as the macro writes it, so a bare `status` + beside a hand-qualified `o.order_id` fails. + +A plain group read inside a subquery, such as `EXISTS (SELECT @weekly.keys FROM refunds AS r ...)`, +could bind its bare columns to the subquery's relations instead of the one the slot reads. The +call fails with `P017` when a relation of an enclosing subquery, other than the relation passed +for the slot, has one of the group's columns, or when its columns are not known. `LATERAL` +subqueries in `FROM` or `JOIN` are checked the same way: + +``` +error[P017]: slot weekly.keys column product_id would bind to refunds in this subquery; write alias.@weekly.keys +``` + +Read the group through its alias there. A group read in a CTE, or in a derived table in `FROM`, +reads that query's own relations and is not checked. + +`alias.@weekly.name` naming no slot, or `alias.@weekly` naming none at all, fails with `P022`. +`@weekly.name` without an alias may still read a column the type does not declare, through the +passed relation, as before. + +## Mapping slots at the call + +A call maps slots with `relation.slot := ...`, next to its other named arguments: + +```sql +WITH weekly AS (SELECT * FROM __ref("weekly_orders")), +@sql.weekly_rates(weekly, weekly.keys := (product_id), weekly.order_count := orders_placed, __cte := by_product) +@sql.weekly_rates(weekly, weekly.keys := (product_id, region), weekly.order_count := orders_placed, __cte := by_product_region) +final AS (SELECT * FROM by_product_region) +SELECT * FROM final +``` + +The first call compiles to the same SQL as a macro written for `product_id`, the second groups by +product and region. The left of `:=` is always a parameter path, and the right side is column +names of the passed relation: + +- A single-column slot takes one column name. An unmapped single-column slot reads the column of + its own name, so `refund_count` above reads `refund_count`. +- A group takes a parenthesised list of column names, such as `(product_id, region)`, and every + call must map it. +- A slot reads a column as it is: compute an expression in the relation you pass. +- A macro that passes its own relation on maps the inner slots to its slot names, as in + `@sql.weekly_rates(weekly := @src, weekly.keys := (dims), weekly.order_count := placed)`, or to + columns of its relation. Each inner slot keeps its kind: a group mapped to columns is still read + bare, or after its alias, and mapping a group to a single-column slot of the outer relation, or + the reverse, fails with `P017`. + +## Rules for groups + +1. A group is pasted verbatim, comma-separated. It takes no per-item expression, indexing, + counting or renaming, and groups are never combined. +2. A group stands only as a whole item of a select list, `PARTITION BY`, `GROUP BY`, `ORDER BY` + (without `ASC`, `DESC` or `NULLS`) or `USING (...)`. Anywhere else fails with `P015` at the + macro file. +3. Grouped columns keep their real names in the output. +4. A group is never empty: `weekly.keys := ()` fails. +5. A relation may have several groups, such as `keys COLUMNS, carried COLUMNS`. Groups of one + relation may not share a column, and a column that fills a single-column slot, mapped or by its + own name, may not also be in a group. + +## Slot errors + +Slot errors fail with `P017` and name the slot and the caller's relation: + +``` +error[P017]: slot weekly.order_count needs a BIGINT column; weekly has no column orders_total +``` + +- Every mapped and grouped column must exist in the passed relation, and a single-column slot needs + a compatible type: `slot weekly.order_count needs a BIGINT column; weekly.region is VARCHAR`, + or `slot weekly.keys groups column channel; weekly has no column channel`. Like other argument + checks, these run when the relation is a CTE or table alias; the model's own column checks still + run on the expanded SQL. +- Overlapping groups, or a grouped column that also fills a single-column slot, fail at the call. +- A grouped column named like a column the macro defines, or like a bare column such as `s.week` + or a column read through the relation such as `@weekly.region` beside the group in the same + select list, fails with `P018`, naming both: + `column refunds of slot weekly.keys clashes with column refunds that macro weekly_rates() defines`. +- A mapped single-column slot read in a select list needs an `AS` in the macro, so output names + don't depend on the caller's mapping. Groups are exempt, because they keep their real names, and + so is an unmapped slot, which keeps its own name. So are reads in `IN`, `EXISTS` and scalar + subqueries, whose column names never reach the macro's output. A `LATERAL` subquery in `FROM` or + `JOIN` is checked like a derived table, because its columns reach the output, as through `l.*`. + +## Checks on every call + +Each macro is checked on its own when the project compiles, and each call is checked against the +caller's column types, inferred like the rest of the model: + +- a `RELATION` argument must have every declared column, each of a compatible type, and every + column its [slots](#slot-errors) are mapped to; +- a typed argument must have a compatible type, so `@sql.cents_to_dollars(o.status)` with a + `VARCHAR` status fails. + +These fail with `P017`, naming the argument, the expected type or column and what the caller has: + +``` +error[P017]: argument src of macro clean_orders() needs column is_deleted BOOLEAN; orders_raw has no column is_deleted + --> models/marts/orders_clean.sql:2:1 + parameter declared here: + --> macros/sql/clean_orders.sql:3:14 + = help: pass a relation with these columns: order_id BIGINT, updated_at TIMESTAMP, is_deleted BOOLEAN +``` + +An argument whose type the compiler cannot infer passes. So do calls inside a subquery or in a +query part with `UNION`, `INTERSECT` or `EXCEPT`, arguments built from `__ref()`, macro calls or +parameters of a calling macro, and calls in models with runtime placeholders; the model's own +column checks still run on the expanded SQL. All other errors also point at the call and at the +macro file. + +## Result types + +An expression macro call is typed wherever it stands, from its expanded SQL, as if the SQL were +written there. `returns` declares the type for when that SQL's type is unknown: + +```sql +MACRO (description "Status bar", arguments (value VARCHAR), returns VARCHAR); + +bar(LENGTH(@value), 0, 10) +``` + +- The declared type applies only where the expanded SQL's type is unknown, including a list + literal such as `[]`, whose type analysis doesn't infer. A known type stands, even a close one such as `VARCHAR(100)` for + `returns VARCHAR(5)`, and contract checks report a mismatch with `K002`. +- The type flows on as an inferred type would: through CTEs, `SELECT *`, subqueries, joins and each + branch of a set operation, through operators and functions around the call, to contract checks + (`K002`, `K003`), `SQBRCONTRACT105` and models that read the column. +- `returns` is used only for analysis. It doesn't appear in the generated SQL, so on its own it doesn't + change the column's physical type. A body of `'ACTIVE'` still creates `VARCHAR(6)` on Snowflake. + For `SQBRCONTRACT105`, a select-list item that is one call needs no `CAST` at the call only when + the macro body's outermost expression is a `CAST(... AS type)` or `...::type` to the column's + contract type, compared as the dialect does, such as `CAST(LEFT(@value, 3) AS VARCHAR)` for a + `VARCHAR` column. Otherwise the rule asks for a cast in the macro or at the call, whether the + macro declares a fixed `returns` or `returns TYPE OF`. A call inside a larger expression always + needs an outer cast. +- Without `returns`, the call is typed by inference alone. When an output is one call whose type is + unknown, `K003` and `SQBRCONTRACT105` suggest declaring `returns` on the macro. +- `returns TYPE OF ` declares the type of that argument at each call, under the same + rule: `arguments (value ANY NUMERIC, places INT CONST DEFAULT 2), returns TYPE OF value`. If the + argument's own type is unknown, so is the call's. The argument must be a scalar argument of the + macro (`P015`). +- A fixed `returns` is checked against the body at the macro file. The body is typed from the + declared argument types; a known type of another family, such as `UPPER(@value)` with + `returns BIGINT`, fails with `P024`. A body whose type cannot be inferred, for example because it + reads `ANY` arguments, is a list literal, or calls a function whose type is unknown or only + guessed, trusts `returns`. +- `returns` and every argument type must be a type the adapter's SQL dialect accepts, including a + decimal precision and scale it can hold, such as at most `DECIMAL(38, s)` on DuckDB (`P015`). +- A call where only a keyword can stand, such as the date part of `EXTRACT`, is typed by inference + alone. +- A call inside a Python macro argument, as in `@py.bracket(@sql.status_bar(o.status))`, is typed + by inference alone. +- `returns` never changes the compiled SQL. + +A `* REPLACE` item takes its source column's type rather than the call's. diff --git a/website/src/content/docs/docs/concepts/macros/sql-macros.mdx b/website/src/content/docs/docs/concepts/macros/sql-macros.mdx index 80976753d4..e296f0263d 100644 --- a/website/src/content/docs/docs/concepts/macros/sql-macros.mdx +++ b/website/src/content/docs/docs/concepts/macros/sql-macros.mdx @@ -61,7 +61,7 @@ CAST(ROUND(@amount_cents / 100, @places) AS DECIMAL(18, 2)) ``` `returns` is optional for expression macros and not allowed for the other forms. See -[Result types](#result-types) for what it does. +[Result types](/docs/concepts/macros/sql-macro-reference#result-types) for what it does. The description follows the same rules as every header string: write text that contains double quotes as `"""..."""`, and a wrapped description is stored folded into one flowing text (see @@ -239,26 +239,6 @@ These fail with `P022`: Parameters may not be named `sql`, `py`, `const`, `enum`, `param` or `__cte`. -### Positions that need a literal - -SQL needs a literal in some positions. A parameter without `CONST` there fails with `P023` when the -macro is checked: - -``` -error[P023]: places is used where a literal is required; declare it places INT CONST -``` - -| Position | Adapters | -|---|---| -| `LIMIT`, `OFFSET`, `TOP` and `FETCH FIRST` counts | all | -| `INTERVAL` and `EXTRACT` units | all | -| `PIVOT ... IN (...)` values | all | -| `DATE_TRUNC` unit (the first argument, or the last on BigQuery) | all | -| `ROUND` precision | DuckDB and MotherDuck (for `DECIMAL` values), Databricks | - -The positions every adapter shares are checked for every macro file. `DATE_TRUNC` and `ROUND` are -checked for the project's adapter at each call. - ## Relation slots Everything a macro reads from a relation is declared in its `RELATION` type as a slot. A slot is @@ -292,112 +272,6 @@ rated AS ( A `COLUMNS` slot is untyped: each column passed to it keeps its own type, so the expanded SQL is fully typed. -### Reading slots - -- `alias.@weekly.order_count` reads the slot from a table alias or CTE, such as `s` above. For a - group, the prefix applies to every column: `s.@weekly.keys` becomes `s.product_id, s.region`. -- A plain read of a group, `@weekly.keys`, always stands as the bare column names, - `product_id, region`. So later CTEs, such as `rated` above, read the group from whatever - relation they select from, without an alias. A bare name that two joined relations share fails - as hand-written SQL does, with `B003`; read it as `alias.@weekly.keys` there. -- A plain read of a single-column slot always reads through the passed relation, mapped or not: - `@weekly.refund_count` becomes `weekly.refund_count`, or `weekly.refunds_n` with - `weekly.refund_count := refunds_n`. So the same body expands the same way for every mapping. -- Every read inside `USING (...)` stands unqualified, as `USING` needs, including - `USING (@weekly.week)` for a column the type does not declare. -- In a column-list macro, `o.@weekly.keys` outputs the group's columns under their own names. -- Rules judge a read as you wrote it. `SQLCLEAN028`, which rejects a query mixing qualified and - unqualified columns, judges a query the macro writes whole, such as a CTE of a CTE-list macro, - once, in the macro file. There every plain read counts as unqualified and every - `alias.@weekly...` read as qualified. A step written only in plain reads, such as - `SELECT @weekly.keys, @weekly.refund_count AS refunds FROM @weekly`, passes; to qualify, read - every slot through an alias. -- Where a call's output joins the caller's query, as with expression and column-list macros, the - query is judged at the call. A read through the relation the call passes, such as `o.created_at` - above, fits either style; any other column counts as the macro writes it, so a bare `status` - beside a hand-qualified `o.order_id` fails. - -A plain group read inside a subquery, such as `EXISTS (SELECT @weekly.keys FROM refunds AS r ...)`, -could bind its bare columns to the subquery's relations instead of the one the slot reads. The -call fails with `P017` when a relation of an enclosing subquery, other than the relation passed -for the slot, has one of the group's columns, or when its columns are not known. `LATERAL` -subqueries in `FROM` or `JOIN` are checked the same way: - -``` -error[P017]: slot weekly.keys column product_id would bind to refunds in this subquery; write alias.@weekly.keys -``` - -Read the group through its alias there. A group read in a CTE, or in a derived table in `FROM`, -reads that query's own relations and is not checked. - -`alias.@weekly.name` naming no slot, or `alias.@weekly` naming none at all, fails with `P022`. -`@weekly.name` without an alias may still read a column the type does not declare, through the -passed relation, as before. - -### Mapping slots at the call - -A call maps slots with `relation.slot := ...`, next to its other named arguments: - -```sql -WITH weekly AS (SELECT * FROM __ref("weekly_orders")), -@sql.weekly_rates(weekly, weekly.keys := (product_id), weekly.order_count := orders_placed, __cte := by_product) -@sql.weekly_rates(weekly, weekly.keys := (product_id, region), weekly.order_count := orders_placed, __cte := by_product_region) -final AS (SELECT * FROM by_product_region) -SELECT * FROM final -``` - -The first call compiles to the same SQL as a macro written for `product_id`, the second groups by -product and region. The left of `:=` is always a parameter path, and the right side is column -names of the passed relation: - -- A single-column slot takes one column name. An unmapped single-column slot reads the column of - its own name, so `refund_count` above reads `refund_count`. -- A group takes a parenthesised list of column names, such as `(product_id, region)`, and every - call must map it. -- A slot reads a column as it is: compute an expression in the relation you pass. -- A macro that passes its own relation on maps the inner slots to its slot names, as in - `@sql.weekly_rates(weekly := @src, weekly.keys := (dims), weekly.order_count := placed)`, or to - columns of its relation. Each inner slot keeps its kind: a group mapped to columns is still read - bare, or after its alias, and mapping a group to a single-column slot of the outer relation, or - the reverse, fails with `P017`. - -### Rules for groups - -1. A group is pasted verbatim, comma-separated. It takes no per-item expression, indexing, - counting or renaming, and groups are never combined. -2. A group stands only as a whole item of a select list, `PARTITION BY`, `GROUP BY`, `ORDER BY` - (without `ASC`, `DESC` or `NULLS`) or `USING (...)`. Anywhere else fails with `P015` at the - macro file. -3. Grouped columns keep their real names in the output. -4. A group is never empty: `weekly.keys := ()` fails. -5. A relation may have several groups, such as `keys COLUMNS, carried COLUMNS`. Groups of one - relation may not share a column, and a column that fills a single-column slot, mapped or by its - own name, may not also be in a group. - -### Checks at the call - -Slot errors fail with `P017` and name the slot and the caller's relation: - -``` -error[P017]: slot weekly.order_count needs a BIGINT column; weekly has no column orders_total -``` - -- Every mapped and grouped column must exist in the passed relation, and a single-column slot needs - a compatible type: `slot weekly.order_count needs a BIGINT column; weekly.region is VARCHAR`, - or `slot weekly.keys groups column channel; weekly has no column channel`. Like other argument - checks, these run when the relation is a CTE or table alias; the model's own column checks still - run on the expanded SQL. -- Overlapping groups, or a grouped column that also fills a single-column slot, fail at the call. -- A grouped column named like a column the macro defines, or like a bare column such as `s.week` - or a column read through the relation such as `@weekly.region` beside the group in the same - select list, fails with `P018`, naming both: - `column refunds of slot weekly.keys clashes with column refunds that macro weekly_rates() defines`. -- A mapped single-column slot read in a select list needs an `AS` in the macro, so output names - don't depend on the caller's mapping. Groups are exempt, because they keep their real names, and - so is an unmapped slot, which keeps its own name. So are reads in `IN`, `EXISTS` and scalar - subqueries, whose column names never reach the macro's output. A `LATERAL` subquery in `FROM` or - `JOIN` is checked like a derived table, because its columns reach the output, as through `l.*`. - ## Compose macros A SQL macro may call other SQL macros, and pass its own parameters on as `@name`: @@ -425,78 +299,6 @@ isn't visible from there fails with `S026`; to use a caller's value, make it an `@const(...)` at the call site. SQL macros a body calls resolve the same way, so a macro can call a folder-local macro beside it that its callers cannot see. -## Checks - -Each macro is checked on its own when the project compiles, and each call is checked against the -caller's column types, inferred like the rest of the model: - -- a `RELATION` argument must have every declared column, each of a compatible type, and every - column its [slots](#checks-at-the-call) are mapped to; -- a typed argument must have a compatible type, so `@sql.cents_to_dollars(o.status)` with a - `VARCHAR` status fails. - -These fail with `P017`, naming the argument, the expected type or column and what the caller has: - -``` -error[P017]: argument src of macro clean_orders() needs column is_deleted BOOLEAN; orders_raw has no column is_deleted - --> models/marts/orders_clean.sql:2:1 - parameter declared here: - --> macros/sql/clean_orders.sql:3:14 - = help: pass a relation with these columns: order_id BIGINT, updated_at TIMESTAMP, is_deleted BOOLEAN -``` - -An argument whose type the compiler cannot infer passes. So do calls inside a subquery or in a -query part with `UNION`, `INTERSECT` or `EXCEPT`, arguments built from `__ref()`, macro calls or -parameters of a calling macro, and calls in models with runtime placeholders; the model's own -column checks still run on the expanded SQL. All other errors also point at the call and at the -macro file. - -## Result types - -An expression macro call is typed wherever it stands, from its expanded SQL, as if the SQL were -written there. `returns` declares the type for when that SQL's type is unknown: - -```sql -MACRO (description "Status bar", arguments (value VARCHAR), returns VARCHAR); - -bar(LENGTH(@value), 0, 10) -``` - -- The declared type applies only where the expanded SQL's type is unknown, including a list - literal such as `[]`, whose type analysis doesn't infer. A known type stands, even a close one such as `VARCHAR(100)` for - `returns VARCHAR(5)`, and contract checks report a mismatch with `K002`. -- The type flows on as an inferred type would: through CTEs, `SELECT *`, subqueries, joins and each - branch of a set operation, through operators and functions around the call, to contract checks - (`K002`, `K003`), `SQBRCONTRACT105` and models that read the column. -- `returns` is used only for analysis. It doesn't appear in the generated SQL, so on its own it doesn't - change the column's physical type. A body of `'ACTIVE'` still creates `VARCHAR(6)` on Snowflake. - For `SQBRCONTRACT105`, a select-list item that is one call needs no `CAST` at the call only when - the macro body's outermost expression is a `CAST(... AS type)` or `...::type` to the column's - contract type, compared as the dialect does, such as `CAST(LEFT(@value, 3) AS VARCHAR)` for a - `VARCHAR` column. Otherwise the rule asks for a cast in the macro or at the call, whether the - macro declares a fixed `returns` or `returns TYPE OF`. A call inside a larger expression always - needs an outer cast. -- Without `returns`, the call is typed by inference alone. When an output is one call whose type is - unknown, `K003` and `SQBRCONTRACT105` suggest declaring `returns` on the macro. -- `returns TYPE OF ` declares the type of that argument at each call, under the same - rule: `arguments (value ANY NUMERIC, places INT CONST DEFAULT 2), returns TYPE OF value`. If the - argument's own type is unknown, so is the call's. The argument must be a scalar argument of the - macro (`P015`). -- A fixed `returns` is checked against the body at the macro file. The body is typed from the - declared argument types; a known type of another family, such as `UPPER(@value)` with - `returns BIGINT`, fails with `P024`. A body whose type cannot be inferred, for example because it - reads `ANY` arguments, is a list literal, or calls a function whose type is unknown or only - guessed, trusts `returns`. -- `returns` and every argument type must be a type the adapter's SQL dialect accepts, including a - decimal precision and scale it can hold, such as at most `DECIMAL(38, s)` on DuckDB (`P015`). -- A call where only a keyword can stand, such as the date part of `EXTRACT`, is typed by inference - alone. -- A call inside a Python macro argument, as in `@py.bracket(@sql.status_bar(o.status))`, is typed - by inference alone. -- `returns` never changes the compiled SQL. - -A `* REPLACE` item takes its source column's type rather than the call's. - ## Tools - `sqb format` formats SQL macro files, bodies of every form and the `MACRO` header, and @@ -508,8 +310,13 @@ A `* REPLACE` item takes its source column's type rather than the call's. [When Rules fail](/docs/cli/compile#when-rules-fail). - Editing a macro recompiles its callers. -Test SQL macros with [macro-mode unit tests](/docs/concepts/testing#macro-tests), and replace one +Test SQL macros with [macro-mode unit tests](/docs/concepts/testing-reference#macro-tests), and replace one in a model test with a [`__macro__` mock](/docs/concepts/testing#macro-mocking). See [Macros](/docs/concepts/macros#choose-the-right-tool) for when to use a SQL macro, a Python macro, a SQL function or a generic audit. + +## Reference + +[SQL macro reference](/docs/concepts/macros/sql-macro-reference) covers positions that need a +literal, reading and mapping relation slots, the checks at each call and result types. diff --git a/website/src/content/docs/docs/concepts/models.mdx b/website/src/content/docs/docs/concepts/models.mdx index 0137c42209..c882fc6f8d 100644 --- a/website/src/content/docs/docs/concepts/models.mdx +++ b/website/src/content/docs/docs/concepts/models.mdx @@ -59,7 +59,7 @@ See [Functions](/docs/concepts/functions) for scalar UDF and table-function refe ## Model guide - [Materializations](/docs/concepts/models/materializations): views, tables, incrementals, snapshots, and custom materializations. -- [Schemas](/docs/concepts/models/schemas): inline columns, reusable schemas, inheritance, audits, and model-local extensions. +- [Reusable column schemas](/docs/concepts/models/schemas): inline columns, reusable schemas, inheritance, audits, and model-local extensions. - [Type Enforcement](/docs/concepts/models/type-enforcement): static type checks and runtime cast behavior by materialization. - [Contracts](/docs/concepts/models/contracts): exact and open output validation, runtime guarantees, nullability, and enum-backed columns. - [Hooks](/docs/concepts/models/hooks): SQL and Python lifecycle hooks. diff --git a/website/src/content/docs/docs/concepts/models/column-migrations.mdx b/website/src/content/docs/docs/concepts/models/column-migrations.mdx index df5dd93edf..b905ff1341 100644 --- a/website/src/content/docs/docs/concepts/models/column-migrations.mdx +++ b/website/src/content/docs/docs/concepts/models/column-migrations.mdx @@ -3,15 +3,34 @@ title: "Column migrations" description: "Keep a column's history when you rename it in an incremental or snapshot model." --- -`on_schema_change` compares columns by name, so renaming a column looks like one column removed and another added. With the default `append_new_columns`, the new column starts empty and older rows keep their values under the old name; with `sync_all_columns`, the old column and its data are dropped. A column migration renames the column in the warehouse instead, so every row keeps its value under the new name, with no leftover column and no replay. +Rename a column with `sqb rename fact_orders.amount revenue`. On the next build, SQLBuild renames +the column in the warehouse table, so every row keeps its value under the new name and nothing is +replayed. -Column migrations apply to incremental models, including microbatch models, and snapshot models. Tables and views are rebuilt on every change and don't need them. +```bash +sqb rename fact_orders.amount revenue +sqb plan +``` -## Declaring a rename +```text +Column migrations (1) +└── fact_orders migrate columns + └── amount -> revenue rename in place +``` -[`sqb rename .`](/docs/cli/rename) renames the column in the model and every query that reads it, and adds the `migrate_from` declaration below for you. +Column migrations apply to incremental models, including microbatch models, and snapshot models. +Tables and views are rebuilt on every build, so they don't need them. -Add `migrate_from` to the renamed column in the model header, naming the old column: +Without a column migration, `on_schema_change` compares columns by name and sees one column removed +and another added. With the default `append_new_columns`, the new column starts empty and older rows +keep their values under the old name. With `sync_all_columns`, the old column and its data are +dropped. + +## Declaring a rename + +[`sqb rename .`](/docs/cli/rename) renames the column in the model and in every query +that reads it, and writes a `migrate_from` declaration on the renamed column. You can also write the +declaration yourself: ```sql MODEL ( @@ -29,30 +48,45 @@ SELECT order_id, order_date, amount AS revenue FROM __source("raw_orders") ``` -On the next build, SQLBuild renames `amount` to `revenue` in the existing table, records the rename, and then builds the model incrementally, so the new delta lands in `revenue`. +On the next build, SQLBuild renames `amount` to `revenue` in the existing table, records the rename, +and then builds the model incrementally, so the new delta lands in `revenue`. -A column entry may contain only `migrate_from`, or combine it with `type`, `nullable`, `description`, and `audits`. When the model binds a reusable schema with `model_schema`, an inherited column may add `migrate_from` without redeclaring its metadata. +Once the rename is recorded, `migrate_from` has no further effect, and SQLBuild reports (`M108`) +that it can be removed. Keep it until every target that needs the rename has built. On a target +where the table doesn't exist yet, the declaration does nothing and the table is created with the +new name. -After the rename is recorded, `migrate_from` has no further effect, and SQLBuild reports (`M108`) that it can be removed. Keep it until every target that needs the rename has built. On a target where the table doesn't exist yet, the declaration does nothing: the table is created with the new column name. +A column entry can hold `migrate_from` on its own or together with `type`, `nullable`, +`description` and `audits`. A column inherited from a [reusable schema](/docs/concepts/models/schemas) +can add `migrate_from` without repeating its metadata. -A declaration is rejected when compiling if: +Compilation rejects a declaration when: -- the model is not incremental or snapshot, unless it is a table or view model that declares its own `migrate_from`; there the column's `migrate_from` only names the old column in the [compatibility view](/docs/concepts/models/migrations#old-names) at the model's old name; -- `migrate_from` names the column itself, or a qualified name instead of one column; -- two columns declare the same `migrate_from`; -- a declared column migrates from another column that itself declares `migrate_from`. Chains and swaps can't be renamed in place safely; rename through a name no other column uses, or rebuild with `--full-refresh`. +- `migrate_from` names the column itself, or a qualified name instead of one column. +- Two columns declare the same `migrate_from`. +- A column migrates from another column that declares its own `migrate_from`. Chains and swaps + can't be renamed in place safely. Rename through a name no other column uses, or rebuild with + `--full-refresh`. +- The model isn't incremental or snapshot, unless the model itself declares `migrate_from` (see + below). -## Automatic detection +On a table or view model that declares its own model-level `migrate_from`, a column's `migrate_from` +only names the old column in the [old-name view](/docs/concepts/models/migrations#old-names). -A pure rename is detected without a declaration: exactly one column that disappeared from the model has exactly the same expression as exactly one new column, for example `amount` becoming `amount AS revenue`, or `ROUND(amount, 2) AS revenue` becoming `ROUND(amount, 2) AS net_revenue`. Expressions are compared ignoring formatting, comments, and keyword and unquoted identifier case. Detection compares the query the table was last built from with the current query, so it applies to the build that makes the change. +## Automatic detection -A rename is only automatic when it is the whole change. Apart from the renamed output names, the query must be the same: the same CTEs, `FROM`, joins, filters, grouping, and other columns. References to the new name in `ORDER BY` or `QUALIFY` count as references to the old name only when SQLBuild can prove they bind to the output column: the columns of every relation the query reads must be known, from the inspected warehouse tables of its sources and upstream models or from CTEs and subqueries with named columns, the new name must not be an input column, and the old name must either not be an input column or be passed through unchanged. Otherwise the change is not treated as a rename, because engines differ on whether such a name refers to the output alias or the input column. The output columns may be reordered unless the query groups, orders, or uses `DISTINCT ON` by position. If anything else changed, an unchanged expression such as `amount` may now read different data, for example from a CTE that redefines `amount`, so nothing is renamed automatically and the change follows `on_schema_change`. Declare `migrate_from` to rename the column anyway. +SQLBuild detects a rename by itself when renames are the only change to the query: each removed +column has exactly one new column with the same expression. For example, `amount` becomes +`amount AS revenue`. If you changed anything else in the query, declare `migrate_from`, or use +`sqb rename`. -Detection never guesses: +Detection never guesses. If more than one removed or new column has the same expression, nothing is +renamed automatically. Two columns that swap names keep their names. Automatically detected renames +are marked `(automatic)` in the plan, and they never stop a build: when one can't be applied, the +columns follow `on_schema_change`. -- Columns are matched by expression, not by position. Two columns renamed at once each pair with the column whose expression they kept; two columns that swap names keep their names and are not renamed. -- If more than one removed column has the same expression, or more than one new column does, nothing is migrated automatically. -- A near match, where the expressions reference the same columns but differ or the rest of the query changed, for example `amount` becoming `ROUND(amount, 2) AS revenue`, is not a rename. It follows the model's `on_schema_change` policy, and `sqb plan` shows a hint next to the added column until the change has been built: +A near match, where the expression changed, is not a rename. It follows `on_schema_change`, and +`sqb plan` suggests the declaration until the change has been built: ```text ├── schema diff: @@ -61,59 +95,37 @@ Detection never guesses: - amount (removed) ``` -Add the suggested `migrate_from` to keep the history. A declared rename with a changed expression or other query changes is still renamed in place, but it is a query change, so `replay_on_change` decides how much history is rebuilt, exactly as it would without the rename. +A declared rename with other query changes still renames the column in place. The other changes are +a query change, so the model's `replay_on_change` decides how much history is rebuilt, as it would +without the rename. ## Decisions -`sqb plan` and `sqb build` list column migrations before the models they change: - -```text -Column migrations (1) -└── fact_orders migrate columns - └── amount -> revenue rename in place -``` - -Automatically detected renames are marked `(automatic)`. The decision is based on the table's current columns first, then on recorded renames: +`sqb plan` and `sqb build` list column migrations before the models they change. The decision is +based on the table's current columns first, then on recorded renames: | Decision | Meaning | |----------|---------| | `rename in place` | The old column exists and the new one doesn't. Rename it. | -| `already renamed, record` | The new column exists and the old one doesn't, but no rename is recorded, which is what an interrupted rename leaves behind. Record it without renaming again. | +| `already renamed, record` | The new column exists, the old one doesn't, and no rename is recorded. An interrupted rename leaves this. Record it without renaming again. | | `done` | The rename is recorded and the new column exists. Nothing to do. | -| `conflict` | Both columns exist. The build stops (`M110`); drop one of them, or remove `migrate_from` and let `on_schema_change` handle the columns. | -| `source missing` | Neither column exists and no rename is recorded. The plan warns (`M109`) and the columns follow `on_schema_change` as if `migrate_from` were absent. A target can require confirmation or stop the build instead with [`missing_migration_origin`](/docs/concepts/project-configuration#missing-migration-origins). | +| `conflict` | Both columns exist. The build stops (`M110`). Drop one of them, or remove `migrate_from` and let `on_schema_change` handle the columns. | +| `source missing` | Neither column exists and no rename is recorded. The plan warns (`M109`) and the columns follow `on_schema_change`. A target can require confirmation or stop instead with [`missing_migration_origin`](/docs/concepts/models/migrations#missing-migration-origins). | | `source still produced` | The query still selects the old column. The build stops (`M111`). | -| `unsupported` | The warehouse can't rename this table's columns in place (see [Adapter support](#adapter-support)). The build stops (`M112`). | - -Automatically detected renames never stop a build: when they can't be applied, the columns follow `on_schema_change` as before. - -## How a rename runs - -Renames run after [model migrations](/docs/concepts/models/migrations) and before any model builds, one model at a time: - -1. The table's current columns are read again. A column that is already renamed is not renamed twice. -2. The column is renamed with `ALTER TABLE ... RENAME COLUMN` or the adapter's equivalent. -3. The rename is recorded. - -On adapters with transactional DDL (PostgreSQL, SQL Server, DuckDB, MotherDuck), each model's renames and their records commit together. On other adapters, a build interrupted between the rename and the record is recovered by the `already renamed, record` decision on the next build. A build that fails after the rename, for example while applying the delta, resumes on the next build without renaming again. - -After the rename, the remaining differences follow `on_schema_change`: `append_new_columns` adds any other new columns, `sync_all_columns` also drops removed ones and changes types, and `fail` only fails when a difference other than the rename remains. - -A pure rename is not a query change. This applies to declared renames too, as long as the rename is the whole change by the rule above. It doesn't trigger `replay_on_change` for the model, and it doesn't cascade a replay to downstream models. Downstream models that you edit to use the new name have query changes of their own and follow their own `replay_on_change`. If the renamed column is the model's cursor, the next delta starts from the watermark stored under the old name. - -## State - -Each rename is recorded in an append-only `_sqlbuild_column_migrations` table in the model's schema, with the model, relation, old and new column, discovery (`manual` or `automatic`), and decision (`rename` or `record`). Janitor never archives it. +| `unsupported` | The warehouse can't rename this table's columns (see [Adapter support](#adapter-support)). The build stops (`M112`). | ## Adapter support | Adapter | Statement | |---------|-----------| | Snowflake, PostgreSQL, DuckDB, MotherDuck, BigQuery | `ALTER TABLE ... RENAME COLUMN ... TO ...` | -| Databricks | `ALTER TABLE ... RENAME COLUMN ... TO ...`, which requires Delta column mapping | +| Databricks | `ALTER TABLE ... RENAME COLUMN ... TO ...`, which needs Delta column mapping | | SQL Server | `EXEC sp_rename '.
.', '', 'COLUMN'` | -Databricks renames a Delta table column only when the table uses name-based column mapping (`delta.columnMapping.mode = 'name'`). SQLBuild checks this during planning and doesn't change the table protocol for you. A declared rename on a table without column mapping stops the build (`M112`) with the statement that enables it: +On Databricks, a Delta table column can be renamed only when the table uses name-based column +mapping. SQLBuild checks this during planning and doesn't change the table for you. A declared +rename on a table without column mapping stops the build (`M112`) with the statement that enables +it: ```sql ALTER TABLE
SET TBLPROPERTIES ( @@ -123,6 +135,33 @@ ALTER TABLE
SET TBLPROPERTIES ( ); ``` -Enabling column mapping upgrades the table's Delta protocol, which older readers may not support. An automatically detected rename on such a table is skipped with a warning (`M112`) and follows `on_schema_change`. - -On PostgreSQL, views that select the renamed column keep working and keep their own output column names. On SQL Server, `sp_rename` warns that renaming can break scripts and stored procedures that reference the old name. On PostgreSQL, the new name is created in lower case, as an unquoted identifier would be. +Enabling column mapping upgrades the table's Delta protocol, which older readers may not support. +An automatically detected rename on such a table is skipped with a warning (`M112`). + +## Details + +- Expressions are compared ignoring formatting, comments, and the case of keywords and unquoted + identifiers. Detection compares the query the table was last built from with the current query, + so it applies to the build that makes the change. +- "The only change" means the same CTEs, `FROM`, joins, filters, grouping and other columns. Output + columns may be reordered, unless the query groups, orders or uses `DISTINCT ON` by position. +- A new name used in `ORDER BY` or `QUALIFY` counts as the old name only when SQLBuild can prove it + refers to the output column: every input relation's columns are known, the new name isn't an + input column, and the old name either isn't one or passes through unchanged. +- A rename that is the whole change is not a query change, whether declared or detected. It doesn't + trigger `replay_on_change` and doesn't replay downstream models. Downstream models you edit to use + the new name follow their own `replay_on_change`. +- If the renamed column is the model's cursor, the next delta starts from the watermark stored under + the old name. +- Renames run after [model migrations](/docs/concepts/models/migrations) and before any model + builds, one model at a time. SQLBuild reads the table's columns again first, so a column is never + renamed twice. On PostgreSQL, SQL Server, DuckDB and MotherDuck, a model's renames and their + records commit together; elsewhere an interrupted run recovers through `already renamed, record`. +- After the rename, other differences follow `on_schema_change`: `append_new_columns` adds new + columns, `sync_all_columns` also drops removed ones and changes types, and `fail` fails only when + a difference other than the rename remains. +- Renames are recorded in a `_sqlbuild_column_migrations` table in the model's schema. Janitor + never archives it. +- On PostgreSQL, views that select the renamed column keep working with their own output names, and + the new name is created in lower case. On SQL Server, `sp_rename` warns that renaming can break + scripts and stored procedures that use the old name. diff --git a/website/src/content/docs/docs/concepts/models/configuration.mdx b/website/src/content/docs/docs/concepts/models/configuration.mdx index 47894ebf00..e4ca878c5b 100644 --- a/website/src/content/docs/docs/concepts/models/configuration.mdx +++ b/website/src/content/docs/docs/concepts/models/configuration.mdx @@ -92,22 +92,21 @@ MODEL ( ); ``` -A `"` inside a `"..."` string ends it, so it is an error with its line and column rather than the -start of an escape. A description has no `\"` escape and no `""` doubling: in a `"..."` -description a backslash is an ordinary character, so `"Files under C:\"` ends with a backslash and -`"Path C:\\temp"` stores both backslashes. In a `'...'` description, `\'` and `''` both stand for -one `'`, as in `'it\'s'` and `'it''s'`. `sqb format --fix` rewrites a description that escapes its -quotes as `\"` or doubles them as `""` into a `"""..."""` string holding real quotes: +Quotes inside strings: + +- A `"` inside a `"..."` string ends it, so it is an error. There is no `\"` escape and no `""` + doubling; use `"""..."""` instead. `sqb format --fix` rewrites such a description for you. +- In a `"..."` string, a backslash is an ordinary character: `"Path C:\\temp"` stores both + backslashes. +- In a `'...'` string, `\'` and `''` both stand for one `'`, as in `'it\'s'` and `'it''s'`. ```text error[D002]: MODEL(...) in 'models/customers.sql:2:34' contains invalid SQLBuild header syntax: a double quote ends this "..." string early; use """…""" for text containing double quotes = help: write the text between triple quotes, where quotes need no escaping: description """Matches "Miss"."""; sqb format --fix rewrites a description that escapes its quotes as \" or doubles them as "" ``` -A `"""..."""` string ends at the first three double quotes in a row. Up to two quotes just before -them belong to the text, so the text may end with a quote, but it cannot contain three in a row. -A `"""` inside the text and a `"""` that is never closed fail with the position of the offending -quote or of the opening `"""`. +- A `"""..."""` string ends at the first three double quotes in a row. The text may end with a + quote but can't contain three in a row. ### Folded descriptions @@ -313,7 +312,7 @@ whole attempt, starting when the model begins executing: its hooks, statements, every microbatch. Time spent waiting for upstream models or for a free worker does not count. At the deadline SQLBuild cancels the model's running statements through the adapter (see the -[adapter table](/docs/concepts/project-configuration#duration-limit)), runs no further statements +[adapter table](/docs/cli/build#duration-limit)), runs no further statements for the attempt, and reports the model as timed out with the elapsed time, the limit, where it was set, the cancellation outcome and the query IDs it saw. Statements that already completed are not undone, but the attempt stops before any later promotion or swap, so a staged table is not @@ -321,5 +320,5 @@ published. Models that depend on it are skipped as for any failure, and independ A model that finishes successfully just after its limit keeps its successful result. The per-model limit is separate from the target's whole-build -[`max_duration`](/docs/concepts/project-configuration#duration-limit). When both apply, each +[`max_duration`](/docs/cli/build#duration-limit). When both apply, each statement is bounded by whichever deadline comes first. diff --git a/website/src/content/docs/docs/concepts/models/contracts.mdx b/website/src/content/docs/docs/concepts/models/contracts.mdx index 7377be5034..b4b5b740de 100644 --- a/website/src/content/docs/docs/concepts/models/contracts.mdx +++ b/website/src/content/docs/docs/concepts/models/contracts.mdx @@ -1,24 +1,10 @@ --- title: "Contracts" -description: "Validate required or exact model output schemas." +description: "Say which columns a model promises to output, and when SQLBuild checks them." --- -A model's declared columns provide output metadata, type enforcement, and column audit attachment. The project `check_declared_columns` setting and model `contract` policy determine whether those declarations also define a statically validated output shape. - -| Model declaration | Behavior | -|-------------------|----------| -| Unspecified | Follow `settings.check_declared_columns`: `true` validates declared columns as an open shape; `false` treats them as metadata and audit attachment only. | -| `none` | Do not activate declared-shape or nullability contract validation for this model. Explicit type enforcement remains active. | -| `enforced` | Treat declared columns as the complete authoritative output shape and enable runtime exact-schema checks on supported materializations. | - -The project default is `check_declared_columns = true`. Projects that use column declarations primarily for metadata and audits, including migrations from dbt, can turn the check off: - -```toml -[settings] -check_declared_columns = false -``` - -With `check_declared_columns = false`, a model must declare `contract enforced` to activate shape validation, the same as every other model declaring `contract none`. A model-level `contract enforced` or `contract none` always overrides the project setting. +A contract is the list of columns a model promises to output. You write it in the model's +`columns (...)`, and SQLBuild checks the model's SQL against it. ```sql MODEL ( @@ -34,128 +20,87 @@ MODEL ( ); ``` -## Validation - -### Compile time - -When declared-shape validation is active and SQL analysis can infer the model's output: - -- A missing declared column fails. -- A proven type mismatch fails when type enforcement or an exact contract is active. -- A declared non-null column fails when its expression is proven nullable. -- Additional inferred columns fail only for `contract enforced`. - -If SQLBuild cannot infer the output shape, these static shape checks cannot run. An enforced contract additionally requires a non-empty column declaration and adds runtime requirements on supported materializations. - -Type enforcement remains independent. Typed columns continue to receive static type checks and supported runtime casts with `check_declared_columns = false` and `contract none`. - -Configuration fields that reference output columns, including `unique_key`, `cursor`, `updated_at`, and `check_columns`, are checked against an enforced declaration. +How strictly the list is checked depends on the model's `contract` setting, which can also come +from `[defaults]` or `[path_defaults]`: -### Runtime - -For supported materialization paths, `contract enforced` inspects the staged relation and rejects: - -- Missing declared columns. -- Additional undeclared columns. -- Warehouse types that differ from declared types. - -Runtime contract validation does not scan data for nulls and does not inspect warehouse nullability metadata. `nullable false` participates in static nullability analysis; add a `not_null` audit when null values must be checked at runtime. - -Runtime exact-schema validation currently runs for: - -- Staged full-table builds. -- Non-microbatch incremental deltas. -- Snapshot deltas. +| `contract` setting | Listed columns | Extra columns | Checked | +|--------------------|----------------|---------------|---------| +| not set (default) | must be in the output | allowed | at compile time | +| `enforced` | must be in the output, with their declared types | not allowed | at compile time, and at runtime where the [table below](#runtime) says so | +| `none` | documentation and audit targets only | allowed | declared types only | -Views and custom materializations rely on compile-time contract analysis. Microbatch incrementals currently apply type enforcement and audits but do not perform the framework runtime exact-schema validation step. +To make models without a `contract` setting treat their columns as documentation and audit targets +only, set `[settings] check_declared_columns = false`. -Staged table validation happens before promotion, so a failure leaves the existing destination untouched. Immediate table promotion is incompatible with `contract enforced` because SQLBuild cannot validate output before replacing the destination. +Do this when column lists may not match the SQL, such as lists carried over from dbt YAML during a +migration. Models that need a contract then opt in with `contract enforced`. -## Type enforcement +Declared types are checked and cast whatever the contract; see +[Type Enforcement](/docs/concepts/models/type-enforcement). -Declaring a model column type enables type enforcement automatically, independently of the contract policy. Type enforcement controls static type compatibility and runtime casts on supported table and incremental paths; contracts control output shape. See [Type Enforcement](/docs/concepts/models/type-enforcement) for the materialization matrix. +## What fails and when -## Reusable schemas - -Contracts apply to the effective declaration, not only the reusable base: +### Compile time -```sql -MODEL ( - model_schema order, - columns ( - ingestion_batch_id (type VARCHAR, nullable false), - ), - contract enforced, -); -``` +When SQLBuild can work out the model's output columns, these fail unless the model's columns are +documentation only: -With `contract enforced`, this requires exactly the resolved `order` columns plus `ingestion_batch_id`. With `contract none`, the resolved columns remain metadata, audit attachment points, and inputs to explicit type enforcement without activating shape validation. See [Schemas](/docs/concepts/models/schemas). +- A listed column is missing from the output (`K001`). +- A `nullable false` column can be null (`K004`). +- With `contract enforced` only: an extra column (`K005`), or a `unique_key`, `cursor`, `updated_at` + or `check_columns` that names an unlisted column. -## Enum columns +A column whose type doesn't match its declared type fails too (`K002`), whatever the contract. If +SQLBuild can't work out the output, these checks are skipped. `contract enforced` needs at least one +listed column (`K006`). -A column may use a declared enum as a portable logical domain type: +### Runtime -```sql -MODEL ( - description "Example model", - contract enforced, - columns ( - fulfillment_method (type fulfillment_method), - ), -); -``` +With `contract enforced`, SQLBuild also checks the built output's column names and types before it +replaces anything: -This does not require, create, or reference a warehouse-native enum type. SQLBuild resolves string-valued enums to `VARCHAR` and integer-valued enums to `INTEGER`. Under `contract enforced`, it also generates an `accepted_values` audit for the declared members. Audit severity and timing follow normal audit configuration and materialization behavior. With the default error severity, it gates staged-table promotion and pre-DML delta paths; views and custom materializations may already have changed their relation when the audit runs. +| Materialization | Runtime check | +|-----------------|---------------| +| Table (staged, the default) | Before the new table replaces the old one | +| Table or non-microbatch incremental with `table_promotion_mode = "immediate"` | Not allowed: a `K011` compile error. Use staged promotion | +| Incremental | On each delta, before it is written | +| Microbatch incremental | None; compile time only | +| Snapshot | On each delta | +| View, custom | None; compile time only | -Enum member references such as `@enum("fulfillment_method").DELIVERY` are a separate feature that render one validated SQL literal. See [Enum Model Contracts](/docs/concepts/enums/model-contracts) for the complete distinction and lowering behavior. +Runtime checks don't look at nulls. Add a `not_null` audit to check for nulls at runtime. -## Adopting an existing schema +## Starting from an existing warehouse -Ordinary `sqb compile` validates authored contracts offline. To compare declarations with existing -warehouse relations, use the explicitly online, read-only contract commands: +To compare your declarations with tables that already exist, or write declarations from them, use +[`sqb contract`](/docs/cli/contract). It reads the warehouse and never changes it: ```bash sqb contract diff --from prod --select tag:commerce -sqb contract generate --from prod --select tag:commerce -``` - -The target named by `--from` supplies the database and schema namespace to inspect. SQLBuild uses -the active project connection and never resolves a second set of origin credentials. - -### Preview and write - -`contract diff` returns exit code `1` when declarations and physical relations disagree. -`contract generate` prints proposed declaration changes without changing files: - -```bash -# Fill missing types and append physical columns missing from code sqb contract generate --from prod --select tag:commerce --write - -# Replace conflicting types/names and remove declarations absent physically -sqb contract generate --from prod --select tag:commerce --write --overwrite ``` -Contract commands never mutate warehouse relations, lifecycle state, or fingerprints. - -### Safety rules - -- Additive generation preserves existing types on conflict, repository-only columns, comments, - descriptions, audits, nullability, enums, tags, freshness, and loader configuration. -- `--overwrite` is an explicit repository replacement policy, not a warehouse write. -- Generation does not add `contract enforced`; activation remains an authored decision. -- A shared `SCHEMA()` is not changed from one model's evidence. Ownership conflicts are reported. -- Writes are atomic and SQLBuild recompiles the project, restoring prior contents if validation - fails. -- Sources are first-class selectable resources, for example `--select source:raw_orders`. - -Enforced upstream contracts are authoritative compile-time interfaces. When input evidence is -complete, missing or ambiguous columns fail in projections, joins, filters, grouping, windows, and -ordering. Partial or opaque schemas remain open; SQLBuild does not turn missing metadata into a -false error. Use `sqb lineage --include-uses --format json` to inspect direct -non-projection uses such as `join_on`, `where`, `group_by`, and `window_order_by`. - -## Related policies - -The project default checks declared columns as an open shape for models that omit `contract`. A repository can require an explicit `contract enforced` instead with `settings.check_declared_columns = false` or configure a custom [Rule](/docs/concepts/rules/custom-rules/overview) that requires enforced contracts for selected model families. - -Contracts also constrain schema-change behavior. For example, `snapshot_schema_change append_new_columns` is incompatible with `contract enforced` because an unannounced appended column would violate the exact declaration. +`generate` fills in missing types and columns but never adds `contract enforced`; that stays your +decision. See [`sqb contract`](/docs/cli/contract) for `--overwrite` and what generation keeps. + +## Models that read an enforced contract + +When a model reads a model or source with `contract enforced`, SQLBuild knows its exact columns. A +reference to a column it doesn't have, or to an ambiguous one, fails at compile time, wherever it +appears: the select list, a join, a filter, grouping, a window or `ORDER BY`. When an upstream's columns aren't fully known, +nothing is reported. To see where a model uses each upstream column, run +`sqb lineage --include-uses --format json`. + +## Details + +- **Reusable schemas.** A contract covers the model's full column list: the columns of its + [reusable schema](/docs/concepts/models/schemas) plus any it adds. With `contract enforced`, the + output must have exactly those. +- **Column order.** Contracts match columns by name; output order isn't checked. +- **Enum columns.** A column typed with an enum, such as `fulfillment_method (type fulfillment_method)`, + is a `VARCHAR` or `INTEGER`. With `contract enforced`, SQLBuild adds an `accepted_values` audit for + its members. See [Enum Model Contracts](/docs/concepts/enums/model-contracts). +- **Schema changes.** `snapshot_schema_change append_new_columns` can't be combined with + `contract enforced`, because an appended column would break the exact list. +- **Requiring contracts.** A custom [Rule](/docs/concepts/rules/custom-rules/overview) can require + `contract enforced` for chosen models. diff --git a/website/src/content/docs/docs/concepts/models/hooks/sql.mdx b/website/src/content/docs/docs/concepts/models/hooks/sql.mdx index bc00cbb979..213a0cc119 100644 --- a/website/src/content/docs/docs/concepts/models/hooks/sql.mdx +++ b/website/src/content/docs/docs/concepts/models/hooks/sql.mdx @@ -199,7 +199,7 @@ The model must still see a named hook under the usual SQLBuild strictly validates hook resource syntax, arguments, interpolation, macros, metadata, and non-empty payloads. It does not classify executable statement kinds or implement vendor SQL grammar. Each rendered hook payload is passed to the adapter in one `execute` call; whether a driver accepts multiple statements or client-side batch separators is adapter and warehouse behavior. Use separate hook entries when portable ordering between statements matters. -After expansion, Polyglot validates the complete payload using the active adapter's analysis dialect when the model's effective `sql_analysis` setting is enabled, which it is by default. The `--no-sql-analysis` flag disables this analysis for the invocation. Polyglot does not understand every administrative or procedural command supported by every warehouse. If it cannot parse valid vendor-specific hook SQL, disable SQL analysis for that model or invocation and let the adapter and warehouse provide the authoritative result. SQLBuild does not compensate with keyword allowlists or handwritten parser fallbacks. +After expansion, SQLBuild's SQL analysis validates the complete payload using the active adapter's analysis dialect when the model's effective `sql_analysis` setting is enabled, which it is by default. The `--no-sql-analysis` flag disables this analysis for the invocation. SQL analysis does not understand every administrative or procedural command supported by every warehouse. If it cannot parse valid vendor-specific hook SQL, disable SQL analysis for that model or invocation and let the adapter and warehouse provide the authoritative result. SQLBuild does not compensate with keyword allowlists or handwritten parser fallbacks. Common errors include: @@ -207,6 +207,6 @@ Common errors include: - Unsupported header keys, empty descriptions, and missing SQL bodies - Missing or unused arguments and unsupported argument values - Unknown named hooks, unquoted hook names, and bare hook strings -- SQL rejected by Polyglot while optional SQL validation is enabled +- SQL rejected by SQL analysis while optional SQL validation is enabled Definition errors point to the hook file. Invocation and argument errors also identify the consuming model entry, such as `post_hooks[1] sql("grant_access")`. Runtime output preserves the authored hook index and identifies named and inline SQL entries. diff --git a/website/src/content/docs/docs/concepts/models/materializations.mdx b/website/src/content/docs/docs/concepts/models/materializations.mdx index 57a9d10aa9..14e18f9791 100644 --- a/website/src/content/docs/docs/concepts/models/materializations.mdx +++ b/website/src/content/docs/docs/concepts/models/materializations.mdx @@ -16,11 +16,11 @@ SELECT id AS order_id, customer_id, status FROM __source("raw__orders") ``` -View audits run after the view has been replaced. A failing audit marks the build as failed but cannot preserve the previous view. Views use compile-time contract and type analysis; SQLBuild does not rewrite the view with runtime casts or run the staged runtime-contract step. +Views are checked at compile time only; SQLBuild doesn't add runtime casts or contract checks to a view. ## Table -By default, creates a staging table, applies supported type and contract enforcement, runs blocking audits, and only then promotes it to the destination. A pre-promotion failure leaves the previous destination unchanged. +Builds the new table in a staging table, checks its types and contract, runs blocking audits, and only then replaces the old table. ```sql MODEL (description "One row per order", materialized table); @@ -30,7 +30,7 @@ FROM __ref("stg_orders") GROUP BY customer_id ``` -Projects may opt into `settings.table_promotion_mode = "immediate"`. Immediate promotion replaces the destination before audits, so a failed audit does not restore the old table. Immediate promotion rejects models that require declared-type enforcement or `contract enforced`; use staged promotion for those guarantees. A table or incremental model with `contract enforced` under immediate promotion is a `K011` compile error, so `sqb compile`, `sqb plan`, `sqb build`, and `sqb scenario test` report it before any warehouse work. +Set `[settings] table_promotion_mode = "immediate"` to build straight into the destination and audit afterwards. Immediate promotion can't enforce declared types or `contract enforced`. A model with `contract enforced` is a `K011` compile error; a model with declared types fails when it builds. ## Incremental @@ -48,7 +48,7 @@ MODEL ( ); ``` -For non-microbatch models, the first run, `--full-refresh`, and a full replay-on-change rebuild use the full-table path rather than incremental DML. Microbatch models retain batched execution: a full rebuild drops the existing target, creates its replacement from the first batch, and applies later batches with incremental DML. A failed full-refresh microbatch does not preserve the previous target. +The first run, `--full-refresh` and a full `replay_on_change` rebuild build the whole table, as a table model does. A microbatch model instead drops the table and rebuilds it batch by batch. See [Incremental](/docs/concepts/incremental) for cursor semantics, replay, schema changes, and microbatch execution. @@ -98,3 +98,14 @@ WHERE ordered_at >= @@@partition_start The `config` block is passed to the Python function through `ctx.config`. Runtime-owned `@@@placeholder` values remain unresolved until materialization execution. The custom function owns staging, runtime type or contract enforcement, audit timing, promotion, and rollback. Framework final audits run after the function returns unless `MaterializationResult.audit_results` is populated. A custom materialization can call `ctx.run_audits` against a staging relation before applying changes and return those results through `MaterializationResult.audit_results`. + +## When a build fails + +| Materialization | What a failed build leaves | +|-----------------|----------------------------| +| View | The new view. Audits run after it is replaced. | +| Table (staged, the default) | The old table, untouched. Checks and audits run before it is replaced. | +| Table (`immediate`) | The new table. Audits run after it is replaced. | +| Incremental | The old table if a delta audit fails. Rows already written stay if a final audit or a later microbatch fails. | +| Microbatch full rebuild | A partly rebuilt table; the old one was dropped first. | +| Custom | Whatever the custom materialization does. | diff --git a/website/src/content/docs/docs/concepts/models/migrations.mdx b/website/src/content/docs/docs/concepts/models/migrations.mdx index 4f2d9af630..4d4a030e7b 100644 --- a/website/src/content/docs/docs/concepts/models/migrations.mdx +++ b/website/src/content/docs/docs/concepts/models/migrations.mdx @@ -3,15 +3,35 @@ title: "Model migrations" description: "Keep a renamed model's history, and keep its old name working for a while." --- -Renaming an incremental or snapshot model normally means rebuilding it from scratch under the new name, which loses any history the source data can no longer reproduce. A model migration moves the existing relation's data to the new name instead, then continues building incrementally. +Rename a model with `sqb rename stg_orders orders`. On the next build, SQLBuild moves the existing +table to the new name and carries on building it incrementally, so no history is lost and nothing is +backfilled. The old name keeps working as a view for 30 days. -Table and view models can declare a migration too. They hold no history, so they are rebuilt under the new name, but they keep their identity for change detection. For every materialization, the old name keeps working for a while through a [compatibility view](#old-names). To keep a column's history when you rename the column, see [Column migrations](/docs/concepts/models/column-migrations). +```bash +sqb rename hourly_order_activity order_activity_by_hour +sqb plan +``` -## Declaring a migration +```text +Migrations (1) +└── order_activity_by_hour migrate dev.hourly_order_activity -> dev.order_activity_by_hour + ├── compatibility compatible + ├── transfer physical copy, promote by transactional rename + └── old name dev.hourly_order_activity + ├── view until 2026-11-06 + └── old table archived +``` + +This matters most for incremental and snapshot models, whose history the source data may no longer +reproduce. Table and view models are rebuilt under the new name, but they keep their identity for +change detection, and their old name keeps working too. To rename a column instead, see +[Column migrations](/docs/concepts/models/column-migrations). -[`sqb rename`](/docs/cli/rename) and `sqb mv` rename or move a model, update every reference to it, and add `migrate_from` whenever the model's relation moves and its materialization keeps warehouse data (table, view, incremental, or snapshot). +## Declaring a migration -To declare a migration by hand, add `migrate_from` to the renamed model's header, naming the old model: +[`sqb rename`](/docs/cli/rename) and `sqb mv` rename or move a model, update every reference to it, +and write `migrate_from` into the model's header. You can also write it yourself, naming the old +model: ```sql MODEL ( @@ -23,13 +43,20 @@ MODEL ( ); ``` -Use `schema.name` when the old relation is in a different schema of the same database. +Use `schema.name` when the old relation is in another schema of the same database. -On the next build, SQLBuild copies the old relation's data to the new name through a staging table (see [How a move runs](#how-a-move-runs)), records the move, and builds the model incrementally from the migrated state. There is no first-run rebuild, and no `replay_on_change` replay caused by the rename. After the model builds, the old relation is archived and a compatibility view takes its name (see [Old names](#old-names)). +On the next build, SQLBuild copies the old relation to the new name, records the move, and builds +the model incrementally from there. There is no first-run rebuild, and the rename alone causes no +`replay_on_change` replay. After the model builds, the old relation is archived and a view takes its +name (see [Old names](#old-names)). -A rename is usually made together with other edits, for example a changed filter or a renamed upstream reference. If the renamed model's query also differs from the old model's, the plan still recognises the rename: the model is listed as `Renamed` with its query diff, never as a first run. The query change follows the model's own `replay_on_change`, exactly as it would without the rename (forward-only by default). Models downstream of the renamed model are not replayed because of the rename; see [Replay decisions](/docs/concepts/planning/replay-decisions). +A rename usually comes with other edits, such as a changed filter. The plan still recognises the +rename and lists the model as `Renamed` with its query diff. The query change follows the model's +own `replay_on_change`, and downstream models are not replayed because of the rename; see +[Replay decisions](/docs/concepts/planning/replay-decisions). -After the move is recorded, `migrate_from` has no further effect, and SQLBuild tells you it can be removed. Keep it until every target that needs the move has built. +Once the move is recorded, `migrate_from` has no further effect, and SQLBuild tells you it can be +removed. Keep it until every target that needs the move has built. ## Decisions @@ -39,78 +66,67 @@ After the move is recorded, `migrate_from` has no further effect, and SQLBuild t |----------|---------| | `migrate` | The new relation doesn't exist. Move the old one's data to it. | | `done` | The move is already recorded. Nothing to do. | -| `redo` | The new relation exists but has no build history and no recorded move, which is what an interrupted migration leaves behind. Move it again. | -| `superseded replace` | The new relation was itself the origin of the latest recorded move, for example after renaming a model back. Replace it with the newer data. | +| `redo` | The new relation exists with no build history and no recorded move. An interrupted migration leaves this. Move it again. | +| `superseded replace` | The new relation was the origin of the latest recorded move, for example after renaming a model back. Replace it with the newer data. | | `forced replace` | The new relation has its own build history and `migrate_force true` is set. Replace it. | | `conflict` | The new relation has its own build history. The build stops (`M103`) until you set `migrate_force true` or remove `migrate_from`. | -| `origin missing` | The old relation doesn't exist and no move was recorded, for example on a target that never built the old name. The plan warns (`M102`) and the model builds fresh. A target can require confirmation or stop the build instead with [`missing_migration_origin`](/docs/concepts/project-configuration#missing-migration-origins). If SQLBuild has build history for the old relation in this target, it should exist, so the build stops (`M102`) whatever the policy: the connection may lack permission to see it, or it was dropped outside SQLBuild. Restore access, or remove `migrate_from` to build fresh deliberately. | +| `origin missing` | The old relation doesn't exist and no move was recorded, for example on a fresh target. The plan warns (`M102`) and the model builds fresh. | -The origin and destination must be compatible under the model's normal `on_schema_change` rules; otherwise the build stops (`M104`). +On a target where the old name was never built, `origin missing` is normal. A target can ask for +confirmation or stop the build instead; see [Missing migration origins](#missing-migration-origins). -`migrate_force true` only affects the `conflict` case and is safe to leave in the header. +If SQLBuild built the old relation in this target but can't see it now, the build always stops with +`M102`: the connection may lack permission, or the relation was dropped outside SQLBuild. Restore +access, or remove `migrate_from` to build fresh on purpose. -Table and view models only ever see `migrate`, `done`, and `origin missing`: nothing is replaced, so there is no conflict. `migrate_force` on a table or view model is a compile error. In `sqb plan --json`, their decision is `renamed`. +The old and new relations must be compatible under the model's `on_schema_change` rules; otherwise +the build stops (`M104`). `migrate_force true` only affects the `conflict` case and is safe to leave +in the header. -## Automatic discovery +Table and view models only see `migrate`, `done` and `origin missing`, because nothing is replaced. +`migrate_force` on a table or view model is a compile error. In `sqb plan --json`, their decision is +`renamed`. -If a selected model has never been built, SQLBuild compares it with models that were removed from the project but whose relations still exist. When exactly one removed model has equivalent logic, it is migrated automatically. Equivalent means the same query and configuration, ignoring the model's own name, CTE and table alias names, comments, formatting, and storage-only settings. Renamed upstream models are matched first, so a renamed chain of models is migrated together. Removed models whose data was already moved on by a recorded migration are not candidates, so a model renamed several times matches its latest table. +### Missing migration origins -Renamed tables and views are matched the same way. They are rebuilt under the new name, since they hold no history, but they keep their identity for change detection. These renames are recorded as `renamed` events. +On a target that never built the old name, such as a fresh development schema, there is nothing to +move. The target's `missing_migration_origin` decides what the build does: -Automatic discovery never guesses. If a match is ambiguous, SQLBuild warns (`M107`) and builds from scratch; declare `migrate_from` to choose. An explicit `migrate_from` always wins. Automatic discovery covers renames within the project's schemas; use `migrate_from` to move a model to another schema. Relations last built before this feature carry no stored fingerprint and aren't matched automatically. +| Value | Behavior | +|-------|----------| +| `allow` | Default. The plan warns (`M102` for models, `M109` for columns) and the model builds as if `migrate_from` were absent. | +| `require_confirmation` | The build asks on a terminal and stops otherwise. Pass `--allow-missing-migration-origin` to confirm in a non-interactive run. | +| `deny` | The build stops with `M102` or `M109`. Use it where a missing origin means the wrong target or a lost table. | -## How a move runs +```toml +[targets.prod] +missing_migration_origin = "deny" +``` -A move never overwrites the destination directly: +`sqlbuild_local.toml` can set it for a target too. `sqb dbt run` and `sqb dbt build` follow it but +have no confirmation flag: they ask on a terminal and stop otherwise. In a non-interactive dbt run, +build the affected models with `sqb build --allow-missing-migration-origin` and leave them out of +the dbt selection, or set the target to `allow`. -1. The old relation is cloned or copied into a new staging table named `_sqb_archive____migration_stage__`, with the destination's configured table type. -2. The staging table is checked: it must exist, have the expected columns and table type, and, for a physical copy, the same row count as the old relation. -3. The staging table is swapped in as the destination. If a destination already existed, for example after `superseded replace` or `forced replace`, it is kept as `_sqb_archive____migration_previous__` rather than dropped. -4. The move is recorded, then the model builds as usual. -5. After the model builds successfully, the old relation is archived and a compatibility view is created at its name (see [Old names](#old-names)). +## Automatic discovery -Until step 5, the old relation is not modified. Staging and replaced tables use the [janitor](/docs/cli/janitor) archive format, so an abandoned staging table or a replaced destination is deleted by janitor after `archive_retention_days`. A staging table left by an interrupted run is never reused; the next run stages again. Don't run janitor at the same time as a build in the same schema. +A model you rename by hand, without `migrate_from`, is still found when its logic is unchanged. If a +selected model has never been built and exactly one removed model has the same query and +configuration, SQLBuild migrates it automatically. Renamed upstream models are matched first, so a +renamed chain of models moves together. -`sqb plan` shows how each move will run, for example `physical copy, transient -> permanent, promote by swap`. +Discovery never guesses. If more than one removed model matches, SQLBuild warns (`M107`) and builds +from scratch; declare `migrate_from` to choose. An explicit `migrate_from` always wins. Discovery +covers renames within the project's schemas; use `migrate_from` to move a model to another schema. ## Old names -A rename breaks every query outside the project that still reads the old name: dashboards, notebooks, other teams' jobs. After the renamed model builds successfully, SQLBuild keeps the old name working for a while: - -1. The old relation is renamed to `_sqb_archive____migration_origin__`. -2. A view is created at the old name that selects from the new relation. -3. The view gets the privileges that were granted on the old relation, read from the archive. Only existing grants are copied; nothing new is granted. - -On DuckDB, MotherDuck, PostgreSQL, and SQL Server these steps and their records commit in one transaction. On Snowflake, BigQuery, and Databricks each step is recorded after it runs, and a re-run resumes from the first unrecorded step without repeating one. A failure is reported as `M117`; re-run the build. - -| Adapter | Privileges copied to the view | -|---------|-------------------------------| -| Snowflake | `SHOW GRANTS` on the archive, replayed to roles and database roles, keeping `WITH GRANT OPTION`. `OWNERSHIP`, grants to shares, and privileges that don't apply to views are skipped | -| PostgreSQL | Table and column privileges from the catalog, including grants to `PUBLIC` and `WITH GRANT OPTION`. The owner's own privileges are not copied | -| BigQuery | Table IAM role bindings from `INFORMATION_SCHEMA.OBJECT_PRIVILEGES`, replayed with `GRANT`. See the note below | -| Databricks | Unity Catalog grants on the table itself that apply to views, such as `SELECT`. `OWN`, `MODIFY`, and inherited catalog or schema grants are skipped | -| SQL Server | Object and column permissions, including `DENY` and `WITH GRANT OPTION`, that apply to views | -| DuckDB, MotherDuck | No object privileges exist | - -Column privileges are copied to the view's column of the same name, which is the old name when the view aliases a renamed column. A column privilege on a column the view no longer exposes is not copied; the view shows nothing of that column. - -The copied grants are recorded on the view's creation. After that, the view's own privileges are what counts: when SQLBuild redefines the view after a rebuild, the view ends up with exactly the privileges it had just before, so a privilege you revoke from the view stays revoked and one you grant on the view is kept. SQLBuild uses `CREATE OR REPLACE VIEW` on PostgreSQL, `COPY GRANTS` on Snowflake, and `ALTER VIEW` on SQL Server and Databricks. - -Creating or replacing a view can also apply privileges you did not give it: PostgreSQL applies `ALTER DEFAULT PRIVILEGES` when it re-creates a view, and Snowflake applies the schema's future grants for views on `CREATE OR REPLACE VIEW … COPY GRANTS`. SQLBuild reads the view's privileges before every create or redefinition and reconciles afterwards: it grants what went missing and revokes what was added. A new compatibility view likewise ends up with only the privileges copied from the old relation. Ownership and the owner's own privileges are never changed. - -A recorded view that no longer exists at the old name is not re-created by a build; [janitor](/docs/cli/janitor) records it as dropped. +A rename breaks every query outside the project that reads the old name: dashboards, notebooks, +other teams' jobs. So after the renamed model builds, SQLBuild archives the old relation and creates +a view at the old name that reads the new one. The view gets the privileges that were granted on the +old relation; nothing new is granted. -On Snowflake, BigQuery, Databricks, DuckDB, and MotherDuck, views read their tables by name when queried, so a build redefines a compatibility view only when its SQL would change, for example when the view lists columns and the new relation's columns changed. PostgreSQL views are bound to the table itself, so they are redefined whenever the table is replaced. - -BigQuery cannot rename views, so an old view model is re-created under the archive name with its IAM bindings, and then dropped. - -On BigQuery, a view reads its tables with the permissions of whoever queries it. A principal that could read the old table keeps access to the old name through the copied bindings, but also needs read access on the new table. SQLBuild does not grant that and does not create authorized views; the build warns with `M118` and names the principals. - -If columns were renamed, the view presents them under their old names, for example `SELECT order_id, revenue AS amount FROM analytics.daily_revenue`. Incremental and snapshot models alias every recorded [column migration](/docs/concepts/models/column-migrations). A table or view model that declares `migrate_from` can declare old column names the same way, as `revenue (migrate_from amount)`; this only aliases the column in the compatibility view, since the table is rebuilt with the new name anyway. The view follows the new relation's current columns after every build of the model. - -`sqb plan` shows what happens at the old name: - -``` +```text Migrations (1) └── daily_revenue migrate analytics.revenue -> analytics.daily_revenue ├── transfer rebuild (table) @@ -121,61 +137,109 @@ Migrations (1) └── old table archived ``` -A move resumed after the archive shows `old table already archived`. Once the view exists it shows `view live until 2026-10-28` and how many grants were copied. When no view is kept, the block shows `left for janitor` and the reason, such as `old_name_view false` or `name reused by model:revenue`. +If columns were renamed, the view shows them under their old names, for example +`SELECT order_id, revenue AS amount FROM analytics.daily_revenue`. Incremental and snapshot models +alias every recorded [column migration](/docs/concepts/models/column-migrations). A table or view +model that declares `migrate_from` can name old columns the same way, as +`revenue (migrate_from amount)`; this only aliases the column in the view. -In `sqb plan --json`, each migration has an `old_name` object with `action` (`archive_and_view`, `view_only`, `live`, or `none`), `view`, `reads`, `expires_at`, `column_aliases`, `grants_copied` (the number of grants copied once the view exists, otherwise `null`), and `reason`. The top-level `old_names` list also includes steps that resume an earlier move. - -The view expires after 30 days by default. After that, [janitor](/docs/cli/janitor) drops it; `sqb janitor --drop-old-name-view analytics.revenue` drops it earlier. Change the default for the project, or turn the views off with `false`: +The view expires after 30 days. Then [janitor](/docs/cli/janitor) drops it; +`sqb janitor --drop-old-name-view analytics.revenue` drops it earlier. Change the default for the +project, or turn views off with `false`: ```toml [migrations] old_name_views = "7d" ``` -A model header overrides the project setting with `old_name_view 7d` or `old_name_view false`. The retention of a move is fixed when the move is recorded. - -With views off, or when a view can't be kept, the old relation is left in place and janitor archives it once it is no longer part of the project, as before. No view is created when: - -- the old relation was not built by SQLBuild; -- another model in the project builds at the old name; -- the old name is itself a compatibility view of an earlier rename. - -Moves recorded before a SQLBuild version that supports old-name views never gain a view later. - -### Code inside the project - -The compatibility view is for consumers outside the project. Project code must use the new model: - -- Python nodes, hooks, and loaders whose literal SQL names the old name fail with `P008`, at compile time for a declared `migrate_from` and at plan time for an automatically discovered rename. SQL models reading it through `__ref` are unaffected, since `__ref` resolves the new name. -- A new model whose name is a live compatibility view stops the build with `M114`, which names the view and its expiry. Drop the view early with janitor, or choose another name. +A model header overrides this with `old_name_view 7d` or `old_name_view false`. A move keeps the +retention it had when it was recorded. -### PostgreSQL views +No view is created when views are off, when the old relation wasn't built by SQLBuild, when another +model builds at the old name, or when the old name is itself a view from an earlier rename. The plan +then shows `left for janitor` and the reason, and janitor archives the old relation once it is no +longer part of the project. -PostgreSQL views are bound to the table they read, not its name. When a build replaces the renamed model's table, or drops or retypes its columns, SQLBuild points its compatibility views at the new table in the same transaction, so they are never missing, even if the build is interrupted. A view is replaced in place when its columns allow it, so views you build on top of the compatibility view keep working; otherwise it is dropped and re-created with the privileges it had, which fails if another view depends on it. +### Privileges by adapter -Views outside SQLBuild that read the old relation follow it into the archive and keep reading its old data; the build warns with `M115` and names them. Re-create them against the old name or the new relation. On the other adapters, views read relations by name, so no rebinding is needed. - -### Upgrading +| Adapter | Privileges copied to the view | +|---------|-------------------------------| +| Snowflake | `SHOW GRANTS` on the archive, replayed to roles and database roles, keeping `WITH GRANT OPTION`. `OWNERSHIP`, grants to shares, and privileges that don't apply to views are skipped | +| PostgreSQL | Table and column privileges, including grants to `PUBLIC` and `WITH GRANT OPTION`. The owner's own privileges are not copied | +| BigQuery | Table IAM role bindings from `INFORMATION_SCHEMA.OBJECT_PRIVILEGES`, replayed with `GRANT` | +| Databricks | Unity Catalog grants on the table that apply to views, such as `SELECT`. `OWN`, `MODIFY`, and inherited catalog or schema grants are skipped | +| SQL Server | Object and column permissions, including `DENY` and `WITH GRANT OPTION`, that apply to views | +| DuckDB, MotherDuck | No object privileges exist | -Upgrade the SQLBuild version that runs `sqb janitor` together with the one that runs builds. An older janitor does not know compatibility views and archives them as stale relations. +On BigQuery, a view reads its tables with the permissions of whoever queries it. A principal that +could read the old table also needs read access on the new one. SQLBuild doesn't grant it and +doesn't create authorized views; the build warns with `M118` and names the principals. -## State +### Code inside the project -Each move is recorded in an append-only `_sqlbuild_migrations` table in the destination schema. A missing record means completion is unrecorded; staging or promotion may already have happened. Re-running after an interruption recovers through the migration decisions above. +The view is for consumers outside the project. Project code must use the new name: -What happens at the old name is recorded in an append-only `_sqlbuild_old_name_views` table next to it, one row per step: `required` (with the move), `origin_archived`, `view_created`, and `view_dropped`. Rows are never updated. +- Python nodes, hooks and loaders whose literal SQL names the old name fail with `P008`. SQL models + that read it through `__ref` are unaffected, since `__ref` resolves the new name. +- A new model named like a live view stops the build with `M114`, which names the view and its + expiry. Drop the view early with janitor, or choose another name. ## Adapter support -| Adapter | Staging | Promotion | -|---------|---------|-----------| -| Snowflake | Zero-copy clone with `COPY GRANTS`. Transient → permanent is a physical copy, because Snowflake can't clone a transient table into a permanent one. | `ALTER TABLE … SWAP WITH` for an existing destination; rename for a missing destination | +| Adapter | Copy | Promotion | +|---------|------|-----------| +| Snowflake | Zero-copy clone with `COPY GRANTS`. Transient → permanent is a physical copy, because Snowflake can't clone a transient table into a permanent one. | `ALTER TABLE … SWAP WITH` for an existing destination; rename for a missing one | | BigQuery | `CREATE TABLE … CLONE`, or `CREATE TABLE … COPY` when BigQuery refuses the clone | Two renames; the destination is briefly missing between them, and an interrupted run recovers on retry | | Databricks | `DEEP CLONE` | Two renames, as for BigQuery | | PostgreSQL, SQL Server, DuckDB, MotherDuck | Physical copy | Renames and the migration record in one transaction | -On PostgreSQL, views that depend on the destination are re-pointed to the new table in the same transaction, keeping their options, owner and grants. Materialized views are not re-pointed. - ## Previewing another target -`sqb plan --as ` compiles the project for another target, for example production, and shows its migration decisions using your current connection. It only inspects; nothing is written. +`sqb plan --as ` compiles the project for another target, such as production, and shows its +migration decisions using your current connection. It only inspects; nothing is written. + +## Details + +- **How a move runs.** The old relation is cloned or copied into a staging table named + `_sqb_archive____migration_stage__`. The staging table is checked for its + columns, table type and, for a physical copy, row count. It is then swapped in as the destination. + A destination that already existed is kept as `_sqb_archive____migration_previous__`. + The move is recorded, the model builds, and only then is the old relation archived as + `_sqb_archive____migration_origin__`. +- Staging and replaced tables use the [janitor](/docs/cli/janitor) archive format, so janitor + deletes an abandoned one after `archive_retention_days`. A staging table left by an interrupted + run is never reused. Don't run janitor at the same time as a build in the same schema. +- Archiving the old relation, creating the view and copying grants commit in one transaction on + DuckDB, MotherDuck, PostgreSQL and SQL Server. On Snowflake, BigQuery and Databricks each step is + recorded after it runs, and a re-run resumes from the first unrecorded step. A failure is reported + as `M117`; re-run the build. +- Column privileges are copied to the view's column of the same name, which is the old name when the + view aliases a renamed column. A privilege on a column the view doesn't expose is not copied. +- After it is created, the view's own privileges are what counts. When SQLBuild redefines the view, + it reads its privileges first and restores them afterwards, so a privilege you revoke stays revoked + and one you grant is kept. This also undoes privileges that PostgreSQL `ALTER DEFAULT PRIVILEGES` + or Snowflake future grants would add. Ownership is never changed. +- On Snowflake, BigQuery, Databricks, DuckDB and MotherDuck, views read tables by name, so a build + redefines the view only when its SQL would change. PostgreSQL views are bound to the table, so + SQLBuild re-points them in the same transaction whenever the table is replaced, and other views + that depend on the destination are re-pointed too, keeping their options, owner and grants. + A view whose columns no longer fit is dropped and re-created with its privileges, which fails if + another view depends on it. Materialized views are not re-pointed. +- On PostgreSQL, views outside SQLBuild that read the old relation follow it into the archive and + keep reading its old data; the build warns with `M115` and names them. Re-create them against the + old name or the new relation. +- BigQuery can't rename views, so an old view model is re-created under the archive name with its + IAM bindings, then dropped. +- A recorded view that no longer exists is not re-created by a build; janitor records it as dropped. +- Run the same SQLBuild version for `sqb janitor` as for builds. An older janitor doesn't recognise + compatibility views and archives them as stale relations. +- In `sqb plan --json`, each migration has an `old_name` object with `action` (`archive_and_view`, + `view_only`, `live` or `none`), `view`, `reads`, `expires_at`, `column_aliases`, `grants_copied` + and `reason`. The top-level `old_names` list also includes steps that resume an earlier move. +- Discovery ignores the model's own name, CTE and table alias names, comments, formatting and + storage-only settings. A model already moved on by a recorded migration is not a candidate, so a + model renamed several times matches its latest table. Relations without a stored fingerprint are + not matched. Renamed tables and views are recorded as `renamed` events. +- Moves are recorded in an append-only `_sqlbuild_migrations` table in the destination schema, and + old-name steps in `_sqlbuild_old_name_views`. A missing record means the move may be partly done; + re-running recovers through the decisions above. diff --git a/website/src/content/docs/docs/concepts/models/schemas.mdx b/website/src/content/docs/docs/concepts/models/schemas.mdx index 311a58cb9f..0bb057ff15 100644 --- a/website/src/content/docs/docs/concepts/models/schemas.mdx +++ b/website/src/content/docs/docs/concepts/models/schemas.mdx @@ -1,9 +1,11 @@ --- -title: "Schemas" -description: "Declare model columns inline or reuse canonical inherited schemas." +title: "Reusable column schemas" +description: "Declare a model's columns in its header, or once in a SCHEMA() that many models share." --- -Schema metadata defines column names, types, nullability, descriptions, and column audits. It can live inline in one `MODEL()` header or in a reusable `SCHEMA()` declaration. +A model's columns, with their types, nullability, descriptions and audits, are declared in its +`MODEL()` header. When several models output the same columns, declare them once in a `SCHEMA()` and +bind each model to it. ## Inline columns @@ -122,13 +124,16 @@ binding a schema. Enum column types must likewise be visible from the schema's o Inheritance may be transitive. Parent columns resolve before child columns. An inherited column cannot be redeclared or overridden in a child schema. SQLBuild rejects unknown parents, cycles, case-insensitive duplicates, and multiple parents. -Physical SQL output order is not currently enforced. Static contract analysis matches names and checks declared types and proven non-nullability. Runtime exact-contract validation matches names and types but does not validate nullability. +## Contracts and rebuilds -## Contracts and planning +A model's [contract](/docs/concepts/models/contracts) covers its full column list: the schema's +columns plus the ones the model adds. -`contract enforced` treats the complete named-plus-local declaration as the exact output shape. An unspecified model follows the project's `check_declared_columns` setting; `contract none` explicitly keeps the resolved columns as metadata, audit attachment, and type-enforcement inputs without activating shape validation. See [Contracts](/docs/concepts/models/contracts). - -For models bound to a reusable schema, effective column names, types, nullability, and enum members participate in model version identity. Changing those fields on a parent therefore affects models bound through descendants. Descriptions do not change model identity. Audits have their own audit-gate identities, so audit changes invalidate reusable audit results without changing the model version itself. +Changing a column's name or type in a schema changes every model bound to it, or to a schema that +extends it. The plan shows those models as `Schema changed` and replays them as described in +[Replay decisions](/docs/concepts/planning/replay-decisions). Changing nullability or a description +doesn't change the plan. +Changing an audit reruns that audit without changing the model. ## Limitations diff --git a/website/src/content/docs/docs/concepts/models/type-enforcement.mdx b/website/src/content/docs/docs/concepts/models/type-enforcement.mdx index 65fc5c323a..3c39cea865 100644 --- a/website/src/content/docs/docs/concepts/models/type-enforcement.mdx +++ b/website/src/content/docs/docs/concepts/models/type-enforcement.mdx @@ -3,7 +3,7 @@ title: "Type Enforcement" description: "Understand declared model types, static checks, and runtime casting." --- -Type enforcement makes authored column types operational. It is a model feature as well as a related source feature, and it is separate from whether a model uses an exact contract. +Declare a column's type and SQLBuild checks it at compile time and, on staged tables and incremental models, casts the column to it at runtime. You don't switch it on: any declared type enables it. ## Enabling it @@ -55,17 +55,10 @@ Runtime cast support currently depends on materialization: Staged promotion is the default table mode. If a project selects immediate table promotion, a typed model fails with guidance to use staged promotion because SQLBuild cannot inspect and reconstruct output before mutating the destination. -## Type enforcement versus contracts +## Type enforcement and contracts -These controls answer different questions: - -| Feature | Question | -|---------|----------| -| Type enforcement | Should typed columns be checked and, on supported paths, cast to their declared physical types? | -| `contract none` | Which declared columns can SQLBuild verify statically while still allowing additional output columns? | -| `contract enforced` | Must the output have exactly the complete declared shape? | - -Type enforcement can be active with `contract none`. Conversely, an enforced contract may contain untyped columns: exact shape validation still checks their names, but there is no declared physical type to cast or compare. +Type enforcement works whatever the model's `contract` setting, including `contract none`. A +contract decides which columns must exist; see [Contracts](/docs/concepts/models/contracts). ## Schema changes diff --git a/website/src/content/docs/docs/concepts/observability.mdx b/website/src/content/docs/docs/concepts/observability.mdx index c93bfa395a..08f04474b1 100644 --- a/website/src/content/docs/docs/concepts/observability.mdx +++ b/website/src/content/docs/docs/concepts/observability.mdx @@ -1,24 +1,10 @@ --- -title: "Execution Observability" -description: "Choose authoritative lifecycle facts, readable logs, or command-output records." +title: "Observability" +description: "Find out what a run did, or send its events to your own systems." --- -SQLBuild keeps execution facts separate from human output. Do not parse terminal text to determine -whether work started, completed, failed, skipped, or retried. - -| Record | Purpose | Authority | -|--------|---------|-----------| -| `LifecycleEvent` | Immutable invocation, run, resource-attempt, operation, statement, retry, and audit facts | Canonical lifecycle evidence once durably stored | -| `DiagnosticLog` | Structured framework diagnostics | Explanatory only | -| [Recall](/docs/concepts/observability/recall) | Queryable history of invocations, statements, SQL, audit results, and output, locally or in ClickHouse | Troubleshooting history; best effort | -| `stdout.log` / `stderr.log` | Exact host-local process output (compute log files) | Troubleshooting transcript; also stored by Recall | -| `CommandOutputRecord` | Bounded, ANSI-free remote stdout/stderr chunks | Potentially sensitive and lossy troubleshooting data | -| Final JSON output | End-of-command aggregate | Result projection, not event history | - -Human-readable CLI output remains the normal operator interface. Structured records are a separate -contract for integrations and durable consumers. - -## Which to use +SQLBuild records what each command did, separately from what it prints. Use these records instead +of parsing terminal output. | You want to | Use | |-------------|-----| @@ -29,12 +15,23 @@ contract for integrations and durable consumers. Recall is on by default and covers investigation for people and agents. Sinks and execution history are for building on SQLBuild's events elsewhere. The `stdout.log`, `stderr.log` and -`diagnostics.jsonl` compute log files under `logs//` are still written alongside Recall and -hold the same bytes; prefer `sqb recall output`, which works the same with a shared store. +`diagnostics.jsonl` files under `logs//` hold the same output as Recall; prefer +`sqb recall output`, which also works with a shared store. + +## What SQLBuild records + +| Record | What it holds | How far to rely on it | +|--------|---------------|-----------------------| +| `LifecycleEvent` | Each invocation, run, resource attempt, operation, statement, retry and audit | Complete once stored; events are never changed | +| `DiagnosticLog` | SQLBuild's structured diagnostics | Explanation only | +| [Recall](/docs/concepts/observability/recall) | Invocations, statements, SQL, audit results and output, locally or in ClickHouse | Best effort, for troubleshooting | +| `stdout.log` / `stderr.log` | The exact output of each process on this machine | For troubleshooting; Recall also stores it | +| `CommandOutputRecord` | Command output in bounded chunks, without colour codes, for remote systems | May be incomplete and may hold sensitive data | +| Final JSON output | The command's result | The result only, not what happened on the way | ## Lifecycle envelope -New events use lifecycle schema version 2: +Events use lifecycle schema version 2: ```json { diff --git a/website/src/content/docs/docs/concepts/observability/execution-history.mdx b/website/src/content/docs/docs/concepts/observability/execution-history.mdx index 95dcb01931..34308d8d2a 100644 --- a/website/src/content/docs/docs/concepts/observability/execution-history.mdx +++ b/website/src/content/docs/docs/concepts/observability/execution-history.mdx @@ -3,14 +3,16 @@ title: "Execution history" description: "Store and query lifecycle events and run history from Python, with SQLite locally or PostgreSQL when deployed." --- -`LifecycleEventLogStorage` stores canonical immutable lifecycle facts. `RunStorage` serves an atomically -published, rebuildable run projection. Diagnostic and compute logs are never accepted by either -contract. The public types, filters, pages, records, errors, and helper functions are exported from -`sqlbuild.execution_history`. +Execution history is a Python API for building your own run history: store SQLBuild's lifecycle +events in SQLite or PostgreSQL and read runs back, for example to show run status in an internal +tool. Everything is exported from `sqlbuild.execution_history`. -This is a Python API for applications that store and serve SQLBuild events themselves. To look into -what past runs did, use [Recall](/docs/concepts/observability/recall), which records every command -automatically and is queried with [`sqb recall`](/docs/cli/recall). +To look into what past runs did, you don't need this API: use +[Recall](/docs/concepts/observability/recall), which records every command automatically and is +queried with [`sqb recall`](/docs/cli/recall). + +`LifecycleEventLogStorage` stores events as they arrive and never changes them. `RunStorage` serves +the current state of each run, rebuilt from those events. Neither stores logs or command output. ## Public contracts @@ -20,10 +22,8 @@ automatically and is queried with [`sqb recall`](/docs/cli/recall). The concrete SQLite and PostgreSQL classes additionally provide transactional `append_and_project`, `reconcile`, and `check_health` operations. -The deprecated `EventLogStorage` alias has been removed; use `LifecycleEventLogStorage`. Storage -tables use the lifecycle-specific names `lifecycle_event_log` in SQLite and -`sqlbuild_lifecycle_event_log` in PostgreSQL; the former `event_log` and `sqlbuild_event_log` names -are retired. +Events are stored in the `lifecycle_event_log` table in SQLite and `sqlbuild_lifecycle_event_log` +in PostgreSQL. Each newly stored event receives a positive, increasing `storage_order`, opaque stable `cursor`, and UTC `received_at`. Pages ascend in storage order. `after_cursor` is exclusive, diff --git a/website/src/content/docs/docs/concepts/observability/recall-clickhouse.mdx b/website/src/content/docs/docs/concepts/observability/recall-clickhouse.mdx new file mode 100644 index 0000000000..90f904456b --- /dev/null +++ b/website/src/content/docs/docs/concepts/observability/recall-clickhouse.mdx @@ -0,0 +1,114 @@ +--- +title: "Sharing Recall in ClickHouse" +description: "Keep one Recall history for a team or CI in ClickHouse: users, grants, setup, retention and reading." +--- + +Point `write_connection` and `read_connection` at ClickHouse connections to keep one history for a +whole team. Every machine +that writes to it, including CI and scheduled production runs, becomes queryable from anywhere: +see what last night's production build ran and printed without access to the machine, find the +first run where an audit started failing across everyone's runs, or compare a production run with +your own by `target`, `git_revision` or `sqlbuild_version`. + +The connection needs `adapter = "clickhouse"` when the project uses another adapter. Recall creates +and upgrades its own store on the first write: the `database` when it does not exist yet, then its +tables, views and schema version. A writer that may run that DDL needs no other setup. + +```toml +[connections.history_writer] +adapter = "clickhouse" +host = "${env:CLICKHOUSE_HOST}" +user = "recall_writer" +password = "${env:RECALL_WRITER_PASSWORD}" +database = "observability" + +[connections.history_reader] +adapter = "clickhouse" +host = "${env:CLICKHOUSE_HOST}" +user = "recall_reader" +password = "${env:RECALL_READER_PASSWORD}" +database = "observability" + +[recall] +write_connection = "history_writer" # every command writes history; creates the store on first use +read_connection = "history_reader" # sqb recall query, schema and output (SELECT only) +retention = "30d" +``` + +`sqb recall query`, `schema` and `output` use only `read_connection` and never create or change +anything in the store, so someone investigating production runs needs no credentials that could +rewrite or delete its history. Both keys may name one connection when separate users are not +wanted. A machine that only reads shared history, such as a laptop investigating production runs, +sets only `read_connection`; its own commands keep recording to the local store. Each side +resolves only its own connection, so environment references in the other connection, such as the +writer's password on a reader's laptop, need not be set. + +## Users and grants + +SQLBuild doesn't enforce grants; it lets you separate the roles. Give the writer the rights to +create and upgrade the store in its database, so the first command sets it up and later SQLBuild +versions upgrade it without anyone running DDL by hand: + +```sql +-- once, as an administrator: the users +CREATE USER recall_writer IDENTIFIED BY '...'; +CREATE USER recall_reader IDENTIFIED BY '...'; +GRANT CREATE DATABASE, CREATE TABLE, CREATE VIEW, DROP TABLE, ALTER ADD COLUMN, SELECT, INSERT + ON observability.* TO recall_writer; +GRANT SELECT ON observability.* TO recall_reader; +``` + +`DROP TABLE` is needed because ClickHouse replaces a view through a temporary table; Recall never +drops a table itself. + +| User | Rights | What happens | +|------|--------|--------------| +| Writer with the grants above | Create and upgrade the store | The first command creates the store; later versions upgrade it | +| Writer with only `SELECT` and `INSERT` | Write history | Works once the store exists. A missing or older store makes Recall print the statements an administrator should run | +| Reader with `SELECT` | Read history | `sqb recall query`, `schema` and `output` work, and can't change the store | + +A reader may be a standard read-only user (`SETTINGS readonly = 1`, or a settings profile with +it). Its queries run with the server's own settings; every other reader gets the +[settings SQLBuild pins](/docs/concepts/adapters/clickhouse#settings-sqlbuild-pins). The views +return the same rows either way. `sqb recall query` runs with `readonly = 2`, which refuses writes +but allows query settings, unless the user is already read-only. A user that may not set the row +limit gets the same truncated result, cut off while reading. A setting Recall needs that the user +may not change fails with an error naming it: set it in the user's profile, or give the user +`readonly = 2`, which keeps it read-only. + +## Setting up and upgrading the store + +- **Setup.** On first use, the writer creates the database if it's missing, printing + ``Recall: created ClickHouse database `observability` for the store.`` once, then the store's + tables and views. An existing database is used as it is, whatever its engine. +- **Clusters.** The database is created on the server the writer connects to, never `ON CLUSTER`. + On a self-managed cluster whose default database engine isn't `Replicated`, create the database + yourself first with `ENGINE = Replicated(...)`; Recall leaves it alone. ClickHouse Cloud needs + nothing. `ON CLUSTER` stores aren't supported: use a single ClickHouse server or ClickHouse Cloud. +- **Without DDL rights.** If the writer may not create or upgrade the store, the command still + runs and exits as it otherwise would. Recall prints the statements an administrator should run, at + most once a day, starting with `CREATE DATABASE IF NOT EXISTS` when the database is missing. + `sqb recall schema --ddl` prints them again. A store behind the current version still receives + history, in the columns that exist. +- **Mixed versions.** A writer never replaces the views of a store that a newer SQLBuild version + created. +- **Many writers.** Many concurrent commands can write one store. A repeated delivery is counted + once. A record written again with different content keeps the stored row, and the rejected copy + is listed in the `conflicts` view. +- **Writes.** History is sent about every five seconds and when the command ends, with the same shutdown + budget and exit-code guarantees as the local store. + +## Retention in ClickHouse + +Retention belongs to the store. Every table has a TTL of `retention`; SQL text is kept a week +longer, and for as long as a kept invocation references it. The views hide expired rows at once, +but ClickHouse drops expired parts in the background, so rows may stay on disk for a while. + +When a project's `retention` differs from the tables' TTL, Recall warns once a day with the +`ALTER TABLE … MODIFY TTL` and view statements that would apply it. It never changes the TTL +itself. + +## Reading a ClickHouse store + +`sqb recall query`, `schema` and `output` read the ClickHouse store, so queries use ClickHouse SQL. +`--json` output includes the server's `rows_read`, `bytes_read` and `query_id`. diff --git a/website/src/content/docs/docs/concepts/observability/recall.mdx b/website/src/content/docs/docs/concepts/observability/recall.mdx index b459907887..c32df9d52b 100644 --- a/website/src/content/docs/docs/concepts/observability/recall.mdx +++ b/website/src/content/docs/docs/concepts/observability/recall.mdx @@ -4,7 +4,7 @@ description: "The local history of every command: what ran, what it executed, an --- Recall records each `sqb` invocation in a local SQLite store, or in a shared -[ClickHouse store](#sharing-history-in-clickhouse), so you can ask what happened after the fact: which command ran, how it ended, which statements it executed and how long they took, and +[ClickHouse store](/docs/concepts/observability/recall-clickhouse), so you can ask what happened after the fact: which command ran, how it ended, which statements it executed and how long they took, and exactly what it printed. Recall is on by default and needs no setup. ## An investigation, start to finish @@ -283,115 +283,9 @@ prints a notice and records nothing. ## Sharing history in ClickHouse -Point `write_connection` and `read_connection` at ClickHouse connections to keep one history for a -whole team. Every machine -that writes to it, including CI and scheduled production runs, becomes queryable from anywhere: -see what last night's production build ran and printed without access to the machine, find the -first run where an audit started failing across everyone's runs, or compare a production run with -your own by `target`, `git_revision` or `sqlbuild_version`. - -The connection needs `adapter = "clickhouse"` when the project uses another adapter. Recall creates -and upgrades its own store on the first write: the `database` when it does not exist yet, then its -tables, views and schema version. A writer that may run that DDL needs no other setup. - -```toml -[connections.history_writer] -adapter = "clickhouse" -host = "${env:CLICKHOUSE_HOST}" -user = "recall_writer" -password = "${env:RECALL_WRITER_PASSWORD}" -database = "observability" - -[connections.history_reader] -adapter = "clickhouse" -host = "${env:CLICKHOUSE_HOST}" -user = "recall_reader" -password = "${env:RECALL_READER_PASSWORD}" -database = "observability" - -[recall] -write_connection = "history_writer" # every command writes history; creates the store on first use -read_connection = "history_reader" # sqb recall query, schema and output (SELECT only) -retention = "30d" -``` - -`sqb recall query`, `schema` and `output` use only `read_connection` and never create or change -anything in the store, so someone investigating production runs needs no credentials that could -rewrite or delete its history. Both keys may name one connection when separate users are not -wanted. A machine that only reads shared history, such as a laptop investigating production runs, -sets only `read_connection`; its own commands keep recording to the local store. Each side -resolves only its own connection, so environment references in the other connection, such as the -writer's password on a reader's laptop, need not be set. - -### Users and grants - -SQLBuild doesn't enforce grants; it lets you separate the roles. Give the writer the rights to -create and upgrade the store in its database, so the first command sets it up and later SQLBuild -versions upgrade it without anyone running DDL by hand: - -```sql --- once, as an administrator: the users -CREATE USER recall_writer IDENTIFIED BY '...'; -CREATE USER recall_reader IDENTIFIED BY '...'; -GRANT CREATE DATABASE, CREATE TABLE, CREATE VIEW, DROP TABLE, ALTER ADD COLUMN, SELECT, INSERT - ON observability.* TO recall_writer; -GRANT SELECT ON observability.* TO recall_reader; -``` - -`DROP TABLE` is needed because ClickHouse replaces a view through a temporary table; Recall never -drops a table itself. - -| User | Rights | What happens | -|------|--------|--------------| -| Writer with the grants above | Create and upgrade the store | The first command creates the store; later versions upgrade it | -| Writer with only `SELECT` and `INSERT` | Write history | Works once the store exists. A missing or older store makes Recall print the statements an administrator should run | -| Reader with `SELECT` | Read history | `sqb recall query`, `schema` and `output` work, and can't change the store | - -A reader may be a standard read-only user (`SETTINGS readonly = 1`, or a settings profile with -it). Its queries run with the server's own settings; every other reader gets the -[settings SQLBuild pins](/docs/concepts/adapters/clickhouse#settings-sqlbuild-pins). The views -return the same rows either way. `sqb recall query` runs with `readonly = 2`, which refuses writes -but allows query settings, unless the user is already read-only. A user that may not set the row -limit gets the same truncated result, cut off while reading. A setting Recall needs that the user -may not change fails with an error naming it: set it in the user's profile, or give the user -`readonly = 2`, which keeps it read-only. - -### Setting up and upgrading the store - -- **Setup.** On first use, the writer creates the database if it's missing, printing - ``Recall: created ClickHouse database `observability` for the store.`` once, then the store's - tables and views. An existing database is used as it is, whatever its engine. -- **Clusters.** The database is created on the server the writer connects to, never `ON CLUSTER`. - On a self-managed cluster whose default database engine isn't `Replicated`, create the database - yourself first with `ENGINE = Replicated(...)`; Recall leaves it alone. ClickHouse Cloud needs - nothing. `ON CLUSTER` stores aren't supported: use a single ClickHouse server or ClickHouse Cloud. -- **Without DDL rights.** If the writer may not create or upgrade the store, the command still - runs and exits as it otherwise would. Recall prints the statements an administrator should run, at - most once a day, starting with `CREATE DATABASE IF NOT EXISTS` when the database is missing. - `sqb recall schema --ddl` prints them again. A store behind the current version still receives - history, in the columns that exist. -- **Mixed versions.** A writer never replaces the views of a store that a newer SQLBuild version - created. -- **Many writers.** Many concurrent commands can write one store. A repeated delivery is counted - once. A record written again with different content keeps the stored row, and the rejected copy - is listed in the `conflicts` view. -- **Writes.** History is sent about every five seconds and when the command ends, with the same shutdown - budget and exit-code guarantees as the local store. - -### Retention in ClickHouse - -Retention belongs to the store. Every table has a TTL of `retention`; SQL text is kept a week -longer, and for as long as a kept invocation references it. The views hide expired rows at once, -but ClickHouse drops expired parts in the background, so rows may stay on disk for a while. - -When a project's `retention` differs from the tables' TTL, Recall warns once a day with the -`ALTER TABLE … MODIFY TTL` and view statements that would apply it. It never changes the TTL -itself. - -### Reading a ClickHouse store - -`sqb recall query`, `schema` and `output` read the ClickHouse store, so queries use ClickHouse SQL. -`--json` output includes the server's `rows_read`, `bytes_read` and `query_id`. +Recall writes to a local SQLite store by default. To keep one history for a team, CI and production +runs, point it at ClickHouse; see +[Sharing Recall in ClickHouse](/docs/concepts/observability/recall-clickhouse). ## Filtering what Recall stores @@ -528,3 +422,4 @@ statement. It is `NULL` on DuckDB and BigQuery; a BigQuery statement's job ID is Compute logs (`logs///stdout.log`, `stderr.log`, and `diagnostics.jsonl`) are written alongside Recall. Recall stores the same output bytes, captured at the same point, so `sqb recall output` matches the compute log files. + diff --git a/website/src/content/docs/docs/concepts/planning/selection-and-staleness.mdx b/website/src/content/docs/docs/concepts/planning/selection-and-staleness.mdx index d8f49682a9..e4bde6be0a 100644 --- a/website/src/content/docs/docs/concepts/planning/selection-and-staleness.mdx +++ b/website/src/content/docs/docs/concepts/planning/selection-and-staleness.mdx @@ -3,19 +3,39 @@ title: "Selection and staleness" description: "How selection interacts with change detection, and the stale warnings that stop silent partial rebuilds." --- -Change detection is selection-aware. When you scope a run with `--select`, SQLBuild only runs (and only displays) the resources you selected, but it still reasons about the *whole* graph to keep the result honest. The key case it handles: a selected model whose upstream changed but is **not** in the selection. +When you select part of the project with `--select`, SQLBuild runs only what you selected, but it +still checks the whole graph. The case it watches for: a selected model whose upstream changed but +isn't selected, so the model would build on stale input. ## Only selected resources appear in the plan -For a scoped run like `sqb dbt build --select agg_daily_revenue`, the plan shows only the selected resources, not their passive upstream closure. SQLBuild still walks upstreams internally for change propagation (a selected model flips to run when an upstream changed), but those passive upstreams are not listed as plan entries or counted in the header. +The plan lists only the selected resources, not the upstreams they read. SQLBuild still looks at +those upstreams: a selected model runs when an upstream changed. `--full-refresh` also applies only +to the selected models. -`--full-refresh` is likewise scoped to the selected models, not the expanded upstream closure. +## Stale upstreams -## Stale warnings instead of silent partial rebuilds +`sqb plan` and `sqb build` run every selected model, even when an unselected upstream changed. Pass +`--selection-diagnostics` to list those stale upstreams. After changing `stg_orders`, +`sqb plan --select fact_orders --selection-diagnostics` shows: -If a selected model has an upstream that changed but is **outside the selection**, rebuilding the selected model alone would run it on top of a stale upstream and silently produce a result that looks current but isn't. SQLBuild does not do that. It leaves the selected model current (it is not rebuilt) and emits a stale warning naming the changed upstreams, with the selector you can use to incorporate them. +``` +Warnings (1) +└── fact_orders + ├── selected model 'fact_orders' will build on 1 stale upstream(s) not selected for rebuild: + ├── - stg_orders + └── rebuild the closure to refresh them: --select +fact_orders +``` -Suppose `stg_orders` (upstream) changed, and you select only the downstream `agg_daily_revenue`: +To include the changed upstreams, select the closure: `--select +fact_orders` builds `stg_orders` +and everything between it and `fact_orders`. + +A changed seed outside the selection is treated the same way as a changed model. + +## With dbt + +`sqb dbt plan`, `run` and `build` are stricter. A selected model whose upstream changed outside the +selection is not rebuilt: it stays current, and the plan warns, naming the changed upstreams. ```bash sqb dbt plan --select agg_daily_revenue @@ -37,23 +57,7 @@ Warnings (1) (e.g. +model) to incorporate it ``` -The model stays current rather than being rebuilt on a stale input, and the warning tells you exactly which upstreams changed and how to pull them in. - -`sqb plan` and `sqb build` work differently: they run every selected model, even when an -unselected upstream changed. Pass `--selection-diagnostics` to list those stale upstreams. After -changing `stg_orders`, `sqb plan --select fact_orders --selection-diagnostics` shows: - -``` -Warnings (1) -└── fact_orders - ├── selected model 'fact_orders' will build on 1 stale upstream(s) not selected for rebuild: - ├── - stg_orders - └── rebuild the closure to refresh them: --select +fact_orders -``` - -## Incorporating the changed upstreams - -Use a closure selector to include the changed upstreams in the run. With `+agg_daily_revenue`, SQLBuild pulls in the changed `stg_orders`, cascades the change through the intermediate models, and rebuilds: +Select the closure to pull in the changed upstreams and rebuild: ```bash sqb dbt plan --select +agg_daily_revenue @@ -72,11 +76,3 @@ Plan ready (8 selected resources) model.jaffle_analytics.int_order_payments upstream changed model.jaffle_analytics.stg_order_statuses upstream changed ``` - -## Seeds follow the same rule - -Seed changes are tracked the same selection-aware way. A changed seed that is outside the current selection does not trigger a partial rebuild of its dependents; instead the dependents are left current and a stale warning is emitted, exactly as for models. This keeps a scoped run from quietly building on a seed that the run didn't reload. - -## Why this matters - -A run that reports success but silently built a model on stale inputs is a correctness hazard - the data looks fresh but isn't. Surfacing staleness as an explicit warning (and refusing the misleading partial rebuild) keeps scoped runs honest: you either get a result built on current inputs, or a clear warning telling you it would not be, with the fix. diff --git a/website/src/content/docs/docs/concepts/planning/source-freshness.mdx b/website/src/content/docs/docs/concepts/planning/source-freshness.mdx index 19fa8ab0af..4bb4ccf610 100644 --- a/website/src/content/docs/docs/concepts/planning/source-freshness.mdx +++ b/website/src/content/docs/docs/concepts/planning/source-freshness.mdx @@ -9,7 +9,9 @@ are affected and choose the correct incremental action. ## Configuration -Source freshness is configured per source in `sources/*.yml` with a `freshness:` block. See [Sources: Source freshness](/docs/concepts/sources#source-freshness) for the full configuration reference. +Configure freshness per source with a `freshness:` block, including `lag_tolerance` for timestamps +that move without the data changing. See [Sources](/docs/concepts/sources#source-freshness) for the +fields. ## How observations work @@ -21,15 +23,11 @@ During planning, SQLBuild observes the current data version of each source that Sources without explicit `freshness:` config are auto-observed using the `adapter` strategy if the adapter supports table metadata and the source has a physical table (not an expression source, not a managed source). -Adapter metadata for all sources is fetched in one batched lookup, but each source gets its own -result. When one source can't be observed, only that source is unknown; the others keep their -freshness. The `Freshness metadata` progress line reports the outcome, for example -`Freshness metadata OK (257 sources, 3 unknown)`. The plan adds one warning per reason, listing up -to 10 sources, such as `source freshness unknown (unavailable) for raw_events, raw_sessions: ...`. -Sources without freshness configuration that the adapter can't observe are not warned about. -Every unknown source is listed with its reason and message under `unknown_source_details` in the -`direct_source_freshness` metadata of `sqb plan --json`; run `sqb freshness` for the same detail -in text form. +When one source can't be observed, only that source is unknown; the others keep their freshness. +The plan warns once per reason, for example +`source freshness unknown (unavailable) for raw_events, raw_sessions: ...`. Sources without +freshness configuration that the adapter can't observe are not warned about. `sqb freshness` shows +each unknown source with its reason, as does `unknown_source_details` in `sqb plan --json`. ## Missing source tables @@ -48,15 +46,10 @@ sources and sources with a loader are exempt, because SQLBuild creates their tab them. Expression sources, sources without a schema, and wildcard tables are also exempt. A missing table that only unselected models read does not fail the plan; its freshness is reported as unknown. -The check first looks for each source in the warehouse listing that planning already reads for -source columns, so it normally issues no extra queries. A source the listing does not find (for -example because of identifier case, a stream, a materialized view, or a table in another database) -is then probed directly with `SELECT 1 FROM WHERE 1=0`, using the same relation name the -model SQL reads. Probes for several such sources run concurrently, within the same limit as other -warehouse inspection reads. Only a not-found error from that probe counts as missing, including a -missing schema, catalog, or attached database; any other error (for example a connection error) -fails planning with that error. On Snowflake the probe may need a running warehouse, but it only -runs for sources the listing did not find. +SQLBuild finds missing tables in the warehouse listing it already reads, then probes the sources it +didn't find there, such as streams or tables in another database, with +`SELECT 1 FROM WHERE 1=0`. On Snowflake the probe may need a running warehouse. Only a not-found error counts as missing; any other error, +such as a lost connection, fails planning with that error. Snowflake reports a missing table and a table the current role can't read with the same error, and its listing hides tables the role can't access. For those sources `S405` says the table "does not @@ -67,10 +60,6 @@ A table that a model's own `pre_hook` creates does not exist yet when the plan i fails with `S405`. Declare such a table as a managed source or a source with a loader, or create it before planning. -## Lag tolerance - -For timestamp-based freshness, `lag_tolerance` controls how much the observed value can drift before being considered a real change. If the current timestamp is within the tolerance of the previous observation, the source is treated as unchanged. This is useful for sources where the freshness timestamp moves by seconds or minutes on every query but the underlying data hasn't meaningfully changed. - ## State storage Source freshness observations are appended to `_sqlbuild_source_freshness` in each target schema. diff --git a/website/src/content/docs/docs/concepts/project-configuration.mdx b/website/src/content/docs/docs/concepts/project-configuration.mdx index 32e5661fa1..33a7d51f30 100644 --- a/website/src/content/docs/docs/concepts/project-configuration.mdx +++ b/website/src/content/docs/docs/concepts/project-configuration.mdx @@ -1,34 +1,13 @@ --- -title: "Project Configuration" -description: "Configure your SQLBuild project with sqlbuild_project.toml and sqlbuild_local.toml." +title: "Configuration reference" +description: "Every key in sqlbuild_project.toml and sqlbuild_local.toml." --- -SQLBuild projects are configured with two files in the project root: +Every key in `sqlbuild_project.toml`, which you commit, and `sqlbuild_local.toml`, which each +developer keeps out of Git. For a walkthrough, see [Set up a project](/docs/project-setup). -- **`sqlbuild_project.toml`** - shared project configuration, committed to version control -- **`sqlbuild_local.toml`** - local developer overrides, gitignored - -Macro, constant, and enum visibility is not configured in either file. Declaration scopes use filesystem conventions; see [Declaration Scopes](/docs/concepts/declaration-scopes). - -Most projects need only one committed `sqlbuild_project.toml`. Define shared targets such -as `dev` and `prod` there, including clone policies and team-wide defaults. Do not maintain -separate complete project files for each environment. - -Create `sqlbuild_local.toml` only when a developer or execution environment needs different -target selection, credentials, schemas, adapter settings, or variables. SQLBuild loads it -automatically and merges its explicitly configured values over the shared project config. - -```text -project/ - sqlbuild_project.toml # committed: shared targets and behavior - sqlbuild_local.toml # gitignored: this developer's overrides -``` - -Add the local file to `.gitignore`: - -```gitignore -sqlbuild_local.toml -``` +Where macros, constants and enums are visible is set by folders, not in these files; see +[Declaration Scopes](/docs/concepts/declaration-scopes). ## sqlbuild_project.toml @@ -67,7 +46,7 @@ materialized = "view" | Field | Description | |-------|-------------| -| `name` | Project name. Used in fingerprint tracking and manifest generation. | +| `name` | Project name | | `adapter` | Database adapter: `duckdb`, `motherduck`, `snowflake`, `bigquery`, `databricks`, `postgres`, `sqlite`, `sqlserver`, or `clickhouse`. See [Adapters](/docs/concepts/adapters). | ### Optional fields @@ -136,11 +115,9 @@ explicitly: database = ":memory:" ``` -A `database` or `schema` present in a named connection is connection/session metadata only; -it does not satisfy the mandatory namespace strategy for a named target. Put the target's -authoritative database and schema on `[targets.]`. SQLBuild validates connection -references while loading configuration, without opening a warehouse connection, and reports -an unknown `targets..connection` name as an offline configuration error. +A `database` or `schema` on a connection doesn't decide where models are built; set them on the +target. A target that names an unknown connection fails when the configuration loads, before +anything connects. ## Targets @@ -155,8 +132,8 @@ A target is a named build context - the database and schema you build into, plus | `vars` | Target-specific project variables | | `defer_sources_to` | Target name to read managed source data from (see [Loaders](/docs/concepts/python-nodes/loaders#source-deferral)) | | `clone` | Clone policy (see below) | -| `missing_migration_origin` | What a build does when a declared `migrate_from` origin does not exist in this target: `allow` (default), `require_confirmation`, or `deny` (see below) | -| `warehouses` | Default Snowflake warehouse per command group, `build` and `query` (see below) | +| `missing_migration_origin` | What a build does when a declared `migrate_from` origin does not exist in this target: `allow` (default), `require_confirmation`, or `deny`. See [Missing migration origins](/docs/concepts/models/migrations#missing-migration-origins) | +| `warehouses` | Snowflake warehouse per command group, `build` and `query`. See [Command-group warehouses](/docs/concepts/adapters/snowflake#command-group-warehouses) | | `concurrency` | Worker limit for commands run against this target, replacing `[settings] concurrency` (see below) | ```toml @@ -251,97 +228,6 @@ The value must be an integer of at least `1`; anything else fails with `D001`. ` `8 configured limit (targets.prod.concurrency)`. `sqb scenario test` has no `--target` option and uses the active target's value. -### Command-group warehouses - -Builds and ad-hoc inspection usually need different compute. A Snowflake target can choose a -default warehouse for each command group, so `sqb build` runs on a large warehouse while -`sqb query` and `sqb diff` stay on a small one: - -```toml -[connections.warehouse] -account = "my_org-my_account" -warehouse = "ANALYTICS_WH" # fallback for every command - -[targets.dev] -connection = "warehouse" -database = "ANALYTICS" -schema = "DEV" - -[targets.dev.warehouses] -build = "BUILD_WH" # commands in the build group -query = "ADHOC_WH" # commands in the query group -``` - -Each command connects with the first warehouse set in this order: - -1. `--warehouse ` on the command line, for that invocation only -2. The target's group warehouse, `[targets..warehouses] build` or `query` -3. The connection's `warehouse` -4. The Snowflake user's default warehouse - -Both groups are optional; a target without `warehouses` uses the connection's `warehouse`. A group -warehouse takes precedence over the connection `warehouse` in every file. In particular, a -shared group overrides a connection `warehouse` you set in `sqlbuild_local.toml`; to change a -group locally, set the group itself, or pass `--warehouse` for one invocation. Group values -accept the same `${ENV:...}` and project-variable (`${my_var}`) templates as connection values. -Group values and `--warehouse` must be Snowflake identifiers: unquoted (`ADHOC_WH`) or -double-quoted (`"Adhoc WH"`). - -Every connection a command opens uses its group, so one invocation never mixes warehouses: -`sqb build` plans, inspects, and executes on the `build` warehouse, and `sqb diff` creates its -temporary comparison tables on the `query` warehouse. The exception is `sqb dbt`: it applies the -`build` group to SQLBuild-owned work, while dbt nodes keep the warehouse from their dbt profile. - -| Group | Commands | -|-------|----------| -| `build` | `audit`, `build`, `check`, `clone`, `dbt` (SQLBuild-owned work), `load`, `plan`, `seed` | -| `query` | `contract`, `cost`, `debug`, `diff`, `freshness`, `janitor`, `query`, `scenario`, `test` | - -Other commands, such as `compile`, `lineage`, and `format`, never connect to a warehouse. -Integrations that run builds, such as Dagster, invoke `sqb build` and use the `build` group. -Standalone `sqb test`, `sqb scenario` and `sqb janitor` use the `query` group because they work on -fixture data or catalogue metadata; tests and audits that run inside `sqb build` use the `build` group. - -Each developer can point ad-hoc commands at their own warehouse without editing shared -configuration: - -```toml -# sqlbuild_local.toml -[targets.dev.warehouses] -query = "ALICE_XS_WH" -``` - -Local group values override the project value for the same group only; the other group keeps -its project value. `sqb debug` shows the resolved warehouse for each group and where it came -from, and `sqb build --verbose` shows it in the execution header. - -Groups need an adapter with a session warehouse: the Snowflake adapter, or a custom adapter -that extends it. `warehouses` in `sqlbuild_project.toml` is checked against the project's -adapter, and `warehouses` in `sqlbuild_local.toml` against the adapter you actually run. Setting -them for an adapter without a session warehouse, or passing `--warehouse`, is an error rather -than a silently ignored setting. The one exception is a local `adapter` override, for example a -developer running a Snowflake project on DuckDB: shared groups then don't apply, and -`sqb debug` reports them as `not used (adapter duckdb)`. - -### Missing migration origins - -A `migrate_from` declaration, on a model or a column, names a relation or column that exists in the targets where the rename is pending. A target that never built the old name, such as a fresh development schema, has nothing to migrate. `missing_migration_origin` decides what happens there: - -| Value | Behavior | -|-------|----------| -| `allow` | Default. The plan warns (`M102` for models, `M109` for columns), nothing is migrated, and the model builds as if `migrate_from` were absent. | -| `require_confirmation` | The build asks for confirmation on a terminal, and stops otherwise; pass `--allow-missing-migration-origin` to confirm in non-interactive runs. | -| `deny` | The build stops with `M102` or `M109`. Use it for targets where a missing origin means the wrong target or a lost relation. | - -```toml -[targets.prod] -missing_migration_origin = "deny" -``` - -`sqlbuild_local.toml` can override the value for a target. - -The policy covers only origins with no build history in the target. A model origin that SQLBuild built in this target but can no longer see, because of permissions or because it was dropped outside SQLBuild, always stops the build with `M102`. Column origins have no such history, so `M109` always follows the policy. `sqb dbt run` and `sqb dbt build` apply the policy too, but have no confirmation flag: they ask on a terminal and stop otherwise. In non-interactive dbt runs, build the affected models with `sqb build --allow-missing-migration-origin` and leave them out of the dbt selection, or set the target's `missing_migration_origin` to `allow`. - ### Clone policies Targets can declare whether they allow cloning to or from: @@ -366,9 +252,8 @@ Both policies default to `false`. `sqb clone --from prod --to dev` requires `all ### Execution limits -Named targets can set optional limits for `sqb build`, and per-item time limits for unit tests and -scenarios. Limits are useful for development targets where an unexpectedly broad selector or -long-running command should stop instead of consuming an unbounded amount of warehouse compute. +A target can cap what one `sqb build` runs, so a broad selector on a development target stops +instead of running up warehouse cost. A target without this section has no limits. ```toml [targets.dev] @@ -387,93 +272,16 @@ Automated tools must request approval before changing this policy. """ ``` -All fields are optional. A target without an `execution_limits` section has no limits, so -production or orchestrated targets need no placeholder values. - -#### Model limit - -`max_models` is a positive integer. SQLBuild expands selectors and counts model entries that would -execute. When the count exceeds the target limit, the build exits before clone, load, or model -execution begins. The error reports the target, selected count, configured maximum, configuration -key, and confirms that no warehouse changes were made. - -`sqb plan` remains unrestricted and can be used to inspect a broad selection safely. - -#### Duration limit - -`max_duration` is a positive fixed duration such as `30m`, `2h`, or `90s`. It covers planning and -warehouse-mutating build phases after target resolution. SQLBuild stops dispatching work at the -deadline, bounds each warehouse statement by the remaining time where the warehouse supports a -statement limit, and cancels statements still running at the deadline. - -Every built-in adapter enforces the limit: - -| Adapter | Warehouse statement limit | Cancellation at the deadline | Longest accepted duration | -|---------|---------------------------|------------------------------|---------------------------| -| Snowflake | `STATEMENT_TIMEOUT_IN_SECONDS` per statement | Driver cancel request for the running query | 7 days | -| DuckDB, MotherDuck | None | Connection interrupt | About 24 days | -| PostgreSQL | `SET LOCAL statement_timeout` inside transactions; statements outside a transaction rely on cancellation | Driver cancel (`cancel_safe`) | About 24 days | -| BigQuery | Job timeout (`job_timeout_ms`) on each query job | `jobs.cancel`; reported as requested, not confirmed | About 24 days | -| Databricks | `STATEMENT_TIMEOUT` session setting, capped at 48 hours | Cursor cancel; reported as requested, not confirmed | About 24 days | -| SQL Server | None; the driver's query timeout is process-wide | Driver cancel | About 24 days | -| SQLite | None | Connection interrupt | About 24 days | - -A project adapter that can't cancel a running statement is rejected for `max_duration`, rather than -showing a limit that may leave warehouse work running. Under a per-item timeout, its items stop -before their next statement and report the cancellation as not supported. - -A duration failure may come after earlier models completed. Its error therefore says it is a -timeout, unlike a model-limit failure before execution, and that warehouse changes may already -have happened. - -#### Per-item time limits - -Unit test, scenario and model limits (`test_timeout`, `scenario_timeout` and a model's -`max_duration`) count running time only. The clock starts at the item's first warehouse statement, -so rendering and setup before it do not count, and it stops during rollback and cleanup, which always -run even after the limit. A timeout is reported only when the limit passed before the item's work -finished. Where a warehouse reports queueing separately, queued time is excluded: - -| Adapter | How the item limit is enforced | Queued time | -|---------|--------------------------------|-------------| -| Snowflake | `STATEMENT_TIMEOUT_IN_SECONDS` per statement, with a client-side backstop after a grace margin of half the limit (at least 30 seconds) | Excluded by the warehouse limit; the backstop counts it | -| DuckDB, MotherDuck | Connection interrupt at the limit | Not applicable | -| PostgreSQL | `SET LOCAL statement_timeout` inside transactions; cancellation at the limit | Lock waits count | -| BigQuery | `jobs.cancel` at the limit; no job timeout, because a job timeout counts from job creation | Excluded: the clock stops while a job is pending | -| Databricks | `STATEMENT_TIMEOUT` session setting and cursor cancel | May count | -| SQL Server | Driver cancel at the limit | Lock waits count | -| SQLite | Connection interrupt at the limit | Waits for the write lock count | - -When a model contains unit tests or nested work with its own limit, the earliest limit applies and -cancels whatever is running under it. After a timeout interrupted a statement, SQLBuild checks the -connection before giving it to the next item and replaces it if it is no longer usable. - -#### Unit test and scenario timeouts - -`test_timeout` and `scenario_timeout` set the per-test and per-scenario time limit on this target, -replacing `[settings] test_timeout` and `[settings] scenario_timeout`. Each takes a positive fixed -duration or `"disabled"`. See [Unit test and scenario limits](#unit-test-and-scenario-limits). - -#### Custom remediation - -`remediation` is optional project-authored text. SQLBuild appends it to model-count, duration, and -unsupported-adapter errors after the factual error details. This lets each project define its own -approval or recovery process without SQLBuild recommending that users weaken local policy. - -#### Local overrides - -The same section is accepted in `sqlbuild_local.toml` and follows normal field-by-field local target -precedence: - -```toml -[targets.dev.execution_limits] -max_models = 200 -max_duration = "90m" -``` +| Field | Description | +|-------|-------------| +| `max_models` | Most models one build may run. A larger selection stops before anything runs; `sqb plan` has no limit | +| `max_duration` | Longest one build may run, such as `"45m"`. At the deadline SQLBuild cancels the statements still running | +| `test_timeout` | Time limit for each unit test on this target, or `"disabled"`; replaces `[settings] test_timeout` | +| `scenario_timeout` | Time limit for each scenario on this target, or `"disabled"`; replaces `[settings] scenario_timeout` | +| `remediation` | Text added to limit errors, such as how your team approves a larger build | -This permits an intentional developer override while keeping the shared target policy visible in -`sqlbuild_project.toml`. Other target fields and any execution-limit fields omitted locally continue -to inherit their project values. +See [Limits](/docs/cli/build#limits) for how each warehouse enforces them. `sqlbuild_local.toml` +accepts the same section: the fields it sets replace the project's, and the rest are inherited. ### Diff limits @@ -645,12 +453,12 @@ delta_compile = true |-------|---------|-------------| | `sql_analysis` | `true` | Enable SQL syntax analysis, column binding, type inference, output inference, and semantic validation at compile time | | `require_sql_analysis` | `false` | Allow `sql_analysis false` (in a MODEL header or a `[path_defaults]` entry) only on models whose SQL the parser cannot read, and reject `[settings] sql_analysis = false` in `sqlbuild_local.toml`. Project file only. See [SQL analysis](/docs/concepts/models/configuration#sql-analysis). | -| `query_change_tracking` | `true` | Track query fingerprints for change detection. A fingerprint covers the SQL tokens of the compiled query and ignores whitespace, comments, and the letter case of reserved keywords and built-in function names, so formatting-only edits do not rebuild. Every other word, including non-reserved keywords, identifiers, aliases, JSON path keys and user-defined function names, counts exactly, as do string literals, quoted identifiers, numbers, and optimizer hints. | -| `check_declared_columns` | `true` | Controls whether column declarations on models without a `contract` declaration activate static shape/nullability validation. `true` checks that every declared column is in the SELECT and checks its nullability; `false` treats columns as metadata and audit attachment unless the model declares `contract enforced`. Model-level `contract enforced` and `contract none` override this setting. Explicit type enforcement remains independent. See [Contracts](/docs/concepts/models/contracts). | +| `query_change_tracking` | `true` | Rebuild a model when its compiled query changes. Whitespace, comments, and the case of reserved keywords and built-in function names don't count as changes | +| `check_declared_columns` | `true` | Check that models without a `contract` setting output their declared columns, and that `nullable false` columns can't be null. `false` makes those columns documentation and audit targets only. See [Contracts](/docs/concepts/models/contracts) | | `concurrency` | `1` | Default number of worker connections for commands that run resources in parallel, such as `sqb build`, `sqb test`, `sqb audit`, and `sqb seed`, and for `sqb janitor` actions on warehouses that run DDL concurrently; a target's `concurrency` and `--concurrency` override it (see [Target concurrency](#target-concurrency)) | -| `microbatch_concurrency` | `false` | Explicitly permit models with `batch_concurrency > 1`; concurrent batches use immutable coordination facts | +| `microbatch_concurrency` | `false` | Allow models with `batch_concurrency > 1` | | `auto_load_sources` | `true` | Automatically run source loaders before building dependent models during `sqb build`. See [Loaders](/docs/concepts/python-nodes/loaders). | -| `table_promotion_mode` | adapter default | `staged` (CTAS to staging, audit, then promote) or `immediate` (CTAS directly to target, then audit) | +| `table_promotion_mode` | `staged` | `staged` builds a table under a staging name and replaces the old one only after audits pass; `immediate` builds in place and audits afterwards. See [Table](/docs/concepts/models/materializations#table) | | `default_audit_severity` | `error` | Default severity for audits: `warn` or `error` | | `default_audit_run_scope` | `delta_and_final` | Default run scope for audits: `final` or `delta_and_final` | | `unit_test_max_real_models` | `10` | Most real (unmocked) models one SQL unit test may execute; a larger test fails compilation. See [Unit test and scenario limits](#unit-test-and-scenario-limits). | @@ -662,22 +470,17 @@ delta_compile = true | Setting | Limits | Details | |---------|--------|---------| -| `unit_test_max_real_models` | Real (unmocked) models one SQL unit test may run; a larger test fails compilation and names each model, the path that pulls it in, and the inputs to mock | [Real-model limit](/docs/concepts/testing#real-model-limit) | -| `test_timeout` | Each unit test's warehouse time, from its first statement | [Time limits](/docs/concepts/testing#time-limits), including precedence | +| `unit_test_max_real_models` | Real (unmocked) models one SQL unit test may run; a larger test fails compilation and names each model, the path that pulls it in, and the inputs to mock | [Real-model limit](/docs/concepts/testing-reference#real-model-limit) | +| `test_timeout` | Each unit test's warehouse time, from its first statement | [Time limits](/docs/concepts/testing-reference#time-limits), including precedence | | `scenario_timeout` | Each scenario's warehouse time; no default | Below | The scenario limit that applies is the first one set of `--scenario-timeout`, `SCENARIO(timeout "...")`, `[targets..execution_limits] scenario_timeout`, and `[settings] scenario_timeout`. At the limit, SQLBuild cancels the scenario's running statement, reports it as `TIMEOUT`, and continues with independent items, as for -[unit tests](/docs/concepts/testing#time-limits). Every value is a positive fixed duration such as +[unit tests](/docs/concepts/testing-reference#time-limits). Every value is a positive fixed duration such as `"45s"` or `"10m"`, or `"disabled"`. `sqlbuild_local.toml` accepts all three settings. -### Table promotion mode - -- **`staged`** (default for most adapters): Materializes into a staging table, runs audits, then swaps into the target. If audits fail, the production table is untouched. -- **`immediate`**: Creates the table directly at the target location. Audits run after materialization. Simpler but no pre-promotion safety net. - ## Rules Rules configuration belongs in the shared `sqlbuild_project.toml` so local and CI compilation use @@ -870,24 +673,8 @@ critical_tags = ["critical"] ## dbt -Configuration for running SQLBuild alongside an existing dbt project: - -```toml -[dbt] -project_dir = "../dbt_project" -profiles_dir = "../profiles" -target_path = "../dbt_project/target" -target = "dev" -``` - -| Field | Description | -|-------|-------------| -| `project_dir` | Path to the dbt project root (where `dbt_project.yml` lives) | -| `profiles_dir` | Path to the directory containing `profiles.yml` | -| `target_path` | Path to dbt's `target/` directory (where `manifest.json` is written) | -| `target` | dbt target name override (optional) | - -Paths can be absolute or relative to the SQLBuild project root. See [Using SQLBuild with dbt](/docs/concepts/dbt-compatibility/overview) for setup and usage details. +`[dbt]` points SQLBuild at an existing dbt project. See +[Using SQLBuild with dbt](/docs/concepts/dbt-compatibility/overview#configuration) for its fields. ## Skills @@ -971,7 +758,3 @@ configured fields. Target blocks merge the same way and may override the connect `database`, `schema`, `loader_schema`, variables, source deferral, and policy fields. Unspecified values continue to come from `sqlbuild_project.toml`; a local reference to an unknown merged connection still fails offline during configuration loading. - -This replaces the common dbt pattern of switching profiles or setting environment variables -to change targets. Each developer sets their target, named connection, and preferences once in -`sqlbuild_local.toml` and it persists across sessions. diff --git a/website/src/content/docs/docs/concepts/python-nodes/loaders.mdx b/website/src/content/docs/docs/concepts/python-nodes/loaders.mdx index 6c538061ad..508c2c89c4 100644 --- a/website/src/content/docs/docs/concepts/python-nodes/loaders.mdx +++ b/website/src/content/docs/docs/concepts/python-nodes/loaders.mdx @@ -156,10 +156,9 @@ Existing rows matching the unique key are updated; new rows are inserted. Snowflake reads a backslash inside a string literal as an escape, so SQLBuild doubles every backslash in the values a loader returns. A value such as `C:\orders\2026` is stored exactly as -returned. Earlier releases wrote those backslashes unescaped, so Snowflake dropped them or read them -as escapes. If a `merge` or `delete_insert` unique key, or a value an `append` load is -deduplicated by, contains a backslash, rows loaded by an earlier release will not match newly -loaded rows. Rewrite or delete those rows before relying on the key. +returned. If a `merge` or `delete_insert` unique key, or a value an `append` load is deduplicated +by, contains a backslash, rows stored with single backslashes won't match newly loaded rows. +Rewrite or delete those rows before relying on the key. ## Self-managed loaders diff --git a/website/src/content/docs/docs/concepts/resource-identities.mdx b/website/src/content/docs/docs/concepts/resource-identities.mdx index 1a0f59282d..844f4726a0 100644 --- a/website/src/content/docs/docs/concepts/resource-identities.mdx +++ b/website/src/content/docs/docs/concepts/resource-identities.mdx @@ -1,61 +1,44 @@ --- -title: "Resource Identities" -description: "Canonical names for SQLBuild resources, selectors, state, and integrations." +title: "Naming rules" +description: "Names of models, tests, macros and other SQLBuild resources are lowercase snake_case." --- -SQLBuild resource identities use lowercase ASCII snake_case. A name must start with a lowercase -letter, end with a lowercase letter or digit, and contain only lowercase letters, digits, and -underscores. Consecutive underscores are valid, so established names such as -`commerce__mart_v_order` remain canonical. - -This contract applies to models, seeds, sources, SQL and Python functions, generic and singular -audits, attached audit definitions and instance names, SQL tests and parameterized cases, -scenarios, SQL and Python hooks, macros, model schemas, enums, constants, materializations, -providers, event exporters, loaders, tasks, assets, and checks. Private scoped declarations use -exactly one leading underscore, such as `_country_codes`; public declarations must not use a -leading underscore. - -Provider classes are the one conventional derivation: when `provider_name` is omitted, -`AnalyticsApiProvider` resolves to `analytics_api_provider`. An explicit `provider_name` is an -authored identity and must already be canonical snake_case; SQLBuild does not normalize it. - -Names derived from files use the filename stem. For example, `models/daily_orders.sql` defines -the model identity `daily_orders`, while `audits/generic/expression_is_true.sql` defines the -generic audit identity `expression_is_true`. Directories organize resources but do not change -their names. - -Physical warehouse identifiers are separate from SQLBuild resource identities. Database, schema, -table, model alias, column, tag, group, and directory names retain their existing adapter-specific -contracts and do not need to follow this profile. - -## Invalid names - -Compilation fails during discovery with `D016` when an authored identity is not canonical: +Names of models, tests, macros and every other SQLBuild resource are lowercase snake_case, such as +`daily_orders`. SQLBuild never renames anything for you; it suggests the right spelling. Renaming a +file by hand creates a new resource with no history, so use [`sqb rename`](/docs/cli/rename) to +keep it. ```text error[D016]: Invalid model identity 'DailyOrders' in models/DailyOrders.sql; use snake_case 'daily_orders' ``` -Discovery also fails with `D016` when the project directory, or a model, unit test, scenario, source -or schema YAML file it reads, has a path that is not valid UTF-8, including a directory above the -file. The error shows each invalid byte as a `\xNN` escape; rename the path to valid UTF-8. Other -files with such names, such as notes or data files, and directories that hold none of those files -are ignored. Commands that tolerate unreadable files, such as `sqb scope`, report the file and -continue. +A name starts with a lowercase letter, ends with a lowercase letter or digit, and contains only +lowercase letters, digits and underscores. Double underscores are fine, as in +`commerce__mart_v_order`. + +A resource named after its file takes the file name without the extension: +`models/daily_orders.sql` is the model `daily_orders`. Folders organise files but are not part of +the name. + +## Renaming a resource -SQLBuild suggests a corrected spelling but never silently normalizes an identity. Silent -normalization would make selectors, manifests, persisted execution state, and integration keys -disagree about which resource ran. +[`sqb rename`](/docs/cli/rename) and `sqb mv` rename a model or column, update every reference, and +keep the warehouse table's history. See [Refactoring](/docs/concepts/refactoring). -## Migrating existing projects +A rename you make by hand creates a new resource. Its build history, fingerprints and audit results +start fresh, and selectors and integrations that use the old name, such as Dagster asset keys, need +updating too. Compile before you build. -1. Rename file-derived resources and explicit `name` values to snake_case. -2. Update `__ref`, `__seed`, `__source`, function, hook, macro, audit, test, and dependency - references. -3. Update selectors and external integrations, including Dagster asset/check keys, that use the - old identity. -4. Compile before building. +## Details -A rename intentionally creates a new resource identity. Existing fingerprints, audit history, -and other persisted state under the old name are not silently reassigned to the new resource. +- A private scoped declaration starts with exactly one underscore, such as `_country_codes`. Public + declarations don't start with an underscore. See [Declaration scopes](/docs/concepts/declaration-scopes). +- A provider class without a `provider_name` gets a snake_case name from its class name: + `AnalyticsApiProvider` is `analytics_api_provider`. An explicit `provider_name` must already be + snake_case. +- Warehouse names are not affected: databases, schemas, tables, aliases, columns, tags and groups + follow the warehouse's own rules. +- Paths SQLBuild reads, such as models, tests, scenarios and source or schema YAML files, must be + valid UTF-8, including the folders above them. Otherwise compilation fails with `D016`, showing + each invalid byte as `\xNN`. `sqb scope` reports such a file and continues. diff --git a/website/src/content/docs/docs/concepts/rules.mdx b/website/src/content/docs/docs/concepts/rules.mdx index a1e0eb1845..59dc5f81e6 100644 --- a/website/src/content/docs/docs/concepts/rules.mdx +++ b/website/src/content/docs/docs/concepts/rules.mdx @@ -188,20 +188,11 @@ built-in Rules that did not run. `L007` never changes an exit code, including fo `sqb build`: it reports a gap in SQLBuild's dialect support, not a problem in your SQL. A syntax error in a supported dialect still fails the Rules that read it. -## One definition of project validity +## When Rules run -`sqb compile` is authoritative. SQLBuild evaluates a project in this order: - -```text -mandatory compiler correctness -→ selected native built-in Rules -→ selected custom Python Rules -→ artifact completion -``` - -Build and planning commands use the same compiler path. Configured findings stop execution before a -warehouse connection is opened. Compile artifacts are not completed from a project that failed its -Rules. +Rules run in every `sqb compile`, `sqb plan` and `sqb build`, after SQLBuild's own checks pass. If a +Rule fails, nothing runs in the warehouse. See +[Execution and caching](/docs/concepts/rules/execution-and-caching). ## Rules compared with tests, audits, and contracts @@ -231,24 +222,12 @@ comments, and mandatory compiler correctness cannot be excepted at all. See [Findings and exceptions](/docs/concepts/rules/findings-and-exceptions) for configuration examples. -## Performance is part of the design - -Model Rules are evaluated and cached per model. Editing one model invalidates affected subjects -rather than every invocation. Project Rules run once over the complete project view and are the -explicit choice for genuinely project-wide requirements, with broader invalidation. SQLBuild tracks -the inputs each Rule depends on so cached findings are reused only while those inputs remain valid. - -The default SQL Rules lint every compiled model, so a cold compile without a cache costs more than -the compiler alone: about 14% more wall time and 28% more CPU on a generated 3,000-model project. -Warm and one-model edit compiles reuse cached findings and cost the same. Set `select = []` to -measure or run the compiler without them. - -Required CI exercises custom Rule packs over generated projects, checking compilation time, -memory, and cache invalidation. Separate 20- and 100-Rule profiles measure the cost of custom -validation at 5,000-model scale. +## Performance -See [Benchmarks](/docs/benchmarks#compiler-guards-in-ci) for the benchmark profiles and reproduction commands, -and [Execution and caching](/docs/concepts/rules/execution-and-caching) for the Rules lifecycle. +Rules are cached per model, so editing one model re-checks only what it affects. The default SQL +Rules add about 14% to a cold compile and nothing to a warm one. See +[Benchmarks](/docs/benchmarks#compiler-guards-in-ci) and +[Execution and caching](/docs/concepts/rules/execution-and-caching). ## Continue diff --git a/website/src/content/docs/docs/concepts/rules/configuration-and-selection.mdx b/website/src/content/docs/docs/concepts/rules/configuration-and-selection.mdx index 883f56f474..0f30c8173e 100644 --- a/website/src/content/docs/docs/concepts/rules/configuration-and-selection.mdx +++ b/website/src/content/docs/docs/concepts/rules/configuration-and-selection.mdx @@ -66,7 +66,7 @@ allowed_tests = ["customers__reviewed_empty_input"] `SQBRTEST203` (`empty-input-only-test`) accepts `allowed_tests`, a list of SQL test names that are reviewed exceptions. Listed tests are not flagged and still count toward `SQBRTEST202`'s `min_tests_per_model`. Entries that name no existing test are reported as stale. See -[Empty-input tests](/docs/concepts/testing#empty-input-tests). +[Empty-input tests](/docs/concepts/testing-reference#empty-input-tests). `SQBRGRAPH103` and `SQBRPROJECT107` accept `published_layers`, and `SQBRPROJECT107` accepts `allowed_qualifiers`, a list of `{ qualifier, reason }` or `{ paths, reason }` tables with an @@ -137,19 +137,19 @@ final AS ( SELECT event_id, order_id, status FROM final ``` -Each `SELECT` is checked on its own: its select list, `FROM`, `JOIN ... ON`, `WHERE`, `GROUP BY`, -`HAVING`, `QUALIFY` and `ORDER BY`. A nested `SELECT`, subquery, CTE body or set-operation branch is -a separate scope. Calls are grouped by function and by the column reference as written, with names -compared the way the warehouse resolves them: on Snowflake unquoted names fold to upper case and -quoted names compare exactly, so `"VALUE"` matches `value` but `"Value"` does not; on DuckDB, -BigQuery and Databricks names compare case-insensitively whether or not they are quoted. References -are not resolved, so `value` and `e.value`, or `events.value` and -`e.value`, are different groups. The argument may be wrapped in parentheses, `LOWER`, `UPPER`, -`TRIM` without trim characters, `CAST`, `TRY_CAST` or a single-argument `COALESCE`; a wrapper with -more arguments, such as `COALESCE(e.value, e.backup)`, is not a plain column and is not counted. One -finding is reported at the second call and names the count with the column as written, for example -`TRY_PARSE_JSON(e.value) is parsed 3 times in one SELECT`. Different parse functions, such as -`PARSE_JSON` and `TRY_PARSE_JSON`, are counted separately. There is no automatic fix. +The finding is reported at the second call, for example +`TRY_PARSE_JSON(e.value) is parsed 3 times in one SELECT`. There is no automatic fix. + +Details: + +- Each `SELECT` is checked on its own. A subquery, CTE body or set-operation branch is a separate + `SELECT`. +- Calls count as the same when they use the same function on the same column as written: `value` + and `e.value` are different. Names compare as the warehouse compares them: on Snowflake, + unquoted names fold to upper case; on DuckDB, BigQuery and Databricks, case never matters. +- The column may be wrapped in parentheses, `LOWER`, `UPPER`, `TRIM`, `CAST`, `TRY_CAST` or a + one-argument `COALESCE`. `COALESCE(e.value, e.backup)` is not a plain column and isn't counted. +- `PARSE_JSON` and `TRY_PARSE_JSON` are counted separately. Known limitations: diff --git a/website/src/content/docs/docs/concepts/rules/execution-and-caching.mdx b/website/src/content/docs/docs/concepts/rules/execution-and-caching.mdx index 3c9f1d8983..a875667f4e 100644 --- a/website/src/content/docs/docs/concepts/rules/execution-and-caching.mdx +++ b/website/src/content/docs/docs/concepts/rules/execution-and-caching.mdx @@ -13,10 +13,7 @@ SQLBuild evaluates a project in this order: Native built-ins always run before custom Rules. Structured compile output reports built-in and custom durations separately, together with cache hits and misses. -## Authoritative enforcement - -`sqb compile` runs the complete configured Rules set. Build and execution commands use the same -compiler path and reject Rule findings before opening a warehouse connection. +## What a selection checks `sqb rules run` is intentionally focused. Use it to inspect one exact code or family, but run the normal compiler before treating the complete project as valid. diff --git a/website/src/content/docs/docs/concepts/rules/findings-and-exceptions.mdx b/website/src/content/docs/docs/concepts/rules/findings-and-exceptions.mdx index d28e8f89f6..020a5e28bd 100644 --- a/website/src/content/docs/docs/concepts/rules/findings-and-exceptions.mdx +++ b/website/src/content/docs/docs/concepts/rules/findings-and-exceptions.mdx @@ -120,14 +120,6 @@ error[SQBRSUPPRESSION000]: Inline Rule suppressions are not supported; declare t Copy the entry into `sqlbuild_project.toml`, adjust the reason for a reviewer, and delete the comment. -## Removed settings - -`[rules] allow_exceptions` was removed. Configured exceptions are always allowed because the project -file is the review gate, and inline suppressions no longer exist. A project that still sets -`allow_exceptions` fails configuration loading with an error that explains the removal: delete the -line, keep reviewed deviations as `[[rules.rule_exceptions]]` entries, and use `[rules] ignore` to -turn a Rule off for the whole project. - ## Mandatory correctness Mandatory compiler correctness is not configurable and cannot be suppressed. A project must first diff --git a/website/src/content/docs/docs/concepts/scenarios.mdx b/website/src/content/docs/docs/concepts/scenarios.mdx index 828b5835d8..6a96616d7d 100644 --- a/website/src/content/docs/docs/concepts/scenarios.mdx +++ b/website/src/content/docs/docs/concepts/scenarios.mdx @@ -22,7 +22,7 @@ Cost is a second reason to move a check into a scenario. A unit test inlines eve chain into one SQL statement, so the warehouse compiles and optimises the whole chain at once, and that cost can grow much faster than the query's size. A scenario materialises each model step by step, so each statement is compiled separately, as in a real build. SQLBuild limits how many real -models one unit test may execute (see [Cost guards](/docs/concepts/testing#cost-guards)); when a test +models one unit test may execute (see [Cost guards](/docs/concepts/testing-reference#cost-guards)); when a test reaches that limit because it checks behaviour across several models, it may belong here. ## Authoring scenarios diff --git a/website/src/content/docs/docs/concepts/semantic-compilation.mdx b/website/src/content/docs/docs/concepts/semantic-compilation.mdx index 8095d04aff..c6ea842771 100644 --- a/website/src/content/docs/docs/concepts/semantic-compilation.mdx +++ b/website/src/content/docs/docs/concepts/semantic-compilation.mdx @@ -1,5 +1,5 @@ --- -title: "Semantic Compilation" +title: "What sqb compile checks" description: "Offline reference, semantic and type checks that sqb compile runs before any warehouse work." --- diff --git a/website/src/content/docs/docs/concepts/sources.mdx b/website/src/content/docs/docs/concepts/sources.mdx index bc0c640b8e..7c143b2ac4 100644 --- a/website/src/content/docs/docs/concepts/sources.mdx +++ b/website/src/content/docs/docs/concepts/sources.mdx @@ -241,15 +241,10 @@ Accepts positive durations: `15m` (minutes), `2h` (hours), `1d` (days). If the c ### Auto-observation -Sources without an explicit `freshness:` block are auto-observed using the `adapter` strategy if: - -- The source has a physical table (not an expression source) -- The source is not managed -- The adapter supports table freshness metadata - -This means most unmanaged table sources get freshness tracking automatically on adapters that support it, with no configuration needed. - -Use [`sqb freshness`](/docs/cli/freshness) to observe source freshness on demand without triggering a build. See [Source freshness](/docs/concepts/planning/source-freshness) for how freshness feeds into change-aware builds. +A table source without a `freshness:` block is observed with the `adapter` strategy when the adapter +supports it, unless it is managed. See [Source freshness](/docs/concepts/planning/source-freshness) +for how observations feed into plans, and [`sqb freshness`](/docs/cli/freshness) to observe sources +without building. ### Freshness config reference diff --git a/website/src/content/docs/docs/concepts/testing-reference.mdx b/website/src/content/docs/docs/concepts/testing-reference.mdx new file mode 100644 index 0000000000..bc243cda2d --- /dev/null +++ b/website/src/content/docs/docs/concepts/testing-reference.mdx @@ -0,0 +1,434 @@ +--- +title: "Testing reference" +description: "Test modes, parameterized cases, cursor windows, cost guards and time limits for SQL unit tests." +--- + +This page lists the less common unit test features. Start with [Unit tests](/docs/concepts/testing) +for writing and running a test. + +## Test modes + +Model tests mock inputs and compare model output. Three more modes test reusable logic directly, +without a model chain: + +| Mode | Actual CTE | Expected CTE | Also allowed | +|------|-----------|--------------|--------------| +| `model` | none: the model chain | `__expected__` | `__source__`, `__ref__`, `__seed__`, `__assert__`, `__macro__` | +| `macro` | `__macro_actual__` | `__macro_expected__` | Helper CTEs, `@py.name()` and `@sql.name()` calls in actual, CTE-list `@sql.name()` calls in the `WITH` list | +| `udf` | `__udf_actual__` | `__udf_expected__` | Helper CTEs, `__udf()` calls in actual | +| `table_fn` | `__table_fn_actual__` | `__table_fn_expected__` | Helper CTEs, `__table_fn()` calls in actual | + +SQLBuild infers the mode from the actual CTE, so `TEST (mode ...)` is optional. An explicit mode +that contradicts the CTEs is a compile error, and so is a prefix from another mode, such as +`__source__` in a macro test. + +Only the actual CTE calls the macro, UDF or table function under test. Helpers don't call it +either, so a helper of a macro test must not call `__udf()` or `__table_fn()`. Expected CTEs must +not call macros, UDFs or table functions, so the expected data stays independent and easy to read. + +### Macro tests + +Call the macro in `__macro_actual__` and give the result in `__macro_expected__`: + +```sql +TEST (mode macro, name "calculates_line_total_cents"); + +WITH +input_values AS ( + SELECT 950 AS price_cents, 3 AS quantity +), +__macro_actual__ AS ( + SELECT @py.line_total_cents("price_cents", "quantity") AS line_total_cents + FROM input_values +), +__macro_expected__ AS ( + SELECT 2850 AS line_total_cents +) +``` + +Macros expand at compile time. During `sqb build`, a macro test runs before any model that uses the +macro. + +[SQL macros](/docs/concepts/macros/sql-macros) are tested the same way. Call an expression macro in +`__macro_actual__`: + +```sql +TEST (mode macro, name "cents_to_dollars_rounds"); + +WITH +input_values AS (SELECT CAST(1250 AS BIGINT) AS amount_cents), +__macro_actual__ AS ( + SELECT @sql.cents_to_dollars(amount_cents) AS dollars FROM input_values +), +__macro_expected__ AS (SELECT CAST(12.5 AS DECIMAL(18, 2)) AS dollars) +``` + +Call a column-list macro as an item of the `__macro_actual__` select list, and pass a helper CTE as +each `RELATION` argument: + +```sql +TEST (mode macro, name "audit_columns_flag_updates"); + +WITH +input_rows AS ( + SELECT TIMESTAMP '2026-01-01' AS created_at, TIMESTAMP '2026-01-02' AS updated_at +), +__macro_actual__ AS (SELECT @sql.audit_columns(input_rows) FROM input_rows), +__macro_expected__ AS ( + SELECT TIMESTAMP '2026-01-01' AS created_at, TIMESTAMP '2026-01-02' AS updated_at, + TRUE AS was_updated +) +``` + +Call a CTE-list macro in the test's `WITH` list, and select from one of its CTEs in +`__macro_actual__`: + +```sql +TEST (mode macro, name "clean_orders_keeps_latest_live"); + +WITH +input_orders AS ( + SELECT 1 AS order_id, TIMESTAMP '2026-01-01' AS updated_at, FALSE AS is_deleted + UNION ALL SELECT 1, TIMESTAMP '2026-01-02', FALSE + UNION ALL SELECT 2, TIMESTAMP '2026-01-01', TRUE +), +@sql.clean_orders(input_orders) +__macro_actual__ AS (SELECT order_id, updated_at FROM latest), +__macro_expected__ AS (SELECT 1 AS order_id, TIMESTAMP '2026-01-02' AS updated_at) +``` + +This is the only place outside `__macro_actual__` where a macro test may call a macro. The call may +pass `__cte := name`, as in a model, to call the macro more than once or to rename its CTEs. Other +modes can't call a CTE-list macro in their `WITH` list. + +### UDF tests + +Call the UDF in `__udf_actual__` and give the result in `__udf_expected__`: + +```sql +TEST (name "detects_completed_orders"); + +WITH +input_values AS ( + SELECT 'completed' AS order_status + UNION ALL + SELECT 'pending' AS order_status +), +__udf_actual__ AS ( + SELECT + order_status, + __udf("udf__is_completed_order")(order_status) AS is_completed_order + FROM input_values +), +__udf_expected__ AS ( + SELECT 'completed' AS order_status, TRUE AS is_completed_order + UNION ALL + SELECT 'pending' AS order_status, FALSE AS is_completed_order +) +``` + +During `sqb build`, a UDF test runs after the function is created and before any model that uses +it. + +### Functions in `sqb test` + +`sqb test` tests the current version of your functions without a build. Before the tests run, +SQLBuild creates a temporary copy of every project UDF and table function the selected tests call, +and of the functions those call. The tests call the copies, so an edited function is tested without +`sqb build`, and the built function in the target schema is never created, replaced or dropped. The +copies disappear when the test connections close. + +| Adapter | Temporary copy | +|---|---| +| DuckDB, MotherDuck | `CREATE TEMP MACRO`, called as `temp.main.`. Python UDFs are registered on each test connection | +| PostgreSQL | SQL function in the session's `pg_temp` schema, called as `pg_temp.` | +| Snowflake | `CREATE TEMPORARY FUNCTION __sqb_test__` in the session's current schema, for SQL and Python UDFs and table functions | +| Databricks | `CREATE TEMPORARY FUNCTION __sqb_test__` for SQL functions | +| BigQuery, SQL Server, ClickHouse | No copy; tests call the built function (ClickHouse functions are server-wide) | +| SQLite | Not applicable; SQLite has no SQL functions | + +When a copy can't be made, because the adapter doesn't support it (a Databricks Python UDF, for +example) or the warehouse refuses (the role lacks permission, or the Snowflake session has no +current schema), the test calls the built function. If that function isn't built either, the test +fails with an error naming the function and the reason, and asks you to run `sqb build` first. A +function whose callee has no copy calls the built functions too, so a test never mixes copies and +built functions. + +Every test connection gets the same copies as the first one. If a later connection can't create a +copy, `sqb test` stops with `T006`, naming the function and the connection. + +On PostgreSQL, the copies live in one server session. Run `sqb test` for projects with functions +through a direct connection or a session-mode pool, not a transaction-mode connection pooler. + +### Table function tests + +Call the table function in `__table_fn_actual__` and give its rows in `__table_fn_expected__`: + +```sql +TEST (name "returns_customer_orders"); + +WITH +__table_fn_actual__ AS ( + SELECT order_id, order_status, is_completed_order + FROM __table_fn("table_fn__customer_orders")(1) +), +__table_fn_expected__ AS ( + SELECT 1 AS order_id, 'completed' AS order_status, TRUE AS is_completed_order + UNION ALL + SELECT 2 AS order_id, 'completed' AS order_status, TRUE AS is_completed_order +) +``` + +Table function tests run after the function is created. Models can't depend on table functions, so +these tests check the function on its own. + +## Repeating test logic + +| Pattern | Use when | Results | +|---------|----------|---------| +| Several `TEST` blocks | Cases need different SQL | One per block | +| A `VALUES` case table or macro | One comparison over all cases is enough | One for the table | +| Parameters and cases | Cases share a template but each needs its own result | One per named case | + +### Native independent cases + +Declare typed parameters and ordered, named cases in the header: + +```sql +TEST ( + name "order_status_maps_source_states", + parameters ( + source_status string, + expected_status string, + ), + cases ( + completed (source_status "completed", expected_status "completed"), + cancelled (source_status "cancelled", expected_status "excluded"), + pending (source_status "pending", expected_status "open"), + ), +); + +WITH +__source__raw__orders AS ( + SELECT 1 AS id, @param("source_status") AS status +), +__expected__stg_orders AS ( + SELECT 1 AS id, @param("expected_status") AS status +) +``` + +Each case reports on its own, in the order written: `order status: maps source states [completed]`, +`[cancelled]` and `[pending]`. Selecting `stg_orders` runs every case. Run one case with `--case`: + +```bash +sqb test --select stg_orders --case cancelled +``` + +An unknown case name lists the available ones. + +Parameter types are `string`, `integer`, `boolean`, `float` and exact `decimal`. Quote decimal +values, such as `tax_rate "0.2000"`, so they never pass through binary floating point. Booleans and +integers stay distinct. Raw SQL and collection parameters are rejected. + +A nullable parameter uses the expanded form, and works where SQL can infer the null's type: + +```sql +parameters ( + cancellation_reason (type string, nullable true), +), +cases ( + completed (cancellation_reason null), +) +``` + +Every case gives exactly the declared parameters, and every parameter appears as `@param("name")` +in the test. Missing, extra, incompatible, undeclared and unused values are rejected before +anything runs. + +Parameters are rendered first, as typed literals for the active adapter, then `@const` and `@enum` +references, then macros. A parameter can therefore be a macro argument, as in +`@py.order_fixture(@param("source_status"))`. Parameters work in every mode and keep every expected +model of a multi-model test. + +Text and JSON output show the test, the case and its typed values. Compile JSON, manifests, DAG +checks and compiled SQL record each case's source, block and fingerprint. Changing one case changes +only that case's fingerprint. + +Scenarios can't be parameterized: keep one scenario per coherent business world. + +### Aggregate SQL case tables + +When cases don't need their own results, a `VALUES` table is shorter: + +```sql +TEST(); + +WITH cases(case_name, source_status, expected_status) AS ( + VALUES + ('completed', 'completed', 'completed'), + ('cancelled', 'cancelled', 'excluded') +), +__expected__status_mapping AS ( + SELECT case_name, expected_status FROM cases +) +``` + +This gives one test result. Quoted strings in `VALUES` rows may contain commas, brackets and +parentheses; escape embedded quotes as usual in SQL. + +## Cursor windows in tests + +Models that call `__cursor_start()` and `__cursor_end()` run in a test over one wide window, +never the model's own `cursor_start` and `cursor_end`: + +| Cursor type | Window | +|-------------|--------| +| Timestamp | `1900-01-01 00:00:00` to `2999-12-31 00:00:00` | +| Integer | `-1000000000000000` to `1000000000000000` | + +The bounds render as the same typed literals as in a real run, so date arithmetic such as +`DATEADD('day', -7, __cursor_start()::DATE)` works and mocked rows aren't filtered out. +Microbatch models run once over the whole window. + +To test windowing itself, set the window in the header. Either key is optional; `cursor_start` is +inclusive and `cursor_end` exclusive: + +```sql +TEST (name "daily_orders_skips_rows_outside_window", cursor_start "2026-02-01", cursor_end "2026-02-03"); +``` + +The window applies to every model in the test that uses the cursor functions, and is checked +against each model's `cursor_type` and `cursor_grain`. Invalid, misaligned or inverted bounds, and a +window on a test whose models don't use the cursor functions, are compile errors. The compiled test +SQL shows the rendered window. + +## Empty-input tests + +A test whose mocks are all empty (`WHERE FALSE`, `WHERE 1 = 0`, `LIMIT 0` or `__EMPTY_FIXTURE()`) +and whose only checks are an empty `__expected__` CTE or a bare `SELECT ... FROM __ref("")` +assertion can't fail for a model whose rows come from its inputs. Rule `SQBRTEST203` rejects it, +and `SQBRTEST202` doesn't count it toward `min_tests_per_model`. Mock representative rows and +assert concrete output instead. + +An empty-input test is fine when it asserts concrete output, such as a global aggregate that must +return one zero row: + +```sql +TEST (name "order_summary__empty_inputs_return_zero_row"); + +WITH +__source__raw__orders AS ( + SELECT CAST(NULL AS INTEGER) AS id, CAST(NULL AS INTEGER) AS amount WHERE FALSE +), +__expected__order_summary AS ( + SELECT 0 AS order_count, 0 AS total_amount +) +``` + +Assertions that filter for invalid rows, compare with expected rows, aggregate or limit their result +are not bare existence checks. List reviewed exceptions by test name in the project file; see +[Rule options](/docs/concepts/rules/configuration-and-selection#rule-options). + +## Cost guards + +A unit test runs the real SQL of every unmocked model between its mocks and the model under test as +one warehouse statement. The warehouse compiles and optimises that whole chain at once, and the cost +can grow much faster than the query's size. SQLBuild bounds each test in two ways. + +### Real-model limit + +`[settings] unit_test_max_real_models` (default `10`) caps how many real models one test may run, +counting the model under test. It can also be set in `sqlbuild_local.toml`. A test over the limit +fails compilation, in `sqb test` and in `sqb build` when it runs tests, before any warehouse +connection opens: + +```text +error[S000]: Invalid SQL test fixtures: +- SQL unit test 'customer_totals_rollup' executes 3 real models (limit 2). + Test file: tests/unit/customer_totals_rollup.sql + Model under test: customer_totals + +Real models executed, and the dependency path that pulls each one in: + customer_totals model under test + order_totals customer_totals -> order_totals + stg_orders customer_totals -> order_totals -> stg_orders + +Options: + 1. Mock the direct inputs of the model under test to cut the chain: + __ref__order_totals AS (SELECT ...) + 2. If this test checks behaviour across several models, consider a scenario + (tests/scenarios/). A unit test inlines every real model into one statement, + so the warehouse compiles and optimises the whole chain at once; a scenario + builds each model step by step, compiling each statement separately. + 3. Raise the limit in sqlbuild_project.toml (or sqlbuild_local.toml): + [settings] + unit_test_max_real_models = 3 +``` + +`sqb test` shows each test's count on its row, such as `3 real models`, so a growing chain is +visible early. Macro, UDF and table function tests run no models and aren't counted. + +### Time limits + +Each unit test has a time limit, `30s` by default. The first of these that is set applies: + +1. `--test-timeout` on `sqb test` or `sqb build` +2. `TEST(timeout "45s")` in the test file +3. `[targets..execution_limits] test_timeout` +4. `[settings] test_timeout`, in `sqlbuild_local.toml` or `sqlbuild_project.toml` +5. The `30s` default + +Each takes a positive fixed duration, such as `"45s"` or `"10m"`, or `"disabled"`. + +The clock starts at the test's first warehouse statement. Rendering, warehouse queueing where the +warehouse reports it, and cleanup after the comparison don't count. See +[Per-item time limits](/docs/cli/build#per-item-time-limits) for how each +adapter enforces the limit, and [Duration limit](/docs/cli/build#duration-limit) +for how each adapter cancels a running statement. + +At the limit, SQLBuild cancels the running statement, reports the test as `TIMEOUT`, and continues +with the other tests. The report gives the elapsed time, the limit, where it was set, the +cancellation outcome and the query IDs it saw. When the test runs real models beyond the model +under test, it also lists them, with the same options as the real-model limit: + +```text +SQL unit test 'order_scans_kept' timed out after 1.0s (limit 1s). + Limit set by: TEST() timeout in tests/unit/order_scans_kept.sql + Cancellation: confirmed; the running statement stopped + +This test inlines 1 real model beyond the model under test into one statement: + stg_orders + +Options: + 1. Mock the inlined models closest to the model under test to cut the chain, + for example __ref__stg_orders AS (SELECT ...). + 2. If this test checks behaviour across several models, consider a scenario + (tests/scenarios/). A unit test inlines every real model into one statement, + so the warehouse compiles and optimises the whole chain at once; a scenario + builds each model step by step, compiling each statement separately. + 3. To allow more time, set TEST(timeout "3s") on this test, [settings] test_timeout = "3s" + in sqlbuild_project.toml (or sqlbuild_local.toml), or --test-timeout for one run. +``` + +A test that runs only the model under test reports the timeout, the limit, where it was set and how +to change it, without the scenario advice. + +Scenarios have their own limit, `scenario_timeout`; see +[Scenario timeouts](/docs/concepts/project-configuration#unit-test-and-scenario-limits). + +## Skipping SQL analysis + +Compile checks test fixtures and assertions like model SQL. For SQL the parser can't read, add +`sql_analysis false` to the `TEST` header; the test still runs. Tests of a model with +`sql_analysis false`, and every test in a `--no-sql-analysis` run, aren't analysed. With +`[settings] require_sql_analysis = true`, the opt-out is accepted only on test SQL that can't be +parsed; see [Requiring SQL analysis](/docs/concepts/models/configuration#requiring-sql-analysis). + +## Details + +- SQL tests need SQLBuild's SQL analysis for the adapter's dialect. On an adapter without it, such + as a custom adapter without `sql_analysis_dialect_name`, planning SQL tests fails with a message + that says SQL tests are unavailable for that dialect. +- On DuckDB, MotherDuck, PostgreSQL and Snowflake, a test evaluates each compared model's full + output once. If that output fails, for example because a fixture value can't be converted in a + column the expected rows leave out, the test errors with the warehouse's message. diff --git a/website/src/content/docs/docs/concepts/testing.mdx b/website/src/content/docs/docs/concepts/testing.mdx index 57afdd22fa..221b09aa98 100644 --- a/website/src/content/docs/docs/concepts/testing.mdx +++ b/website/src/content/docs/docs/concepts/testing.mdx @@ -1,6 +1,6 @@ --- -title: "Testing" -description: "SQL unit tests and multi-model tests with macro support, assertions, and model chaining." +title: "Unit tests" +description: "Check a model's logic on rows you write, by mocking its inputs and comparing its output." --- A unit test checks a model's logic on rows you write. You mock the model's inputs with CTEs, @@ -36,11 +36,6 @@ Tests can chain several models, mock models, seeds, macros and table functions, that must hold for every row, and test macros, UDFs and table functions directly. For end-to-end tests over physical warehouse relations, see [Scenarios](/docs/concepts/scenarios). -SQL tests are planned with SQLBuild's native parser for the adapter's SQL analysis dialect. On an -adapter whose dialect has no native parser, including a custom adapter without -`sql_analysis_dialect_name`, a project with SQL tests fails to plan them, with a message that says -SQL tests are unavailable for that dialect. - ## How tests work The test above: @@ -54,9 +49,9 @@ The test above: Every test starts with a `TEST()` header. All of its fields are optional: `TEST();` is a complete header, and a test without a `name` is named after its file. Fields name the test, set its -[mode](#test-modes) or [cursor window](#cursor-windows-in-tests), declare -[parameters](#native-independent-cases), set a [time limit](#time-limits), or -[skip SQL analysis](#skipping-sql-analysis). +[mode](/docs/concepts/testing-reference#test-modes) or [cursor window](/docs/concepts/testing-reference#cursor-windows-in-tests), declare +[parameters](/docs/concepts/testing-reference#native-independent-cases), set a [time limit](/docs/concepts/testing-reference#time-limits), or +[skip SQL analysis](/docs/concepts/testing-reference#skipping-sql-analysis). A test ends after its last CTE. A trailing `SELECT 1` is accepted; nothing else may follow the last CTE. @@ -133,7 +128,7 @@ __expected__stg_orders AS ( Test names are unique across the project: a test can't share a name with another test, a scenario, a model, a source, a seed, a function or a Python node. Name a test after the behaviour it checks, such as `orders_status_rules`, rather than after the model. The cases of one -[parameterized test](#native-independent-cases) share its name. +[parameterized test](/docs/concepts/testing-reference#native-independent-cases) share its name. ## CTE conventions @@ -285,7 +280,7 @@ The plan lists the mocked sources, refs and seeds, the real models in execution models, and any unsatisfied leaf dependencies. A mocked ref is called out, because its real SQL won't run. Inspection exits non-zero when the plan has unresolved dependencies. -Each real model adds to the cost of the test; see [Cost guards](#cost-guards). +Each real model adds to the cost of the test; see [Cost guards](/docs/concepts/testing-reference#cost-guards). ## Mocking refs and seeds @@ -385,7 +380,7 @@ __expected__customer_order_totals AS ( The fixture name must match a declared table function. Fixtures can read helper CTEs, and an assertion can read a table-function fixture directly. To test the table function itself, use a -[table function test](#table-function-tests). +[table function test](/docs/concepts/testing-reference#table-function-tests). ## Multiple expected models @@ -512,329 +507,6 @@ An assertion must read a model the test runs. A test whose checks read only mock `__ref("stg_orders")` in a test that also defines `__ref__stg_orders`, has nothing to run, and compile rejects it with `P013`. Mock the model's inputs instead, as above, or remove the test. -## Cursor windows in tests - -Models that call `__cursor_start()` and `__cursor_end()` run in a test over one wide window, -never the model's own `cursor_start` and `cursor_end`: - -| Cursor type | Window | -|-------------|--------| -| Timestamp | `1900-01-01 00:00:00` to `2999-12-31 00:00:00` | -| Integer | `-1000000000000000` to `1000000000000000` | - -The bounds render as the same typed literals as in a real run, so date arithmetic such as -`DATEADD('day', -7, __cursor_start()::DATE)` works and mocked rows aren't filtered out. -Microbatch models run once over the whole window. - -To test windowing itself, set the window in the header. Either key is optional; `cursor_start` is -inclusive and `cursor_end` exclusive: - -```sql -TEST (name "daily_orders_skips_rows_outside_window", cursor_start "2026-02-01", cursor_end "2026-02-03"); -``` - -The window applies to every model in the test that uses the cursor functions, and is checked -against each model's `cursor_type` and `cursor_grain`. Invalid, misaligned or inverted bounds, and a -window on a test whose models don't use the cursor functions, are compile errors. The compiled test -SQL shows the rendered window. - -## Empty-input tests - -A test whose mocks are all empty (`WHERE FALSE`, `WHERE 1 = 0`, `LIMIT 0` or `__EMPTY_FIXTURE()`) -and whose only checks are an empty `__expected__` CTE or a bare `SELECT ... FROM __ref("")` -assertion can't fail for a model whose rows come from its inputs. Rule `SQBRTEST203` rejects it, -and `SQBRTEST202` doesn't count it toward `min_tests_per_model`. Mock representative rows and -assert concrete output instead. - -An empty-input test is fine when it asserts concrete output, such as a global aggregate that must -return one zero row: - -```sql -TEST (name "order_summary__empty_inputs_return_zero_row"); - -WITH -__source__raw__orders AS ( - SELECT CAST(NULL AS INTEGER) AS id, CAST(NULL AS INTEGER) AS amount WHERE FALSE -), -__expected__order_summary AS ( - SELECT 0 AS order_count, 0 AS total_amount -) -``` - -Assertions that filter for invalid rows, compare with expected rows, aggregate or limit their result -are not bare existence checks. List reviewed exceptions by test name in the project file; see -[Rule options](/docs/concepts/rules/configuration-and-selection#rule-options). - -## Test modes - -Model tests mock inputs and compare model output. Three more modes test reusable logic directly, -without a model chain: - -| Mode | Actual CTE | Expected CTE | Also allowed | -|------|-----------|--------------|--------------| -| `model` | none: the model chain | `__expected__` | `__source__`, `__ref__`, `__seed__`, `__assert__`, `__macro__` | -| `macro` | `__macro_actual__` | `__macro_expected__` | Helper CTEs, `@py.name()` and `@sql.name()` calls in actual, CTE-list `@sql.name()` calls in the `WITH` list | -| `udf` | `__udf_actual__` | `__udf_expected__` | Helper CTEs, `__udf()` calls in actual | -| `table_fn` | `__table_fn_actual__` | `__table_fn_expected__` | Helper CTEs, `__table_fn()` calls in actual | - -SQLBuild infers the mode from the actual CTE, so `TEST (mode ...)` is optional. An explicit mode -that contradicts the CTEs is a compile error, and so is a prefix from another mode, such as -`__source__` in a macro test. - -Only the actual CTE calls the macro, UDF or table function under test. Helpers don't call it -either, so a helper of a macro test must not call `__udf()` or `__table_fn()`. Expected CTEs must -not call macros, UDFs or table functions, so the expected data stays independent and easy to read. - -### Macro tests - -Call the macro in `__macro_actual__` and give the result in `__macro_expected__`: - -```sql -TEST (mode macro, name "calculates_line_total_cents"); - -WITH -input_values AS ( - SELECT 950 AS price_cents, 3 AS quantity -), -__macro_actual__ AS ( - SELECT @py.line_total_cents("price_cents", "quantity") AS line_total_cents - FROM input_values -), -__macro_expected__ AS ( - SELECT 2850 AS line_total_cents -) -``` - -Macros expand at compile time. During `sqb build`, a macro test runs before any model that uses the -macro. - -[SQL macros](/docs/concepts/macros/sql-macros) are tested the same way. Call an expression macro in -`__macro_actual__`: - -```sql -TEST (mode macro, name "cents_to_dollars_rounds"); - -WITH -input_values AS (SELECT CAST(1250 AS BIGINT) AS amount_cents), -__macro_actual__ AS ( - SELECT @sql.cents_to_dollars(amount_cents) AS dollars FROM input_values -), -__macro_expected__ AS (SELECT CAST(12.5 AS DECIMAL(18, 2)) AS dollars) -``` - -Call a column-list macro as an item of the `__macro_actual__` select list, and pass a helper CTE as -each `RELATION` argument: - -```sql -TEST (mode macro, name "audit_columns_flag_updates"); - -WITH -input_rows AS ( - SELECT TIMESTAMP '2026-01-01' AS created_at, TIMESTAMP '2026-01-02' AS updated_at -), -__macro_actual__ AS (SELECT @sql.audit_columns(input_rows) FROM input_rows), -__macro_expected__ AS ( - SELECT TIMESTAMP '2026-01-01' AS created_at, TIMESTAMP '2026-01-02' AS updated_at, - TRUE AS was_updated -) -``` - -Call a CTE-list macro in the test's `WITH` list, and select from one of its CTEs in -`__macro_actual__`: - -```sql -TEST (mode macro, name "clean_orders_keeps_latest_live"); - -WITH -input_orders AS ( - SELECT 1 AS order_id, TIMESTAMP '2026-01-01' AS updated_at, FALSE AS is_deleted - UNION ALL SELECT 1, TIMESTAMP '2026-01-02', FALSE - UNION ALL SELECT 2, TIMESTAMP '2026-01-01', TRUE -), -@sql.clean_orders(input_orders) -__macro_actual__ AS (SELECT order_id, updated_at FROM latest), -__macro_expected__ AS (SELECT 1 AS order_id, TIMESTAMP '2026-01-02' AS updated_at) -``` - -This is the only place outside `__macro_actual__` where a macro test may call a macro. The call may -pass `__cte := name`, as in a model, to call the macro more than once or to rename its CTEs. Other -modes can't call a CTE-list macro in their `WITH` list. - -### UDF tests - -Call the UDF in `__udf_actual__` and give the result in `__udf_expected__`: - -```sql -TEST (name "detects_completed_orders"); - -WITH -input_values AS ( - SELECT 'completed' AS order_status - UNION ALL - SELECT 'pending' AS order_status -), -__udf_actual__ AS ( - SELECT - order_status, - __udf("udf__is_completed_order")(order_status) AS is_completed_order - FROM input_values -), -__udf_expected__ AS ( - SELECT 'completed' AS order_status, TRUE AS is_completed_order - UNION ALL - SELECT 'pending' AS order_status, FALSE AS is_completed_order -) -``` - -During `sqb build`, a UDF test runs after the function is created and before any model that uses -it. - -### Functions in `sqb test` - -`sqb test` tests the current version of your functions without a build. Before the tests run, -SQLBuild creates a temporary copy of every project UDF and table function the selected tests call, -and of the functions those call. The tests call the copies, so an edited function is tested without -`sqb build`, and the built function in the target schema is never created, replaced or dropped. The -copies disappear when the test connections close. - -| Adapter | Temporary copy | -|---|---| -| DuckDB, MotherDuck | `CREATE TEMP MACRO`, called as `temp.main.`. Python UDFs are registered on each test connection | -| PostgreSQL | SQL function in the session's `pg_temp` schema, called as `pg_temp.` | -| Snowflake | `CREATE TEMPORARY FUNCTION __sqb_test__` in the session's current schema, for SQL and Python UDFs and table functions | -| Databricks | `CREATE TEMPORARY FUNCTION __sqb_test__` for SQL functions | -| BigQuery, SQL Server, ClickHouse | No copy; tests call the built function (ClickHouse functions are server-wide) | -| SQLite | Not applicable; SQLite has no SQL functions | - -When a copy can't be made, because the adapter doesn't support it (a Databricks Python UDF, for -example) or the warehouse refuses (the role lacks permission, or the Snowflake session has no -current schema), the test calls the built function. If that function isn't built either, the test -fails with an error naming the function and the reason, and asks you to run `sqb build` first. A -function whose callee has no copy calls the built functions too, so a test never mixes copies and -built functions. - -Every test connection gets the same copies as the first one. If a later connection can't create a -copy, `sqb test` stops with `T006`, naming the function and the connection. - -On PostgreSQL, the copies live in one server session. Run `sqb test` for projects with functions -through a direct connection or a session-mode pool, not a transaction-mode connection pooler. - -### Table function tests - -Call the table function in `__table_fn_actual__` and give its rows in `__table_fn_expected__`: - -```sql -TEST (name "returns_customer_orders"); - -WITH -__table_fn_actual__ AS ( - SELECT order_id, order_status, is_completed_order - FROM __table_fn("table_fn__customer_orders")(1) -), -__table_fn_expected__ AS ( - SELECT 1 AS order_id, 'completed' AS order_status, TRUE AS is_completed_order - UNION ALL - SELECT 2 AS order_id, 'completed' AS order_status, TRUE AS is_completed_order -) -``` - -Table function tests run after the function is created. Models can't depend on table functions, so -these tests check the function on its own. - -## Repeating test logic - -| Pattern | Use when | Results | -|---------|----------|---------| -| Several `TEST` blocks | Cases need different SQL | One per block | -| A `VALUES` case table or macro | One comparison over all cases is enough | One for the table | -| Parameters and cases | Cases share a template but each needs its own result | One per named case | - -### Native independent cases - -Declare typed parameters and ordered, named cases in the header: - -```sql -TEST ( - name "order_status_maps_source_states", - parameters ( - source_status string, - expected_status string, - ), - cases ( - completed (source_status "completed", expected_status "completed"), - cancelled (source_status "cancelled", expected_status "excluded"), - pending (source_status "pending", expected_status "open"), - ), -); - -WITH -__source__raw__orders AS ( - SELECT 1 AS id, @param("source_status") AS status -), -__expected__stg_orders AS ( - SELECT 1 AS id, @param("expected_status") AS status -) -``` - -Each case reports on its own, in the order written: `order status: maps source states [completed]`, -`[cancelled]` and `[pending]`. Selecting `stg_orders` runs every case. Run one case with `--case`: - -```bash -sqb test --select stg_orders --case cancelled -``` - -An unknown case name lists the available ones. - -Parameter types are `string`, `integer`, `boolean`, `float` and exact `decimal`. Quote decimal -values, such as `tax_rate "0.2000"`, so they never pass through binary floating point. Booleans and -integers stay distinct. Raw SQL and collection parameters are rejected. - -A nullable parameter uses the expanded form, and works where SQL can infer the null's type: - -```sql -parameters ( - cancellation_reason (type string, nullable true), -), -cases ( - completed (cancellation_reason null), -) -``` - -Every case gives exactly the declared parameters, and every parameter appears as `@param("name")` -in the test. Missing, extra, incompatible, undeclared and unused values are rejected before -anything runs. - -Parameters are rendered first, as typed literals for the active adapter, then `@const` and `@enum` -references, then macros. A parameter can therefore be a macro argument, as in -`@py.order_fixture(@param("source_status"))`. Parameters work in every mode and keep every expected -model of a multi-model test. - -Text and JSON output show the test, the case and its typed values. Compile JSON, manifests, DAG -checks and compiled SQL record each case's source, block and fingerprint. Changing one case changes -only that case's fingerprint. - -Scenarios can't be parameterized: keep one scenario per coherent business world. - -### Aggregate SQL case tables - -When cases don't need their own results, a `VALUES` table is shorter: - -```sql -TEST(); - -WITH cases(case_name, source_status, expected_status) AS ( - VALUES - ('completed', 'completed', 'completed'), - ('cancelled', 'cancelled', 'excluded') -), -__expected__status_mapping AS ( - SELECT case_name, expected_status FROM cases -) -``` - -This gives one test result. Quoted strings in `VALUES` rows may contain commas, brackets and -parentheses; escape embedded quotes as usual in SQL. - ## Running tests Tests run during `sqb build` in DAG order, before the model they test is built. Run them on their @@ -845,103 +517,7 @@ sqb test sqb test --select stg_orders ``` -## Cost guards - -A unit test runs the real SQL of every unmocked model between its mocks and the model under test as -one warehouse statement. The warehouse compiles and optimises that whole chain at once, and the cost -can grow much faster than the query's size. SQLBuild bounds each test in two ways. - -### Real-model limit - -`[settings] unit_test_max_real_models` (default `10`) caps how many real models one test may run, -counting the model under test. It can also be set in `sqlbuild_local.toml`. A test over the limit -fails compilation, in `sqb test` and in `sqb build` when it runs tests, before any warehouse -connection opens: - -```text -error[S000]: Invalid SQL test fixtures: -- SQL unit test 'customer_totals_rollup' executes 3 real models (limit 2). - Test file: tests/unit/customer_totals_rollup.sql - Model under test: customer_totals - -Real models executed, and the dependency path that pulls each one in: - customer_totals model under test - order_totals customer_totals -> order_totals - stg_orders customer_totals -> order_totals -> stg_orders - -Options: - 1. Mock the direct inputs of the model under test to cut the chain: - __ref__order_totals AS (SELECT ...) - 2. If this test checks behaviour across several models, consider a scenario - (tests/scenarios/). A unit test inlines every real model into one statement, - so the warehouse compiles and optimises the whole chain at once; a scenario - builds each model step by step, compiling each statement separately. - 3. Raise the limit in sqlbuild_project.toml (or sqlbuild_local.toml): - [settings] - unit_test_max_real_models = 3 -``` - -`sqb test` shows each test's count on its row, such as `3 real models`, so a growing chain is -visible early. Macro, UDF and table function tests run no models and aren't counted. - -### Time limits - -Each unit test has a time limit, `30s` by default. The first of these that is set applies: - -1. `--test-timeout` on `sqb test` or `sqb build` -2. `TEST(timeout "45s")` in the test file -3. `[targets..execution_limits] test_timeout` -4. `[settings] test_timeout`, in `sqlbuild_local.toml` or `sqlbuild_project.toml` -5. The `30s` default - -Each takes a positive fixed duration, such as `"45s"` or `"10m"`, or `"disabled"`. - -The clock starts at the test's first warehouse statement. Rendering, warehouse queueing where the -warehouse reports it, and cleanup after the comparison don't count. See -[Per-item time limits](/docs/concepts/project-configuration#per-item-time-limits) for how each -adapter enforces the limit, and [Duration limit](/docs/concepts/project-configuration#duration-limit) -for how each adapter cancels a running statement. - -At the limit, SQLBuild cancels the running statement, reports the test as `TIMEOUT`, and continues -with the other tests. The report gives the elapsed time, the limit, where it was set, the -cancellation outcome and the query IDs it saw. When the test runs real models beyond the model -under test, it also lists them, with the same options as the real-model limit: - -```text -SQL unit test 'order_scans_kept' timed out after 1.0s (limit 1s). - Limit set by: TEST() timeout in tests/unit/order_scans_kept.sql - Cancellation: confirmed; the running statement stopped - -This test inlines 1 real model beyond the model under test into one statement: - stg_orders - -Options: - 1. Mock the inlined models closest to the model under test to cut the chain, - for example __ref__stg_orders AS (SELECT ...). - 2. If this test checks behaviour across several models, consider a scenario - (tests/scenarios/). A unit test inlines every real model into one statement, - so the warehouse compiles and optimises the whole chain at once; a scenario - builds each model step by step, compiling each statement separately. - 3. To allow more time, set TEST(timeout "3s") on this test, [settings] test_timeout = "3s" - in sqlbuild_project.toml (or sqlbuild_local.toml), or --test-timeout for one run. -``` - -A test that runs only the model under test reports the timeout, the limit, where it was set and how -to change it, without the scenario advice. - -Scenarios have their own limit, `scenario_timeout`; see -[Scenario timeouts](/docs/concepts/project-configuration#unit-test-and-scenario-limits). - -## Skipping SQL analysis - -Compile checks test fixtures and assertions like model SQL. For SQL the parser can't read, add -`sql_analysis false` to the `TEST` header; the test still runs. Tests of a model with -`sql_analysis false`, and every test in a `--no-sql-analysis` run, aren't analysed. With -`[settings] require_sql_analysis = true`, the opt-out is accepted only on test SQL that can't be -parsed; see [Requiring SQL analysis](/docs/concepts/models/configuration#requiring-sql-analysis). - -## Details +## More test features -- On DuckDB, MotherDuck, PostgreSQL and Snowflake, a test evaluates each compared model's full - output once. If that output fails, for example because a fixture value can't be converted in a - column the expected rows leave out, the test errors with the warehouse's message. +[Testing reference](/docs/concepts/testing-reference) covers macro, UDF and table function tests, +parameterized cases, cursor windows, empty-input tests, cost guards and time limits. diff --git a/website/src/content/docs/docs/feature-comparison.mdx b/website/src/content/docs/docs/feature-comparison.mdx index e17b1606e5..71b4674b3d 100644 --- a/website/src/content/docs/docs/feature-comparison.mdx +++ b/website/src/content/docs/docs/feature-comparison.mdx @@ -35,9 +35,9 @@ SQLBuild, dbt, and SQLMesh are all SQL pipeline frameworks. They share common gr | Feature | SQLBuild | dbt | SQLMesh | |---------|----------|-----|---------| -| SQL analysis | Offline syntax, binding, type inference, semantic validation, and lineage (Polyglot) | dbt Core: none; dbt Fusion engine: compile-time (proprietary; built on Apache-2.0 dbt Core v2) | Compile-time (SQLGlot) | +| SQL analysis | Offline syntax, binding, type inference, semantic validation, and lineage | dbt Core: none; dbt Fusion: compile-time | Compile-time (SQLGlot) | | Focused compilation | Full graph integrity with deep analysis limited to selection and required upstream closure | Selected compilation | Selected planning | -| Column-level lineage | Compile-time, fast and rich modes | dbt Core: post-hoc via docs; dbt Fusion engine: compile-time | Compile-time | +| Column-level lineage | Compile-time, fast and rich modes | dbt Core: post-hoc via docs; dbt Fusion: compile-time | Compile-time | | Column contract validation | Compile-time inference plus runtime enforcement with `contract enforced` | YAML schema contracts at runtime | Schema contracts via plan | | Existing-schema contracts | Online read-only diff and safe repository generation | Codegen/packages | External schema tooling | | Rules and formatting | Compiler-integrated native and custom diagnostics, reasoned suppressions, and separate canonical formatting | External tools | Built-in formatter plus external linting | @@ -79,7 +79,7 @@ SQLBuild, dbt, and SQLMesh are all SQL pipeline frameworks. They share common gr | Feature | SQLBuild | dbt | SQLMesh | |---------|----------|-----|---------| | SQL models | `MODEL()` header with inline config | Jinja-templated SQL + YAML sidecar | `MODEL` DDL | -| Python models | Coming soon | Pandas, PySpark, Snowpark, BigFrames | Pandas, PySpark, Snowpark, BigFrames | +| Python models | Python nodes (tasks, assets, loaders, checks); no dataframe models | Pandas, PySpark, Snowpark, BigFrames | Pandas, PySpark, Snowpark, BigFrames | | Custom materializations | Python with full framework hooks | Jinja-based | Python-based custom model kinds | | Lifecycle hooks | Typed inline SQL, reusable parameterized SQL resources, and Python hooks with compile-time validation and `HookContext` | Jinja pre/post hooks | Python pre/post hooks | @@ -127,6 +127,4 @@ SQLBuild, dbt, and SQLMesh are all SQL pipeline frameworks. They share common gr | **dbt** | The most widely adopted SQL transformation framework with the largest adapter and community ecosystem. | | **SQLMesh** | State-managed pipelines with interval tracking and cross-dialect transpilation. | -## Not yet in SQLBuild - -- **Broader adapter support** - Redshift, Trino, Spark, Athena +See the [Roadmap](/docs/roadmap) for what comes next. diff --git a/website/src/content/docs/docs/index.mdx b/website/src/content/docs/docs/index.mdx index d3e35fcc80..7dece95f34 100644 --- a/website/src/content/docs/docs/index.mdx +++ b/website/src/content/docs/docs/index.mdx @@ -60,9 +60,8 @@ audit results and everything it printed. Agents and people query that history wi | Review | [`sqb review`](/docs/cli/review) | Builds a branch's changes in a temporary schema, tests them and compares them with a target | | Investigate | [`sqb recall`](/docs/cli/recall) | Queries the history of past runs, then live warehouse state | -SQLBuild works with DuckDB, MotherDuck, Snowflake, BigQuery, Databricks, PostgreSQL, SQL Server, -ClickHouse and SQLite. -DuckDB runs locally, so you can try everything without warehouse credentials. See +SQLBuild works with Snowflake, DuckDB, MotherDuck and PostgreSQL. BigQuery, Databricks, SQL Server, +ClickHouse and SQLite are in beta. DuckDB runs locally, so you can try everything without warehouse credentials. See [Adapters](/docs/concepts/adapters) for connection setup. ## Where to start diff --git a/website/src/content/docs/docs/integrations/dagster.mdx b/website/src/content/docs/docs/integrations/dagster.mdx index 34d319eced..c64e10b955 100644 --- a/website/src/content/docs/docs/integrations/dagster.mdx +++ b/website/src/content/docs/docs/integrations/dagster.mdx @@ -105,7 +105,7 @@ metadata to command-output and lifecycle records: Lifecycle events remain SQLBuild's immutable execution facts. Human stdout/stderr remains a correlated troubleshooting transcript; the integration does not parse terminal text to establish -resource status. See [Execution Observability](/docs/concepts/observability). +resource status. See [Observability](/docs/concepts/observability). ## Scenarios as checks diff --git a/website/src/content/docs/docs/project-setup.mdx b/website/src/content/docs/docs/project-setup.mdx new file mode 100644 index 0000000000..6198da05fc --- /dev/null +++ b/website/src/content/docs/docs/project-setup.mdx @@ -0,0 +1,86 @@ +--- +title: "Set up a project" +description: "Connect a SQLBuild project to your warehouse with dev and prod targets and a personal local file." +--- + +A project is configured by `sqlbuild_project.toml`, which you commit, and an optional +`sqlbuild_local.toml`, which each developer keeps to themselves. Start a project with +[`sqb init`](/docs/cli/init), or explore one with [`sqb playground`](/docs/cli/playground). + +```text +project/ + sqlbuild_project.toml # committed: shared targets and settings + sqlbuild_local.toml # gitignored: your own target, schema and credentials + models/ + sources/ + tests/ +``` + +## Connect to the warehouse + +A connection holds how to reach the warehouse. A target names where models are built: a database +and schema, such as `dev` or `prod`. Several targets can share one connection. + +```toml +name = "shop" +adapter = "snowflake" +default_target = "dev" + +[connections.warehouse] +account = "my_org-my_account" +user = "${ENV:SNOWFLAKE_USER}" +password = "${ENV:SNOWFLAKE_PASSWORD}" +warehouse = "TRANSFORM_WH" + +[targets.dev] +connection = "warehouse" +database = "ANALYTICS" +schema = "DEV" + +[targets.prod] +connection = "warehouse" +database = "ANALYTICS" +schema = "PROD" +``` + +`${ENV:NAME}` reads an environment variable, so passwords and tokens stay out of the file. Each +adapter page lists its connection fields; see [Adapters](/docs/concepts/adapters). + +Check the setup with [`sqb debug`](/docs/cli/debug), which validates the config and opens a +connection. Commands that don't need the warehouse, such as `sqb compile`, `sqb test --inspect` and +`sqb lineage`, work without one. + +## Choose a target + +Commands build into the target given by `--target`, then the `target` in `sqlbuild_local.toml`, then +`default_target`: + +```bash +sqb build # default_target, or your local target +sqb build --target prod +``` + +## Keep your own settings local + +Put what differs for you in `sqlbuild_local.toml` and add it to `.gitignore`. It merges over the +shared file, key by key, so it holds only what you change: + +```toml +# sqlbuild_local.toml +target = "dev" + +[targets.dev] +schema = "DEV_ALICE" + +[connections.warehouse] +user = "alice" +``` + +The local file can also switch `adapter`, for example to build on DuckDB while the team uses +Snowflake. + +## Next steps + +- [Models](/docs/concepts/models) - write your first model. +- [Configuration reference](/docs/concepts/project-configuration) - every project setting, including + defaults, path defaults, clone policies and execution limits. diff --git a/website/src/content/docs/docs/quickstart.mdx b/website/src/content/docs/docs/quickstart.mdx index 71541534ea..22a0721f82 100644 --- a/website/src/content/docs/docs/quickstart.mdx +++ b/website/src/content/docs/docs/quickstart.mdx @@ -21,11 +21,7 @@ cd waffle-shop This creates a self-contained waffle shop project with everything you need to explore SQLBuild. -The playground creates the shared `sqlbuild_project.toml`; that is sufficient for this -quickstart. In a team project, you can additionally create a gitignored -`sqlbuild_local.toml` to select your usual target and override your personal development -schema or credentials without changing the committed project config. See -[Project Configuration](/docs/concepts/project-configuration#sqlbuild_localtoml). +To connect your own warehouse later, see [Set up a project](/docs/project-setup). ## 2. Plan @@ -73,7 +69,7 @@ Seeds (1) Execute the full build: ```bash -sqb --no-color build +sqb build ``` SQLBuild materializes all models in DAG order. SQL unit tests run before their target models. Error-severity audits run against a staging table before promotion - if any fail, the production table is untouched: @@ -89,50 +85,14 @@ Execution sqb build (concurrency: 1) audit not_null (customer_id) PASS audit unique (customer_id) PASS audit not_null (email) PASS - 7/17 view stg_orders OK 0.04s - test test_daily_revenue_chain PASS - expect expected stg_orders PASS - expect expected stg_payments PASS - expect expected daily_revenue PASS - expect assertion daily_revenue_is_non_negative PASS - test test_stg_orders PASS - expect expected stg_orders PASS - expect assertion order_ids_are_not_null PASS - audit not_null (order_id) PASS - audit unique (order_id) PASS - audit not_null (customer_id) PASS - audit relationships (waffle_type_id) PASS - audit accepted_values (status) PASS - 8/17 custom daily_order_partitioned OK 0.21s - audit waffles_ordered_is_positive PASS 4/4 - audit not_null (order_date) PASS 4/4 - 9/17 table daily_revenue OK 0.07s - test test_daily_revenue_chain PASS - expect expected stg_orders PASS - expect expected stg_payments PASS - expect expected daily_revenue PASS - expect assertion daily_revenue_is_non_negative PASS - audit revenue_is_non_negative PASS - 11/17 table fact_orders OK 0.05s - test test_fact_orders PASS - expect expected fact_orders PASS - expect assertion line_totals_are_non_negative PASS - audit not_null (order_id) PASS - 12/17 table customer_status_snapshot (merge) OK 0.05s - audit not_null (customer_id) PASS - audit unique (customer_id) PASS + ... 13/17 table hourly_order_activity (delete_insert) OK 0.24s 5 batches (1d) range 2026-03-31T09:00:00 → 2026-04-04T14:00:00 8 rows audit (d) orders_placed_is_non_negative PASS 5/5 audit (d) not_null (activity_hour) PASS 5/5 audit (f) orders_placed_is_non_negative PASS audit (f) not_null (activity_hour) PASS - 17/17 table hourly_activity_with_daily_context (delete_insert) OK 0.63s - 13 batches (6h) range 2026-03-31T18:00:00 → 2026-04-03T23:00:00 0 rows - audit (d) day_orders_cover_hourly_orders PASS 13/13 - audit (d) not_null (activity_hour) PASS 13/13 - audit (f) day_orders_cover_hourly_orders PASS - audit (f) not_null (activity_hour) PASS + ... ✓ Completed successfully PASS=99 WARN=0 FAIL=0 INSUFFICIENT=0 SKIP=0 TOTAL=99 (2.13s) ``` @@ -180,7 +140,40 @@ Seeds (1) The models and functions are no longer listed as first run, and the seed is current. The two loader-managed sources load on every build. -## 5. Common operations +## 5. Change a model + +Edit `models/staging/stg_orders.sql` to drop cancelled orders, by adding this line at the end: + +```sql +WHERE status <> 'cancelled' +``` + +Then plan again: + +```bash +sqb plan +``` + +``` +Query changed (1) +└── stg_orders recreate view + └── query diff: + --- previous + +++ current + @@ -5,4 +5,5 @@ + quantity, + ordered_at, + status + -FROM __source("raw__orders") + +FROM __source("raw__orders") + +WHERE status <> 'cancelled' +``` + +The plan shows which model changed and the exact diff, before anything runs. Downstream models +still build as usual; each incremental model's own `replay_on_change` decides whether it reprocesses +history. See [Planning](/docs/concepts/planning). + +## 6. Common operations ```bash # Rebuild a specific model diff --git a/website/src/navigation.mjs b/website/src/navigation.mjs index a06feae12d..7f0fafd3fb 100644 --- a/website/src/navigation.mjs +++ b/website/src/navigation.mjs @@ -5,5 +5,6 @@ export const sidebar = sidebarData; export const redirects = { "/docs/concepts/rules/overview": "/docs/concepts/rules", "/docs/concepts/rules/custom-rules": "/docs/concepts/rules/custom-rules/overview", - "/docs/concepts/planning/cascade-propagation": "/docs/concepts/planning/replay-decisions" + "/docs/concepts/planning/cascade-propagation": "/docs/concepts/planning/replay-decisions", + "/docs/concepts/enums-and-constants": "/docs/concepts/enums" }; diff --git a/website/src/sidebar.json b/website/src/sidebar.json index 8e33f35aac..10ca9b2fcf 100644 --- a/website/src/sidebar.json +++ b/website/src/sidebar.json @@ -4,81 +4,134 @@ "items": [ "docs", "docs/quickstart", + "docs/project-setup", + { + "label": "Coming from dbt", + "collapsed": true, + "items": [ + "docs/concepts/dbt-compatibility/overview", + "docs/concepts/dbt-compatibility/selection", + "docs/concepts/dbt-compatibility/adding-sqlbuild-models" + ] + }, "docs/feature-comparison", "docs/roadmap", "docs/benchmarks" ] }, { - "label": "dbt Compatibility", + "label": "Build models", + "items": [ + { + "label": "Models", + "slug": "docs/concepts/models" + }, + "docs/concepts/models/materializations", + "docs/concepts/sources", + "docs/concepts/seeds", + "docs/concepts/incremental", + "docs/concepts/snapshots", + "docs/concepts/functions", + { + "label": "Macros", + "collapsed": true, + "items": [ + { + "label": "Overview", + "slug": "docs/concepts/macros" + }, + "docs/concepts/macros/sql-macros", + "docs/concepts/macros/sql-macro-reference", + "docs/concepts/macros/python-macros", + "docs/concepts/macros/composition-and-context" + ] + }, + "docs/concepts/selectors" + ] + }, + { + "label": "Testing", "items": [ - "docs/concepts/dbt-compatibility/overview", - "docs/concepts/dbt-compatibility/selection", - "docs/concepts/dbt-compatibility/adding-sqlbuild-models" + "docs/concepts/testing", + "docs/concepts/scenarios", + "docs/concepts/audits", + "docs/concepts/diff" ] }, { - "label": "Concepts", + "label": "Correctness", "items": [ - "docs/concepts/project-configuration", - "docs/concepts/resource-identities", "docs/concepts/semantic-compilation", + "docs/concepts/models/contracts", + "docs/concepts/models/type-enforcement", + "docs/concepts/models/schemas", { - "label": "Refactoring", - "collapsed": false, + "label": "Rules", + "collapsed": true, "items": [ { "label": "Overview", - "slug": "docs/concepts/refactoring" + "slug": "docs/concepts/rules" }, - "docs/concepts/models/migrations", - "docs/concepts/models/column-migrations" + "docs/concepts/rules/configuration-and-selection", + "docs/concepts/rules/findings-and-exceptions", + "docs/concepts/rules/execution-and-caching" ] + } + ] + }, + { + "label": "Ship", + "items": [ + { + "label": "Planning", + "slug": "docs/concepts/planning" }, + "docs/concepts/planning/replay-decisions", + "docs/concepts/planning/source-freshness", + "docs/concepts/planning/selection-and-staleness", { - "label": "Adapters", - "collapsed": true, - "items": [ - "docs/concepts/adapters", - "docs/concepts/adapters/snowflake", - "docs/concepts/adapters/duckdb", - "docs/concepts/adapters/motherduck", - "docs/concepts/adapters/postgres", - "docs/concepts/adapters/sqlite", - "docs/concepts/adapters/bigquery", - "docs/concepts/adapters/databricks", - "docs/concepts/adapters/sqlserver", - "docs/concepts/adapters/clickhouse" - ] + "label": "Review a branch", + "link": "/docs/cli/review/" }, - "docs/concepts/sources", - "docs/concepts/seeds", + "docs/concepts/column-lineage" + ] + }, + { + "label": "Refactor", + "items": [ + "docs/concepts/refactoring", + "docs/concepts/models/migrations", + "docs/concepts/models/column-migrations" + ] + }, + { + "label": "Investigate", + "items": [ + "docs/concepts/observability/recall", + "docs/concepts/observability/recall-clickhouse", + "docs/concepts/observability", + "docs/concepts/observability/sinks" + ] + }, + { + "label": "Advanced", + "items": [ { - "label": "Models", + "label": "Declaration scopes", "collapsed": true, "items": [ - "docs/concepts/models", - "docs/concepts/models/materializations", - "docs/concepts/models/schemas", - "docs/concepts/models/type-enforcement", - "docs/concepts/models/contracts", { - "label": "Hooks", - "collapsed": true, - "items": [ - { - "label": "Overview", - "slug": "docs/concepts/models/hooks" - }, - "docs/concepts/models/hooks/sql", - "docs/concepts/models/hooks/python" - ] + "label": "Overview", + "slug": "docs/concepts/declaration-scopes" }, - "docs/concepts/models/configuration" + "docs/concepts/declaration-scopes/visibility", + "docs/concepts/declaration-scopes/placement", + "docs/concepts/declaration-scopes/explorer" ] }, { - "label": "Enums and Constants", + "label": "Enums and constants", "collapsed": true, "items": [ "docs/concepts/enums", @@ -88,94 +141,64 @@ "docs/concepts/model-private-values" ] }, + "docs/concepts/interpolation", { - "label": "Macros", + "label": "Hooks", "collapsed": true, "items": [ - "docs/concepts/macros", - "docs/concepts/macros/sql-macros", - "docs/concepts/macros/python-macros", - "docs/concepts/macros/composition-and-context" + { + "label": "Overview", + "slug": "docs/concepts/models/hooks" + }, + "docs/concepts/models/hooks/sql", + "docs/concepts/models/hooks/python" ] }, - "docs/concepts/interpolation", - "docs/concepts/functions", - "docs/concepts/incremental", { - "label": "Planning and Change Detection", + "label": "Custom Rules", "collapsed": true, "items": [ - "docs/concepts/planning", - "docs/concepts/planning/replay-decisions", - "docs/concepts/planning/source-freshness", - "docs/concepts/planning/selection-and-staleness" + "docs/concepts/rules/custom-rules/overview", + "docs/concepts/rules/custom-rules/rule-context", + "docs/concepts/rules/custom-rules/testing-and-determinism", + "docs/concepts/rules/published-layer-rules" ] }, - "docs/concepts/snapshots", - "docs/concepts/audits", { - "label": "Rules", + "label": "Python nodes", "collapsed": true, "items": [ - "docs/concepts/rules", - "docs/concepts/rules/configuration-and-selection", - "docs/concepts/rules/findings-and-exceptions", - "docs/concepts/rules/published-layer-rules", - "docs/concepts/rules/execution-and-caching" + "docs/concepts/python-nodes/overview", + "docs/concepts/python-nodes/loaders", + "docs/concepts/python-nodes/tasks", + "docs/concepts/python-nodes/assets", + "docs/concepts/python-nodes/checks", + "docs/concepts/python-nodes/factories", + "docs/concepts/python-nodes/providers", + "docs/concepts/python-nodes/sql-references" ] }, - "docs/concepts/testing", - "docs/concepts/scenarios", - "docs/concepts/selectors", - "docs/concepts/column-lineage", - "docs/concepts/diff" + "docs/concepts/adapters/custom", + "docs/concepts/observability/execution-history" ] }, { - "label": "Advanced Concepts", + "label": "Adapters", + "collapsed": true, "items": [ { - "label": "Observability", - "collapsed": true, - "items": [ - "docs/concepts/observability", - "docs/concepts/observability/recall", - "docs/concepts/observability/sinks", - "docs/concepts/observability/execution-history" - ] - }, - { - "label": "Declarations and Scopes", - "collapsed": true, - "items": [ - "docs/concepts/declaration-scopes", - "docs/concepts/declaration-scopes/visibility", - "docs/concepts/declaration-scopes/placement", - "docs/concepts/declaration-scopes/explorer" - ] + "label": "Overview", + "slug": "docs/concepts/adapters" }, - { - "label": "Custom Rules", - "collapsed": true, - "items": [ - "docs/concepts/rules/custom-rules/overview", - "docs/concepts/rules/custom-rules/rule-context", - "docs/concepts/rules/custom-rules/testing-and-determinism" - ] - } - ] - }, - { - "label": "Python Nodes", - "items": [ - "docs/concepts/python-nodes/overview", - "docs/concepts/python-nodes/loaders", - "docs/concepts/python-nodes/tasks", - "docs/concepts/python-nodes/assets", - "docs/concepts/python-nodes/checks", - "docs/concepts/python-nodes/factories", - "docs/concepts/python-nodes/providers", - "docs/concepts/python-nodes/sql-references" + "docs/concepts/adapters/snowflake", + "docs/concepts/adapters/duckdb", + "docs/concepts/adapters/motherduck", + "docs/concepts/adapters/postgres", + "docs/concepts/adapters/sqlite", + "docs/concepts/adapters/bigquery", + "docs/concepts/adapters/databricks", + "docs/concepts/adapters/sqlserver", + "docs/concepts/adapters/clickhouse" ] }, { @@ -194,39 +217,85 @@ "docs/integrations/ingestr" ] }, + { + "label": "Reference", + "items": [ + { + "label": "Configuration reference", + "slug": "docs/concepts/project-configuration" + }, + { + "label": "MODEL() fields", + "slug": "docs/concepts/models/configuration" + }, + "docs/concepts/testing-reference", + "docs/concepts/resource-identities" + ] + }, { "label": "CLI Reference", "items": [ - "docs/cli/init", - "docs/cli/playground", - "docs/cli/skills", - "docs/cli/server", - "docs/cli/login", - "docs/cli/compile", - "docs/cli/format", - "docs/cli/contract", - "docs/cli/scope", - "docs/cli/rename", - "docs/cli/rules", - "docs/cli/plan", - "docs/cli/build", - "docs/cli/load", - "docs/cli/seed", - "docs/cli/test", - "docs/cli/scenario", - "docs/cli/audit", - "docs/cli/freshness", - "docs/cli/check", - "docs/cli/clone", - "docs/cli/diff", - "docs/cli/lineage", - "docs/cli/dag", - "docs/cli/query", - "docs/cli/recall", - "docs/cli/debug", - "docs/cli/janitor", - "docs/cli/clean", - "docs/cli/dbt" + { + "label": "Set up", + "items": [ + "docs/cli/init", + "docs/cli/playground", + "docs/cli/skills", + "docs/cli/debug" + ] + }, + { + "label": "Develop", + "items": [ + "docs/cli/compile", + "docs/cli/format", + "docs/cli/test", + "docs/cli/scenario", + "docs/cli/lineage", + "docs/cli/scope", + "docs/cli/rename", + "docs/cli/rules", + "docs/cli/contract" + ] + }, + { + "label": "Build", + "items": [ + "docs/cli/plan", + "docs/cli/build", + "docs/cli/load", + "docs/cli/seed", + "docs/cli/audit", + "docs/cli/check", + "docs/cli/freshness" + ] + }, + { + "label": "Compare and ship", + "items": [ + "docs/cli/diff", + "docs/cli/review", + "docs/cli/clone" + ] + }, + { + "label": "Operate", + "items": [ + "docs/cli/janitor", + "docs/cli/recall", + "docs/cli/query", + "docs/cli/clean", + "docs/cli/dag", + "docs/cli/dbt" + ] + }, + { + "label": "Server", + "items": [ + "docs/cli/server", + "docs/cli/login" + ] + } ] } ]