Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
166 changes: 91 additions & 75 deletions src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb
| `--concurrency` | Maximum number of audits to run concurrently (default: `1`) |
| `--json` | Print structured audit results |
| `--json-output` | Write structured audit results to a file |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse |

## Concurrency

Expand Down
57 changes: 55 additions & 2 deletions src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ Online: https://sqlbuild.com/docs/cli/build/
- Output
- Deferred builds
- Failure behavior
- Limits
- Fingerprints
- Runtime artifacts

Expand Down Expand Up @@ -61,7 +62,7 @@ own option. `--version` works only directly after `sqb`, as in `sqb --version`:
| `--defer-sources-to` | Read managed source data from another target |
| `--fail-fast` | Stop on first failure and skip remaining nodes |
| `--concurrency` | Number of worker connections (default: 1) |
| `--test-timeout <duration>` | Time limit for each unit test the build runs, such as `30s`, or `disabled`; see [Time limits](../concepts/testing.md#time-limits) |
| `--test-timeout <duration>` | Time limit for each unit test the build runs, such as `30s`, or `disabled`; see [Time limits](../concepts/testing-reference.md#time-limits) |
| `--verbose`, `-v` | Show lifecycle SQL inline after each model |
| `--start-cursor` | Override the start cursor, typed from the selected incremental models' cursor type; fails with `S303` if the selection mixes timestamp and integer cursors. With no cursor model selected, an integer value is read as an integer cursor and anything else as an ISO timestamp |
| `--end-cursor` | Override the end cursor, typed the same way as `--start-cursor` |
Expand All @@ -77,7 +78,7 @@ own option. `--version` works only directly after `sqb`, as in `sqb --version`:
| `SELECTOR ...` | Positional selectors, the same as `--select` |
| `--select`, `-s` | Select specific models, or unit tests by name (`test:<name>`); selecting only unit tests runs them without building models |
| `--exclude` | Exclude specific models |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse |

## Fast iteration

Expand Down Expand Up @@ -219,6 +220,58 @@ skipped as for any other failure. Its row shows `TIMEOUT`. In the summary line,
failures were timeouts. A timeout is a failure for the exit code: the build exits non-zero and its
JSON `status` is `failed`.

## Limits

A target's [`execution_limits`](../concepts/project-configuration.md#execution-limits) cap a whole
build, and a model's [`max_duration`](../concepts/models/configuration.md#execution-time-limit) caps
one model.

### Model limit

When the build would run more models than `max_models`, the build stops before any clone, load or
model runs, and says that nothing in the warehouse changed. `sqb plan` has no limit, so use it to
check a broad selection.

### Duration limit

`max_duration` covers planning and the build. At the deadline SQLBuild starts no more work and
cancels the statements still running. Where the warehouse has a statement time limit, each
statement also gets only the time that remains.

| Adapter | Warehouse statement limit | Cancellation at the deadline | Longest accepted duration |
|---------|---------------------------|------------------------------|---------------------------|
| Snowflake | `STATEMENT_TIMEOUT_IN_SECONDS` per statement | Driver cancel request for the running query | 7 days |
| DuckDB, MotherDuck | None | Connection interrupt | About 24 days |
| PostgreSQL | `SET LOCAL statement_timeout` inside transactions; statements outside a transaction rely on cancellation | Driver cancel (`cancel_safe`) | About 24 days |
| BigQuery | Job timeout (`job_timeout_ms`) on each query job | `jobs.cancel`; reported as requested, not confirmed | About 24 days |
| Databricks | `STATEMENT_TIMEOUT` session setting, capped at 48 hours | Cursor cancel; reported as requested, not confirmed | About 24 days |
| SQL Server | None; the driver's query timeout is process-wide | Driver cancel | About 24 days |
| SQLite | None | Connection interrupt | About 24 days |

A duration timeout can come after earlier models finished, so its error says that warehouse changes
may already have been made. A custom adapter that can't cancel a running statement can't be used
with `max_duration`.

### Per-item time limits

Unit test, scenario and model limits (`test_timeout`, `scenario_timeout` and a model's
`max_duration`) count running time only, from the item's first warehouse statement. Rollback and
cleanup always run and don't count. Where a warehouse reports queueing separately, queued
time doesn't count:

| Adapter | How the item limit is enforced | Queued time |
|---------|--------------------------------|-------------|
| Snowflake | `STATEMENT_TIMEOUT_IN_SECONDS` per statement, with a client-side backstop after a grace margin of half the limit (at least 30 seconds) | Excluded by the warehouse limit; the backstop counts it |
| DuckDB, MotherDuck | Connection interrupt at the limit | Not applicable |
| PostgreSQL | `SET LOCAL statement_timeout` inside transactions; cancellation at the limit | Lock waits count |
| BigQuery | `jobs.cancel` at the limit; no job timeout, because a job timeout counts from job creation | Excluded: the clock stops while a job is pending |
| Databricks | `STATEMENT_TIMEOUT` session setting and cursor cancel | May count |
| SQL Server | Driver cancel at the limit | Lock waits count |
| SQLite | Connection interrupt at the limit | Waits for the write lock count |

When limits nest, such as a model's unit tests inside its own limit, the earliest deadline applies.
After a timeout, SQLBuild checks the connection and replaces it if it is no longer usable.

## Fingerprints

After a successful build, SQLBuild writes version identities to `_sqlbuild_fingerprints` in the target schema. These are used on subsequent runs to detect changes and skip unchanged work. A build writes a row only for a resource whose identity differs from its latest row, so a build that changes no definitions writes none. In a project with model migration history, every build writes a row for each resource it builds. See [Planning and Change Detection](../concepts/planning.md) for details.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb
| `--json` | Print check results as JSON |
| `--json-output` | Write check results JSON to a file path |
| `--vars` | Override project variables |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse |

Selecting a non-check node is rejected; use [`sqb build`](build.md) to run tasks and assets.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ sqb --project-dir <path> clone --from <target> [--to <target>] [flags]
| `--hard-copy` | Force physical table copies instead of zero-copy cloning |
| `--select`, `-s` | Select specific models to clone |
| `--exclude` | Exclude specific models from cloning |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse |

## Examples

Expand Down Expand Up @@ -95,4 +95,4 @@ allow_as_clone_destination = true
The origin target must already contain the built relations being cloned. The destination
connection's credentials must be able to read the origin namespace and create relations in the
destination. Managed physical sources may bootstrap a destination that has not been built. See
[Project Configuration](../concepts/project-configuration.md) for details.
[Configuration reference](../concepts/project-configuration.md) for details.
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb
2. **Graph.** Resolves `__ref()`, `__source()` and `__seed()`, expands macros, and orders models
by dependency.
3. **SQL checks.** Parses and checks SQL, when SQL analysis is on. See
[Semantic Compilation](../concepts/semantic-compilation.md).
[What `sqb compile` checks](../concepts/semantic-compilation.md).
4. **Column lineage.** Traces column-level dependencies across models.
5. **Contracts.** Checks declared columns against the inferred query output.
6. **Rules.** Runs the selected built-in Rules, then the selected custom Rules.
Expand All @@ -89,8 +89,7 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb
Rules that need inferred types, such as `SQBRCONTRACT105`, are skipped when SQL analysis is off,
and compile prints one note naming them.

`sqb compile` is authoritative for correctness, contracts, lineage and configured Rules. Rule
findings block the artifacts. To check one Rule code or family without changing project
Rule findings block the artifacts. To check one Rule code or family without changing project
configuration, use [`sqb rules run`](rules.md).

### When Rules fail
Expand Down Expand Up @@ -170,19 +169,14 @@ error[K001]: declared column 'total_cents' was not found in statically inferred
| `K001` | A declared column is missing from the query output |
| `K002` | A column's inferred type doesn't match the declared type |
| `K003` | A column's type can't be proven (with `type_enforcement` on) |
| `K004` | A `nullable false` column can be null |
| `K005` | A model with `contract enforced` outputs a column it doesn't list |
| `K006` | A model with `contract enforced` lists no columns |
| `K013` | A string literal fills an enum-typed column of an enforced contract but isn't a member of the enum (not checked on SQL Server) |

Which models are checked depends on `[settings] check_declared_columns`:

- `true` (default): a model with declared columns and no `contract` setting must output every
declared column.
- `false`: only models with `contract enforced` are shape-checked; declared columns elsewhere
are metadata and audit targets.

Explicit type enforcement works with either value. Setting `false` is a project policy, not a
way to silence an error: if a model is meant to have a contract, fix the query or the
declaration. Errors on `contract enforced` models don't suggest changing the setting. See
[Contracts](../concepts/models/contracts.md).
Models without a `contract` setting are checked unless the project sets
`[settings] check_declared_columns = false`; see [Contracts](../concepts/models/contracts.md). Don't
turn the check off to silence one error: fix the query or the column list.

## Output

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@

Online: https://sqlbuild.com/docs/cli/contract/

Existing-schema comparison is explicitly online and read-only with respect to the warehouse.
`sqb contract` compares a model's or source's declared columns with the table that already exists in
the warehouse, or writes declarations from it. It reads the warehouse and never changes it. The
target named by `--from` supplies the database and schema to read, using the project's connection.

## diff

Expand All @@ -22,8 +24,22 @@ Reports declaration differences. Exit code `0` means agreement and `1` means dif
sqb contract generate --from <target> [flags]
```

Prints proposed declaration updates. Add `--write` to change repository files and `--overwrite` to
replace conflicts rather than preserving authored declarations.
Prints proposed declaration updates without changing files.

```bash
# Fill in missing types and add warehouse columns missing from code
sqb contract generate --from prod --select tag:commerce --write

# Also replace conflicting types and names, and remove columns the warehouse doesn't have
sqb contract generate --from prod --select tag:commerce --write --overwrite
```

Without `--overwrite`, generation keeps everything you wrote: existing types where they conflict,
columns only in code, comments, descriptions, audits, nullability, enums, tags, freshness and loader
settings. It never adds `contract enforced`; that stays your decision.

A shared `SCHEMA()` isn't changed from one model's columns; the conflict is reported instead. Writes
are atomic, and SQLBuild recompiles the project afterwards and restores the old files if it fails.

## Flags

Expand All @@ -37,7 +53,7 @@ replace conflicts rather than preserving authored declarations.
| `--vars` | both | Override project variables |
| `--write` | generate | Apply generated repository changes |
| `--overwrite` | generate | Replace conflicting declarations; requires `--write` |
| `--warehouse <name>` | both | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |
| `--warehouse <name>` | both | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse |

See [Contracts](../concepts/models/contracts.md#adopting-an-existing-schema) for ownership and safety
guarantees.
Sources can be selected like models, for example `--select source:raw_orders`. See
[Contracts](../concepts/models/contracts.md) for what a contract checks.
48 changes: 46 additions & 2 deletions src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dbt.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,16 @@

Online: https://sqlbuild.com/docs/cli/dbt/

## Contents

- sqb dbt plan
- sqb dbt run
- sqb dbt build
- Flags
- sqb dbt debug
- Selectors
- Configuration

Orchestrate dbt and SQLBuild together. Each subcommand runs dbt first, then SQLBuild, with selection logic across both project graphs. dbt remains responsible for dbt-owned models; SQLBuild validates and executes SQLBuild-owned models downstream. See [Using SQLBuild with dbt](../concepts/dbt-compatibility/overview.md) for scope and selection behavior.

## sqb dbt plan
Expand Down Expand Up @@ -48,6 +58,40 @@ confirmation asks for the typed confirmation on an interactive terminal and refu
build the affected models with `sqb build` and its allow flag instead, or relax the policy. A
refusal at this point leaves completed dbt work in place and builds no SQLBuild models.

## Flags

`sqb dbt plan/run/build` declare the common flags directly. Anything declared goes **before** a `--` separator; any other raw dbt flag goes **after** it and is forwarded verbatim. A flag placed on the wrong side errors rather than silently reaching dbt.

```bash
sqb dbt build --select path:models/marts --full-refresh
```

Declared flags are routed to the right place automatically:

| Flag | Routed to | Notes |
| ---- | --------- | ----- |
| `--select` / `--exclude` | dbt | dbt resolves selection (see [Selection](../concepts/dbt-compatibility/selection.md)). |
| `--vars` | dbt **and** SQLBuild | The same vars feed dbt's compile and SQLBuild's own variable resolution, so both sides see identical values. |
| `--full-refresh` | dbt **and** SQLBuild | Requests a full rebuild on both sides. dbt and native SQLBuild models independently apply their model-level `full_refresh` setting, including `false` opt-outs. |
| `--threads` | dbt (`--threads`) and SQLBuild (`--concurrency`) | |
| `--target` / `--project-dir` / `--profiles-dir` / `--profile` / `--target-path` | dbt | Standard dbt locators. |

### `--vars`

`--vars` accepts the same JSON object dbt accepts, and SQLBuild passes it to **both** the underlying `dbt` invocation and its own variable resolution. That means a value referenced as `@@my_var` in SQLBuild model SQL (see [Variables and placeholders](../concepts/interpolation.md)) and as `{{ var('my_var') }}` in a dbt model both resolve to the value you passed. CLI vars take precedence over project and local config vars. You do not need to declare vars twice.

```bash
sqb dbt build --select path:models/marts --vars '{"my_var": 1}'
```

### Forwarding other dbt flags

For any native dbt flag SQLBuild does not declare, put it after `--` and it is passed straight to the `dbt` invocation untouched:

```bash
sqb dbt build -- --log-level debug
```

## sqb dbt debug

Run dbt diagnostics followed by SQLBuild diagnostics.
Expand Down Expand Up @@ -93,9 +137,9 @@ profiles_dir = "../profiles"
target_path = "../dbt_project/target"
```

See [Project Configuration](../concepts/project-configuration.md#dbt) for all fields.
See [Using SQLBuild with dbt](../concepts/dbt-compatibility/overview.md#configuration) for all fields.

SQLBuild-owned work in `sqb dbt plan`, `run` and `build` connects with the target's `build`
[warehouse group](../concepts/project-configuration.md#command-group-warehouses). dbt itself
[warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses). dbt itself
keeps using the warehouse from its profile. These commands do not accept `--warehouse`, because
their remaining arguments pass through to dbt.
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<!-- generated-by: sqlbuild skills -->

# debug
# sqb debug

> Validate project configuration and test the warehouse connection.

Expand All @@ -22,7 +22,7 @@ The command reports four groups of checks:

**Runtime** - SQLBuild version, Python version, Python path, OS info.

**Configuration** - Finds and validates `sqlbuild_project.toml`, loads `sqlbuild_local.toml` if present, resolves the adapter, active target, connection, database, and schema. For Snowflake it also shows the warehouse each [command group](../concepts/project-configuration.md#command-group-warehouses) connects with and where that value came from.
**Configuration** - Finds and validates `sqlbuild_project.toml`, loads `sqlbuild_local.toml` if present, resolves the adapter, active target, connection, database, and schema. For Snowflake it also shows the warehouse each [command group](../concepts/adapters/snowflake.md#command-group-warehouses) connects with and where that value came from.

**Providers** - Lists the [providers](../concepts/python-nodes/providers.md) the project defines.

Expand All @@ -34,7 +34,7 @@ The command reports four groups of checks:
|------|-------------|
| `--no-connection` | Skip the connection and query tests. Useful for validating config without warehouse access. |
| `--json` | Output results as JSON instead of formatted text. |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/adapters/snowflake.md#command-group-warehouses) and the connection warehouse |

## Example

Expand Down
Loading
Loading