Skip to content

feat(state): record each build and seed run in _sqlbuild_runs - #1579

Merged
kvlonge merged 7 commits into
mainfrom
feat/observed-runs-main
Nov 18, 2026
Merged

kvlonge merged 7 commits into
mainfrom
feat/observed-runs-main

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Why

Nothing in the warehouse records who ran a build or seed, when, or how it ended; a killed run looks like one that never started.

Changes

  • Each sqb build that runs nodes, and each sqb seed, appends a started row to _sqlbuild_runs in the target schema before its first node, and a finished row with its status and node counts at the end. A run with only a started row was killed or crashed.
  • CLI and server workers share the row writers; a worker adds its server run ID, attempt, retry link and starter.
  • Each row records the selection as typed, named jobs, target, warehouse, sqb login and OS users, machine name, SQLBuild version, Git commit, and only --vars names. [settings] record_run_host = false leaves out the machine name.
  • The started row sets the table up once per run; the finished row inserts directly. A preserve target schema records no run. A failed write warns and never fails the run; Ctrl+C stops it.
  • Finished counts include Python nodes and checks; a build that fails without a failed node is recorded as failed.
  • Runs-table adapter methods live in a mixin, so the strict adapter contract still covers every base method.

Verification

  • CLI integration tests on DuckDB: rows for builds, seeds and a killed build; partial and failed statuses (including a failing task with no failed model); --vars names only; host opt-out; an unwritable table warns and the build succeeds; Ctrl+C while recording stops the build. The interrupt and task-status cases fail without the third commit.
  • Worker integration test: a scheduled run's rows carry its server run, attempt and starter.
  • Recall tests: a first build logs no failed statement. Snowflake DDL, strict adapter and run status tests.
  • Golden local-config-unknown-key now lists record_run_host among allowed keys.
  • No DuckShop edge case: its edge area holds SQL shapes, not targets; the audit storage test covers a preserve target.
  • make check-ci passed on the final head.

Every sqb build that runs nodes, and every sqb seed, now appends a
started row to _sqlbuild_runs in its target schema before its first node
runs, and a finished row with its status (succeeded, partial, failed or
cancelled) and node counts when it ends. A run with only a started row
was killed or crashed.

The rows are written in the build and seed paths the command line and
server workers share, so both record the same way; a worker adds the
server run ID, attempt, retry link and who started it. Each row records
the selection as typed, the jobs it names, the target, the warehouse,
sqb login and operating-system users, the machine name, the SQLBuild
version and the Git commit, and only the names of --vars, never their
values. [settings] record_run_host = false leaves out the machine name.

Writing a row never fails the build: a failed write prints a warning.
… that fails

A build's started row created the schema and the runs table before every
insert, about 0.4 s of Snowflake statements on every run after a schema's
first. Both rows now insert directly and set the table up only when the
insert fails, as on a schema's first run.
… tasks or checks fail

- Ctrl+C or a termination while a run row is written now stops the
  build or seed. Only an ordinary error counts as a failed write, which
  still only warns, and only it triggers creating the runs table.
- A finished build's row now counts Python nodes and checks as the
  footer does, and a build that failed without a failed node, such as
  on an end-audit error, is recorded as failed instead of succeeded.
- The runs-table adapter methods move into their own mixin, so the
  strict adapter contract still covers every base adapter method.
…order with its started row

Counting a Python node or check into a run's node counts moves onto
RunNodeCounts in sqlbuild-state, using the build's own Python node
statuses. Opening a run recorder now writes its started row, and
RunRecorder::raised records a run that stopped on an error, so the build
and seed commands no longer repeat either step. The Python bindings
crate stays under its code-line cap; behaviour is unchanged.
@kvlonge
kvlonge enabled auto-merge November 18, 2026 15:22
…a preserve schema

A run's first insert into a missing runs table failed before the table
was created, so the first build in a schema logged a failed statement
in recall although the build succeeded. The started row now sets the
schema and table up, as fingerprint writes do once per run, and the
finished row inserts directly, setting the table up again only if that
insert fails.

A target whose schema is preserve keeps each model's own schema, so it
names no schema of its own: the runs table was created in a new schema
called preserve. Such a run is now not recorded, and a preserve
database falls back to the connection's database.
@kvlonge
kvlonge added this pull request to the merge queue Nov 18, 2026
…keys

The unknown-key diagnostic lists every allowed [settings] key, and runs
recording adds record_run_host.
A build now writes _sqlbuild_runs to its target schema, and the janitor
skips it like the other state tables.
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 18, 2026
@kvlonge
kvlonge added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 41cf38f Nov 18, 2026
35 checks passed
@kvlonge
kvlonge deleted the feat/observed-runs-main branch November 18, 2026 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant