Skip to content

feat: count definition changes and stop --changed at job edges - #1458

Merged
kvlonge merged 4 commits into
mainfrom
feat/changed-counts-reach
Nov 13, 2026
Merged

kvlonge merged 4 commits into
mainfrom
feat/changed-counts-reach

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Why

--changed counted every new source row as a change, so a deploy-then-build rebuilt everything downstream of a streaming source, and it ran models that another scheduled job refreshes. Node status could also show unchanged models as Changed after a fingerprint scheme change.

Changes

  • --changed[=definition,source_data] picks what counts; [changes] counts sets it for a project. Without it, sqb build/plan --changed counts definition, and a job's run with changed: True (schedule, sensor or launched by hand) counts both, as before. A rebuilt upstream counts as a definition change only when its recorded executions show one, followed back through older executions.
  • --reach job|all ([changes] reach, default job): a change stops at a model another triggered job refreshes. A model still runs when any job holding one of its changes, however many levels up, contains it. A job's own run, or a build that selects the job, never holds back that job's models.
  • Planner reasons other than query, function or config edits always count. An unselected ancestor runs on its records, no longer as a missing relation.
  • Build prints the policy and "Changed: X of N run; U unchanged and H held back"; plan --changed lists each section and adds changed to the JSON. The UI command builder handles --flag=value.
  • Node status and the catalog re-fingerprint recorded definitions under the snapshot's dialect.
  • Docs for build and plan; DuckShop cases changed_builds_reach, changed_builds_two_up, changed_source_orders, fingerprint_scheme_change.

Verification

  • sqlbuild-state decision cases, each checked to fail without its fix; sqlbuild-cli/config tests; the orchestration catalog test.
  • E2E test_changed_builds.py (DuckDB, SQLite) with the two-job reach scenario and changes held by two jobs two levels up; integration: a sensor on new source data still rebuilds a changed: True job.
  • UI vitest and svelte-check; clippy, fensu, make check-ci.

--changed[=definition,source_data] chooses what counts as a change, and
--reach job|all stops a change at nodes another triggered job refreshes.
Node status re-fingerprints recorded definitions under the snapshot's
dialect before reporting a model as changed.
…own models

A job's run with changed: True counts definition and source data unless
[changes] counts says otherwise, and reaches its whole selection. A build
that selects a job never holds back that job's models. Every counted
change contributes its holding jobs, read-back follows deep chains, and
planner reasons other than definition edits always count.
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 13, 2026
@kvlonge
kvlonge disabled auto-merge November 13, 2026 21:06
…odel

A rebuilt upstream that read several redefined relations now reports all of
them, so a model runs when any job holding one of those changes contains it,
whatever order the inputs were recorded in.
@kvlonge
kvlonge enabled auto-merge November 13, 2026 21:30
@kvlonge
kvlonge added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit 442b9a8 Nov 13, 2026
53 checks passed
@kvlonge
kvlonge deleted the feat/changed-counts-reach branch November 13, 2026 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant