Skip to content

fix!: atomic keyed writes, indexed control store and matching loader locations - #1478

Merged
kvlonge merged 18 commits into
mainfrom
fix/keyed-writes-and-loader-locations
Nov 14, 2026
Merged

kvlonge merged 18 commits into
mainfrom
fix/keyed-writes-and-loader-locations

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Why

Unindexed control-store foreign keys held the SQLite write lock through scans. Databricks and Snowflake keyed writes were not atomic. sqb load placed managed sources differently from build, and a loader under a target schema of preserve wrote into a schema named preserve. A slow store commit delayed every scheduler wake.

Changes

  • Every control-store foreign key is indexed, created in place when a store opens.
  • Databricks keyed delete_insert is one MERGE; Snowflake runs its staging CTAS before the transaction.
  • sqb load puts a managed source where build does: its own location, else the target's loader_schema, then schema. It expands only the target value it falls back to, with --vars, and refuses ${CTX:...} as build does.
  • A target schema of preserve stops an intermediate loader with no destination schema (S103) and a managed source with no schema (P001).
  • S103 and S104 messages and sqb debug on DuckDB are corrected.
  • The scheduler sleeps to the next due time read before its last commit, records per-phase cycle timings and logs cycles over 500 ms.
  • Deployment docs show the server and a location as Podman Quadlet units.

BREAKING CHANGE: loaders that wrote into a schema named preserve now fail with S103 or P001 until they name a schema. sqb load now writes managed sources to loader_schema, like build.

Verification

  • Each fix has a test that fails without it, including store upgrades and e2e placement cases.
  • A 0.223.0 store gained all 10 indexes and still opened in 0.223.0.
  • The latency test bounds wake time, not store time. With a launcher taking 700 ms after each decision it passes; the old loop woke 702 ms late.
  • The Quadlet units ran under systemd with rootful Podman 5.4.2: both containers healthy, a schedule launched a worker,, a killed server restarted. The files pass the Quadlet generator.
  • make check-ci passes. DuckShop is unchanged: these are target settings and command placement, not project shapes.

Debug passed the DuckDB connection file path as the target database,
while models build in the file's own catalog. Debug now drops it for
namespaceless adapters, as diff already does.
A managed source's loader writes to the source's own schema or the
target's loader_schema, so a target schema of preserve no longer stops
it. The check now applies when the selected target's own schema is
preserve and the loader's destination names no schema, the same way for
compile, build and load.
sqb load filled a managed source's missing schema from the target's
schema, while build uses the target's loader_schema first, so the two
commands wrote the same source to different schemas. Load now resolves
it the same way, and its missing-schema error names a managed source as
one.
…preserve

With loader_schema or schema set to preserve and no schema on the
source, build stops with P001 while load wrote into a schema literally
named preserve. Load now raises the same error.
sqb load read the target's database, schema and loader_schema as
written when it placed a managed source, so a value such as
${raw_schema} failed with a parser error while build expanded it. Load
now expands them with the project and command-line vars, as build does.
…k to

Expanding the whole target up front failed load on unused templates that
build ignores, and accepted run context templates that build refuses.
Load now expands just the fallback value, with build's own options.
…og slow cycles

The loop slept for the next due time's distance measured before the cycle's
last commit, so a slow commit or WAL checkpoint pushed every wake back by its
own length. It now sleeps until that deadline. Each cycle records per-phase
timings, the worker supervisor logs cycles slower than 500 ms with them, and
the latency test bounds how late the scheduler wakes while printing store time
separately.
Named-target checks reuse the compile attachment's resolved_target instead of
a copy, the named-target schema binding declares its keyword arguments
directly, and both load-selection bindings read their request through one
helper.
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 14, 2026
@kvlonge
kvlonge enabled auto-merge November 14, 2026 11:08
@kvlonge
kvlonge added this pull request to the merge queue Nov 14, 2026
Merged via the queue into main with commit 53d8395 Nov 14, 2026
50 of 75 checks passed
@kvlonge
kvlonge deleted the fix/keyed-writes-and-loader-locations branch November 14, 2026 12:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant