Skip to content

fix(cost): sync on the cheapest warehouse and finish runs before collecting costs - #1576

Merged
kvlonge merged 3 commits into
mainfrom
fix/sync-warehouse-and-cost-collection
Nov 18, 2026
Merged

kvlonge merged 3 commits into
mainfrom
fix/sync-warehouse-and-cost-collection

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Why

A location's warehouse syncs ran under no command group, so they could wake a large warehouse just to read metadata and small state tables, paying a minimum billed minute each time. Run-end cost collection also kept builds and server runs open for 30 to 50 s, and every build re-read older runs whose costs could not change.

Changes

  • A location syncs on a new sync warehouse group: the target's sync warehouse, else query, else the connection's own warehouse, as before. The target's build warehouse is used only when the connection names none, so a sync can still run. This deliberately differs from the earlier local commit, which tried build before the connection's warehouse. sqb debug lists the sync warehouse and its source. Runs a location executes keep their own command's group.
  • A command-line build prints its summary, then Collecting warehouse costs..., with a 6 s run-end budget. A worker build records its cost as pending and collects it after the run publishes its outcome.
  • A finished query that still has no warehouse size two minutes after it ended ran without warehouse compute and no longer marks the run partial.
  • A partial run is re-read only while a statement was unseen, running or awaiting metadata, once it ended two minutes ago, over its own window, within an 8 s shared budget per build.

Verification

  • Warehouse unit cases: a sync uses sync, then query, then the connection's warehouse; a target with only build = "XL_WH" and a connection warehouse syncs on the connection's (fails on the earlier order); with no connection warehouse it uses build.
  • Cost refresh tests (sqlbuild-state) and Snowflake cost unit tests: missing metadata, final partial runs, per-run windows, shared budget.
  • No DuckShop edge case: this changes warehouse choice and cost timing, not a project shape, data value or lifecycle.
  • make check-ci passed on the final head.

…house

A location's warehouse syncs ran under no command group, so they used
the connection's warehouse, often the largest, to read metadata and
small state tables, waking it for a minimum billed minute each time.

The location command now uses a new 'sync' warehouse group: the
target's [targets.<name>.warehouses] sync, else query, else the
connection's own warehouse, as before. The target's build warehouse is
used only when the connection names none, so a sync can still run. sqb
debug lists the sync warehouse with where it comes from, and the
location reports it when a sync session opens. Runs a location executes
keep their own command's group.
… edge

The conversions between CostRunRecord objects and plain cost runs call
into Python, so they move from the bindings crate to
sqlbuild-python-front's resource_costs area, a configured Python edge,
next to the resource costs table they feed. Behaviour is unchanged. The
bindings crate drops below its code-line cap again, and the two pending
edge budgets for the old file go away.
…ange

Run-end cost collection kept builds and server runs open for 30-50 s and
re-read the same runs on every build without changing them.

- A finished query still without a warehouse size two minutes after it
  ended ran without warehouse compute (for example a result Snowflake
  served from metadata); it no longer marks the run partial. A newer
  one is still missing metadata, which a later build's re-read settles.
- A partial run is re-read only while a statement was unseen, still
  running or awaiting metadata; a run that saw every statement with
  only permanent limitations is final. Runs are re-read only once
  they ended two minutes ago.
- An older run's re-read reads only its own window instead of every
  query since it started, and re-reads share an 8 s budget per build.
- The run-end sweep does not retry for missing metadata alone, and a
  command-line build's sweep has a 6 s budget.
- A command-line build prints its summary, then 'Collecting warehouse
  costs...'. A worker build records its cost pending and collects it
  after the run publishes its outcome, so the run finishes and
  releases its claims first.
@kvlonge
kvlonge marked this pull request as ready for review November 18, 2026 13:54
@kvlonge
kvlonge added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit fe92b8e Nov 18, 2026
91 of 93 checks passed
@kvlonge
kvlonge deleted the fix/sync-warehouse-and-cost-collection branch November 18, 2026 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant