Skip to content

feat(orchestration): report failed runs to the platform webhook and fix job write sets - #1441

Merged
kvlonge merged 3 commits into
mainfrom
fix/tag-job-write-set
Nov 13, 2026
Merged

kvlonge merged 3 commits into
mainfrom
fix/tag-job-write-set

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Why

  • A job whose models read a managed source failed: the build loads it, but the write set did not claim it.
  • Tag-selected layer jobs share the functions their models call, so their group reported a false cycle.
  • Bearer tokens went over plain HTTP silently.
  • No push signal existed for failed runs.

Changes

  • Write sets include the managed sources selected models read. Group children and reruns compute their own; --no-load-sources claims none; ad-hoc permission checks cover them.
  • Known gaps, for a follow-up: sources loaded through loader dependencies or hook reads are still unclaimed, as the snapshot doesn't carry those reads yet.
  • A job that builds a node itself no longer waits for its other builders. A real cycle names the group and the edges that close it.
  • A non-loopback http:// server gets one warning per process; HTTP stays accepted, loopback silent.
  • New run_failed platform webhook event (same envelope, Idempotency-Key, at-least-once). It names the job or ad-hoc run, run ID, location, target, outcome and up to 20 failed nodes; error text is redacted and capped. Sent for failed and partial runs, which have failed nodes. It stays out of person history, as error text can quote warehouse data.
  • Docs cover the warning and webhook; TS client and docs skill regenerated; UI skips the event.

Verification

  • Orchestration crate on SQLite and PostgreSQL, with one run_failed per partial run through faults and retries, and a group rerun with a loaded source.
  • Publication and orchestration integration suites. A worker build loading a managed source passes; on the base binary it fails with planned writer source:raw_orders is outside the frozen run write set. The warning shows for 0.0.0.0, not 127.0.0.1.
  • The DuckShop copy that showed the false cycle now publishes.
  • DuckShop: adds edge_cases/orchestration/shared_function_layers.py. The warning and webhook event have no project shape.
  • make check-ci; UI check, tests and build.

A build that loads a managed source for a selected model now claims that
source in its frozen write set, so the worker no longer refuses the run.
Group children compute their own write sets, and reruns recompute writers
for their narrowed selection.

A job that builds a node itself, such as a function its models share with
another layer's job, no longer waits for that node's other builders, so
tag-selected layer jobs stop reporting a false group cycle. A real cycle
now names the group and the edges that close it.

Locations and workers warn once per process when a bearer token would
travel over plain HTTP to a non-loopback server.
A failed or partial run now publishes a run_failed platform event naming
its job, run, location, target, outcome and first failed nodes, with node
error text redacted and bounded as for person reads. It reaches only the
platform webhook, never person platform history, because error text can
quote warehouse data. The server page documents the webhook stream.
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 13, 2026
Person reads never return run_failed, but the schema's event union now
includes it, so the UI history mapping and kind filter drop it.
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 13, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 13, 2026
@kvlonge
kvlonge added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit 6d80a42 Nov 13, 2026
54 checks passed
@kvlonge
kvlonge deleted the fix/tag-job-write-set branch November 13, 2026 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant