Skip to content

feat(orchestration): job owner, skip overlap and on_failure retries - #1453

Merged
kvlonge merged 5 commits into
mainfrom
feat/job-run-settings
Nov 13, 2026
Merged

kvlonge merged 5 commits into
mainfrom
feat/job-run-settings

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Why

Jobs need a visible owner, a way to stop scheduled requests piling up behind a long run, and retries for transient failures. The job pages showed placeholder policies the server never had.

Changes

  • Jobs and groups accept owner, overlap="queue" | "skip" and on_failure=on_failure(retries=1..10, retry_delay_seconds=0..86400). Publication rejects a blank owner, an unknown overlap and out-of-range retries.
  • With overlap="skip", a schedule tick or sensor request that arrives while a run of the job is active starts no run. The tick records skipped with the run it yielded to. Sensor checks list skipped requests; sensors on new source data, stale sources or custom code ask again once the job is idle. Manual runs, reruns, backfills and catch-up ticks still queue.
  • A failed or partial top-level run of a job with on_failure queues a rerun of its failed and skipped nodes after the delay, shown as a retry_delay queue wait. Cancelled runs and group members don't retry on their own; groups retry with their own policy.
  • Job list and detail reads return owner, overlap and retries; tick and sensor reads return what a skip yielded to. Skipped ticks follow covered ticks' 30-day retention.
  • UI: the jobs list and job page show overlap and retries, the tick log and sensor checks show skips, and the queue shows retry waits.
  • The control store schema changes; pre-release stores are recreated as before.
  • Docs: a new server page section; docs skill and client regenerated.

DuckShop: this adds settings rather than fixing a project shape, value or lifecycle, so no edge case is added.

Verification

  • Orchestration tests on SQLite and PostgreSQL: tick skip and catch-up, sensor skip, retry chain, cancel, and the job list and detail reads.
  • Integration tests: settings publish into the job document, and invalid values are rejected.
  • UI: svelte-check, vitest, and screenshots of the jobs list, job page and a skipped tick.
  • make check-ci on the final head.

Work in progress, not ready for review.

Done: owner/overlap/on_failure authoring and validation (retries 1-10,
delay 0s-24h), snapshot definitions and job reads; schedule ticks and
sensor requests skip while a run of a skip job is active (catch-up ticks
still queue); failed or partial top-level runs queue a rerun from failure
after the delay, shown as a RetryDelay queue wait.

Validated so far: targeted orchestration tests on SQLite and PostgreSQL
(tick planning, schedule skip, sensor skip, retry chain and cancel).
Not yet done: Python boundary tests, docs, regenerated client, review,
full gate.
Python Job and JobGroup publication reads plain owner, overlap and
on_failure fields. A person's rerun of a retry starts without the retry
delay, and a retry is queued at most once per failed run. Docs cover
which sensors ask again after a skip.
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 13, 2026
@kvlonge
kvlonge enabled auto-merge November 13, 2026 18:26
@kvlonge
kvlonge added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit dba15f4 Nov 13, 2026
98 of 100 checks passed
@kvlonge
kvlonge deleted the feat/job-run-settings branch November 13, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant