Skip to content

Allow CLI load runs to continue after terminal iteration failures - #461

Open
THardy98 wants to merge 2 commits into
mainfrom
fix/graceful-driver-failure
Open

Allow CLI load runs to continue after terminal iteration failures#461
THardy98 wants to merge 2 commits into
mainfrom
fix/graceful-driver-failure

Conversation

@THardy98

@THardy98 THardy98 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

What was changed

omes run-scenario and run-scenario-with-worker now accept --iteration-failure-policy=fail-fast|continue. Existing commands remain fail-fast by default; operators running sustained load can explicitly select continue. Retry count and terminal-failure policy remain independent.

The loadgen library still defaults to fail-fast. When continuation is enabled, it returns a typed IterationFailuresError containing attempted, succeeded, failed, and elapsed values. The CLI treats only that degraded completion as successful in continue mode, so cancellation, timeout, validation, connection, joined, and other run-level errors remain fatal. Every terminal failure remains an error log, and a completed degraded run emits a structured warning with its outcome counts.

The operational observability changes are intentionally separated into stacked follow-up PR #486.

Why?

A terminal iteration failure currently ends an omes run-scenario invocation. In a Kubernetes load-driver deployment, that turns a transient workflow failure into a container restart and stops the configured load run early. This PR adds an explicit continuation mode for that use case while preserving existing fail-fast behavior for unflagged CLI, CI, and automation callers.

Checklist

  1. Closes [Feature Request] Gracefully handle load driver failures #453

  2. How was this tested:

    • Focused loadgen tests cover fixed-iteration and duration-based degraded results.
    • CLI tests cover the zero-value and registered fail-fast defaults, explicit continue, explicit fail-fast, invalid policy, wrapped degraded result, joined error, and unrelated error cases.
    • A logger-observer test verifies the final degraded-run warning and its attempted, succeeded, and failed fields.
    • Full cmd/omes tests and focused race-detector coverage pass locally.
    • All 46 GitHub Actions checks pass on the revised head.
  3. Any docs updates needed?

    • Updated docs/running.md and docs/authoring-scenarios.md for the opt-in policy, warning behavior, and relationship to retries.

@THardy98
THardy98 force-pushed the fix/graceful-driver-failure branch 3 times, most recently from 2a8c379 to 1d9f0ae Compare September 10, 2026 13:57
@THardy98
THardy98 marked this pull request as ready for review September 10, 2026 14:39
@THardy98
THardy98 requested review from a team as code owners September 10, 2026 14:39
@THardy98
THardy98 marked this pull request as draft September 10, 2026 14:40
@THardy98 THardy98 changed the title Gracefully handle load driver iteration failures Continue CLI load runs and report terminal iteration failures Sep 10, 2026
@THardy98
THardy98 marked this pull request as ready for review September 10, 2026 17:10
@THardy98
THardy98 force-pushed the fix/graceful-driver-failure branch from 1d9f0ae to 9797459 Compare September 11, 2026 14:01
@THardy98 THardy98 changed the title Continue CLI load runs and report terminal iteration failures Continue CLI load runs after terminal iteration failures Sep 11, 2026
@THardy98
THardy98 added this pull request to stack #487 September 11, 2026 14:02
ContinueOnIterationFailure already lets GenericExecutor finish all configured work, but callers can only identify the degraded completion by matching an error string.

Return IterationFailuresError with outcome counts and elapsed time so callers can apply policy without changing the library default. Emit a final warning with the outcome counts so degraded completion remains visible.
Expose GenericExecutor continuation through a CLI policy flag while keeping fail-fast as the default for existing commands, CI, and automation.

Continue mode treats only the typed degraded completion as successful; cancellation, timeout, joined, and other run-level errors remain fatal.
@THardy98
THardy98 force-pushed the fix/graceful-driver-failure branch from 9797459 to 36a18f4 Compare September 11, 2026 16:20
@THardy98 THardy98 changed the title Continue CLI load runs after terminal iteration failures Allow CLI load runs to continue after terminal iteration failures Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Gracefully handle load driver failures

1 participant