Report terminal iteration outcomes in logs and Prometheus - #486
Open
THardy98 wants to merge 3 commits into
Open
Conversation
THardy98
added this pull request to stack #487
September 11, 2026 14:02
THardy98
changed the title
THardy98
marked this pull request as ready for review
September 11, 2026 14:48
Prometheus vectors bind label values positionally. The upcoming iteration outcome metric reuses one cached vector across tagged handlers, so nondeterministic Go map iteration can attach values to the wrong labels. Sort merged tag keys before deriving the value slice to keep the label schema stable.
Report attempted, succeeded, and failed logical iterations with rates and achieved successful throughput when a run reaches its configured end. Keep this operational reporting separate from the CLI failure-tolerance policy.
Expose terminal logical-iteration outcomes through omes_iterations_total with bounded scenario, outcome, and normalized status_code labels. Record once after retries finish and omit cancellation-abandoned iterations. Keep full terminal errors in structured logs while the metric labels remain aggregation-safe.
THardy98
force-pushed
the
feat/iteration-outcome-observability
branch
from
September 11, 2026 16:20
a7c7c28 to
2e8c45b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
The load driver extends PR #461's basic degraded-run warning with structured terminal iteration failures and an aggregate attempted, succeeded, failed, success-rate, failure-rate, and successful-throughput summary. It also exports
omes_iterations_totalwith boundedscenario,outcome, and normalizedstatus_codelabels.The local Prometheus handler orders tag names and values deterministically because cached metric vectors bind label values positionally. Metrics record one terminal outcome per logical iteration after retries finish; iterations abandoned by cancellation are omitted. Full wrapped errors remain in logs rather than metric labels.
This is the top layer of native stack #487 and depends on PR #461. Review PR #461 first for the iteration-tolerance behavior, then this PR for observability.
Why?
When operators opt into continuing after terminal iteration failures, they need explicit visibility into degraded runs. Structured logs retain actionable errors, while the bounded Prometheus dimensions support aggregation without introducing unbounded cardinality.
Checklist
Closes: N/A. This is the observability follow-up to PR Allow CLI load runs to continue after terminal iteration failures #461.
How was this tested:
cmd/omes, loadgen, metrics, and race-detector tests pass locally on the rebased stack.Any docs updates needed?
docs/running.mdwith the aggregate summary and Prometheus metric contract.