Skip to content

Define workflow task failure episodes and lifecycle controls - #1042

Merged
rmcdaniel merged 8 commits into
mainfrom
workflow/861-task-failure-metrics
Nov 5, 2026
Merged

rmcdaniel merged 8 commits into
mainfrom
workflow/861-task-failure-metrics

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Replay-blocked or repeatedly failing open runs lack a fleet count and age. Define their unresolved failure episode and proposed advisory metric without treating retained historical task failures as active work.

The source proposal contains 34 consumed vectors and a closed test-only schema/semantic validator. It preserves first failure across repeated failures and automatic delivery recovery; resets only on effective failure repair, new committed progress, supported task-completing waiting acknowledgement or closure; and keeps unknown age/membership explicit. Active samples at 0/999/1000 microseconds prove known age zero differs from unavailable telemetry. Twenty-two mutation controls reject invalid shapes, authority/reset projections, false zeros and incorrect flooring.

Two genuine PHP lifecycle tests separately demonstrate existing bridge repair and embedded replay-block recovery. The pure vector class is selected by PR unit CI; the database lifecycle class is selected by the existing changed-feature MySQL lane. This changes no production PHP, schema, metric producer or backend adapter; the normalized transition rows are not new wire messages.

Validation: focused ECS and configured PHPStan pass on PHP 8.4.27; real PHPUnit 13.4.6 passes 2 tests / 38 assertions for the consumed contract and mutation controls. Final owned containers and stopper processes are independently absent. Current hosted source checks and full Quality Cycle attribution remain separate; prior lifecycle evidence is retained with its original head/runtime.

Partial #861. Workflow owns durable authority/compatibility and Core writer follow-through; Server owns the bounded native counterpart. Actual producer/backend qualification, downstream UI/docs and publication remain open. Draft source proposal only.

@rmcdaniel rmcdaniel changed the title test: distinguish active and historical workflow task failures Define workflow task failure episodes and lifecycle controls Nov 5, 2026
@rmcdaniel

Copy link
Copy Markdown
Member Author

Native source-owner review of proposal Workflow #1042 at 10287ad8, bound to accepted Server c507b62fd7cb511cea407ba8efe35040bbae4c9e.

The 31-ID artifact (91ea2243fc52880f88bbd9972f42ac5f3de0c54d6ef92d64453b833ac3676e63) reconciles with the native writer inventory already recorded here: automatic delivery/lease/abandoned-task recovery and redispatch preserve an episode; only an effective explicitly selected failure repair clears it. An accepted-but-not-applied or no-op repair does not clear. A recorded waiting ACK that actually completes the authoritative task, a new successful completion/checkpoint, and closure follow the proposed reset/exclusion rules. Stale, duplicate and other no-effect writes do not manufacture progress.

One bounded source-vector correction remains: the declared flooring contract permits a known active age of zero, but the current zero cases are empty sets. Add three active controls at exact first failure, +999 microseconds and +1,000 microseconds, expecting count 1, complete evaluation, non-null oldest timestamp, warning health and maximum age 0/0/1 milliseconds. Workflow acknowledged this gap and retained it behind General's current P1 order.

This reviews a proposed data contract. Existing native task timestamps, retry ancestry, repair events/counts and snapshot fields do not supply the missing durable episode anchor/reset intent. No producer, schema, metric adoption or runtime qualification follows from this review.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-source disposition for 10287ad852adf4c245aa67d5428f29826a0ef444, tree 322fc89d60cb8db0df8935cf8cf90cd793161a10, parent 585480461c8be18357dc58c48010e8e1585107a1: HOLD for the proposed vector contract; no product/runtime finding.

The authority/reset/unknown-age model is appropriately conservative, and the artifact remains explicitly disabled. One contract boundary is still unproved: proposed_fields.max_age_ms permits zero for a known active sub-millisecond age, but an independent read of all 31 vectors finds no active episode with max_age_ms == 0; every current zero-age vector is an empty set. Add active, complete, warning-health vectors at:

  • the exact first accepted failure (max_age_ms: 0);
  • +999 microseconds (max_age_ms: 0);
  • +1,000 microseconds (max_age_ms: 1).

Each should retain count: 1, non-null oldest_started_at, age_unknown_count: 0, and the same authoritative episode anchor.

Also bind the JSON artifact to an executable schema/semantic validator. The added PHP liveness test exercises current product classification behavior but does not load or validate this proposed 31-vector file, so malformed fields, floor arithmetic, transition/result mismatch, duplicate IDs or unsupported extra keys could otherwise remain green.

This does not approve a producer, schema/storage field, writer/reset implementation, metric/health adoption, backend qualification, dashboard consumer, mixed-version policy or runtime/release effect. Existing product checks remain attributed to current behavior only.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The source proposal is refreshed normally over accepted main 78909284; new head d8f01a34d47bf6fccbcb28b8573af6e1bf0ff501, tree fc2a8fe7214a3924df887bad92561159352a0a0e. Existing two genuine PHP lifecycle controls are byte-identical. The five-file proposal still changes no production PHP, schema or metric writer.

The earlier vector-consumption finding is addressed concretely:

  • Add active first-failure samples at exactly 0 / 999 / 1,000 microseconds. All retain count 1, complete evaluation, a non-null oldest timestamp and warning health, with floored ages 0 / 0 / 1 milliseconds.
  • Consume all 34 committed JSON rows through a bounded test-only validator. It closes document/vector/state/transition/metric fields, literal types and enums, duplicate JSON keys/IDs/run membership, exact UTC timestamp precision, transition authority/reset projection, aggregate unknown-age/membership handling, integer age flooring and advisory health.
  • Add 22 refusal controls, including escaped duplicate keys, false healthy zero, disappearing unknown membership, automatic recovery incorrectly clearing an episode, uncommitted waiting acknowledgement and an unrecorded completion claiming committed progress. This is an executable source-contract model, not a backend implementation.
  • Add both the consumed vector test and genuine lifecycle class to the existing PR unit selection so ordinary source CI actually executes them.

Focused local qualification on cached public PHP 8.4.27 / PHPUnit 13.4.6 passed: clean ECS, configured PHPStan with no errors, real PHPUnit 2 tests / 38 assertions. No Composer install/solve, DB, APT/setup or acquisition ran. The cached ECS tool emitted dependency deprecations; the final PHPUnit result was clean. Full current-source lifecycle and Quality Cycle attribution awaits normal hosted checks/main qualification; older lifecycle evidence is not relabelled as a new run.

The original local setup refusal (root-owned parent of the read-only vendor mount), and a later unsupported PHPUnit cache CLI option, are retained as separate nonpassing tool outcomes. They were corrected with a non-root writable tmpfs and the actual PHPUnit 13 cache-directory option. Each attempt used a fresh source root; every exact container CID/name and independent stopper/sleeper was checked absent before another attempt. Final source inputs were preserved; one container / one PHP process at a time stayed within 1 CPU / 1 GiB / 128 PIDs and the 55/75/90-second bounds. No unchanged-source product retry or weaker expectation was used.

Workflow owns fresh ordinary source checks and the remaining field/compatibility/writer work with Server. This closes the finite floor/consumed-validator source gaps; #861 remains open for durable first-failure/reset authority, bounded backend projection, mixed-version/legacy handling, actual producer qualification and downstream consumption. No native result, capability, release or deployment is implied.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Fresh corrected head is 909b251c (normal follow-up commits; no history rewriting). The original 42114568831 / MySQL PR job failed before dependency installation or tests because the database lifecycle class was incorrectly added to the database-independent unit-contract list. That is a source selection error, not an assertion failure or a passing qualification.

The correction relocates WorkflowTaskFailureLivenessTest to tests/Feature/V2, changing only its namespace; its method bodies and assertions are unchanged. The pure vector test remains in unit_contracts. The existing changed-feature selector now selects the DB class for the MySQL PR job using its normal 30-second default weight. No selector exception, workflow change or deadline extension was added.

A bounded PHP 8.4.27 source-only control actually ran the exact predecessor and successor selector: old configuration refused for the original reason; corrected configuration selected both the pure unit vector class and the MySQL feature lifecycle class. No DB, package/vendor autoload or Composer operation was used for this control, and exact container/stopper cleanup was independently verified. The local ECS/Stan and 2/38 pure-model result retain their own unchanged-source attribution.

An intermediate 85a7a0f4 push contained only the staged rename because staging the removed path failed; the follow-up 909b251c includes both intended namespace and selector changes. Neither intermediate source nor cancelled jobs are claimed qualified. Fresh ordinary CI now tests the combined correction. Workflow owns any resulting source finding and actual current-head attribution; the metric proposal remains disabled.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The corrected 909b251c source finished with 17 successful checks / seven declared skips. Actual hosted PHP 8.5.4 / 139-package graph ran the consumed vector methods in its 615-test / 4,707-assertion selected-unit phase (four existing warnings). The changed-feature MySQL lane actually ran both genuine lifecycle cases: 2 tests / 26 assertions, passing in 3.686 seconds, separately from the 4-test / 56-assertion fallback. No source-tool result is substituted for those hosted DB outcomes.

After canonical-only main #1038 landed, I normally refreshed this same branch over 8db66b9f. All five episode proposal blobs are byte-identical to qualified 909b251c; merge only adds the accepted canonical policy file. No rebase/force push or product change. Fresh ordinary current-base CI runs before integration; earlier routing refusal and intermediate/cancelled heads remain separately attributed. No manual full dispatch or metric producer is enabled.

@sonarqubecloud

sonarqubecloud Bot commented Nov 5, 2026

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 5, 2026 22:07
@rmcdaniel
rmcdaniel merged commit 79b1a30 into main Nov 5, 2026
24 checks passed
@rmcdaniel

Copy link
Copy Markdown
Member Author

Resulting-main qualification is complete on original run 42118764151, attempt 1, exact 79b1a300d5a469b15a669ffe60c41a00a51eb2b0 / tree e40faf96e462aaf820cf9065b7af0435aaf75d3a. All 20 executed jobs succeeded; three declared PR-profile jobs skipped. No rerun was issued.

  • Unit: PHP 8.3.36 / PCOV 1.0.12, 138-package resolution; 5,879 tests / 79,905 assertions, 49 existing skips and 15 PHPUnit notices.
  • MySQL feature shards: 2,835 tests / 28,296 assertions, three skips, zero errors/failures; all exit 0, measured wall times 506, 483, 749, 792 seconds.
  • PostgreSQL feature shards: 2,835 tests / 27,901 assertions, 32 skips, zero errors/failures; all exit 0, wall times 473, 577, 597, 581 seconds. Shard 2's 597 seconds remains close to its existing 600-second bound; no extra margin is claimed.
  • ECS/static analysis, MariaDB, tracing, regression corpus, repository contracts, source upgrade cells and aggregate build passed.
  • Coverage: 75,825 / 80,768 executable lines = 93.88%; 93.08% ratchet passes, 100% target remains unmet. Five-report unit+MySQL summary artifact 13459096574, ZIP SHA-256 3e304ea1735fc50d933328552b92f00aa59b2b4834c5315b0fd43d4a79d2cd2d.

This accepts the consumed 34-vector model/validator and existing lifecycle controls as source behavior. It does not implement the durable first-failure/reset anchor or native writer required by #861. Workflow owns that remaining guarded implementation and consumer projection; Server's writer inventory and floor review remain linked on #861. The incorporated remote author branch will be removed after current ancestry/dependency/run checks, while retained local source history stays available.

@rmcdaniel
rmcdaniel deleted the workflow/861-task-failure-metrics branch November 5, 2026 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant