[echoomegaprime]

Echo fleet #49604. Not deployed. CertForge goes out through its own governed release.

Problem

When the last dispatch attempt errored (for example subscriber_worker_claim_denied) while the run's reservation was still BOUND, finish_dispatch() marked only the dispatch FAILED. The run itself was never transitioned. As a result:

• the run stayed QUEUED forever;
• the metered run was never compensated;
• every "queue idle" check stayed blocked, including echo-swarm-recert (#46741), which has been exiting with STOP CertForge queue busy while the swarm's production certificate has expired.

On FORGE, 7 runs from 2026-10-09 to 10-30 are stuck exactly this way: dispatch FAILED after 3 attempts, reservation BOUND, execution never started, no compensation.

Fix

• New helper _terminalize_unstarted_run(): gives the same outcome as the existing lease-expiry exhaustion path. The run moves to INFRASTRUCTURE_FAILURE with outcome INFRA_FAILED, the reservation is RELEASED, and the meter is compensated using the same dispatch-crash key, so a run can never be compensated twice.
• finish_dispatch(): calls it for an error on the last attempt while the reservation is BOUND. The other branches are unchanged.
• _recover_expired_dispatches(), which runs on every claim_dispatch: reconciles runs the old code already orphaned (reason dispatch_failed_run_orphaned, with an audit event). It is idempotent.

The existing lease-expiry block is untouched.

Tests

• 4 new tests in tests/test_dispatch_orphaned_runs.py. On unmodified main, 3 of them fail, with the run left QUEUED.
• The 7 subscriber and dispatch test modules pass, 107/107, and 6/6 mutants are killed.
• A read-only dry run of the reconciler's selection against the live FORGE DB matches exactly the 7 orphans and nothing else.

Release notes

• The deployed release d8ad9fd8 has the same subscriber.py as main (sha 482a0b4b…), so this applies cleanly.
• On the first claim_dispatch after deploy, the 7 orphans move to INFRASTRUCTURE_FAILURE and each one's metered run is compensated once.
• After that, re-run echo-swarm-recert.
• PR subscriber: do not mask a failed BEGIN with 'cannot rollback' (fleet #46912) #40 (#46912) is also still waiting for a release.

🤖 Generated with Claude Code

https://claude.ai/code/session_01C4YSVhYLwZidNF5snQ4eec

[echoomegaprime]

Integration check — 2026-11-10, isolated FORGE worktree

PR #40 (subscriber failed-BEGIN rollback guard) and PR #41 (terminalize exhausted QUEUED dispatches) were applied sequentially onto the current origin/main commit 6515544d0083edde1eefba8bd89ab3d5c95ccf4b in a separate worktree /tmp/cf-pr40-pr41-compat-lmpqxtw9. The source checkout with unrelated tracked edits was not modified.

Both git apply --check passes succeeded and both patches applied without conflicts. The focused tests tests/test_subscriber_migration_rollback.py and tests/test_dispatch_orphaned_runs.py ran together: 7 passed in 4.23s. PR #40 patch SHA-256 55c8377de5d2e3672291fcab777b29766526f6aa2799d61838dd8236451824c1; PR #41 patch SHA-256 1b30ca42dd810a32ebe5deda92592578a0f0e7ea8a1a53fa46066e761d301103.

A broader subscriber-governance test invocation did not return a reliable tool result, so a full 107-test regression claim is NOT made. Current exact-head hosted CI, release certification, and production promotion were not run; the PRs remain unmerged.

Current Certification Forge readback: most recent sampled release verdict is signed, signature-valid and NOT_READY for mandatory_rule_failed:production_e2e / production_e2e_attestation_missing. Successful patch compatibility does not satisfy that release gate.

Next: rerun bounded subscriber suite with inspectable result; establish exact merged candidate and passing CI/security; use CertForge signed PRODUCTION_READY and rollback evidence before any merge/promotion.

[echoomegaprime]

Independent composite sandbox checkpoint — 2026-11-10 21:31 UTC

On the current upstream main 4f7d99e2dd4422f3d81cf5b709581be963f5818a, a disposable FORGE /tmp checkout fetched exact PR #40 6b53ea8fe9c285f4cdcf2f2a02ef6b64a9bbccea and this PR #41 3adb7e0c4931a61254a6786cf40bc53416f29aa7. Both commits applied sequentially via git cherry-pick --no-commit, without conflicts. No production files, DB, service or GitHub refs changed.

Combined src/echo_certification_forge/subscriber.py SHA-256: c8f088b2888c171e80b3ebdc6370752f35113fde288b2cfc894178c1bfe282a1. Combined staged diff passed git diff --cached --check; Python AST parse passed.

Combined-head focused regression results (not just separate PR CI):

python3 -B -m pytest -q -p no:cacheprovider tests/test_subscriber_migration_rollback.py tests/test_dispatch_orphaned_runs.py
....... [100%]
7 passed in 11.75s

A broader governance-module test attempt failed at the remote execution tool boundary and returned no reliable pytest result; do not interpret it as a test failure or pass. Separate PR CI runs are green, but the combined candidate does not yet have current hosted CI, complete regression coverage or production-ready signed certification.

Next gated sequence: isolate and retain one exact merged candidate, run complete subscriber/dispatch regression suite and current hosted CI, review critical release diff, verify backup/rollback and owner release authorization, then run target-specific Certification Forge signed end-to-end verification before production deployment. Seven historical orphaned runs should be reconciled only by the reviewed deployed code with meter compensation readback and no ad-hoc DB edits.

ECF captured and independently acknowledged this sandbox evidence: evt_214da1fb6c9442eeb4015573a73953ba.

[echoomegaprime]

Cross-PR validation update — 2026-11-11 00:20 UTC

Reused the existing isolated FORGE checkout with main 4f7d99e2dd4422f3d81cf5b709581be963f5818a, this PR head 3adb7e0c4931a61254a6786cf40bc53416f29aa7, and PR #40 head 6b53ea8fe9c285f4cdcf2f2a02ef6b64a9bbccea. A single idempotency-guarded detached run wrote a durable receipt and 67 tests passed, exit 0, in 71.54s across test_subscriber_migration_rollback.py, test_dispatch_orphaned_runs.py, and test_p7_subscriber_governance.py. Independent readback verified the receipt, combined subscriber.py SHA-256 c8f088b2888c171e80b3ebdc6370752f35113fde288b2cfc894178c1bfe282a1, and git diff --cached --check exit 0. This supersedes only the earlier test-result-unavailable scratch attempt, not the release gates.

Not merged/deployed: this is focused sandbox evidence, not full combined CI, reviewed admission, backup/rollback, or signed production end-to-end attestation. Latest inspected CertForge run remains NOT_READY. No production mutation or protected approval was performed.

ECF receipt evt_4f2830418e7f4a80a851d2bf107113ae.

[echoomegaprime]

November 10, 2026 — expanded PR #40 + #41 integration verification on FORGE

Scope: isolated worktree /tmp/cf-pr40-pr41-compat-lmpqxtw9 based on echo-certification-forge current main commit 6515544d0083edde1eefba8bd89ab3d5c95ccf4b. The open PR #40 and #41 patches were applied in that order with clean git apply --check preflights; no production source, services, signed evidence, or branch references were changed.

Focused patch regressions: tests/test_subscriber_migration_rollback.py + tests/test_dispatch_orphaned_runs.py → 7 passed (previous recorded execution).

Broader subscriber-governance suite: tests/test_p7_subscriber_governance.py, 60 collected, executed deterministically in four consecutive 15-test bounded batches using python3 -B -m pytest -q -p no:cacheprovider with worktree-local PYTHONPATH:

• Tests 1–15: 15 passed in 12.70s

• Tests 16–30: 15 passed in 17.77s

• Tests 31–45: 15 passed in 14.84s

• Tests 46–60: 15 passed in 22.88s

Result: 60/60 broader tests passed, plus 7/7 focused patch regressions. The tests ran in independent bounded processes; no single-process whole-suite claim is made. Both PRs remain OPEN and UNMERGED. This is source regression evidence, not exact-candidate hosted CI, production end-to-end attestation, or signed PRODUCTION_READY certification.

Remaining gates: reconcile both PR heads against the current default branch, obtain a reviewed immutable integration candidate and exact-head CI/security test results, determine whether current CertForge production-e2e attestation requirements can be satisfied, verify backup/rollback admission, and promote only with signed release evidence. The live CertForge API reports healthy custody/signing architecture, but recent inspected signed verdicts are NOT_READY because production_e2e_attestation_missing.

Next: independent code-review/CI on the combined candidate; no protected release action until all gates pass.

[echoomegaprime]

Integration coordination (Nov 10, 2026): The exact-target combined draft is now PR #49: #49, commit 0ab617ab6ed97e1a40e5cb3d160215dc93dc86d5, parent/main 4f7d99e2dd4422f3d81cf5b709581be963f5818a. It includes precisely the changes from #40 and #41; independent exact-canonical clone tests passed 67/67 (7 focused + 60 subscriber-governance). Local Git tree and GitHub tree both 38542565dc2725132d44571ca4883afaa958ada0. Use #49 for combined CI and integration review rather than independently merging #40 and #41 without reconciling the overlap. No merge or production deploy has occurred. Exact-head CI is in progress; signed CertForge PRODUCTION_READY and production_e2e attestation remain required.