Skip to content

fix(subscriber): preserve execution ownership during historical orphan recovery - #61

Draft
echoomegaprime wants to merge 6 commits into
agent/certforge-worker-failure-reason-redaction-20261112from
agent/certforge-orphan-ownership-guard-20261112
Draft

echoomegaprime wants to merge 6 commits into
agent/certforge-worker-failure-reason-redaction-20261112from
agent/certforge-orphan-ownership-guard-20261112

Conversation

@echoomegaprime

Copy link
Copy Markdown
Owner

Independent CertForge release-gate safety patch

Stacked on draft #58 at exact parent 76b85df4185addbba0f8b700750c38c019e083ec, preserving #49 and #58 heads.

The historical orphan selection in #49 previously matched any QUEUED/FAILED/BOUND run, even an inconsistent record with an execution-start timestamp, worker ownership, worker attestation metadata, dispatch lease, or claim token. Such a record must not be terminalized and compensated as unstarted.

This least-change patch adds negative, fail-closed SQL predicates for ownership and execution markers before recovery, preserving run state, reservation, metering and the existing audit evidence for inconsistent cases. No production database modification, heartbeat fabrication or policy override.

Exact local test evidence (FORGE disposable checkout): 16/16 tests passed in 35.96 seconds, including 12 new parameterized contradictory-state cases plus 4 existing recovery tests. git diff --check and compileall passed. Verified source Git blob 66ff83dc4d2e527369a96f45fb86670bf9d900a5 and test blob 7286c2bcd96444ca971d8d27aee3d7a65f0cb3a7 against the exact tested bytes before publishing.

Live read-only predicate safety: seven existing QUEUED/FAILED/BOUND records have none of the 12 guarded ownership markers, so this correction preserves their eligibility. The eight separate CANCELLED/PENDING/BOUND reservation cases are not selected and require independent accounting review.

Gates: DRAFT, requires independent review, hosted exact-head CI and CodeQL, integration with deployed-ancestry PR #60, formal release authorization and authentic signed-worker recovery before any promotion. This PR does not approve or deploy the fix.

Related: #49, #50, #51, #52, Swarm PR #16.

Copy link
Copy Markdown
Owner Author

Independent CertForge gate review (2026-11-12, read-only; current head 272bea3674f6cdecd4ca0423b30b745cbf755cab): hosted run https://github.com/echoomegaprime/echo-certification-forge/actions/runs/43220151628 is now completed/success, including the full deterministic suite, P1, P6 and P7. Exact-head check-runs contain test=success and ECHO Release Sentinel=neutral, no CodeQL result; GitHub review list contains zero independent approvals. This closes the previously pending CI question, not release admission.

Additional fail-closed review item: _recover_expired_dispatches() protects worker/lease/attestation/start fields, but the historical-orphan selection predicate does not check subscriber_run_reservations.consumed_at. That field may independently indicate prior consumption despite contradictory QUEUED/FAILED/BOUND states. Production read-only inventory shows consumed_at is null for all 15 currently BOUND rows and populated for 2,526/2,560 RELEASED rows; no existing stranded row is currently implicated. Before review, establish the column's lifecycle semantics and add a negative synthetic regression for a QUEUED/FAILED/BOUND record with only consumed_at non-null; if it denotes consumption, add AND r.consumed_at IS NULL to the selector. This is a source-level potential inconsistency, not proof of a production misclassification. No production data was edited, and no new tests are claimed in this update.

Retain draft/HOLD until independent review, real exact-SHA CodeQL, CertForge policy certification and reversible deployment authorization. No approval or merge performed.

Copy link
Copy Markdown
Owner Author

Independent CertForge gate repair — 2026-11-12 14:34 UTC. New exact draft head b98d1cedd6532ac29fd7dfb33adb340a953975fc (source change 7119fe7, test commit b98d1ce), superseding 272bea3. Two independently verified fail-closed cases now covered: (a) historical QUEUED/FAILED/BOUND orphan reconciliation excludes consumed_at and execution/ownership markers; (b) final-attempt finish_dispatch refuses to terminalize/compensate contradictory evidence of prior consumption, execution start or worker ownership. Synthetic negative tests use only disposable SQLite fixtures; both source files' Git blobs were fetched back at the exact head. Independently fetched this SHA into a disposable FORGE checkout and ran python3 -m pytest -q -p no:cacheprovider tests/test_dispatch_orphaned_runs.py tests/test_subscriber_migration_rollback.py: 23 passed in 23.10 seconds, zero failures; workspace auto-cleaned. Hosted exact-head workflow 43238392872 was in progress, not yet passing at last read. No CodeQL exact-head success and no independent review approval established. No production change. Request independent security/owner review and hosted test + CodeQL closure before any recovery release; do not self-approve/merge.

Copy link
Copy Markdown
Owner Author

Independent CertForge release-gate review — 2026-11-12 15:25 UTC. HOLD; new reproducible lease-expiry ownership defect at exact head b98d1cedd6532ac29fd7dfb33adb340a953975fc.

The #61 guards protect FAILED historical orphans and final finish_dispatch() errors, but _recover_expired_dispatches() has a separate CLAIMED lease-expiry path. Its SELECT currently includes every CLAIMED + expired dispatch with r.state='BOUND' and runs.state='QUEUED', regardless of reservation consumed_at, execution_started_at, worker_id, attestation, signing, or heartbeat fields. With exhausted attempts, that path transitions the run to INFRASTRUCTURE_FAILURE, releases the reservation, and compensates metering, even when those fields contain contradictory execution/ownership evidence. With attempts remaining, it can requeue work despite that evidence.

Actual isolated repro (no production mutation): On FORGE, shallow-cloned this exact SHA into a disposable temp directory. A synthetic pytest used existing _run() fixture (max_dispatch_attempts=1), claimed a dispatch, populated subscriber_run_reservations.consumed_at, set only the dispatch lease expiry to year 2000, then called claim_dispatch() to trigger expiry recovery. The expected safety assertion (run still QUEUED, reservation and usage retained for review) FAILED: run became INFRASTRUCTURE_FAILURE. Pytest returned exit 1 in 1.19 s; the assertion was 'INFRASTRUCTURE_FAILURE' == 'QUEUED'. The temporary checkout was deleted. This is a genuine negative test finding, not a passing regression suite.

Least-change correction requested before promotion: Apply the same fail-closed reservation ownership/consumption predicate to the expired-CLAIMED recovery query. If any contradictory evidence exists, do not terminalize, compensate, or requeue; put dispatch into an auditable manual-review state while preserving run, reservation, and immutable evidence. Also check the pre-exhaustion finish_dispatch() retry path, which currently sets PENDING without consulting the new ownership predicate. Add synthetic tests for both exhausted and retryable dispatches with consumed_at, execution_started_at and worker identity; preserve successful behavior for genuinely unstarted attempts.

Separately, current exact-head hosted Actions test completed successfully (https://github.com/echoomegaprime/echo-certification-forge/actions/runs/43238392872), while check-run API shows only test=success and ECHO Release Sentinel=neutral, no CodeQL. Zero independent review submissions. Do not merge/deploy pending regression correction, external approval, CodeQL and normal release gates.

Fail closed on lease expiry and retry errors when worker ownership, reservation consumption or execution markers exist. Preserve run/reservation/metering evidence and add synthetic negative and positive recovery controls. Reviewed locally: 30 focused pytest cases passed; Git diff check clean.

Copy link
Copy Markdown
Owner Author

Independent CertForge release-gate update — 2026-11-12 15:37 UTC.

New tested draft head published: c95dd7d2a971ea26d827382a4f1d5ec96fa327d6 (parent b98d1cedd6532ac29fd7dfb33adb340a953975fc). The two changed files' Git blob SHAs (57b0a45a040ec6a31d3f46bb696f3f9ab3dc8d2f, fee03642b3ff9f311244d2fc213a2a60a66beed0) and tree 31e5337caf8d20e1132ea3aa7c525cf236f06e90 exactly match the independently tested disposable FORGE checkout.

Defect reproduced first: Expired CLAIMED dispatch with populated consumed_at and exhausted attempts was incorrectly terminalized as INFRASTRUCTURE_FAILURE (negative pytest failed on prior head). Fix: The separate lease-expiry path now fails closed on contradictory reservation execution/worker ownership metadata; it preserves the original run, bound reservation and metering, sets dispatch failed for review, and records the review requirement in audit. Pre-exhaustion finish_dispatch() no longer retries with contradictory ownership and audits this hold. No production mutation.

Verified: 30/30 focused tests/test_dispatch_orphaned_runs.py tests passed in 28.62 seconds on FORGE; Python compile and git diff --check passed. Tests include eight new negative cases plus two positive controls for truly unstarted expired leases. The hosted exact-head CI run is in progress, not yet passed: https://github.com/echoomegaprime/echo-certification-forge/actions/runs/43249647796 .

Review gates: Still a draft; no independent approvals or established exact-head CodeQL. Do not merge or deploy until independent review/security certification. Source PR #49's head is unchanged and does not yet include this dependent guard.

Copy link
Copy Markdown
Owner Author

Independent CertForge release-gate update — 2026-11-12 16:20 UTC (read-only; HOLD).

  • Exact PR fix(subscriber): preserve execution ownership during historical orphan recovery #61 head c95dd7d. GitHub Actions run 43249647796 is now completed/success; its test job completed the full deterministic suite and P1, P6, P7 steps successfully.
  • Direct exact-SHA check-runs API returned only successful test and neutral ECHO Release Sentinel. No CodeQL check-run at this head; GitHub's code-scanning/default-setup reports not-configured, and .github/workflows currently contains only ci.yml. Older CodeQL analyses on other PR SHAs are not exact-head evidence.
  • Independent approvals: 0. Deployed CertForge marker still d8ad9fd, i.e. this fix is not deployed.
  • Read-only production corroboration: 7 QUEUED/FAILED/BOUND historical jobs (3 attempts each, no selected execution-ownership markers); 8 separate CANCELLED/PENDING/BOUND reservations outside this fix; 3 ACTIVE enrolled runner identities and 0 fresh authenticated heartbeats (latest 2026-07-26T14:05:14.630191Z).
  • Next gate: independent review of all ownership/retry/compensation transitions; enable reviewed exact-head CodeQL analysis; integrate against the reviewed deployed-ancestry candidate DRAFT: reconcile deployed CertForge ancestry with main; retain readiness + production safeguards #60; confirm staging and rollback with authentic worker enrollment and protected approvals. No manual DB correction, unauthorized queue cleanup, service promotion, forged attestation or self-approval.

The FORGE connector's repeat disposable local-test invocation encountered an execution error in this inspection; no new local passing tests are claimed here. Prior separately recorded focused-test evidence and current successful hosted CI remain distinct.

Copy link
Copy Markdown
Owner Author

Independent release-gate source review, 2026-11-12 18:30 UTC. Exact reviewed stack: PR #61 head c95dd7d2a971ea26d827382a4f1d5ec96fa327d6; stacked PR #62 head 9417dc5a06159291384d11f569a2ffdb5b15c108.

Additional ownership-custody review requirement (source-level, not a demonstrated production fault): In SubscriberGovernance._recover_expired_dispatches, a CLAIMED/expired dispatch with reservation execution evidence is correctly blocked from metering compensation and set to review-required FAILED, but its unconditional UPDATE subscriber_run_dispatches SET state=?, claim_token=NULL, lease_owner=NULL, lease_expires_at=NULL erases dispatcher-claim ownership fields. The subsequent audit entry records ownership_review_required, attempts and compensation flag, not the prior lease owner/lease timestamp or a non-secret durable claim correlation. finish_dispatch similarly clears claim fields after marking ownership review. Existing test_expired_claim_with_ownership_requires_review verifies no terminalization/retry/compensation, but does not assert durable independent custody of the original dispatcher claimant.

This is a potential forensic/ownership-preservation gap, not grounds to infer the seven historical QUEUED/FAILED/BOUND production jobs were misclassified. Please determine whether an existing immutable event already captures sufficient claim identity and time. If not, record a bounded, redacted owner/lease correlation before clearing the active claim; do not persist a replayable raw claim token or keep the stale token executable (since finish_dispatch matches it). Add synthetic tests for owned expired CLAIMED, stale-worker attempt, unstarted positive control, audit custody and no leaked secrets. Independent maintainer/security review is required; this comment is not an approval or authorization to deploy.

Fresh FORGE read-only check: seven historical QUEUED/FAILED/BOUND jobs still have zero selected ownership/consumption markers; eight separate CANCELLED/PENDING/BOUND records remain outside this patch. No production DB changes made.

Copy link
Copy Markdown
Owner Author

Independent custody follow-up (2026-11-12 ~19:35 UTC): new draft PR #63, stacked on exact-head #62 9417dc5a06159291384d11f569a2ffdb5b15c108, repairs the prior finding in this thread. The expired CLAIMED dispatch recovery still invalidates replayable claim/lease fields, but now records prior dispatcher owner, lease expiry, and only a SHA-256 digest of the high-entropy claim token in the append-only, hash-chained audit. Raw claim token is never logged. FORGE disposable checkout tests: 32/32 passed in 32.83s; compilation and whitespace checks passed. GitHub file blob SHAs exactly match independently tested files (a6f0de931b81b1714d161bede82c2bea0eb127bd and 388687ab93b1cc2a707543954d34ff4db11c3c45). #63 exact commit a0515d875c2a384965229eb5716331a776ab8190; hosted CodeQL job passed, full CI still running at last check. Draft only: no production or approval implications.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant