Skip to content

release(draft): integrate deployed ancestry, guarded queue recovery and exact-head CodeQL - #64

Draft
echoomegaprime wants to merge 41 commits into
mainfrom
agent/certforge-ancestry-recovery-integration-20261112
Draft

echoomegaprime wants to merge 41 commits into
mainfrom
agent/certforge-ancestry-recovery-integration-20261112

Conversation

@echoomegaprime

@echoomegaprime echoomegaprime commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Independent CertForge release-gate integration — DRAFT / NO DEPLOYMENT

Purpose: Combine the verified deployed/GitHub ancestry reconciliation (#60) with the guarded historical queue recovery, ownership custody, deterministic regressions, and exact-head CodeQL workflow (#63, stacked on #62/#61/#49/#41). This is a review-only integration branch, not a production authorization or Swarm #16 deployment.

Immutable identity and reproducibility

Safeguards and release gates

  • Keep immutable historical run, dispatch, reservation, metering and worker-ownership evidence; never manually mutate SQLite, arbitrarily cancel or replay claim tokens, fabricate heartbeat, or drop independent audit custody.
  • Source includes expiration-claim contradictory-execution guards, released-reservation accounting and audit records limited to non-replayable hashes.
  • Review the full deployed ancestry incorporated by DRAFT: reconcile deployed CertForge ancestry with main; retain readiness + production safeguards #60, not just the small GitHub PR diff. Independently verify deployment script rollback, attestation custody and trust-root behavior.
  • Require independent protected-PR review and exact-head CI/CodeQL, signed enrolled worker capacity, real policy-owner decision on docs: governed two-phase admission proposal for Swarm Meta OAuth (review required) #52's two-phase admission, reversible pre-production canary and explicit production window before any promotion.
  • Swarm chore(deps): bump the pip group across 2 directories with 2 updates #16 f65935510f35b035536e08b6ae292cd821f02327 remains independent and NOT_READY; authentic full-40 proof, signed HAMMER production attestation, and exact-target PRODUCTION_READY are absent. No costly 40-seat validation without approved budget/cost preflight.

This PR creates no production restart, signing operation, release exception, queue mutation or protected merge.

Current-head security follow-up and release blocker (2026-11-12)

The current SHA e6b51c683c0027cab3ed446a26fdd614476a32c6 updates two files: prior dispatcher claim-token custody now uses the existing server-peppered HMAC-SHA256 helper rather than unkeyed SHA-256, with explicit negative regressions. An isolated FORGE disposable checkout of the exact modified source/test blobs passed 2 focused tests (30 deselected). This new head has its own hosted CI run #43311519833.

Important: the GitHub Actions CodeQL job passed, but the separate GitHub Advanced Security CodeQL check is failing with one HIGH-severity alert (alert #1, py/weak-sensitive-data-hashing) at subscriber.py:889. That line is the existing _token_digest HMAC-SHA256 API-key helper, not the added historical custody code. API-key secret material is generated with secrets.token_urlsafe(policy.api_key_secret_bytes) where policy requires at least 24 bytes, and _token_pepper requires at least 32 bytes. It may be a classification false positive for random API credentials, but do not suppress or dismiss it without independent security-owner verification. Passing a scanner job does not mean the security analysis gate passed. Production promotion remains HOLD, independent approvals remain missing, and signed production E2E, fresh worker capacity and full-40 evidence are outstanding.

ECHO-OMEGA-PRIME and others added 30 commits August 30, 2026 05:22
Successful hostile acceptance reports omitted the completed_phase_gate field
required by the master collector. Publish P4 only after every core check and
cleanup succeeds, and keep exact-wrapper reports pending until enrichment
returns. Failed verification, malformed capture, interruption and nonzero
exit cannot leave a newly completed report for the master consumer.

Keep master validation, exact-source and signing rules, release verdicts and
historical acceptance reports unchanged. Add producer/wrapper/consumer
integration coverage including pending publication and failure paths.

Validation: 24 focused and adjacent tests pass; full suite 637 pass, 2 skip,
1 existing symlink test blocked by Windows privilege 1314. P1 acceptance
passes all 3 scenarios using a byte-identical source copy outside historical
artifacts. Hosted CI, dual certification and deployment remain pending.
…#22)

Release hardening so the P4 hostile acceptance gate can seal again:

- Bump the pinned python:3.12-alpine base for all six role images, the
  sandbox default image and the P4 gate to
  sha256:4c47124a8391cb7a9f571164147d154777cf012a4ece5f86097130d7a4478111
  (Trivy: 0 HIGH/CRITICAL findings on the new base).
- Bump the hash-locked cryptography pin to 50.0.0 (all PyPI wheel and sdist
  hashes locked).
- deploy_forge.sh: the dispatcher unit uses Wants= instead of Requires= on
  the API service so an API restart no longer tears down in-flight runs; the
  source-fetch git credential helper is configurable
  (CERTFORGE_GIT_CREDENTIAL_HELPER); the trusted manifest default is the
  canonical-JSON digest that run_worker pins (08ba068c...), not the raw
  file-bytes digest.
- Opt the repository into its own CertForge gate with a stdlib-only journey
  (scripts/certforge_journey.py) and pin tests (tests/test_release_pins.py).

The worker environment identity digest is label-derived and does not change
with this release (8e6466d2...).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0152mN1wMwj9vE2YmV2xZF4F
…space (#22)

The fresh P4 hostile-acceptance run for bc1b8ad built and sealed all 12
images and passed the scans, the static and runtime attack matrices and
the custody and anchor service probes. It then stopped as INFRA_FAILED:
the worker container exited before /healthz answered.

Cause: since P6 (bfc2159) the worker app opens its deployment ledger at
import time. Its default path, <package root>/var/deployments.sqlite3,
resolves to /opt/var on the read-only image root, so uvicorn died with
"Read-only file system: '/opt/var'". The last green P4 (9c07eb7) predates
P6, so no P6-era source could pass P4 until now.

Fix: the hardened probe sets ECHO_CERTFORGE_DEPLOYMENT_LEDGER to
/workspace/state/deployments.sqlite3, next to the other state paths.
Only the harness changes; production and the image stay as they are.
Reproduced with the sealed worker image: exit 1 without the variable,
healthy with it. A new test pins every worker state path to the writable
workspace; it fails without this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0152mN1wMwj9vE2YmV2xZF4F
…ndependent review

Preserve deployed trust roots, production DB safeguards, worker dispatcher resilience, and the mainline 90-second configurable startup/rollback readiness probes with staging failure log custody. Source-only reconciliation; NO deployment or gate approval.
Fail closed on lease expiry and retry errors when worker ownership, reservation consumption or execution markers exist. Preserve run/reservation/metering evidence and add synthetic negative and positive recovery controls. Reviewed locally: 30 focused pytest cases passed; Git diff check clean.
…CodeQL

Review-only two-parent integration of PR #60 and PR #63. Exact tree 3745b34 independently passed 61 focused regression tests on isolated FORGE checkout. No production release admission, deployment or reservation mutation.

echoomegaprime commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner Author

Independent exact-head gate update (2026-11-12 UTC): candidate ccf59365cb2987f2e2bc2da5e33f4fbd30eb985b, immutable tree 3745b34cb593134dccd99db46f34ff4892879a47; this tree exactly matches the earlier 61/61-test FORGE disposable integration result (prior evidence, not newly re-run). Hosted run https://github.com/echoomegaprime/echo-certification-forge/actions/runs/43310137751 has CodeQL job 131503747888 SUCCESS; CI job 131503748295 was still in progress at this check. Zero independent approvals. Actual FORGE deployment still d8ad9fd881365801354b78a758632d2a31ab6828, with both running CertForge services pointed to that exact immutable release directory. No promotion or queue mutation. Keep this draft until full CI passes, independent reviewers approve the complete deployed ancestry, canary/rollback policy #52 is formally accepted, and runtime release conditions are real.

Corrective exact-head update, 2026-11-12 UTC: Integration parent ccf59365cb2987f2e2bc2da5e33f4fbd30eb985b finished hosted CI successfully, but its separate GitHub Advanced Security CodeQL analysis check FAILED (HIGH) despite a passing GitHub Actions CodeQL job. The draft PR #64 head has advanced, by a bounded two-file HMAC custody follow-up, to e6b51c683c0027cab3ed446a26fdd614476a32c6 (tree dc7e69a253d586eee578d2560bf12eabb51ea63f); 2 focused tests passed on the exact changed blob bytes. At this update, new-head GitHub Actions CodeQL job passed, full CI was still in progress, and the separate GHAS CodeQL check remained FAILED, alert #1, py/weak-sensitive-data-hashing at the pre-existing API-key HMAC helper (subscriber.py:889). Generated API-key secrets are randomly created with a 24-byte minimum and server pepper is 32-byte minimum; independent security review must determine whether this alert is a false positive. Do not suppress, dismiss, or bypass it. No independent approvals or production release authorization. This note supersedes any impression above that all CodeQL gates had passed.

Final current-head check reconciliation (2026-11-12 UTC): CertForge draft PR #64 head e6b51c683c0027cab3ed446a26fdd614476a32c6: hosted CI SUCCESS (job 131508526846), GitHub Actions CodeQL job SUCCESS (131508527314), but independent GitHub Advanced Security CodeQL check FAILURE (131509086490, one HIGH alert py/weak-sensitive-data-hashing at subscriber.py:889). ECHO Release Sentinel reports SUCCESS (131511834665) on the same exact head despite the failed security-analysis check. Acceptance regression required: Actions CodeQL succeeded && GHAS CodeQL failed MUST block release, regardless of Sentinel status. The flagged pre-existing helper uses HMAC-SHA256 with a >=32-byte pepper and random >=24-byte API-key secret; potential false positive requires independent security-owner adjudication, not self-dismissal. CertForge signed staging result remains NOT_READY without production E2E; zero independent approvals. No promotion or Swarm deployment.

Replace bare SHA-256 on the replayable dispatch claim token with existing server-peppered HMAC-SHA256, and assert the audit record never contains the raw token or its unkeyed digest. Addresses high CodeQL finding py/weak-sensitive-data-hashing at prior integrated head. Exact focused FORGE test: 2 passed, 30 deselected; immutable source/test blobs verified.

Copy link
Copy Markdown
Owner Author

Independent CertForge release-gate readback, 2026-11-12 22:23 UTC, exact PR #64 head e6b51c683c0027cab3ed446a26fdd614476a32c6: the current ECHO Release Sentinel check 131535880699 reports SUCCESS while on the same immutable head GitHub Advanced Security CodeQL check 131509086490 reports FAILURE / 1 high; GitHub Actions CodeQL 131508527314 and CI 131508526846 are successful; ECHO Certification Forge check 131508687933 remains NEUTRAL with signed NOT_READY and absent production E2E. The GHAS failure is py/weak-sensitive-data-hashing at subscriber.py:889, subject to independent security-owner adjudication; do not dismiss or suppress it without review.

This re-confirms fail-open cross-check behavior after GHAS reported failure. To address it without touching CertForge or Swarm production, I published a separate DRAFT, source-only ECHO Release Sentinel PR #4 at exact head 633e2fba35288d2f55158ed5dfeb82354ec0ef77 requiring distinct same-SHA Actions CI / Actions CodeQL / GHAS CodeQL / CertForge checks and a fail-closed complete inventory. Six synthetic regressions are committed; hosted run #43321054550 began in progress, not yet claimed passing at this comment.

HOLD. Sentinel patch is not reviewed, merged or deployed and cannot retroactively validate #64. CI and CodeQL workflow success are not security or production release admission. No protected merge, production service restart, DB change, signer operation or Swarm #16 deployment.

Copy link
Copy Markdown
Owner Author

Independent exact-head security triage (2026-11-12 23:29 UTC; release remains HOLD)

Candidate e6b51c6 still has GHAS CodeQL/failure (check 131509086490), while Actions CodeQL (131508527314) and CI (131508526846) pass; CertForge is neutral (131508687933), and deployed ECHO Release Sentinel incorrectly reports success (131542803913, 22:36:18 UTC). No independent PR review is submitted/approved.

I read the exact-commit source of the flagged HMAC-SHA256 helper at src/echo_certification_forge/subscriber.py:888-889. Evidence relevant to security-owner adjudication, not an automatic dismissal: the policy bounds api_key_secret_bytes to at least 24 (line 158), token issuance uses secrets.token_urlsafe(self.policy.api_key_secret_bytes) (line 1156), initialization rejects a token pepper shorter than 32 bytes (lines 267-279), and lookup uses hmac.compare_digest (lines 1248-1250, 1350-1352). The helper is a keyed digest of generated API-key material, not evidence by itself of password storage. However, source checks alone do not prove production pepper entropy, rotation/custody, or actual scanner classification; no trust or secret value was inspected.

Required independent disposition: verify the GHAS alert's exact dataflow and threat model, confirm runtime pepper provenance and credential lifecycle without exposing secret values, choose a reviewed source fix or a narrowly justified security-owner disposition, rerun GHAS at the frozen candidate SHA, and require GHAS success plus CertForge production certification before releasing. Do not suppress scan, change required-check semantics, or treat the Actions CodeQL uploader as the GHAS result. The outstanding Sentinel fix is draft echo-release-sentinel#4; its hosted tests do not change the deployed verdict. No database/service/runner/signing operations performed.

Copy link
Copy Markdown
Owner Author

Independent release-gate evidence (2026-11-13 01:22–01:27 UTC; read-only on FORGE), pinned to PR #64 head e6b51c683c0027cab3ed446a26fdd614476a32c6:

  • Replayed the exact historical-orphan WHERE predicates from this PR as a SELECT against the live CertForge database, without invoking recovery or making SQLite edits. 7/7 runs.QUEUED / dispatches.FAILED / reservations.BOUND records satisfy all null execution/lease/claim/consumption guards. They span one tenant and have 3 dispatch attempts each; all seven have exactly one positive certification-run metering event, with 0 previously recorded dispatch-crash compensation events. This is eligibility evidence, not authorization to compensate/terminalize production records.
  • 8 separate CANCELLED / PENDING / BOUND reservations are outside the predicate; must remain untouched by orphan recovery.
  • Verified /home/forge/echo-certification-forge-current resolves to the release whose on-disk .certforge-release-sha is d8ad9fd881365801354b78a758632d2a31ab6828. API and dispatcher systemd units are active and configured with that current working directory. The proposed integrated source is not deployed.
  • Exact-head CI and Actions CodeQL: SUCCESS; GitHub Advanced Security CodeQL: FAILURE (github-advanced-security); CertForge certification: NEUTRAL; deployed Release Sentinel: misleading SUCCESS. The HIGH analysis alert and Sentinel PR docs: ECHO repository health receipt #4 remain independent release blockers. No approving PR review exists.

Source-review follow-up: this PR fetches all eligible historical orphans and reconciles them in one transaction. Present live cohort is small (7), but please explicitly review the batch/lock budget or establish a bounded batch acceptance test before enabling on a larger tenant set. HOLD pending independent security and release review. No deployment, cancellations, worker simulation, or paid 40-seat execution was attempted.

Copy link
Copy Markdown
Owner Author

Independent CertForge release-gate source review — 2026-11-13 02:30 UTC; pinned exact draft head e6b51c683c0027cab3ed446a26fdd614476a32c6. HOLD; no production mutation or approval.

New scope-safety acceptance question (separate from the earlier batch-size concern): in src/echo_certification_forge/subscriber.py around lines 4209–4265, the historical-orphan SELECT matches dispatch=FAILED / reservation=BOUND / run=QUEUED with absent execution/claim/lease/consumption markers. It does not restrict d.attempts to the exhausted-attempt threshold, restrict d.last_error to the documented pre-execution failure codes, or require a documented legacy-age cutoff. Every selected row is sent to _terminalize_unstarted_run (with compensation). The recovery also runs synchronously inside the claim_dispatch immediate transaction (around lines 4322–4323), with fetchall() and no row cap.

Fresh read-only FORGE verification shows the existing cohort is narrowly consistent with expected legacy history: exactly seven QUEUED/FAILED/BOUND, all three attempts, one tenant, with zero selected ownership markers and seven positive certification-run meter events; eight separate CANCELLED/PENDING/BOUND records are excluded. No new production incident is inferred from this source risk.

Review/acceptance needed before promotion: document why every otherwise-unowned FAILED dispatch should immediately terminalize, even if it is recent, has fewer than maximum attempts, or has an unrecognized failure reason; otherwise narrow the predicate to the proven terminal pre-execution cohort. Add negative regression(s) using disposable fixtures for recent one-attempt FAILED/QUEUED/BOUND and unknown error, explicit ownership conflicts, and a stable historical exhausted cohort. Preserve exactly-once metering/audit and lock-budget guarantees; test a cohort larger than the current five-row batch fixture. No raw DB edits, arbitrary cancellations, fabricated heartbeats, or policy bypass.

CI success at this SHA cannot satisfy the failed GitHub Advanced Security CodeQL check (ID 131509086490), neutral CertForge check, or independent review. Deployed CertForge remains d8ad9fd881365801354b78a758632d2a31ab6828.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants