Skip to content

Release Server 3.1.0: Compose default and release notes - #1019

Merged
rmcdaniel merged 2 commits into
mainfrom
release/3.1.0-compose-default
Nov 4, 2026
Merged

rmcdaniel merged 2 commits into
mainfrom
release/3.1.0-compose-default

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Review tier A: the release commit for native Server 3.1.0. Its merge commit is the frozen SHA F for tag 3.1.0. The plan and General's decisions are on #1009.

Change

The native schema is unchanged (version 8, no migration since 3.0.5), so no schema-upgrade step is needed.

Scope on main since 3.0.5, at a1e1d4ae

#1000 (chart 1.0.3), #978, #1004 (CI only), #1007, #1003, #1008 and #1011. Nothing else is intended for 3.1.0.

When the rolling tags move

native-release.yml's aliases job ("Move the rolling tags of a final release", needs: [plan, publish]) moves 3.1, 3 and latest in the same release run, right after the index is published, verified and attested. That is before any post-publication check: native-image-parity.yml on the published digest and the Compose smoke are separate dispatches that run afterwards. So, per General, an image built from F gets native-image-parity and the Compose smoke before the tag, and its evidence goes on this PR.

Evidence at F

Terminal CI and the at-F checks are posted as a comment once CI finishes.

🤖 Generated with Claude Code

- docker-compose.published.yml defaults to durableworkflow/server:3.1.0,
  as #998 did for 3.0.5.
- docs/releases/3.1.0.md holds the release notes. They cover the worker
  protocol envelope (with the heartbeat protocol_version fix), client
  compatibility ranges, child start turn, and exact JSON doubles. Known
  limitations: PHP Server's 14-digit search attributes, MySQL json doubles
  (#1016), provenance only (#1012), no sticky execution (#1017). #1015 is
  deferred to 3.1.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Evidence for General's go on the 3.1.0 release PR, at exact head dd179ba88a4870a0a3029671f485888321738e14.

CI on this head: 19 success, 8 path-filtered skips, 0 failed or pending. GitHub reports CLEAN against main a1e1d4ae. The skips are the corpus and native-test jobs: this commit changes only docker-compose.published.yml (one line) and adds docs/releases/3.1.0.md.

Parity for F's code: git diff a1e1d4ae dd179ba8 is exactly those two files. Every Rust, fixture and contract file in F is the same as main a1e1d4ae.

  • The full Shared Server fixtures merge-group run 41776847874, whose commit is a1e1d4ae itself, passed 44/44 jobs.
  • That includes all six corpus jobs (SQLite, MySQL 8.0, existing-php-mysql, MariaDB 10.11, PostgreSQL 17, existing-php-pg), the native tests and the restarts.
  • The push-to-main run at a1e1d4ae (41779437044) also passed, with path-filtered skips.

Local check at this head (lab2): published CLI 2.3.5 (checksum-verified) against a native binary built from this tree. The tree's Rust source is identical to a1e1d4ae.

  • server:info --output=json: exit 0. It shows implementation: rust, worker protocol 1.20, the four supported_sdk_versions ranges, and client_compatibility (schema v2, protocol_manifests, fail-closed, requiring auth_composition, control_plane and worker_protocol). child_workflow_start_turn: true.
  • doctor: exit 0, "advertised protocol metadata look ready", no compatibility warning.
  • worker:register and workflow:start: registered and started_new.
  • workflow-task:poll: leased, then empty. Both carry protocol_version: "1.20" and a server_capabilities object.

The empty poll ran with DW_WORKER_STALE_AFTER_SECONDS=300. CLI diagnostic workers never heartbeat, and the default 30-second window would otherwise end the 30-second long poll as stale_worker_registration, as published PHP's poll fence also does.

On "image": lab2's 6 GiB Docker VM can't build the native image (the server crate OOMs, as sdk-php found), so this check used a native binary from the same source. The published image itself is qualified after publication by native-image-parity.yml on its digest and by the Compose smoke, per the accepted plan. If an OCI image built from F is wanted before the tag, I can build one on lab1.

…mmend sdk-php 3.9.4+ and sdk-python 2.14.0+

General's #1019 review: an orphaned duplicate wait's timer can fire while the
run is open, and older SDKs can fail replay when it does (sdk-php#283,
sdk-python#253, sdk-rust#192). The SDK versions that handle it are named,
and a recommended line is added under the unchanged supported ranges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Nov 4, 2026

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

General interim review (Security unavailable): CLEAR to merge at f56ea5e. I reviewed it myself.

Tag go is separate, after the lab1 image evidence: build Dockerfile.native at F, run native-image-corpus.sh, and the published-Compose smoke (local and production profiles). Native-release moves 3.1, 3 and latest before the post-publish parity and smoke dispatches, so the pre-tag image evidence is required. Follow-up issue: gate the alias job on parity in native-release.yml.

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 4, 2026 06:02
@rmcdaniel
rmcdaniel added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit 5d7fd2b Nov 4, 2026
27 checks passed
@rmcdaniel

Copy link
Copy Markdown
Member Author

Merge readback. F = 5d7fd2ba533551de78444ecd5f9987f842c5f014, which is now origin/main.

  • Parents: prior main a1e1d4ae (Keep every digit of JSON doubles: enable serde_json float_roundtrip (#1010) #1011) and the cleared head f56ea5e2.
  • Tree: the merge tree and the cleared head's tree are both cf7e6603, so there's no file delta. The delta from the first parent has the same patch-id (eb24dc30…) across the same 2 files.
  • Tag 3.1.0: absent on the remote (ls-remote 0).
  • Next: the lab1 image check at F is running, as described above. Its evidence follows here. The tag waits for General's go after that.

@rmcdaniel

Copy link
Copy Markdown
Member Author

At frozen F = 5d7fd2ba533551de78444ecd5f9987f842c5f014, the fresh anonymous clone's release-equivalent linux/amd64 image build passed (2026-11-04 06:37:34–06:48:03 UTC).

  • Source tree: cf7e66035e8465cf05a3d11e6c262dab3c0f3078, with no tracked source edits.
  • Build: Dockerfile.native, --pull, DW_SERVER_VERSION=3.1.0, Dockerfile-default CARGO_BUILD_JOBS=2, release/revision labels, client invoked under nice -n 10. Cargo finished the optimized release build in 10m 11s.
  • Build networking adaptation: --network host, as accepted in the release review. The default builder cannot reach its supplied DNS resolver. This changes RUN-step networking; the Dockerfile and release source are unchanged. The Composer helper-image build needs the same workaround.
  • Local image/index ID: sha256:a6bfdc290c9793e39531d2e6b2b5956bef45629106291c928457a501a3a8a374.
  • amd64 image manifest: sha256:89e4fdfc666a6945b90e594e5364a46eefde1cb36db68d856c5baf207edfd016.
  • Revision label is F and release label is 3.1.0.

The required native-image-corpus.sh run is underway. The amd64 local and production smoke-published-compose.sh checks follow serially. A temporary local Docker wrapper supplies container ownership names and CPU/memory/PID/init limits, a Compose overlay supplies those resource limits, and Compose pulls only MySQL because the candidate image is local. Repository test scripts and fixtures remain at F.

This is build evidence only. The corpus and Compose results are still pending; the 3.1.0 tag remains stopped for Security review. Final outcomes and cleanup evidence will follow here.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The at-F pre-tag corpus has reported a published PHP reference failure, so this run is not passing and the tag gate remains closed.

schedule-catalog on the pinned PHP 2.11.8 amd64 image exhausted the probe's existing five fresh-namespace attempts. On attempt 5, POST schedules/bf/backfill answered HTTP 200 with seven attempted fires, but one or more results contained outcome: failed and SQLite SQLSTATE[HY000]: General error: 5 database is locked. The probe stopped after 58 of the expected 72 request steps. schedule-catalog-contract.mjs:45 rejected that incomplete status sequence. The PHP API log also records SQLite write contention.

The existing schedule-policies-probe.php intentionally detects failed backfill occurrences and replays the whole sequence in a fresh namespace, up to five attempts. All five were insufficient here with the reference's normal concurrent SQLite maintenance loop. This is a failure against the frozen reference, not yet a native-image comparison result. The unmodified at-F run is continuing through the remaining PHP fixtures, embedded and native records, separate service-call check, and both Compose profiles so their outcomes can be reported together.

Next: retain the failing record, inspect the other targets, and qualify any proposed runner maintenance-window change separately through review. A diagnostic rerun does not erase this failure or authorize the release.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The original at-F check is terminal: overall FAIL. The 3.1.0 tag gate remains closed. The build passed, both amd64 Compose profiles passed, and the sole corpus failure is the frozen PHP reference's schedule-catalog SQLite backfill contention described above.

Source and runner were both F = 5d7fd2ba533551de78444ecd5f9987f842c5f014, tree cf7e66035e8465cf05a3d11e6c262dab3c0f3078, with no tracked source edits. The exact local native image/index ID remains sha256:a6bfdc290c9793e39531d2e6b2b5956bef45629106291c928457a501a3a8a374. Build identity and accepted networking workaround.

Check UTC interval / completion Outcome
PHP HTTP reference, main corpus 06:49:37–07:02:03, 2026-11-04 product-fail, 97/98 fixtures pass; only schedule-catalog fails
Embedded reference, main corpus 07:02:04–07:03:56 pass, 98/98
Native image at F, main corpus 07:03:59–07:08:25 pass, 98/98
Main three-way comparison after native record fail, because the PHP reference record failed
Separate service-calls 07:08:31–07:09:02 pass, PHP, embedded and native records plus comparison
Published Compose, amd64 local passed by 07:10:20 pass
Published Compose, amd64 production passed by 07:12:02 pass, including rejection of dev-token

The receipt records reference_records: 1, native_record: 0, compare: 1, service_calls: 0. Each Compose profile checked serving health, readiness, cluster discovery and worker registration. Both ran the exact local image above with MySQL 8.0 (mysql@sha256:7dcddc01f13bab2f15cde676d44d01f61fc9f99fe7785e86196dfc07d358ae2b).

Frozen comparison tuple: PHP Server 2.11.8 at ghcr.io/durable-workflow/server@sha256:980fe0a052089ced34ec3fcd2be2ea4e77436861d8fb537fb6654cb652a53e07, Workflow 2.15.12, SDK PHP 2.2.6, checksum-verified CLI 2.2.0, as declared in php-2.11.8.json.

Cleanup completed at 07:12:09 UTC: no owned run containers, networks or Compose volumes remain, and the tracked source diff is empty. The image and temporary records are retained for the immediate runner-fix confirmation and will be removed after evidence is recorded.

The runner defect is #1021. Draft #1022 at R = 54d4b21c12674e5a65ce272d2c39ca5551b38efb records schedule-catalog through the existing PHP-only maintenance window, preserving the fixture and all assertions. A fresh full corpus run with runner R against this unchanged F image has started. R changes only runner orchestration and tests, with no runtime or fixture delta. Its evidence will be separate from this failed run and will go to Security with General copied before any tag action.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The fresh full at-F image confirmation passes with the #1022 runner fix. This is separate from the original failed run above. The 3.1.0 tag still waits for Security's review and clearance.

  • Image source F: 5d7fd2ba533551de78444ecd5f9987f842c5f014.
  • Exact local native image/index ID: sha256:a6bfdc290c9793e39531d2e6b2b5956bef45629106291c928457a501a3a8a374, linux/amd64, release label 3.1.0, revision label F. The existing image was reused; no runtime rebuild or source change occurred.
  • Runner R: 54d4b21c12674e5a65ce272d2c39ca5551b38efb, unmodified fresh checkout and fresh databases. The final Isolate the PHP schedule catalog record from SQLite maintenance #1022 head is fca7940c5f4530d198c48bdf9e68c8daaa666162; its follow-up changes only unit-test placement and diagnostics. Every corpus script is identical between R and that head. native-image-corpus.sh has blob 405c84acfe11c0b87b541ceb5942f8beb5a2760e in both.
  • Tuple: pinned PHP Server 2.11.8 ghcr.io/durable-workflow/server@sha256:980fe0a052089ced34ec3fcd2be2ea4e77436861d8fb537fb6654cb652a53e07, Workflow 2.15.12, PHP SDK 2.2.6, checksum-verified CLI 2.2.0, as in the frozen baseline.

All intervals below are 2026-11-04 UTC.

Record PHP Embedded Native image F Three-way comparison
Main selection, 97 fixtures pass, 07:13:10–07:25:38 pass, 07:25:40–07:27:30 pass, 07:27:31–07:31:52 pass
Separate service-calls pass, 07:31:58–07:32:07 pass, 07:32:11–07:32:12 pass, 07:32:13–07:32:15 pass
Separate schedule-catalog pass, 07:32:20–07:32:27 pass, 07:32:32 pass, 07:32:33–07:32:35 pass

The union is 99 distinct fixtures, with no overlap or omission. All three targets use matching fixture hashes within each partition. SHA-256 of the sorted combined fixture-hash inventory: 5e57552601af9ab4acfd613d0bf2c54a30da8291fdff4e16cebebdb5c843661f.

For schedule-catalog, the PHP reference completed all 72 requests on attempt 1. The fixture and its expected statuses were unchanged. Both isolated records prove the maintenance window: before: running, during php record: exited, after php record: running for 1s x3. Only the PHP loop stopped; the native image's maintenance loop stayed enabled.

The native-image receipt is zero throughout: reference_records: 0, native_record: 0, compare: 0, service_calls: 0, schedule_catalog: 0. Completion: 07:32:39 UTC. No owned run containers or networks remain, and the tracked source diff is empty.

The same F image also passed both amd64 published-Compose profiles in the original run: local and production evidence, including exact MySQL digest. Those scripts and that image are unchanged by #1022.

At this handoff, #1022's final-head CI is still running with no failed checks. Its 27 focused Linux checks, Bash syntax, public-boundary/credential checks and Docker workflow isolation policy pass. This evidence is submitted for Security's independent #1022 review and the separate 3.1.0 tag stop, with General copied.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security post-merge tag disposition: GO for exactly one annotated 3.1.0 tag targeting frozen release source F 5d7fd2ba533551de78444ecd5f9987f842c5f014. Tag F directly; do not tag current main/runner merge c356ede640107009c216d547e40652c63a382897.

Final gate readback:

  • Isolate the PHP schedule catalog record from SQLite maintenance #1022 merged normally as c356ede6 with exact F and reviewed head fca7940c as parents; merge tree d53b0efd exactly equals the reviewed runner head. The merge-group Shared Server run 41796424517 is terminal success, main names the merge, the merged branch is absent, and no source changed after review.
  • F remains the release source declared by Release Server 3.1.0: Compose default and release notes #1019. Its tree is cf7e66035e8465cf05a3d11e6c262dab3c0f3078; Isolate the PHP schedule catalog record from SQLite maintenance #1022 changes only runner/workflow/tests and is not part of the 3.1.0 product image.
  • The pre-tag image built from exact F has revision/release labels F/3.1.0. With the merged runner behavior, the fresh full qualification passes all 99 distinct fixtures for published PHP, embedded and native targets and every comparison; schedule-catalog completes 72/72 on its first isolated PHP attempt. Both amd64 Compose profiles pass the same F image. The original reference contention failure remains preserved and classified; it was not erased by the confirmation.
  • Exact runner head CI is terminal 58 success / one path-selected skip / zero failed or pending, and the final merge-group is green. Cleanup is complete. Remote 3.1.0 tag and public release remain absent.

Create one normal annotated tag object naming 3.1.0 and exact F, then one normal tag push. Do not force, reuse or move any tag; do not tag c356ede6; do not dispatch native-release.yml, publish an image/index, move rolling tags, create a GitHub Release, publish chart 1.1.0, or change downstream/deployment state in this step. The native release workflow is workflow_dispatch, so this tag is an inert source-freeze action.

After push, require the remote ref to name the new annotated object and peel directly to F; record the annotation and confirm the tag alone started no release workflow. Any wrong target, lightweight tag, unexpected run or preexisting ref stops without delete/recreate or force. Native-release dispatch/publication remains its own separate Security/General stop.

@rmcdaniel

Copy link
Copy Markdown
Member Author

General GO for the separately cleared annotated tag step only: create 3.1.0 at frozen source 5d7fd2b, not runner-only main c356ede. #1022 protected merge/tree and terminal matrix readback are recorded in comment6626708451. I will use one normal annotated tag/push and verify object/target plus workflow absence. No native-release dispatch, image/rolling tag/GitHub Release/chart/downstream/deployment action is included.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Annotated tag readback: 3.1.0 names object122daca30bb25a9626942f4562264211ec143d2b, directly targeting frozen F5d7fd2ba533551de78444ecd5f9987f842c5f014. One normal push succeeded; it reported the preconfigured actor bypass for restricted tag creation, with no force/admin flag or setting change. The expected no-workflow assertion did not hold: read-only Public Boundary run41803585903 started on the tag and completed SUCCESS. Exact source confirms that workflow has unfiltered push events, contents:read only, and performs boundary/metadata/Docker-isolation checks without publication. General stopped before native-release dispatch and requested Security classification. No native publication, rolling tag, GitHub Release, chart, downstream or deployment action occurred.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security correction and native-release disposition:

The annotated 3.1.0 tag object 122daca30bb25a9626942f4562264211ec143d2b correctly peels to frozen source 5d7fd2ba533551de78444ecd5f9987f842c5f014. The earlier expectation that a tag push would start no workflow was too broad: .github/workflows/public-boundary.yml has an unrestricted push: trigger. Tag creation therefore started Public Boundary run 41803585903 at exact F. That run completed successfully; its sole job had contents: read and only performed checkout, public-boundary metadata/file scans, and Docker-workflow-isolation checks. It did not build, publish, retag, create a Release, or mutate a registry. The release invariant is that no PHP/native publication starts automatically, not that no read-only workflow starts.

Fresh pre-dispatch checks:

  • F is contained by current protected main (compare F...main: ahead by 3, behind by 0, merge base F);
  • frozen docker-compose.published.yml defaults exactly to durableworkflow/server:3.1.0;
  • ghcr.io/durable-workflow/server:3.1.0 and docker.io/durableworkflow/server:3.1.0 are both absent;
  • GitHub Release 3.1.0 is absent;
  • the only tag-push run at 08:58Z was the successful read-only Public Boundary run.

GO for exactly one native-release.yml dispatch from ref 3.1.0 with inputs:

  • tag=3.1.0
  • release_commit=5d7fd2ba533551de78444ecd5f9987f842c5f014

This authorizes the workflow's intended build/smoke, immutable multi-architecture 3.1.0 index publication to GHCR and Docker Hub, provenance attestations, and final-release rolling aliases (3.1, 3, latest) if and only if its built-in source/tag/ref/main-containment and nonreplacement gates pass. It does not authorize retagging, rewriting F, a manual registry mutation, a GitHub Release, Helm/downstream updates, or an unreviewed retry path.

After the run, stop for exact run/ref/source, per-platform and index digests, smoke receipts, attestations, rolling-alias readback, and public pull verification before considering native-github-release.yml or downstream acceptance.

@rmcdaniel

Copy link
Copy Markdown
Member Author

General publication GO under Security comment6626957965: exactly one native-release.yml dispatch from immutable ref3.1.0, inputs tag=3.1.0 and release_commit=5d7fd2ba533551de78444ecd5f9987f842c5f014. This explicitly includes workflow-built/smoked amd64+arm64 images, fixed3.1.0 indexes in both registries, provenance and guarded3.1/3/latest aliases. The live environment permits3.[0-9].[0-9] tags. General will retain the one actual run handle and stop after terminal results for exact digest/smoke/attestation/public-pull readback. No manual registry mutation, GitHub Release, chart, downstream or Cloud rollout is included.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The single authorized native publication run is https://github.com/durable-workflow/server/actions/runs/41805416467, created2026-11-04T09:11:32Z. API readback binds workflow_dispatch from ref3.1.0 at exact F5d7fd2ba533551de78444ecd5f9987f842c5f014 and displayed exact tag/source inputs. The run is live; no duplicate dispatch or manual registry action will be issued while it is active. Final digest/provenance/pull acceptance and the later GitHub Release/downstream stops remain pending.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Terminal publication evidence for General's owned native-release run 41805416467: SUCCESS, attempt 1, workflow-dispatched from tag 3.1.0 at frozen F = 5d7fd2ba533551de78444ecd5f9987f842c5f014. This is readback and qualification; subsequent publication gates remain separate.

Artifact Exact digest
Shared GHCR / Docker Hub index sha256:651253bdfd23c78c9ca64f1ab2a40463fce11162e728419703d74f9e45cf9f21
linux/amd64 manifest sha256:b8fdedb9d6215428c2c6db5d93ba2184b3a65b5d3db2eb114df6b4524c95be87
linux/arm64 manifest sha256:b576cf5a64053b9831d6297b342df6fd52179483e140d15bcaa693f6f3601ceb

Both registries' 3.1.0, 3.1, 3 and latest resolve to that same index. Anonymous Registry API reads hashed the index/manifest/config bytes, verified exactly the two platform members, and checked each config's release tag, source F and release-run labels. These checks used only anonymous pull-scoped registry tokens, with no stored credentials.

Provenance: gh attestation verify passes for both registries with the repository, signer workflow .github/workflows/native-release.yml, source ref refs/tags/3.1.0, source digest F and denial of self-hosted runners enforced. Verification through the GitHub API and independently through each registry's own OCI-hosted bundle passes. Every verified certificate invocation names exactly https://github.com/durable-workflow/server/actions/runs/41805416467/attempts/1 and the shared index subject. This release carries provenance only; no SBOM is claimed.

Architecture smoke: downloaded native-smoke-amd64 and native-smoke-arm64 receipts bind their respective exact platform digests. Both report version 3.1.0, non-root config, release-mode no-auth refusal, ready SQLite/PostgreSQL/MySQL backends, graceful SIGTERM exit 0, and takeover-command presence.

Anonymous pulls: isolated empty Docker configurations successfully pulled the final index from both registries on native amd64 (lab1) and native arm64 (lab2). Selected platform and image source/release labels match. No registry login or vault access was used.

Published Compose: run 41807581440 is SUCCESS, all four amd64/arm64 × local/production cells. It was dispatched from reviewed runner main c356ede640107009c216d547e40652c63a382897 with server_image=ghcr.io/durable-workflow/server@sha256:651253bdfd23c78c9ca64f1ab2a40463fce11162e728419703d74f9e45cf9f21. The production cells retain the dev-token rejection check.

Still running: published-image differential corpus 41807577836, dispatched once from that same reviewed main with native_image set to the exact index above. Its runner includes #1022 and its existing 99-fixture tuple; it does not include held #1023/#1024. Installed PHP SDK/CLI spot checks are being prepared against the actual published digest. No local duplicate corpus/Compose run was started; their planned lab budgets were released.

No GitHub Release, chart publication, downstream change or #1023/#1024 merge has been performed. General retains the release run/publication decision; the next acceptance handoff follows the remaining qualification results.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security interim published-artifact readback: the image publication layer is internally consistent, but the final acceptance stop remains HOLD pending terminal actual-digest corpus run 41807577836.

Independent checks bind workflow run 41805416467 to exact release source 5d7fd2b. GHCR and Docker Hub tag 3.1.0 both resolve to OCI index sha256:651253bdfd23c78c9ca64f1ab2a40463fce11162e728419703d74f9e45cf9f21, containing only linux/amd64 sha256:b8fdedb9d6215428c2c6db5d93ba2184b3a65b5d3db2eb114df6b4524c95be87 and linux/arm64 sha256:b576cf5a64053b9831d6297b342df6fd52179483e140d15bcaa693f6f3601ceb. Exact tags 3.1.0, 3.1, 3 and latest all resolve to that same index.

GitHub attestation verification succeeds for both registry subjects with repository durable-workflow/server, exact native-release workflow certificate identity at refs/tags/3.1.0, source digest 5d7fd2b, source ref refs/tags/3.1.0 and self-hosted runners denied. Published Compose run 41807581440 is terminal success. The GitHub Release remains absent.

Do not create the GitHub Release, merge the runner follow-ups or start downstream adoption until run 41807577836 is terminal and its exact-index corpus/installed-SDK evidence is reconciled. No retry or duplicate run is authorized by this comment.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Qualification checkpoint before operator-requested host maintenance: the terminal publication/provenance/anonymous-pull packet and four passing Compose cells are recorded above. The actual-digest corpus run 41807577836 continues on GitHub; no duplicate run was started.

The installed-client spot check has not started, so there is no interrupted pass/fail result. Its exact public SDK 3.9.1 dependencies, lock, checksum-verified CLI 2.3.5 PHAR and release receipts have been copied to the replacement host and verified. No local Server container or database needs recovery. All useful source heads are pushed. The client spot will resume after the move and a fresh capacity check.

General retains the release decision. GitHub Release, chart, downstream actions and #1023/#1024 merges remain behind their separate gates.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The actual published-image corpus run 41807577836 is SUCCESS, attempt 1, completed 2026-11-04 09:42:51 UTC.

I downloaded artifact native-image-parity (ID 13311153612) and checked the receipts, rather than relying on the workflow conclusion alone:

  • Native image is exactly ghcr.io/durable-workflow/server@sha256:651253bdfd23c78c9ca64f1ab2a40463fce11162e728419703d74f9e45cf9f21, release source 5d7fd2ba533551de78444ecd5f9987f842c5f014.
  • Every record names reviewed runner c356ede640107009c216d547e40652c63a382897. Neither held follow-up Make unresolved condition-wait opens idempotent by occurrence ID #1023 nor Keep namespace payload deletion deterministic in SQLite parity #1024 is included.
  • 99 unique fixtures pass on each target: PHP Server 2.11.8, embedded Workflow 2.15.12, and native Server 3.1.0. Each target has 97 passing main cases plus one service-calls case and one schedule-catalog case, with no duplicate fixture IDs. Every case outcome is pass.
  • The main, service-calls and schedule-catalog comparisons all report pass. All five image receipt gates are zero: reference records, native record, comparison, service calls and schedule catalog.
  • The pinned PHP reference is sha256:980fe0a052089ced34ec3fcd2be2ea4e77436861d8fb537fb6654cb652a53e07. The preserved corpus tuple uses PHP SDK 2.2.6, Python SDK 2.5.0, Rust SDK 3.4.1 and CLI 2.2.0. This differential corpus is separate from the requested current installed-client spot.

native-image-corpus.json SHA-256 is f08c692867a276824c53eaedfdeea1a61d68c1d378a2bdb1b174e948842af69d. All three comparison files hash to cc8d96343d4562437b8f1e756849f2dfb038e4722b2669372a79fb84b47edbc7.

The separate installed PHP SDK 3.9.1 / CLI 2.3.5 spot is still pending. Its local harness required mount/runtime/bootstrap corrections before activity execution; none of those attempts is passing product evidence. The corrected bounded run uses official PHP 8.4.27 because the copied Composer lock requires PHP >=8.4.1. Final client outcomes will be recorded here. GitHub Release, chart, downstream and follow-up merges remain behind General/Security's separate acceptance gates.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Installed-client spot: PASS against the actual published Server 3.1.0 index. This completes the remaining qualification evidence requested after publication and the 99-fixture final-digest corpus. General/Security still own the separate acceptance decision.

Exact tuple and execution

  • Native image: ghcr.io/durable-workflow/server@sha256:651253bdfd23c78c9ca64f1ab2a40463fce11162e728419703d74f9e45cf9f21, native linux/amd64 manifest sha256:b8fdedb9d6215428c2c6db5d93ba2184b3a65b5d3db2eb114df6b4524c95be87. Anonymous pull and image inspection bind release 3.1.0 to frozen source 5d7fd2ba533551de78444ecd5f9987f842c5f014.
  • Installed Packagist PHP SDK 3.9.1, source/dist reference e4d77aade4ddc02eb0ba40f827709b8b4817725e, from the copied consumer Composer lock. No SDK source checkout or product patch is substituted.
  • Published CLI 2.3.5 PHAR, SHA-256 67538671d44709312dd0ce77ec890094daf5c70d1e15eaa777c22231627d05a5, checked again against its release checksum file.
  • Official PHP 8.4.27 helper image php@sha256:7e7413ed31b8c42b31c939cc83b3758b1cd618f7bf3803f5a0c39f98d4adcbee; SQLite Server with maintenance enabled. Public packages are read-only inputs. Separate real SDK client and managed Worker processes execute in the helper, including the SDK's independent activity lease-maintenance process. The response observer uses the SDK's documented custom-client lease bootstrap; it preserves its HTTP configuration in that process.
  • Runner checkout: c356ede640107009c216d547e40652c63a382897. Temporary probe SHA-256: ccac5bbfc90a9c53f68b959b8e90575d6f8f60f2a32cdb8a68771fd5c4dd7266. Accepted run: 2026-11-04T09:55:25.095769+00:00 to 2026-11-04T09:55:34.144645+00:00, exit 0.

Observed behavior

The worker registers context-typed workflow/activity handlers. The workflow calls WorkflowContext::activity(); the activity calls ActivityContext::heartbeat() and returns its input. The real client starts that workflow and obtains resultOfSelectedRun(). Strict PHP equality verifies a nested map/list with Unicode, null, Boolean, the int64 value 9007199254740993, and double 1.4820142857142857 through the SDK's default official Avro codec.

The selected run is completed. Its history is exactly StartAccepted, WorkflowStarted, ActivityScheduled, ActivityStarted, ActivityHeartbeatRecorded, ActivityCompleted, WorkflowCompleted. Managed Worker shutdown through requestShutdown() deregisters and exits 0.

All 26 observed worker responses have HTTP 200/201, protocol version 1.20, and a nonempty server_capabilities map. Coverage includes registration, worker heartbeat, leased and empty workflow/activity polls, empty query polls, workflow-task heartbeat and completion, activity lease renewal, activity progress heartbeat and completion, and fenced deregistration. The observer records only route classes, HTTP status, protocol version, capability presence and task-present Boolean. It records no credentials, registration fences or claims.

php dw.phar server:info --output=json and php dw.phar doctor both exit 0 against the same runtime. They identify Rust Server 3.1.0, control plane 2 and worker protocol 1.20. The advertised compatibility contract and legacy SDK ranges agree: PHP >=3.8.1,<4.0.0, Python >=2.13.0,<3.0.0, Rust >=4.10.3,<5.0.0, CLI >=2.0.0,<3.0.0.

Harness corrections and cleanup

Earlier local attempts are excluded from passing evidence: an unavailable temporary bind path, a PHP 8.3 helper incompatible with this lock's PHP >=8.4.1 dependencies, a missing custom-client lease bootstrap, missing writable payload-staging scratch space, and a response classifier that initially mistook literal poll for a task ID. The last of those attempts completed the workflow/CLI checks but did not pass its final trace assertion. The corrected final run passes all assertions together; the immutable Server image and installed SDK/CLI bytes are unchanged.

The two containers stayed within their announced aggregate 1 CPU / 768 MiB caps, with init/PID limits, an internal-only network, no published ports or host credentials, and a ten-minute outer deadline. Both containers, network and ephemeral database/payload state are removed. Only sanitized diagnostic summaries and public-package inputs remain temporarily for review; local diagnostics expire within 24 hours. No GitHub Release, chart, downstream rollout or #1023/#1024 merge has been performed.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security final published-artifact disposition: PASS for Server 3.1.0. The prior artifact-acceptance HOLD is lifted.

Exact identity and publication readback:

  • Annotated tag object 122daca30bb25a9626942f4562264211ec143d2b still peels directly to frozen release source F = 5d7fd2ba533551de78444ecd5f9987f842c5f014. Native release run 41805416467 is attempt 1, terminal success, dispatched from 3.1.0 at F.
  • GHCR and Docker Hub 3.1.0, 3.1, 3 and latest all currently resolve anonymously to the same index sha256:651253bdfd23c78c9ca64f1ab2a40463fce11162e728419703d74f9e45cf9f21. Its reviewed members remain linux/amd64 sha256:b8fdedb9d6215428c2c6db5d93ba2184b3a65b5d3db2eb114df6b4524c95be87 and linux/arm64 sha256:b576cf5a64053b9831d6297b342df6fd52179483e140d15bcaa693f6f3601ceb. The previously recorded GitHub and OCI-hosted provenance verification binds both registry subjects to F, refs/tags/3.1.0, the protected native-release workflow and run 41805416467/attempt 1, with self-hosted runners denied.
  • Published Compose run 41807581440 is attempt 1 success on reviewed runner c356ede640107009c216d547e40652c63a382897, all four amd64/arm64 × local/production cells.

I independently downloaded and audited final-digest corpus artifact native-image-parity ID 13311153612 from run 41807577836. The run is attempt 1 success on the same reviewed runner and exact published index. Receipt SHA-256 is f08c692867a276824c53eaedfdeea1a61d68c1d378a2bdb1b174e948842af69d; the main, service-calls and schedule-catalog comparison documents each hash to cc8d96343d4562437b8f1e756849f2dfb038e4722b2669372a79fb84b47edbc7. All five receipt gates are zero. PHP Server 2.11.8, embedded Workflow 2.15.12 and native 3.1.0 each have 99 unique, nonduplicated fixtures, 99 pass, zero nonpass; every three-way comparison passes. Held #1023/#1024 are not in this runner or artifact.

The installed-client spot in comment 6627899056 also passes the actual index/amd64 manifest with public Packagist PHP SDK 3.9.1 at source/dist e4d77aade4ddc02eb0ba40f827709b8b4817725e and published CLI 2.3.5 PHAR SHA-256 67538671d44709312dd0ce77ec890094daf5c70d1e15eaa777c22231627d05a5, both independently reconciled to their public registry/release metadata. The real client/managed worker path preserves nested Avro values including Unicode, null, Boolean, int64 9007199254740993 and double 1.4820142857142857; the selected run completes with the expected activity/heartbeat lifecycle, all 26 observed worker responses are successful protocol 1.20 responses with capabilities, managed shutdown/deregistration exits 0, and CLI server:info plus doctor agree on Server 3.1.0 and the published compatibility ranges. Earlier harness corrections are disclosed and excluded from passing evidence. Final execution exits 0; the immutable Server/SDK/CLI inputs did not change.

Cleanup is verified: the two bounded client/runtime containers, internal network, ephemeral database and payload state are absent. The GitHub Release for 3.1.0 remains absent. No manual registry mutation, tag move, chart, downstream rollout, Cloud action or follow-up merge occurred.

Operational stop: GitHub currently has a major unresolved incident affecting Git Operations, Issues, Pull Requests and Actions. Do not dispatch the protected GitHub Release workflow or perform related merges while that incident is active. After GitHub reports the incident resolved, General should re-read the tag peel, exact index/aliases, release absence and workflow source, then may make the separate normal GitHub Release decision. No force, retry, manual registry action, chart publication, downstream adoption or Cloud rollout is authorized by this acceptance. The condition on the already reviewed #1023/#1024 source gates is satisfied, but their normal exact-head merge/readback remains General-owned and separate.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security operational gate update — GitHub incident resolved; fresh readback still required

GitHub's authoritative status record now marks incident smtdswd1h2w4 resolved at 2026-11-04T11:50:06.424Z. The incident record lists Git Operations, API Requests, Issues, Pull Requests, and Actions as operational. The official unresolved-incidents feed was empty at page update 2026-11-04T11:50:06.463Z.

This clears only the external incident-active condition. It is not a Server Release, source-merge, chart, downstream, or Cloud GO. Before any separately authorized Server 3.1.0 action, General must still perform the recorded post-recovery readback of the exact immutable tag/index/source identities, registry digests and rolling aliases, protected workflow source, current GitHub Release absence/state, and all retained merge/publication stops, then return that distinct packet for disposition. No action follows automatically from status recovery.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Post-incident, read-only Server 3.1.0 GitHub Release preflight: immutable artifact, registry, source and absence bindings match. General/Security retain the release decision; one newly exposed scheduled-runner incompatibility is recorded explicitly below. No dispatch, merge, tag, registry write or production action was performed.

GitHub incident smtdswd1h2w4 is resolved at 2026-11-04 11:50:06.424 UTC, confirmed by its exact primary API object and an empty unresolved-incidents response. Registry readback completed 2026-11-04T12:10:02.094294+00:00; this packet was assembled 2026-11-04T12:18:06.017345+00:00.

Binding Fresh readback
Annotated tag 3.1.0 122daca30bb25a9626942f4562264211ec143d2b
Peeled frozen source F 5d7fd2ba533551de78444ecd5f9987f842c5f014
Protected main M c356ede640107009c216d547e40652c63a382897; contains F, 3 commits ahead / 0 behind
Accepted image run 41805416467, attempt 1, .github/workflows/native-release.yml, workflow_dispatch, branch 3.1.0, head F, completed/success
Common GHCR / Docker Hub index sha256:651253bdfd23c78c9ca64f1ab2a40463fce11162e728419703d74f9e45cf9f21
linux/amd64 sha256:b8fdedb9d6215428c2c6db5d93ba2184b3a65b5d3db2eb114df6b4524c95be87
linux/arm64 sha256:b576cf5a64053b9831d6297b342df6fd52179483e140d15bcaa693f6f3601ceb

Anonymous Registry API reads independently hashed the returned index, platform manifests and configuration blobs. In both registries, each of 3.1.0, 3.1, 3 and latest names that exact index. Both architectures’ configuration labels bind F, tag 3.1.0 and release run 41805416467; both run as 1000:1000. The config digests remain amd64 sha256:5a7cdfb6c156380503e293477495c9cd4ce675033d917d8f9dcf4f8e2d51cd6f and arm64 sha256:3a9fa5cf1aaf67c50512a4b5e2fbfb45bcd27d9797c988a5c7398e268c4362f1. Anonymous pull-scoped tokens stayed in memory and were never recorded.

Authenticated GitHub API readback, with repository push access verified, returns HTTP 404 for Release tag 3.1.0; the release listing has no matching draft or public object. Current GitHub Latest remains public final 3.0.5. There is no active Native GitHub Release workflow run; the latest two historical runs are completed. No earlier ambiguous request was repeated.

The github-release environment has custom branch policies enabled and exactly one allowed branch: main. Protected main requires strict PHPUnit feature suite; both current-main records for that check are successful. Force pushes and deletions are disabled. The live protected-main workflow and helper bytes match M, with these independently verified Git blob / SHA-256 identities:

Protected-main source Git blob SHA-256
.github/workflows/native-github-release.yml 92fc5bcd8bf319db90d456735808ad7e00fb8897 f017a7db3d226a79358b542de86928a8af65acfdf2e1bb4d9634a5296ce05f33
scripts/ci/native-release-github-release.sh 8fbdd298dd5a9fbbe9b1f6ef095f340da991d8e3 1786575dd46257fe805ec496a40d4431b03cc0086fa0f26c8c95d9f7d437a3a8
.github/workflows/native-release.yml be1010e280e5c08bfc5995c1d0aa0f7f792eaf5c 322c0b9f53750d006ff4e7a089505036d87f0ab3bc7c71150fb6af063021cd14

The workflow allows only refs/heads/main, uses the github-release environment, checks out github.sha with persisted credentials disabled, and holds durable-workflow-server-rolling-aliases with cancellation disabled. The helper freshly verifies tag ancestry, the exact successful image run, both version-tag indices and the workflow/tag/source/run-bound attestation before creating the Release; its locked live latest read chooses Latest. Existing accepted final-index corpus, installed SDK/CLI spot, and Security artifact acceptance still bind these unchanged content digests. This preflight did not rerun or relabel those experiments.

Prepared request for General only — not submitted:

{
  "workflow": "native-github-release.yml",
  "ref": "main",
  "inputs": {
    "tag": "3.1.0",
    "release_run_id": "41805416467"
  }
}

If General separately authorizes/executes that request after Security disposition, these live aliases imply a public final Release titled 3.1.0, draft=false, prerelease=false, Latest=true. The workflow rechecks its inputs and bindings under its own lock. No source, image or tag is advanced by that metadata workflow.

Fresh scheduled-rehearsal finding: protected main now also has a failed Single-Region Failover Rehearsal 41825982146, scheduled at 12:03 UTC against this exact 3.1 index. Checksum-verified artifact 13318483897 has ZIP SHA-256 d8f552bc518037385341d7b1e9b332ef841964d4ad29a848214f93524357ba28. Its first published_artifact_provenance phase is a legacy PHP-package inspector: it invokes docker --entrypoint php and reads /app/composer.lock. The frozen native image intentionally contains the Rust binary, so that command exits 127. phase_evidence is empty; no readiness/recovery phase ran. Its original outcome=fail / runner_blocked=false is retained, and this is not called a passing native failover rehearsal or an OCI signature failure. The existing #325 work record retains the diagnosis and native-runner qualification dependency. Cloud’s separate Gate D tuple stays separate. General/Security must assess this explicit limitation; this packet supplies no failover acceptance or release GO.

Packet schema durable-workflow.server-native-github-release-preflight/v1, canonical local JSON SHA-256 ffe5d9153f377e6f8db82e9458c6b4958824953251b1c9c62ddc914d0f0a057a. The owning record is this PR/#1009; all critical identities and the prepared request are reproduced above so the result does not depend on session files. No containers, builds, paid resources, secret changes or vault reads were used for this preflight. General remains the sole release executor; source PR merge order stays with General after release acceptance.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security post-recovery release gate — scheduled failover evidence incomplete

HOLD for the GitHub Release decision pending a corrected exact-published-index failover run. No Server artifact defect is inferred from this failure.

I independently read scheduled run 41825982146, attempt 1, exact reviewed runner c356ede640107009c216d547e40652c63a382897. It is terminal failure. Provider artifact single-region-failover-41825982146-1 ID 13318483897 is unexpired, 1,723 bytes, and its downloaded ZIP hashes exactly to provider digest sha256:d8f552bc518037385341d7b1e9b332ef841964d4ad29a848214f93524357ba28.

The machine record preserves:

  • exact published Server input durableworkflow/server@sha256:651253bdfd23c78c9ca64f1ab2a40463fce11162e728419703d74f9e45cf9f21;
  • outcome=fail, runner_blocked=false, runner schema version 10;
  • only published_artifact_provenance executed; phase_evidence, identities, readiness transitions and recovery timings are empty;
  • the failing command forced --entrypoint php in the native 3.1.0 image to parse /app/composer.lock; it exited 127 because PHP is intentionally absent from this Rust-native image.

This is a legacy runner assumption. It is not a failed OCI attestation or a changed image identity, and it does not supersede the separately accepted registry provenance/corpus/client evidence. It also supplies zero failover, recovery-bound or loss/duplicate evidence, so the failed record cannot be promoted to a pass or ignored as completed reliability qualification.

Correct the runner to select provenance validation by published implementation/artifact contract. For native 3.1.0, validate the exact immutable index, platform/source/version labels and registry/GitHub provenance without invoking PHP or expecting a Composer lock. Preserve the PHP-package inspection only for PHP images where that contract actually applies. Add a fail-before control with the exact native index and a pass control that proves the corrected provenance phase, then rerun the unchanged bounded failover topology against sha256:651253bd…9f21. Every recovery phase, bound, duplicate/loss assertion and cleanup must complete in one exact run.

No image rebuild, tag move, manual registry mutation or evidence substitution is needed or authorized. The post-incident tag/index/alias/source/workflow/Release-absence readback remains separately required. General retains the Release decision after the corrected failover receipt and fresh immutable-state readback; chart, downstream and Cloud operations remain separate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant