Repository navigation
Replace PHP Server with a compatible Rust runtime and safe in-place upgrade #325
Description
Objective
Replace the PHP/Apache Durable Workflow Server with a faster Rust successor in
this repository. Existing published SDKs, CLI, Waterline and sample applications
must work unchanged. Operators experience a major version upgrade of the same
product, with the same API, behavior, data and operational surfaces.
The PHP Server remains working as a reference throughout development. The
embedded Laravel Workflow package remains a first-class product. This is accepted
implementation work assigned to a dedicated maintainer, not a new architecture
comparison or a claim that Rust is already faster.
Constraints
Repository and delivery
- No new repositories. Rust lives in
durable-workflow/server. - Land small, coherent PRs on
mainfrequently. Do not accumulate the rewrite
on a long-lived branch or break the PHP Server while landing foundations. - Keep a running, contributor-readable log in this repository with the current
plan, significant design decisions, verified outcomes, filed issues, the
fixture cases they block and the next concrete action. - Choose the architecture and established Rust libraries with engineering
judgment. Explain significant choices briefly and link the proving fixtures. - Follow ordinary review, CI, release and branch-cleanup rules.
Contract, parity and embedded mode
- Published specifications and shared fixtures define observable behavior.
Neither implementation wins by default. When they disagree, settle the
contract first, then correct whichever implementation is wrong. - Establish a shared fixture corpus, recorder and differential runner that can
target a server URL. Reuse existing protocol specifications, conformance
scenarios and benchmark tooling rather than creating a second software factory. - Run the shared fixtures against embedded Laravel, PHP Server and Rust Server.
Workflow package changes also remain tested against a real HTTP Server. - Default to parity for existing and new observable capabilities. Define the
contract and fixtures first, implement a feature in both servers (including
necessary Workflow package changes), and pass the same fixtures on both.
A simple, slower PHP implementation is acceptable. - Internal scheduling, sharding and pooling mechanics without an observable
contract need no PHP counterpart. - If an observable capability cannot reasonably exist in PHP, advertise an
explicit capability flag, mark its fixtures Rust-only and record why.
Do not disguise incomplete parity as an internal optimization. - Preserve the existing Avro payload protocol, errors, authentication and
authorization, namespaces, durable identities and history, worker leases and
fencing, retries, cancellation, timers, schedules, visibility and recovery.
Execute the required PHP/Python/Rust worker directions, rather than assuming
an HTTP schema comparison proves them.
Database isolation and one-way upgrade
- Support SQLite, MariaDB/MySQL and PostgreSQL using the current database tables
and stored data. Rust owns its versioned forward migrations. - Ordinary development and differential runs use separate isolated databases
for PHP, Rust and embedded targets. Never run different implementations
concurrently against one database. - The only cross-implementation shared-database exception is the dedicated
sequential upgrade test: PHP writes and stops, then Rust takes over that
same database. Multi-node replicas are tested within their own isolated
deployment, not against another implementation's state. - Upgrade existing deployments in place. HTTP export/import remains a way to
move data between deployments, not the upgrade route. - Require a backup before mutation and stop the PHP fleet before takeover.
Make interruption resume safely or fail without corrupting state. - Record a schema marker. All affected PHP startup/worker/bootstrap paths must
refuse an upgraded database before migration or durable writes can occur. - There is no PHP rollback after upgrade and no mixed PHP/Rust fleet.
Rust migrations need not preserve post-upgrade PHP schema compatibility.
The PHP reference continues to work against its own non-upgraded database. - Inventory stored PHP-serialized values, class names and framework conventions.
Handle each with a read-compatibility path or documented one-time conversion.
Do not silently discard data that Rust cannot interpret. - Preserve acknowledged work, durable history, pending workflows, timers,
leases, credentials and external payload references/bytes across takeover.
Operations and performance
- Preserve existing image names, supported image architectures,
DW_*
configuration, health/readiness, bootstrap, Compose and Helm layouts, metrics,
diagnostics, graceful shutdown and recovery. - Document a migration path for anything that cannot carry over directly,
such as a PHP authentication-provider class. - Record a reproducible PHP baseline before judging Rust. Use exact published
artifacts, workloads, hardware, resource limits, commands and raw results. - Rust must meet or beat PHP throughput, memory and idle long-poll cost.
Verify timers and lease renewals remain healthy during CPU-heavy load.
Do not accept apparent gains from lost work, backlog, swapping or an
unmeasured drain. Repeat promising measurements and recheck baseline drift. - Do not change customer pricing, Cloud production or managed rollout as part
of the Server rewrite. Exact Rust artifacts need their own Cloud qualification.
Work order
- Establish the safety net: current contract inventory, shared fixtures,
recorder, URL-targeted runner, meaningful embedded/HTTP coverage and
reproducible PHP baseline. - Land the Rust foundations and vertical slices in small reviewed merges,
keeping PHP operational. - Run the differential suite after every change. Record intentional contract
decisions and convert every defect into a focused shared regression. - Complete all capabilities and actual SDK/worker directions, database and
multi-node qualification, performance proof and operational compatibility. - Finish migration tooling, backup-first upgrade tests, operator documentation
and major-version cutover preparation. Release through the repository's
existing decision and publication process, then verify published artifacts.
Acceptance
- Shared fixtures, recorder and URL-targeted differential runner are usable
by a contributor from the public repositories. - PHP Server stays operational and embedded Laravel remains first class.
Shared contracts pass on embedded, PHP Server and Rust Server. - Every required fixture passes with actual published PHP/Python/Rust SDKs
and every meaningful supported workflow/activity/child direction executed.
Existing CLI, Waterline and sample apps work unchanged. - Multi-node kill/restart tests prove no lost durable work, no duplicate
committed outcomes and no acceptance of stale worker attempts. External
effects retain their explicit application-idempotency contract. - Conformance and multi-node suites pass on SQLite, MariaDB/MySQL and
PostgreSQL. Database topology and limits are explicit. - Rust meets or beats the recorded PHP baseline for throughput, memory and
idle long polls. Timers and lease renewals remain healthy under CPU-heavy load. - Existing images/configuration/bootstrap/Compose/Helm/health/readiness/
shutdown/diagnostics remain usable, with documented migrations for exceptions. - A backup-first in-place upgrade starts from data written by the last PHP
release, with pending workflows, timers and leases, and continues under Rust
on each supported database. - Interrupted upgrade resumes or fails safely. PHP reliably refuses the
upgraded schema. No concurrent mixed fleet or post-upgrade PHP rollback. - Stored PHP-specific values and external payloads have tested compatible
reads or explicit one-time conversions. - Operator upgrade/cutover documentation, major-release readiness,
exact publication identities and downstream verification are complete. - The running repository log lets a maintainer see decisions, blockers,
blocked fixture cases and the next action without private workspace context.
Ownership
A dedicated Rust Server maintainer owns this issue and its scoped implementation,
tests and cutover preparation. Coordinate necessary cross-repository contract
and Workflow changes through linked owning records. Other maintainers should
avoid competing implementation of this rewrite while that assignment is active.
Activity
rmcdaniel commentedon Oct 9, 2026
The first safety-net slice is in #326, with the public running log, capability inventory and frozen published tuple. The default PHP artifact remains unchanged; no Rust runtime or performance improvement is claimed.
At source
8182ad8b5df8a8266f52e0a368fb3abea53a19e9, three reviewed cases run through actual PHP SDK workers and real embedded Laravel database queue jobs on independent SQLite files: typed echo, an exact int64 beyond JSON number precision, and one external activity. The recorder checks completed work, the complete expected event inventory, history pagination, deadline budgets, registered types, decoded values/types and start/run/activity/attempt relationships. The differential comparison anchors to current fixture hashes and rechecks observations; matching partial recordings and saved pass labels cannot qualify a run. There are 25 focused comparator checks.Hosted fixture qualification passed and retains both records and comparison for 90 days. Repository contracts and boundary checks passed; the full PHP feature suite is still running. This is bounded correctness evidence, not a full three-runtime/database or performance qualification.
Next action after the first merge: establish a current reproducible PHP baseline with the existing capacity workload and commands. The current standard launcher freezes older packages and a Linux 6.8 profile; a versioned current artifact tuple and matching qualification profile are needed before recording comparable measurements. Keep workload/admission/completion/drain rules intact. Also complete the representation audit: Laravel queue payloads serialize PHP job objects, and internal async payloads can contain signed serialized closures inside Avro binary values. Avro-only wire transport does not by itself make those representations Rust-readable. Upgrade fixtures must exercise explicit tested compatibility/conversion or safe preflight refusal, without discarding acknowledged work.
All unproven #325 acceptance items remain open, including full SDK/consumer directions, all database families, multi-node interruption, Rust throughput/memory/idle long polls, in-place takeover, schema-marker refusal, operator cutover, publication and downstream verification. Rust development will record target-directory size and clean/incremental compile time and use bounded task-owned build state.
rmcdaniel commentedon Oct 9, 2026
The fixture foundation is merged in #326 at
ed49098b3db6b43827c9d90ba9b38f0104f2342d. Three shared cases pass on the pinned published PHP image and embedded Workflow through real SDK worker and Laravel queue execution; all 25 comparator counterexamples pass. Fixture qualification and full feature/source qualification passed at the exact PR head. Merged branch deletion is verified.Next is the PHP baseline: the existing capacity 1.9.0 tuple pins Server 2.1.1 and the standard local profile requires a different kernel. I will preserve those historical inputs and add a versioned current-artifact input/profile through the existing capacity commands, retaining their offered-load, completion, full-window and drain rules. Throughput, memory and idle mixed long-poll measurements remain unrecorded; Rust implementation and all remaining #325 gates stay open.
rmcdaniel commentedon Oct 9, 2026
The pinned PHP measurement profile merged in #327 after all source checks passed. Three 120-second windows at 0.25 offered starts/s each completed and verified all 30 planned workflows with zero pending work or poll/result/history errors. Higher-rate repetitions and the following drift control are running; no maximum-capacity result or Rust improvement is claimed.
Three mixed-idle repetitions each at 6, 12 and 24 polls completed with clean worker deregistration, no claimed tasks and no runtime OOM/restarts. Outcomes were consistently 6 empty; 8 empty/4 explicit capacity rejections; and 8 empty/16 rejections, respectively. Query waits lasted about five seconds despite requesting ten. All 24-poll bursts produced at least one two-second health/readiness timeout; an Apache worker-limit warning was observed. The failed API headroom observations are retained, rather than treating admission correctness as a complete pass. Changed background workload also requires a matched-condition PHP/Rust resampling before any performance gain claim.
Draft #328 records these bounded results and a stored-representation audit checkpoint. Avro transport does not remove PHP-specific queue jobs, signed serialized closures, encrypted history cursors or local class execution. Their compatibility/conversion, backup-first migration, schema marker and PHP refusal tests remain explicit open gates.
Next: finish offered-rate/drift observations, retain their complete limits and failed evidence, remove the reference experiment resources, then implement the first Rust slice without changing the default PHP artifact.
rmcdaniel commentedon Oct 9, 2026
The PHP development reference is complete, with bounded results in #328. Three 120-second windows at each of 0.25 and 0.5 offered starts/s completed and verified every workflow in-window. At 1/s, only 48–51 of 120 completed in-window before the remainder drained; this does not establish sustained 1/s throughput. The original second high-rate run's configured hourly worker exit invalidated its resource comparison. Its raw result remains retained, together with the replacement's one readiness 503.
All three 24-poll registration bursts produced ordinary API timeouts. Smaller mixed-poll runs passed admission and API checks; every registration was cleaned up. All 810 offered workflows, including the invalid resource sample and two following drift windows, completed and verified. Final synthetic counts were 846 completed runs including setup, 2530 completed tasks, zero failed jobs and zero idle registrations. Input identities, failures and sampling limits are retained; disposable experiment resources and dependencies are removed.
The result is diagnostic: changing background workload, admission sampling gaps, incomplete client resource costs and the development host's different capacity topology still require a matched PHP/Rust comparison and full qualification. Stored PHP jobs/closures, cursors and local execution also remain explicit conversion/compatibility gates in the audit. Next: merge the exact-head-qualified report, then add the first Rust slice with clean/incremental build-time and physical disk measurements. PHP remains the published default.
rmcdaniel commentedon Oct 9, 2026
#329 merged the first unpublished Rust foundation at
d684f2ff1a85c1aa49b7febd877139bf2db631ab. All final-head source checks passed, including the PHP feature suite, four Rust codec tests, 15 two-way PHP/Rust codec cases and the real PHP/embedded workflow fixtures. Fixture evidence is retained for 90 days. No Rust HTTP workflow, migration, runtime gain or cutover is qualified yet.The build observations keep the tradeoff explicit: native clean builds around 22 seconds/182 MiB; native no-op/comment edits 0.063/0.374 seconds; Kache fresh-target reconstruction 3.462 seconds, with slower cold builds and added cache/target disk. It remains optional. Build/cache/download state, task tool images, Kache sandbox, clean worktree and merged local/remote branch are removed; thin receipts remain.
Next: add the HTTP/storage slice with honest readiness and safe upgraded-database boundaries. Full malformed-input/resource qualification precedes exposing payload decode at HTTP ingress. The complete capability/client/database/performance/one-way-upgrade gates remain under #325; PHP stays the published default.
rmcdaniel commentedon Oct 9, 2026
PR #330 merged as
d2feae2a54c3630271e1d4e5217740f09031d2d6. It adds an explicitly experimental Rust HTTP/SQLite runtime that starts real workflows, leases workflow/activity tasks, persists typed history and opaque Avro envelopes, fences stale attempts, and commits/replays completion through the unchanged published PHP SDK.The three reviewed fixtures match across isolated published PHP, Rust and embedded Laravel targets. The final-head shared Action also passes four codec tests, seven transaction/HTTP tests, 15 two-way codec values and a real SIGKILL/replacement-container check: the persisted activity lease expires, the old attempt is refused, and a fresh SDK worker completes attempt two with one committed result and the original identities/history. The complete PHP feature/source qualification, repository contracts and public-boundary checks pass.
Bounded compile measurements at two CPUs/2 GiB: a clean test/binary build took 54.65 s, with 521.4 MiB of targets plus 202.6 MiB of downloads; binary-only no-op/comment edit took 0.16/1.51 s. Inputs, exclusions and cache growth are in
rust/runtime-build-observations-2026-10-09.md. This is a development observation, not a full-runtime or performance acceptance claim. Disposable resources and the merged worktree/branch have been removed.PHP remains the published default; native Rust development databases do not qualify for PHP takeover. Next: enforce PHP refusal before any upgraded-database mutation, then extend the native schema and backup-first sequential migration foundation. MySQL/PostgreSQL, the remaining API/client/capability surface, strict ingress/external payloads, full recovery/CPU-load/performance qualification and published artifacts remain open. #325 stays active.
584 remaining items
Rust #325: keep probe retry notes when the probe succeeds
Rust #325: compare the recorder note as plain text
rmcdaniel commentedon Oct 29, 2026
Checkpoint, October 29: #854 merged (
4e97c4ea), with Cloud's and Security's reviews.many-updates: 65 completed updates leave the signal wait, and a 66th accepted update is described as the open wait. Published PHP and native both pass.request_id. The handoff is in Investigate intermittent approved-update HTTP 500 in published PHP MariaDB corpus #825;METHOD pathlabel with a status.In review: #858 (worker-plane error bodies). Ready locally: parents with several open children.
Rust #325: report duplicate poll task kinds with the other failures
Rust #325: compare distinct values with PHP's loose equality
Rust #325: compare numeric strings and distinct candidates as Laravel…
rmcdaniel commentedon Oct 29, 2026
Checkpoint, October 29: #858 merged (
609fde24), with Cloud's review.validation_failedin the worker envelope. This includes Laravel'sdistinctontask_kinds.*with PHP 8's loose==:zendi_smart_str_equalsfor numeric strings, andArr::dotcandidates.worker-error-bodies: 20 requests. Published 2.11.5 and native pass, and a native test replays them all.Also today: the 2.11.6 reference survey (#863) found published regression #861, which General owns; the move waits on the corrected release. Next: parents with several open children.
rmcdaniel commentedon Oct 29, 2026
Checkpoint, October 29: #865 merged (
c2cdbaf9), with Cloud's review.child-descriptions: 5 cases; published 2.11.5 and native pass. Its probe terminates the children and the parent after recording.rmcdaniel commentedon Oct 29, 2026
Checkpoint, October 29: #866 merged (
d8418877).child-descriptions: 8 cases. Added: an activity before a child, a child before an activity, and a child started with a cancellation policy, which both refuse with409 child_cancellation_policy_not_supported. Published 2.11.5 and native pass.In CI: #867, the parity reference move to published 2.11.7 (Closes #863) with the #770 canonical history meter. Ready locally after it:
bridge-adapterscompared exactly, because 2.11.7 answers #747 and #777 as intended.rmcdaniel commentedon Oct 29, 2026
Checkpoint, October 29: three PRs merged.
6af963db): the parity reference moved to published 2.11.7 (Closes Rust #325: move the parity reference to the corrected published release (2.11.7) #863), with native's v1 history-byte meter in Core's canonical unit (Waterline v1 projection import has no event or history byte limit #770). Onlywaterline-import's fivehistory_size_bytesvalues changed.bridge-adapterscompares exactly. 2.11.7 answers Bridge adapter update_workflow skips external update admission, so SDK-worker updates are always rejected #747 and Bridge adapters: foreign-instance type disclosure, non-atomic dedupe and colliding update keys, unbounded adapter name #777 as intended. The four other published-defect tolerances stay, because 2.11.7 still shows them.8ace5c15): child retry and timeout options.retry_policy,execution_timeout_secondsandrun_timeout_secondsonstart_child_workflow.startChildRetryIfAvailable).ChildRunFailednamesclosed_reason: timed_out.child-retries. The first hosted run caught a MySQL-only decode and placeholder bug, which is fixed and passes locally on MariaDB 10.11, PostgreSQL 17 and SQLite.Tracked: #873, cooperative cancellation delivered to a retried child. Native refuses it, and a fix needs a PHP recording first.
In review: #874. Child runs and retries inherit the parent's memos and search attributes. It has a new
child-visibilityfixture, which passes on 2.11.7 and native.rmcdaniel commentedon Oct 30, 2026
Security pre-publication environment readback — 2026-10-30: current main .github/workflows/native-release.yml assigns all three manual, main-only native publication jobs to dockerhub-native, but the repository environment API does not yet list that environment. Before the first native RC/publication, provision it with an explicit main-only deployment-branch policy and the intended scoped Docker Hub credentials, then verify the dispatch guard and retained public checks. This is preparation for the first publication, not a current Security hold; pre-RC development and qualification should continue. No secret value or secret metadata was queried, and no repository/environment setting changed.
rmcdaniel commentedon Oct 30, 2026
Checkpoint, October 30: #874 and #876 merged.
9670e037): child runs and retries inherit the parent's memos and search attributes (inheritTypedVisibilityMetadata). New fixturechild-visibility.73bc484d, closes Rust #325: cooperative cancellation delivered to a retried child #873): a parent's cooperative cancellation reaches a retried child. The delivery follows the call to its current run and link. Undertry_cancel, the retry run, not the failed first run, is asked to cancel. New 2-case fixtureretried-child-cancellation, recorded against 2.11.7 first.Also handed off and closed: workflow#931's published-PHP acceptance. General ran it and Security accepted it.
Remaining before a first release candidate (rough estimate 2 to 4 weeks at the current CI pace; the least certain item is performance):
dockerhub-nativeenvironment must be provisioned first. The operator's scope decision on prepared local activities, workflow-bound Nexus handlers and external payload takeover is also needed.Next: item 2, takeover of the 2.11.7 layout.
rmcdaniel commentedon Oct 30, 2026
Operator direction (October 30), for cross-host recovery: the goal is speed. The October 24 fix-CI-first rule applies only when CI is measurably backed up (jobs waiting for runners), not as a standing one-PR-at-a-time limit. Otherwise, open PRs as soon as they are ready, keep several in flight, and work independent slices in parallel. Measured then: the last two Shared Server fixtures runs had zero runner queue wait (43–44 jobs, longest about 18 minutes). The same text is in the Rust dev host's workspace AGENTS.md.
Also from the operator, the same day: publish native release candidates (
native-X.Y.Z-rc.N) to GHCR now and add Docker Hub later (#883). The Docker Hub repositorydurableworkflow/server-nativeand its scoped token do not exist yet.rmcdaniel commentedon Oct 30, 2026
Native release candidate 1 published (October 30). Following the operator's decision, it is on GHCR only.
native-3.0.0-rc.1(annotated), source03a3e7b7c02281ee8ff58fb53180e428e866ad39: main through Rust #325: cover takeover of published PHP Server 2.11.7 databases #881, Rust #325: answer a repeated activity result as published PHP does #882 and Rust #325: publish native release candidates to GHCR without Docker Hub #883.ghcr.io/durable-workflow/server-native:3.0.0-rc.1, indexsha256:43718810bb9d1657aefe5fa157dfdd894c8f2a6d3e7edd6449a2ae3d0add6c86.docker pullof the amd64 image works, and its revision label is the tagged commit.Next: the matched PHP/native performance qualification against this image. It runs on the Rust dev VPS (6 vCPU), because the disposable-host path through Cloud is paused. The results and host facts will go in
benchmarks/server-port/rc1/.rmcdaniel commentedon Oct 30, 2026
Release-candidate performance gate: met on the published image (#886,
e824331e). Three alternating pairs ran on one host: the publishedserver-native:3.0.0-rc.1(sha256:43718810…) against the profile's recorded PHP baseline (Server 2.6.2). At 1 workflow/s:Idle polls: native answered every poll empty with no health failures. PHP answered with backpressure and failed health checks at 12 and 24 idle polls.
Caveats:
codec-deadline-health, not this run.Observations.