Skip to content

Versioned activity workers never see unpinned activity tasks #908

Open

Description

@rmcdaniel

Current delivery

Workflow 2.11.2 is published at a730773a; all published upgrade checks
pass. PHP Server #689 is merged at c36710e8 / whole d4b4fa43, with
independent main source, image-readiness, topology, CodeQL and Sonar checks
passing. Detailed evidence.

The replacement Server image is not published. Remaining delivery requires
the existing protected/security gates (shared #237
and Server #673), both
published architectures and the actual rolling-deployment consumer check.
Sample #274 also
qualifies both embedded locks at 2.11.2. This issue stays open until those
affected consumer outcomes are verified; a library tag does not update an
existing Server image.

What happens

An activity task with no build pin (workflow_tasks.compatibility is null) is never offered to a worker that registered a build_id. If every worker on the queue is versioned, the activity stays ready with attempt_count 0 and the run waits forever.

This happens in the normal rolling-deploy overlap. While builds v1 and v2 are both live on a queue, WorkflowStartVersionPin finds no single cohort and leaves new runs unpinned. A versioned worker takes the unpinned run's workflow task, because the workflow poller treats a blank pin as compatible with any build. The activities that workflow task schedules are then visible to no versioned worker.

Reproduction

Published Server 2.11.1 (ghcr.io/durable-workflow/server@sha256:bf8bed13f5f809ee3425233180a581353eac4683adbee5c4ecbf63bb18bbd102, Workflow 2.10.2), SQLite, protocol 1.20.

  1. Register wA with build_id: "v1" and wB with build_id: "v2" on queue q. Both support workflow type echo and activity type a.
  2. Start p2 (echo) on q. Describe shows compatibility: null.
  3. wB polls the workflow task, gets p2, and completes it with schedule_activity a.
  4. wB polls activity tasks three times with timeout_seconds: 1, then once with timeout_seconds: 5. wA also polls. Every poll returns no task.
  5. The workflow_tasks row for the activity is ready, compatibility null, and available_at is in the past.
  6. Register an unversioned worker wC. Its first activity poll returns p2's activity.

Where

DefaultActivityTaskBridge::poll adds ->where('compatibility', $compatibility) whenever the poller has a build id. That excludes null rows. The workflow side does not: WorkflowTaskPoller::matchesCompatibility returns true for a blank task pin, and nextVisibleReadyAt includes null rows for a versioned poller. The activity poller's own matchesCompatibility also returns true for a blank pin, so only the bridge query is stricter. This is the same on main.

Expected

An unpinned activity task can be claimed by any worker on the queue, as unpinned workflow tasks are. A pinned task is claimed only by its build.

The Rust server (#325 rewrite) routes activities by that rule. It records this as a contract decision rather than copying the stranded behavior.

Activity

  1. rmcdaniel commented on Oct 23, 2026

    @rmcdaniel
    MemberAuthor

    Accepted P1: versioned workers must receive unpinned activities matching their
    namespace, queue and activity type; pinned tasks remain build-isolated and
    claims retain existing execution/attempt/lease authority.

    Core 2.11.2 is published
    at a730773a3dcdf4c6d4ea0297f6c9010ccc0603f9, exact qualified whole
    579372329eb58d34aed5fa7b4d071fc51479d10f. All candidate/main checks and
    all seventeen published package/upgrade checks in
    39906612424
    pass. Packagist, normal installation and the loaded bridge hash match.
    The complete 68 bridge cases / 412 assertions pass on each actual SQLite,
    PostgreSQL 17.11 and MySQL 8.0.46 with no skips. Five test-first failures and
    the original style-only CI failure remain recorded in #909.

    PHP Server #689 normally
    merged c36710e860a44c5d51f016eadabd49f5b85c451e, whole
    d4b4fa4305041f94b69d57938981e1a8d418369b. It changes only the published
    package pin/lock, matching future-ready selector and the two-build regression.
    All fourteen applicable candidate checks pass, one explicit skip. Independent
    main now passes all twelve applicable checks and one explicit skip, including
    feature/source,
    MySQL topology,
    CodeQL, Sonar and boundary. The remote head is verified absent.

    Nine actual PHP HTTP-kernel cases / 353 assertions pass with normally installed
    Core 2.11.2: two live builds, default/named, unpinned/pinned, exact activity and
    parent completion, plus future-ready scope refusals. Old Core 2.11.1 has five
    failures while all four pinned full-completion controls pass. This is protocol
    qualification, not TCP or a published Server image. PR689 has zero open or
    confirmed Sonar issues; that does not claim main has no historical findings.

    Remaining delivery: Server 2.11.2 is not published. Its existing protected
    CI/security gates (#237 and Server #673), exact rebuilt architectures and
    published rolling-deployment consumer acceptance must finish before this issue
    closes. Tagging Core does not change an existing Server image. The exact PHP
    source handoff has been delivered to the native owner for independent combined
    qualification, preserving all shared/frozen identities.

    Sample #274 updates
    both embedded Core locks and the embedded tuple to published 2.11.2, preserving
    Server 2.10.0 / Core 2.9.0 and every SDK/CLI/Waterline selection. The first image
    guard caught a missed microservice lock; its failed result remains recorded.
    Corrected 95e3640 requires fresh normal qualification. Waterline's Core pin is
    development-only; embedded hosts own their runtime upgrade. No unchanged SDK
    publication is implied.

    Published Sample consumer follow-through

    Published consumer acceptance is complete for this Sample change.

    • Candidate 95e3640c passed all 32 checks. Merge de8a36108dd6ea0b723578b5a128ef2c7149d0df preserves whole tree 14b93ad8edc1c4138443b57f479075cd48194f09; all 30 independent main checks pass.
    • Normal image publication 39913495254 passes all six jobs, including credential-free published-image acceptance on native amd64 and arm64 and promotion to both main channels.
    • Independent anonymous registry inspection now verifies both ghcr.io/durable-workflow/sample-app-devcontainer:main and durableworkflow/sample-app-devcontainer:main. Both have amd64 child sha256:71e097365c64fadaa01ecd13e72a4db64099079c5b222b50bba118f80c79cb65 and arm64 child sha256:b759104af17ff7aac239fcc3852b9abce55cd2a4cee2e06b849e4e4f22b15701. Both child configurations identify exact merge revision de8a36108dd6ea0b723578b5a128ef2c7149d0df.
    • Both embedded application locks use published Core 2.11.2 / a730773a. Every other package and the separate Server/SDK/frozen tuple components remain unchanged.

    The first candidate's missed nested-lock seed failure remains failed. Initial manual registry readers also failed before correcting redirect handling; these verifier failures are separate from the final anonymous registry result. This delivers the embedded Sample consumer, not a replacement Server release or a new published Server worker-loss qualification.

  2. added
    priority:P1High-priority product or release risk
    status:in-progressApproved work is actively being implemented or validated
    completion:evidence-requiredClose only after all explicit acceptance and operational evidence is public
    on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    completion:evidence-requiredClose only after all explicit acceptance and operational evidence is publicpriority:P1High-priority product or release riskstatus:in-progressApproved work is actively being implemented or validated

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions