Auto-merge gate can silently skip an evaluation: adopt concurrency.queue: max

Why

GitHub Actions shipped a queue key for concurrency, and it closes a gap this repo is currently exposed to.

cancel-in-progress: false does not mean "every run eventually executes." It protects the run that is already executing; the run waiting behind it is still discarded when a newer one arrives:

So on a burst of three events in one group, run 1 executes, run 2 is silently cancelled, run 3 executes. There is no failure, no annotation, and nothing in the run list to say a run was dropped.

`queue`	Meaning
`single` (default)	"At most one job or workflow run can be `pending` in the concurrency group."
`max`	"Up to 100 jobs or workflow runs can be `pending` in the concurrency group." — FIFO

queue: max with cancel-in-progress: true is a validation error, so it only ever applies where cancelling is already switched off.

Announcement: https://github.com/orgs/community/discussions/12835#discussioncomment-16602100
Reference: https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#concurrency

Proven in this org already: devantler-tech/agent-skills has run queue: max in its release.yaml since 2026-07-11, with every Release run green since. This is propagating a fix we have already validated, not a trial.

Why it matters here specifically

.github/workflows/enable-auto-merge.yaml sets a job-level group:

concurrency:
  group: enable-auto-merge-${{ github.event.pull_request.number || github.event.issue.number || github.run_id }}
  cancel-in-progress: false

The group is per-PR, and this workflow is deliberately triggered by a wide event set — pull_request, pull_request_review, issue_comment, merge_group — precisely so the gate re-evaluates when reviewer results land after the original push. Those events routinely arrive in quick succession on one PR: a synchronize, then a CodeRabbit review, then a Codex result comment.

Under the default queue: single the middle evaluation is cancelled. The file's own comments state the requirement that makes this serious:

A dropped evaluation is therefore not just a missed arming — it can be a missed disarm, leaving a PR armed on evidence that has since been withdrawn. That is a correctness defect in a fail-closed gate.

This is also the highest-leverage instance in the portfolio: this is a reusable workflow consumed across repos, so one fix propagates to every consumer.

What

Add queue: max to the enable-auto-merge job's concurrency block so every triggering event gets its own evaluation, FIFO.

Audit the other workflows in this repo while here — lint.yaml and validate-go-project.yaml use cancel-in-progress: true, which is correct for superseded CI and must stay as-is.

Acceptance criteria

The enable-auto-merge job carries queue: max; cancel-in-progress: false is retained.
A burst of PR events on one PR produces one evaluation run per event, none cancelled.
cancel-in-progress: true workflows are left unchanged.

Part of devantler-tech/monorepo#2453 · Rough size: XS

Blocker: #277 | upstream | last-verified 2026-10-10: the three repository opt-ins still read true; this repository has now run two eligible evaluations under its opt-in (two Dependabot pull requests on 2026-10-09, both auto-merge jobs succeeded, nothing cancelled in 127 complete runs), but each pull request had exactly one, so no burst of real evaluations has been observed; the blocker issue records the decision to rely on the native three-job queue check plus this steady state, and this issue closes when every repository under the gate has the opt-in

🔗 Related PRs

devantler-tech/actions#412 - ci: skip the suite on the release-commit push to dodge the self-pin tag race [merged]
devantler-tech/actions#427 - fix: resolve first-party self-references at the same commit [merged]
devantler-tech/actions#575 - fix(auto-merge): harden mutable review evidence [merged]
devantler-tech/actions#594 - fix(validate-go-project): raise govulncheck timeout to survive cold-cache scans [merged]
devantler-tech/actions#599 - fix(auto-merge): complete ineligible required runs [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

chore: agent-claim #725 nonce=fba5e8c4b21fcd29f03f90e287c34c26

This is already implemented — do not build it again

I picked this up as the oldest unblocked bug and found the code already in place. Recording that here so the next run does not re-derive it, which is the waste pattern this portfolio keeps paying for.

It shipped in 81ada4a, "add opt-in retention of queued evaluations" (#1151). Verified at the current tip of the default branch: all three concurrency blocks in the auto-merge workflow carry the key — the workflow-level one and both job-level ones — and cancel-in-progress: false is retained on the job-level pair.

Against the acceptance criteria

The job carries the key, with cancellation still off — met, with one deliberate refinement worth noting. It is not the literal constant this issue asked for; it is an expression that resolves to the queued behaviour when opted in and to the default otherwise. That is this portfolio's feature-flag-first convention rather than a shortfall: the change lands latent and activation is a separate, reversible step.
Workflows that cancel in progress are left unchanged — met. The three that cancel are untouched, which is also required, since the two keys are a validation error together.
A burst of events produces one run per event, none cancelled — not met, and cannot be while the opt-in is off. This is the only criterion still open.

What actually remains

Activation, then the behavioural proof. There are two ways in, and they differ in who can do them:

A repository variable, which is a settings change and so the maintainer's.
An input on the call, which a caller repository can set in its own workflow — squarely inside what an agent may change.

The second is the interesting one, because it means this does not have to wait on a settings toggle. A single caller could opt in, the burst behaviour could be observed there, and the rollout could widen from that evidence.

Suggestion

Re-scope this issue to activation and verification rather than implementation, or close it and open a small successor for the rollout. I have not changed either, since which caller goes first is a judgement about blast radius that belongs with whoever picks the rollout up.

Unassigning myself and releasing the claim — there is no code to write here.

Decision: activate with a canary on `devantler-tech/actions`, then roll out org-wide

I checked the one route I could have taken myself, and it is not available.

The caller-input route does not exist. No repository calls enable-auto-merge.yaml through workflow_call. It runs in ksail, platform, monorepo and elsewhere as a required workflow, and that ruleset is managed in the UI rather than declaratively in .github/deploy/. A direct required-workflow run has an empty inputs context, so the only switch it reads is the QUEUE_PENDING_EVALUATIONS variable. Adding a workflow_call caller just to set the input would run the gate twice on every event, which is worse than leaving it off.
Nothing declares org Actions variables in Git. .github/deploy/ manages rulesets, repositories, teams and labels, but not variables. So setting the variable is a settings change, and settings changes belong to the maintainer.

Recommended order, smallest blast radius first:

Set the repository variable QUEUE_PENDING_EVALUATIONS=true on devantler-tech/actions only. This is the canary: the workflow's own PRs see the most review churn.
Trigger a burst on one PR there (push, then a CodeRabbit review, then a Codex comment) and confirm every evaluation runs, with none cancelled. That is the one criterion still open.
Set it at the organisation level and delete the repository variable.

Rollback at any point is deleting the variable. It changes only how many pending evaluations are kept (up to 100 instead of 1). Cancellation behaviour is unchanged, and GitHub rejects queue: max together with cancel-in-progress: true, so the two cannot combine.

Unblocked — the canary is live

Read the Slack thread you pointed at. Your answer:

Verified against the API rather than taking it on trust: devantler-tech/actions now carries the repository Actions variable QUEUE_PENDING_EVALUATIONS=true, updated 2026-09-21T18:01:32Z. The blocked label and the **Blocker:** line are removed — this is ordinary actionable work again, and the authority half of it is resolved for good.

What is left

Verify the canary on this repository. The variable only changes behaviour when several evaluations land on one PR close together, so the check is whether a queued re-evaluation is now retained instead of dropped. That is the next step and it needs no further permission.
Promote org-wide once it holds. You have explicitly allowed me to set the variable myself, so the remaining rollout is mine to do — I will do it after step 1 produces evidence, not before, since running the canary first was the whole point of asking for one repository rather than all of them.

Nothing here needs you again unless step 1 says the behaviour is wrong.

Passive observation does not verify the canary — recording that before anyone reads it as verification

I looked for the drop in the runs this repository has already produced, and that approach cannot answer the question. Saying so rather than reporting the clean numbers as a pass.

Measured over the last 100 enable-auto-merge.yaml runs on devantler-tech/actions:

window	runs	conclusions
before the variable was set	90	`success=90`
since `2026-09-21T18:01:32Z`	10	`success=10`

Zero cancellations on either side, so the conclusion field does not discriminate — the ten post-canary runs include a dense cluster (seven in fourteen minutes, three inside twenty seconds) and it looks exactly like the ninety before it.

The reason is in the workflow itself: the concurrency group is keyed per pull request, so evaluations only contend when they land on the same PR. The bursts in this window were spread across different PRs and never contended at all. A green reading here is therefore evidence that nothing regressed, and evidence of nothing else.

What verification actually needs

A controlled burst on one pull request in this repository: two or more evaluation-triggering events on the same PR inside the window where the first is still running, then a check that the second was retained rather than superseded. That is constructible here and needs no permission — it is the next step on this issue.

Until that runs, the canary is live but unverified, and the org-wide promotion stays behind it.

Where this stands, and the one constraint that would have invalidated the obvious experiment

Two of the three acceptance criteria are already met on main, so only the burst is outstanding:

✅ the enable-auto-merge job carries the queue expression with cancel-in-progress: false retained (workflow-level at L170–176, and on both mutation jobs at L361–364 and L424–427);
✅ QUEUE_PENDING_EVALUATIONS=true on this repository (set 2026-09-21T18:01:32Z);
⬜ a burst of PR events on one PR produces one evaluation run per event, none cancelled.

The constraint: the canary variable is only read by DIRECT runs of this workflow in this repository. The queue expression gates the vars.QUEUE_PENDING_EVALUATIONS branch behind

toJSON(inputs) == '{}' && startsWith(github.workflow_ref, 'devantler-tech/actions/.github/workflows/enable-auto-merge.yaml@')

so a consumer repository's PR resolves queue to single unless it passes queue-pending-evaluations as an input. Bursting on a consumer PR therefore measures the unfixed path and returns a false negative — which reads exactly like a failed canary. The burst must run on a pull request in devantler-tech/actions itself, and there is currently no open PR here to carry it.

Executable protocol for whoever picks this up

Use a real PR in this repository — not a throwaway. The experiment needs no particular content, so the next substantive change here should carry it rather than manufacturing CI noise.
On that one PR, fire three issue_comment events in rapid succession (well inside one evaluation's runtime, so runs 2 and 3 are genuinely pending rather than sequential). Record each comment's created_at.

Read the runs for that PR's concurrency group:

gh api "repos/devantler-tech/actions/actions/runs?event=issue_comment&per_page=100" \
  --jq '.workflow_runs[] | select(.path==".github/workflows/enable-auto-merge.yaml") | {id, created_at, status, conclusion}'

PASS = three runs reach a terminal state, none with conclusion: cancelled. FAIL = a middle run is cancelled, which is the silent skip this issue is about.

Why it has to be done this way

Recorded on 2026-09-22 so it is not re-derived: passive observation cannot verify this, and its numbers look like a pass. The concurrency group is keyed per pull request, so evaluations only ever contend when they land on the same PR — and a sample of 100 enable-auto-merge runs (90 before the canary, 10 after) showed success throughout with zero cancellations simply because that traffic was spread across different PRs and never contended. The conclusion field does not discriminate until the burst is deliberately concentrated on one PR.

Not started in this run: the burst needs a PR here to ride on, and a partial run would produce exactly the clean-looking numbers above. Claim retired; unassigned and unblocked for the next run.

Correction: the recorded burst protocol cannot fail, so it cannot verify anything

Picked this up again and re-checked the premise and the plan against main at 0e07438 before building anything.

The key is real. GitHub's workflow-syntax and concurrency docs define concurrency.queue with single (default, at most one pending run, a newer one replaces it) and max (up to 100 pending, FIFO), and reject queue: max together with cancel-in-progress: true. The implementation is on main (all three concurrency blocks), so there is still no code to write.

The protocol in the previous comment would report a PASS whatever the queue setting is. Three issue_comment events from devantler on a PR here never share a concurrency group:

Workflow level: a comment run only joins the shared per-PR state lane when it is a deleted review-bot comment on a trusted-bot PR with merge gates enforced. Every other comment run's group ends in its own github.run_id, so each run has a private group.
Mutation jobs (the only other lane): they run only when eligibility classifies the event as eligible. A comment is eligible only when its author is a trusted review bot (CodeRabbit or Codex) and the PR author is a trusted bot. A devantler comment is never eligible, so both mutation jobs are skipped and never enter the lane.

So "three runs, none cancelled" is guaranteed by construction. The same applies to any devantler-authored PR, which covers every agent PR in this repository.

What a valid burst needs

Two or more eligible evaluations must be pending in the same group at once. In this repository that means one non-draft PR authored by a trusted bot (Dependabot, Renovate, etc.) and either:

several pull_request lifecycle events close together (for example a close and reopen repeated while the first evaluation is still running). This is only meaningful while actor-trust enforcement is off here: with it on, those events are deliberately cancel-in-progress: true with queue: single, so a cancelled middle run would be expected, not a defect; or
with merge gates enforced, a CodeRabbit review followed by a Codex result comment on that bot PR, both landing inside the first evaluation's runtime. This is the scenario the issue was written about.

PASS = every run in that burst has a job that waited on the group (its started_at is after the previous job's completed_at) and none is cancelled. Runs that never overlapped prove nothing, as the 2026-09-22 sample already showed.

I have not triggered either burst. Both mean manipulating a live Dependabot PR, and whether actor-trust and merge-gate enforcement are on in this repository has to be read from its settings first, which I cannot see from here. Leaving this open for the canary verification. Claim released.

Generated by Claude Code

Canary exposure measured: zero eligible evaluations so far

I read the job list of every enable-auto-merge.yaml run on this repository since the variable was set (2026-09-21T18:01:32Z → 2026-09-23T20:50Z): 155 runs, 131 issue_comment and 24 pull_request_review, and not one of them ran a mutation job. Only eligibility executed in each.

That is why the earlier "zero cancellations" reading proves nothing: nothing ever entered the queued lane. The reason is who authored the PRs in the window. Every review-bot event landed on a botantler-1 PR, which is not in TRUSTED_BOT_AUTHORS, and the reviews on the two Dependabot PRs came from botantler-1, which is not in TRUSTED_REVIEW_ACTORS. So the canary has had no exposure, and waiting passively will not change that on its own.

Verification plan, recorded so the next run can execute it without re-deriving: the next time a Dependabot PR is open and non-draft here, post a CodeRabbit review request and a Codex review request on it back to back. Both reviewers are trusted review actors and the PR author is a trusted bot, so both result comments are eligible evaluations in the same per-PR mutation group. PASS = every resulting mutation job either ran or waited (its started_at is after the previous job's completed_at), and none is cancelled.

No code change is needed; this stays open for that measurement.

Current canary readback on 2026-09-24: the repository variable QUEUE_PENDING_EVALUATIONS=true remains present. ENFORCE_ACTOR_TRUST and ENFORCE_MERGE_GATES have no repository entries and do not appear in the complete applicable organization-variable listing. The only open Actions PR is devantler-tech/actions#1335, authored by devantler, so it is not an eligible trusted-bot canary. No burst or settings mutation was performed, and this remains unverified.

One correction to the latest proposed protocol: requesting CodeRabbit and Codex back to back conflicts with the consumer contract's rule that only one review-provider request may be active at a time. Do not manufacture a second review after a successful first review just to produce queue traffic. A future canary should use an eligible, real trusted-bot PR and a permitted event sequence that demonstrably overlaps in the actual mutation concurrency group. Record pending/start/completion timestamps and eligibility for every event; green eligibility-only runs still prove nothing about retention.

Canary re-measured 2026-10-01: the actions-repo canary has retired, and it never got any exposure

devantler-tech/actions: 400 enable-auto-merge.yaml runs between 2026-09-23T20:52Z and 2026-09-25T21:12Z (total_count 400, so the list is complete). There were 345 issue_comment runs and 55 pull_request_review runs. All 1,200 jobs were eligibility successes, plus auto-merge and disarm-untrusted-update skipped. Not one evaluation reached the queued mutation lane. There have been no runs since 2026-09-25T21:12Z, which matches the actions → .github consolidation. The QUEUE_PENDING_EVALUATIONS=true variable there now guards nothing.
devantler-tech/.github (the workflow's home now): enable-auto-merge.yaml reads QUEUE_PENDING_EVALUATIONS only for direct runs whose workflow_ref is this file, and only from a repository or organization variable. This repository has no repository variables. The organization variable listing needs admin:org, which this run does not have, so whether the queue is armed here is UNKNOWN, not "no". There have been 883 runs here since 2026-09-25.

Next step: set the repository variable QUEUE_PENDING_EVALUATIONS=true on .github, or confirm the organization-level one, so the canary runs where the traffic now is. That is a persistent settings change, so it is not taken from an unattended run. Then repeat the job-level measurement above against .github's runs, using organic trusted-bot traffic only (per the 2026-09-24 correction). No code change is needed.

Navigation Menu

Repository navigation

Auto-merge gate can silently skip an evaluation: adopt `concurrency.queue: max` #286

Why

Why it matters here specifically

What

Acceptance criteria

This is already implemented — do not build it again

Against the acceptance criteria

What actually remains

Suggestion

Decision: activate with a canary on `devantler-tech/actions`, then roll out org-wide

3 remaining items

Unblocked — the canary is live

What is left

Passive observation does not verify the canary — recording that before anyone reads it as verification

What verification actually needs

Where this stands, and the one constraint that would have invalidated the obvious experiment

Executable protocol for whoever picks this up

Why it has to be done this way

Correction: the recorded burst protocol cannot fail, so it cannot verify anything

What a valid burst needs

Canary exposure measured: zero eligible evaluations so far

Canary re-measured 2026-10-01: the actions-repo canary has retired, and it never got any exposure

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Auto-merge gate can silently skip an evaluation: adopt concurrency.queue: max #286

Description

Why

Why it matters here specifically

What

Acceptance criteria

Activity

coderabbitai commented on Jul 25, 2026

devantler commented on Sep 20, 2026

This is already implemented — do not build it again

Against the acceptance criteria

What actually remains

Suggestion

devantler commented on Sep 21, 2026

Decision: activate with a canary on devantler-tech/actions, then roll out org-wide

3 remaining items

devantler commented on Sep 22, 2026

Unblocked — the canary is live

What is left

devantler commented on Sep 22, 2026

Passive observation does not verify the canary — recording that before anyone reads it as verification

What verification actually needs

devantler commented on Sep 22, 2026

Where this stands, and the one constraint that would have invalidated the obvious experiment

Executable protocol for whoever picks this up

Why it has to be done this way

devantler commented on Sep 22, 2026

Correction: the recorded burst protocol cannot fail, so it cannot verify anything

What a valid burst needs

devantler commented on Sep 23, 2026

Canary exposure measured: zero eligible evaluations so far

devantler commented on Sep 24, 2026

devantler commented on Oct 1, 2026

Canary re-measured 2026-10-01: the actions-repo canary has retired, and it never got any exposure

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Auto-merge gate can silently skip an evaluation: adopt `concurrency.queue: max` #286

Decision: activate with a canary on `devantler-tech/actions`, then roll out org-wide