Repository navigation
Roll out queued auto-merge evaluations and retire the temporary opt-in #277
Description
Activity
🤖 Generated by the Agentic Engineer
Review date 2026-09-12 passed: step 1 as written has no consumers
Step 1 cannot run as written. It says to enable
queue-pending-evaluationson a small set of trustedworkflow_callconsumers. No such consumers exist.What I measured on 2026-09-15:
- No repository calls
enable-auto-merge.yamlwithworkflow_call, except this repository's own two self-tests inci.yaml. One has the flag off (actor-trust-self-test) and one has it on (queued-self-test). Checked with an org code search for the filename, and a broader search forenable-auto-merge.- homebrew-tap only mentions the workflow in a comment.
- world-at-ruin's CD arms its tap PRs itself, because the org workflow only accepts allow-listed bot authors.
- Production traffic comes from the org-required runs. homebrew-tap and ksail have no workflow file for it;
contents/.github/workflows/enable-auto-merge.yamlreturns 404 in both. Yet both run🔀 Enable Auto-Mergeon everypull_requestevent: 17 of homebrew-tap's last 100 runs and 11 of ksail's. Those runs are the required workflow, where the flag's default applies.
So the real first step is this issue's own item 3: define and validate direct/required-workflow activation, while keeping an explicit
workflow_callfalsehonoured while the flag exists. The observation in item 1 (a real event burst plus steady state) then applies to that path. Release traffic makes homebrew-tap and ksail a natural first pair.New bounded review date: 2026-09-22. By then, either a draft PR defines direct/required activation behind the flag, or a named blocker is recorded here. #286 stays open until production acceptance is demonstrated, as this issue already says.
- No repository calls
- added a commit that references this issue
on Sep 15, 2026 🤖 Generated by the Agentic Engineer
Item 3 has a draft: devantler-tech/actions#1269. This meets the 2026-09-22 review date set above.
Direct and organization-required runs now opt in with a
QUEUE_PENDING_EVALUATIONSrepository or organization variable. The workflow reads the variable only when the run has no inputs and this workflow file is the one running. Aworkflow_callrun always carries its declared inputs and its caller's workflow reference, so an explicitfalseinput still wins.What the tests can and cannot show. The author-gate test pins the new expression on all three queue settings. It fails against the current
mainworkflow and passes on the branch. This repository cannot start an organization-required run of itself, so the direct path takes effect only once a repository sets the variable. That is item 1's observation step, and homebrew-tap plus ksail remain the natural first pair.Setting the variable changes repository or organization configuration, so it follows once devantler-tech/actions#1269 merges and ships.
- added a commit that references this issue
on Sep 15, 2026 🤖 Generated with Claude Code
The rollout variable is set, so this issue is no longer blocked.
QUEUE_PENDING_EVALUATIONS=truewas set as a repository variable on homebrew-tap and ksail on 2026-10-06 at 16:33Z. The maintainer's direction, given in an interactive session the same day, is that setting it is the engineer's own step rather than his. Direct and organization-required🔀 Enable Auto-Mergeruns on those two repositories now read the variable astrue; every other repository is unchanged.Next: the observation in item 1 on these two repositories: a real event burst plus steady state, recording retained, completed and cancelled evaluations, execution time and API rate-limit errors. To roll back, delete the variable on both repositories.
🤖 Generated by the Agentic Engineer
Fresh pilot observation, bounded to the records available on 2026-10-07 at 22:47Z: both existing repository opt-ins still read
true. Complete paginated run records since their 2026-10-06 16:33Z activation contain 84 KSail and 64 Homebrew Tap evaluations, all with a successful run conclusion.Those run conclusions alone do not satisfy this rollout. Complete job/step graphs for KSail run 37693667025, the earlier delayed Homebrew run 37519232004, and later Homebrew run 37696786137 all show successful ineligible admission with privileged classification, arming and disarming skipped. They are useful no-write controls, not a real eligible event burst or evidence of retained authorization evaluations.
No default was changed, and no pull request was mutated to manufacture a burst. The remaining observation must join an actually eligible consumer evaluation to its immutable producer and collect complete queued/executed job evidence and API outcomes. The legacy required-workflow metadata endpoint is deprecated (422); the complete organization ruleset inventory remains inaccessible to this session, so source attribution is not guessed from a workflow filename. Keep the temporary opt-in and #286 open; review this evidence gap by 2026-10-15.
🤖 Generated by the Agentic Engineer (interactive Claude Code session)
A third opt-in is on:
devantler-tech/.githubitself. The maintainer set the repository variableQUEUE_PENDING_EVALUATIONS=truethere on 2026-10-08 at 14:54Z; read back just after. The two existing opt-ins (ksail and homebrew-tap, both since 2026-10-06 16:33Z) still read true.This repository is where the auto-merge workflow runs directly and has the most traffic, so it is the likeliest place for a real eligible burst to appear. No such run has happened here since the opt-in (the newest run is from 10:19Z, before it), so nothing is proven yet and the evidence gap recorded above is unchanged. No pull request was touched to produce traffic.
Correction to my own note elsewhere: I told the maintainer #286 was blocked only on this variable. That was out of date; the pilot had already moved to two consumer repositories. The variable adds a pilot, it does not close the gap.
🤖 Generated by the Agentic Engineer
Pilot reading on this repository, 2026-10-10 (ahead of the 2026-10-15 review)
All three opt-ins still read
truetoday (this repository since 2026-10-08 14:54Z, ksail and homebrew-tap since 2026-10-06 16:33Z).What ran here since the opt-in. 127 runs of this repository's own
🔀 Enable Auto-Mergeworkflow. The listing's total matches what was read, so it is complete. All 127 finished successfully and none was cancelled.The first eligible evaluations under the opt-in. Two of the 127 were real, eligible evaluations, one for each Dependabot pull request opened on 2026-10-09:
Pull request Run auto-mergejobOutcome #562 37903680988 queued 08:14:11Z, ran 08:15:04Z to 08:15:28Z, success armed at 08:15:24Z, still open (parked on its own blocker) #563 37903955045 queued 08:21:30Z, ran 08:24:04Z to 08:24:25Z, success merged at 08:34:57Z I read the job list of every run on those two pull requests (19 runs). The other 17 admitted nothing: they were triggered by the organization bot's reviews, the coverage bot's comments and one engineer comment, none of which is an eligible actor, so both privileged jobs were skipped. No job in the 19 was cancelled. The remaining 108 runs were on agent-authored pull requests or on issues, which the rule never admits; I read their run results only, not their job lists.
What this shows, and what it does not.
- Steady state works with the queue on: an eligible evaluation runs to completion and arms the pull request.
- It still shows no burst. Each pull request had exactly one eligible evaluation, so two never waited in the same lane.
Why waiting longer is unlikely to produce one. The queue only matters when one eligible evaluation is running, a second is waiting, and a third arrives before the second starts. From queued to finished, the two evaluations above took 77 and 175 seconds. The only events that qualify on a bot-authored pull request are another lifecycle event (a new push, a reopen) or a CodeRabbit or Codex result. Reviews are requested by hand and one provider at a time, and both pull requests above were approved by the organization bot, which is not an eligible actor. Every earlier reading on this issue and on #286 since 2026-09-21 found no eligible evaluation at all.
What already covers retention. Every full CI run in this repository puts three jobs into one lane that uses the same two settings as the gate (
cancel-in-progress: false,queue: max), and the required check reads the native job list and fails unless all three completed successfully at that head. At the head CI tested for #562 the three ran one after another with no overlap (08:22:36Z to 08:23:00Z, 08:24:51Z to 08:25:19Z, 08:27:14Z to 08:27:36Z) and none was cancelled. That is the same scheduler behaviour the gate depends on, observed natively and on every full run, without touching a pull request.Decision for the review
Stop waiting for an organic double event. Item 1 is met by the two parts together: eligible evaluations complete correctly with the queue on (above), and the scheduler keeps every waiting job in such a lane (the native three-job check). Neither part alone would be enough, and I am not claiming a burst of real evaluations was observed.
Next steps (item 4), in this order. Neither starts from this note: the board's In Review and Verifying stages are both over their limits today (26 against 12 and 9 against 3, read at 11:02 UTC), so new starts hold until they drain.
- Widen the opt-in to the other repositories that run the required workflow, a few first (monorepo, platform, world-at-ruin), then the rest. The organization still requires the frozen copy of this workflow from
devantler-tech/actions(Repoint organization-required workflows from actions to .github #237). That copy cannot change, but I checked today that it reads the same repository variable. So the variable is the only way those repositories get the queue now, it needs no release, and deleting it is the way back. This is what closes the exposure in Auto-merge gate can silently skip an evaluation: adoptconcurrency.queue: max#286, and Auto-merge gate can silently skip an evaluation: adoptconcurrency.queue: max#286 closes when every repository under the gate has it. I did not set it in this run, which was limited to this repository andactions. - Make the queue the default here once Repoint organization-required workflows from actions to .github #237 points the requirement at this repository: on for runs of this workflow that have no caller, an explicit
falsefrom a caller still honoured, input and variable kept for one release as the way back. A later change removes the temporary plumbing and the variables.
One thing to watch while widening: retaining waiting jobs can add runner work during a burst (#475). In the 2026-10-09 burst the gate's own jobs waited between one and a half and two and a half minutes before starting. Eligible evaluations are rare (two in two days in this repository), so I expect no visible change; read the gate's job waits back after each step.
If you would rather keep waiting for a real double event, say so here and I will set a new date instead.
The opt-in
queue-pending-evaluationsflag introduced by devantler-tech/actions#1151 stages the #286 repair because retaining pending jobs can increase runner/API work. Existing callers and direct/required runs keep their current queue until rollout.Review date / target expiry: 2026-09-12. This is a temporary release flag, not a permanent setting. The expiry is a review deadline, never permission to remove it without evidence.
Acceptance:
concurrency.queue: max#286 only once its production acceptance is demonstrated. If rollout is deferred at the review date, record a specific blocker and a new bounded date here.GitHub's queue limit remains 100 pending jobs/runs; ordering follows the start of waiting rather than event dispatch. No App authority expansion is part of this task.