Repository navigation
CI takes hours to start #25088
Description
Describe the bug
Created more for visibility to others. CI now takes a long time to start across many PRs, both on push and in the merge queue. This seems to have started sometime between 2026-09-08 07:30 and 08:44 UTC.
Examples:
https://github.com/apache/datafusion/actions/runs/34238609607/job/102102700226?pr=25081
#25081
https://github.com/apache/datafusion/actions/runs/34238609420
Additional context
I believe that's ASF issue, https://issues.apache.org/jira/browse/INFRA-28374
Activity
blaginin commentedon Sep 8, 2026
I think that is related to other ASF projects too:
alamb commentedon Sep 8, 2026
I hit a bunch of timeouts when trying to merge PRs using the merge queue in arrow-rs-object-store:
alamb commentedon Sep 10, 2026
The root cause appears to be our DataFusion github runner usage is consuming all the github runners:
alamb commentedon Oct 20, 2026
This has been happening again a lot recently (taking hours to start running)
I have been exploring what apache projects are consuming jobs and Claude came up with this one
The current output is
It looks like a few PRs consumed a lot of the runner time (each launches 14 jobs via some matrix)
alamb commentedon Oct 20, 2026
The TLDR of my poking around is that some projects have many queued jobs (often from matrix'd runs) that are consuming significant numbers of runners and the DataFusion (and arrow-rs backlogs) are queued.
I'll ask the projects politely if there is some way to reduce the runner usage
mbutrovich commentedon Oct 20, 2026
Thank you!
alamb commentedon Oct 20, 2026
Current queue across projects
tballison commentedon Oct 21, 2026
Over on Tika, we're also seeing > hour long queues. It felt like there was a change a bit more than two weeks ago. The lengthy queues cannot be attributed to post Community Over Code exuberance.
This is not good.
Thank you for opening this ticket.
alamb commentedon Oct 21, 2026
I suspect it may be the fact that it is far easier than ever to make PRs via ai agents, and each PR often starts large amounts of CI
Also the new github stacks feature was released around Oct 6 (approximately the right timeline): https://github.blog/changelog/2026-10-06-stacked-pull-requests-generally-available/
And I found at least one example where a stack of PRs ended up running the full CI suite for each stacked PR each time the stack got rebased:
I don't have more data to back up that yet, but will keep searching
alamb commentedon Oct 21, 2026
Here is another snapshot
alamb commentedon Oct 21, 2026
I spent some time on some more analytics this morning on how many runner minutes are used by different projects
Here is the data + script I used: Archive.zip
Note this only includes the github hosted runners (not the RunsOn runners):
My numbers show that airflow has used used 77.2 days of runner time in the last 17h
By comparison DataFusion only used 11.7 days of runner time in the last 2 days
arrow-rs used 2.7 days over 17 hours
alamb commentedon Oct 21, 2026
There are also some great discussions in the ASF slack
https://the-asf.slack.com/archives/CBX4TSBQ8/p1792591324182189?thread_ts=1792520326.623569&cid=CBX4TSBQ8
That have come to the same conclusion
28 remaining items
potiuk commentedon Oct 24, 2026
BTW. Our (stacked - 5 PRS) "Your PR, Your CI" is also going through last round of reviews apache/airflow#74825 and I hope we merge it today and start piloting moving some volunteer contributors to be switched to it to test things out (after our testing of course). I believe Arrow had (and still has ?) some issues with their CI after they switched to "use runners of contributors" so we definitely do not want to rush it with 400 PRs a week.
alamb commentedon Oct 24, 2026
The DataFusion version is working well for the last week or so (though it took us a while to refine it)
I do think @sdf-jkl is still refining our arrow-rs PR:
potiuk commentedon Oct 24, 2026
Our version in Airflow (and one that will hopefully turn into reusable action set) - is way more sophisticated. We aim for practically no change in the workflow from the maintainer / contributor point of view:
The PR's status will be also set depending on the status of the fork run.
This all will be done via scheduled workflow that will check and update status for such open PRs. Generally we aim for practically 0 change in how you interact with GH PRs via UI. This all shoudl work - no need (such as in rust workflow) for contributors to post their workflow run links - and even know that something changed.
Once they enable the workflows in their fork - things will work for them "as usual" (with little delay on refreshing status of their jobs to the PR).
Once I get it into reusable action, maybe you could try to switch to it. I will post it here once I have it.
alamb commentedon Oct 24, 2026
i think this is basically what arrow and datafusion have also achieved. The only difference is the contributor needs to enable workflows on their fork
potiuk commentedon Oct 24, 2026
Cool - I saw workflow-runs not started - but possibly it was before
So maybe @alamb -> YOU would like to make it a shared action from datafusion - I am happy to review and approve new actions in https://github.com/apache/infrastructure-actions - where shared actions for ASF projects are added.
People seem to be interested - and you seem to have quite a bit more generic option in your PR that will be easier to make generic?
I think that would help others.
potiuk commentedon Oct 24, 2026
I did it for the "pull_request_target" replacement (doc-only) and few other actions and it seems that other ASF projects are picking it up - so it might be a good idea to use the opportunity when they need it.
Happy to promote it among others.
alamb commentedon Oct 25, 2026
Thanks for the suggestion; I haven't been doing the actual github action work (that is others with agents) so I don't think I have a lot to add as a shared action at this point (and just having my agent vomit a PR into that repo is probably less effectively than someone who knows what they are doing)
potiuk commentedon Oct 25, 2026
It's just a shared repo with actions - nothing special and it has a lot of harrness - it's very friendly for people who know how to use theor agents and would like to help other projects in ASF - with README/ AGENTS.md guiding the agents. And people like me reviewing it - so that shoudl be possible to add it by anyone.
"
I personally would like to have all the ASF maintainers to think in "How can I help infra and other projects" rather than "What I can demand from infra". That's my personal goal to increase a number of people who take care about more than their own garden.
alamb commentedon Oct 25, 2026
I agree
I was just trying to say I didn't think my adding a vibe coded github action to a shared repo is the best way to help other projects (their agents can probably do just as well, if not better)
kumarUjjawal commentedon Oct 25, 2026
@potiuk would something like #25591 be a good addition to the asf infra repo? I would be open to make it more generic and contribute towards the shared actions.
potiuk commentedon Oct 25, 2026
If generalized and made ready to use by others with instructions, readme and examples how to use it- absolutely.
potiuk commentedon Oct 25, 2026
FYI: apache/infrastructure-website#323 - another optimization I added to Airflow and can be implemented by anyone.
Magpie will automatically use it as a proposed optimization when it is merged to infra website.
alamb commentedon Oct 26, 2026
Anecdotally I see PRs from arrow-rs and datafusion running with shorter queue times.
Compared to last week, Airflow usage is down to about 1/3 of where it was (35days vs 100days) -- thank you @potiuk
alamb commentedon Oct 26, 2026
So while I am certainly a biased observer, my conclusion is that the work the Datafusion and Arrow communities have done over the last week have made our projects CI usage an order of magnitude lower than other large Apache projects (e.g. 1.6 runner days vs over 30 runner days)
potiuk commentedon Oct 26, 2026