Skip to content

Move docker-builds to OSDC runners - #22106

Draft
huydhn wants to merge 5 commits into
mainfrom
osdc/docker-builds
Draft

Move docker-builds to OSDC runners#22106
huydhn wants to merge 5 commits into
mainfrom
osdc/docker-builds

Conversation

@huydhn

@huydhn huydhn commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Part 1 of moving ExecuTorch's Linux CI to OSDC. #22107 migrates linux_job_v2 -> v3, #22108 the native jobs.

OSDC pods have no docker daemon, so the builds move to the in-cluster BuildKit pool: build.sh switches to docker buildx build --push under REMOTE_BUILDKIT, driven by test-infra's docker-build-remote-buildkit action. The tag is computed in the workflow now, but it is the same string calculate-docker-image produced, so jobs still on v2 keep resolving.

Also adds the ciflow/docker trigger, registers it in pytorch-probot.yml and auto-labels the paths that need a rebuild: a fork PR gets no OIDC token, so a pull_request-triggered build cannot push on OSDC. pull_request stays for now.

Authored with Claude Code.

OSDC runner pods have no docker daemon, so the ci-image builds move to the
in-cluster BuildKit pool. build.sh switches to `docker buildx build --push`
when REMOTE_BUILDKIT is set and is otherwise unchanged, and the workflow drives
it through test-infra's docker-build-remote-buildkit action, which registers the
per-architecture remote builder and rides out a cold builder pool.

The tag is now computed in the workflow rather than by calculate-docker-image,
but it is the same string that action produced for `ci-image:<name>` consumers
(executorch/ci-image:<name>-$(git rev-parse HEAD:.ci/docker)), so jobs still on
linux_job_v2 keep resolving their images while the v3 migration lands
separately.

The orchestrator runner only streams the .ci/docker context to BuildKit and
waits, so it drops to the smallest label of the matching architecture, and
executorch-ubuntu-22.04-cuda-windows moves off the GPU runner it never used.

Authored with Claude Code.
@pytorch-bot

pytorch-bot Bot commented Aug 24, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22106

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 167 Pending

As of commit 5723f87 with merge base 9a2d135 (image):

NEW FAILURE - The following job has failed:

  • Build Presets / apple (macos) / build (gh)
    ##[error]fatal: unable to access 'https://github.com/Maratyszcza/FP16.git/': Failed to connect to github.com port 443 after 231 ms: Couldn't connect to server

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 24, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@huydhn

huydhn commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

Correcting the arm64 label. mt-l-arm64g2-6-25 does not schedule — the three arm64 legs of the first run sat queued from 19:09 to 22:56 and were cancelled without a runner ever picking them up.

The label naming is contradictory across sources: arc.yaml maps linux.arm64.2xlarge to l-arm64g2-6-32, ci-infra renamed that def to l-arm64g2-6-25 in pytorch/ci-infra#625, and ci-infra's smoke test still references c-mt-l-arm64g2-6-32 — but c-mt- is the staging prefix, so that says nothing about prod. The only recent runs on any mt-l-arm64g2-* label I could find anywhere are test-infra's build-clang/build-gcc from March.

So both PRs now use mt-l-arm64g4-16-62, which is what pytorch/pytorch's own OSDC docker-builds uses and which picks up in seconds in prod today. #22107 also folds in the one job that was on mt-l-arm64g3-16-62 — same size, equally unattested — so all arm64 work lands on one proven label.

huydhn added 3 commits August 24, 2026 18:24
A fork PR gets no OIDC token, so on OSDC a pull_request-triggered build cannot
assume role/arc to push. Applying the ciflow/docker label tags the PR head and
fires the push trigger with the PR's own SHA, which does work. pull_request
stays for now and comes out once that path is proven.

Authored with Claude Code.
Applying the label only pushes a tag if the tag is listed here, so the trigger
added in the previous commit is inert without it.

Authored with Claude Code.
Removing the pull_request trigger otherwise made rebuilding the images a thing
contributors have to remember. The labeler applies ciflow/docker on the same
paths docker-builds triggers on, which pytorchbot turns into the tag push.

Authored with Claude Code.
arc.yaml maps linux.arm64.2xlarge to l-arm64g2-6-32 and ci-infra renamed that
def to l-arm64g2-6-25, but neither name schedules: the three arm64 legs of the
first run of this workflow sat queued for nearly four hours and were cancelled
without a runner ever picking them up. mt-l-arm64g4-16-62 is the only arm64
label with live production evidence, and it is what pytorch/pytorch's own OSDC
docker-builds uses.

Authored with Claude Code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant