Skip to content

Move the remaining native Linux CI jobs off EC2 #22108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
huydhn wants to merge 1 commit into
base: osdc/linux-job-v3
Choose a base branch
from
Draft

Move the remaining native Linux CI jobs off EC2 #22108

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion .github/workflows/_test_backend.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,13 @@ jobs:
package-golden-artifacts:
if: ${{ inputs.run-linux }}
needs: test-backend-linux
runs-on: linux.2xlarge
# Repackaging artifacts needs no ExecuTorch toolchain, so this runs on a
# GitHub-hosted runner rather than moving to OSDC; the S3 upload that used
# to ride on the EC2 instance role assumes role/arc through OIDC instead.
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- name: Download model test artifacts
uses: actions/download-artifact@v4
Expand Down Expand Up @@ -127,6 +133,13 @@ jobs:
path: golden_artifacts_*.zip
if-no-files-found: ignore

- name: Configure AWS credentials
if: ${{ hashFiles('golden_artifacts_*.zip') != '' }}
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::308535385114:role/arc
aws-region: us-east-1

- name: Upload golden artifacts to S3
uses: seemethere/upload-artifact-s3@v5
if: ${{ hashFiles('golden_artifacts_*.zip') != '' }}
Expand Down
77 changes: 35 additions & 42 deletions .github/workflows/_xtensa_build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
# Reusable: cross-compile cadence_executor_runner for one Cadence Xtensa core.
#
# A native job (not linux_job_v2) because the GitHub OIDC token must be minted on
# the runner host: the ACTIONS_ID_TOKEN_REQUEST_* vars do not cross into
# linux_job_v2's docker exec. So the role is assumed on the host, then the build
# runs inside the CI image via docker run with the creds passed in. Binding the
# A native job rather than linux_job_v3, because the build has to assume the
# Cadence artifacts role rather than the role/arc that linux_job_v3 assumes for
# itself, and only a native job can run configure-aws-credentials. Binding the
# environment also gives the OIDC token the environment claim. The licensed
# toolchain + core configs are fetched at runtime from an auth-gated store;
# role/region/store come from CI variables and are not committed.
Expand All @@ -23,68 +22,62 @@ on:
default: ""

jobs:
# The runner pod pulls the container before any step runs, so the image has to
# be a fully qualified reference resolved by a job this one depends on.
docker-image:
name: Resolve CI docker image
uses: ./.github/workflows/_docker-image.yml

build:
name: ${{ inputs.backend }}
runs-on: linux.2xlarge
needs: docker-image
runs-on: mt-l-x86iavx512-8-64
container:
image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/executorch/ci-image:executorch-ubuntu-22.04-clang12-${{ needs.docker-image.outputs.ci-docker-hash }}
environment: cadence
permissions:
id-token: write
contents: read
steps:
- name: Clean workspace
shell: bash
run: |
set -eux
rm -rf "${GITHUB_WORKSPACE}"
mkdir -p "${GITHUB_WORKSPACE}"

- name: Checkout executorch
uses: actions/checkout@v4
with:
submodules: recursive
ref: ${{ inputs.ref }}

- name: Calculate docker image
id: calculate-docker-image
uses: pytorch/test-infra/.github/actions/calculate-docker-image@main
with:
docker-image-name: ci-image:executorch-ubuntu-22.04-clang12

- name: Pull docker image
run: docker pull "${{ steps.calculate-docker-image.outputs.docker-image }}"

- name: Assume Cadence artifacts role (host OIDC)
- name: Assume Cadence artifacts role
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ vars.CADENCE_CI_AWS_ROLE }}
aws-region: ${{ vars.CADENCE_CI_AWS_REGION }}

- name: Cross-compile cadence_executor_runner
env:
DOCKER_IMAGE: ${{ steps.calculate-docker-image.outputs.docker-image }}
BACKEND: ${{ inputs.backend }}
XTENSA_S3_BUCKET: ${{ vars.CADENCE_CI_S3_BUCKET }}
shell: bash
run: |
set -eux
# OIDC/role assumption already happened on the host above; pass the
# resulting AWS creds and the store/backend into the CI image, where
# the toolchain download + cross-compile run.
docker run --rm \
-e BACKEND -e XTENSA_S3_BUCKET \
-e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \
-e AWS_DEFAULT_REGION -e AWS_REGION \
-v "${GITHUB_WORKSPACE}:/work/executorch" -w /work/executorch \
"${DOCKER_IMAGE}" \
bash -c '
set -exo pipefail
eval "$(/opt/conda/bin/conda shell.bash hook)"
conda activate "$(conda env list --json | jq -r ".envs | .[-1]")"
./install_requirements.sh > /dev/null
pip install --quiet awscli
# hifi4/fusion_g3 optimized kernels need the foss-xtensa nnlib
# sources, which are not vendored in executorch; the cadence
# installer clones them. vision has no nnlib dependency.
if [ "${BACKEND}" != "vision" ]; then
backends/cadence/install_requirements.sh
fi
source .ci/scripts/setup-xtensa-tools.sh "${BACKEND}"
.ci/scripts/build-cadence-xtensa.sh --no-run
chmod -R a+rX cmake-out
'
set -exo pipefail
eval "$(/opt/conda/bin/conda shell.bash hook)"
conda activate "$(conda env list --json | jq -r ".envs | .[-1]")"
./install_requirements.sh > /dev/null
pip install --quiet awscli
# hifi4/fusion_g3 optimized kernels need the foss-xtensa nnlib
# sources, which are not vendored in executorch; the cadence
# installer clones them. vision has no nnlib dependency.
if [ "${BACKEND}" != "vision" ]; then
backends/cadence/install_requirements.sh
fi
source .ci/scripts/setup-xtensa-tools.sh "${BACKEND}"
.ci/scripts/build-cadence-xtensa.sh --no-run
chmod -R a+rX cmake-out

- name: Upload runner
uses: actions/upload-artifact@v4
Expand Down
70 changes: 32 additions & 38 deletions .github/workflows/_xtensa_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Reusable: build + run the Cadence Xtensa op-level gtest tests for one core on
# the Instruction Set Simulator (xt-run). Mirrors _xtensa_build.yml's native
# OIDC + docker-run skeleton (running xt-run needs the same licensed toolchain),
# OIDC skeleton (running xt-run needs the same licensed toolchain),
# then builds the gtest op-test ELF and runs it on the simulator. The runner
# cross-compile and these op tests are separate build configs (the tests need
# exceptions/RTTI that the runner build disables), so this is a self-contained
Expand All @@ -21,64 +21,58 @@ on:
default: ""

jobs:
# The runner pod pulls the container before any step runs, so the image has to
# be a fully qualified reference resolved by a job this one depends on.
docker-image:
name: Resolve CI docker image
uses: ./.github/workflows/_docker-image.yml

test:
name: ${{ inputs.backend }}
runs-on: linux.2xlarge
needs: docker-image
runs-on: mt-l-x86iavx512-8-64
container:
image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/executorch/ci-image:executorch-ubuntu-22.04-clang12-${{ needs.docker-image.outputs.ci-docker-hash }}
environment: cadence
permissions:
id-token: write
contents: read
steps:
- name: Clean workspace
shell: bash
run: |
set -eux
rm -rf "${GITHUB_WORKSPACE}"
mkdir -p "${GITHUB_WORKSPACE}"

- name: Checkout executorch
uses: actions/checkout@v4
with:
submodules: recursive
ref: ${{ inputs.ref }}

- name: Calculate docker image
id: calculate-docker-image
uses: pytorch/test-infra/.github/actions/calculate-docker-image@main
with:
docker-image-name: ci-image:executorch-ubuntu-22.04-clang12

- name: Pull docker image
run: docker pull "${{ steps.calculate-docker-image.outputs.docker-image }}"

- name: Assume Cadence artifacts role (host OIDC)
- name: Assume Cadence artifacts role
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ vars.CADENCE_CI_AWS_ROLE }}
aws-region: ${{ vars.CADENCE_CI_AWS_REGION }}

- name: Build and run op tests on xt-run
env:
DOCKER_IMAGE: ${{ steps.calculate-docker-image.outputs.docker-image }}
BACKEND: ${{ inputs.backend }}
XTENSA_S3_BUCKET: ${{ vars.CADENCE_CI_S3_BUCKET }}
shell: bash
run: |
set -eux
# OIDC/role assumption already happened on the host above; pass the
# resulting AWS creds and the store/backend into the CI image, where
# the toolchain download + op-test build + xt-run happen.
docker run --rm \
-e BACKEND -e XTENSA_S3_BUCKET \
-e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \
-e AWS_DEFAULT_REGION -e AWS_REGION \
-v "${GITHUB_WORKSPACE}:/work/executorch" -w /work/executorch \
"${DOCKER_IMAGE}" \
bash -c '
set -exo pipefail
eval "$(/opt/conda/bin/conda shell.bash hook)"
conda activate "$(conda env list --json | jq -r ".envs | .[-1]")"
./install_requirements.sh > /dev/null
pip install --quiet awscli
# hifi4/fusion_g3 optimized kernels need the foss-xtensa nnlib
# sources, which are not vendored in executorch; the cadence
# installer clones them. vision has no nnlib dependency.
if [ "${BACKEND}" != "vision" ]; then
backends/cadence/install_requirements.sh
fi
source .ci/scripts/setup-xtensa-tools.sh "${BACKEND}"
.ci/scripts/test-cadence-xtensa.sh
'
set -exo pipefail
eval "$(/opt/conda/bin/conda shell.bash hook)"
conda activate "$(conda env list --json | jq -r ".envs | .[-1]")"
./install_requirements.sh > /dev/null
pip install --quiet awscli
# hifi4/fusion_g3 optimized kernels need the foss-xtensa nnlib
# sources, which are not vendored in executorch; the cadence
# installer clones them. vision has no nnlib dependency.
if [ "${BACKEND}" != "vision" ]; then
backends/cadence/install_requirements.sh
fi
source .ci/scripts/setup-xtensa-tools.sh "${BACKEND}"
.ci/scripts/test-cadence-xtensa.sh
18 changes: 16 additions & 2 deletions .github/workflows/apple.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,13 @@ jobs:
# Upload the test demo app to S3
upload-demo-ios:
needs: build-demo-ios
runs-on: linux.2xlarge
# Moving an artifact from GitHub to S3 needs no ExecuTorch toolchain, so
# this runs on a GitHub-hosted runner rather than moving to OSDC; the upload
# that used to ride on the EC2 instance role assumes role/arc through OIDC.
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- name: Download the artifacts from GitHub
uses: actions/download-artifact@v4
Expand All @@ -138,6 +144,12 @@ jobs:
set -eux
ls -lah ./

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::308535385114:role/arc
aws-region: us-east-1

- name: Upload the artifacts to S3
uses: seemethere/upload-artifact-s3@v5
with:
Expand All @@ -159,7 +171,9 @@ jobs:
uses: pytorch/test-infra/.github/workflows/mobile_job.yml@main
with:
device-type: ios
# For iOS testing, the runner just needs to call AWS Device Farm, so there is no need to run this on macOS
# For iOS testing, the runner just needs to call AWS Device Farm, so there is no need to run this on macOS.
# Still an EC2 label: mobile_job.yml resolves its actions by relative path
# (./test-infra/.github/actions/...), which does not work on OSDC.
runner: linux.2xlarge
test-infra-ref: main
# This is the ARN of ExecuTorch project on AWS
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/stale.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ on:
jobs:
stale:
if: ${{ github.repository == 'pytorch/executorch' }}
runs-on: linux.large
# Pure GitHub API work, so it needs neither an EC2 nor an OSDC runner.
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
Expand Down
Loading