Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,9 @@ updates:
interval: monthly
cooldown:
default-days: 7
- package-ecosystem: "github-actions"
directory: "/actions/python-lambda-zip"
schedule:
interval: monthly
cooldown:
default-days: 7
16 changes: 14 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,21 @@ avoid breaking things. If you feel lucky you might use `v0` branch.
# actions
Number of helpful actions for GitHub Actions

* [build-push-ecr](./actions/build-push-ecr/README.md)
* [build-push-ecr](./actions/build-push-ecr/README.md):
builds and pushes an image to AWS ECR using buildx

* [download-private-asset](./actions/download-private-asset/README.md)
* [download-private-asset](./actions/download-private-asset/README.md):
installs a release asset from a private GitHub repository

* [python-lambda-zip](./actions/python-lambda-zip/README.md):
builds a zip deployment package for a Python AWS Lambda using runtime-compatible
wheels, and checks that it loads in the AWS Lambda base image

* [terraform](./actions/terraform/README.md):
Terraform format linting, required-tag checks and tfsec security scanning

* [upgrade-erlang-elixir](./actions/upgrade-erlang-elixir/README.md):
resolves target Erlang/Elixir versions, updates them in CI files and opens a pull request

# reusable workflows
Workflows that a caller repo runs with `uses` at job level. Intentionally sit at
Expand Down
78 changes: 78 additions & 0 deletions actions/python-lambda-zip/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Python Lambda zip

Builds a zip-file deployment package for a Python AWS Lambda and checks that it actually loads in
the AWS Lambda runtime.

Dependencies with compiled extensions (`cryptography`, `numpy`, ...) ship platform-specific wheels.
A plain `pip install` on a GitHub runner picks wheels built for the runner's (new) glibc, which the
Lambda runtime may be too old to load, giving errors like
``version `GLIBC_2.28' not found`` only at import time in AWS. This action

1. installs the dependencies with `pip --platform ... --only-binary=:all:`, so that only wheels
compatible with the Lambda runtime are used, and
2. imports your modules inside the matching `public.ecr.aws/lambda/python` image, so that any
remaining problem fails the build instead of the deployment.

The action does not upload the zip anywhere; use the `zip-path` output (or `output-dir`) for that.

## Usage

```yaml
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: '3.11'
- run: pip install pipenv
- uses: gameanalytics/github/actions/python-lambda-zip@v0
with:
python-version: '3.11'
source-dir: app
requirements-command: pipenv requirements
smoke-test-glob: 'lambda_*.py'
```

Run it in your pull request workflow too, not only on releases, to find problems before release.
The runner needs `pip`, `zip`, `unzip` and Docker, which `ubuntu-latest` has.

## Inputs

| Input | Default | Description |
|---|---|---|
| `python-version` | (required) | Python version of the Lambda runtime, e.g. `3.11` |
| `architecture` | (required) | `x86_64` or `arm64` (arm64 is smoke tested using QEMU emulation) |
| `source-dir` | | Directory with the Lambda code; its contents end up at the root of the zip |
| `include` | | Newline-separated paths copied into the zip keeping their relative path, e.g. `app` |
| `exclude` | | Newline-separated patterns of files to leave out of the zip, as for `zip -x`, e.g. `boto3/*` |
| `requirements-file` | `requirements.txt` | Fully resolved requirements file |
| `requirements-command` | | Command printing the requirements, e.g. `pipenv requirements`; used instead of the file when set |
| `smoke-test-glob` | `*.py` | Modules (relative to the zip root) to import; build fails if nothing matches |
| `output-dir` | `build` | Where to put the zip |
| `zip-name` | `package.zip` | File name of the zip |

At least one of `source-dir` and `include` is required. `__pycache__` directories and `.pyc` files are left out of the zip.

Output: `zip-path`, the absolute path of the zip.

## How `python-version` maps to wheels and image

| Python version | Lambda OS | glibc | Wheels installed | Smoke test image |
|---|---|---|---|---|
| 3.9 - 3.11 | Amazon Linux 2 | 2.26 | `manylinux2014` | `public.ecr.aws/lambda/python:<version>` |
| 3.12 and later | Amazon Linux 2023 | 2.34 | `manylinux_2_28` | `public.ecr.aws/lambda/python:<version>` |

The dependencies are installed with `--no-deps`, so the requirements must already be fully
resolved, as with a lock file. Only pure Python and manylinux wheels are accepted: a package that
has only a source distribution is built from source if it turns out to be pure Python, but if it
is compiled, or only has newer manylinux wheels, the build fails instead of building something
that breaks at runtime.

pip evaluates environment markers in the requirements (such as `python_version`) against the host
Python, so the host Python must be the same version as `python-version` (the build fails
otherwise): set it up with `actions/setup-python` first. Building `arm64` on an `x86_64` runner
works for the wheels, but platform markers are then evaluated for the runner, so a warning is issued.

When a new Lambda runtime moves to a newer OS, update the mapping in `build.sh`.
80 changes: 80 additions & 0 deletions actions/python-lambda-zip/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
name: Python Lambda zip
description: >
Builds a zip-file deployment package for a Python AWS Lambda, using wheels compatible with the
Lambda runtime, and checks that it can be imported inside the AWS Lambda base image.

inputs:
python-version:
description: >
Python version of the Lambda runtime, e.g. '3.11'. Determines the Lambda base image used for
the smoke test and which manylinux wheels are installed.
required: true
architecture:
description: "Lambda architecture: 'x86_64' or 'arm64'"
required: true
source-dir:
description: 'Directory with the Lambda code. Its contents end up at the root of the zip.'
required: false
include:
description: >
Newline-separated paths (files or directories) to copy into the zip keeping their relative
path, e.g. 'app' to get an 'app' package at the root of the zip. Can be combined with source-dir.
At least one of source-dir and include is required.
required: false
exclude:
description: >
Newline-separated patterns of files to leave out of the zip, as for 'zip -x', matched against
paths in the zip, e.g. 'boto3/*' for dependencies the Lambda runtime provides. The smoke test
runs in the Lambda image, where such packages are available.
required: false
requirements-file:
description: 'Fully resolved requirements file, e.g. generated from a lock file.'
required: false
default: 'requirements.txt'
requirements-command:
description: >
Shell command that prints the requirements to stdout, e.g. 'pipenv requirements'. Used
instead of requirements-file when set. The tool it needs must already be installed.
required: false
smoke-test-glob:
description: >
Glob, relative to the root of the zip, of the modules to import in the smoke test, e.g.
'*.py' or 'app/*.py' (module names starting with an underscore are skipped). The build
fails if nothing matches.
required: false
default: '*.py'
output-dir:
description: 'Directory to put the zip in'
required: false
default: 'build'
zip-name:
description: 'File name of the zip'
required: false
default: 'package.zip'

outputs:
zip-path:
description: 'Absolute path of the built zip'
value: ${{ steps.build.outputs.zip-path }}

runs:
using: composite
steps:
- name: Set up QEMU
if: inputs.architecture == 'arm64'
uses: docker/setup-qemu-action@v4.4.0
- name: Build zip and smoke test it
id: build
shell: bash
env:
PYTHON_VERSION: ${{ inputs.python-version }}
ARCHITECTURE: ${{ inputs.architecture }}
SOURCE_DIR: ${{ inputs.source-dir }}
INCLUDE: ${{ inputs.include }}
EXCLUDE: ${{ inputs.exclude }}
REQUIREMENTS_FILE: ${{ inputs.requirements-file }}
REQUIREMENTS_COMMAND: ${{ inputs.requirements-command }}
SMOKE_TEST_GLOB: ${{ inputs.smoke-test-glob }}
OUTPUT_DIR: ${{ inputs.output-dir }}
ZIP_NAME: ${{ inputs.zip-name }}
run: ${{ github.action_path }}/build.sh
109 changes: 109 additions & 0 deletions actions/python-lambda-zip/build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
#!/usr/bin/env bash
# Builds a Lambda deployment zip for the given Python version and architecture,
# then imports it inside the matching AWS Lambda base image as a smoke test.
# Configured through environment variables, see action.yml.
set -euo pipefail

: "${PYTHON_VERSION:?}" "${ARCHITECTURE:?}" "${OUTPUT_DIR:?}" "${ZIP_NAME:?}"
: "${SMOKE_TEST_GLOB:?}"

if [[ -z "${SOURCE_DIR:-}" && -z "${INCLUDE:-}" ]]; then
echo "at least one of source-dir and include must be set" >&2
exit 1
fi

if [[ ! "$PYTHON_VERSION" =~ ^3\.[0-9]+$ ]]; then
echo "python-version must look like '3.11', got '$PYTHON_VERSION'" >&2
exit 1
fi
minor="${PYTHON_VERSION#3.}"

# Lambda runtimes up to python3.11 run on Amazon Linux 2 (glibc 2.26), so only
# manylinux2014 (glibc 2.17) wheels can be loaded. python3.12 and later run on
# Amazon Linux 2023 (glibc 2.34), which can load manylinux_2_28 wheels.
if (( minor <= 11 )); then
manylinux=manylinux2014
else
manylinux=manylinux_2_28
fi

case "$ARCHITECTURE" in
x86_64) docker_platform=linux/amd64 ;;
arm64) docker_platform=linux/arm64 ;;
*) echo "architecture must be x86_64 or arm64, got '$ARCHITECTURE'" >&2; exit 1 ;;
esac
arch=$([[ $ARCHITECTURE == arm64 ]] && echo aarch64 || echo x86_64)
# Also accept manylinux2014 wheels, which pip does not do implicitly for newer tags
wheel_platforms=(--platform "manylinux2014_${arch}")
[[ $manylinux == manylinux2014 ]] || wheel_platforms+=(--platform "${manylinux}_${arch}")
image="public.ecr.aws/lambda/python:${PYTHON_VERSION}"

# pip evaluates environment markers in the requirements (python_version etc.) against the
# host Python, so a mismatch would silently drop or add dependencies
host_version="$(python3 -c 'import sys; print("%d.%d" % sys.version_info[:2])')"
if [[ "$host_version" != "$PYTHON_VERSION" ]]; then
echo "Host Python is $host_version but python-version is $PYTHON_VERSION: set up the same version first, e.g. with actions/setup-python" >&2
exit 1
fi
if [[ "$ARCHITECTURE" != "$(uname -m)" ]]; then
echo "::warning::Building for $ARCHITECTURE on a $(uname -m) host: platform markers in the requirements are evaluated for the host"
fi

stage="$(mktemp -d)"
trap 'rm -rf "$stage"' EXIT
mkdir -p "$stage/package" "$OUTPUT_DIR"
zip_path="$(cd "$OUTPUT_DIR" && pwd)/$ZIP_NAME"
rm -f "$zip_path"

if [[ -n "${REQUIREMENTS_COMMAND:-}" ]]; then
bash -c "$REQUIREMENTS_COMMAND" > "$stage/requirements.txt"
else
cp "$REQUIREMENTS_FILE" "$stage/requirements.txt"
fi

# --no-deps because the requirements are expected to be fully resolved (a lock file). It also
# allows pip to fall back to building a source distribution when a package has no wheel
# for the target; the check below makes sure that this only happens for pure Python packages.
echo "::group::Install dependencies for $manylinux $arch, Python $PYTHON_VERSION"
python3 -m pip install -r "$stage/requirements.txt" --no-deps -t "$stage/package" \
"${wheel_platforms[@]}" --python-version "$PYTHON_VERSION" \
--implementation cp
echo "::endgroup::"

bad=0
for wheel_file in "$stage"/package/*.dist-info/WHEEL; do
if grep '^Tag: ' "$wheel_file" | grep -qv -e '-any$' -e 'manylinux'; then
echo "::error::$(dirname "$wheel_file" | xargs basename) is not a pure Python or manylinux package (built from source for the host?): $(grep '^Tag: ' "$wheel_file" | tr '\n' ' ')"
bad=1
fi
done
(( bad == 0 ))

if [[ -n "${SOURCE_DIR:-}" ]]; then
cp -r "$SOURCE_DIR"/. "$stage/package/"
fi
while IFS= read -r path; do
[[ -n "$path" ]] && cp -r --parents "$path" "$stage/package/"
done <<< "${INCLUDE:-}"
zip_excludes=(-x "*__pycache__*" -x "*.pyc")
while IFS= read -r pattern; do
[[ -n "$pattern" ]] && zip_excludes+=(-x "$pattern")
done <<< "${EXCLUDE:-}"
(cd "$stage/package" && zip -qr "$zip_path" . "${zip_excludes[@]}")
echo "Built $zip_path"

mkdir -p "$stage/smoke"
unzip -q "$zip_path" -d "$stage/smoke"
echo "::group::Smoke test in $image ($docker_platform)"
docker run --rm --platform "$docker_platform" --entrypoint python \
-v "$stage/smoke:/var/task:ro" -w /var/task -e GLOB="$SMOKE_TEST_GLOB" "$image" \
-c "import glob, importlib, os
files = glob.glob(os.environ['GLOB'], recursive=True)
mods = sorted(f[:-3].replace('/', '.') for f in files if not os.path.basename(f).startswith('_'))
assert mods, 'no modules matched ' + os.environ['GLOB']
for m in mods:
importlib.import_module(m)
print('imported', m)"
echo "::endgroup::"

echo "zip-path=$zip_path" >> "${GITHUB_OUTPUT:-/dev/null}"
Loading