Migrate dependency management from pip to uv

[oto-macenauer-absa]

The task

Migrate the project's Python dependency and environment management from pip + requirements*.txt to uv.

Motivation

• Reproducible builds via a committed uv.lock (today nothing pins transitive dependencies — only direct ones are pinned in requirements*.txt).
• Single source of truth for metadata + dependencies in pyproject.toml (currently it holds only tool config: black, coverage, mypy).
• Much faster resolve/install in CI and Docker builds.
• Managed interpreter provisioning, so local setup and CI agree on Python 3.13.

Current state (what touches pip today)

File	Usage
requirements.txt	9 pinned production deps (confluent-kafka, psycopg2, boto3, ...)
requirements-dev.txt	-r requirements.txt + 11 test/lint deps, incl. psycopg2-binary
pyproject.toml	No [project] table at all — tool config only
Dockerfile:65	pip install -r requirements.txt --no-binary confluent-kafka
.github/actions/setup-dev-python-env/action.yml	setup-python with cache: 'pip' + pip install -r requirements-dev.txt
.github/workflows/check_python.yml:42,44	Change detection filters on requirements*.txt
.github/dependabot.yml:22	package-ecosystem: "pip"
Makefile:3	PYTHON := .venv/bin/python
DEVELOPER.md:31-36	python3 -m venv + pip3 install -r requirements-dev.txt
.github/copilot-instructions.md:56	Documents the two-requirements-files layout

Scope of work

1. pyproject.toml

• Add a [project] table: name, version, requires-python = ">=3.13,<3.14", and dependencies migrated from requirements.txt (keep the exact == pins on the first pass so the migration is a no-op dependency-wise).
• Add [dependency-groups] dev with the contents of requirements-dev.txt.
• Decide how to handle the psycopg2 (prod) vs psycopg2-binary (dev) split — both are currently installed in a dev env. Either keep both (they are distinct distributions and resolve fine) or move to a single one plus a build-time flag. Document whichever is chosen.
• Add [tool.uv] config if needed (e.g. no-binary-package = ["confluent-kafka"] for the Docker path).

2. Lockfile

• Generate and commit uv.lock.
• Add a .python-version file pinning 3.13.

3. Dockerfile — the highest-risk part of this migration

This Dockerfile is consumed by an external deployment pipeline, not by this repo's CI. That pipeline clones EventGate, supplies the config file, the trusted certificates (TRUSTED_SSL_CERTS) and the SASL/SSL artifacts (SASL_SSL_ARTIFACTS), then runs docker/build-push-action on top. Consequences:

• The build environment is not a plain GitHub-hosted runner with open egress. Whatever assumptions uv makes about network, TLS trust and package indexes have to hold there, and that environment is not visible from this repo.
• A break lands in the deployment pipeline, in another repo, after merge here — not on the PR. See item 4b for coordination.

Treat this section as the critical path.

Blockers to resolve before writing the Dockerfile change

These are places where uv does not behave like pip, and each is invisible until the deployment pipeline runs.

(a) TLS trust — uv ignores the system CA bundle by default. The image's first build step exists precisely to append the pipeline-supplied certs to /etc/pki/tls/certs/ca-bundle.crt (typical of an inspecting corporate proxy). uv ships its own TLS stack and does not read the system trust store unless told to. Any uv network operation must run with UV_NATIVE_TLS=1 (or --native-tls), otherwise dependency resolution fails with certificate errors in the deployment pipeline while working fine on an unproxied laptop. Set it as an ENV before the install step.

(b) Package index — uv does not read pip.conf / PIP_INDEX_URL. If the pipeline points pip at an internal mirror (Artifactory/Nexus) via pip.conf, PIP_INDEX_URL or PIP_EXTRA_INDEX_URL, uv will silently ignore all of it and go to public PyPI — which may be blocked, or worse, may not be the intended source. uv needs UV_DEFAULT_INDEX / UV_INDEX env vars or a [[tool.uv.index]] entry in pyproject.toml. Confirm with the pipeline owners whether an internal index is in play before implementing.

(c) Getting the uv binary into the image. COPY --from=ghcr.io/astral-sh/uv:... needs a pull from ghcr.io at build time. If the deployment environment only permits a mirrored registry, that fails. Fallbacks, in order of preference: mirror the uv image internally; or pip install uv==<pin> from the already-configured index as the first step (keeps a single network source, costs one pip invocation); or vendor the binary. Pick based on (b)'s answer.

Two further constraints must survive the migration:

1. --no-binary confluent-kafka is load-bearing. It forces a source build against the system librdkafka compiled earlier in the image, which is what provides Kerberos/GSSAPI support. The PyPI wheel is built without GSSAPI. The uv equivalent is --no-binary-package confluent-kafka.
2. Deps currently land in the base image's system site-packages (plain pip install, no --target), which is already on the Lambda import path. Keep that placement — uv pip install --system is the direct equivalent. Do not switch to --target "${LAMBDA_TASK_ROOT}" as part of this change; that alters the import path for no benefit and widens the blast radius.

Proposed shape:

# uv binary — pinned by digest, matching the repo's action-pinning convention.
# Base image is arm64; the ghcr.io/astral-sh/uv image is multi-arch.
COPY --from=ghcr.io/astral-sh/uv:<version>@sha256:<digest> /uv /bin/uv

# Dependency manifests only, copied before src so this layer stays cached
# until dependencies actually change (same caching property as today's
# `COPY requirements.txt`).
COPY pyproject.toml uv.lock ${LAMBDA_TASK_ROOT}/

and, replacing the pip install line inside the existing RUN:

    cd "${LAMBDA_TASK_ROOT}" && \
    uv export --frozen --no-dev --no-emit-project \
      --format requirements-txt -o /tmp/requirements.lock.txt && \
    uv pip install --system --no-binary-package confluent-kafka \
      -r /tmp/requirements.lock.txt && \

Why export-then-install rather than uv sync: uv sync builds a .venv, which is not on the Lambda import path and would need PYTHONPATH fixing. Why export rather than uv pip install -r pyproject.toml: the latter re-resolves at build time and ignores uv.lock, throwing away the reproducibility this migration is for. --frozen makes the build fail loudly if uv.lock is stale relative to pyproject.toml — a desirable build gate.

Additional Docker checks:

• Only the prod dependency group must be installed — --no-dev above. No pytest/moto/testcontainers in the shipped image.
• --no-emit-project assumes src/ is COPY'd rather than installed as a package (as today). If [project] ends up declaring packages, set [tool.uv] package = false instead.
• uv export emits hashes by default; confirm hash verification still passes for the confluent-kafka sdist given --no-binary-package.
• Remove /bin/uv in the existing cleanup step so the build tool is not part of the shipped image (attack surface + AquaSec/Trivy findings).
• Re-check the non-root USER 1000 block — chown -R must still cover everything uv writes.
• Confirm the image still passes the AquaSec night scan.

4. CI

• .github/actions/setup-dev-python-env/action.yml: swap actions/setup-python + pip install for astral-sh/setup-uv (SHA-pinned, enable-cache: true) + uv sync --group dev.
• check_python.yml: change detection currently keys on requirements*.txt — must key on pyproject.toml and uv.lock instead, otherwise dependency-only PRs stop triggering the quality gates.
• Decide whether tool invocations become uv run pylint ... / uv run pytest ... or whether the synced venv is activated once. Note uv sync creates .venv in the workspace, so the existing bare pylint/mypy/pytest calls will break unless the venv is put on PATH.

4b. Coordinate with the external deployment pipeline

The pipeline that builds and pushes this image lives in another repo and runs docker/build-push-action against this Dockerfile after injecting config, certs and SASL/SSL artifacts. It is the only thing that exercises the changed install path, and it does so after merge here.

• Confirm with the pipeline owners: internal package index in use? (blocker b) Registry egress restrictions? (blocker c) Any step that reads requirements.txt directly — pre-fetching wheels, vulnerability scanning, an SBOM step, or an offline/air-gapped install stage? If so, deleting requirements*.txt (item 8) breaks it, and the cutover must be sequenced with a pipeline change rather than merged independently.
• Agree a verification run: build the migrated image through the real pipeline, against the real certs and index, before this issue is closed. A local docker build on an unproxied machine does not exercise blockers (a) or (b) at all.
• Note that docker/build-push-action layer caching keys off the COPY set; swapping requirements.txt for pyproject.toml + uv.lock invalidates the dependency layer once, which is expected (librdkafka recompiles).

Optionally, add a build-only smoke job to this repo's CI (path-filtered on Dockerfile / pyproject.toml / uv.lock, --platform=linux/arm64, generous timeout-minutes since librdkafka compiles from source). It would catch plain Dockerfile syntax/step regressions on the PR, but it will not reproduce the proxy or index conditions of the deployment environment, so it is a complement to the pipeline verification, not a substitute.

5. Makefile

• Replace PYTHON := .venv/bin/python with uv run so it works on Windows too (.venv/bin/python does not exist there — it is .venv/Scripts/python.exe).

6. Dependabot

• .github/dependabot.yml: package-ecosystem: "pip" → "uv". Verify the groups/allow: direct config carries over and that dependabot_auto.yml still matches the resulting PRs.

7. Docs

• DEVELOPER.md "Set Up Python Environment" → uv sync --group dev.
• .github/copilot-instructions.md dependency-layout paragraph.
• README.md if it references requirements files.

8. Cleanup

• Delete requirements.txt / requirements-dev.txt once nothing references them — including the external deployment pipeline (see 4b). If a consumer outside this repo needs them, generate on demand with uv export --format requirements-txt rather than keeping them hand-maintained, and sequence the deletion after the consumer is updated.

Open questions

• Should the == pins in pyproject.toml be relaxed to >=/~= now that uv.lock provides exact reproducibility? Relaxing means Dependabot updates the lock rather than churning the manifest. Recommend a follow-up, not part of this migration.
• Is uv.lock covered by the AquaSec/Trivy scanning currently in use, or does the scan need uv export output to see the dependency tree?

Acceptance criteria

• uv sync --group dev produces a working dev environment from a clean clone.
• make qa passes locally.
• All jobs in check_python.yml pass, and are still triggered by a dependency-only change.
• Docker image builds for linux/arm64 and confluent-kafka is compiled from source against the system librdkafka — verify GSSAPI is actually present in the built image, not just that the build succeeded.
• Shipped image contains no dev dependencies and no uv binary.
• The image has been built through the real deployment pipeline, with the real certs and package index — not only locally.
• The deployment pipeline has been confirmed to have no remaining dependency on requirements*.txt.
• Lambda still cold-starts and serves both handlers (src.event_gate_lambda.lambda_handler, event stats) from the rebuilt image.
• uv.lock is committed and Dependabot opens PRs against it.
• No references to requirements*.txt remain in the repo.