Skip to content

ci: run fork-PR unit/lint/type checks offline from a prewarmed cache (#831)#845

Merged
vikrantpuppala merged 1 commit into
mainfrom
fork/831-warm-deps-cache
Jul 8, 2026
Merged

ci: run fork-PR unit/lint/type checks offline from a prewarmed cache (#831)#845
vikrantpuppala merged 1 commit into
mainfrom
fork/831-warm-deps-cache

Conversation

@vikrantpuppala

Copy link
Copy Markdown
Contributor

Summary

Fork PRs currently show all required checks red regardless of the diff (#831, reported against the trivial fork PR #671). Root cause: every job in code-quality-checks.yml installs deps via setup-poetrysetup-jfrog, which mints a JFrog PyPI token from a GitHub OIDC token. GitHub deliberately withholds the OIDC token (ACTIONS_ID_TOKEN_REQUEST_TOKEN) from fork pull_request runs, so setup-jfrog's set -u script dies with unbound variable before any test runs. Public PyPI is unreachable from the protected runners, so a fork run can obtain no package source at all.

This adopts the Databricks SOP for this exact problem (mirrors databricks/dbt-databricks): invert the flow so a trusted workflow — which does get OIDC — pre-builds the environment and caches it, and the fork run replays it offline.

  • .github/workflows/warm-deps-cache.yml (new) — the sole writer of the forkvenv-* cache. Builds the in-project .venv for the full python × depset × extras matrix from JFrog and saves each leg. Triggers: push:main (dep files), a daily schedule (beats GitHub's 7-day cache eviction), and workflow_dispatch with an optional pr_number that warms a fork's changed lockfiles (fetches only poetry.lock/pyproject.toml — no source, no workflows). Primes mypy stub packages too.
  • .github/actions/restore-deps (new) — restores the newest warmed .venv for the leg via a restore-keys prefix and reports cache-hit.
  • .github/workflows/code-quality-checks.yml — each job tries restore-deps first and, on a hit, runs the tools directly from .venv/bin (not poetry run, because even bootstrapping Poetry would hit JFrog). On a miss — every same-repo PR, or a fork whose lockfiles changed before a maintainer re-warmed — it falls back to the original setup-poetry JFrog flow, unchanged.

Why this is safe

The fork run holds no credentials and needs none — it only reads a GitHub Actions cache and runs offline. Everything still flows through JFrog (the warmer populates the cache from it). No pull_request_target, no secret exposure to fork code.

Scope

Unit + lint + type checks (the fork-reachable required checks). e2e / kernel-e2e / integration / DBR-LTS stay merge-queue-gated — they need live warehouse secrets and are out of scope for #831.

Fork dependency-change flow

If a fork PR changes poetry.lock/pyproject.toml, its cache key won't match and CI misses the cache. A maintainer reviews the dep change, runs Actions → Warm Dependency Cache → Run workflow → pr_number = N, then the contributor re-runs CI.

Test plan

  • actionlint clean on both new/changed workflows (only pre-existing custom-runner-label warnings remain, shared by all workflows).
  • Verified every poetry run on the fork path is guarded by cache-hit != 'true'; .venv is relocatable / interpreter restored via actions/setup-python on the hit path.
  • Live (needs maintainer): run the warmer once on main to seed the cache, then open a fork PR and confirm (a) the .venv restores, (b) poetry install is skipped, (c) pytest / black / mypy genuinely run and report pass/fail, (d) no JFrog call on the fork job.

Follow-ups for a maintainer

  • Seed the cache by running the warmer on main before the first fork PR relies on it.
  • Confirm branch-protection required-check names still match the job names here (unchanged in this PR).

Closes #831

This pull request and its description were written by Isaac.

…831)

Fork pull_request runs never receive a GitHub OIDC token, so
.github/actions/setup-jfrog (which mints a JFrog PyPI token from that OIDC
token) dies with "ACTIONS_ID_TOKEN_REQUEST_TOKEN: unbound variable" and every
required check goes red on every fork PR — regardless of the diff (#831,
reported against the trivial fork PR #671). Public PyPI is unreachable from the
protected runners, so a fork run can obtain no package source at all.

Adopt the Databricks SOP (mirrors databricks/dbt-databricks): a trusted warmer
workflow that DOES get OIDC pre-builds the in-project .venv for every matrix
leg and saves it to the Actions cache; fork PRs restore that .venv and run the
tools directly from .venv/bin, fully offline, never touching JFrog. Everything
still flows through JFrog (the warmer populates the cache from it); the fork
just holds no credentials, so there is no pull_request_target / secret-exposure
risk.

- .github/workflows/warm-deps-cache.yml: new, sole writer of the forkvenv-*
  cache. Triggers on push:main (dep files), a daily schedule (beats GitHub's
  7-day cache eviction), and workflow_dispatch with an optional pr_number that
  warms a fork's changed lockfiles (fetches ONLY poetry.lock/pyproject.toml, no
  source). Warms the full py x depset x extras matrix, priming mypy stubs too.
- .github/actions/restore-deps: new composite action; restores the newest
  forkvenv-* entry for the leg via a restore-keys prefix and reports cache-hit.
- .github/workflows/code-quality-checks.yml: each job tries restore-deps first
  and runs offline from .venv/bin on a hit; on a miss (all same-repo PRs, or a
  fork whose lockfiles changed pre-warm) it falls back to the original
  setup-poetry JFrog flow, unchanged. Same-repo PR behavior is unaffected.

Scope: unit + lint + type checks (the fork-reachable required checks). e2e /
kernel-e2e / integration / DBR-LTS stay merge-queue-gated (they need live
warehouse secrets).

Closes #831

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
@vikrantpuppala vikrantpuppala added this pull request to the merge queue Jul 8, 2026
Merged via the queue into main with commit a238dc9 Jul 8, 2026
53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Various tests things fail on PR

2 participants