Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,16 @@ updates:
- "*"
labels:
- "github-actions"
Comment thread
larsoner marked this conversation as resolved.
- "chore"

# Maintain the pinned dependencies of the Python tools
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "monthly"
groups:
python:
patterns:
- "*"
labels:
- "chore"
62 changes: 62 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,32 @@ concurrency:
cancel-in-progress: true

jobs:
test-stale-wheels:
name: "test stale wheel check"
# No secrets needed, so this one can also run on forks
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Install the latest version of uv
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
python-version: '3.14'

- name: Run the tests
run: uv run --with-requirements requirements.txt tests/test_check_stale_wheels.py

test:
name: "test upload via action"
runs-on: ubuntu-latest
if: github.repository == 'scientific-python/upload-nightly-action'
# Pull requests from forks and from Dependabot are not given secrets, so there
# they check that the action refuses to run rather than really uploading
env:
HAS_TOKEN: ${{ secrets.UPLOAD_TOKEN != '' }}
outputs:
uploaded: ${{ steps.record.outputs.uploaded }}

steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand Down Expand Up @@ -47,49 +69,80 @@ jobs:
- name: List contents of wheel
run: python -m zipfile --list dist/test_package-*.whl

- name: Test that an upload without a token is refused
id: no_token
if: env.HAS_TOKEN != 'true'
continue-on-error: true
uses: ./_action_path/
with:
artifacts_path: dist
anaconda_nightly_upload_token: ''

- name: Check that the upload was refused
if: env.HAS_TOKEN != 'true'
env:
OUTCOME: ${{ steps.no_token.outcome }}
run: |
if [ "${OUTCOME}" != "failure" ]; then
echo "::error::the action did not refuse to upload without a token"
exit 1
fi

- name: Test upload
if: env.HAS_TOKEN == 'true'
uses: ./_action_path/
with:
artifacts_path: dist
anaconda_nightly_upload_token: ${{ secrets.UPLOAD_TOKEN }}

- name: Test upload that forces removal first
if: env.HAS_TOKEN == 'true'
uses: ./_action_path/
with:
artifacts_path: dist
anaconda_nightly_upload_token: ${{ secrets.UPLOAD_TOKEN }}

- name: Build v0.0.2 wheel and sdist
if: env.HAS_TOKEN == 'true'
run: |
# Bump version to avoid wheel name conflicts
sed -i 's/0.0.1/0.0.2/g' _action_path/tests/test_package/pyproject.toml
rm ./dist/*
uv tool run --from build pyproject-build --installer=uv --outdir ./dist _action_path/tests/test_package

- name: Test upload with non-main label
if: env.HAS_TOKEN == 'true'
uses: ./_action_path/
with:
artifacts_path: dist
anaconda_nightly_upload_token: ${{ secrets.UPLOAD_TOKEN }}
anaconda_nightly_upload_labels: test

- name: Build v0.0.3 wheel and sdist
if: env.HAS_TOKEN == 'true'
run: |
# Bump version to avoid wheel name conflicts
sed -i 's/0.0.2/0.0.3/g' _action_path/tests/test_package/pyproject.toml
rm ./dist/*
uv tool run --from build pyproject-build --installer=uv --outdir ./dist _action_path/tests/test_package

- name: Test upload with multiple labels
if: env.HAS_TOKEN == 'true'
uses: ./_action_path/
with:
artifacts_path: dist
anaconda_nightly_upload_token: ${{ secrets.UPLOAD_TOKEN }}
anaconda_nightly_upload_labels: dev,test

- name: Record whether anything was uploaded
id: record
run: echo "uploaded=${HAS_TOKEN}" | tee -a "${GITHUB_OUTPUT}"

cleanup:
runs-on: ubuntu-latest
needs: [test]
# Nothing to clean up when the uploads were skipped for lack of a token
if: needs.test.outputs.uploaded == 'true'
# Set required workflow secrets in the environment for additional security
# https://github.com/scientific-python/upload-nightly-action/settings/environments
environment:
Expand All @@ -107,3 +160,12 @@ jobs:
anaconda --token ${{ secrets.ANACONDA_TOKEN }} remove \
--force \
"scientific-python-nightly-wheels/test-package"

report-failure:
needs: [test-stale-wheels, test, cleanup]
if: failure() && github.event_name == 'schedule'
permissions:
issues: write
uses: ./.github/workflows/report-failure.yml
with:
title: 'The scheduled CI run is failing'
9 changes: 9 additions & 0 deletions .github/workflows/keep-alive.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,12 @@ jobs:
user_name: 'github-actions[bot]'
user_email: 'github-actions[bot]@users.noreply.github.com'
commit_message: Keep repo GitHub Actions alive

report-failure:
needs: [deploy]
if: failure() && github.event_name == 'schedule'
permissions:
issues: write
uses: ./.github/workflows/report-failure.yml
with:
title: 'The workflow keep-alive is failing'
9 changes: 9 additions & 0 deletions .github/workflows/remove-wheels.yml
Original file line number Diff line number Diff line change
Expand Up @@ -109,3 +109,12 @@ jobs:

done
fi

report-failure:
needs: [remove]
if: failure() && github.event_name == 'schedule'
permissions:
issues: write
uses: ./.github/workflows/report-failure.yml
with:
title: 'The old wheel cleanup is failing'
46 changes: 46 additions & 0 deletions .github/workflows/report-failure.yml

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will be a nice short action to reuse elsewhere, too. So ultimately, I think it would make sense to actually document/mention all of the actions, suitable for direct downstream usage, in the README.

Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
name: Report a failing workflow

# Called by the scheduled workflows in this repository when they fail. A cron job
# that quietly stops working can go unnoticed for months, which is the very thing
# this repository nags other projects about.

on:
workflow_call:
inputs:
title:
description: 'Title of the issue to open, and to reuse while it stays open'
required: true
type: string

jobs:
report:

runs-on: ubuntu-latest
permissions:
issues: write

steps:
- name: Open or update an issue
env:
GH_TOKEN: ${{ github.token }}
TITLE: ${{ inputs.title }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
body="\`${GITHUB_WORKFLOW}\` failed, most recently in [this run](${RUN_URL}).

Until it is fixed, whatever that workflow keeps an eye on is unattended."

# Only ever match our own issues, so the default page of 30 cannot hide the
# one we are looking for. Without --search this is the REST creator filter,
# which wants the literal login: "app/github-actions" matches nothing here.
number=$(gh issue list --repo "${GITHUB_REPOSITORY}" --state open \
--author "github-actions[bot]" --json number,title \
--jq 'map(select(.title == env.TITLE)) | .[0].number // ""')

# Rewrite rather than comment, so a workflow that fails every day does not
# collect a comment a day
if [ -n "${number}" ]; then
gh issue edit "${number}" --repo "${GITHUB_REPOSITORY}" --body "${body}"
else
gh issue create --repo "${GITHUB_REPOSITORY}" --title "${TITLE}" --body "${body}"
fi
49 changes: 49 additions & 0 deletions .github/workflows/stale-wheels.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
name: Report stale wheels

on:
# Run daily at 0:23 UTC, an hour before "Remove old wheels"
schedule:
- cron: '23 0 * * *'
workflow_dispatch:
inputs:
dry_run:
description: 'Report findings without opening or closing any issues'
type: boolean
default: true

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
report:

runs-on: ubuntu-latest
if: github.repository_owner == 'scientific-python'
# Needed to open an issue here if we cannot reach a project's own tracker
permissions:
issues: write

steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Install the latest version of uv
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
python-version: '3.14'

- name: Check for stale wheels
env:
ISSUE_OPENER_TOKEN: ${{ secrets.ISSUE_OPENER_TOKEN }}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have this token yet?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep I added it!

GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
uv run --with-requirements requirements.txt tools/check_stale_wheels.py ${{ inputs.dry_run && '--dry-run' || '' }}

report-failure:
needs: [report]
if: failure() && github.event_name == 'schedule'
permissions:
issues: write
uses: ./.github/workflows/report-failure.yml
with:
title: 'The stale nightly wheel check is failing'
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# pixi environments
.pixi
*.egg-info
__pycache__
.pytest_cache
61 changes: 61 additions & 0 deletions AGENTS.md

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't reviewed or even read this file.

Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Notes for agents

Guidance for automated contributors working in this repository.
Written for someone who can read the code but has no way to know the conventions or the reasoning behind them.

## What is here

`action.yml` and `upload_wheels.sh` are the published action itself, which projects call from their own workflows to upload nightly wheels.
Everything else exists to look after the [`scientific-python-nightly-wheels`](https://anaconda.org/scientific-python-nightly-wheels) channel those uploads land on.

`.github/workflows/remove-wheels.yml` deletes wheels beyond the retention policy, daily at 01:23 UTC.
`tools/check_stale_wheels.py` warns projects before that deletion strands them, daily at 00:23 UTC.
The hour between the two is deliberate: a package must be flagged before it can be deleted, and a fully deleted package disappears from the channel listing where we would no longer see it at all.

`.github/workflows/report-failure.yml` is a reusable workflow every scheduled workflow calls when it fails.
This repository nags other projects about silently broken automation, so its own cron jobs must not fail quietly.
It is called as a job rather than used as a composite action because `ci.yml` checks out to `_action_path` and `remove-wheels.yml` does not check out at all.

## Conventions

Pin third-party actions to a full commit SHA with a `# vX.Y.Z` comment; Dependabot updates them monthly as a single group.

Run the Python tools with `uv run --with-requirements requirements.txt <script>`, which is how the workflows invoke them; the scripts carry no PEP 723 header, so that their dependencies are declared in exactly one place.
That file pins exact versions rather than a `uv.lock`, because Dependabot has no `uv` ecosystem and an unmaintainable lock would only rot, while it does understand `requirements.txt` and proposes updates monthly.
`scientific-python/issue-from-pytest-log-action` is worth reading for the rest of the house style, though it locks its scripts individually where this repository does not.

Pin versions exactly, as `pixi.toml` and `requirements.txt` both do.
Do not add a `[tool.uv] exclude-newer` window to a script whose dependencies Dependabot pins: uv then refuses to resolve any pin newer than the cutoff, leaving every Dependabot pull request unresolvable until the release ages past it.
The window in `pixi.toml` is fine, because those versions are bumped by hand.

Format Python with `ruff format --line-length 100`; there is no ruff configuration in the repository yet, so pass the length explicitly.

## Things that will catch you out

Pull requests from forks *and* from Dependabot receive no secrets, so a job keyed on `github.event.pull_request.head.repo.fork` still breaks for Dependabot.
Gate on the secret instead, as `ci.yml` does with `HAS_TOKEN: ${{ secrets.UPLOAD_TOKEN != '' }}`, and have the tokenless path assert the action refuses to upload.

A called workflow's token is capped by the calling job's, so a job that calls `report-failure.yml` must itself declare `permissions: issues: write`.

The anaconda.org API's `latest_version` is not the most recently uploaded version, because nightly version strings do not sort consistently across projects.
Take the maximum `upload_time` across every version instead.

That API carries no repository URL for these packages — `home`, `dev_url`, and `source_git_url` are all null — so `check_stale_wheels.py` resolves the target repository from PyPI `Project-URL` metadata, with `PYPI_MAP` covering the one package that has none.

PyGithub serializes every request through an internal connection lock, so threading does not speed up the GitHub half of a run.
A full stale wheel check takes about two minutes, which is fine for a daily job.

## Working on the stale wheel check

Run it with `ISSUE_OPENER_TOKEN` and `GITHUB_TOKEN` set, and keep `--dry-run` on unless you intend to open issues in other projects' trackers.
Dry runs still authenticate and still read issues; they only skip writes.

The thresholds are constants at the top of the script rather than command line options, by request: add an option only when something actually needs to vary.
`RETENTION_DAYS` must stay in step with the 30 days in `remove-wheels.yml` and the policy section of `README.md`.

Tests live in `tests/test_check_stale_wheels.py` and run with `uv run --frozen tests/test_check_stale_wheels.py`.
They stub the network, so they are fast and safe to run anywhere.
Every case in them is a real package whose metadata would break a naive implementation; add to that table rather than replacing it when the resolution logic changes.

Issues opened in other projects are posted by [@scientific-python-bot](https://github.com/scientific-python-bot) using a classic token with the `public_repo` scope, stored as the `ISSUE_OPENER_TOKEN` secret.
Be conservative with anything that posts outward: those messages arrive unsolicited in other maintainers' inboxes.
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
See [AGENTS.md](AGENTS.md).
47 changes: 47 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,53 @@ In addition cron jobs in GitHub repositories will be disabled after a certain am
We are not aware of a solution other than regular commit activity to prevent the deactivation of
scheduled cron jobs.

## Stale wheel reminders

So that the removal above does not come as a surprise, a daily job in this repository checks how
long ago each package was last uploaded. If it has been more than **15 days**, the job opens an
issue on that project's own issue tracker naming the date its wheels will be removed, comments once
more at **25 days**, and closes the issue automatically once a new wheel is uploaded. Issues are opened
by [@scientific-python-bot][bot]; the repository to report to is taken from the project's PyPI
metadata (`Project-URL`), so keeping those URLs pointed at your GitHub repository is enough to be
reachable.

[bot]: https://github.com/scientific-python-bot

The check runs daily from `tools/check_stale_wheels.py` in this repository, an hour before the
cleanup job that does the deleting. If your project's PyPI metadata carries no GitHub URL we can
follow, we have no way to reach you: the run records that in an issue here instead, and the fix is
either to add a `Project-URL` upstream or to ask us to map the package by hand. Maintainers of this
repository: the workflow posts with `ISSUE_OPENER_TOKEN`, a token belonging to the bot account with
the `public_repo` scope, and falls back to the workflow's own `GITHUB_TOKEN` when reporting here.

## Reporting a failing scheduled workflow

A cron job that quietly stops working can go unnoticed for months, which is the same problem the
reminders above exist to solve. The scheduled workflows here each call `report-failure.yml`, a small
reusable workflow that opens one issue when a run fails and rewrites that issue on later failures
rather than adding a comment a day.

Other projects are welcome to call it too:

```yml
jobs:
nightly:
...

report-failure:
needs: [nightly]
if: failure() && github.event_name == 'schedule'
permissions:
issues: write
uses: scientific-python/upload-nightly-action/.github/workflows/report-failure.yml@main
with:
title: 'The nightly wheel build is failing'
```

The issue is opened in the calling repository by `github-actions[bot]`, using that workflow's own
`GITHUB_TOKEN`, so no secret is needed. The `title` doubles as the identity of the issue, so keep it
stable. As with the action itself, we recommend pinning to a specific SHA rather than to `main`.


# Using nightly builds in CI

Expand Down
Loading
Loading