Repository navigation
ci: stop a TestPyPI outage from costing the release its wheels - #498
Merged
Merged
Conversation
The 1.5.0b2 release reached PyPI with all 8 files, but its GitHub Release
is missing three wheels -- cp312, pp310 and pp311 -- and the cause is
ordering, not the upload.
In the pypi job, `Publish package to TestPyPI` sits between `Publish to
PyPI` and `Upload package to GitHub Release`. On the Python 3.12 leg the
step log reads:
[Publish to PyPI] outcome=success
[Publish package to TestPyPI] outcome=failure
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(
host='test.pypi.org', port=443): Read timed out. (read timeout=5)
PyPI had already accepted the upload. A transient timeout against a
secondary index then failed the job, so the GitHub Release step never ran
and that leg's wheel was never attached. With no `fail-fast: false` on the
matrix, the failure also cancelled the pypy3.10 and pypy3.11 legs, which
lost their wheels the same way.
Two changes:
- `continue-on-error: true` on the TestPyPI step. It is a secondary target;
its availability must not gate the real release.
- `fail-fast: false` on the pypi matrix. Each leg has its own wheel to
attach, and a cancelled leg never attaches it.
Note `skip-existing: true` was already set on both publish steps, so this
was never a duplicate-upload conflict -- that was my first hypothesis and
the log refuted it.
Verified the file still parses and both changes landed where intended:
pypi.strategy.fail-fast=False, 7 matrix legs, TestPyPI
continue-on-error=True, PyPI and GitHub Release steps untouched at
continue-on-error=False, step order unchanged.
Does not backfill the three wheels already missing from the v1.5.0b2
release; that is a published artefact and the maintainer's call.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1.5.0b2reached PyPI with all 8 files, but its GitHub Release is missing three wheels —cp312,pp310andpp311. The upload was never the problem; the step ordering was.What actually happened
The
pypijob runs three steps in this order:Publish to PyPI, thenPublish package to TestPyPI, thenUpload package to GitHub Release. On the Python 3.12 leg of run35412937138, the per-step outcomes were:with:
PyPI had already accepted the upload. A transient timeout against a secondary index then failed the job, and because
Upload package to GitHub Releasecomes after it, that step never ran — the log contains zero lines for it. The 3.12 wheel went to PyPI and was never attached to the release.The matrix has no
fail-fast: false, so the same failure cancelled thepypy3.10andpypy3.11legs, which lost their wheels the same way.Measured asset counts:
v1.5.0a1(clean)v1.5.0b2(
v1.5.0a1also carries three conda build numbers per Python whereb1/b2carry one; that difference is unrelated to this failure and is not attributed to it. The attributable loss is exactly the three wheels.)The fix
continue-on-error: trueon the TestPyPI step. It is a secondary target. Its availability must not gate the real release, and it certainly must not prevent the primary artefacts being attached.fail-fast: falseon thepypimatrix. Each leg has its own wheel to attach; a cancelled leg never attaches it. One leg's bad luck should not cost three wheels.A hypothesis this refutes, recorded because it was mine
My first reading was a duplicate-upload race — every leg building the same universal wheel, the first winning and the losers rejected as already existing. That was wrong, and the fix it implied was already in place:
skip-existing: trueis set on both publish steps (create-release.yml:234and:242). The failing step was TestPyPI, not PyPI, and the cause was a network timeout. The log settled it; the wheel-size coincidence that suggested the race (every wheel is 1,279,537 bytes, because they are the same universal wheel) was real but explained nothing about the failure.Verification
Re-parsed the workflow and asserted both changes landed where intended, with the neighbouring steps untouched:
What this does not do
It does not backfill the three wheels already absent from the
v1.5.0b2release, and it does not touch that release or its tag — those are published artefacts and the maintainer's call. PyPI is unaffected either way:pip install --pre pypcapkitgets the complete set, since all 8 files are there.It also changes nothing about how the release is triggered or gated.