Skip to content

ci: stop a TestPyPI outage from costing the release its wheels - #498

Merged
JarryShaw merged 1 commit into
mainfrom
ci/testpypi-must-not-gate-release
Sep 19, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
ci/testpypi-must-not-gate-release

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

1.5.0b2 reached PyPI with all 8 files, but its GitHub Release is missing three wheels — cp312, pp310 and pp311. The upload was never the problem; the step ordering was.

What actually happened

The pypi job runs three steps in this order: Publish to PyPI, then Publish package to TestPyPI, then Upload package to GitHub Release. On the Python 3.12 leg of run 35412937138, the per-step outcomes were:

[Publish to PyPI]             outcome=success
[Publish package to TestPyPI] outcome=failure

with:

TimeoutError: _ssl.c:1015: The handshake operation timed out
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='test.pypi.org', port=443):
    Read timed out. (read timeout=5)

PyPI had already accepted the upload. A transient timeout against a secondary index then failed the job, and because Upload package to GitHub Release comes after it, that step never ran — the log contains zero lines for it. The 3.12 wheel went to PyPI and was never attached to the release.

The matrix has no fail-fast: false, so the same failure cancelled the pypy3.10 and pypy3.11 legs, which lost their wheels the same way.

Measured asset counts:

release assets wheels present
v1.5.0a1 (clean) 23 cp310, cp311, cp312, cp313, cp314, pp310, pp311
v1.5.0b2 10 cp310, cp311, cp313, cp314 — cp312, pp310, pp311 missing

(v1.5.0a1 also carries three conda build numbers per Python where b1/b2 carry one; that difference is unrelated to this failure and is not attributed to it. The attributable loss is exactly the three wheels.)

The fix

  • continue-on-error: true on the TestPyPI step. It is a secondary target. Its availability must not gate the real release, and it certainly must not prevent the primary artefacts being attached.
  • fail-fast: false on the pypi matrix. Each leg has its own wheel to attach; a cancelled leg never attaches it. One leg's bad luck should not cost three wheels.

A hypothesis this refutes, recorded because it was mine

My first reading was a duplicate-upload race — every leg building the same universal wheel, the first winning and the losers rejected as already existing. That was wrong, and the fix it implied was already in place: skip-existing: true is set on both publish steps (create-release.yml:234 and :242). The failing step was TestPyPI, not PyPI, and the cause was a network timeout. The log settled it; the wheel-size coincidence that suggested the race (every wheel is 1,279,537 bytes, because they are the same universal wheel) was real but explained nothing about the failure.

Verification

Re-parsed the workflow and asserted both changes landed where intended, with the neighbouring steps untouched:

pypi.strategy.fail-fast: False
pypi matrix legs: 7
step 'Publish to PyPI':                continue-on-error=False
step 'Publish package to TestPyPI':    continue-on-error=True
step 'Upload package to GitHub Release': continue-on-error=False
step order tail: ['Publish to PyPI', 'Publish package to TestPyPI', 'Upload package to GitHub Release']

What this does not do

It does not backfill the three wheels already absent from the v1.5.0b2 release, and it does not touch that release or its tag — those are published artefacts and the maintainer's call. PyPI is unaffected either way: pip install --pre pypcapkit gets the complete set, since all 8 files are there.

It also changes nothing about how the release is triggered or gated.

The 1.5.0b2 release reached PyPI with all 8 files, but its GitHub Release
is missing three wheels -- cp312, pp310 and pp311 -- and the cause is
ordering, not the upload.

In the pypi job, `Publish package to TestPyPI` sits between `Publish to
PyPI` and `Upload package to GitHub Release`. On the Python 3.12 leg the
step log reads:

  [Publish to PyPI]            outcome=success
  [Publish package to TestPyPI] outcome=failure
  urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(
      host='test.pypi.org', port=443): Read timed out. (read timeout=5)

PyPI had already accepted the upload. A transient timeout against a
secondary index then failed the job, so the GitHub Release step never ran
and that leg's wheel was never attached. With no `fail-fast: false` on the
matrix, the failure also cancelled the pypy3.10 and pypy3.11 legs, which
lost their wheels the same way.

Two changes:

- `continue-on-error: true` on the TestPyPI step. It is a secondary target;
  its availability must not gate the real release.
- `fail-fast: false` on the pypi matrix. Each leg has its own wheel to
  attach, and a cancelled leg never attaches it.

Note `skip-existing: true` was already set on both publish steps, so this
was never a duplicate-upload conflict -- that was my first hypothesis and
the log refuted it.

Verified the file still parses and both changes landed where intended:
pypi.strategy.fail-fast=False, 7 matrix legs, TestPyPI
continue-on-error=True, PyPI and GitHub Release steps untouched at
continue-on-error=False, step order unchanged.

Does not backfill the three wheels already missing from the v1.5.0b2
release; that is a published artefact and the maintainer's call.
@JarryShaw
JarryShaw merged commit 39116e0 into main Sep 19, 2026
24 checks passed
@JarryShaw
JarryShaw deleted the ci/testpypi-must-not-gate-release branch September 19, 2026 02:17
@JarryShaw JarryShaw added the ci Pull requests that change CI or workflow configuration (ci: subject prefix) label Sep 22, 2026
@JarryShaw JarryShaw added this to the 1.5 milestone Oct 6, 2026
@JarryShaw JarryShaw moved this to Done in PyPCAPKit Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Pull requests that change CI or workflow configuration (ci: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant