ci: regenerate sample captures through util/regenerate_samples.py (#1967)

.github/workflows/unit-tests.yml

@@ -508,33 +508,14 @@ jobs:
         continue-on-error: ${{ matrix.ktlo == true }}
         run: python -m pip install -e '.[test,Scapy,DPKT,crypto,cli,NGAP,vendor,docs,PyShark]'
 
-      # `shell: bash` rather than the default, because it runs with -o pipefail:
-      # without it the `tee` below would swallow the generator's exit status.
+      # util/regenerate_samples.py annotates a failure as a fixture failure, not
+      # a test one, and warns when the PCAP-NG captures are stand-ins.
       - name: Regenerate sample captures
         id: samples
         if: ${{ steps.install.outcome == 'success' }}
         continue-on-error: ${{ matrix.ktlo == true }}
         shell: bash
-        run: |
-          # Keep a failure here legible as what it is -- the fixtures could not
-          # be built, so no test has run yet and nothing is wrong with the code
-          # under test.
-          if ! python examples/generators/make_samples.py 2>&1 | tee "$RUNNER_TEMP/make-samples.log"; then
-            echo "::${ANNOTATION_LEVEL} title=Sample fixture generation failed::examples/generators/make_samples.py could not rebuild examples/captures/. This is a fixture-generation failure, not a test failure -- the test suite has not run."
-            exit 1
-          fi
-
-          # examples/generators/pcapng.py fetches captures from the Wireshark
-          # repository and synthesises stand-ins when the download fails. The
-          # stand-ins parse, and tests/protocols/test_pcapng_regression.py only
-          # asserts that extraction succeeds, so a degraded run goes green and
-          # is otherwise indistinguishable from a clean one. Say which happened.
-          if grep -q '(download unavailable)' "$RUNNER_TEMP/make-samples.log"; then
-            echo "::warning title=Sample fixtures degraded::The upstream Wireshark captures were unreachable, so synthesised stand-ins were used. The PCAP-NG tier ran, but not against the upstream bytes."
-            grep '(download unavailable)' "$RUNNER_TEMP/make-samples.log"
-          else
-            echo "Upstream Wireshark captures were used, matching their pinned SHA-256 digests."
-          fi
+        run: python util/regenerate_samples.py
 
       # See the `test` job above for why this step exists.
       - name: Report available parallelism
@@ -1323,25 +1304,13 @@ jobs:
           python -m pip install -U pip setuptools wheel
           python -m pip install -e '.[test,Scapy,DPKT,cli,PyShark,PyPCAP,PyPCAPFile]'
 
-      # See the `integration` job above for why this step is shaped the way it
-      # is.
+      # See the `integration` job above.
       - name: Regenerate sample captures
         id: samples
         if: ${{ steps.install.outcome == 'success' }}
         continue-on-error: ${{ matrix.ktlo == true }}
         shell: bash
-        run: |
-          if ! python examples/generators/make_samples.py 2>&1 | tee "$RUNNER_TEMP/make-samples.log"; then
-            echo "::${ANNOTATION_LEVEL} title=Sample fixture generation failed::examples/generators/make_samples.py could not rebuild examples/captures/. This is a fixture-generation failure, not a test failure -- the test suite has not run."
-            exit 1
-          fi
-
-          if grep -q '(download unavailable)' "$RUNNER_TEMP/make-samples.log"; then
-            echo "::warning title=Sample fixtures degraded::The upstream Wireshark captures were unreachable, so synthesised stand-ins were used. The PCAP-NG tier ran, but not against the upstream bytes."
-            grep '(download unavailable)' "$RUNNER_TEMP/make-samples.log"
-          else
-            echo "Upstream Wireshark captures were used, matching their pinned SHA-256 digests."
-          fi
+        run: python util/regenerate_samples.py
 
       # See the `test` job above for why this step exists.
       - name: Report available parallelism
@@ -1835,21 +1804,13 @@ jobs:
           python -m pip install -U pip setuptools wheel
           python -m pip install -e '.[test,Scapy,DPKT,crypto,cli,NGAP,vendor,docs,PyShark]'
 
-      # See the integration job above for why this step is shaped the way it is.
+      # See the `integration` job above. This job has no KTLO leg, so a
+      # failure is always an error.
       - name: Regenerate sample captures
         shell: bash
-        run: |
-          if ! python examples/generators/make_samples.py 2>&1 | tee "$RUNNER_TEMP/make-samples.log"; then
-            echo "::error title=Sample fixture generation failed::examples/generators/make_samples.py could not rebuild examples/captures/. This is a fixture-generation failure, not a test failure -- the test suite has not run."
-            exit 1
-          fi
-
-          if grep -q '(download unavailable)' "$RUNNER_TEMP/make-samples.log"; then
-            echo "::warning title=Sample fixtures degraded::The upstream Wireshark captures were unreachable, so synthesised stand-ins were used. The PCAP-NG tier ran, but not against the upstream bytes."
-            grep '(download unavailable)' "$RUNNER_TEMP/make-samples.log"
-          else
-            echo "Upstream Wireshark captures were used, matching their pinned SHA-256 digests."
-          fi
+        env:
+          ANNOTATION_LEVEL: error
+        run: python util/regenerate_samples.py
 
       # See the `test` job above for why this step exists.
       - name: Report available parallelism

docs/source/contributing/releasing.rst

@@ -52,7 +52,7 @@ Editing ``__version__`` by hand without also moving :file:`CITATION.cff`'s
 asserts the two agree, and ``create-release.yml``'s ``unit-tests`` job calls
 ``unit-tests.yml`` with
 ``gate-only: true``, which runs the **full** suite rather than the tiered
-subset an ordinary push runs (its ``gate`` job, ``unit-tests.yml:1798-1873``).
+subset an ordinary push runs (its ``gate`` job).
 That gate runs whenever a release will -- a moved ``__version__`` is exactly
 what makes ``version_check``'s evidence read "not yet published" -- and every
 publishing job's ``needs:`` requires it to have succeeded, so a
@@ -62,8 +62,8 @@ Either run the script, or move both fields by hand in the same commit.
 This is the **only** edit a person makes to get a release started -- the tag,
 the Release, and every upload are the workflow's job from here. Other
 checks run unconditionally on the same path, though none is a step in *this*
-process: the ``changelog`` job (``unit-tests.yml:1641-1667``) fails outright if
-``CHANGELOG.md`` has drifted from its source entry under
+process: the ``changelog`` job's "Check CHANGELOG.md is in step with its source
+entry" step fails outright if ``CHANGELOG.md`` has drifted from its source entry under
 :file:`docs/source/changelog/`, and ``github``'s "Select release body" step
 warns, without failing, if that entry's heading still reads "unreleased". Each
 enforces what the changelog convention

docs/source/contributing/workflows.rst

@@ -375,8 +375,8 @@ the same commit several times over. It also skips ``changes``,
 `Project Pins`_ below), and ``required-checks`` (see `Required Status
 Checks`_ below), which are gated out by their own ``if:`` rather than by having
 already run. ``changelog`` carries no
-``if:`` at all and runs on every path regardless -- deliberately, per its own
-comment (``unit-tests.yml:1629-1635``): ``create-release.yml`` feeds
+``if:`` at all and runs on every path regardless -- deliberately, per the comment
+above the job: ``create-release.yml`` feeds
 ``CHANGELOG.md`` to the GitHub Release body, so the release path is exactly
 where a drifted file must not go unchecked. Confirmed on run `36210743295
 <https://github.com/JarryShaw/PyPCAPKit/actions/runs/36210743295>`__ (a
@@ -688,7 +688,7 @@ grepping every workflow file for its name:
      - Where
    * - ``Required checks passed``
      - job ``required-checks``
-     - ``unit-tests.yml:1964``
+     - ``unit-tests.yml``
    * - ``Compat Python 3.11``
      - job ``compatibility``, matrix leg ``3.11``
      - ``python-compatibility.yml``, ``.github/python-versions.toml:20``
@@ -709,23 +709,22 @@ That is the list after `#1633 <https://github.com/JarryShaw/PyPCAPKit/issues/163
 which added ``Compat Python 3.15`` and removed ``Compat Python 3.10``. Moving a version
 between the tiers below means editing the ruleset in the same change.
 
-``Required checks passed`` is defined by exactly one job
-(``unit-tests.yml:1964``). ``Compat Python`` is defined by one, the
-``name:`` of the ``compatibility`` job in ``python-compatibility.yml``, whose
+``Required checks passed`` is defined by exactly one, the ``name:`` of the
+``required-checks`` job in ``unit-tests.yml``. ``Compat Python`` is defined by
+one, the ``name:`` of the ``compatibility`` job in ``python-compatibility.yml``, whose
 ``matrix`` comes from :file:`.github/python-versions.toml`: the required legs
 are its supported tier (``python-versions.toml:20-24``), and its KTLO legs
 (``python-versions.toml:16-19``) have names ending in ``(KTLO)``, so no required
-context matches one. A comment at
-``unit-tests.yml:1876`` mentions the string without defining it.
+context matches one. The comment above ``required-checks`` mentions the string
+without defining it.
 
 ``Required checks passed`` is an aggregate, not a single check run. The ruleset
 requires it and the ``Compat Python 3.11``-``3.15`` legs, nothing else, as
-``unit-tests.yml``'s own comment (``unit-tests.yml:1875-1885``) records. It
-``needs:`` the gating jobs of ``unit-tests.yml`` -- ``test``, ``integration``,
-``engine-tests`` and ``pypcap-parity``, plus ``project-tests``, the one run of
-the project pins, and ``changes`` for the docs-only path -- and checks each
-result explicitly
-(``unit-tests.yml:1963-2019``). Rulesets match check names literally, with no
+that comment records. It ``needs:`` the gating jobs of ``unit-tests.yml`` --
+``test``, ``integration``, ``engine-tests`` and ``pypcap-parity``, plus
+``project-tests``, the one run of the project pins, and ``changes`` for the
+docs-only path -- and checks each result explicitly, in its "Check every
+required job succeeded" step. Rulesets match check names literally, with no
 wildcard, so requiring the matrix cells one by one would mean editing the
 ruleset whenever a matrix changes; the aggregate keeps the required list fixed.
 It accepts the gating jobs as ``skipped`` only on a docs-only pull request, where
@@ -734,8 +733,8 @@ It accepts the gating jobs as ``skipped`` only on a docs-only pull request, wher
 ``python-compatibility.yml``, and a job cannot ``needs:`` a job in another
 workflow file. This job runs on Unit Tests' own ``push``/``pull_request``
 triggers; the ``gate-only: true`` reusable calls documented above skip it via
-``if: ${{ always() && inputs.gate-only != true }}`` (``unit-tests.yml:1966``),
-so it is never produced -- and never expected -- on those paths.
+its ``if: ${{ always() && inputs.gate-only != true }}``, so it is never
+produced -- and never expected -- on those paths.
 
 .. _ktlo-tier:
 
@@ -758,9 +757,9 @@ and keeps its legs' check names, ``Python 3.10 (KTLO)`` and so on. In
 with supported legs. Every job holding KTLO legs works the same way:
 
 * Every step that can fail on a KTLO leg has an ``id`` and
-  ``continue-on-error: ${{ matrix.ktlo == true }}`` (``unit-tests.yml:394-411``),
-  unless it is masked on every leg, as the timings uploads are. Its ``outcome`` is
-  then ``failure`` but its ``conclusion`` ``success``, so the job, its check run
+  ``continue-on-error: ${{ matrix.ktlo == true }}`` (see the comment above
+  ``test-ktlo``), unless it is masked on every leg, as the timings uploads are.
+  Its ``outcome`` is then ``failure`` but its ``conclusion`` ``success``, so the job, its check run
   and ``required-checks`` read ``success``. A step that needs an earlier one's work
   waits on that step's ``outcome``: a leg that cannot install stops there, and a
   ``pypi`` leg stops at its first failure.
@@ -1204,14 +1203,20 @@ does; each has tests under :file:`tests/project/` that run it.
        (`#1963 <https://github.com/JarryShaw/PyPCAPKit/issues/1963>`__), and in
        ``unit-tests.yml``'s ``test`` and ``test-ktlo``, on PyPy's legs (`#2005
        <https://github.com/JarryShaw/PyPCAPKit/issues/2005>`__).
+   * - :file:`util/regenerate_samples.py`
+     - ``Regenerate sample captures`` in ``unit-tests.yml``'s ``integration`` and
+       ``pypcap-parity`` (and so their KTLO twins), at the job's ``ANNOTATION_LEVEL``,
+       and ``gate``, at ``error``.
 
 The summaries run on the image's ``/usr/bin/python3`` rather than the leg's
 interpreter, which may have failed to set up or be too broken to run anything, so
 they need only the standard library, as ``classify_changes.py`` does on the
 runner's ``python3``. :file:`util/pytest_shards.py` runs on ``/usr/bin/python3``
 too, standard library only, so on GraalPy's leg the leg's interpreter runs pytest
 alone; on PyPy's legs the leg's interpreter runs the script as well, under the
-file-size cap every test command of ``unit-tests.yml`` takes. Both ``changes`` jobs check out on every event, since the script is in the
+file-size cap every test command of ``unit-tests.yml`` takes.
+:file:`util/regenerate_samples.py` is standard library only but runs on the leg's
+interpreter, since the generator it runs imports ``pcapkit``. Both ``changes`` jobs check out on every event, since the script is in the
 tree. ``create-release.yml``'s ``pypi`` job keeps its own copy of
 the KTLO summary's shell until its shell moves to scripts too; a project pin runs
 that copy and the script on the same steps and requires the same output.