ci(experimental): skip or stop superseded runs instead of cancelling them (#1938)

.github/workflows/experimental.yml

@@ -25,26 +25,88 @@ on:
 permissions:
   contents: read
 
-# #1903, the owner's option 1: a run is superseded by the next one on its ref,
-# whether a pull request's or a push to main's. Nothing waits on these legs, so
-# main's newest commit is the one worth testing, and a merge wave no longer
-# queues a two-hour GraalPy leg per push. unit-tests.yml still gives each push
-# a run of its own, since its `unittest-ordering` leg runs only on main. Any
-# other event -- a push to another branch, should `on:` ever allow one, or a
-# trigger added later -- keeps a group of its own, by run id, and is never
-# cancelled.
-concurrency:
-  group: experimental-tests-${{ (github.event_name == 'pull_request' || (github.event_name == 'push' && github.ref == 'refs/heads/main')) && github.ref || github.run_id }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && github.ref == 'refs/heads/main') }}
+# #1903 has a newer commit supersede a run, a pull request's or a push to
+# main's: nothing waits on these legs, so the newest commit is the one worth
+# testing. #1938, the owner's choice: a superseded run is skipped or stopped,
+# never cancelled, since GitHub counts a cancelled check against its commit and
+# marks it red. So there is no `concurrency:` block at all: without
+# `cancel-in-progress` a group still holds one pending run and cancels it once
+# a newer one arrives (docs.github.com, "Control the concurrency of workflows
+# and jobs"), and `queue: max` would hold a new commit's run behind the
+# superseded one until that one stops. Instead `current` skips every other job
+# of a run whose commit is superseded before it starts, and `Run unit tests`
+# stops one superseded while its tests run; util/superseded.py holds the rule
+# for both. A run whose commit stays current is unaffected.
 
 jobs:
+  # #1938: the gate. util/superseded.py's `check` asks the API whether this
+  # run's commit is still the head of main or of its open pull request, and
+  # writes `current`; any other event, or an API it cannot read, is current.
+  # Every other job needs this one and skips where `current` is `false`, so a
+  # superseded run's checks read `skipped`, which marks no commit red. Every
+  # step is `continue-on-error`: without `latest`, setup-python or requests the
+  # check fails or answers current, so a step's failure leaves `current` empty
+  # or `true`, which runs the rest; each job's `if:` opens with `!cancelled()`,
+  # so a failure of the job itself, which replaces the implicit `success()`,
+  # runs the rest too.
+  current:
+    name: Check the commit is still current
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions:
+      contents: read
+      pull-requests: read
+    outputs:
+      current: ${{ steps.check.outputs.current }}
+
+    steps:
+      - uses: actions/checkout@v7
+        continue-on-error: true
+        with:
+          sparse-checkout: |
+            .github/python-versions.toml
+            util/python_versions.py
+            util/superseded.py
+          sparse-checkout-cone-mode: false
+          persist-credentials: false
+
+      # #1660: `latest` from .github/python-versions.toml, read with the
+      # runner's own python3, for the interpreter util/superseded.py fetches
+      # with requests on.
+      - name: Read the Python versions
+        id: versions
+        continue-on-error: true
+        run: python3 util/python_versions.py --github-output
+
+      - uses: actions/setup-python@v7
+        id: python
+        continue-on-error: true
+        with:
+          python-version: ${{ steps.versions.outputs.latest }}
+
+      - name: Install requests
+        id: requests
+        continue-on-error: true
+        run: python -m pip install requests
+
+      - name: Ask whether this run's commit is still current
+        id: check
+        continue-on-error: true
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: python util/superseded.py check
+
   # unit-tests.yml's job of the same name, reading the one output this file needs.
   python-versions:
     name: Read the Python versions
+    needs: current
+    if: ${{ !cancelled() && needs.current.outputs.current != 'false' }}
     runs-on: ubuntu-latest
     timeout-minutes: 5
     outputs:
       experimental: ${{ steps.read.outputs.experimental }}
+      # #1938: the watcher's own interpreter, apart from the leg's.
+      latest: ${{ steps.read.outputs.latest }}
 
     steps:
       - uses: actions/checkout@v7
@@ -65,6 +127,8 @@ jobs:
   # pull_request run.
   changes:
     name: Classify changed files
+    needs: current
+    if: ${{ !cancelled() && needs.current.outputs.current != 'false' }}
     runs-on: ubuntu-latest
     timeout-minutes: 5
     outputs:
@@ -127,11 +191,12 @@ jobs:
   #
   # A job past its `timeout-minutes` is cancelled (same page,
   # `jobs.<job_id>.timeout-minutes`), and the page does not say whether the
-  # job-level guard covers that. So setup-python, the install and the tests
-  # carry caps of their own, 5, 10 and 120 minutes, under the job's 140: a step
-  # past its cap fails rather than cancels, and `continue-on-error` then applies
-  # as to any failure (actions/runner src/Runner.Worker/StepsRunner.cs:322-353,
-  # at 5f0b3ddcc). That leaves 5 minutes for the rest, which took at most 0.32,
+  # job-level guard covers that. So setup-python, the install, the watcher's
+  # setup-python and requests (#1938) and the tests carry caps of their own,
+  # 5, 10, 1, 1 and 120 minutes, under the job's 140: a step past its cap
+  # fails rather than cancels, and `continue-on-error` then applies as to any
+  # failure (actions/runner src/Runner.Worker/StepsRunner.cs:322-353, at
+  # 5f0b3ddcc). That leaves 3 minutes for the rest, which took at most 0.32,
   # setup-python included, on every uncovered leg of runs
   # 38569321543-38569522442.
   #
@@ -151,11 +216,20 @@ jobs:
   # expected, blocks nothing, and leaves the caps as they are.
   experimental:
     name: Python ${{ matrix.python-version }} (experimental)
-    if: ${{ needs.changes.outputs.code != 'false' }}
-    needs: [changes, python-versions]
+    # #1938: `!cancelled()` so a failed `current` skips nothing, and then the
+    # two results stand in for the implicit `success()` it replaces: a failed
+    # `changes` or `python-versions` still skips the legs.
+    if: >-
+      ${{ !cancelled() && needs.changes.result == 'success' && needs.python-versions.result == 'success'
+      && needs.current.outputs.current != 'false' && needs.changes.outputs.code != 'false' }}
+    needs: [current, changes, python-versions]
     runs-on: ${{ matrix.os || 'ubuntu-latest' }}
     timeout-minutes: 140
     continue-on-error: true
+    # #1938: `Run unit tests` reads the pull request, as `current` does.
+    permissions:
+      contents: read
+      pull-requests: read
     # #1672: every annotation a failing step writes is a warning.
     env:
       ANNOTATION_LEVEL: warning
@@ -202,6 +276,35 @@ jobs:
           nproc
           python -c "import os; print('cpu_count', os.cpu_count())"
 
+      # #1938: the interpreter `Run unit tests` runs util/superseded.py on, a
+      # CPython of its own: `latest`, never the leg's GraalPy or free-threaded
+      # build, which would also have to take requests. `update-environment:
+      # false` leaves PATH, `pythonLocation` and `LD_LIBRARY_PATH` as the leg's
+      # setup-python set them, so the tests see the leg's interpreter alone;
+      # this one is run by its `python-path`, and finds its own libpython by
+      # the rpath actions/python-versions builds with
+      # (builders/ubuntu-python-builder.psm1), since its soname is not the
+      # leg's. No `cache:` and `--no-cache-dir`, so the leg's pip cache is
+      # untouched. Each step has a cap of 1 minute (see the job's comment),
+      # and a failure leaves the tests to run unwatched.
+      - uses: actions/setup-python@v7
+        id: watcher-python
+        if: ${{ steps.install.outcome == 'success' }}
+        continue-on-error: true
+        timeout-minutes: 1
+        with:
+          python-version: ${{ needs.python-versions.outputs.latest }}
+          update-environment: false
+
+      - name: Install the watcher's requests
+        id: watcher
+        if: ${{ steps.install.outcome == 'success' && steps.watcher-python.outcome == 'success' }}
+        continue-on-error: true
+        timeout-minutes: 1
+        env:
+          WATCHER_PYTHON: ${{ steps.watcher-python.outputs.python-path }}
+        run: '"$WATCHER_PYTHON" -m pip install --no-cache-dir requests'
+
       # #1740: GraalPy's leg loses its runner 18-30 minutes into its tests ("The
       # runner has received a shutdown signal"), under a 120-minute cap. So every
       # 30 s this samples `free -m`, `df -h` of `/` and the workspace's volume,
@@ -313,6 +416,26 @@ jobs:
       # crashed, by xdist's `[gwN] node down:` lines (under `-q` it prints no
       # `replacing crashed worker`, xdist/dsession.py:78-80 at v3.8.0), for
       # `Containment summary`.
+      #
+      # #1938: util/superseded.py's `watch` runs the whole command -- the scope's
+      # sudo where there is one, then prlimit, timeout and pytest -- in a
+      # session of its own, outside the scope, and asks `current`'s question
+      # every 60 s. Once the commit is superseded it sends SIGTERM, SIGKILL 30 s
+      # on, to that session and, with `--scope`, to every process of
+      # pcapkit-tests.scope through `systemctl kill`: the scope's processes run
+      # as the runner's user under a root-owned sudo, which a signal to the
+      # session cannot pass. Ubuntu's own /etc/sudoers sets `use_pty` (sudo
+      # 1.9.9-1ubuntu2.7 and 1.9.15p5-3ubuntu5.24.04.4), and sudoers(5) says it
+      # has no effect where sudo has no terminal, as in a step; the scope is
+      # reached by name either way. Once the command exits it writes a
+      # `Superseded` notice and a job-summary line and exits 0, so the step
+      # succeeds and frees the runner; where it outlives SIGKILL by 15 s, a
+      # warning and exit 70, since the tests may run on until `timeout` ends
+      # them. Otherwise it exits with the command's own status, so a failure on
+      # a current commit reads as it would unwatched, and an API it cannot read
+      # counts as current. It runs on `watcher-python`, with requests; where
+      # that interpreter or its requests did not install, the tests run
+      # unwatched, as they would without #1938.
       - name: Run unit tests
         id: tests
         if: ${{ steps.install.outcome == 'success' }}
@@ -321,6 +444,9 @@ jobs:
         env:
           GRAAL_PYTHON_VM_ARGS: ${{ startsWith(matrix.python-version, 'graalpy') && '--vm.Xmx2g --vm.XX:+ExitOnOutOfMemoryError' || '' }}
           WORKERS: ${{ startsWith(matrix.python-version, 'graalpy') && '3' || 'auto' }}
+          GH_TOKEN: ${{ github.token }}
+          WATCHER: ${{ steps.watcher.outcome }}
+          WATCHER_PYTHON: ${{ steps.watcher-python.outputs.python-path }}
         run: |
           tail -n +1 -F "$RUNNER_TEMP/resource-monitor.log" 2>/dev/null &
           monitor=$!
@@ -350,6 +476,18 @@ jobs:
             # A log line, not an annotation: on a lost runner this log is all that is left.
             echo "No 12G scope: the probe failed (${probe}), so the tests run uncapped."
           fi
+          if [ "$WATCHER" = success ] && [ -x "$WATCHER_PYTHON" ]; then
+            watched=
+            if [ "$scope" = 12G ]; then
+              # `scoped` is a function, which the watcher runs through a shell.
+              export -f scoped
+              set -- bash -c '"$@"' bash "$@"
+              watched=pcapkit-tests.scope
+            fi
+            set -- "$WATCHER_PYTHON" util/superseded.py watch --interval 60 --grace 30 --scope "$watched" -- "$@"
+          else
+            echo "No watcher: its interpreter or requests did not install (${WATCHER:-skipped}), so the tests run unwatched."
+          fi
           oom_kills() { awk '$1 == "oom_kill" { print $2 }' /proc/vmstat; }
           before=$(oom_kills)
           started=$SECONDS

docs/source/contributing/workflows.rst

@@ -758,13 +758,50 @@ Tests run completes without it, and with it ``Required checks passed``,
 ``unittest-ordering`` and the workflows that wait on Unit Tests completing
 (``coverage-comment.yml``, ``deploy-pages.yml``, ``cron-vendor.yml`` and
 ``cron-conda.yml``). It runs on Unit Tests' ``push`` and ``pull_request`` events,
-skips a docs-only pull request by the same ``changes`` script, and cancels a
-superseded run, a pull request's or, unlike Unit Tests, a push to ``main``'s
+skips a docs-only pull request by the same ``changes`` script, and lets a newer
+commit supersede a run, a pull request's or, unlike Unit Tests, a push to ``main``'s
 (`#1903 <https://github.com/JarryShaw/PyPCAPKit/issues/1903>`__): since nothing
 waits on the legs, ``main``'s newest commit is the one worth testing, whereas Unit
-Tests keeps a run per push for ``unittest-ordering``, which runs only there. Any
-other event keeps a concurrency group of its own, by run id. The legs never
-block, by the KTLO mechanics above: every step that can fail has
+Tests keeps a run per push for ``unittest-ordering``, which runs only there.
+
+A superseded run is skipped or stopped, never cancelled (`#1938
+<https://github.com/JarryShaw/PyPCAPKit/issues/1938>`__): GitHub counts a
+``cancelled`` check against its commit, so a cancelled run marks the commit red.
+The workflow has no ``concurrency`` group either, since a group cancels its pending
+run once a newer one queues. Its first job, ``current`` ("Check the commit is still
+current"), sets up ``latest`` and installs ``requests``, which the script fetches with,
+then runs :file:`util/superseded.py`'s ``check``. A push's commit is current
+while it is the head of ``main``; a pull request's while the pull request is open and
+its head is still the event's ``pull_request.head.sha``; any other event's always.
+Every other job needs ``current`` and skips where its output is ``false``, so a run
+superseded before it starts ends with ``skipped`` checks and a ``Superseded`` notice.
+Each of those ``if:`` conditions opens with ``!cancelled()``, which drops the implicit
+``success()``, so a failed ``current`` step, which leaves the output empty, or a failed
+``current`` job runs the rest; the legs still need ``changes`` and ``python-versions``
+to succeed. ``Run unit tests`` runs its whole command -- the scope's ``sudo`` where
+there is one, then ``prlimit``, ``timeout`` and pytest -- under the script's ``watch``,
+outside the scope, which asks again every 60 seconds. The watcher runs on a CPython of
+its own, ``latest`` from a second setup-python step, ``watcher-python``, with
+``update-environment: false``, so the leg's ``PATH``, ``LD_LIBRARY_PATH`` and pip
+cache are as its own setup-python left them, and ``requests`` goes into that
+interpreter, never the leg's GraalPy or free-threaded build. The two steps are capped
+at 1 minute each, and where either fails the tests run unwatched. Once the commit is superseded,
+or the pull request merged or closed, it sends SIGTERM, then SIGKILL 30 seconds on:
+to ``pcapkit-tests.scope`` by name (``sudo -n systemctl kill``) where the tests run
+in it, since they run as the runner's user under a root-owned ``sudo`` and in
+``timeout``'s own process group, and to the command's session otherwise. Ubuntu's
+``/etc/sudoers`` sets ``use_pty``, which sudoers(5) says has no effect where ``sudo``
+has no terminal, as in a step; the scope is reached by name either way. Once the
+command exits, the watcher writes a ``Superseded`` notice and a job-summary line and
+exits 0, so the leg ends green and frees its runner. Every wait is bounded: where the
+command outlives SIGKILL by 15 seconds, it writes a ``Superseded run not stopped``
+warning and exits 70, not 0, since the tests may run on, holding the step's output,
+until ``timeout`` or the step's cap ends them, and the failed step is then a warning
+like any other on these legs. Otherwise the step ends with the tests' own status. An
+API the script cannot read counts as current, so it never stops the tests, and it
+asks with ``If-None-Match``, as a ``304`` answer costs no rate limit.
+
+The legs never block, by the KTLO mechanics above: every step that can fail has
 ``continue-on-error: true``, their annotations are warnings, and ``Experimental
 summary``, the job's last step, reports each failed step. A failure no step
 contains, such as a lost runner, fails the job, and the job's own
@@ -773,8 +810,10 @@ run from failing when a job fails" (`Workflow syntax
 <https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idcontinue-on-error>`__),
 though the leg's check run still reads ``failure``. A job past its
 ``timeout-minutes`` is cancelled, and that page does not say whether the job-level
-guard covers a cancellation. So setup-python, the install and the tests have caps
-of their own, of 5, 10 and 120 minutes, under the job's 140: a step at its cap
+guard covers a cancellation. So setup-python, the install, the watcher's setup-python
+and ``requests`` (`#1938 <https://github.com/JarryShaw/PyPCAPKit/issues/1938>`__,
+below) and the tests have caps of their own, of 5, 10, 1, 1 and 120 minutes, under
+the job's 140, which leaves 3 for the rest: a step at its cap
 fails rather than cancels, and the guard holds (actions/runner
 ``src/Runner.Worker/StepsRunner.cs:322-353``). The runner's ``has timed out`` error
 stays, changing no conclusion, as ``Process completed with exit code N.`` does.