Skip to content

harness: measure a runner's MLLP ingress rate instead of pass/failing one point - #118

Merged
wshallwshall merged 3 commits into
mainfrom
ingress-rate-probe
Aug 1, 2026
Merged

harness: measure a runner's MLLP ingress rate instead of pass/failing one point#118
wshallwshall merged 3 commits into
mainfrom
ingress-rate-probe

Conversation

@wshallwshall

Copy link
Copy Markdown
Collaborator

Diagnostic tooling for item #320. Two new files plus a three-line note; nothing existing changes behaviour.

Why

test_load_runner red the required windows-2025 leg twice on main (9b03057f, 56f7d240) with byte-identical counters — 90 sent, 44 acked, 46 stranded, 52 read — on runs that lost nothing. #115 widened the reconcile's stranding budget so a saturated-but-lossless run stops failing. That fixed the symptom and says nothing about the cause.

The open question is whether the windows-2025 image is slow or Windows generally is, and a pass/fail test at one fixed offered rate structurally cannot answer it. That's a rate, not a verdict.

What this adds

  • harness/load/ingress_probe.py — sweeps offered rate, prints one machine-parseable RESULT line per run.
  • .github/workflows/ingress-rate-probe.ymlworkflow_dispatch only, runs it across ubuntu-latest / windows-2022 / windows-2025 and writes a side-by-side table to the step summary.

ubuntu is the control, not a deployment target: it establishes what "not saturated" looks like on hosted hardware so a slow Windows row can be read against something.

The correction this PR is built on

I have to flag this, because I got it wrong first: I ran 600/s once on a developer box, got 456 stranded of 900 (50.7%) — a near-exact match for the windows-2025 signature — and wrote it up as a clean reproduction.

Four repeats of the same command on the same box then produced 0 stranded, every time. The outlier was taken while an unrelated test suite was running.

So stranding here is a contention artifact, not a clean function of offered rate, and n=1 is not a measurement. Hence --repeat, defaulted to 3, with the correction written into the module docstring so the next reader doesn't redo it.

What survives: an unloaded box strands zero at up to 10× the CI profile's rate, while windows-2025 stranded ~51% at the profile's own 60/s, twice, byte-identically. That's weaker than what I first claimed and still worth measuring properly.

Deliberately not a gate

No pull_request trigger, so it can never report on a PR or become the required-but-absent trap. And it is structurally unfit to gate anything: the probe exits 0 even when the reconcile fails, because a machine too slow to keep up is the finding, not an error. Recorded in .github/required-contexts.txt alongside the other advisory workflows.

It also prints no derived per-second figure. engine_read / wall looks like a service rate and isn't one — wall includes the stop grace, drain and settle-poll, so it lands at ~25/s whether the run offered 60/s or 600/s. I had that in a draft and removed it; reporting offered/ingested/stranded and letting the reader compare rows is honest, a fabricated rate is not.

What it still cannot answer

Both legs are hosted VMs. A slow-2025 / fast-2022 result narrows the cause to the 2025 image — it cannot clear or convict Windows Server 2025 as a deployment target, because a hosted image can be slow for reasons a deployed server isn't (Defender policy, VM disk backing, noisy neighbours).

Only the self-hosted WS2025 rig settles that, and it is unregisteredactions/runners returns total_count: 0 and selfhosted-win2025-sql.yml has never run. That remains the decisive experiment, and it needs the VM brought up.

Verification

  • ruff check + ruff format clean; pre-commit passed including the Actions linter.
  • YAML asserted to have no pull_request trigger.
  • Probe exercised locally at 60/s and 600/s, and with --repeat.
  • 222 workflow/required-context/posture guard tests pass, including the invariants that police the advisory-workflow list.

wshallwshall and others added 2 commits August 1, 2026 08:19
… one point

test_load_runner red the required windows-2025 leg twice on main (9b03057,
56f7d24) with byte-identical counters -- 90 sent, 44 acked, 46 stranded, 52 read
-- on runs that lost nothing. #115 widened the reconcile's stranding budget so a
saturated-but-lossless run stops failing. That fixed the symptom and said nothing
about the cause, and a pass/fail test at ONE fixed offered rate structurally
cannot: the question is whether windows-2025 is slow or windows is, and that is a
rate, not a verdict.

Adds `harness.load.ingress_probe` (sweeps offered rate, prints one machine-
parseable RESULT line per run) and a workflow_dispatch-only workflow that runs it
across ubuntu-latest / windows-2022 / windows-2025 and writes a side-by-side table
to the step summary. ubuntu is the CONTROL: it establishes what "not saturated"
looks like on hosted hardware so a slow Windows row can be read against it.

REPEATS ARE THE POINT, and this is a correction to my own earlier claim. One 600/s
run on a developer box produced 456 stranded of 900 (50.7%) -- a near-exact match
for the windows-2025 CI signature -- and I wrote it up as a clean reproduction.
Four repeats of the same command on the same box then produced 0 stranded, every
time; the outlier was taken while an unrelated test suite was running. So
stranding here is a CONTENTION artifact, not a clean function of offered rate, and
n=1 is not a measurement. Hence --repeat, defaulted to 3, with the correction
recorded in the module docstring so the next reader does not redo it.

What survives that correction is the weaker, still-useful claim: an unloaded box
strands ZERO at up to 10x the CI profile's rate, while windows-2025 stranded ~51%
at the profile's own 60/s, twice, byte-identically.

Deliberately NOT a required context and structurally unfit to become one: no
pull_request trigger, and the probe exits 0 even when the reconcile fails, because
a machine too slow to keep up is the finding rather than an error. Recorded in
.github/required-contexts.txt with the other advisory workflows. Prints no derived
per-second figure either -- engine_read/wall looks like a service rate and is not
one (wall includes stop grace, drain and settle, so it lands at ~25/s whether the
run offered 60/s or 600/s).

Scope: both legs are hosted VMs, so a slow-2025/fast-2022 result narrows the cause
to the 2025 IMAGE. It cannot clear or convict Windows Server 2025 as a deployment
target -- only the self-hosted WS2025 rig can, and it was unregistered
(actions/runners -> total_count 0) when this was written.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wshallwshall
wshallwshall enabled auto-merge (squash) August 1, 2026 13:27
auto-merge was automatically disabled August 1, 2026 13:31

Pull Request is not mergeable

wshallwshall added a commit that referenced this pull request Aug 1, 2026
test_backlog_status_check::test_the_real_backlog_satisfies_the_invariant fails on
origin/main at ea05525 itself. #320 (filed in #117) opened with a `📋 Filed …`
banner, which is not one of the five the invariant accepts (SHIPPED / DECLINED /
RETIRED / Re-scored / Status). An invented emoji instead of the defined vocabulary.

It fails in 0.6s inside every required test leg, so it red every PR that compiles
the suite -- already blocking auto-merge on #119 and would have blocked #118.

Corrected to `🚧 Status: OPEN INVESTIGATION …`, which is what the item is. The
banner now also carries the two facts a reader needs: measurement tooling landed
in #118, and the decisive experiment is blocked on an unregistered self-hosted
WS2025 runner.

Coverage hole recorded, not fixed here: #117 was docs-only, and ci.yml's `changes`
job short-circuits that case (code == 'false' skips install/lint/type/test), so
the guard that polices BACKLOG.md does not run on a PR that only changes
BACKLOG.md. The one class of change the invariant exists to catch is the class
that skips it. Follow-up: either count docs/BACKLOG.md as `code` for the
short-circuit, or run the ledger/backlog guards in a cheap always-on leg.
@wshallwshall
wshallwshall enabled auto-merge (squash) August 1, 2026 14:16
@wshallwshall
wshallwshall merged commit 534b067 into main Aug 1, 2026
34 checks passed
@wshallwshall
wshallwshall deleted the ingress-rate-probe branch August 1, 2026 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant