harness: measure a runner's MLLP ingress rate instead of pass/failing one point - #118
Merged
Conversation
… one point test_load_runner red the required windows-2025 leg twice on main (9b03057, 56f7d24) with byte-identical counters -- 90 sent, 44 acked, 46 stranded, 52 read -- on runs that lost nothing. #115 widened the reconcile's stranding budget so a saturated-but-lossless run stops failing. That fixed the symptom and said nothing about the cause, and a pass/fail test at ONE fixed offered rate structurally cannot: the question is whether windows-2025 is slow or windows is, and that is a rate, not a verdict. Adds `harness.load.ingress_probe` (sweeps offered rate, prints one machine- parseable RESULT line per run) and a workflow_dispatch-only workflow that runs it across ubuntu-latest / windows-2022 / windows-2025 and writes a side-by-side table to the step summary. ubuntu is the CONTROL: it establishes what "not saturated" looks like on hosted hardware so a slow Windows row can be read against it. REPEATS ARE THE POINT, and this is a correction to my own earlier claim. One 600/s run on a developer box produced 456 stranded of 900 (50.7%) -- a near-exact match for the windows-2025 CI signature -- and I wrote it up as a clean reproduction. Four repeats of the same command on the same box then produced 0 stranded, every time; the outlier was taken while an unrelated test suite was running. So stranding here is a CONTENTION artifact, not a clean function of offered rate, and n=1 is not a measurement. Hence --repeat, defaulted to 3, with the correction recorded in the module docstring so the next reader does not redo it. What survives that correction is the weaker, still-useful claim: an unloaded box strands ZERO at up to 10x the CI profile's rate, while windows-2025 stranded ~51% at the profile's own 60/s, twice, byte-identically. Deliberately NOT a required context and structurally unfit to become one: no pull_request trigger, and the probe exits 0 even when the reconcile fails, because a machine too slow to keep up is the finding rather than an error. Recorded in .github/required-contexts.txt with the other advisory workflows. Prints no derived per-second figure either -- engine_read/wall looks like a service rate and is not one (wall includes stop grace, drain and settle, so it lands at ~25/s whether the run offered 60/s or 600/s). Scope: both legs are hosted VMs, so a slow-2025/fast-2022 result narrows the cause to the 2025 IMAGE. It cannot clear or convict Windows Server 2025 as a deployment target -- only the self-hosted WS2025 rig can, and it was unregistered (actions/runners -> total_count 0) when this was written. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
wshallwshall
enabled auto-merge (squash)
August 1, 2026 13:27
auto-merge was automatically disabled
August 1, 2026 13:31
Pull Request is not mergeable
wshallwshall
added a commit
that referenced
this pull request
Aug 1, 2026
test_backlog_status_check::test_the_real_backlog_satisfies_the_invariant fails on origin/main at ea05525 itself. #320 (filed in #117) opened with a `📋 Filed …` banner, which is not one of the five the invariant accepts (SHIPPED / DECLINED / RETIRED / Re-scored / Status). An invented emoji instead of the defined vocabulary. It fails in 0.6s inside every required test leg, so it red every PR that compiles the suite -- already blocking auto-merge on #119 and would have blocked #118. Corrected to `🚧 Status: OPEN INVESTIGATION …`, which is what the item is. The banner now also carries the two facts a reader needs: measurement tooling landed in #118, and the decisive experiment is blocked on an unregistered self-hosted WS2025 runner. Coverage hole recorded, not fixed here: #117 was docs-only, and ci.yml's `changes` job short-circuits that case (code == 'false' skips install/lint/type/test), so the guard that polices BACKLOG.md does not run on a PR that only changes BACKLOG.md. The one class of change the invariant exists to catch is the class that skips it. Follow-up: either count docs/BACKLOG.md as `code` for the short-circuit, or run the ledger/backlog guards in a cheap always-on leg.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Diagnostic tooling for item #320. Two new files plus a three-line note; nothing existing changes behaviour.
Why
test_load_runnerred the required windows-2025 leg twice onmain(9b03057f,56f7d240) with byte-identical counters — 90 sent, 44 acked, 46 stranded, 52 read — on runs that lost nothing. #115 widened the reconcile's stranding budget so a saturated-but-lossless run stops failing. That fixed the symptom and says nothing about the cause.The open question is whether the windows-2025 image is slow or Windows generally is, and a pass/fail test at one fixed offered rate structurally cannot answer it. That's a rate, not a verdict.
What this adds
harness/load/ingress_probe.py— sweeps offered rate, prints one machine-parseableRESULTline per run..github/workflows/ingress-rate-probe.yml—workflow_dispatchonly, runs it across ubuntu-latest / windows-2022 / windows-2025 and writes a side-by-side table to the step summary.ubuntuis the control, not a deployment target: it establishes what "not saturated" looks like on hosted hardware so a slow Windows row can be read against something.The correction this PR is built on
I have to flag this, because I got it wrong first: I ran 600/s once on a developer box, got 456 stranded of 900 (50.7%) — a near-exact match for the windows-2025 signature — and wrote it up as a clean reproduction.
Four repeats of the same command on the same box then produced 0 stranded, every time. The outlier was taken while an unrelated test suite was running.
So stranding here is a contention artifact, not a clean function of offered rate, and n=1 is not a measurement. Hence
--repeat, defaulted to 3, with the correction written into the module docstring so the next reader doesn't redo it.What survives: an unloaded box strands zero at up to 10× the CI profile's rate, while windows-2025 stranded ~51% at the profile's own 60/s, twice, byte-identically. That's weaker than what I first claimed and still worth measuring properly.
Deliberately not a gate
No
pull_requesttrigger, so it can never report on a PR or become the required-but-absent trap. And it is structurally unfit to gate anything: the probe exits 0 even when the reconcile fails, because a machine too slow to keep up is the finding, not an error. Recorded in.github/required-contexts.txtalongside the other advisory workflows.It also prints no derived per-second figure.
engine_read / walllooks like a service rate and isn't one —wallincludes the stop grace, drain and settle-poll, so it lands at ~25/s whether the run offered 60/s or 600/s. I had that in a draft and removed it; reporting offered/ingested/stranded and letting the reader compare rows is honest, a fabricated rate is not.What it still cannot answer
Both legs are hosted VMs. A slow-2025 / fast-2022 result narrows the cause to the 2025 image — it cannot clear or convict Windows Server 2025 as a deployment target, because a hosted image can be slow for reasons a deployed server isn't (Defender policy, VM disk backing, noisy neighbours).
Only the self-hosted WS2025 rig settles that, and it is unregistered —
actions/runnersreturnstotal_count: 0andselfhosted-win2025-sql.ymlhas never run. That remains the decisive experiment, and it needs the VM brought up.Verification
ruff check+ruff formatclean; pre-commit passed including the Actions linter.pull_requesttrigger.--repeat.