Skip to content

feat: harden measurement + scoring pipeline (audit A1/A3/A4) - #3

Merged
BlueBirdBack merged 5 commits into
mainfrom
devin/quality-hardening
Oct 11, 2026
Merged

BlueBirdBack merged 5 commits into
mainfrom
devin/quality-hardening

Conversation

@BlueBirdBack

@BlueBirdBack BlueBirdBack commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Hardens the measurement and scoring pipeline per the source-audit findings (A1, A3, A4):

  • Memory test measures DRAM, not a shared cache buffer: --memory-scope=local gives each sysbench thread its own working set, auto-sized as RAM/threads/4 clamped to [32M, 256M] (overridable via MEM_BLOCK). The previous --memory-block-size=1M + default global scope benchmarked a 1 MiB hot buffer.
  • Parse failures can no longer masquerade as zeros: malformed/jobless fio JSON and missing p95 metrics report FAILED and degrade the run to PARTIAL, instead of scoring a fake 0.00.
  • --strict flag / ABS_STRICT=1: exit non-zero when the verdict is INCOMPLETE, so bash abs.sh && ... is trustworthy.
  • Command timeouts: sysbench/fio calls wrapped in a portable watchdog (ABS_CMD_TIMEOUT, default 900 s) so a hung benchmark cannot stall the script forever.
  • Reproducible JSON: new provenance block records script SHA-256, sysbench/fio/python3 versions, and the score_model id; mode now includes cpu_prime, mem_block, strict.
  • Regression tests: coverage for per-thread scope selection, malformed fio JSON → FAILED, and --strict exit codes. 22/22 pass, shellcheck clean.
  • Scoring reads structured metrics.tsv, not display text: every measured value is captured as a numeric field at add() time; all three scorers (local score, network score, verdict) consult it first and only fall back to text parsing — the whole A2-class of "digit inside a label gets scored" bugs is eliminated, not just its instance.
  • All benchmark calls watchdoged: fio (including --end_fsync=1 prepare), iperf3 (portable abs_timeout, no GNU-timeout dependency), and the dd fallback are now time-bounded.
  • Scoring model doc (docs/scoring.md): abs-local-v1 formulas, the provenance of every reference constant, verdict thresholds, comparability contract, and honest calibration limits — linked from both READMEs.

Verified end-to-end on macOS (Apple Silicon): full run completes, memory now reports ~79 GiB/s per-thread DRAM bandwidth (was ~183 GiB/s cache-hot), FULL score 2532.

Notes

--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -9,13 +9,23 @@
 jobs:
   regression:
-    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest]
+    runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v5
-      - name: Check shell
+      - name: Check shell (Linux)
+        if: runner.os == 'Linux'
         run: |
           sudo apt-get update
           sudo apt-get install -y shellcheck
           shellcheck abs.sh
+      - name: Check shell (macOS)
+        if: runner.os == 'macOS'
+        run: |
+          brew install shellcheck
+          shellcheck abs.sh
       - name: Run regression suite
         run: python3 -m unittest -v tests/test_abs.py

Devin session: https://app.devin.ai/sessions/c74df4e507504d63aa16cf898640ae58
Requested by: BlueBirdBack

BlueBirdBack and others added 5 commits October 10, 2026 23:17
…sults

A sysbench run that exits 0 without an 'events per second' field produced
'CPU single thread: events/s, p95 1.00 ms', and the scorer then parsed the
'95' inside the 'p95' label as CPU throughput — a missing measurement was
scored as a real (FULL) result.

Now each sysbench metric validates its parsed value: missing/empty EPS or
memory-throughput fields mark the metric FAILED ('no events-per-second
field'), so the score degrades to PARTIAL/not-comparable instead of
borrowing digits from display text.

Adds a regression test with a p95-only sysbench fixture.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- memory: per-thread working set (--memory-scope=local, auto-sized
  MEM_BLOCK = RAM/threads/4 clamped to [32M,256M]) so the 15% memory
  subscore measures DRAM, not a shared 1MiB hot buffer
- fio: malformed/jobless JSON and missing p95 metrics now report
  FAILED instead of being scored as a real zero
- add --strict flag + ABS_STRICT env: exit non-zero on INCOMPLETE
- wrap sysbench/fio calls in a portable timeout (ABS_CMD_TIMEOUT,
  default 900s)
- JSON: provenance block (script sha256, sysbench/fio/python3
  versions, score_model id) and mode fields cpu_prime/mem_block/strict
- tests: regression coverage for per-thread scope, malformed fio
  JSON, and strict exit behavior

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Documents abs-local-v1 formulas, reference constants and their
heuristic provenance, verdict thresholds with the component-floor
override, the comparability contract, and known limits. Linked from
both READMEs.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- add() records an optional raw numeric value to metrics.tsv; all
  three Python scorers (local score, network score, verdict) consult
  it first and only fall back to display-text parsing, eliminating
  the class of bug where display text like 'p95 1.00' was scored
- wrap fio (incl. prepare), iperf3, and dd fallback calls in
  abs_timeout; iperf watchdog is now portable (no GNU timeout dep)
- regression test: scorer must prefer metrics.tsv over display text
  (23/23 pass, shellcheck clean, real run FULL 2466)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@BlueBirdBack
BlueBirdBack merged commit 88b0bf7 into main Oct 11, 2026
2 checks passed
@BlueBirdBack
BlueBirdBack deleted the devin/quality-hardening branch October 11, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant