Repository navigation
feat: harden measurement + scoring pipeline (audit A1/A3/A4) - #3
Merged
Merged
Conversation
…sults
A sysbench run that exits 0 without an 'events per second' field produced
'CPU single thread: events/s, p95 1.00 ms', and the scorer then parsed the
'95' inside the 'p95' label as CPU throughput — a missing measurement was
scored as a real (FULL) result.
Now each sysbench metric validates its parsed value: missing/empty EPS or
memory-throughput fields mark the metric FAILED ('no events-per-second
field'), so the score degrades to PARTIAL/not-comparable instead of
borrowing digits from display text.
Adds a regression test with a p95-only sysbench fixture.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- memory: per-thread working set (--memory-scope=local, auto-sized MEM_BLOCK = RAM/threads/4 clamped to [32M,256M]) so the 15% memory subscore measures DRAM, not a shared 1MiB hot buffer - fio: malformed/jobless JSON and missing p95 metrics now report FAILED instead of being scored as a real zero - add --strict flag + ABS_STRICT env: exit non-zero on INCOMPLETE - wrap sysbench/fio calls in a portable timeout (ABS_CMD_TIMEOUT, default 900s) - JSON: provenance block (script sha256, sysbench/fio/python3 versions, score_model id) and mode fields cpu_prime/mem_block/strict - tests: regression coverage for per-thread scope, malformed fio JSON, and strict exit behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Documents abs-local-v1 formulas, reference constants and their heuristic provenance, verdict thresholds with the component-floor override, the comparability contract, and known limits. Linked from both READMEs. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- add() records an optional raw numeric value to metrics.tsv; all three Python scorers (local score, network score, verdict) consult it first and only fall back to display-text parsing, eliminating the class of bug where display text like 'p95 1.00' was scored - wrap fio (incl. prepare), iperf3, and dd fallback calls in abs_timeout; iperf watchdog is now portable (no GNU timeout dep) - regression test: scorer must prefer metrics.tsv over display text (23/23 pass, shellcheck clean, real run FULL 2466) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens the measurement and scoring pipeline per the source-audit findings (A1, A3, A4):
--memory-scope=localgives each sysbench thread its own working set, auto-sized asRAM/threads/4clamped to [32M, 256M] (overridable viaMEM_BLOCK). The previous--memory-block-size=1M+ default global scope benchmarked a 1 MiB hot buffer.FAILEDand degrade the run to PARTIAL, instead of scoring a fake 0.00.--strictflag /ABS_STRICT=1: exit non-zero when the verdict is INCOMPLETE, sobash abs.sh && ...is trustworthy.ABS_CMD_TIMEOUT, default 900 s) so a hung benchmark cannot stall the script forever.provenanceblock records script SHA-256, sysbench/fio/python3 versions, and thescore_modelid;modenow includescpu_prime,mem_block,strict.--strictexit codes. 22/22 pass, shellcheck clean.metrics.tsv, not display text: every measured value is captured as a numeric field atadd()time; all three scorers (local score, network score, verdict) consult it first and only fall back to text parsing — the whole A2-class of "digit inside a label gets scored" bugs is eliminated, not just its instance.--end_fsync=1prepare), iperf3 (portableabs_timeout, no GNU-timeout dependency), and the dd fallback are now time-bounded.docs/scoring.md):abs-local-v1formulas, the provenance of every reference constant, verdict thresholds, comparability contract, and honest calibration limits — linked from both READMEs.Verified end-to-end on macOS (Apple Silicon): full run completes, memory now reports ~79 GiB/s per-thread DRAM bandwidth (was ~183 GiB/s cache-hot), FULL score 2532.
Notes
workflowscope). Apply this patch to enable a ubuntu+macOS test matrix:Devin session: https://app.devin.ai/sessions/c74df4e507504d63aa16cf898640ae58
Requested by: BlueBirdBack