You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Memory-bound rows flip their verdict between two valid runs of the same code (load 6-8) #20
Two valid -perf.run reports of go-simd/floats, on the same code, disagree on the verdict of memory-bound rows by more than either report's interval. The verdict printed in perf-table.md (faster / tie / slower) is therefore not reproducible for those rows at loads between 6 and 8, although each run passes the gate of 8.
The two reports:
A: main's table, measured at 83132fd on 2026-11-15, load 6.5 to 7.96, 13 retries, refbench v0.4.1.
B: branch perf-remeasure-low-load (5edc8f2), measured at 9c45587 on 2026-11-16, load 6.0 to 7.94, 3 retries, refbench v0.4.1.
The code of the rows below is identical between the two commits.
row
A: Go / numpy, ratio
B: Go / numpy, ratio
Mul/1000000
3.309e5 [2.987e5, 3.342e5] / 2.291e5, 1.44 slower
2.016e5 [1.61e5, 2.167e5] / 2.0e5, 1.01 tie
Sub/1000000
2.021e5 / 1.951e5, 1.04 slower
1.76e5 / 1.999e5, 0.88 tie
Mul/100000
faster or tie
1.997e4 [1.905e4, 2.111e4] / 1.517e4, 1.32 slower
Add/100000
faster or tie
1.867e4 / 1.513e4, 1.23 slower
Sub/100000
faster or tie
1.883e4 / 1.5e4, 1.26 slower
CDotC/100000
faster or tie
1.746e4 / 1.457e4, 1.20 slower
Compute-bound rows (Dot, Sum, CosineSimilarity…) agree between A and B. The rows that flip move data at roughly memory bandwidth (1e5 to 1e6 float64, two or three operands). That makes them sensitive to what the ~25 other processes on the machine do to the memory bus and the shared caches, which the 1-minute load average does not measure.
Consequences:
A repository's table can turn a row "slower" without any code change. Under the fleet rule "slower = open defect", that opens a false defect; it can also hide a true one.
The per-row interval describes the variance inside one run, not between runs, so it cannot tell the two apart.
Possible directions (not a design):
Record a between-run spread, for example by re-measuring each row in a second, separated pass and keeping both. Mark a row whose verdict differs between the two as unstable rather than faster or slower.
A stricter gate for rows flagged memory-bound, for example by bytes moved per call.
At minimum, print in the table that a tie band wider than the interval applies to these rows.
Activity