Skip to content

Latest commit

 

History

History
213 lines (189 loc) · 11.5 KB

perf.md

File metadata and controls

213 lines (189 loc) · 11.5 KB

Performance: go-ndarray against NumPy

The table is perf-table.md, generated by refbench (go-scikit/refbench v0.3.0) from perf.json; nothing in it is edited by hand. The rules are those of the go-scikit performance campaign (go-scikit/roadmap#27):

  • One core on both sides. GOMAXPROCS=1 for Go, and the reference worker runs with OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, VECLIB_MAXIMUM_THREADS at 1. GOMAXPROCS=1 must also be in the environment of the test binary: ndarray/fft rests on go-fft/fft, which reads it once at init (go-fft/fft#107).
  • The pinned reference: NumPy 2.5.4 (bench/refs.txt, hashes included), Python 3.12.12 through uv. The earlier page of this repository compared multicore Go with single-threaded NumPy 2.2.4 and is kept as perf-history.md for its kernel-level analysis only.
  • Interleaved rounds, one statistic: a round is one sample per side, alternating which runs first; the verdict is on the median with quartiles.
  • Load gate: no timing verdict at a 1-minute load of 8 or more (-perf.run refuses and writes nothing).
  • Same data: every array argument is lo + (hi-lo) * frac(i * 0.618...), built in Go and in NumPy; TestDataParity compares the SHA-256 of both sides' bytes (24 distinct argument sets, complex ones included, all equal), and TestRefSmoke runs every NumPy call once.
  • One case per public function (a function with several shapes of use has several rows, Array.SumAxis/axis0 and /axis1), or a named exemption with its reason in perf-exempt.md. CI fails on a public function with neither.

What a row times

Each row calls the Go function once on a vector of 100 000 elements (a 1000 x 100 matrix for the axis reductions, 128 x 128 for the matrix products, 64 x 64 for linalg, 65 536 points or 256 x 256 or 32 x 32 x 32 for fft), and the NumPy call on the same data. The result is allocated by the call on both sides; the *Into rows write into a preallocated array on both sides (out=). The random rows draw 100 000 values from a generator built once outside the timed loop (default_rng(1) / DefaultRNG(1), RandomState(1), np.random.seed(1)), so the constructor is not in the interval. Views (Reshape, Slice, Transpose, Ravel) are views in NumPy; where Go copies, the row says so.

fft rows include the plan on both sides: ndarray/fft builds or fetches a plan per call, numpy.fft keeps its own plan cache warm after calibration. The scipy.fft comparison of the transforms themselves is go-fft/fft's.

Value parity is not checked here: each package's own tests compare its results with NumPy's fixtures (random/numpy_test.go, linalg/testdata, fft/testdata).

Running it

cd bench
GOMAXPROCS=1 go test -run TestPerf -cpu 1 .               # CI mode: allocations, table, public-function inventory
GOMAXPROCS=1 go test -run TestPerf -cpu 1 -perf.run .     # measure; writes docs/perf.json and docs/perf-table.md
GOMAXPROCS=1 go test -run TestPerf -perf.record .         # allocation counts only (status: deferred)
go test -run 'TestDataParity|TestRefSmoke' -perf.parity . # needs uv

Status

The table is measured at 37c1b86 (the head of the perf/fftshift-one-pass branch, clean) on 2026-11-15, on one pinned core of cfarm420 (AMD EPYC 7773X, Zen 3), NumPy 2.5.4. The 1-minute load over the rounds was 2.62 to 3.99, and all 256 rows are valid. It replaces the table measured at v0.41.0 (de3f41ac, 2026-11-14, load 4.12 to 5.56), and the section below lists what differs.

185 rows are slower than NumPy, 54 are faster and 17 tie (v0.41.0: 188, 54 and 14). Every slower row is an open defect. They are tracked by cause in #118 rather than in one issue per row:

  • result allocation: the allocating Add is 3.9× slower, while AddInto is within 5% (1.05);
  • elementwise operations: the unary Ceil (6.0×), Floor (5.8×), Square (5.5×), Abs (5.5×), Neg (5.4×) and Round (4.7×) now run on vector kernels and what remains is the result allocation; Maximum, Minimum and Power (5.3–5.8×) and Log2 (4.0×) are the worst of the others;
  • copies where NumPy returns a view or wraps the buffer (Flatten 5.7×, Copy 5.8×, Imag 5.2×);
  • linalg at 64 × 64 (1.5–3.9×);
  • the fft shifts, now 2.0× (they were 9.7× and 9.9×);
  • the random draws built on log/exp.

Faster, for example: SqrtInto (0.50), fft.FFT (0.52), Sum (0.80), Log10 (0.57). Exp (0.98) ties.

Since the v0.41.0 table

The changes below were made after v0.41.0 and are included in the table above. The ratios of this table against the v0.41.0 table (Go time / NumPy time, 10⁵ float64):

row v0.41.0 now allocations
fft.FFTShift 9.72 2.03 29 → 9
fft.IFFTShift 9.90 2.02 29 → 9
Ravel 566.8 0.36 4 → 3
Real 565.9 0.58 4 → 3
Ceil 20.4 5.95 5
Floor 19.2 5.75 5
Round 13.8 4.67 5
Abs 11.9 5.48 5
Square 11.0 5.51 5
Neg 10.3 5.44 5
AddScalar 8.56 5.40 8
Full 8.00 5.51 5
Ones 7.34 5.30 5
fft.RFFT2 1.41 1.17 31 → 30
fft.RFFT2With 1.50 1.15 31 → 30
fft.IHFFT 1.33 1.04 35 → 34
fft.IHFFTWith 1.22 1.05 35 → 34
fft.RFFTN 0.95 0.75 32 → 31
fft.RFFTNWith 0.92 0.73 32 → 31
fft.RFFT 0.90 0.75 30 → 29
fft.RFFTWith 0.95 0.79 30 → 29

fft.FFTShift and fft.IFFTShift now shift in one pass (the allocation count falls from 29 to 9), and their ratio from about 9.8 to 2.0. This change is well above the run-to-run spread below. They remain slower than NumPy.

Ravel and Real are now faster than NumPy, and the fft real-input transforms each save one allocation because they call x.Real(), which is a view. The unary rows remain slower than NumPy, by about 5× instead of 10–20×: they still allocate the result, which AddInto shows to be the main part of the remaining gap (Add 3.85, AddInto 1.05).

Treat small ratio changes with care: the tables were measured at different loads, and rows whose code did not change moved by as much as 0.2 to 0.3 (Sqrt 1.01 to 1.32, fft.IRFFT 0.72 to 0.74 in the third run but 1.06 in the second, SubScalar 4.81 to 7.11 between two runs of the same code). Only the rows above whose change comes from a known code change, and whose size exceeds that spread, are claimed as effects: FFTShift and IFFTShift, Ravel and Real, and the allocation counts. The other fft real-input ratios (0.1 to 0.3 lower than at v0.41.0) are within or near that spread, so only their allocation counts are attributed to Real, and the elementwise ratios do not show a difference between kernels.

The measurements that follow were made at the time by go test -bench on the same core. On 10⁵ float64 at load 2.2–2.4, over three interleaved rounds:

  • The scalar forms (AddScalar … DivScalar, AddInto … DivInto, and any Float64 operation with a one-element operand against a contiguous array of the result's shape) no longer copy the scalar to the result's size. A strided operand, such as a transpose, still takes the general path. They stream it through the same SIMD kernel, so the result is bit for bit the same. AddScalar went from 183–203 µs to 95–110 µs, and from 1.6 MB to 0.8 MB per call.

  • Full, Ones and FullOf fill by doubling copy instead of storing one element at a time. Full went from 131–135 µs to 84–104 µs. Ones calls Full. FullOf, which fills the other dtypes, was not timed.

  • Float64 Abs, Neg, Square, Floor, Ceil and Round run on the go-simd/floats v0.23.0 kernels (above the parallel threshold, split across cores as before) instead of a scalar loop through Map. The results are bit for bit those of math.Abs, -x, x*x, math.Floor, math.Ceil and math.RoundToEven. Abs and Neg keep NaN bits exactly; for Square, Floor, Ceil and Round a signalling NaN may come back quieted (TestUnaryKernelsBitIdentical: edge table, lengths 0–33 and one above the threshold). Round is now half to even, as np.round; the old half away from zero is RoundHalfAway, on the scalar path (see the README). go test -bench Unary (benchmarks/unary_bench_test.go), 10⁵ float64 including the result allocation, one pinned core of cfarm420 (GOMAXPROCS=1, taskset -c 9), a test binary of v0.42.0 and one of this change, four interleaved rounds of 1 s, 1-minute load 2.34 at the start and 3.32 at the end. Before → after, range over the rounds:

    operation before (µs) after (µs)
    Abs 244–258 70–75
    Neg 243–249 72–75
    Square 243–247 72–80
    Floor 432–457 73–81
    Ceil 437–464 72–81
    Round (the old one was half away from zero) 325–340 71–80

    Round was measured in a second run of the same kind (four interleaved rounds, load 2.25 to 2.42), v0.42.0 against this change. RoundHalfAway, the control on the scalar path, takes 324–345 µs in the same run. These are times of the Go code against itself; the NumPy ratios are in the table above.

  • Ravel of a contiguous array and Real of a real array return views, as NumPy does, instead of copying (the table's rows were 566× slower than NumPy, which returns views). On 10⁵ float64, cfarm420, one pinned core, three interleaved rounds at load 2.1 to 2.3: Ravel went from 83–90 µs to 93–101 ns and Real from 84–91 µs to 101–104 ns, 803 kB to 128 B per call. The new table measures them against NumPy at 63 ns and 97 ns against 174 ns and 168 ns (ratios 0.36 and 0.58). This is a behaviour change (see the README).

  • Float64 Maximum and Minimum run on the go-simd/floats v0.26.0 kernels instead of the scalar builtin loop, in every path that calls them (same shape, scalar and repeated operands, broadcast). The results are bit for bit those of the builtin max and min for numbers and zeros; a NaN gives a NaN, as before, and which NaN is target-dependent (TestMaximumMinimumFloats). On 10⁵ float64 including the result allocation, cfarm420, one pinned core, a test binary of main against one of this change, four interleaved rounds of 1 s at load 3.6 to 4.0 (benchmarks/maxmin_bench_test.go): Maximum went from 139–150 µs to 76–82 µs and Minimum from 141–151 µs to 76–81 µs. The allocation is unchanged, 2 per call and 803 kB. No NumPy ratio is claimed for these numbers: the new table has not been run on this change.

What remains of the allocation cause is Go's own cost. A CPU profile of a bare make([]float64, 100000) shows about 60% garbage-collector marking (assists in the allocating goroutine) and about 23% zeroing. NumPy frees each result when its last reference goes and reuses that memory warm. In Go, the way to avoid both costs is to not allocate: use the *Into forms or a Workspace.

The run was made from the measured commit's own code. The test binary was cross-compiled from a clean checkout of 37c1b86, the commit was shipped to the host as a git bundle, and the binary ran from outside the clone so that the tree stayed clean. Done that way, the recorded commit is 37c1b86 itself.