Skip to content

Commit 0659ebc

Browse files
tannevaledclaude
andauthored
perf: fold contiguous reductions directly; NumPy's pairwise sum for float32/complex (#51)
A float32 Sum of 2^20 elements took 6.85 ms on POWER9 against 0.23 ms for float64: 6.8 ns an element, while Add was only 1.5x slower. Every non-float64 reduction walked [outer][axisLen][inner] with the reduced axis outside, so a whole-array sum (inner == 1) took a slice and ran a one-iteration loop per element. A contiguous run is now folded directly, as NumPy's add and multiply reductions do: the first element, then the rest added to it, pairwise for float32 and complex (NumPy's FLOAT/CFLOAT_pairwise_sum, loops_utils.h.src v2.5.3), in order for integers and products. POWER9, 1 thread, n=2^20, benchstat over 6 interleaved runs (p=0.002): float32 Sum -76%, int64 -64%, int32 -54%, float64 (control) ~. More accurate too: error vs the exact sum 75 -> 0.87 on 2^16 mixed values, and 2^24 + 1000 ones no longer saturates at 2^24; the new tests fail on the old code. Coverage 100%; compiles for 386 and arm. Also: BenchmarkDTypeSum/Add, a perf.md section, and the roadmap brought up to date (fft, linalg and the Ruby binding shipped; float32 SIMD is what is open). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent a1f82eb commit 0659ebc

5 files changed

Lines changed: 334 additions & 10 deletions

File tree

‎benchmarks/dtype_bench_test.go‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
package benchmarks
2+
3+
import (
4+
"fmt"
5+
"testing"
6+
7+
nd "github.com/go-ndarray/ndarray"
8+
)
9+
10+
// The same reduction and elementwise operation on the dtypes that have SIMD
11+
// kernels (float64) and on those that run the pure-Go loops (float32 and the
12+
// integers), at a cache-resident and a memory-bound size. Run them with
13+
// GOMAXPROCS=1 to compare kernels rather than the multicore fan-out. They
14+
// measure what a float32 kernel would be worth before anyone writes one.
15+
16+
var dtypeBenchTypes = []nd.DType{nd.Float64, nd.Float32, nd.Int64, nd.Int32}
17+
18+
func BenchmarkDTypeSum(b *testing.B) {
19+
for _, dt := range dtypeBenchTypes {
20+
for _, n := range []int{1 << 14, 1 << 20} {
21+
x := vec(n).AsType(dt)
22+
b.Run(fmt.Sprintf("%v/n=%d", dt, n), func(b *testing.B) {
23+
b.SetBytes(int64(n * dt.Size()))
24+
for i := 0; i < b.N; i++ {
25+
_ = x.Sum()
26+
}
27+
})
28+
}
29+
}
30+
}
31+
32+
func BenchmarkDTypeAdd(b *testing.B) {
33+
for _, dt := range dtypeBenchTypes {
34+
for _, n := range []int{1 << 14, 1 << 20} {
35+
x, y := vec(n).AsType(dt), vec(n).AsType(dt)
36+
b.Run(fmt.Sprintf("%v/n=%d", dt, n), func(b *testing.B) {
37+
b.SetBytes(int64(n * dt.Size()))
38+
for i := 0; i < b.N; i++ {
39+
_, _ = x.Add(y)
40+
}
41+
})
42+
}
43+
}
44+
}

‎docs/perf.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -797,6 +797,42 @@ the morning. Only ratios taken in the same run are comparable. Not measured on
797797
Apple M-series, whose machine was shared with other jobs at the time. The bands
798798
stay dynamic, so its slower efficiency cores are handled as before.
799799

800+
## Sums of the other dtypes (2026-10-18)
801+
802+
Measured first, because nothing had: `BenchmarkDTypeSum` and
803+
`BenchmarkDTypeAdd` (`benchmarks/dtype_bench_test.go`) run the same operation
804+
on float64, float32, int64 and int32. On POWER9 (cfarm29, one thread pinned to
805+
the idlest CPU by `hostload -pick 1`, the host 3% busy before and after), a
806+
float32 `Sum` of 2²⁰ elements took **6.85 ms against 0.23 ms for float64**,
807+
6.8 ns an element, while `Add` was only 1.5× slower than float64. A missing
808+
SIMD kernel explains a factor of 4 to 8, not 30.
809+
810+
The cause was the reduction loop. Every non-float64 reduction walked the
811+
`[outer][axisLen][inner]` layout with the reduced axis outside and the kept
812+
elements inside, so a whole-array sum (`inner` = 1) took a slice and ran a
813+
one-iteration loop per element. A contiguous run is now folded directly, as
814+
NumPy's add and multiply reductions do: the first element, then the rest
815+
added to it, **pairwise for float32 and the complex dtypes** (NumPy's
816+
`FLOAT/CFLOAT_pairwise_sum`, `loops_utils.h.src` at v2.5.3: eight
817+
accumulators, blocks of 128, halved above), in order for the integers (exact
818+
in any order) and for products (NumPy multiplies in order).
819+
820+
| POWER9, 1 thread, n = 2²⁰ | before | after | |
821+
|---|---:|---:|---:|
822+
| float32 `Sum` | 6.850 ms | 1.637 ms | −76% |
823+
| int64 `Sum` | 5.195 ms | 1.870 ms | −64% |
824+
| int32 `Sum` | 6.170 ms | 2.868 ms | −54% |
825+
| float64 `Sum` (its own kernels; the control) | 225.7 µs | 225.7 µs | ~ |
826+
827+
Six runs each, old and new binaries alternating, `benchstat` p = 0.002. The
828+
float32 sum is also more accurate: on 2¹⁶ values of mixed sign and magnitude
829+
its error against the exact sum (`math/big`) fell from 75 to 0.87, and 2²⁴
830+
followed by 1000 ones no longer saturates at 2²⁴ (`sum_pairwise_test.go`).
831+
832+
Left: float32 has no SIMD kernel yet (still 7× float64 for `Sum`), and the
833+
integer sums convert the whole array to int64 before adding (NumPy's
834+
accumulator type), an extra pass and allocation.
835+
800836
## SIMD coverage
801837

802838
- **amd64 (SSE2)** ships hand-vectorized `sum` (4-accumulator `ADDPD`), `sqrt`

‎docs/plan-ndarray.md‎

Lines changed: 16 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -153,16 +153,22 @@ suite (plus the per-arch CI jobs already wired in `.github/workflows/ci.yml`).
153153
first-class bool dtype (so masks are real booleans) remains Phase 2.
154154
- **Phase 2 — dtypes — DONE.** Thirteen dtypes (bool, signed and unsigned
155155
integers, float32/64, complex64/128) as a runtime property of `Array`,
156-
with NumPy 2.x promotion; see [design-dtypes.md](design-dtypes.md). Next:
157-
SIMD kernels for `float32`, and the `fft` bridge.
158-
- **Phase 3 — broadcasting ufuncs.** A general ufunc framework over views
159-
(the axis-reduction primitives above are the first step); more elementwise
160-
and reduction ufuncs.
161-
- **Phase 4 — linear algebra.** `MatMul`/`Dot`, transpose-aware GEMM,
162-
decompositions; SIMD/blocked kernels.
163-
- **Phase 5 — Ruby binding.** Expose the array type to Ruby through
164-
[go-embedded-ruby](https://github.com/go-embedded-ruby/ruby) as a cgo-free
165-
alternative to `Numo::NArray`.
156+
with NumPy 2.x promotion; see [design-dtypes.md](design-dtypes.md). The
157+
`fft` bridge shipped in v0.10.0 (package `fft`, over go-fft). Still open:
158+
SIMD kernels for `float32` (float32 runs the pure-Go code today).
159+
- **Phase 3 — broadcasting ufuncs — MOSTLY DONE.** Broadcasting arithmetic,
160+
comparisons returning Bool masks, `Maximum`/`Minimum`, `Logical*`, `Power`,
161+
`Where` and the per-axis reductions, in every dtype. Not done: a public
162+
framework for user-defined ufuncs (`np.frompyfunc`, `np.vectorize`).
163+
- **Phase 4 — linear algebra — DONE (v0.11.0).** `MatMul`/`Dot` on a
164+
panel-packed GEMM with SIMD micro-kernels on six architectures, and package
165+
`linalg` (`numpy.linalg`: solve, inverse, determinants, Cholesky, QR, SVD,
166+
eigenproblems, least squares, norms); see [linalg.md](linalg.md) for what
167+
is left there.
168+
- **Phase 5 — Ruby binding — DONE.**
169+
[go-embedded-ruby](https://github.com/go-embedded-ruby/ruby) binds the array
170+
as `NDArray` (since its v0.5.0, the API the go-ndarray playground runs in the
171+
browser), a cgo-free alternative to `Numo::NArray`.
166172

167173
## 5. Decisions
168174

‎sum_pairwise_test.go‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
package ndarray
2+
3+
import (
4+
"math"
5+
"math/big"
6+
"testing"
7+
)
8+
9+
// Sums along a contiguous axis of float32 and complex arrays are pairwise, as
10+
// NumPy's are (FLOAT/CFLOAT_pairwise_sum, numpy/_core/src/umath/
11+
// loops_utils.h.src). The expectations below come from exact arithmetic, not
12+
// from running the code.
13+
14+
// TestFloat32SumDoesNotSaturate: 2^24 followed by 1000 ones. Added in order
15+
// in float32, every 2^24 + 1 rounds back to 2^24, so the sum stays 2^24.
16+
// Pairwise, the ones are summed among themselves first (1000, exact), and
17+
// 2^24 + 1000 is representable (float32's spacing at 2^24 is 2).
18+
func TestFloat32SumDoesNotSaturate(t *testing.T) {
19+
const n = 1000
20+
v := make([]float32, n+1)
21+
v[0] = 1 << 24
22+
for i := 1; i <= n; i++ {
23+
v[i] = 1
24+
}
25+
a := must(FromSlice(v, len(v)))
26+
if got, want := a.Sum(), float64(1<<24+n); got != want {
27+
t.Errorf("Sum = %v, want %v (in order it would be %v)", got, want, float64(1<<24))
28+
}
29+
// The same along the last axis of a 2-D array, row by row.
30+
two := must(FromSlice(append(append([]float32{}, v...), v...), 2, len(v)))
31+
rows, ok := Data[float32](must(two.SumAxis(1, false)))
32+
if !ok || len(rows) != 2 {
33+
t.Fatalf("SumAxis(1) of a 2-row float32 array: %v, %v", rows, ok)
34+
}
35+
for i, got := range rows {
36+
if got != 1<<24+n {
37+
t.Errorf("row %d: %v, want %v", i, got, 1<<24+n)
38+
}
39+
}
40+
c := make([]complex64, n+1)
41+
c[0] = complex(1<<24, -(1 << 24))
42+
for i := 1; i <= n; i++ {
43+
c[i] = complex(1, -1)
44+
}
45+
got := Item[complex64](must(FromSlice(c, len(c))).SumAll())
46+
if want := complex64(complex(1<<24+n, -(1<<24 + n))); got != want {
47+
t.Errorf("complex64 sum = %v, want %v", got, want)
48+
}
49+
}
50+
51+
// TestFloat32SumError compares the sum of 2^16 float32 values of mixed sign
52+
// and magnitude with the exact sum (math/big): pairwise must be far closer
53+
// than a sum in order.
54+
func TestFloat32SumError(t *testing.T) {
55+
const n = 1 << 16
56+
v := make([]float32, n)
57+
x := uint32(12345)
58+
for i := range v {
59+
x = x*1664525 + 1013904223 // a fixed LCG: reproducible, no seed to drift
60+
v[i] = float32(int32(x)>>8) / float32(1<<8) * float32(1+int(x%7))
61+
}
62+
exact := new(big.Float).SetPrec(256)
63+
var inOrder float32
64+
for _, f := range v {
65+
exact.Add(exact, new(big.Float).SetFloat64(float64(f)))
66+
inOrder += f
67+
}
68+
e, _ := exact.Float64()
69+
got := must(FromSlice(v, n)).Sum()
70+
errPairwise, errInOrder := math.Abs(got-e), math.Abs(float64(inOrder)-e)
71+
t.Logf("exact %v: pairwise %v (error %g), in order %v (error %g)", e, got, errPairwise, inOrder, errInOrder)
72+
if errPairwise*10 > errInOrder {
73+
t.Errorf("pairwise error %g is not ten times below the in-order error %g", errPairwise, errInOrder)
74+
}
75+
}
76+
77+
// TestProdAlongANonContiguousAxis: a product along axis 0 of a 2-D integer
78+
// array keeps the row-by-row loop (the reduced axis is not contiguous), and
79+
// accumulates in int64 as NumPy does for int32.
80+
func TestProdAlongANonContiguousAxis(t *testing.T) {
81+
a := must(FromSlice([]int32{1, 2, 3, 4, 5, -6}, 2, 3))
82+
p := must(a.ProdAxis(0, false))
83+
got, ok := Data[int64](p)
84+
if !ok || len(got) != 3 || got[0] != 4 || got[1] != 10 || got[2] != -18 {
85+
t.Errorf("ProdAxis(0) = %v %v, want int64 [4 10 -18]", p.DType(), got)
86+
}
87+
}
88+
89+
// TestPairwiseSumBranches covers each length class of NumPy's scheme and
90+
// checks it against the exact sum of small integers (exact in float32), and
91+
// that a sum of only -0 stays -0.
92+
func TestPairwiseSumBranches(t *testing.T) {
93+
for _, n := range []int{1, 2, 7, 8, 9, 15, 16, 127, 128, 129, 130, 1000, 4099} {
94+
v := make([]float32, n)
95+
c := make([]complex64, n)
96+
want := 0
97+
for i := range v {
98+
v[i] = float32(i%13 - 6)
99+
c[i] = complex(float32(i%5), float32(-(i % 3)))
100+
want += i%13 - 6
101+
}
102+
if got := must(FromSlice(v, n)).Sum(); got != float64(want) {
103+
t.Errorf("n=%d: float32 sum %v, want %d", n, got, want)
104+
}
105+
var wre, wim int
106+
for i := range c {
107+
wre += i % 5
108+
wim -= i % 3
109+
}
110+
if got := Item[complex64](must(FromSlice(c, n)).SumAll()); got != complex(float32(wre), float32(wim)) {
111+
t.Errorf("n=%d: complex64 sum %v, want (%d%+di)", n, got, wre, wim)
112+
}
113+
}
114+
negZero := float32(math.Copysign(0, -1))
115+
if got := must(FromSlice([]float32{negZero, negZero, negZero}, 3)).Sum(); !math.Signbit(got) {
116+
t.Errorf("a sum of -0 is %v, want -0", got)
117+
}
118+
nz := complex(negZero, negZero)
119+
if got := Item[complex64](must(FromSlice([]complex64{nz, nz}, 2)).SumAll()); !math.Signbit(float64(real(got))) || !math.Signbit(float64(imag(got))) {
120+
t.Errorf("a complex sum of -0 is %v, want -0-0i", got)
121+
}
122+
}

‎typed.go‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -345,6 +345,16 @@ const (
345345

346346
// reduceNum sums or multiplies along the axis.
347347
func reduceNum[T numT](op reduceCode, dst, src []T, outer, axisLen, inner int) {
348+
if inner == 1 {
349+
// The reduced axis is contiguous (the last axis, or the whole
350+
// array): fold each run directly. The loop below would take a
351+
// slice and run a one-iteration loop per element there, 6.8 ns an
352+
// element for a float32 sum on POWER9.
353+
for o := 0; o < outer; o++ {
354+
dst[o] = foldRun(op, src[o*axisLen:(o+1)*axisLen])
355+
}
356+
return
357+
}
348358
for o := 0; o < outer; o++ {
349359
row := dst[o*inner : (o+1)*inner]
350360
copy(row, src[o*axisLen*inner:(o*axisLen+1)*inner])
@@ -363,6 +373,112 @@ func reduceNum[T numT](op reduceCode, dst, src []T, outer, axisLen, inner int) {
363373
}
364374
}
365375

376+
// foldRun reduces one contiguous run (len(s) > 0) as NumPy's add and
377+
// multiply reductions do: the result starts as the first element and the
378+
// rest is added to it, pairwise for a floating sum (pairwiseSum), in order
379+
// otherwise. Integer sums are exact in any order, and NumPy multiplies in
380+
// order.
381+
func foldRun[T numT](op reduceCode, s []T) T {
382+
acc := s[0]
383+
rest := s[1:]
384+
if op == rSum {
385+
switch r := any(rest).(type) {
386+
case []float32: // Float64 sums never come here: they have kernels
387+
return any(any(acc).(float32) + pairwiseSum(r)).(T)
388+
case []complex64:
389+
return any(any(acc).(complex64) + pairwiseSumComplex(r)).(T)
390+
case []complex128:
391+
return any(any(acc).(complex128) + pairwiseSumComplex(r)).(T)
392+
}
393+
for _, v := range rest {
394+
acc += v
395+
}
396+
return acc
397+
}
398+
for _, v := range rest {
399+
acc *= v
400+
}
401+
return acc
402+
}
403+
404+
// pwBlock is NumPy's PW_BLOCKSIZE: runs up to this length are summed with
405+
// eight accumulators, longer ones are halved.
406+
const pwBlock = 128
407+
408+
// pairwiseSum is NumPy's FLOAT_pairwise_sum
409+
// (numpy/_core/src/umath/loops_utils.h.src, v2.5.3): rounding error O(lg n)
410+
// instead of O(n), in the same order of additions.
411+
func pairwiseSum[F float32](a []F) F {
412+
n := len(a)
413+
switch {
414+
case n < 8:
415+
// From -0, so that a sum of only -0 stays -0.
416+
res := F(math.Copysign(0, -1))
417+
for _, v := range a {
418+
res += v
419+
}
420+
return res
421+
case n <= pwBlock:
422+
var r [8]F
423+
copy(r[:], a[:8])
424+
i := 8
425+
for ; i < n-n%8; i += 8 {
426+
r[0] += a[i]
427+
r[1] += a[i+1]
428+
r[2] += a[i+2]
429+
r[3] += a[i+3]
430+
r[4] += a[i+4]
431+
r[5] += a[i+5]
432+
r[6] += a[i+6]
433+
r[7] += a[i+7]
434+
}
435+
res := ((r[0] + r[1]) + (r[2] + r[3])) + ((r[4] + r[5]) + (r[6] + r[7]))
436+
for ; i < n; i++ {
437+
res += a[i]
438+
}
439+
return res
440+
}
441+
n2 := n / 2
442+
n2 -= n2 % 8
443+
return pairwiseSum(a[:n2]) + pairwiseSum(a[n2:])
444+
}
445+
446+
// pairwiseSumComplex is NumPy's CFLOAT/CDOUBLE_pairwise_sum: the same scheme
447+
// over the interleaved real and imaginary parts, counted in floats (two per
448+
// element), so four accumulators per part and blocks of 64 elements.
449+
func pairwiseSumComplex[C complex64 | complex128](a []C) C {
450+
n := 2 * len(a) // NumPy's n counts floats
451+
switch {
452+
case n < 8:
453+
re, im := math.Copysign(0, -1), math.Copysign(0, -1)
454+
var acc C = C(complex(re, im))
455+
for _, v := range a {
456+
acc += v
457+
}
458+
return acc
459+
case n <= pwBlock:
460+
// r[0], r[2], r[4], r[6] hold real parts and r[1], r[3], r[5], r[7]
461+
// imaginary parts, of elements 0..3 of each block of four.
462+
var r [4]C
463+
copy(r[:], a[:4])
464+
i := 4
465+
for ; i < len(a)-len(a)%4; i += 4 {
466+
r[0] += a[i]
467+
r[1] += a[i+1]
468+
r[2] += a[i+2]
469+
r[3] += a[i+3]
470+
}
471+
res := (r[0] + r[1]) + (r[2] + r[3])
472+
for ; i < len(a); i++ {
473+
res += a[i]
474+
}
475+
return res
476+
}
477+
n2 := n / 2
478+
n2 -= n2 % 8
479+
return pairwiseSumComplex(a[:n2/2]) + pairwiseSumComplex(a[n2/2:])
480+
}
481+
366482
// reduceOrdered takes the maximum or minimum along the axis; for floats a NaN
367483
// wins, as in NumPy.
368484
func reduceOrdered[T realT](op reduceCode, dst, src []T, outer, axisLen, inner int) {

0 commit comments

Comments
 (0)