Skip to content

Commit 90d0a5c

Browse files
tannevaledclaude
andauthored
perf: axis reductions split columns across cores and use the SIMD kernels (#13)
On a Zen 3 (16 cores) the axis reductions of a 1024x1024 matrix lost to NumPy 0.23-0.43x, for two reasons: - An axis-0 reduction ran on one core: it has a single outer slab, and RunAxisP only split outer slabs, on the belief that a column split needs a strided gather. A band of columns is a contiguous run of every row, so the kernels now take a column band [lo, hi) and RunAxisP hands bands of at least 512 columns to the workers when there are fewer slabs than workers. Rows are combined with the elementwise SIMD add/mul from 16 columns on, element by element in axis order, so the result is unchanged. - A row reduction (inner == 1) was a sequential scalar loop, latency-bound. Each row now goes through the SIMD sum/max/min. Max and min are exact; the sum is regrouped lane-parallel, as NumPy's pairwise sum is. Zen 3, 16 cores pinned, min of three interleaved runs: SumAxis(0) 696 -> 186 us (0.97x NumPy 1 thread) SumAxis(1) 531 -> 92 us (2.4x NumPy) MaxAxis(1) 601 -> 93 us (2.0x NumPy) New test: every RunAxisP schedule (serial, slab split, column bands for one slab and for fewer slabs than workers, narrow fallback, row sums) for Sum/Prod/Max/Min against a naive in-order oracle, exact except the regrouped row sum (1e-12). Shifting the band offset makes it fail. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent c67dabd commit 90d0a5c

6 files changed

Lines changed: 222 additions & 96 deletions

File tree

‎docs/perf.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -411,6 +411,32 @@ amd64 (Zen 3, FMA path), arm64 (cfarm185), ppc64le (cfarm120), riscv64 (cfarm94)
411411
and loong64 (cfarm401). The s390x host (LinuxONE) did not answer; s390x stays
412412
covered by the qemu lane in CI.
413413

414+
## Axis reductions (2026-10-04)
415+
416+
On the Zen 3 the axis reductions of a 1024×1024 matrix lost to NumPy 0.23–0.43×.
417+
Two causes, both measured:
418+
419+
- **An axis-0 reduction ran on one core.** It has a single outer slab, and the
420+
driver only split outer slabs; the comment justifying that said a column split
421+
would need a strided gather. It does not: a band of columns is a contiguous run
422+
of every row. `RunAxisP` now splits the columns into bands of at least 512
423+
when there are fewer slabs than workers, and each row run is added with the
424+
elementwise SIMD kernel (exact per element, so the sum is still in axis order).
425+
- **A row reduction (axis 1, `inner` = 1) was a sequential scalar loop**,
426+
latency-bound at one add per 4 cycles. Each row now goes through the SIMD
427+
sum/max/min. Max and min are exact; the sum is regrouped lane-parallel, as
428+
NumPy's own pairwise sum is.
429+
430+
Zen 3, 16 cores pinned, minimum of three interleaved runs:
431+
432+
| op (1024×1024) | before | after | NumPy 1 thread | after vs NumPy |
433+
|----|--:|--:|--:|:--:|
434+
| SumAxis(0) | 696 µs | **186 µs** | 180 µs | 0.97× |
435+
| SumAxis(1) | 531 µs | **92 µs** | 224 µs | 2.4× |
436+
| MaxAxis(1) | 601 µs | **93 µs** | 189 µs | 2.0× |
437+
438+
On M4: MaxAxis(1) ×6–8, SumAxis(0) ×1.9.
439+
414440
## SIMD coverage
415441

416442
- **amd64 (SSE2)** ships hand-vectorized `sum` (4-accumulator `ADDPD`), `sqrt`

‎internal/kernels/kernels.go‎

Lines changed: 73 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -200,80 +200,105 @@ func Dot1D(a, b []float64) float64 {
200200
// The parent package materialises a strided view into a contiguous []float64
201201
// laid out conceptually as [outer][axisLen][inner], i.e. the reduced axis sits
202202
// in the middle and `inner` trailing elements are contiguous. Each kernel
203-
// reduces the middle axis, writing outer*inner results into dst (also laid out
204-
// as [outer][inner]).
205-
//
206-
// The innermost loop over `inner` walks contiguous memory with unit stride,
207-
// which is the shape a SIMD kernel wants: Phase 1 can replace each step (the
208-
// `inner`-length combine of dst[base:] with src[off:]) with a vectorised
209-
// fused operation across all six 64-bit targets, leaving this scalar version as
210-
// the reference and fallback. axisLen >= 1 is guaranteed by the caller.
211-
212-
// SumAxis reduces the middle axis by summation.
213-
func SumAxis(dst, src []float64, outer, axisLen, inner int) {
203+
// reduces the middle axis for the inner columns [lo, hi) of every outer slab,
204+
// writing those columns of dst (laid out as [outer][inner]). A column band is
205+
// contiguous within each row, so RunAxisP can hand disjoint bands to different
206+
// workers when there are too few outer slabs to split (an axis-0 reduction has
207+
// one). The full reduction is the band [0, inner). axisLen >= 1 is guaranteed
208+
// by the caller.
209+
210+
// rowRun is the band width from which a row combine goes through the SIMD
211+
// elementwise kernel; below it the call costs more than the loop.
212+
const rowRun = 16
213+
214+
// SumAxis reduces the middle axis by summation. Each row of the band is added
215+
// into dst with the elementwise SIMD add, element by element in axis order, so
216+
// the result is the sequential sum. With inner == 1 every output is the sum of
217+
// one contiguous row, taken with the lane-parallel SIMD sum (a regrouping, as
218+
// numpy's pairwise sum is): a sequential sum there is latency-bound.
219+
func SumAxis(dst, src []float64, outer, axisLen, inner, lo, hi int) {
220+
if inner == 1 {
221+
for o := 0; o < outer; o++ {
222+
dst[o] = sumSIMD(src[o*axisLen : o*axisLen+axisLen])
223+
}
224+
return
225+
}
214226
for o := 0; o < outer; o++ {
215-
base := o * inner
227+
d := dst[o*inner+lo : o*inner+hi]
216228
block := o * axisLen * inner
217-
for i := 0; i < inner; i++ {
218-
dst[base+i] = src[block+i]
219-
}
229+
copy(d, src[block+lo:block+hi])
220230
for k := 1; k < axisLen; k++ {
221-
off := block + k*inner
222-
for i := 0; i < inner; i++ {
223-
dst[base+i] += src[off+i]
231+
r := src[block+k*inner+lo : block+k*inner+hi]
232+
if len(d) >= rowRun {
233+
addBin(d, d, r)
234+
continue
235+
}
236+
for i := range d {
237+
d[i] += r[i]
224238
}
225239
}
226240
}
227241
}
228242

229-
// ProdAxis reduces the middle axis by multiplication.
230-
func ProdAxis(dst, src []float64, outer, axisLen, inner int) {
243+
// ProdAxis reduces the middle axis by multiplication, in axis order.
244+
func ProdAxis(dst, src []float64, outer, axisLen, inner, lo, hi int) {
231245
for o := 0; o < outer; o++ {
232-
base := o * inner
246+
d := dst[o*inner+lo : o*inner+hi]
233247
block := o * axisLen * inner
234-
for i := 0; i < inner; i++ {
235-
dst[base+i] = src[block+i]
236-
}
248+
copy(d, src[block+lo:block+hi])
237249
for k := 1; k < axisLen; k++ {
238-
off := block + k*inner
239-
for i := 0; i < inner; i++ {
240-
dst[base+i] *= src[off+i]
250+
r := src[block+k*inner+lo : block+k*inner+hi]
251+
if len(d) >= rowRun {
252+
mulBin(d, d, r)
253+
continue
254+
}
255+
for i := range d {
256+
d[i] *= r[i]
241257
}
242258
}
243259
}
244260
}
245261

246262
// MaxAxis reduces the middle axis by taking the maximum. Like Max it is
247-
// NaN-propagating (numpy.max(axis=...)).
248-
func MaxAxis(dst, src []float64, outer, axisLen, inner int) {
263+
// NaN-propagating (numpy.max(axis=...)); max is exact and associative, so the
264+
// inner == 1 row reduction through the SIMD max kernel is the same value.
265+
func MaxAxis(dst, src []float64, outer, axisLen, inner, lo, hi int) {
266+
if inner == 1 {
267+
for o := 0; o < outer; o++ {
268+
dst[o] = maxSIMD(src[o*axisLen : o*axisLen+axisLen])
269+
}
270+
return
271+
}
249272
for o := 0; o < outer; o++ {
250-
base := o * inner
273+
d := dst[o*inner+lo : o*inner+hi]
251274
block := o * axisLen * inner
252-
for i := 0; i < inner; i++ {
253-
dst[base+i] = src[block+i]
254-
}
275+
copy(d, src[block+lo:block+hi])
255276
for k := 1; k < axisLen; k++ {
256-
off := block + k*inner
257-
for i := 0; i < inner; i++ {
258-
dst[base+i] = max(dst[base+i], src[off+i])
277+
r := src[block+k*inner+lo : block+k*inner+hi]
278+
for i := range d {
279+
d[i] = max(d[i], r[i])
259280
}
260281
}
261282
}
262283
}
263284

264285
// MinAxis reduces the middle axis by taking the minimum, NaN-propagating
265286
// (numpy.min(axis=...)).
266-
func MinAxis(dst, src []float64, outer, axisLen, inner int) {
287+
func MinAxis(dst, src []float64, outer, axisLen, inner, lo, hi int) {
288+
if inner == 1 {
289+
for o := 0; o < outer; o++ {
290+
dst[o] = minSIMD(src[o*axisLen : o*axisLen+axisLen])
291+
}
292+
return
293+
}
267294
for o := 0; o < outer; o++ {
268-
base := o * inner
295+
d := dst[o*inner+lo : o*inner+hi]
269296
block := o * axisLen * inner
270-
for i := 0; i < inner; i++ {
271-
dst[base+i] = src[block+i]
272-
}
297+
copy(d, src[block+lo:block+hi])
273298
for k := 1; k < axisLen; k++ {
274-
off := block + k*inner
275-
for i := 0; i < inner; i++ {
276-
dst[base+i] = min(dst[base+i], src[off+i])
299+
r := src[block+k*inner+lo : block+k*inner+hi]
300+
for i := range d {
301+
d[i] = min(d[i], r[i])
277302
}
278303
}
279304
}
@@ -319,11 +344,11 @@ func ArgMin(a []float64) int {
319344
// ArgMaxAxis writes into dst the index (along the middle axis) of the first
320345
// maximum for each [outer][inner] position, or of the first NaN (see ArgMax).
321346
// Layout matches the *Axis kernels.
322-
func ArgMaxAxis(dst []float64, src []float64, outer, axisLen, inner int) {
347+
func ArgMaxAxis(dst []float64, src []float64, outer, axisLen, inner, lo, hi int) {
323348
for o := 0; o < outer; o++ {
324349
base := o * inner
325350
block := o * axisLen * inner
326-
for i := 0; i < inner; i++ {
351+
for i := lo; i < hi; i++ {
327352
best := src[block+i]
328353
bi := 0
329354
for k := 1; k < axisLen && !math.IsNaN(best); k++ {
@@ -338,11 +363,11 @@ func ArgMaxAxis(dst []float64, src []float64, outer, axisLen, inner int) {
338363

339364
// ArgMinAxis writes into dst the index (along the middle axis) of the first
340365
// minimum for each [outer][inner] position, or of the first NaN.
341-
func ArgMinAxis(dst []float64, src []float64, outer, axisLen, inner int) {
366+
func ArgMinAxis(dst []float64, src []float64, outer, axisLen, inner, lo, hi int) {
342367
for o := 0; o < outer; o++ {
343368
base := o * inner
344369
block := o * axisLen * inner
345-
for i := 0; i < inner; i++ {
370+
for i := lo; i < hi; i++ {
346371
best := src[block+i]
347372
bi := 0
348373
for k := 1; k < axisLen && !math.IsNaN(best); k++ {

‎internal/kernels/kernels_test.go‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -68,33 +68,33 @@ func TestAxisKernels(t *testing.T) {
6868
src := []float64{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}
6969
dst := make([]float64, 2*2)
7070

71-
SumAxis(dst, src, 2, 3, 2)
71+
SumAxis(dst, src, 2, 3, 2, 0, 2)
7272
eqSlice(t, dst, []float64{9, 12, 27, 30})
7373

74-
ProdAxis(dst, src, 2, 3, 2)
74+
ProdAxis(dst, src, 2, 3, 2, 0, 2)
7575
eqSlice(t, dst, []float64{15, 48, 693, 960})
7676

77-
MaxAxis(dst, src, 2, 3, 2)
77+
MaxAxis(dst, src, 2, 3, 2, 0, 2)
7878
eqSlice(t, dst, []float64{5, 6, 11, 12})
7979

80-
MinAxis(dst, src, 2, 3, 2)
80+
MinAxis(dst, src, 2, 3, 2, 0, 2)
8181
eqSlice(t, dst, []float64{1, 2, 7, 8})
8282
// Descending data so MinAxis takes its "found a smaller value" branch.
83-
MinAxis(dst, []float64{6, 5, 4, 3, 2, 1, 12, 11, 10, 9, 8, 7}, 2, 3, 2)
83+
MinAxis(dst, []float64{6, 5, 4, 3, 2, 1, 12, 11, 10, 9, 8, 7}, 2, 3, 2, 0, 2)
8484
eqSlice(t, dst, []float64{2, 1, 8, 7})
8585
// Descending data for MaxAxis exercises its non-update path too.
86-
MaxAxis(dst, []float64{6, 5, 4, 3, 2, 1, 12, 11, 10, 9, 8, 7}, 2, 3, 2)
86+
MaxAxis(dst, []float64{6, 5, 4, 3, 2, 1, 12, 11, 10, 9, 8, 7}, 2, 3, 2, 0, 2)
8787
eqSlice(t, dst, []float64{6, 5, 12, 11})
8888

8989
// axisLen == 1: the reduce loop body never runs; result is the input.
9090
one := make([]float64, 2)
91-
SumAxis(one, []float64{4, 9}, 2, 1, 1)
91+
SumAxis(one, []float64{4, 9}, 2, 1, 1, 0, 1)
9292
eqSlice(t, one, []float64{4, 9})
93-
ProdAxis(one, []float64{4, 9}, 2, 1, 1)
93+
ProdAxis(one, []float64{4, 9}, 2, 1, 1, 0, 1)
9494
eqSlice(t, one, []float64{4, 9})
95-
MaxAxis(one, []float64{4, 9}, 2, 1, 1)
95+
MaxAxis(one, []float64{4, 9}, 2, 1, 1, 0, 1)
9696
eqSlice(t, one, []float64{4, 9})
97-
MinAxis(one, []float64{4, 9}, 2, 1, 1)
97+
MinAxis(one, []float64{4, 9}, 2, 1, 1, 0, 1)
9898
eqSlice(t, one, []float64{4, 9})
9999
}
100100

@@ -108,7 +108,7 @@ func BenchmarkSumAxis(b *testing.B) {
108108
b.ReportAllocs()
109109
b.SetBytes(int64(len(src) * 8))
110110
for i := 0; i < b.N; i++ {
111-
SumAxis(dst, src, outer, axisLen, inner)
111+
SumAxis(dst, src, outer, axisLen, inner, 0, inner)
112112
}
113113
}
114114

@@ -133,9 +133,9 @@ func TestArgAndScan(t *testing.T) {
133133
// Axis variants over [outer=1][axisLen=3][inner=2]: columns {3,1},{9,5} etc.
134134
src := []float64{3, 1, 9, 5, 0, 7}
135135
dst := make([]float64, 2)
136-
ArgMaxAxis(dst, src, 1, 3, 2)
136+
ArgMaxAxis(dst, src, 1, 3, 2, 0, 2)
137137
eqSlice(t, dst, []float64{1, 2}) // col0 max at k=1 (9); col1 max at k=2 (7)
138-
ArgMinAxis(dst, src, 1, 3, 2)
138+
ArgMinAxis(dst, src, 1, 3, 2, 0, 2)
139139
eqSlice(t, dst, []float64{2, 0}) // col0 min at k=2 (0); col1 min at k=0 (1)
140140

141141
cs := make([]float64, 6)

‎internal/kernels/parallel.go‎

Lines changed: 40 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -180,45 +180,52 @@ func reduceP(a []float64, red func([]float64) float64) float64 {
180180

181181
// axisKernel is the shape of the [outer][axisLen][inner] axis reducers
182182
// (SumAxis, MaxAxis, …).
183-
type axisKernel func(dst, src []float64, outer, axisLen, inner int)
184-
185-
// RunAxisP runs an axis-reduction kernel, fanning the `outer` slabs across cores
186-
// above ParThreshold (measured by the total element count outer*axisLen*inner).
187-
// Each worker owns a disjoint band of outer rows and writes the matching disjoint
188-
// [band*inner] region of dst, reading only its own src slab — so the parallel
189-
// result is identical to the serial kernel. Splitting `outer` keeps every
190-
// worker's inner traversal contiguous (the cache-friendly axis), which is the
191-
// case that matters: an axis-1 reduction of an (R x C) matrix has outer=R,
192-
// inner=1, and the serial kernel leaves all but one core idle.
183+
type axisKernel func(dst, src []float64, outer, axisLen, inner, lo, hi int)
184+
185+
// RunAxisP runs an axis-reduction kernel across cores above ParThreshold
186+
// (measured by the total element count outer*axisLen*inner). With at least one
187+
// outer slab per worker it splits the slabs; with fewer (an axis-0 reduction
188+
// has one slab) it splits the inner columns into bands of at least axisBand,
189+
// which the old driver ran serially. Either way each worker writes disjoint
190+
// elements of dst, each computed exactly as the serial kernel would, so the
191+
// parallel result is identical to it.
193192
func RunAxisP(k axisKernel, dst, src []float64, outer, axisLen, inner int) {
194193
total := outer * axisLen * inner
195-
if total < ParThreshold {
196-
k(dst, src, outer, axisLen, inner)
197-
return
198-
}
199-
// Split the outer slabs: each worker owns a disjoint band of outer rows and
200-
// writes the matching disjoint dst rows, reading only its own contiguous src
201-
// slab — identical to the serial kernel. This is the case that matters most:
202-
// an axis-1 reduction of an (R x C) matrix has outer=R, inner=1, so the serial
203-
// kernel leaves all but one core idle, and an inner-split would need a strided
204-
// column gather whose copy cost cancels the parallel gain. A single-outer-slab
205-
// reduction (axis 0: outer=1, inner=C) already streams contiguous inner runs
206-
// per accumulation step and stays on the (fast) serial path.
207-
if outer < 2 {
208-
k(dst, src, outer, axisLen, inner)
209-
return
210-
}
211194
w := numWorkers(total)
212-
if w > outer {
213-
w = outer
195+
switch {
196+
case total < ParThreshold || w < 2:
197+
k(dst, src, outer, axisLen, inner, 0, inner)
198+
case outer >= w:
199+
// Enough outer slabs: each worker owns a band of them and reads only its
200+
// own contiguous src slab. An axis-1 reduction of an (R x C) matrix
201+
// (outer=R, inner=1) is this case.
202+
parallelFor(outer, w, func(lo, hi int) {
203+
k(dst[lo*inner:hi*inner], src[lo*axisLen*inner:hi*axisLen*inner], hi-lo, axisLen, inner, 0, inner)
204+
})
205+
case inner >= 2*axisBand:
206+
// Too few slabs (an axis-0 reduction has one): split the columns. A band
207+
// of columns is a contiguous run of every row, so each worker streams
208+
// its runs and writes its own columns of dst; no gather is needed.
209+
bands := min(w, inner/axisBand)
210+
width := roundUp((inner+bands-1)/bands, 8)
211+
parallelFor(bands, bands, func(b0, b1 int) {
212+
for b := b0; b < b1; b++ {
213+
if lo := b * width; lo < inner {
214+
k(dst, src, outer, axisLen, inner, lo, min(lo+width, inner))
215+
}
216+
}
217+
})
218+
default:
219+
parallelFor(outer, min(w, outer), func(lo, hi int) {
220+
k(dst[lo*inner:hi*inner], src[lo*axisLen*inner:hi*axisLen*inner], hi-lo, axisLen, inner, 0, inner)
221+
})
214222
}
215-
parallelFor(outer, w, func(lo, hi int) {
216-
dstSlab := dst[lo*inner : hi*inner]
217-
srcSlab := src[lo*axisLen*inner : hi*axisLen*inner]
218-
k(dstSlab, srcSlab, hi-lo, axisLen, inner)
219-
})
220223
}
221224

225+
// axisBand is the narrowest column band RunAxisP gives a worker (512 doubles,
226+
// 4 KiB per row): narrower bands make each row run too short to stream.
227+
const axisBand = 512
228+
222229
// MaxP returns the maximum element of a (non-empty), parallelised above
223230
// ParThreshold. It runs the SIMD max kernel (packed MAXPD + NaN scan on amd64;
224231
// the intrinsic-lowered scalar FMAXD reducer on arm64/others) on each chunk and

0 commit comments

Comments
 (0)