Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,36 @@
`MaskSelect` (`a[mask]`), `Nonzero` (`flatnonzero`), and `Take`.
- **Manipulation** — `Concatenate`/`Stack`/`VStack`/`HStack`.
- **Linear algebra** — `MatMul`/`Dot`/`Inner`/`Outer`.
- **Memory reuse** — `Workspace`, an arena for loops: bind the inputs with
`ws.Use`, compute as usual, `ws.Reset()` at the end of each pass. Results stop
costing the garbage collector anything (see below).

### Loops: `Workspace`

Every operation returns a new array, and in a loop those results are garbage
the collector must chase. NumPy frees its temporaries the moment their
reference count drops; Go cannot. On a Zen 3 that made `x + y` on 1 024
elements 3× slower than NumPy, and a chain like `sqrt(x*y + x)` 3.4× slower.
A `Workspace` gives the results of one pass a single arena that `Reset` hands
back:

```go
ws := ndarray.NewWorkspace()
for step := 0; step < steps; step++ {
x := ws.Use(state) // results computed from x come from ws
p, _ := x.Mul(w)
q, _ := p.Add(bias)
state = q.Sqrt().Detach() // the one result that outlives the pass
ws.Reset() // everything else is recycled
}
```

After the first pass nothing is allocated, nothing is zeroed that the kernel
overwrites anyway, and the memory stays warm in cache: `x + y` on 1 024
elements goes from 2.6 µs to 0.38 µs (2.1× NumPy), `sqrt(x*y + x)` on 256 Ki
from 1.18 ms to 0.39 ms (7.5× NumPy). ⚠ An array allocated from the workspace
is invalid after `Reset`; `Detach` is how a result leaves it. The `*Into`
methods (`AddInto`, `SqrtInto`, …) remain the way to reuse one named buffer.

The numeric inner loops are kept behind a narrow kernel API. Behind it,
**large** elementwise ops, reductions and matmul run **multicore** (across
Expand Down
1 change: 1 addition & 0 deletions benchmarks/bench_numpy.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ def main():
("AddInto", "np.add(x, y, out=z)"),
("MulInto", "np.multiply(x, y, out=z)"),
("SqrtInto", "np.sqrt(x, out=z)"),
("Chain", "np.sqrt(x * y + x)"),
("Sum", "x.sum()"),
("Mean", "x.mean()"),
("Max", "x.max()"),
Expand Down
51 changes: 51 additions & 0 deletions benchmarks/bench_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,57 @@ func BenchmarkAddInto(b *testing.B) {
}
}

// BenchmarkAddWS is Add with its operands bound to a Workspace that is Reset
// every iteration: the allocating form, minus the garbage collector and the
// zeroing (see Workspace). Compare with BenchmarkAdd and BenchmarkAddInto.
func BenchmarkAddWS(b *testing.B) {
for _, n := range elemSizes {
ws := nd.NewWorkspace()
x, y := ws.Use(vec(n)), vec(n)
b.Run(fmt.Sprintf("n=%d", n), func(b *testing.B) {
b.SetBytes(int64(n * 8))
for i := 0; i < b.N; i++ {
_, _ = x.Add(y)
ws.Reset()
}
})
}
}

// BenchmarkChain is sqrt(x*y + x) written the plain way (three allocating
// operations, two temporaries), the shape of real array code; numpy's row
// "Chain" is np.sqrt(x * y + x).
func BenchmarkChain(b *testing.B) {
for _, n := range elemSizes {
x, y := vec(n), vec(n)
b.Run(fmt.Sprintf("n=%d", n), func(b *testing.B) {
b.SetBytes(int64(n * 8))
for i := 0; i < b.N; i++ {
p, _ := x.Mul(y)
q, _ := p.Add(x)
_ = q.Sqrt()
}
})
}
}

// BenchmarkChainWS is BenchmarkChain in a Workspace reset every iteration.
func BenchmarkChainWS(b *testing.B) {
for _, n := range elemSizes {
ws := nd.NewWorkspace()
x, y := ws.Use(vec(n)), vec(n)
b.Run(fmt.Sprintf("n=%d", n), func(b *testing.B) {
b.SetBytes(int64(n * 8))
for i := 0; i < b.N; i++ {
p, _ := x.Mul(y)
q, _ := p.Add(x)
_ = q.Sqrt()
ws.Reset()
}
})
}
}

func BenchmarkMulInto(b *testing.B) {
for _, n := range elemSizes {
x, y := vec(n), vec(n)
Expand Down
47 changes: 46 additions & 1 deletion docs/perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ with no result array, so they have no separate *into* row.)
path is the supported way to hit/beat NumPy there (and at large n even the alloc
form wins, because the kernel time then dwarfs the one allocation). (Sum and Max are
the exceptions — their reduction kernels need no result allocation and win even
serially.)
serially.) **Superseded for loops by `Workspace`, see below.**

### NaN convention for Max / Min (and the SIMD max kernel)

Expand Down Expand Up @@ -313,6 +313,51 @@ The SIMD kernels are also validated against the scalar oracle per-arch in CI:
| `MatMul` vs tuned BLAS (OpenBLAS) | **FIXED — parity at n=1024 (~0.99×, ~203 GFLOP/s); 0.97× at n=512** | by-lane FMLA micro-kernel (`FMLA Vd.2D,Vn.2D,Vm.D[i]` via `WORD`) closed the prior 0.76× gap. Only small n=256 trails (0.67×) on per-call overhead, not throughput — see above |
| other `Map` ufuncs (`Exp`, `Log`, `Sin`…) | NumPy ~parity (libm-bound) | a packed `VEXP`/`VLOG` is libm-accuracy work; the math, not the dispatch, dominates here |

## Allocation: `Workspace` (2026-10-04)

On real amd64 hardware the allocating forms were far worse than on the M4: on
a Zen 3 (16 cores), `x + y` lost to NumPy 0.13–0.25× at every size up to 256 Ki
elements, while `AddInto` won. Slicing copies, `Concatenate` and broadcasting
adds (whose results also allocate) lost 5–9×. Measured causes, on M4:

- **The garbage collector.** A benchmark's live heap is tiny, so a 128 KiB
result per call triggered a GC cycle every ~22 calls (912 cycles in 20 000).
With 256 MiB–2 GiB of live heap, which is closer to a real program, the
overhead halves: 8.3 → 3.6 µs at 16 Ki, against 2.4 µs for `AddInto`.
- **The rest:** `make` zeroes memory the kernel overwrites anyway, and fresh
memory is cold or page-faulted. `GOGC=off` does not help: it trades the
cycles for page faults (9.4 µs).

A library must not set `GOGC`, and Go has no unzeroed allocation. What NumPy's
reference counting gets for free is scoped reuse. ND4J, the Java ndarray
library, has the same problem and answers it with *workspaces*: an arena that
results of one pass come from, recycled at the end of the pass. `Workspace` is
that. `ws.Use(x)` binds an input, every result computed from a bound array is
carved from the arena (64-byte aligned, capped so an `append` cannot spill),
results stay bound so a chain stays in it, and `Reset` recycles everything,
folding a pass that needed several blocks into one block. Sites whose kernel
writes every element (elementwise, ufuncs, copies, broadcasts, concatenation,
`Where`, `Clip`, scans, `Take`, `Outer`, mat·vec) skip the zeroing. Those that
accumulate (GEMM, vec·mat, axis reductions) clear their slot.

Zen 3, 16 cores pinned, NumPy 2.5.3 single-threaded, median of two runs:

| op | n | heap | **Workspace** | `*Into` | NumPy | Workspace vs NumPy |
|----|--:|--:|--:|--:|--:|:--:|
| Add | 1 024 | 2.6 µs | **0.38 µs** | 0.31 µs | 0.82 µs | 2.1× |
| Add | 16 Ki | 35 µs | **4.6 µs** | 4.3 µs | 5.1 µs | 1.1× |
| Add | 256 Ki | 498 µs | **67 µs** | 75 µs | 150 µs | 2.2× |
| Add | 4 Mi | 8.0 ms | **2.2 ms** | 2.2 ms | 6.3 ms | 2.9× |
| sqrt(x·y+x) | 1 024 | 12.2 µs | **2.6 µs** | — | 3.6 µs | 1.4× |
| sqrt(x·y+x) | 16 Ki | 112 µs | **33 µs** | — | 34 µs | 1.0× |
| sqrt(x·y+x) | 256 Ki | 1.18 ms | **0.39 ms** | — | 2.97 ms | 7.5× |
| sqrt(x·y+x) | 4 Mi | 23.4 ms | **9.1 ms** | — | 19.2 ms | 2.1× |

On M4 the same chain gains ×5 at 1 024 and ×1.6 at 4 Mi. Correctness: every
operation is checked bit for bit against its heap result, including on passes
whose recycled memory was first filled with NaN (a site that skipped zeroing it
needed would leak the NaN; planting that bug in the GEMM made the test fail).

## amd64 on real hardware (2026-10-04)

Every figure above was taken on Apple silicon. This round measured amd64 for
Expand Down
8 changes: 4 additions & 4 deletions indexing.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ func (a *Array) MaskSelect(mask *Array) (*Array, error) {
out = append(out, av[i])
}
}
return &Array{data: out, shape: []int{len(out)}, strides: []int{1}}, nil
return &Array{data: out, shape: []int{len(out)}, strides: []int{1}, ws: a.ws}, nil
}

// Nonzero returns a 1-D array of the flat (row-major) indices at which a is
Expand All @@ -46,7 +46,7 @@ func (a *Array) Nonzero() *Array {
out = append(out, float64(i))
}
}
return &Array{data: out, shape: []int{len(out)}, strides: []int{1}}
return &Array{data: out, shape: []int{len(out)}, strides: []int{1}, ws: a.ws}
}

// Take returns a 1-D array gathering a's flattened (row-major) elements at the
Expand All @@ -55,7 +55,7 @@ func (a *Array) Nonzero() *Array {
func (a *Array) Take(indices ...int) (*Array, error) {
flat := a.contiguousData()
n := len(flat)
out := make([]float64, len(indices))
out := a.alloc(len(indices), false)
for k, idx := range indices {
j := idx
if j < 0 {
Expand All @@ -67,5 +67,5 @@ func (a *Array) Take(indices ...int) (*Array, error) {
}
out[k] = flat[j]
}
return &Array{data: out, shape: []int{len(out)}, strides: []int{1}}, nil
return &Array{data: out, shape: []int{len(out)}, strides: []int{1}, ws: a.ws}, nil
}
18 changes: 9 additions & 9 deletions linalg.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ var ErrLinalg = fmt.Errorf("ndarray: incompatible shapes for linear algebra")
// array. Either operand may be any 2-D view: the GEMM packs it through its
// strides (matView), so nothing is materialised first.
func matmul2D(a, b *Array, m, k, n int) *Array {
dst := make([]float64, m*n)
dst := a.ws.alloc(m*n, true) // MatMulStridedP accumulates into dst
kernels.MatMulStridedP(dst, a.matView(), b.matView(), m, k, n)
shape := []int{m, n}
return &Array{data: dst, shape: shape, strides: rowMajorStrides(shape)}
return &Array{data: dst, shape: shape, strides: rowMajorStrides(shape), ws: a.ws}
}

// matView is a 2-D array as the GEMM's strided operand view. The packing
Expand Down Expand Up @@ -60,7 +60,7 @@ func (a *Array) Dot(b *Array) (*Array, error) {
ErrLinalg, a.shape[0], b.shape[0])
}
s := kernels.Dot1DP(a.contiguousData(), b.contiguousData())
return &Array{data: []float64{s}, shape: []int{}, strides: []int{}}, nil
return &Array{data: []float64{s}, shape: []int{}, strides: []int{}, ws: a.ws}, nil

case na == 2 && nb == 2:
return a.MatMul(b)
Expand All @@ -73,9 +73,9 @@ func (a *Array) Dot(b *Array) (*Array, error) {
ErrLinalg, a.shape, b.shape)
}
m, k := a.shape[0], a.shape[1]
dst := make([]float64, m)
dst := a.alloc(m, false)
kernels.MatVecP(dst, a.contiguousData(), b.contiguousData(), m, k)
return &Array{data: dst, shape: []int{m}, strides: []int{1}}, nil
return &Array{data: dst, shape: []int{m}, strides: []int{1}, ws: a.ws}, nil

case na == 1 && nb == 2:
// (k,) · (k x n) = (n,): cache-friendly row-streaming accumulation, no
Expand All @@ -85,9 +85,9 @@ func (a *Array) Dot(b *Array) (*Array, error) {
ErrLinalg, a.shape, b.shape)
}
k, n := b.shape[0], b.shape[1]
dst := make([]float64, n)
dst := a.alloc(n, true) // VecMatP accumulates into dst
kernels.VecMatP(dst, a.contiguousData(), b.contiguousData(), k, n)
return &Array{data: dst, shape: []int{n}, strides: []int{1}}, nil
return &Array{data: dst, shape: []int{n}, strides: []int{1}, ws: a.ws}, nil

default:
return nil, fmt.Errorf("%w: Dot supports 1-D and 2-D operands, got %d-D and %d-D",
Expand Down Expand Up @@ -126,7 +126,7 @@ func (a *Array) Outer(b *Array) *Array {
u := a.contiguousData()
v := b.contiguousData()
m, n := len(u), len(v)
dst := make([]float64, m*n)
dst := a.alloc(m*n, false)
for i := 0; i < m; i++ {
ui := u[i]
row := dst[i*n : i*n+n]
Expand All @@ -135,5 +135,5 @@ func (a *Array) Outer(b *Array) *Array {
}
}
shape := []int{m, n}
return &Array{data: dst, shape: shape, strides: rowMajorStrides(shape)}
return &Array{data: dst, shape: shape, strides: rowMajorStrides(shape), ws: a.ws}
}
11 changes: 6 additions & 5 deletions manipulation.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import "fmt"
// always-copy contract).
func (a *Array) Flatten() *Array {
data := a.materialize()
return &Array{data: data, shape: []int{len(data)}, strides: []int{1}}
return &Array{data: data, shape: []int{len(data)}, strides: []int{1}, ws: a.ws}
}

// ExpandDims returns a view with a new length-1 axis inserted at the given
Expand All @@ -31,7 +31,7 @@ func (a *Array) ExpandDims(axis int) (*Array, error) {
// The inserted axis has length 1; its stride is irrelevant, use 0.
strides = append(strides, 0)
strides = append(strides, a.strides[axis:]...)
return &Array{data: a.data, shape: shape, strides: strides, offset: a.offset}, nil
return &Array{data: a.data, shape: shape, strides: strides, offset: a.offset, ws: a.ws}, nil
}

// Squeeze returns a view with length-1 axes removed. With no axes given, every
Expand Down Expand Up @@ -70,7 +70,7 @@ func (a *Array) Squeeze(axes ...int) (*Array, error) {
strides = append(strides, a.strides[i])
}
}
return &Array{data: a.data, shape: shape, strides: strides, offset: a.offset}, nil
return &Array{data: a.data, shape: shape, strides: strides, offset: a.offset, ws: a.ws}, nil
}

// Concatenate joins the given arrays along an existing axis, matching
Expand Down Expand Up @@ -115,7 +115,7 @@ func concatInto(arrays []*Array, ax int, out []int) *Array {
inner := prod(out[ax+1:])
outer := prod(out[:ax])
axisTotal := out[ax]
data := make([]float64, outer*axisTotal*inner)
data := wsOf(arrays...).alloc(outer*axisTotal*inner, false) // every slab is copied in
// Column offset (in the joined axis) where the next array's slab begins.
colBase := 0
for _, arr := range arrays {
Expand All @@ -131,7 +131,7 @@ func concatInto(arrays []*Array, ax int, out []int) *Array {
}
colBase += aLen
}
return &Array{data: data, shape: out, strides: rowMajorStrides(out)}
return &Array{data: data, shape: out, strides: rowMajorStrides(out), ws: wsOf(arrays...)}
}

// Stack joins the given arrays along a new axis, matching numpy.stack. Every
Expand Down Expand Up @@ -191,6 +191,7 @@ func atLeast2D(a *Array) *Array {
shape: []int{1, a.shape[0]},
strides: []int{a.shape[0] * a.strides[0], a.strides[0]},
offset: a.offset,
ws: a.ws,
}
}
return a
Expand Down
Loading
Loading