diff --git a/README.md b/README.md index d18e977..13fe942 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,8 @@ The numeric inner loops are kept behind a narrow kernel API. Behind it, **large** elementwise ops, reductions and matmul run **multicore** (across `GOMAXPROCS`); the sum reduction uses a **go-asmgen SIMD kernel** (NEON on arm64, SSE2 on amd64); and `MatMul` is a **panel-packed, cache-blocked GEMM** -with a go-asmgen SIMD-FMA micro-kernel (NEON 4×8 on arm64, SSE2 4×4 on amd64) — +with a go-asmgen SIMD-FMA micro-kernel (NEON 4×8 on arm64; AVX2/FMA 6×8 on +amd64, chosen at run time, with an SSE2 fallback) — the OpenBLAS/BLIS structure. So go-ndarray **beats single-threaded NumPy** on its core ops on large arrays (Add/Mul ~2×, Sum ~2.4×), and `MatMul` reaches **tuned-BLAS parity at 1024²**: ≈0.99× vs multi-threaded OpenBLAS 0.3.29 on an diff --git a/docs/perf.md b/docs/perf.md index bb8bb10..600dcce 100644 --- a/docs/perf.md +++ b/docs/perf.md @@ -212,22 +212,25 @@ round implements the full OpenBLAS/BLIS structure and ships it for **all** sizes The micro-kernel then streams conflict-free memory **regardless of the source stride** — this is the precise fix for the power-of-two L1 set-conflicts that defeated the unpacked attempt. Edge tiles are zero-padded so the kernel always - sees a full MR×NR block; the ragged right/bottom borders take a scalar path. + sees a full MR×NR block; a ragged right/bottom tile runs the same micro-kernel + into a scratch tile, and only its valid corner is added to C. - **Cache blocking** — an `NC` (columns) → `KC` (contraction) → `MC` (rows) loop nest keeps the packed B panel L2-resident and each packed A panel in L1. Defaults `MC=256, KC=256, NC=512` (tuned on this VM). - **SIMD-FMA micro-kernel** (go-asmgen) — **NEON 4×8 on arm64** (16 D2 - accumulators, the source of the win), **SSE2 4×4 on amd64** (8 XMM - accumulators; SSE2 has no FMA, so explicit MULPD+ADDPD), **scalar 4×4** on the + accumulators, the source of the win), **AVX2/FMA 6×8 on amd64** (12 YMM + accumulators, picked at run time by a CPUID probe; an SSE2 6×4 pair without + FMA is the fallback), **scalar 4×4** on the four arches without vector-double asm — which still gain from contiguous packed data + blocking + multicore. - **Parallelism** — the M rows are split into MR-aligned bands, one per core, each writing a disjoint region of C. -The arithmetic is the standard ikj order `dst[i][j] += a[i][p]·b[p][j]`; packing -only relocates the operands, so the result is **bit-for-bit identical** to the -scalar oracle and to NumPy's `A@B` (max abs diff **0.0**, verified at 128×128 -against OpenBLAS). +The arithmetic is the ikj order `dst[i][j] += a[i][p]·b[p][j]`, summed per KC +block; for k ≤ KC that is the scalar oracle's order exactly, and above it the +block partials regroup the sum (a valid reordering, as in every blocked BLAS). +At 128×128 the result matched NumPy's `A@B` against OpenBLAS with max abs diff +**0.0**. ### Packed GEMM vs the PRIOR kernel (why it ships) @@ -261,8 +264,8 @@ there the gap is *fixed per-call overhead*, not compute: This is a small-matrix overhead ceiling. Closing it further would mean a serial-vs-parallel crossover tuned per size and a cheaper pack for tiny panels; it is not a Go-assembler or algorithm limit, and it does not affect the -large-matrix parity result. (`GOAMD64=v3`/AVX2-FMA would likewise lift the amd64 -micro-kernel, which today is SSE2-only; the arm64 path already has its FMA.) +large-matrix parity result. (amd64 now has its AVX2/FMA kernel too; see the +real-hardware section below.) ### Footnote: vs reference BLAS @@ -310,15 +313,70 @@ The SIMD kernels are also validated against the scalar oracle per-arch in CI: | `MatMul` vs tuned BLAS (OpenBLAS) | **FIXED — parity at n=1024 (~0.99×, ~203 GFLOP/s); 0.97× at n=512** | by-lane FMLA micro-kernel (`FMLA Vd.2D,Vn.2D,Vm.D[i]` via `WORD`) closed the prior 0.76× gap. Only small n=256 trails (0.67×) on per-call overhead, not throughput — see above | | other `Map` ufuncs (`Exp`, `Log`, `Sin`…) | NumPy ~parity (libm-bound) | a packed `VEXP`/`VLOG` is libm-accuracy work; the math, not the dispatch, dominates here | +## amd64 on real hardware (2026-10-04) + +Every figure above was taken on Apple silicon. This round measured amd64 for +the first time, on an AMD EPYC 7773X (Zen 3, GCC Compile Farm cfarm421), with +go-ndarray and NumPy both pinned to the same 16 cores of one NUMA node +(`taskset -c 16-31`; Go's GOMAXPROCS follows the affinity mask). NumPy 2.5.3 +with its bundled OpenBLAS 0.3.34 (DYNAMIC_ARCH, Haswell kernels on Zen 3), +once with `OPENBLAS_NUM_THREADS=1` and once with 16. go rows are the minimum +of three runs interleaved with the previous version, so drift on a shared host +hits both alike. + +| op | before | after | gain | vs OpenBLAS 1 thread | vs OpenBLAS 16 threads | +|----|------:|------:|:--:|:--:|:--:| +| MatMul 1024² | 21.87 ms | **8.05 ms** | ×2.72 | 5.93× | 0.60× | +| MatMul 512² | 3.69 ms | 1.56 ms | ×2.36 | 3.89× | 0.66× | +| MatMul 256² | 0.661 ms | 0.558 ms | ×1.19 | 1.36× | 0.30× | +| MatMul 128² | 0.210 ms | 0.141 ms | ×1.49 | 0.73× | 0.36× | +| MatMul 1024×256·256×1024 | 5.72 ms | 2.71 ms | ×2.11 | 4.55× | 0.54× | +| Inner 512² (a·bᵀ) | 5.87 ms | **1.47 ms** | ×4.00 | 4.19× | **0.96×** | + +Four changes, each measured on its own before the next: + +1. **AVX2/FMA 6×8 micro-kernel** (the BLIS Haswell shape). In L1 it runs at + 43 GFLOP/s on one Cascade Lake core, the FMA ports' peak at that clock; the + SSE2 4×4 it replaces had no FMA and half the vector width. +2. **Edge tiles through the micro-kernel.** With MR=6, every 256-row MC block + ends in a 4-row edge panel, and the scalar edge loop took **40%** of a 512² + product (profiled). A partial tile now runs the kernel into a scratch tile. +3. **Prefetch** of the C tile at kernel entry and of the A panel eight steps + ahead: serial 768² went from 27.5 to 34.7 GFLOP/s (+26%); 90% of the time was + already in the kernel, stalled on memory. A sweep of MC (48–252) and KC + (128–512) moved nothing (±3%), so the blocking is unchanged. +4. **B packed by all workers.** MatMulP packs the shared B block once per + (jc, pc) round, and one goroutine did it while the others waited: +30–70% + in parallel. Below 64 Ki elements the launch costs more than the copy (M4: + −8–15% at 96³–128²), so small blocks are still packed by one. + +`Inner` gained the most because it used to materialise bᵀ before multiplying. +The GEMM now reads any 2-D view through its strides while packing (the copy it +makes anyway), so a transposed, sliced, reversed or broadcast operand costs no +extra pass. + +**Still behind OpenBLAS with 16 threads** (0.54–0.66× on large products, 0.3× +below 256²): per-core throughput in the parallel run is about a third of the +serial kernel's, so the next lever is the parallel schedule (a persistent +worker pool instead of a goroutine launch per round, and the BLIS jr-loop +split for short m), then an AVX-512 kernel for the hosts that have it. + +Correctness was checked on real hardware for five of the six 64-bit targets: +amd64 (Zen 3, FMA path), arm64 (cfarm185), ppc64le (cfarm120), riscv64 (cfarm94) +and loong64 (cfarm401). The s390x host (LinuxONE) did not answer; s390x stays +covered by the qemu lane in CI. + ## SIMD coverage - **amd64 (SSE2)** ships hand-vectorized `sum` (4-accumulator `ADDPD`), `sqrt` (packed `SQRTPD`), `max`/`min` (`MAXPD`/`MINPD` + `CMPPD` NaN scan), the **elementwise `add`/`sub`/`mul`/`div`** (packed `ADD/SUB/MUL/DIVPD`, 8 - doubles/iter + scalar tail), and the **GEMM micro-kernel** (`gemmMicro4x4`: a - 4×4 SSE2 `MULPD`+`ADDPD` tile, no FMA at the v1 baseline), generated by - go-asmgen and validated per-arch in CI (and cross-run under qemu-x86_64 — the - GEMM tile included). + doubles/iter + scalar tail), all at the GOAMD64=v1 baseline, and the **GEMM + micro-kernel**: `gemmMicro6x8FMA` (6×8, 12 YMM accumulators, `VBROADCASTSD` + + `VFMADD231PD`, C and A prefetched) when go-asmgen's CPUID probe reports FMA + with the OS saving YMM state, else a pair of SSE2 6×4 `MULPD`+`ADDPD` tiles + over the same packing. Generated by go-asmgen and validated per-arch in CI; + the FMA kernel on real Cascade Lake and Zen 3 hosts. - **arm64 (NEON)** ships hand-vectorized `sum`, **packed `sqrt`** (`FSQRT V.2D`), the **elementwise `add`/`sub`/`mul`** (via `VFMLA`/`VFMLS` against a `1.0` vector — `add: b+a·1`, `sub: a−b·1`, `mul: 0+a·b`, each FMA exact so @@ -344,5 +402,5 @@ per-arch job regenerates the committed `.s`, fails if it is stale, vets (asmdecl), builds (cmd/asm encodes), and runs the bit/NaN-correctness suite. The multicore path and the packed/cache-blocked GEMM driver are architecture-independent; only the GEMM micro-kernel is per-arch (NEON 4×8 on -arm64, SSE2 4×4 on amd64, scalar 4×4 on the other four — all bit-identical to the +arm64, AVX2/FMA 6×8 or SSE2 on amd64, scalar 4×4 on the other four — all bit-identical to the scalar ikj oracle, validated per-arch in CI incl. amd64 under qemu-x86_64). diff --git a/docs/plan-ndarray.md b/docs/plan-ndarray.md index 91a7895..262ddc5 100644 --- a/docs/plan-ndarray.md +++ b/docs/plan-ndarray.md @@ -114,7 +114,8 @@ suite (plus the per-arch CI jobs already wired in `.github/workflows/ci.yml`). `sum` (4-accumulator), `sqrt` (packed `SQRTPD` / NEON `FSQRT V.2D`), `max`/`min` (`MAXPD`/`MINPD` + NaN scan / four-accumulator builtin), the elementwise `add`/`sub`/`mul`/`div`, and the **GEMM micro-kernel** (NEON 4×8 - by-lane FMLA on arm64, SSE2 4×4 on amd64), generated by go-asmgen and + by-lane FMLA on arm64, AVX2/FMA 6×8 with an SSE2 fallback on amd64), + generated by go-asmgen and validated per-arch in CI (bit-identical to the scalar oracle for the elementwise/sqrt/max ops; tight-tolerance for the sum reduction, whose lane-parallel grouping is a valid reordering — the same trade-off as diff --git a/internal/kernels/asmgen/amd64/gen.go b/internal/kernels/asmgen/amd64/gen.go index 5fe92c6..b0493a1 100644 --- a/internal/kernels/asmgen/amd64/gen.go +++ b/internal/kernels/asmgen/amd64/gen.go @@ -45,7 +45,9 @@ func main() { f.Add(binKernel("subSSE2", "SUBPD", "SUBSD")) f.Add(binKernel("mulSSE2", "MULPD", "MULSD")) f.Add(binKernel("divSSE2", "DIVPD", "DIVSD")) - f.Add(gemmKernel()) + f.Add(gemmFMAKernel()) + f.Add(gemmSSE2HalfKernel()) + f.Add(amd64.FeatureProbe("hasFMA", amd64.FMA)) if err := os.WriteFile("sum_amd64.s", []byte(f.String()), 0o644); err != nil { fmt.Fprintln(os.Stderr, err) @@ -290,97 +292,160 @@ func binKernel(name, vec, sca string) *emit.Function { return b.Func() } -// gemmKernel builds gemmMicro4x4(kc int, pa, pb, c *float64, ldc int): the SSE2 -// 4x4 register-blocked GEMM micro-kernel for the packed GEMM. -// -// It computes a 4-row x 4-col tile of C from packed panels and ADDS it into C: -// -// for p in [0,kc): C[r][col] += pa[p*4+r] * pb[p*4+col] (r<4, col<4) -// -// pa is the packed A panel (MR=4 contiguous A values per k step), pb the packed -// B panel (NR=4 contiguous B values per k step) — both unit-stride, so the loads -// are conflict-free regardless of the source matrices' strides (the point of -// packing: it removes the power-of-two L1 set-conflicts that defeat an unpacked -// register-blocked kernel). -// -// amd64 baseline is SSE2 (GOAMD64=v1) which has NO fused multiply-add (FMA is -// FMA3/AVX2), so each step is an explicit MULPD then ADDPD. SSE2 *does* have a -// packed FP add, so the running accumulation is the ordinary dst += a*b, in the -// same ikj order as the scalar oracle — the tile is bit-identical to it. -// -// Register map: the 4x4 C tile lives in X0..X7 (4 rows x 2 XMM, 2 doubles each); -// the per-step B row in X8,X9 (4 doubles); the broadcast A value in X10; the two -// products in X11,X12. -func gemmKernel() *emit.Function { - sig := amd64.Layout( +// gemmMR, gemmNR are the micro-tile the packing in gemm.go produces on amd64: +// 6 rows of packed A per k step, 8 columns of packed B. +const ( + gemmMR = 6 + gemmNR = 8 +) + +func gemmSig() amd64.Signature { + return amd64.Layout( []string{"kc", "pa", "pb", "c", "ldc"}, []amd64.Type{amd64.Int64, amd64.Ptr, amd64.Ptr, amd64.Ptr, amd64.Int64}, nil, nil, ) - b := amd64.NewFunc("gemmMicro4x4", sig, 0) +} + +// gemmFMAKernel builds gemmMicro6x8FMA(kc int, pa, pb, c *float64, ldc int): +// +// for p in [0,kc): C[r][col] += pa[p*6+r] * pb[p*8+col] (r<6, col<8) +// +// the BLIS Haswell dgemm shape (6x8, row-major here). Each k step loads the +// 8-double B row into two YMM, broadcasts each of the 6 A values with +// VBROADCASTSD straight from the panel, and issues 12 VFMADD231PD into the 12 +// accumulators Y0..Y11 (row r in Y(2r), Y(2r+1)). Twelve independent chains +// cover the FMA latency on both ports (Haswell: 5 cycles x 2 ports needs >= 10), +// and 8 loads per 12 FMAs stay under the 2-loads-per-cycle budget, so the loop +// is FMA-throughput bound — the property that makes it a real dgemm kernel. +// +// Fused: each step rounds a*b+c once, as tuned BLAS (and gc's own scalar code at +// GOAMD64=v3, and arm64's) do. The SSE2 fallback below does not fuse, so on +// non-integer data the two can differ in the last place; both are the ikj sum. +// The caller gates this kernel on hasFMA (CPUID FMA+AVX+OSXSAVE and XCR0 YMM). +func gemmFMAKernel() *emit.Function { + b := amd64.NewFunc("gemmMicro6x8FMA", gemmSig(), 0) b.LoadArg("kc", "CX") b.LoadArg("pa", "SI") // A panel b.LoadArg("pb", "DI") // B panel b.LoadArg("c", "DX") // C base b.LoadArg("ldc", "R8") b.Raw("SHLQ $3, R8") // ldc -> bytes - // Zero the eight C accumulators X0..X7. - b.Raw("XORPS X0, X0") - b.Raw("XORPS X1, X1") - b.Raw("XORPS X2, X2") - b.Raw("XORPS X3, X3") - b.Raw("XORPS X4, X4") - b.Raw("XORPS X5, X5") - b.Raw("XORPS X6, X6") - b.Raw("XORPS X7, X7") - b.Raw("TESTQ CX, CX") - b.Raw("JZ gstore") - b.Raw("gkloop:") - // B row: 4 contiguous doubles into X8 (cols 0,1) and X9 (cols 2,3). - b.Raw("MOVUPD (DI), X8") - b.Raw("MOVUPD 16(DI), X9") - b.Raw("ADDQ $32, DI") - // For each of the 4 A values, broadcast and fuse into that row's two XMM. - gemmRow(b, "X0", "X1", 0) - gemmRow(b, "X2", "X3", 8) - gemmRow(b, "X4", "X5", 16) - gemmRow(b, "X6", "X7", 24) - b.Raw("ADDQ $32, SI") // advance A panel by 4 doubles - b.Raw("DECQ CX") - b.Raw("JNZ gkloop") - // C += accumulators, row by row (ordinary packed ADDPD, bit-identical). - b.Raw("gstore:") - gemmStore(b, "X0", "X1", true) // row0 at (DX) - gemmStore(b, "X2", "X3", false) - gemmStore(b, "X4", "X5", false) - gemmStore(b, "X6", "X7", false) + // Prefetch the C tile now: its rows are ldc apart, so they are 6 different + // cache lines (12 when a row straddles two), usually not in L1, and the + // store at the end would otherwise stall on each of them in turn. Issued + // first, they arrive while the k loop runs. The last byte of each row is + // fetched too, for a row that does not start on a line boundary. + b.Raw("MOVQ DX, R9") + for r := 0; r < gemmMR; r++ { + b.Raw("PREFETCHT0 (R9)") + b.Raw("PREFETCHT0 63(R9)") + if r < gemmMR-1 { + b.Raw("ADDQ R8, R9") + } + } + for r := 0; r < 12; r++ { + b.Raw(fmt.Sprintf("VXORPD Y%d, Y%d, Y%d", r, r, r)) + } + // Unrolled by two k steps, with a one-step tail. The A panel streams from + // L2 (the B micro-panel stays in L1 across the whole A block), so each pair + // of steps prefetches the A line 8 steps ahead. + b.Raw("MOVQ CX, R10") + b.Raw("SHRQ $1, R10") + b.Raw("JZ ftail") + b.Raw("fkloop:") + b.Raw(fmt.Sprintf("PREFETCHT0 %d(SI)", 8*gemmMR*8)) + fmaStep(b, 0) + fmaStep(b, 1) + b.Raw(fmt.Sprintf("ADDQ $%d, SI", 2*8*gemmMR)) + b.Raw(fmt.Sprintf("ADDQ $%d, DI", 2*8*gemmNR)) + b.Raw("DECQ R10") + b.Raw("JNZ fkloop") + b.Raw("ftail:") + b.Raw("TESTQ $1, CX") + b.Raw("JZ fstore") + fmaStep(b, 0) + b.Raw("fstore:") + for r := 0; r < gemmMR; r++ { + b.Raw(fmt.Sprintf("VADDPD (DX), Y%d, Y15", 2*r)) + b.Raw("VMOVUPD Y15, (DX)") + b.Raw(fmt.Sprintf("VADDPD 32(DX), Y%d, Y15", 2*r+1)) + b.Raw("VMOVUPD Y15, 32(DX)") + if r < gemmMR-1 { + b.Raw("ADDQ R8, DX") + } + } + b.Raw("VZEROUPPER") // no AVX-SSE transition penalty in the Go code after b.Ret() return b.Func() } -// gemmRow fuses one A value (at offset off in the A panel SI) into one C row's -// two XMM accumulators (lo cols 0,1 = accLo; hi cols 2,3 = accHi): broadcast -// a -> X10, X11 = Brow_lo*a, X12 = Brow_hi*a, then add into the accumulators. -func gemmRow(b *amd64.Builder, accLo, accHi string, off int) { - b.Raw(fmt.Sprintf("MOVDDUP %d(SI), X10", off)) // X10 = [a,a] - b.Raw("MOVAPS X8, X11") - b.Raw("MULPD X10, X11") // X11 = Brow_lo * a - b.Raw("ADDPD X11, " + accLo) - b.Raw("MOVAPS X9, X12") - b.Raw("MULPD X10, X12") // X12 = Brow_hi * a - b.Raw("ADDPD X12, " + accHi) +// fmaStep emits one k step of the 6x8 FMA tile, step reading the A and B +// values at that many steps past SI and DI. +func fmaStep(b *amd64.Builder, step int) { + ao, bo := 8*gemmMR*step, 8*gemmNR*step + b.Raw(fmt.Sprintf("VMOVUPD %d(DI), Y12", bo)) // B cols 0..3 + b.Raw(fmt.Sprintf("VMOVUPD %d(DI), Y13", bo+32)) // B cols 4..7 + for r := 0; r < gemmMR; r++ { + b.Raw(fmt.Sprintf("VBROADCASTSD %d(SI), Y14", ao+8*r)) + b.Raw(fmt.Sprintf("VFMADD231PD Y12, Y14, Y%d", 2*r)) + b.Raw(fmt.Sprintf("VFMADD231PD Y13, Y14, Y%d", 2*r+1)) + } } -// gemmStore adds one C row's two accumulators into memory at the current row -// pointer (DX), then advances DX by ldc bytes (R8) unless this is the last row. -func gemmStore(b *amd64.Builder, accLo, accHi string, first bool) { - b.Raw("MOVUPD (DX), X11") - b.Raw("ADDPD " + accLo + ", X11") - b.Raw("MOVUPD X11, (DX)") - b.Raw("MOVUPD 16(DX), X12") - b.Raw("ADDPD " + accHi + ", X12") - b.Raw("MOVUPD X12, 16(DX)") - b.Raw("ADDQ R8, DX") +// gemmSSE2HalfKernel builds gemmMicro6x4SSE2(kc int, pa, pb, c *float64, ldc +// int), the baseline (GOAMD64=v1, no FMA) fallback over the same 6x8 packing: +// it computes the 6-row x 4-col half of the tile whose B columns start at pb, +// so the Go side calls it twice (pb, c) and (pb+4, c+4). A whole 6x8 tile in +// XMM would need 24 accumulators; the half needs 12 (X0..X11, row r in X(2r), +// X(2r+1)) plus the B half-row X12,X13, the broadcast A X14 and a product X15. +// The B stride is still the full NR=8 doubles per k step. +// +// SSE2 has no fused multiply-add, so each step is MULPD then ADDPD: the +// running sum is the plain dst += a*b in ikj order, like the scalar oracle at +// GOAMD64=v1. +func gemmSSE2HalfKernel() *emit.Function { + b := amd64.NewFunc("gemmMicro6x4SSE2", gemmSig(), 0) + b.LoadArg("kc", "CX") + b.LoadArg("pa", "SI") + b.LoadArg("pb", "DI") + b.LoadArg("c", "DX") + b.LoadArg("ldc", "R8") + b.Raw("SHLQ $3, R8") + for r := 0; r < 12; r++ { + b.Raw(fmt.Sprintf("XORPS X%d, X%d", r, r)) + } + b.Raw("TESTQ CX, CX") + b.Raw("JZ hstore") + b.Raw("hkloop:") + b.Raw("MOVUPD (DI), X12") + b.Raw("MOVUPD 16(DI), X13") + for r := 0; r < gemmMR; r++ { + b.Raw(fmt.Sprintf("MOVDDUP %d(SI), X14", 8*r)) // X14 = [a,a] + b.Raw("MOVAPS X12, X15") + b.Raw("MULPD X14, X15") + b.Raw(fmt.Sprintf("ADDPD X15, X%d", 2*r)) + b.Raw("MULPD X13, X14") // the broadcast is spent: reuse it for the product + b.Raw(fmt.Sprintf("ADDPD X14, X%d", 2*r+1)) + } + b.Raw(fmt.Sprintf("ADDQ $%d, SI", 8*gemmMR)) + b.Raw(fmt.Sprintf("ADDQ $%d, DI", 8*gemmNR)) + b.Raw("DECQ CX") + b.Raw("JNZ hkloop") + b.Raw("hstore:") + for r := 0; r < gemmMR; r++ { + b.Raw("MOVUPD (DX), X14") + b.Raw(fmt.Sprintf("ADDPD X%d, X14", 2*r)) + b.Raw("MOVUPD X14, (DX)") + b.Raw("MOVUPD 16(DX), X15") + b.Raw(fmt.Sprintf("ADDPD X%d, X15", 2*r+1)) + b.Raw("MOVUPD X15, 16(DX)") + if r < gemmMR-1 { + b.Raw("ADDQ R8, DX") + } + } + b.Ret() + return b.Func() } // nanScan emits: X9 = (reg unordered reg) ? all-ones : 0 ; X8 |= X9, i.e. it diff --git a/internal/kernels/asmgen/go.mod b/internal/kernels/asmgen/go.mod index 9e9cec2..bafd420 100644 --- a/internal/kernels/asmgen/go.mod +++ b/internal/kernels/asmgen/go.mod @@ -6,4 +6,4 @@ module github.com/go-ndarray/ndarray/internal/kernels/asmgen go 1.26.4 -require github.com/go-asmgen/asmgen v0.10.0 +require github.com/go-asmgen/asmgen v0.11.0 diff --git a/internal/kernels/asmgen/go.sum b/internal/kernels/asmgen/go.sum index 21b2ac1..5813d72 100644 --- a/internal/kernels/asmgen/go.sum +++ b/internal/kernels/asmgen/go.sum @@ -1,2 +1,2 @@ -github.com/go-asmgen/asmgen v0.10.0 h1:LyCKpGRcug2ynAyOKHwxYxVgd4sNd9S2Oy9IFOR9lTQ= -github.com/go-asmgen/asmgen v0.10.0/go.mod h1:Hj29fXioLRe9oSnQ18SuDBiOcCbu8Qwxba6/5TMr4+Y= +github.com/go-asmgen/asmgen v0.11.0 h1:BQKTBAPEZ1LDS/gdvSemHxUzUGKx9pfHNQtbcln8q8g= +github.com/go-asmgen/asmgen v0.11.0/go.mod h1:Hj29fXioLRe9oSnQ18SuDBiOcCbu8Qwxba6/5TMr4+Y= diff --git a/internal/kernels/gemm.go b/internal/kernels/gemm.go index 84f2765..7673063 100644 --- a/internal/kernels/gemm.go +++ b/internal/kernels/gemm.go @@ -17,13 +17,17 @@ import ( // matrices' strides. A cache-blocking loop nest (NC over columns, KC over the // contraction, MC over rows) keeps the packed B panel resident in L2/L3 and each // packed A panel in L1, and a register-blocked SIMD-FMA micro-kernel -// (gemmMicro, arch-specific: NEON 4x8 on arm64, SSE2 4x4 on amd64, scalar 4x4 -// elsewhere) does the inner work. Parallelised across row bands. +// (gemmMicro, arch-specific: NEON 4x8 on arm64, AVX2/FMA 6x8 or SSE2 on amd64, +// scalar 4x4 elsewhere) does the inner work. Parallelised across row bands. // -// The arithmetic is the standard ikj accumulation dst[i][j] += a[i][p]*b[p][j]; -// the packed layout only relocates the operands, it does not reorder the sum, so -// the result is bit-identical to the scalar oracle (validated in CI and against -// numpy's A@B). +// The arithmetic is the ikj accumulation dst[i][j] += a[i][p]*b[p][j]: within +// one KC block the products are summed in p order, and each block's sum is then +// added into dst. For k <= KC that is the scalar oracle's order exactly; above +// it the block partials regroup the sum, a valid reordering that can move the +// last place (as every blocked BLAS does). Kernels that fuse the multiply-add +// (arm64, amd64 with FMA) round once per step where an unfused one rounds +// twice. On data whose products and sums are exact (the integer-valued tests) +// every path equals the oracle bit for bit. // Block sizes for the cache-blocking loop nest. They are vars so tests can pin // them (forcing the multi-block paths on small inputs) and so they can be tuned @@ -44,6 +48,20 @@ var ( // goroutine-loop overhead stays amortized. A var so it can be tuned/pinned. var bandTilesMR = 8 +// Mat is a read-only strided view of a matrix: element (i, j) is +// Data[Off + i*RS + j*CS]. Packing reads the operands through it, so a +// transposed, sliced or offset view is multiplied in place — the packing copy +// the GEMM makes anyway does the gather — instead of being materialised into +// a contiguous copy first (what BLAS's trans flags buy). Strides may be +// negative or zero; every (i, j) in range must index Data. +type Mat struct { + Data []float64 + Off, RS, CS int +} + +// rowMajor views a contiguous row-major matrix with ld elements per row. +func rowMajor(data []float64, ld int) Mat { return Mat{Data: data, RS: ld, CS: 1} } + // packBuf is one worker's reusable pair of pack buffers: paBuf for the MC*KC A // panel, pbBuf for the KC*NC B panel. Pooling them keeps the per-call GEMM // allocation-free above the goroutine launch, which matters at small n. @@ -95,14 +113,14 @@ func getPackBuf() *packBuf { // r0 (the parallel splitter guarantees this) so packed A panels never straddle a // band boundary; r1 may be m. This is the single-worker entry: it packs its own // B panel. The multi-worker path (gemmBand) shares one packed B across workers. -func packGemmRows(r0, r1 int, dst, a, b []float64, k, n int, buf *packBuf) { +func packGemmRows(r0, r1 int, dst []float64, a, b Mat, k, n int, buf *packBuf) { pa, pb := buf.pa, buf.pb for jc := 0; jc < n; jc += blockNC { // L3 column block of B nc := min(blockNC, n-jc) for pc := 0; pc < k; pc += blockKC { // L2 contraction block kc := min(blockKC, k-pc) - packB(b, n, pc, kc, jc, nc, pb) // pack B(kc x nc) -> NR-wide panels - gemmBand(r0, r1, jc, nc, pc, kc, dst, a, k, n, pa, pb) + packB(b, pc, kc, jc, nc, pb) // pack B(kc x nc) -> NR-wide panels + gemmBand(r0, r1, jc, nc, pc, kc, dst, a, n, pa, pb) } } } @@ -116,10 +134,10 @@ func packGemmRows(r0, r1 int, dst, a, b []float64, k, n int, buf *packBuf) { // is what lets several workers share one packed B: the redundant full-B repack // each worker used to do — 1 MiB copied GOMAXPROCS times per kc block — was the // dominant non-scaling cost of the old row-split parallelism. -func gemmBand(r0, r1, jc, nc, pc, kc int, dst, a []float64, k, n int, pa, pb []float64) { +func gemmBand(r0, r1, jc, nc, pc, kc int, dst []float64, a Mat, n int, pa, pb []float64) { for ic := r0; ic < r1; ic += blockMC { // L1 row block of A mc := min(blockMC, r1-ic) - packA(a, k, ic, mc, pc, kc, pa) // pack A(mc x kc) -> MR-tall panels + packA(a, ic, mc, pc, kc, pa) // pack A(mc x kc) -> MR-tall panels for jr := 0; jr < nc; jr += NR { nr := min(NR, nc-jr) for ir := 0; ir < mc; ir += MR { @@ -136,18 +154,35 @@ func gemmBand(r0, r1, jc, nc, pc, kc int, dst, a []float64, k, n int, pa, pb []f } // packB copies B[pc:pc+kc][jc:jc+nc] into NR-wide column panels: for each panel -// jr the layout is pb[jr*kc + p*NR + c] = B[(pc+p)*ldb + jc+jr+c], with the c >= -// nr columns of an edge panel zero-filled so the micro-kernel reads a full NR. -func packB(b []float64, ldb, pc, kc, jc, nc int, pb []float64) { +// jr the layout is pb[jr*kc + p*NR + c] = B(pc+p, jc+jr+c), with the c >= nr +// columns of an edge panel zero-filled so the micro-kernel reads a full NR. +// A unit column stride copies row runs; any other (a transposed B) walks each +// column down its own unit-or-not row stride, writing the panel column-wise. +func packB(b Mat, pc, kc, jc, nc int, pb []float64) { for jr := 0; jr < nc; jr += NR { nr := min(NR, nc-jr) dst := pb[jr*kc:] + if b.CS == 1 { + for p := 0; p < kc; p++ { + srcRow := b.Data[b.Off+(pc+p)*b.RS+jc+jr:] + d := dst[p*NR:] + for c := 0; c < nr; c++ { + d[c] = srcRow[c] + } + for c := nr; c < NR; c++ { + d[c] = 0 + } + } + continue + } + for c := 0; c < nr; c++ { + src := b.Off + pc*b.RS + (jc+jr+c)*b.CS + for p := 0; p < kc; p++ { + dst[p*NR+c] = b.Data[src+p*b.RS] + } + } for p := 0; p < kc; p++ { - srcRow := b[(pc+p)*ldb+jc+jr:] d := dst[p*NR:] - for c := 0; c < nr; c++ { - d[c] = srcRow[c] - } for c := nr; c < NR; c++ { d[c] = 0 } @@ -155,17 +190,43 @@ func packB(b []float64, ldb, pc, kc, jc, nc int, pb []float64) { } } +// packBParMin is the B-block size, in elements, from which packBP splits the +// packing across workers (64 Ki doubles = 512 KiB). A var so tests can pin it. +var packBParMin = 1 << 16 + +// packBP is packB with the NR-wide panels split across w goroutines. MatMulP +// packs the shared B block once per (jc, pc) round, and done by one goroutine +// it is a serial step while every other worker waits at the round's start: +// a 1 MiB strided copy, eight times in a 1024^2 product. +// +// Below packBParMin elements the goroutine launch costs more than the copy +// (measured on M4: a 128x128 block packed in parallel made 96^3..128^2 +// products 8-15% slower), so a small block is packed by the caller alone. +func packBP(b Mat, pc, kc, jc, nc int, pb []float64, w int) { + if kc*nc < packBParMin { + packB(b, pc, kc, jc, nc, pb) + return + } + panels := (nc + NR - 1) / NR + parallelFor(panels, min(w, panels), func(lo, hi int) { + j0 := lo * NR + packB(b, pc, kc, jc+j0, min(hi*NR, nc)-j0, pb[j0*kc:]) + }) +} + // packA copies A[ic:ic+mc][pc:pc+kc] into MR-tall row panels: for each panel ir -// the layout is pa[ir*kc + p*MR + r] = A[(ic+ir+r)*lda + pc+p], with the r >= mr -// rows of an edge panel zero-filled so the micro-kernel reads a full MR. -func packA(a []float64, lda, ic, mc, pc, kc int, pa []float64) { +// the layout is pa[ir*kc + p*MR + r] = A(ic+ir+r, pc+p), with the r >= mr rows +// of an edge panel zero-filled so the micro-kernel reads a full MR. +func packA(a Mat, ic, mc, pc, kc int, pa []float64) { for ir := 0; ir < mc; ir += MR { mr := min(MR, mc-ir) dst := pa[ir*kc:] + src := a.Off + (ic+ir)*a.RS + pc*a.CS for p := 0; p < kc; p++ { d := dst[p*MR:] + at := src + p*a.CS for r := 0; r < mr; r++ { - d[r] = a[(ic+ir+r)*lda+pc+p] + d[r] = a.Data[at+r*a.RS] } for r := mr; r < MR; r++ { d[r] = 0 @@ -174,19 +235,20 @@ func packA(a []float64, lda, ic, mc, pc, kc int, pa []float64) { } } -// gemmEdge is the scalar fallback for a partial (mr < MR or nr < NR) tile: it -// does the same dst[c0 + r*ldc + c] += sum_p pa[p*MR+r]*pb[p*NR+c] the micro- -// kernel does, reading the already-packed (and zero-padded) panels, so it stays -// contiguous and produces the identical ikj-order result. Only the matrix's -// ragged right/bottom edges take this path. +// gemmEdge computes a partial (mr < MR or nr < NR) tile with the micro-kernel +// itself, the way BLIS does: the packed panels are already zero-padded to a full +// MR x NR, so the kernel runs into a zeroed MR x NR scratch tile and only the +// valid mr x nr corner is added into dst. Every element of C is therefore summed +// by the same kernel, edge or not. The scalar loop this replaces ran ~20x slower +// per element, and on amd64 (MR=6) every 256-row MC block ends in a 4-row edge +// panel: it measured 40% of a 512^2 product there. func gemmEdge(kc int, pa, pb, dst []float64, c0, ldc, mr, nr int) { + var tile [MR * NR]float64 + gemmMicro(kc, pa, pb, tile[:], NR) for r := 0; r < mr; r++ { - for c := 0; c < nr; c++ { - var s float64 - for p := 0; p < kc; p++ { - s += pa[p*MR+r] * pb[p*NR+c] - } - dst[c0+r*ldc+c] += s + row := dst[c0+r*ldc : c0+r*ldc+nr] + for c := range row { + row[c] += tile[r*NR+c] } } } @@ -239,6 +301,14 @@ var matVecThreshold = 1 << 14 // straggling). With many small bands the P-cores simply grab more of them, so // adding the E-cores helps instead of hurting. func MatMulP(dst, a, b []float64, m, k, n int) { + MatMulStridedP(dst, rowMajor(a, k), rowMajor(b, n), m, k, n) +} + +// MatMulStridedP is MatMulP over strided operand views (see Mat): dst (m x n, +// contiguous, zeroed by the caller) = a (m x k) * b (k x n). The views are +// only read, through the packing, so a transposed or sliced operand costs no +// copy beyond the one the packed GEMM makes anyway. +func MatMulStridedP(dst []float64, a, b Mat, m, k, n int) { if m*n < GemmThreshold { buf := getPackBuf() packGemmRows(0, m, dst, a, b, k, n, buf) @@ -280,7 +350,7 @@ func MatMulP(dst, a, b []float64, m, k, n int) { nc := min(blockNC, n-jc) for pc := 0; pc < k; pc += blockKC { // L2 contraction block kc := min(blockKC, k-pc) - packB(b, n, pc, kc, jc, nc, pb) // pack B once for all workers + packBP(b, pc, kc, jc, nc, pb, w) // pack B once, by all workers var next atomic.Int64 // shared band cursor var wg sync.WaitGroup @@ -296,7 +366,7 @@ func MatMulP(dst, a, b []float64, m, k, n int) { } r0 := bi * bandRows r1 := min(r0+bandRows, m) - gemmBand(r0, r1, jc, nc, pc, kc, dst, a, k, n, ab.pa, pb) + gemmBand(r0, r1, jc, nc, pc, kc, dst, a, n, ab.pa, pb) } packPool.Put(ab) }() @@ -312,13 +382,14 @@ func MatMulP(dst, a, b []float64, m, k, n int) { // non-parallel computation. func MatMul(dst, a, b []float64, m, k, n int) { buf := getPackBuf() - packGemmRows(0, m, dst, a, b, k, n, buf) + packGemmRows(0, m, dst, rowMajor(a, k), rowMajor(b, n), k, n, buf) packPool.Put(buf) } // dotRange returns sum(a[i]*b[i]) over equal-length slices, using four -// independent accumulators so the compiler keeps the FMA chain unrolled and (on -// amd64/arm64) auto-vectorises the multiply-add. The four-way grouping is a fixed +// independent accumulators, so four multiply-add chains are in flight instead of +// one (gc does not vectorise this loop; the gain is latency hiding). The +// four-way grouping is a fixed // reassociation of the sum; MatVecP/Dot1DP document the ULP trade-off where it is // observable. It is the contiguous building block for both mat·vec and the 1-D // dot — neither needs to go through the packing GEMM. diff --git a/internal/kernels/gemm_amd64.go b/internal/kernels/gemm_amd64.go index 27bd683..2b1a993 100644 --- a/internal/kernels/gemm_amd64.go +++ b/internal/kernels/gemm_amd64.go @@ -1,25 +1,49 @@ package kernels -// amd64 GEMM micro-kernel: an SSE2 4x4 register-blocked tile (gemm in -// sum_amd64.s, generated by go-asmgen). amd64 baseline is SSE2 (GOAMD64=v1) with -// no FMA, so the kernel uses explicit MULPD+ADDPD; SSE2's packed FP add keeps the -// accumulation in the same ikj order as the scalar oracle (bit-identical). -// MR x NR = 4 x 4 fits the 16 XMM registers (8 for the C tile). +// amd64 GEMM micro-kernel: a 6x8 register-blocked tile (gemm in sum_amd64.s, +// generated by go-asmgen), the BLIS Haswell dgemm shape. Two kernels share the +// one 6x8 packing: +// +// - gemmMicro6x8FMA, AVX2-era FMA3: 12 YMM accumulators, VBROADCASTSD of each +// A value and VFMADD231PD against the two-YMM B row. Taken when hasFMA +// (CPUID FMA, AVX and OSXSAVE, and XCR0 saving YMM state) says so — every +// Intel and AMD part since 2013. +// - gemmMicro6x4SSE2, the GOAMD64=v1 baseline, called twice per tile (columns +// 0-3, then 4-7): explicit MULPD+ADDPD, since SSE2 has no FMA. +// +// The FMA kernel rounds each a*b+c once, as tuned BLAS and arm64's kernel do; the +// SSE2 one rounds twice, as the scalar oracle does at GOAMD64=v1. On data whose +// products and partial sums are exact (the integer-valued tests) the two agree +// bit-for-bit with the oracle; elsewhere they can differ in the last place. // MR, NR are the micro-kernel's register-block dimensions: it computes an // MR-row x NR-col tile of C per call. The packing in gemm.go matches. const ( - MR = 4 - NR = 4 + MR = 6 + NR = 8 ) +// gemmUseFMA selects the FMA kernel. It is a var so tests can force the SSE2 +// path on hardware that has FMA (never the reverse: that would fault). +var gemmUseFMA = hasFMA() + // gemmMicro adds the MR x NR tile sum_p pa[p*MR..]*pb[p*NR..] into the C block at -// dst[0:], whose rows are ldc apart, using the SSE2 kernel. pa, pb hold at least -// kc*MR and kc*NR packed (zero-padded) doubles; dst has at least (MR-1)*ldc + NR -// elements. Bit-identical to the scalar ikj accumulation (see gemmEdge). +// dst[0:], whose rows are ldc apart. pa, pb hold at least kc*MR and kc*NR packed +// (zero-padded) doubles; dst has at least (MR-1)*ldc + NR elements. func gemmMicro(kc int, pa, pb, dst []float64, ldc int) { - gemmMicro4x4(kc, &pa[0], &pb[0], &dst[0], ldc) + if gemmUseFMA { + gemmMicro6x8FMA(kc, &pa[0], &pb[0], &dst[0], ldc) + return + } + gemmMicro6x4SSE2(kc, &pa[0], &pb[0], &dst[0], ldc) + gemmMicro6x4SSE2(kc, &pa[0], &pb[4], &dst[4], ldc) } //go:noescape -func gemmMicro4x4(kc int, pa, pb, c *float64, ldc int) +func gemmMicro6x8FMA(kc int, pa, pb, c *float64, ldc int) + +//go:noescape +func gemmMicro6x4SSE2(kc int, pa, pb, c *float64, ldc int) + +// hasFMA reports CPU and OS support for the FMA kernel (go-asmgen FeatureProbe). +func hasFMA() bool diff --git a/internal/kernels/gemm_amd64_test.go b/internal/kernels/gemm_amd64_test.go new file mode 100644 index 0000000..72001a3 --- /dev/null +++ b/internal/kernels/gemm_amd64_test.go @@ -0,0 +1,16 @@ +package kernels + +import "testing" + +// TestGemmSSE2Fallback runs the GEMM suite on the baseline SSE2 kernel whatever +// the host supports, so a CPU with FMA still exercises the path a pre-2013 CPU +// (or a hypervisor that masks FMA) takes. The FMA path is the default and is +// exercised by the rest of the suite wherever hasFMA is true. +func TestGemmSSE2Fallback(t *testing.T) { + saved := gemmUseFMA + gemmUseFMA = false + defer func() { gemmUseFMA = saved }() + t.Run("correctness", TestGemmCorrectness) + t.Run("bands", TestGemmBandSizing) + t.Run("float", TestGemmFloatTolerance) +} diff --git a/internal/kernels/gemm_bench_test.go b/internal/kernels/gemm_bench_test.go new file mode 100644 index 0000000..fcf2904 --- /dev/null +++ b/internal/kernels/gemm_bench_test.go @@ -0,0 +1,44 @@ +package kernels + +import ( + "strconv" + "testing" +) + +// BenchmarkGemmMicro times the register-blocked micro-kernel alone on +// L1-resident packed panels (kc=256): its GFLOP/s is the per-core ceiling the +// blocked driver can approach, separate from packing and scheduling. +func BenchmarkGemmMicro(b *testing.B) { + const kc = 256 + pa := make([]float64, kc*MR) + pb := make([]float64, kc*NR) + for i := range pa { + pa[i] = float64(i%7) * 0.5 + } + for i := range pb { + pb[i] = float64(i%5) * 0.25 + } + c := make([]float64, MR*NR) + b.SetBytes(0) + b.ResetTimer() + for i := 0; i < b.N; i++ { + gemmMicro(kc, pa, pb, c, NR) + } + b.ReportMetric(float64(2*MR*NR*kc)*float64(b.N)/b.Elapsed().Seconds()/1e9, "GFLOP/s") +} + +// BenchmarkMatMulSerial times the whole serial GEMM (packing + blocking + +// micro-kernel, one core) at a few sizes. +func BenchmarkMatMulSerial(b *testing.B) { + for _, n := range []int{32, 64, 128, 256, 512} { + a, bb := intMat(n, n, 1), intMat(n, n, 2) + dst := make([]float64, n*n) + b.Run(strconv.Itoa(n), func(b *testing.B) { + for i := 0; i < b.N; i++ { + clear(dst) + MatMul(dst, a, bb, n, n, n) + } + b.ReportMetric(float64(2*n*n*n)*float64(b.N)/b.Elapsed().Seconds()/1e9, "GFLOP/s") + }) + } +} diff --git a/internal/kernels/gemm_test.go b/internal/kernels/gemm_test.go index 37d9a16..d759aa6 100644 --- a/internal/kernels/gemm_test.go +++ b/internal/kernels/gemm_test.go @@ -140,6 +140,30 @@ func TestGemmBandSizing(t *testing.T) { }) } +// TestGemmParallelPackB forces packBP's split path (normally only for blocks of +// 64 Ki elements or more) on small products, so the NR-panel ranges handed to +// each goroutine — including a ragged last panel and more workers than panels +// — must reassemble the exact packed block the serial packB writes. +func TestGemmParallelPackB(t *testing.T) { + o := packBParMin + packBParMin = 1 + defer func() { packBParMin = o }() + withMaxProcs(4, func() { + withThresholds(1<<14, 1, func() { // force MatMulP's parallel path + for _, s := range []struct{ m, k, n int }{ + {13, 9, 3 * NR}, // whole panels + {20, 17, 5*NR + 3}, // ragged last panel + {7, 5, NR - 1}, // one partial panel, more workers than panels + {33, 40, 11*NR + 1}, + } { + a, b := intMat(s.m, s.k, int64(s.n)), intMat(s.k, s.n, int64(s.m)) + assertGemm(t, s.m, s.k, s.n, a, b) + withBlocks(MR, 3, 2*NR, func() { assertGemm(t, s.m, s.k, s.n, a, b) }) + } + }) + }) +} + // TestGemmFloatTolerance validates the packed GEMM against the oracle for // genuinely floating-point (non-integer) operands. The packed kernel uses the // identical ikj summation order, so it is in fact bit-identical here too; we diff --git a/internal/kernels/sum_amd64.s b/internal/kernels/sum_amd64.s index 49d0901..6ee227e 100644 --- a/internal/kernels/sum_amd64.s +++ b/internal/kernels/sum_amd64.s @@ -393,7 +393,149 @@ TEXT ·divSSE2(SB), NOSPLIT, $0-32 bdone: RET -TEXT ·gemmMicro4x4(SB), NOSPLIT, $0-40 +TEXT ·gemmMicro6x8FMA(SB), NOSPLIT, $0-40 + MOVQ kc+0(FP), CX + MOVQ pa+8(FP), SI + MOVQ pb+16(FP), DI + MOVQ c+24(FP), DX + MOVQ ldc+32(FP), R8 + SHLQ $3, R8 + MOVQ DX, R9 + PREFETCHT0 (R9) + PREFETCHT0 63(R9) + ADDQ R8, R9 + PREFETCHT0 (R9) + PREFETCHT0 63(R9) + ADDQ R8, R9 + PREFETCHT0 (R9) + PREFETCHT0 63(R9) + ADDQ R8, R9 + PREFETCHT0 (R9) + PREFETCHT0 63(R9) + ADDQ R8, R9 + PREFETCHT0 (R9) + PREFETCHT0 63(R9) + ADDQ R8, R9 + PREFETCHT0 (R9) + PREFETCHT0 63(R9) + VXORPD Y0, Y0, Y0 + VXORPD Y1, Y1, Y1 + VXORPD Y2, Y2, Y2 + VXORPD Y3, Y3, Y3 + VXORPD Y4, Y4, Y4 + VXORPD Y5, Y5, Y5 + VXORPD Y6, Y6, Y6 + VXORPD Y7, Y7, Y7 + VXORPD Y8, Y8, Y8 + VXORPD Y9, Y9, Y9 + VXORPD Y10, Y10, Y10 + VXORPD Y11, Y11, Y11 + MOVQ CX, R10 + SHRQ $1, R10 + JZ ftail + fkloop: + PREFETCHT0 384(SI) + VMOVUPD 0(DI), Y12 + VMOVUPD 32(DI), Y13 + VBROADCASTSD 0(SI), Y14 + VFMADD231PD Y12, Y14, Y0 + VFMADD231PD Y13, Y14, Y1 + VBROADCASTSD 8(SI), Y14 + VFMADD231PD Y12, Y14, Y2 + VFMADD231PD Y13, Y14, Y3 + VBROADCASTSD 16(SI), Y14 + VFMADD231PD Y12, Y14, Y4 + VFMADD231PD Y13, Y14, Y5 + VBROADCASTSD 24(SI), Y14 + VFMADD231PD Y12, Y14, Y6 + VFMADD231PD Y13, Y14, Y7 + VBROADCASTSD 32(SI), Y14 + VFMADD231PD Y12, Y14, Y8 + VFMADD231PD Y13, Y14, Y9 + VBROADCASTSD 40(SI), Y14 + VFMADD231PD Y12, Y14, Y10 + VFMADD231PD Y13, Y14, Y11 + VMOVUPD 64(DI), Y12 + VMOVUPD 96(DI), Y13 + VBROADCASTSD 48(SI), Y14 + VFMADD231PD Y12, Y14, Y0 + VFMADD231PD Y13, Y14, Y1 + VBROADCASTSD 56(SI), Y14 + VFMADD231PD Y12, Y14, Y2 + VFMADD231PD Y13, Y14, Y3 + VBROADCASTSD 64(SI), Y14 + VFMADD231PD Y12, Y14, Y4 + VFMADD231PD Y13, Y14, Y5 + VBROADCASTSD 72(SI), Y14 + VFMADD231PD Y12, Y14, Y6 + VFMADD231PD Y13, Y14, Y7 + VBROADCASTSD 80(SI), Y14 + VFMADD231PD Y12, Y14, Y8 + VFMADD231PD Y13, Y14, Y9 + VBROADCASTSD 88(SI), Y14 + VFMADD231PD Y12, Y14, Y10 + VFMADD231PD Y13, Y14, Y11 + ADDQ $96, SI + ADDQ $128, DI + DECQ R10 + JNZ fkloop + ftail: + TESTQ $1, CX + JZ fstore + VMOVUPD 0(DI), Y12 + VMOVUPD 32(DI), Y13 + VBROADCASTSD 0(SI), Y14 + VFMADD231PD Y12, Y14, Y0 + VFMADD231PD Y13, Y14, Y1 + VBROADCASTSD 8(SI), Y14 + VFMADD231PD Y12, Y14, Y2 + VFMADD231PD Y13, Y14, Y3 + VBROADCASTSD 16(SI), Y14 + VFMADD231PD Y12, Y14, Y4 + VFMADD231PD Y13, Y14, Y5 + VBROADCASTSD 24(SI), Y14 + VFMADD231PD Y12, Y14, Y6 + VFMADD231PD Y13, Y14, Y7 + VBROADCASTSD 32(SI), Y14 + VFMADD231PD Y12, Y14, Y8 + VFMADD231PD Y13, Y14, Y9 + VBROADCASTSD 40(SI), Y14 + VFMADD231PD Y12, Y14, Y10 + VFMADD231PD Y13, Y14, Y11 + fstore: + VADDPD (DX), Y0, Y15 + VMOVUPD Y15, (DX) + VADDPD 32(DX), Y1, Y15 + VMOVUPD Y15, 32(DX) + ADDQ R8, DX + VADDPD (DX), Y2, Y15 + VMOVUPD Y15, (DX) + VADDPD 32(DX), Y3, Y15 + VMOVUPD Y15, 32(DX) + ADDQ R8, DX + VADDPD (DX), Y4, Y15 + VMOVUPD Y15, (DX) + VADDPD 32(DX), Y5, Y15 + VMOVUPD Y15, 32(DX) + ADDQ R8, DX + VADDPD (DX), Y6, Y15 + VMOVUPD Y15, (DX) + VADDPD 32(DX), Y7, Y15 + VMOVUPD Y15, 32(DX) + ADDQ R8, DX + VADDPD (DX), Y8, Y15 + VMOVUPD Y15, (DX) + VADDPD 32(DX), Y9, Y15 + VMOVUPD Y15, 32(DX) + ADDQ R8, DX + VADDPD (DX), Y10, Y15 + VMOVUPD Y15, (DX) + VADDPD 32(DX), Y11, Y15 + VMOVUPD Y15, 32(DX) + VZEROUPPER + RET + +TEXT ·gemmMicro6x4SSE2(SB), NOSPLIT, $0-40 MOVQ kc+0(FP), CX MOVQ pa+8(FP), SI MOVQ pb+16(FP), DI @@ -408,71 +550,112 @@ TEXT ·gemmMicro4x4(SB), NOSPLIT, $0-40 XORPS X5, X5 XORPS X6, X6 XORPS X7, X7 + XORPS X8, X8 + XORPS X9, X9 + XORPS X10, X10 + XORPS X11, X11 TESTQ CX, CX - JZ gstore - gkloop: - MOVUPD (DI), X8 - MOVUPD 16(DI), X9 - ADDQ $32, DI - MOVDDUP 0(SI), X10 - MOVAPS X8, X11 - MULPD X10, X11 - ADDPD X11, X0 - MOVAPS X9, X12 - MULPD X10, X12 - ADDPD X12, X1 - MOVDDUP 8(SI), X10 - MOVAPS X8, X11 - MULPD X10, X11 - ADDPD X11, X2 - MOVAPS X9, X12 - MULPD X10, X12 - ADDPD X12, X3 - MOVDDUP 16(SI), X10 - MOVAPS X8, X11 - MULPD X10, X11 - ADDPD X11, X4 - MOVAPS X9, X12 - MULPD X10, X12 - ADDPD X12, X5 - MOVDDUP 24(SI), X10 - MOVAPS X8, X11 - MULPD X10, X11 - ADDPD X11, X6 - MOVAPS X9, X12 - MULPD X10, X12 - ADDPD X12, X7 - ADDQ $32, SI + JZ hstore + hkloop: + MOVUPD (DI), X12 + MOVUPD 16(DI), X13 + MOVDDUP 0(SI), X14 + MOVAPS X12, X15 + MULPD X14, X15 + ADDPD X15, X0 + MULPD X13, X14 + ADDPD X14, X1 + MOVDDUP 8(SI), X14 + MOVAPS X12, X15 + MULPD X14, X15 + ADDPD X15, X2 + MULPD X13, X14 + ADDPD X14, X3 + MOVDDUP 16(SI), X14 + MOVAPS X12, X15 + MULPD X14, X15 + ADDPD X15, X4 + MULPD X13, X14 + ADDPD X14, X5 + MOVDDUP 24(SI), X14 + MOVAPS X12, X15 + MULPD X14, X15 + ADDPD X15, X6 + MULPD X13, X14 + ADDPD X14, X7 + MOVDDUP 32(SI), X14 + MOVAPS X12, X15 + MULPD X14, X15 + ADDPD X15, X8 + MULPD X13, X14 + ADDPD X14, X9 + MOVDDUP 40(SI), X14 + MOVAPS X12, X15 + MULPD X14, X15 + ADDPD X15, X10 + MULPD X13, X14 + ADDPD X14, X11 + ADDQ $48, SI + ADDQ $64, DI DECQ CX - JNZ gkloop - gstore: - MOVUPD (DX), X11 - ADDPD X0, X11 - MOVUPD X11, (DX) - MOVUPD 16(DX), X12 - ADDPD X1, X12 - MOVUPD X12, 16(DX) + JNZ hkloop + hstore: + MOVUPD (DX), X14 + ADDPD X0, X14 + MOVUPD X14, (DX) + MOVUPD 16(DX), X15 + ADDPD X1, X15 + MOVUPD X15, 16(DX) + ADDQ R8, DX + MOVUPD (DX), X14 + ADDPD X2, X14 + MOVUPD X14, (DX) + MOVUPD 16(DX), X15 + ADDPD X3, X15 + MOVUPD X15, 16(DX) ADDQ R8, DX - MOVUPD (DX), X11 - ADDPD X2, X11 - MOVUPD X11, (DX) - MOVUPD 16(DX), X12 - ADDPD X3, X12 - MOVUPD X12, 16(DX) + MOVUPD (DX), X14 + ADDPD X4, X14 + MOVUPD X14, (DX) + MOVUPD 16(DX), X15 + ADDPD X5, X15 + MOVUPD X15, 16(DX) ADDQ R8, DX - MOVUPD (DX), X11 - ADDPD X4, X11 - MOVUPD X11, (DX) - MOVUPD 16(DX), X12 - ADDPD X5, X12 - MOVUPD X12, 16(DX) + MOVUPD (DX), X14 + ADDPD X6, X14 + MOVUPD X14, (DX) + MOVUPD 16(DX), X15 + ADDPD X7, X15 + MOVUPD X15, 16(DX) ADDQ R8, DX - MOVUPD (DX), X11 - ADDPD X6, X11 - MOVUPD X11, (DX) - MOVUPD 16(DX), X12 - ADDPD X7, X12 - MOVUPD X12, 16(DX) + MOVUPD (DX), X14 + ADDPD X8, X14 + MOVUPD X14, (DX) + MOVUPD 16(DX), X15 + ADDPD X9, X15 + MOVUPD X15, 16(DX) ADDQ R8, DX + MOVUPD (DX), X14 + ADDPD X10, X14 + MOVUPD X14, (DX) + MOVUPD 16(DX), X15 + ADDPD X11, X15 + MOVUPD X15, 16(DX) + RET + +TEXT ·hasFMA(SB), NOSPLIT, $0-1 + MOVB $0, ret+0(FP) + MOVL $1, AX + CPUID + ANDL $0x18001000, CX // FMA, AVX and OSXSAVE + CMPL CX, $0x18001000 + JNE hasFMA_unsupported + XORL CX, CX + XGETBV + ANDL $6, AX // XMM and YMM state enabled in XCR0 + CMPL AX, $6 + JNE hasFMA_unsupported + MOVB $1, ret+0(FP) +hasFMA_unsupported: RET diff --git a/linalg.go b/linalg.go index 056dde5..a64119f 100644 --- a/linalg.go +++ b/linalg.go @@ -11,16 +11,23 @@ import ( // multiply). var ErrLinalg = fmt.Errorf("ndarray: incompatible shapes for linear algebra") -// matmul2D multiplies a contiguous (m x k) by a contiguous (k x n), returning a -// fresh (m x n) array. The operands are materialised so strided/transposed -// views work transparently. +// matmul2D multiplies an (m x k) by a (k x n), returning a fresh (m x n) +// array. Either operand may be any 2-D view: the GEMM packs it through its +// strides (matView), so nothing is materialised first. func matmul2D(a, b *Array, m, k, n int) *Array { dst := make([]float64, m*n) - kernels.MatMulP(dst, a.contiguousData(), b.contiguousData(), m, k, n) + kernels.MatMulStridedP(dst, a.matView(), b.matView(), m, k, n) shape := []int{m, n} return &Array{data: dst, shape: shape, strides: rowMajorStrides(shape)} } +// matView is a 2-D array as the GEMM's strided operand view. The packing +// reads it in place, so a transposed or sliced operand (Inner's bᵀ, a[:, ::2]) +// is never materialised into a contiguous copy first. +func (a *Array) matView() kernels.Mat { + return kernels.Mat{Data: a.data, Off: a.offset, RS: a.strides[0], CS: a.strides[1]} +} + // MatMul returns the matrix product of two 2-D arrays, matching numpy.matmul // (the @ operator) for the 2-D case: a is (m x k), b is (k x n), the result is // (m x n). Both operands must be 2-D and the inner dimensions must agree. diff --git a/linalg_test.go b/linalg_test.go index 5055b8c..f31afe0 100644 --- a/linalg_test.go +++ b/linalg_test.go @@ -132,3 +132,59 @@ func TestOuter(t *testing.T) { eqFloats(t, o2.materialize(), []float64{1, 10, 2, 20, 3, 30, 4, 40}) } + +// TestMatMulStridedViews multiplies views that are not contiguous — both +// transposes, a step-2 column slice, a reversed (negative-stride) row slice, an +// offset sub-block and a zero-stride broadcast row — and requires the exact +// result of multiplying their contiguous copies. The GEMM packs views through +// their strides instead of materialising them, so the packed values, and hence +// every product and sum, must be the same. +func TestMatMulStridedViews(t *testing.T) { + grid := func(r, c int, seed float64) *Array { + d := make([]float64, r*c) + for i := range d { + d[i] = float64((i*7+int(seed))%23) - 11 + } + return mustArr(t, ok(FromData(d, r, c))) + } + big := grid(40, 30, 1) + sl := func(x *Array, ix ...Index) *Array { + v, err := x.Slice(ix...) + if err != nil { + t.Fatal(err) + } + return v + } + row := sl(grid(1, 30, 5), A(0), All()) + bcast := &Array{data: row.data, shape: []int{13, 30}, strides: []int{0, 1}} + cases := []struct { + name string + a, b *Array + }{ + {"aT", grid(30, 17, 2).Transpose(), grid(30, 21, 3)}, + {"bT", grid(17, 30, 2), grid(21, 30, 3).Transpose()}, + {"both T", grid(30, 17, 2).Transpose(), grid(21, 30, 3).Transpose()}, + {"step cols", sl(big, All(), Step(2)), grid(15, 9, 4)}, + {"reversed rows", sl(big, Step(-1), All()), grid(30, 11, 6)}, + {"offset block", sl(big, R(3, 29), R(5, 25)), sl(big, R(7, 27), R(2, 19))}, + {"broadcast rows", bcast, grid(30, 10, 7)}, + } + for _, c := range cases { + got, err := c.a.MatMul(c.b) + if err != nil { + t.Fatalf("%s: %v", c.name, err) + } + want, err := c.a.Copy().MatMul(c.b.Copy()) + if err != nil { + t.Fatalf("%s copy: %v", c.name, err) + } + if !sameShape(got.shape, want.shape) { + t.Fatalf("%s: shape %v, want %v", c.name, got.shape, want.shape) + } + for i := range want.data { + if got.data[i] != want.data[i] { + t.Fatalf("%s [%d]: %v, want %v", c.name, i, got.data[i], want.data[i]) + } + } + } +}