Skip to content

perf: AVX2/FMA GEMM on amd64, strided operands, parallel B packing - #11

Merged
tannevaled merged 1 commit into
mainfrom
perf/amd64-fma-gemm
Oct 4, 2026
Merged

tannevaled merged 1 commit into
mainfrom
perf/amd64-fma-gemm

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

amd64 had never been measured on real hardware, and its GEMM micro-kernel
was a 4x4 SSE2 tile without FMA. On a Zen 3 (16 cores, pinned) against
NumPy 2.5.3 + OpenBLAS 0.3.34, interleaved with the previous version:

MatMul 1024^2 21.87 -> 8.05 ms x2.72 (5.9x OpenBLAS 1 thread, 0.60x 16)
MatMul 512^2 3.69 -> 1.56 ms x2.36
Inner 512^2 5.87 -> 1.47 ms x4.00 (0.96x OpenBLAS 16 threads)
1024x256.256x1024 5.72 -> 2.71 ms x2.11

  • AVX2/FMA 6x8 micro-kernel (the BLIS Haswell shape, 12 YMM accumulators),
    generated by go-asmgen and chosen at run time by its new FMA probe
    (go-asmgen v0.11.0); the GOAMD64=v1 fallback is two SSE2 6x4 tiles over
    the same packing. 43 GFLOP/s per Cascade Lake core in L1, the FMA peak.
  • Edge tiles run the micro-kernel into a scratch tile instead of a scalar
    loop, which took 40% of a 512^2 product once MR=6 left a 4-row edge in
    every 256-row block.
  • The kernel prefetches its C tile on entry and the A panel eight steps
    ahead: serial 768^2 27.5 -> 34.7 GFLOP/s. An MC/KC sweep changed nothing.
  • MatMulP's shared B block is packed by all workers instead of one while
    the others wait (+30-70% in parallel); blocks under 64 Ki elements stay
    serial, where the launch cost more than it saved on M4.
  • The GEMM reads its operands through a strided view (kernels.Mat), so a
    transposed, sliced, reversed or broadcast 2-D operand is packed in place
    rather than materialised first. That is most of Inner's gain.

Tests pass on real amd64 (Zen 3, FMA path), arm64, ppc64le, riscv64 and
loong64 hardware; s390x's host did not answer and stays on the qemu lane.
Coverage is 100% on a real FMA host (the FMA branch is unreachable under
Rosetta). New tests: the SSE2 fallback forced on FMA hardware, the
parallel-pack split, and products of strided views against their
contiguous copies; both new tests were checked to fail on a planted bug.

docs/perf.md gets the real-hardware amd64 section, and comments that were
no longer true are corrected: the GEMM is bit-identical to the ikj oracle
only for k <= KC (above it the block partials regroup the sum), and gc does
not auto-vectorise dotRange.

🤖 Generated with Claude Code

amd64 had never been measured on real hardware, and its GEMM micro-kernel
was a 4x4 SSE2 tile without FMA. On a Zen 3 (16 cores, pinned) against
NumPy 2.5.3 + OpenBLAS 0.3.34, interleaved with the previous version:

  MatMul 1024^2   21.87 -> 8.05 ms  x2.72  (5.9x OpenBLAS 1 thread, 0.60x 16)
  MatMul 512^2     3.69 -> 1.56 ms  x2.36
  Inner 512^2      5.87 -> 1.47 ms  x4.00  (0.96x OpenBLAS 16 threads)
  1024x256.256x1024 5.72 -> 2.71 ms x2.11

- AVX2/FMA 6x8 micro-kernel (the BLIS Haswell shape, 12 YMM accumulators),
  generated by go-asmgen and chosen at run time by its new FMA probe
  (go-asmgen v0.11.0); the GOAMD64=v1 fallback is two SSE2 6x4 tiles over
  the same packing. 43 GFLOP/s per Cascade Lake core in L1, the FMA peak.
- Edge tiles run the micro-kernel into a scratch tile instead of a scalar
  loop, which took 40% of a 512^2 product once MR=6 left a 4-row edge in
  every 256-row block.
- The kernel prefetches its C tile on entry and the A panel eight steps
  ahead: serial 768^2 27.5 -> 34.7 GFLOP/s. An MC/KC sweep changed nothing.
- MatMulP's shared B block is packed by all workers instead of one while
  the others wait (+30-70% in parallel); blocks under 64 Ki elements stay
  serial, where the launch cost more than it saved on M4.
- The GEMM reads its operands through a strided view (kernels.Mat), so a
  transposed, sliced, reversed or broadcast 2-D operand is packed in place
  rather than materialised first. That is most of Inner's gain.

Tests pass on real amd64 (Zen 3, FMA path), arm64, ppc64le, riscv64 and
loong64 hardware; s390x's host did not answer and stays on the qemu lane.
Coverage is 100% on a real FMA host (the FMA branch is unreachable under
Rosetta). New tests: the SSE2 fallback forced on FMA hardware, the
parallel-pack split, and products of strided views against their
contiguous copies; both new tests were checked to fail on a planted bug.

docs/perf.md gets the real-hardware amd64 section, and comments that were
no longer true are corrected: the GEMM is bit-identical to the ikj oracle
only for k <= KC (above it the block partials regroup the sum), and gc does
not auto-vectorise dotRange.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 0490f55 into main Oct 4, 2026
12 checks passed
@tannevaled
tannevaled deleted the perf/amd64-fma-gemm branch October 4, 2026 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant