Repository navigation
perf: AVX2/FMA GEMM on amd64, strided operands, parallel B packing - #11
Merged
Merged
Conversation
amd64 had never been measured on real hardware, and its GEMM micro-kernel was a 4x4 SSE2 tile without FMA. On a Zen 3 (16 cores, pinned) against NumPy 2.5.3 + OpenBLAS 0.3.34, interleaved with the previous version: MatMul 1024^2 21.87 -> 8.05 ms x2.72 (5.9x OpenBLAS 1 thread, 0.60x 16) MatMul 512^2 3.69 -> 1.56 ms x2.36 Inner 512^2 5.87 -> 1.47 ms x4.00 (0.96x OpenBLAS 16 threads) 1024x256.256x1024 5.72 -> 2.71 ms x2.11 - AVX2/FMA 6x8 micro-kernel (the BLIS Haswell shape, 12 YMM accumulators), generated by go-asmgen and chosen at run time by its new FMA probe (go-asmgen v0.11.0); the GOAMD64=v1 fallback is two SSE2 6x4 tiles over the same packing. 43 GFLOP/s per Cascade Lake core in L1, the FMA peak. - Edge tiles run the micro-kernel into a scratch tile instead of a scalar loop, which took 40% of a 512^2 product once MR=6 left a 4-row edge in every 256-row block. - The kernel prefetches its C tile on entry and the A panel eight steps ahead: serial 768^2 27.5 -> 34.7 GFLOP/s. An MC/KC sweep changed nothing. - MatMulP's shared B block is packed by all workers instead of one while the others wait (+30-70% in parallel); blocks under 64 Ki elements stay serial, where the launch cost more than it saved on M4. - The GEMM reads its operands through a strided view (kernels.Mat), so a transposed, sliced, reversed or broadcast 2-D operand is packed in place rather than materialised first. That is most of Inner's gain. Tests pass on real amd64 (Zen 3, FMA path), arm64, ppc64le, riscv64 and loong64 hardware; s390x's host did not answer and stays on the qemu lane. Coverage is 100% on a real FMA host (the FMA branch is unreachable under Rosetta). New tests: the SSE2 fallback forced on FMA hardware, the parallel-pack split, and products of strided views against their contiguous copies; both new tests were checked to fail on a planted bug. docs/perf.md gets the real-hardware amd64 section, and comments that were no longer true are corrected: the GEMM is bit-identical to the ikj oracle only for k <= KC (above it the block partials regroup the sum), and gc does not auto-vectorise dotRange. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
amd64 had never been measured on real hardware, and its GEMM micro-kernel
was a 4x4 SSE2 tile without FMA. On a Zen 3 (16 cores, pinned) against
NumPy 2.5.3 + OpenBLAS 0.3.34, interleaved with the previous version:
MatMul 1024^2 21.87 -> 8.05 ms x2.72 (5.9x OpenBLAS 1 thread, 0.60x 16)
MatMul 512^2 3.69 -> 1.56 ms x2.36
Inner 512^2 5.87 -> 1.47 ms x4.00 (0.96x OpenBLAS 16 threads)
1024x256.256x1024 5.72 -> 2.71 ms x2.11
generated by go-asmgen and chosen at run time by its new FMA probe
(go-asmgen v0.11.0); the GOAMD64=v1 fallback is two SSE2 6x4 tiles over
the same packing. 43 GFLOP/s per Cascade Lake core in L1, the FMA peak.
loop, which took 40% of a 512^2 product once MR=6 left a 4-row edge in
every 256-row block.
ahead: serial 768^2 27.5 -> 34.7 GFLOP/s. An MC/KC sweep changed nothing.
the others wait (+30-70% in parallel); blocks under 64 Ki elements stay
serial, where the launch cost more than it saved on M4.
transposed, sliced, reversed or broadcast 2-D operand is packed in place
rather than materialised first. That is most of Inner's gain.
Tests pass on real amd64 (Zen 3, FMA path), arm64, ppc64le, riscv64 and
loong64 hardware; s390x's host did not answer and stays on the qemu lane.
Coverage is 100% on a real FMA host (the FMA branch is unreachable under
Rosetta). New tests: the SSE2 fallback forced on FMA hardware, the
parallel-pack split, and products of strided views against their
contiguous copies; both new tests were checked to fail on a planted bug.
docs/perf.md gets the real-hardware amd64 section, and comments that were
no longer true are corrected: the GEMM is bit-identical to the ikj oracle
only for k <= KC (above it the block partials regroup the sum), and gc does
not auto-vectorise dotRange.
🤖 Generated with Claude Code