Skip to content

perf(gemm): read a row-major A in place (BLIS sup kernels) - #77

Merged
tannevaled merged 2 commits into
mainfrom
perf/gemm-unpacked-a
Oct 27, 2026
Merged

tannevaled merged 2 commits into
mainfrom
perf/gemm-unpacked-a

Conversation

@tannevaled

@tannevaled tannevaled commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

A row-major A (unit column stride, positive row stride) is now read in place by unpacked-A GEMM kernels: the "rv" gemmsup kernels of BLIS, kernels/haswell/3/sup/bli_gemmsup_rv_haswell_asm_s6x16m.c. B stays packed, and there is one kernel per tile height.

  • amd64 (FMA): gemmUA<mr>x16FMA (float32) and gemmUA<mr>x8FMA (float64), each A value broadcast from its row through a scaled index.
  • arm64: gemmUA<mr>x8S and gemmUA<mr>x8D. One vector load takes 4 floats (2 doubles) per row, and the following k steps read them by lane with the by-element FMLA. A load-and-replicate (VLD1R) version was 26–78% slower on N1 and was replaced.
MatMul vs v0.26.1, abbench 10 rounds, 1 core 64² 256² 1024²
Zen 3 float32 −28% −9.5% −3.9%
Zen 3 float64 −17% −5.5% ~
N1 float32 −20% −7.7% ~
N1 float64 −12% −4.5% ~

Against NumPy 2.5.4 (one core) we are now at 1.03–1.20× its time, and 0.97× for float64 at 1024² on Zen 3. The remaining gap stays listed as an open defect in docs/perf.md.

Sabotage check: amd64 row 1 reading row 2, and arm64 lane 3 reading lane 0. CI failed in arch amd64, arch arm64, and the macOS, ubuntu and windows test jobs, while every target without these kernels passed. The sabotage commit has been removed.

Security: the wrappers bounds-check the whole reach, and the path is limited to a unit column stride and a positive row stride. The kernels are fenced for every tile height and two row strides. The audit entry is in SECURITY.md.

🤖 Generated with Claude Code

tannevaled and others added 2 commits October 27, 2026 12:40
Small products spent a fifth of their time packing A. A row-major A
(unit column stride, positive row stride) is now read in place by
unpacked-A kernels, the "rv" gemmsup kernels of BLIS
(kernels/haswell/3/sup/bli_gemmsup_rv_haswell_asm_s6x16m.c). B stays
packed, so the last panel is still zero-padded, and there is one
kernel per tile height, so the bottom rows of a band never read past
A's last row.

- amd64 (FMA): gemmUA<mr>x16FMA (float32) and gemmUA<mr>x8FMA
  (float64), mr 1..6. Each A value is broadcast straight from its row
  through a scaled index.
- arm64: gemmUA<mr>x8S (mr 1..8) and gemmUA<mr>x8D (mr 1..4). One
  vector load takes 4 floats (2 doubles) of a row, and the following k
  steps read them by lane with the by-element FMLA. A first version
  using load-and-replicate (VLD1R) was 26-78% SLOWER on a Neoverse N1
  and was replaced.

abbench, 10 rounds, one core, against v0.26.1 (64²/256²/1024²):
  Zen 3  float32 -28/-9.5/-3.9%, float64 -17/-5.5/~
  N1     float32 -20/-7.7/~,     float64 -12/-4.5/~

Tests: every GEMM case runs on both A paths (unpackA), plus an A view
with an offset and lda != k, for float32 and float64. The new kernels
are fenced for every height and two lda values. The SSE2 fallback test
pins the packed path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tannevaled
tannevaled force-pushed the perf/gemm-unpacked-a branch from d96fee7 to b0870af Compare October 27, 2026 11:51
@tannevaled
tannevaled marked this pull request as ready for review October 27, 2026 11:51
@tannevaled
tannevaled merged commit 1a1c6b1 into main Oct 27, 2026
12 checks passed
@tannevaled
tannevaled deleted the perf/gemm-unpacked-a branch October 27, 2026 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant