From 12304cc8efab885bb80142dd1f91945d8512d6c2 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Mon, 26 Oct 2026 12:07:41 +0100 Subject: [PATCH] perf: RVV Max/Min (float64, float32); fix a small-n RVV sum; run RVV in CI - RVV extremum kernels: vfmax.vv/vfmin.vv into a tail-undisturbed group, a vmfne.vv + vcpop.m NaN count (fmax drops NaN), vfredmax.vs at the end. SpacemiT X60 (abbench): Max float64 -89% / -83%, float32 -93% / -90%; Sum controls flat. A sabotaged kernel (no NaN count) fails the sequential-oracle test. - Defect, found by running the root suite on a real X60: MeanAxis(1) of int64 [MaxInt64, MinInt64, -3] was 0, NumPy -1. The strip-mined sum took 3 elements as one vector and its unordered reduction cancelled MaxInt64 + (-3). Below 8 elements the sum is sequential now, as on the other targets. - CI: the riscv64 qemu lane ran without V, so no RVV kernel had ever run in CI. It now uses QEMU_CPU=rv64,v=true,vlen=256 and sets NDARRAY_REQUIRE_RVV, which TestRVVWhenRequired turns into a failure if the CPU lacks V. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 11 +- README.md | 2 +- SECURITY.md | 17 +++ docs/perf.md | 36 +++++- docs/plan-ndarray.md | 6 +- internal/kernels/asmgen/riscv64/gen.go | 61 +++++++++ internal/kernels/extrema32_generic.go | 2 +- internal/kernels/extrema32_riscv64.go | 24 ++++ internal/kernels/reduce_riscv64.go | 33 ++++- internal/kernels/rvv_required_riscv64_test.go | 18 +++ internal/kernels/sum_riscv64.s | 116 ++++++++++++++++++ 11 files changed, 315 insertions(+), 11 deletions(-) create mode 100644 internal/kernels/extrema32_riscv64.go create mode 100644 internal/kernels/rvv_required_riscv64_test.go diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index db5bff6..46db1ad 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -116,7 +116,10 @@ jobs: fail-fast: false matrix: include: - - { goarch: riscv64, qemu: qemu-riscv64-static } + # V enabled: qemu's default riscv64 CPU has none, and the RVV kernels + # went untested here until 2026-10-26. NDARRAY_REQUIRE_RVV makes a + # CPU without V fail the lane (TestRVVWhenRequired). + - { goarch: riscv64, qemu: qemu-riscv64-static, cpu: "rv64,v=true,vlen=256", require: NDARRAY_REQUIRE_RVV } - { goarch: loong64, qemu: qemu-loongarch64-static } - { goarch: ppc64le, qemu: qemu-ppc64le-static } - { goarch: s390x, qemu: qemu-s390x-static } @@ -164,7 +167,11 @@ jobs: fi # Printed on purpose: without it, "ran under emulation" and "skipped" are # indistinguishable from a green tick. - echo "running ${{ matrix.goarch }} under $q" + if [ -n "${{ matrix.cpu }}" ]; then + export QEMU_CPU="${{ matrix.cpu }}" + export ${{ matrix.require }}=1 + fi + echo "running ${{ matrix.goarch }} under $q (QEMU_CPU=${QEMU_CPU:-default})" GOARCH=${{ matrix.goarch }} go test -exec="$q" ./... # 386 runs natively on the amd64 runner (the kernel executes 32-bit x86 diff --git a/README.md b/README.md index 534495d..d606f15 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ `ZerosOf`, `FromSlice`, typed access with `Data[T]`/`Item[T]`, weak Go scalars with `Scalar`). Float64 is the default and the most SIMD-accelerated one. Float32 has SIMD `Sum`, `Add`/`Sub`/`Mul`/`Div` and `Sqrt` on every - 64-bit target, and `Max`/`Min` on amd64 and arm64, bit-identical to its + 64-bit target, and `Max`/`Min` on amd64, arm64 and riscv64, bit-identical to its scalar loops; its `MatMul` runs on the float64 GEMM (accumulated in float64, rounded once) ([docs/design-dtypes.md](docs/design-dtypes.md)). diff --git a/SECURITY.md b/SECURITY.md index 7332799..89e8059 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -66,6 +66,23 @@ standard library's known vulnerabilities; `govulncheck ./...` reports none. ## Audit log +**2026-10-26 (v0.22.0).** + +- **Correctness defect.** On riscv64 with V, `MeanAxis(1)` of the int64 row + `[MaxInt64, MinInt64, -3]` was 0, NumPy's -1. The strip-mined float64 sum + took even 3 elements as one vector, and its unordered reduction cancelled + `MaxInt64 + (-3)` against `MinInt64`. Below 8 elements the sum is now + sequential, as on the other targets. Found by running the root suite on a + real SpacemiT X60. +- **CI gap behind it.** The riscv64 qemu lane ran on qemu's default CPU, + which has no V, so the RVV kernels had never run in CI. It now enables V + and sets `NDARRAY_REQUIRE_RVV`, so a CPU without V fails the lane + (`TestRVVWhenRequired`). +- **New kernels.** The riscv64 extremum kernels (`maxRVV`, `minRVV`, + `max32RVV`, `min32RVV`) read `len(a)` elements of a non-empty slice, behind + the `useRVV` gate. They are fenced and tested against the sequential oracle + on the X60. A sabotaged one, which drops its NaN count, fails that test. + **2026-10-26 (v0.20.0).** The float32 extremum kernels (`max32SSE2`, `min32SSE2`, `max32NEON`, `min32NEON`) read `len(a)` elements from a non-empty slice. The Go wrappers pass `len(a)` and never an empty slice diff --git a/docs/perf.md b/docs/perf.md index 69e71a4..636387f 100644 --- a/docs/perf.md +++ b/docs/perf.md @@ -1048,7 +1048,41 @@ Still to do: assemblers have no VSX float compare (for the NaN mask), and on loong64 no vector float max, min or compare at all. - s390x (`VFMAXSB` mode 1) waits for a z machine to test on. -- riscv64 (`vfmax.vv` with a `vmfne.vv` NaN mask) is next. +- riscv64: done in v0.22.0, below. + +### riscv64 `Max`/`Min` and a sum defect (2026-10-26) + +The RVV kernels for float64 and float32 work in three steps: + +- **Accumulate:** each strip is folded into a register group with `vfmax.vv` + (or `vfmin.vv`). The tail-undisturbed policy keeps a short last strip from + touching the other lanes. +- **Count NaNs:** RISC-V's `fmax` returns the non-NaN operand, so each strip + also counts its NaN lanes (`vmfne.vv x, x`, then `vcpop.m`), and a non-zero + count gives NaN. +- **Fold:** `vfredmax.vs` reduces the group at the end. `fmax` already orders + −0 below +0. + +On a SpacemiT X60 (`abbench`, 6 rounds, alternated): + +| `Max`, 2¹⁴ / 2²⁰ | change | +|---|---:| +| float64 | **−89% / −83%** | +| float32 | **−93% / −90%** | +| `Sum` (controls) | ~ | + +Running the root suite on that real X60 found a defect: +`MeanAxis(1)` of the int64 row `[MaxInt64, MinInt64, −3]` was 0, NumPy's −1. + +- **Cause:** the strip-mined sum kernel took even 3 elements as one vector, + and its unordered reduction added `MaxInt64 + (−3)` first. In float64 that + rounds back to `MaxInt64` and cancels. +- **Fix:** below 8 elements the sum is now sequential, the order NumPy and the + other targets' kernels (whose scalar tails take n < 8 whole) use there. +- **Why CI never saw it:** its riscv64 lane runs under qemu, whose default CPU + has no V, so no RVV kernel had ever run in CI. The lane now enables V + (`QEMU_CPU=rv64,v=true,vlen=256`). It also sets `NDARRAY_REQUIRE_RVV`, so a + CPU without V fails the lane instead of passing on the scalar paths. ### float32 `MatMul` on the float64 GEMM (2026-10-26) diff --git a/docs/plan-ndarray.md b/docs/plan-ndarray.md index e528c7d..8224633 100644 --- a/docs/plan-ndarray.md +++ b/docs/plan-ndarray.md @@ -157,9 +157,9 @@ suite (plus the per-arch CI jobs already wired in `.github/workflows/ci.yml`). `fft` bridge shipped in v0.10.0 (package `fft`, over go-fft). float32 has SIMD kernels on all six 64-bit targets for `Sum` (v0.14.0–v0.17.0) and for `Add`/`Sub`/`Mul`/`Div` (v0.18.0) and `Sqrt` (v0.19.0). Still pure Go - for float32: `Max`/`Min` on ppc64le, loong64, s390x and riscv64 (v0.20.0 - vectorised them on amd64 and arm64 and stopped converting to float64 - everywhere), and the transcendental ufuncs. `MatMul` runs on the float64 + for float32: `Max`/`Min` on ppc64le, loong64 and s390x (v0.20.0 vectorised + them on amd64 and arm64 and stopped converting to float64 everywhere, + v0.22.0 on riscv64 for float32 and float64), and the transcendental ufuncs. `MatMul` runs on the float64 GEMM since v0.21.0, 77–90% faster than the triple loop; a native float32 GEMM is still to write. - **Phase 3 — broadcasting ufuncs — MOSTLY DONE.** Broadcasting arithmetic, diff --git a/internal/kernels/asmgen/riscv64/gen.go b/internal/kernels/asmgen/riscv64/gen.go index b859dd5..f8ae17b 100644 --- a/internal/kernels/asmgen/riscv64/gen.go +++ b/internal/kernels/asmgen/riscv64/gen.go @@ -39,6 +39,16 @@ func main() { f.Add(gemmKernel()) f.Add(blocks8Kernel()) f.Add(sqrt32Kernel()) + for _, k := range []struct { + name string + bits int + vop, rd string + }{ + {"maxRVV", 64, "VFMAXVV", "VFREDMAXVS"}, {"minRVV", 64, "VFMINVV", "VFREDMINVS"}, + {"max32RVV", 32, "VFMAXVV", "VFREDMAXVS"}, {"min32RVV", 32, "VFMINVV", "VFREDMINVS"}, + } { + f.Add(extremumKernel(k.name, k.bits, k.vop, k.rd)) + } if err := os.WriteFile("sum_riscv64.s", []byte(f.String()), 0o644); err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) @@ -130,6 +140,57 @@ func binKernel(name, op string) *emit.Function { return b.Func() } +// extremumKernel builds NAME(a *floatN, n int) floatN for n >= 1, the +// NaN-propagating maximum (or minimum), N = bits: +// +// - The accumulator group V16 (LMUL=4) is seeded with a[0] in every lane. +// - Each strip of vl elements is folded in with vfmax.vv/vfmin.vv under +// the tail-undisturbed policy, so a short last strip leaves the other +// lanes alone. +// - At the end the group is reduced with vfredmax.vs/vfredmin.vs. +// +// RISC-V's fmax/fmin order -0 below +0, as Go does, but return the non-NaN +// operand when one is NaN. So each strip also counts its NaN lanes (vmfne.vv +// x, x, then vcpop.m), and a non-zero count makes the result a quiet NaN. +func extremumKernel(name string, bits int, vop, red string) *emit.Function { + ret, mov, ew, shift, fmv := riscv64.Float64, "MOVD", "E64", 3, "FMVDX" + nan := "$0x7ff8000000000000" + if bits == 32 { + ret, mov, ew, shift, fmv = riscv64.Float32, "MOVF", "E32", 2, "FMVWX" + nan = "$0x7fc00000" + } + sig := riscv64.Layout([]string{"a", "n"}, []riscv64.Type{riscv64.Ptr, riscv64.Int64}, + []string{"ret"}, []riscv64.Type{ret}) + b := riscv64.NewFunc(name, sig, 0) + b.LoadArg("a", "X10").LoadArg("n", "X11") + b.Raw("%s (X10), F0", mov). + Raw("VSETVLI X0, %s, M4, TA, MA, X16", ew). // vl = VLMAX + Raw("VFMVVF F0, V16"). + Raw("MOV $0, X17"). // NaN lanes seen + Raw("loop:"). + Raw("VSETVLI X11, %s, M4, TU, MA, X14", ew). + Raw("VLE%dV (X10), V8", bits). + Raw("VMFNEVV V8, V8, V4"). // V4 = the NaN lanes of the strip + Raw("VCPOPM V4, X15"). + Raw("ADD X15, X17, X17"). + Raw("%s V8, V16, V16", vop). + Raw("SLLI $%d, X14, X18", shift). + Raw("ADD X18, X10, X10"). + Raw("SUB X14, X11, X11"). + Raw("BNEZ X11, loop"). + Raw("VSETVLI X0, %s, M4, TA, MA, X16", ew). + Raw("VFMVSF F0, V1"). + Raw("%s V1, V16, V1", red). // V1[0] = extremum of V1[0] and every lane of V16 + Raw("VFMVFS V1, F0"). + Raw("BEQZ X17, done"). + Raw("MOV %s, X5", nan). + Raw("%s X5, F0", fmv). + Raw("done:"). + StoreRet("F0", "ret"). + Ret() + return b.Func() +} + // sqrt32Kernel builds sqrt32RVV(dst, src *float32, n int) for n >= 1: // vfsqrt.v on 32-bit elements, strip-mined; the correctly rounded float32 // square root, bit-identical to float32(math.Sqrt(float64(x))). diff --git a/internal/kernels/extrema32_generic.go b/internal/kernels/extrema32_generic.go index 69ec906..f7baef8 100644 --- a/internal/kernels/extrema32_generic.go +++ b/internal/kernels/extrema32_generic.go @@ -1,4 +1,4 @@ -//go:build !amd64 && !arm64 +//go:build !amd64 && !arm64 && !riscv64 package kernels diff --git a/internal/kernels/extrema32_riscv64.go b/internal/kernels/extrema32_riscv64.go new file mode 100644 index 0000000..760768b --- /dev/null +++ b/internal/kernels/extrema32_riscv64.go @@ -0,0 +1,24 @@ +package kernels + +// max32 runs the RVV kernel (sum_riscv64.s: vfmax.vv with a vmfne NaN count) +// when the CPU reports V, the four-chain Go loop otherwise. +func max32(a []float32) float32 { + if !useRVV { + return max32Go(a) + } + return max32RVV(&a[0], len(a)) +} + +// min32 is the dual of max32, with vfmin.vv. +func min32(a []float32) float32 { + if !useRVV { + return min32Go(a) + } + return min32RVV(&a[0], len(a)) +} + +//go:noescape +func max32RVV(a *float32, n int) float32 + +//go:noescape +func min32RVV(a *float32, n int) float32 diff --git a/internal/kernels/reduce_riscv64.go b/internal/kernels/reduce_riscv64.go index c456ad9..cf39bde 100644 --- a/internal/kernels/reduce_riscv64.go +++ b/internal/kernels/reduce_riscv64.go @@ -32,8 +32,15 @@ func rvvFromAuxv() bool { return err == nil && hwcapFrom(auxv)&hwcapV != 0 } +// sumSIMD is the RVV sum from 8 elements up. Below that it is the sequential +// sum, the order NumPy's pairwise sum and the other targets' kernels (whose +// scalar tails take n < 8 whole) use there. The strip-mined kernel would take +// even 3 elements as one vector, and its unordered reduction regrouped +// [MaxInt64, MinInt64, -3] as float64 so that max + (-3) rounded back to max +// and cancelled: the mean was 0, NumPy's -1. Seen on a real SpacemiT X60, +// never under CI's qemu, whose default CPU has no V. func sumSIMD(a []float64) float64 { - if !useRVV || len(a) == 0 { + if !useRVV || len(a) < 8 { return Sum(a) } return sumRVV(&a[0], len(a)) @@ -48,8 +55,28 @@ func sqrtSIMD(dst, src []float64) { sqrtRVV(&dst[0], &src[0], len(dst)) } -func maxSIMD(a []float64) float64 { return maxUnrolled(a) } -func minSIMD(a []float64) float64 { return minUnrolled(a) } +// maxSIMD / minSIMD are the NaN-propagating extrema of a (non-empty): the RVV +// kernels (vfmax/vfmin with a vmfne NaN count) when the CPU reports V, the +// four-chain scalar reducers otherwise. +func maxSIMD(a []float64) float64 { + if !useRVV { + return maxUnrolled(a) + } + return maxRVV(&a[0], len(a)) +} + +func minSIMD(a []float64) float64 { + if !useRVV { + return minUnrolled(a) + } + return minRVV(&a[0], len(a)) +} + +//go:noescape +func maxRVV(a *float64, n int) float64 + +//go:noescape +func minRVV(a *float64, n int) float64 func addBin(dst, a, b []float64) { if !useRVV || len(dst) == 0 { diff --git a/internal/kernels/rvv_required_riscv64_test.go b/internal/kernels/rvv_required_riscv64_test.go new file mode 100644 index 0000000..1599e05 --- /dev/null +++ b/internal/kernels/rvv_required_riscv64_test.go @@ -0,0 +1,18 @@ +package kernels + +import ( + "os" + "testing" +) + +// TestRVVWhenRequired fails when NDARRAY_REQUIRE_RVV is set and the CPU does +// not report V. CI sets it on the riscv64 qemu lane, which runs with V +// enabled: without this check, a qemu whose CPU lost V would take the scalar +// paths and the RVV kernels would go untested behind a green tick, as they +// did until 2026-10-26. +func TestRVVWhenRequired(t *testing.T) { + t.Logf("useRVV = %v", useRVV) + if os.Getenv("NDARRAY_REQUIRE_RVV") != "" && !useRVV { + t.Fatal("NDARRAY_REQUIRE_RVV is set but the CPU does not report V: the RVV kernels did not run") + } +} diff --git a/internal/kernels/sum_riscv64.s b/internal/kernels/sum_riscv64.s index f2760a1..03c43bc 100644 --- a/internal/kernels/sum_riscv64.s +++ b/internal/kernels/sum_riscv64.s @@ -305,3 +305,119 @@ TEXT ·sqrt32RVV(SB), NOSPLIT, $0-24 BNEZ X13, loop RET +TEXT ·maxRVV(SB), NOSPLIT, $0-24 + MOV a+0(FP), X10 + MOV n+8(FP), X11 + MOVD (X10), F0 + VSETVLI X0, E64, M4, TA, MA, X16 + VFMVVF F0, V16 + MOV $0, X17 + loop: + VSETVLI X11, E64, M4, TU, MA, X14 + VLE64V (X10), V8 + VMFNEVV V8, V8, V4 + VCPOPM V4, X15 + ADD X15, X17, X17 + VFMAXVV V8, V16, V16 + SLLI $3, X14, X18 + ADD X18, X10, X10 + SUB X14, X11, X11 + BNEZ X11, loop + VSETVLI X0, E64, M4, TA, MA, X16 + VFMVSF F0, V1 + VFREDMAXVS V1, V16, V1 + VFMVFS V1, F0 + BEQZ X17, done + MOV $0x7ff8000000000000, X5 + FMVDX X5, F0 + done: + MOVD F0, ret+16(FP) + RET + +TEXT ·minRVV(SB), NOSPLIT, $0-24 + MOV a+0(FP), X10 + MOV n+8(FP), X11 + MOVD (X10), F0 + VSETVLI X0, E64, M4, TA, MA, X16 + VFMVVF F0, V16 + MOV $0, X17 + loop: + VSETVLI X11, E64, M4, TU, MA, X14 + VLE64V (X10), V8 + VMFNEVV V8, V8, V4 + VCPOPM V4, X15 + ADD X15, X17, X17 + VFMINVV V8, V16, V16 + SLLI $3, X14, X18 + ADD X18, X10, X10 + SUB X14, X11, X11 + BNEZ X11, loop + VSETVLI X0, E64, M4, TA, MA, X16 + VFMVSF F0, V1 + VFREDMINVS V1, V16, V1 + VFMVFS V1, F0 + BEQZ X17, done + MOV $0x7ff8000000000000, X5 + FMVDX X5, F0 + done: + MOVD F0, ret+16(FP) + RET + +TEXT ·max32RVV(SB), NOSPLIT, $0-20 + MOV a+0(FP), X10 + MOV n+8(FP), X11 + MOVF (X10), F0 + VSETVLI X0, E32, M4, TA, MA, X16 + VFMVVF F0, V16 + MOV $0, X17 + loop: + VSETVLI X11, E32, M4, TU, MA, X14 + VLE32V (X10), V8 + VMFNEVV V8, V8, V4 + VCPOPM V4, X15 + ADD X15, X17, X17 + VFMAXVV V8, V16, V16 + SLLI $2, X14, X18 + ADD X18, X10, X10 + SUB X14, X11, X11 + BNEZ X11, loop + VSETVLI X0, E32, M4, TA, MA, X16 + VFMVSF F0, V1 + VFREDMAXVS V1, V16, V1 + VFMVFS V1, F0 + BEQZ X17, done + MOV $0x7fc00000, X5 + FMVWX X5, F0 + done: + MOVF F0, ret+16(FP) + RET + +TEXT ·min32RVV(SB), NOSPLIT, $0-20 + MOV a+0(FP), X10 + MOV n+8(FP), X11 + MOVF (X10), F0 + VSETVLI X0, E32, M4, TA, MA, X16 + VFMVVF F0, V16 + MOV $0, X17 + loop: + VSETVLI X11, E32, M4, TU, MA, X14 + VLE32V (X10), V8 + VMFNEVV V8, V8, V4 + VCPOPM V4, X15 + ADD X15, X17, X17 + VFMINVV V8, V16, V16 + SLLI $2, X14, X18 + ADD X18, X10, X10 + SUB X14, X11, X11 + BNEZ X11, loop + VSETVLI X0, E32, M4, TA, MA, X16 + VFMVSF F0, V1 + VFREDMINVS V1, V16, V1 + VFMVFS V1, F0 + BEQZ X17, done + MOV $0x7fc00000, X5 + FMVWX X5, F0 + done: + MOVF F0, ret+16(FP) + RET +