diff --git a/README.md b/README.md index d292a0e..0100c59 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ `ZerosOf`, `FromSlice`, typed access with `Data[T]`/`Item[T]`, weak Go scalars with `Scalar`). Float64 is the default and the most SIMD-accelerated one. Float32 has SIMD `Sum`, `Add`/`Sub`/`Mul`/`Div` and `Sqrt` on every - 64-bit target, and `Max`/`Min` on every 64-bit target but loong64, bit-identical to its + 64-bit target, and `Max`/`Min` on every 64-bit target, bit-identical to its scalar loops; its `MatMul` runs on the float64 GEMM (accumulated in float64, rounded once) ([docs/design-dtypes.md](docs/design-dtypes.md)). diff --git a/SECURITY.md b/SECURITY.md index f7ab405..8386786 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -66,6 +66,14 @@ standard library's known vulnerabilities; `govulncheck ./...` reports none. ## Audit log +**2026-10-26 (v0.25.0).** The loong64 extremum kernels (`maxLSX`, `minLSX`, +`max32LSX`, `min32LSX`) read the whole blocks of `a` behind the `useLASX` +gate. They write one 16-byte vector to an `out` array on the wrapper's stack, +and read two 16-byte read-only constants. A test gap was found and closed: no +extremum test used a negative NaN, so a kernel whose NaN test failed could +pass, as a sabotaged one did. Both signs are tested now. The sabotage then +failed CI's loong64 lane, and every other kernel passed. + **2026-10-26 (v0.24.0).** The ppc64le extremum kernels (`maxVSX`, `minVSX`, `max32VSX`, `min32VSX`) read the whole blocks of `a` (`len(a) &^ 7` or `&^ 15`, never 0, since the wrapper takes the scalar path below a block). They diff --git a/docs/perf.md b/docs/perf.md index 6b8ce57..4a025a0 100644 --- a/docs/perf.md +++ b/docs/perf.md @@ -1045,8 +1045,38 @@ Go's NaN-correct `max` is scalar there. Still to do: - ppc64le: done in v0.24.0, below, without a registry entry. -- loong64 needs registry entries in go-asmgen first: Go's assembler has no - vector float max, min or compare at all. +- loong64: done in v0.25.0, below, without a registry entry. + +### loong64 `Max`/`Min`, and negative NaNs (2026-10-26) + +Go's loong64 assembler has no vector float max, min or compare, so the LSX +kernels work entirely in integer arithmetic: + +- **Order:** `t = x XOR ((x >>a N−1) >>l 1)` flips the magnitude bits of the + negatives. The signed order of `t` is the float order, with −0 below +0 + (Go's rule), and the map is its own inverse. +- **Extremum:** `max(acc, t) = acc XOR ((acc XOR t) AND (acc < t))`, with + `vslt`. +- **NaN:** a lane is NaN iff `(x AND abs) >u +Inf's bits` (`vsltu`), + OR-accumulated. + +**What the sabotage run found.** A sabotaged kernel, with the NaN mask of +`max32LSX`'s first lane dropped, *passed* every test in CI's loong64 lane. In +that integer order a **positive** NaN sorts above +Inf, so a max finds it +without any NaN test. Every NaN in the tests was `math.NaN()`, which is +positive. A **negative** NaN sorts below everything, and only the NaN test +catches it. That is no corner case: x86's default NaN (from `0×∞` or `∞−∞`) +is negative. + +The extremum tests now cover both signs, for float32 and float64 on every +architecture. With them the sabotage failed CI's loong64 lane +(`special Max32 n=16: -0, sequential NaN`), and every other architecture's +kernels passed. + +**Validated under emulation only.** The kernels pass on qemu `la464` (LSX and +LASX, required since #67). The Loongson they would be measured on +(cfarm401) accepted TCP but never sent its SSH banner on 2026-10-26, so their +speed is not measured yet. ### ppc64le `Max`/`Min` (2026-10-26) diff --git a/docs/plan-ndarray.md b/docs/plan-ndarray.md index 062e208..9337eb8 100644 --- a/docs/plan-ndarray.md +++ b/docs/plan-ndarray.md @@ -157,9 +157,8 @@ suite (plus the per-arch CI jobs already wired in `.github/workflows/ci.yml`). `fft` bridge shipped in v0.10.0 (package `fft`, over go-fft). float32 has SIMD kernels on all six 64-bit targets for `Sum` (v0.14.0–v0.17.0) and for `Add`/`Sub`/`Mul`/`Div` (v0.18.0) and `Sqrt` (v0.19.0). Still pure Go - for float32: `Max`/`Min` on loong64 (v0.20.0 vectorised them on amd64 and - arm64 and stopped converting to float64 everywhere, v0.22.0 on riscv64, - v0.23.0 on s390x, v0.24.0 on ppc64le for float32 and float64), and the transcendental ufuncs. `MatMul` runs on the float64 + for float32: nothing in `Max`/`Min` any more (v0.20.0 to v0.25.0 + vectorised them on all six 64-bit targets, for float32 and float64), and the transcendental ufuncs. `MatMul` runs on the float64 GEMM since v0.21.0, 77–90% faster than the triple loop; a native float32 GEMM is still to write. - **Phase 3 — broadcasting ufuncs — MOSTLY DONE.** Broadcasting arithmetic, diff --git a/internal/kernels/asmgen/loong64/gen.go b/internal/kernels/asmgen/loong64/gen.go index 0dd17d0..6d39ca4 100644 --- a/internal/kernels/asmgen/loong64/gen.go +++ b/internal/kernels/asmgen/loong64/gen.go @@ -43,6 +43,14 @@ func main() { f.Add(gemmKernel()) f.Add(blocks8Kernel()) f.Add(sqrt32Kernel()) + abs64 := f.Data("abs64", lanes(8, 0x7fffffffffffffff)) + inf64 := f.Data("inf64", lanes(8, 0x7ff0000000000000)) + abs32 := f.Data("abs32", lanes(4, 0x7fffffff)) + inf32 := f.Data("inf32", lanes(4, 0x7f800000)) + f.Add(extremumKernel("maxLSX", 64, true, abs64, inf64)) + f.Add(extremumKernel("minLSX", 64, false, abs64, inf64)) + f.Add(extremumKernel("max32LSX", 32, true, abs32, inf32)) + f.Add(extremumKernel("min32LSX", 32, false, abs32, inf32)) if err := os.WriteFile("sum_loong64.s", []byte(f.String()), 0o644); err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) @@ -171,6 +179,96 @@ func binKernel(name, vec, sca string) *emit.Function { return b.Func() } +// lanes is a 16-byte vector constant: v in every lane of size bytes, each lane +// little-endian. +func lanes(size int, v uint64) []byte { + b := make([]byte, 0, 16) + for len(b) < 16 { + for i := 0; i < size; i++ { + b = append(b, byte(v>>(8*i))) + } + } + return b +} + +// extremumKernel builds NAME(a *floatN, n int, out *[16]byte) int64 for n a +// positive multiple of the block (8 doubles or 16 floats), entirely in LSX +// integer arithmetic: Go's loong64 assembler has no vector float max, min or +// compare. +// +// - Order: t = x XOR ((x >>a N-1) >>l 1) flips the magnitude bits of the +// negatives, so the signed order of t is the float order, with -0 below +// +0, Go's rule. The map is its own inverse. +// - Extremum: the max of t and acc is acc XOR ((acc XOR t) AND (acc < t)) +// (vslt.d/vslt.w), and the min swaps the compare. +// - NaN: a lane is NaN iff its magnitude bits (x AND abs) exceed +Inf's +// bits as an unsigned integer (vsltu). The masks are OR-accumulated, and +// a non-zero result is returned so the wrapper can answer NaN. +// +// Four accumulators (V0-V3) are folded the same way into one vector, which is +// stored, still in the t order, at out. The Go wrapper maps it back and folds +// the lanes and the tail with Go's max/min. Registers: V4-V7 the block, +// V8-V19 scratch, V20 abs, V21 +Inf, V22 the NaN mask. +func extremumKernel(name string, bits int, isMax bool, abs, inf string) *emit.Function { + sig := loong64.Layout([]string{"a", "n", "out"}, + []loong64.Type{loong64.Ptr, loong64.Int64, loong64.Ptr}, []string{"ret"}, []loong64.Type{loong64.Int64}) + b := loong64.NewFunc(name, sig, 0) + b.LoadArg("a", "R4").LoadArg("n", "R5").LoadArg("out", "R6") + w, block := "V", 8 + if bits == 32 { + w, block = "W", 16 + } + sra, srl, slt, sltu := "VSRA"+w, "VSRL"+w, "VSLT"+w, "VSLT"+w+"U" + b.Raw("MOVV $%s+0(SB), R12", abs).Raw("VMOVQ (R12), V20"). + Raw("MOVV $%s+0(SB), R12", inf).Raw("VMOVQ (R12), V21"). + Raw("VXORV V22, V22, V22") + // sel sets acc to the extremum of acc and t, with m and d as scratch. + sel := func(acc, t, m, d int) { + if isMax { + b.Raw("%s V%d, V%d, V%d", slt, t, acc, m) // m = acc < t + } else { + b.Raw("%s V%d, V%d, V%d", slt, acc, t, m) // m = t < acc + } + b.Raw("VXORV V%d, V%d, V%d", t, acc, d). + Raw("VANDV V%d, V%d, V%d", m, d, d). + Raw("VXORV V%d, V%d, V%d", d, acc, acc) + } + // load reads one block into V4-V7, ORs its NaN lanes into V22 and leaves + // the ordered integers in V8-V11. + load := func() { + for k := 0; k < 4; k++ { + b.Raw("VMOVQ %d(R4), V%d", 16*k, 4+k) + } + for k := 0; k < 4; k++ { + b.Raw("VANDV V20, V%d, V%d", 4+k, 12+k). + Raw("%s V%d, V21, V%d", sltu, 12+k, 12+k). // +Inf < |x|: NaN + Raw("VORV V%d, V22, V22", 12+k). + Raw("%s $%d, V%d, V%d", sra, bits-1, 4+k, 8+k). + Raw("%s $1, V%d, V%d", srl, 8+k, 8+k). + Raw("VXORV V%d, V%d, V%d", 4+k, 8+k, 8+k) + } + b.Raw("ADDV $64, R4").Raw("ADDV $-%d, R5", block) + } + load() + for k := 0; k < 4; k++ { + b.Raw("VORV V%d, V%d, V%d", 8+k, 8+k, k) // the first block seeds acc + } + b.Label(name+"loop").Raw("BEQ R5, %sfold", name) + load() + for k := 0; k < 4; k++ { + sel(k, 8+k, 12+k, 16+k) + } + b.Raw("JMP %sloop", name) + b.Label(name + "fold") + sel(0, 1, 12, 16) + sel(2, 3, 13, 17) + sel(0, 2, 14, 18) + b.Raw("VMOVQ V0, (R6)"). + Raw("VMOVQ V22.V[0], R13").Raw("VMOVQ V22.V[1], R14").Raw("OR R14, R13"). + StoreRet("R13", "ret").Ret() + return b.Func() +} + // sqrt32Kernel builds sqrt32LSX(dst, src *float32, n int): vfsqrt.s over four // 128-bit registers, 16 floats per iteration, SQRTF for the tail; the // correctly rounded float32 square root, bit-identical to diff --git a/internal/kernels/extrema32_generic.go b/internal/kernels/extrema32_generic.go index f77b2a2..8012876 100644 --- a/internal/kernels/extrema32_generic.go +++ b/internal/kernels/extrema32_generic.go @@ -1,4 +1,4 @@ -//go:build !amd64 && !arm64 && !ppc64le && !riscv64 && !s390x +//go:build !amd64 && !arm64 && !loong64 && !ppc64le && !riscv64 && !s390x package kernels diff --git a/internal/kernels/extrema32_loong64.go b/internal/kernels/extrema32_loong64.go new file mode 100644 index 0000000..50ee3c3 --- /dev/null +++ b/internal/kernels/extrema32_loong64.go @@ -0,0 +1,50 @@ +package kernels + +import "math" + +// max32 reduces whole blocks of 16 with the LSX kernel on the floats' bits in +// integer order (see fromOrdered32), then folds the four lanes and the tail +// with Go's max. A NaN anywhere, reported by the kernel, is NaN. +func max32(a []float32) float32 { + n := len(a) &^ 15 + if !useLASX || n == 0 { + return max32Go(a) + } + var out [4]int32 + if max32LSX(&a[0], n, &out) != 0 { + return float32(math.NaN()) + } + m := max(max(fromOrdered32(out[0]), fromOrdered32(out[1])), max(fromOrdered32(out[2]), fromOrdered32(out[3]))) + for _, v := range a[n:] { + m = max(m, v) + } + return m +} + +// min32 is the dual of max32. +func min32(a []float32) float32 { + n := len(a) &^ 15 + if !useLASX || n == 0 { + return min32Go(a) + } + var out [4]int32 + if min32LSX(&a[0], n, &out) != 0 { + return float32(math.NaN()) + } + m := min(min(fromOrdered32(out[0]), fromOrdered32(out[1])), min(fromOrdered32(out[2]), fromOrdered32(out[3]))) + for _, v := range a[n:] { + m = min(m, v) + } + return m +} + +// fromOrdered32 undoes the kernels' order map t = x XOR ((x >>a 31) >>l 1). +func fromOrdered32(t int32) float32 { + return math.Float32frombits(uint32(t ^ int32(uint32(t>>31)>>1))) +} + +//go:noescape +func max32LSX(a *float32, n int, out *[4]int32) int64 + +//go:noescape +func min32LSX(a *float32, n int, out *[4]int32) int64 diff --git a/internal/kernels/extrema32_test.go b/internal/kernels/extrema32_test.go index 9381c77..e5914c8 100644 --- a/internal/kernels/extrema32_test.go +++ b/internal/kernels/extrema32_test.go @@ -31,6 +31,12 @@ func TestExtrema32MatchSequential(t *testing.T) { nz := float32(math.Copysign(0, -1)) inf := float32(math.Inf(1)) nan := float32(math.NaN()) + // A negative NaN too: x86's default NaN (0*Inf, Inf-Inf) has the sign bit + // set. A kernel that orders the bits as integers sorts a positive NaN above + // +Inf, so a max finds it without any NaN test; only a negative NaN shows + // whether the NaN test works. (A sabotaged loong64 NaN mask passed every + // test until this case was added.) + negNaN := math.Float32frombits(0xffc00000) check := func(name string, a []float32) { t.Helper() for _, f := range []struct { @@ -51,7 +57,7 @@ func TestExtrema32MatchSequential(t *testing.T) { check("mixed", operands32(n, 5)) for pos := 0; pos < n; pos++ { for _, fill := range []float32{nz, 0, -1, 1} { - for _, v := range []float32{nz, 0, inf, -inf, nan} { + for _, v := range []float32{nz, 0, inf, -inf, nan, negNaN} { a := make([]float32, n) for i := range a { a[i] = fill diff --git a/internal/kernels/reduce_loong64.go b/internal/kernels/reduce_loong64.go index d5b12ae..59f182f 100644 --- a/internal/kernels/reduce_loong64.go +++ b/internal/kernels/reduce_loong64.go @@ -1,6 +1,9 @@ package kernels -import "os" +import ( + "math" + "os" +) // loong64 LASX kernels generated by go-asmgen (sum_loong64.s). LASX (256-bit, // four float64 lanes) is not on every LoongArch CPU, so each wrapper uses it @@ -42,8 +45,54 @@ func sqrtSIMD(dst, src []float64) { } // max/min stay on the four-accumulator scalar reducers. -func maxSIMD(a []float64) float64 { return maxUnrolled(a) } -func minSIMD(a []float64) float64 { return minUnrolled(a) } +// maxSIMD / minSIMD reduce whole blocks of 8 with the LSX kernels +// (sum_loong64.s), which work on the floats' bits mapped to an integer order +// (see fromOrdered64) because Go's loong64 assembler has no vector float max. +// The wrapper maps the two lanes back and folds them and the tail with Go's +// max/min; a NaN anywhere, reported by the kernel, is NaN. +func maxSIMD(a []float64) float64 { + n := len(a) &^ 7 + if !useLASX || n == 0 { + return maxUnrolled(a) + } + var out [2]int64 + if maxLSX(&a[0], n, &out) != 0 { + return math.NaN() + } + m := max(fromOrdered64(out[0]), fromOrdered64(out[1])) + for _, v := range a[n:] { + m = max(m, v) + } + return m +} + +func minSIMD(a []float64) float64 { + n := len(a) &^ 7 + if !useLASX || n == 0 { + return minUnrolled(a) + } + var out [2]int64 + if minLSX(&a[0], n, &out) != 0 { + return math.NaN() + } + m := min(fromOrdered64(out[0]), fromOrdered64(out[1])) + for _, v := range a[n:] { + m = min(m, v) + } + return m +} + +// fromOrdered64 undoes the kernels' order map t = x XOR ((x >>a 63) >>l 1), +// which is its own inverse. +func fromOrdered64(t int64) float64 { + return math.Float64frombits(uint64(t ^ int64(uint64(t>>63)>>1))) +} + +//go:noescape +func maxLSX(a *float64, n int, out *[2]int64) int64 + +//go:noescape +func minLSX(a *float64, n int, out *[2]int64) int64 func addBin(dst, a, b []float64) { if !useLASX || len(dst) == 0 { diff --git a/internal/kernels/reduce_simd_test.go b/internal/kernels/reduce_simd_test.go index 0d435ac..529f314 100644 --- a/internal/kernels/reduce_simd_test.go +++ b/internal/kernels/reduce_simd_test.go @@ -183,19 +183,24 @@ func TestMaxMinSIMD(t *testing.T) { if g, w := minSIMD(a), Min(a); !bitEq(g, w) { t.Fatalf("n=%d: minSIMD=%v, Min=%v", n, g, w) } - // Inject a NaN at several positions; both must propagate it. - for _, pos := range []int{0, n / 2, n - 1} { - b := append([]float64(nil), a...) - b[pos] = math.NaN() - if g := maxSIMD(b); !math.IsNaN(g) { - t.Fatalf("n=%d NaN@%d: maxSIMD=%v, want NaN", n, pos, g) - } - if g := minSIMD(b); !math.IsNaN(g) { - t.Fatalf("n=%d NaN@%d: minSIMD=%v, want NaN", n, pos, g) - } - // And the oracle agrees it is NaN (semantics match). - if !math.IsNaN(Max(b)) || !math.IsNaN(Min(b)) { - t.Fatalf("n=%d NaN@%d: oracle did not propagate NaN", n, pos) + // Inject a NaN at several positions; both must propagate it. Both + // signs: x86's default NaN is negative, and a kernel that orders the + // bits as integers finds a positive NaN by its size alone, so only the + // negative one shows whether its NaN test works. + for _, nan := range []float64{math.NaN(), math.Float64frombits(0xfff8000000000000)} { + for _, pos := range []int{0, n / 2, n - 1} { + b := append([]float64(nil), a...) + b[pos] = nan + if g := maxSIMD(b); !math.IsNaN(g) { + t.Fatalf("n=%d NaN@%d: maxSIMD=%v, want NaN", n, pos, g) + } + if g := minSIMD(b); !math.IsNaN(g) { + t.Fatalf("n=%d NaN@%d: minSIMD=%v, want NaN", n, pos, g) + } + // And the oracle agrees it is NaN (semantics match). + if !math.IsNaN(Max(b)) || !math.IsNaN(Min(b)) { + t.Fatalf("n=%d NaN@%d: oracle did not propagate NaN", n, pos) + } } } } diff --git a/internal/kernels/sum_loong64.s b/internal/kernels/sum_loong64.s index cb279d1..1c7257a 100644 --- a/internal/kernels/sum_loong64.s +++ b/internal/kernels/sum_loong64.s @@ -651,3 +651,523 @@ qtail: qdone: RET +TEXT ·maxLSX(SB), NOSPLIT, $0-32 + MOVV a+0(FP), R4 + MOVV n+8(FP), R5 + MOVV out+16(FP), R6 + MOVV $abs64<>+0(SB), R12 + VMOVQ (R12), V20 + MOVV $inf64<>+0(SB), R12 + VMOVQ (R12), V21 + VXORV V22, V22, V22 + VMOVQ 0(R4), V4 + VMOVQ 16(R4), V5 + VMOVQ 32(R4), V6 + VMOVQ 48(R4), V7 + VANDV V20, V4, V12 + VSLTVU V12, V21, V12 + VORV V12, V22, V22 + VSRAV $63, V4, V8 + VSRLV $1, V8, V8 + VXORV V4, V8, V8 + VANDV V20, V5, V13 + VSLTVU V13, V21, V13 + VORV V13, V22, V22 + VSRAV $63, V5, V9 + VSRLV $1, V9, V9 + VXORV V5, V9, V9 + VANDV V20, V6, V14 + VSLTVU V14, V21, V14 + VORV V14, V22, V22 + VSRAV $63, V6, V10 + VSRLV $1, V10, V10 + VXORV V6, V10, V10 + VANDV V20, V7, V15 + VSLTVU V15, V21, V15 + VORV V15, V22, V22 + VSRAV $63, V7, V11 + VSRLV $1, V11, V11 + VXORV V7, V11, V11 + ADDV $64, R4 + ADDV $-8, R5 + VORV V8, V8, V0 + VORV V9, V9, V1 + VORV V10, V10, V2 + VORV V11, V11, V3 +maxLSXloop: + BEQ R5, maxLSXfold + VMOVQ 0(R4), V4 + VMOVQ 16(R4), V5 + VMOVQ 32(R4), V6 + VMOVQ 48(R4), V7 + VANDV V20, V4, V12 + VSLTVU V12, V21, V12 + VORV V12, V22, V22 + VSRAV $63, V4, V8 + VSRLV $1, V8, V8 + VXORV V4, V8, V8 + VANDV V20, V5, V13 + VSLTVU V13, V21, V13 + VORV V13, V22, V22 + VSRAV $63, V5, V9 + VSRLV $1, V9, V9 + VXORV V5, V9, V9 + VANDV V20, V6, V14 + VSLTVU V14, V21, V14 + VORV V14, V22, V22 + VSRAV $63, V6, V10 + VSRLV $1, V10, V10 + VXORV V6, V10, V10 + VANDV V20, V7, V15 + VSLTVU V15, V21, V15 + VORV V15, V22, V22 + VSRAV $63, V7, V11 + VSRLV $1, V11, V11 + VXORV V7, V11, V11 + ADDV $64, R4 + ADDV $-8, R5 + VSLTV V8, V0, V12 + VXORV V8, V0, V16 + VANDV V12, V16, V16 + VXORV V16, V0, V0 + VSLTV V9, V1, V13 + VXORV V9, V1, V17 + VANDV V13, V17, V17 + VXORV V17, V1, V1 + VSLTV V10, V2, V14 + VXORV V10, V2, V18 + VANDV V14, V18, V18 + VXORV V18, V2, V2 + VSLTV V11, V3, V15 + VXORV V11, V3, V19 + VANDV V15, V19, V19 + VXORV V19, V3, V3 + JMP maxLSXloop +maxLSXfold: + VSLTV V1, V0, V12 + VXORV V1, V0, V16 + VANDV V12, V16, V16 + VXORV V16, V0, V0 + VSLTV V3, V2, V13 + VXORV V3, V2, V17 + VANDV V13, V17, V17 + VXORV V17, V2, V2 + VSLTV V2, V0, V14 + VXORV V2, V0, V18 + VANDV V14, V18, V18 + VXORV V18, V0, V0 + VMOVQ V0, (R6) + VMOVQ V22.V[0], R13 + VMOVQ V22.V[1], R14 + OR R14, R13 + MOVV R13, ret+24(FP) + RET + +TEXT ·minLSX(SB), NOSPLIT, $0-32 + MOVV a+0(FP), R4 + MOVV n+8(FP), R5 + MOVV out+16(FP), R6 + MOVV $abs64<>+0(SB), R12 + VMOVQ (R12), V20 + MOVV $inf64<>+0(SB), R12 + VMOVQ (R12), V21 + VXORV V22, V22, V22 + VMOVQ 0(R4), V4 + VMOVQ 16(R4), V5 + VMOVQ 32(R4), V6 + VMOVQ 48(R4), V7 + VANDV V20, V4, V12 + VSLTVU V12, V21, V12 + VORV V12, V22, V22 + VSRAV $63, V4, V8 + VSRLV $1, V8, V8 + VXORV V4, V8, V8 + VANDV V20, V5, V13 + VSLTVU V13, V21, V13 + VORV V13, V22, V22 + VSRAV $63, V5, V9 + VSRLV $1, V9, V9 + VXORV V5, V9, V9 + VANDV V20, V6, V14 + VSLTVU V14, V21, V14 + VORV V14, V22, V22 + VSRAV $63, V6, V10 + VSRLV $1, V10, V10 + VXORV V6, V10, V10 + VANDV V20, V7, V15 + VSLTVU V15, V21, V15 + VORV V15, V22, V22 + VSRAV $63, V7, V11 + VSRLV $1, V11, V11 + VXORV V7, V11, V11 + ADDV $64, R4 + ADDV $-8, R5 + VORV V8, V8, V0 + VORV V9, V9, V1 + VORV V10, V10, V2 + VORV V11, V11, V3 +minLSXloop: + BEQ R5, minLSXfold + VMOVQ 0(R4), V4 + VMOVQ 16(R4), V5 + VMOVQ 32(R4), V6 + VMOVQ 48(R4), V7 + VANDV V20, V4, V12 + VSLTVU V12, V21, V12 + VORV V12, V22, V22 + VSRAV $63, V4, V8 + VSRLV $1, V8, V8 + VXORV V4, V8, V8 + VANDV V20, V5, V13 + VSLTVU V13, V21, V13 + VORV V13, V22, V22 + VSRAV $63, V5, V9 + VSRLV $1, V9, V9 + VXORV V5, V9, V9 + VANDV V20, V6, V14 + VSLTVU V14, V21, V14 + VORV V14, V22, V22 + VSRAV $63, V6, V10 + VSRLV $1, V10, V10 + VXORV V6, V10, V10 + VANDV V20, V7, V15 + VSLTVU V15, V21, V15 + VORV V15, V22, V22 + VSRAV $63, V7, V11 + VSRLV $1, V11, V11 + VXORV V7, V11, V11 + ADDV $64, R4 + ADDV $-8, R5 + VSLTV V0, V8, V12 + VXORV V8, V0, V16 + VANDV V12, V16, V16 + VXORV V16, V0, V0 + VSLTV V1, V9, V13 + VXORV V9, V1, V17 + VANDV V13, V17, V17 + VXORV V17, V1, V1 + VSLTV V2, V10, V14 + VXORV V10, V2, V18 + VANDV V14, V18, V18 + VXORV V18, V2, V2 + VSLTV V3, V11, V15 + VXORV V11, V3, V19 + VANDV V15, V19, V19 + VXORV V19, V3, V3 + JMP minLSXloop +minLSXfold: + VSLTV V0, V1, V12 + VXORV V1, V0, V16 + VANDV V12, V16, V16 + VXORV V16, V0, V0 + VSLTV V2, V3, V13 + VXORV V3, V2, V17 + VANDV V13, V17, V17 + VXORV V17, V2, V2 + VSLTV V0, V2, V14 + VXORV V2, V0, V18 + VANDV V14, V18, V18 + VXORV V18, V0, V0 + VMOVQ V0, (R6) + VMOVQ V22.V[0], R13 + VMOVQ V22.V[1], R14 + OR R14, R13 + MOVV R13, ret+24(FP) + RET + +TEXT ·max32LSX(SB), NOSPLIT, $0-32 + MOVV a+0(FP), R4 + MOVV n+8(FP), R5 + MOVV out+16(FP), R6 + MOVV $abs32<>+0(SB), R12 + VMOVQ (R12), V20 + MOVV $inf32<>+0(SB), R12 + VMOVQ (R12), V21 + VXORV V22, V22, V22 + VMOVQ 0(R4), V4 + VMOVQ 16(R4), V5 + VMOVQ 32(R4), V6 + VMOVQ 48(R4), V7 + VANDV V20, V4, V12 + VSLTWU V12, V21, V12 + VORV V12, V22, V22 + VSRAW $31, V4, V8 + VSRLW $1, V8, V8 + VXORV V4, V8, V8 + VANDV V20, V5, V13 + VSLTWU V13, V21, V13 + VORV V13, V22, V22 + VSRAW $31, V5, V9 + VSRLW $1, V9, V9 + VXORV V5, V9, V9 + VANDV V20, V6, V14 + VSLTWU V14, V21, V14 + VORV V14, V22, V22 + VSRAW $31, V6, V10 + VSRLW $1, V10, V10 + VXORV V6, V10, V10 + VANDV V20, V7, V15 + VSLTWU V15, V21, V15 + VORV V15, V22, V22 + VSRAW $31, V7, V11 + VSRLW $1, V11, V11 + VXORV V7, V11, V11 + ADDV $64, R4 + ADDV $-16, R5 + VORV V8, V8, V0 + VORV V9, V9, V1 + VORV V10, V10, V2 + VORV V11, V11, V3 +max32LSXloop: + BEQ R5, max32LSXfold + VMOVQ 0(R4), V4 + VMOVQ 16(R4), V5 + VMOVQ 32(R4), V6 + VMOVQ 48(R4), V7 + VANDV V20, V4, V12 + VSLTWU V12, V21, V12 + VORV V12, V22, V22 + VSRAW $31, V4, V8 + VSRLW $1, V8, V8 + VXORV V4, V8, V8 + VANDV V20, V5, V13 + VSLTWU V13, V21, V13 + VORV V13, V22, V22 + VSRAW $31, V5, V9 + VSRLW $1, V9, V9 + VXORV V5, V9, V9 + VANDV V20, V6, V14 + VSLTWU V14, V21, V14 + VORV V14, V22, V22 + VSRAW $31, V6, V10 + VSRLW $1, V10, V10 + VXORV V6, V10, V10 + VANDV V20, V7, V15 + VSLTWU V15, V21, V15 + VORV V15, V22, V22 + VSRAW $31, V7, V11 + VSRLW $1, V11, V11 + VXORV V7, V11, V11 + ADDV $64, R4 + ADDV $-16, R5 + VSLTW V8, V0, V12 + VXORV V8, V0, V16 + VANDV V12, V16, V16 + VXORV V16, V0, V0 + VSLTW V9, V1, V13 + VXORV V9, V1, V17 + VANDV V13, V17, V17 + VXORV V17, V1, V1 + VSLTW V10, V2, V14 + VXORV V10, V2, V18 + VANDV V14, V18, V18 + VXORV V18, V2, V2 + VSLTW V11, V3, V15 + VXORV V11, V3, V19 + VANDV V15, V19, V19 + VXORV V19, V3, V3 + JMP max32LSXloop +max32LSXfold: + VSLTW V1, V0, V12 + VXORV V1, V0, V16 + VANDV V12, V16, V16 + VXORV V16, V0, V0 + VSLTW V3, V2, V13 + VXORV V3, V2, V17 + VANDV V13, V17, V17 + VXORV V17, V2, V2 + VSLTW V2, V0, V14 + VXORV V2, V0, V18 + VANDV V14, V18, V18 + VXORV V18, V0, V0 + VMOVQ V0, (R6) + VMOVQ V22.V[0], R13 + VMOVQ V22.V[1], R14 + OR R14, R13 + MOVV R13, ret+24(FP) + RET + +TEXT ·min32LSX(SB), NOSPLIT, $0-32 + MOVV a+0(FP), R4 + MOVV n+8(FP), R5 + MOVV out+16(FP), R6 + MOVV $abs32<>+0(SB), R12 + VMOVQ (R12), V20 + MOVV $inf32<>+0(SB), R12 + VMOVQ (R12), V21 + VXORV V22, V22, V22 + VMOVQ 0(R4), V4 + VMOVQ 16(R4), V5 + VMOVQ 32(R4), V6 + VMOVQ 48(R4), V7 + VANDV V20, V4, V12 + VSLTWU V12, V21, V12 + VORV V12, V22, V22 + VSRAW $31, V4, V8 + VSRLW $1, V8, V8 + VXORV V4, V8, V8 + VANDV V20, V5, V13 + VSLTWU V13, V21, V13 + VORV V13, V22, V22 + VSRAW $31, V5, V9 + VSRLW $1, V9, V9 + VXORV V5, V9, V9 + VANDV V20, V6, V14 + VSLTWU V14, V21, V14 + VORV V14, V22, V22 + VSRAW $31, V6, V10 + VSRLW $1, V10, V10 + VXORV V6, V10, V10 + VANDV V20, V7, V15 + VSLTWU V15, V21, V15 + VORV V15, V22, V22 + VSRAW $31, V7, V11 + VSRLW $1, V11, V11 + VXORV V7, V11, V11 + ADDV $64, R4 + ADDV $-16, R5 + VORV V8, V8, V0 + VORV V9, V9, V1 + VORV V10, V10, V2 + VORV V11, V11, V3 +min32LSXloop: + BEQ R5, min32LSXfold + VMOVQ 0(R4), V4 + VMOVQ 16(R4), V5 + VMOVQ 32(R4), V6 + VMOVQ 48(R4), V7 + VANDV V20, V4, V12 + VSLTWU V12, V21, V12 + VORV V12, V22, V22 + VSRAW $31, V4, V8 + VSRLW $1, V8, V8 + VXORV V4, V8, V8 + VANDV V20, V5, V13 + VSLTWU V13, V21, V13 + VORV V13, V22, V22 + VSRAW $31, V5, V9 + VSRLW $1, V9, V9 + VXORV V5, V9, V9 + VANDV V20, V6, V14 + VSLTWU V14, V21, V14 + VORV V14, V22, V22 + VSRAW $31, V6, V10 + VSRLW $1, V10, V10 + VXORV V6, V10, V10 + VANDV V20, V7, V15 + VSLTWU V15, V21, V15 + VORV V15, V22, V22 + VSRAW $31, V7, V11 + VSRLW $1, V11, V11 + VXORV V7, V11, V11 + ADDV $64, R4 + ADDV $-16, R5 + VSLTW V0, V8, V12 + VXORV V8, V0, V16 + VANDV V12, V16, V16 + VXORV V16, V0, V0 + VSLTW V1, V9, V13 + VXORV V9, V1, V17 + VANDV V13, V17, V17 + VXORV V17, V1, V1 + VSLTW V2, V10, V14 + VXORV V10, V2, V18 + VANDV V14, V18, V18 + VXORV V18, V2, V2 + VSLTW V3, V11, V15 + VXORV V11, V3, V19 + VANDV V15, V19, V19 + VXORV V19, V3, V3 + JMP min32LSXloop +min32LSXfold: + VSLTW V0, V1, V12 + VXORV V1, V0, V16 + VANDV V12, V16, V16 + VXORV V16, V0, V0 + VSLTW V2, V3, V13 + VXORV V3, V2, V17 + VANDV V13, V17, V17 + VXORV V17, V2, V2 + VSLTW V0, V2, V14 + VXORV V2, V0, V18 + VANDV V14, V18, V18 + VXORV V18, V0, V0 + VMOVQ V0, (R6) + VMOVQ V22.V[0], R13 + VMOVQ V22.V[1], R14 + OR R14, R13 + MOVV R13, ret+24(FP) + RET + +DATA abs64<>+0(SB)/1, $0xff +DATA abs64<>+1(SB)/1, $0xff +DATA abs64<>+2(SB)/1, $0xff +DATA abs64<>+3(SB)/1, $0xff +DATA abs64<>+4(SB)/1, $0xff +DATA abs64<>+5(SB)/1, $0xff +DATA abs64<>+6(SB)/1, $0xff +DATA abs64<>+7(SB)/1, $0x7f +DATA abs64<>+8(SB)/1, $0xff +DATA abs64<>+9(SB)/1, $0xff +DATA abs64<>+10(SB)/1, $0xff +DATA abs64<>+11(SB)/1, $0xff +DATA abs64<>+12(SB)/1, $0xff +DATA abs64<>+13(SB)/1, $0xff +DATA abs64<>+14(SB)/1, $0xff +DATA abs64<>+15(SB)/1, $0x7f +GLOBL abs64<>(SB), RODATA|NOPTR, $16 + +DATA inf64<>+0(SB)/1, $0x00 +DATA inf64<>+1(SB)/1, $0x00 +DATA inf64<>+2(SB)/1, $0x00 +DATA inf64<>+3(SB)/1, $0x00 +DATA inf64<>+4(SB)/1, $0x00 +DATA inf64<>+5(SB)/1, $0x00 +DATA inf64<>+6(SB)/1, $0xf0 +DATA inf64<>+7(SB)/1, $0x7f +DATA inf64<>+8(SB)/1, $0x00 +DATA inf64<>+9(SB)/1, $0x00 +DATA inf64<>+10(SB)/1, $0x00 +DATA inf64<>+11(SB)/1, $0x00 +DATA inf64<>+12(SB)/1, $0x00 +DATA inf64<>+13(SB)/1, $0x00 +DATA inf64<>+14(SB)/1, $0xf0 +DATA inf64<>+15(SB)/1, $0x7f +GLOBL inf64<>(SB), RODATA|NOPTR, $16 + +DATA abs32<>+0(SB)/1, $0xff +DATA abs32<>+1(SB)/1, $0xff +DATA abs32<>+2(SB)/1, $0xff +DATA abs32<>+3(SB)/1, $0x7f +DATA abs32<>+4(SB)/1, $0xff +DATA abs32<>+5(SB)/1, $0xff +DATA abs32<>+6(SB)/1, $0xff +DATA abs32<>+7(SB)/1, $0x7f +DATA abs32<>+8(SB)/1, $0xff +DATA abs32<>+9(SB)/1, $0xff +DATA abs32<>+10(SB)/1, $0xff +DATA abs32<>+11(SB)/1, $0x7f +DATA abs32<>+12(SB)/1, $0xff +DATA abs32<>+13(SB)/1, $0xff +DATA abs32<>+14(SB)/1, $0xff +DATA abs32<>+15(SB)/1, $0x7f +GLOBL abs32<>(SB), RODATA|NOPTR, $16 + +DATA inf32<>+0(SB)/1, $0x00 +DATA inf32<>+1(SB)/1, $0x00 +DATA inf32<>+2(SB)/1, $0x80 +DATA inf32<>+3(SB)/1, $0x7f +DATA inf32<>+4(SB)/1, $0x00 +DATA inf32<>+5(SB)/1, $0x00 +DATA inf32<>+6(SB)/1, $0x80 +DATA inf32<>+7(SB)/1, $0x7f +DATA inf32<>+8(SB)/1, $0x00 +DATA inf32<>+9(SB)/1, $0x00 +DATA inf32<>+10(SB)/1, $0x80 +DATA inf32<>+11(SB)/1, $0x7f +DATA inf32<>+12(SB)/1, $0x00 +DATA inf32<>+13(SB)/1, $0x00 +DATA inf32<>+14(SB)/1, $0x80 +DATA inf32<>+15(SB)/1, $0x7f +GLOBL inf32<>(SB), RODATA|NOPTR, $16 +