Latent int16 overflow in ggml_vec_dot_i2_i8_s_1x1: 128 maddubs results per lane against a 32767 ceiling

[purpleskulll]

Summary

ggml_vec_dot_i2_i8_s_1x1 accumulates 128 vpmaddubsw results into one
int16 lane before folding. The worst case is 128 * 508 = 65024 against a
32767 ceiling. It is correct in practice only because signed activations
cancel, and it is latent on a default build because the sgemm path takes
the work instead.

I am filing it as hardening, not as a wrong-output bug, and the distinction is
deliberate — see Severity below.

The arithmetic

ggml/src/ggml-cpu/quants.c:

const int group32_num = nb / 32;                     // :1396
...
__m256i accu32 = _mm256_setzero_si256();             // :1413  (int16 lanes)
for (int j = 0; j < 32; j++) {                       // :1414
    ...
    accu32 = _mm256_add_epi16(accu32, _mm256_add_epi16(xq8_0, xq8_1));   // :1439
    accu32 = _mm256_add_epi16(accu32, _mm256_add_epi16(xq8_2, xq8_3));   // :1440
}
accu = _mm256_add_epi32(_mm256_madd_epi16(accu32, one16), accu);         // :1445

32 iterations × 4 planes = 128 vpmaddubsw results per int16 lane. Each is at
most 2 * 2 * 127 = 508, so the lane can reach 65024, roughly twice the
ceiling. The name accu32 suggests 32-bit; the type is __m256i used with
_mm256_add_epi16.

What actually happens

With all activations the same sign the sum does not cancel and the lane wraps.
Measured on the real weights of BitNet-b1.58-2B-4T — 30 tensors at
K = 6912, driven against int8 activations pinned to +127 — 11,998 of
12,000 rows disagree with exact int64 arithmetic, and every error is a
multiple of 65536, the signature of a wrapped int16.

With ordinary int8 activations, 0 of 12,000 rows wrap. That is why it has
never been seen.

Severity: low, and I want to be accurate about why

• On a build where llamafile_sgemm handles these shapes this function is not
  called at all. The kernel's own counters on my build read
  calls=0 sgemm=52068.
• It becomes reachable when that path is disabled, which is a supported switch
  rather than an exotic one.
• Even then it needs same-signed activations across a whole group of 32 blocks.

So: a latent overflow in a fallback, not a correctness bug users are hitting.
I would not have filed it on its own; I am filing it because it is the same
missing bound-derivation as the fold issue I opened alongside this one (#628), in the
same file family, and a fix for one is naturally a fix for both.

Not a duplicate of the ARM reports

#411 and the audit in #525 report an int16 overflow, but it is the NEON
vmlal_s8 accumulation in src/ggml-bitnet-mad.cpp, which does produce visible
garbage. This is a different function, a different instruction set and a
different failure mode — this one is invisible precisely because it cancels.

Suggested fix

Fold every 8 groups rather than every 32, or accumulate in int32 directly.
Upstream ggml's equivalent kernels derive the bound explicitly and stay inside
it — TQ2_0 in arch/x86/quants.c carries // 16-bit sums, because 256*127 still fits — which is the pattern worth copying here.

Reproducible from https://github.com/purpleskulll/bitnet-t5b.

[purpleskulll]

Correction to my own arithmetic above, before anyone spends time on it.

I wrote that each vpmaddubsw result is "at most 2 * 2 * 127 = 508". That is
what the encoder emits — i2_s packs ternary ${-1,0,+1}$ as 2-bit codes
0, 1, 2 and never uses 3 — but it is not the bound the kernel carries. Its
own mask is _mm256_set1_epi8(0x03) (quants.c:1400), so the operand range it
guarantees is 0..3:

per lane, per plane      2 * 3 * 127 = 762
128 results per lane     128 * 762   = 97,536     (kernel's own bound)
                         128 * 508   = 65,024     (what the encoder can produce)

Either figure is at least twice the 32767 ceiling, so the conclusion is
unchanged and the 11,998-of-12,000 measurement stands — it was taken on the real
encoder output, hence on the 508 basis.

I am correcting it because it was inconsistent with #628, which derives the
same kernel family's bound from the mask and gets 762. Two issues from the same
reporter should not quote two different operand ranges for the same
representation, and the mask is the right basis for a bound.

Apologies for the noise.