Integer by zero crash when trying FP8 quantization inference

[kar-dim]

NOTE: I use TensorRT-RTX, so naturally I submitted it on TensorRT-RTX repo, but I think it is more relevant here because it seems to be TRT specific and TRT-RTX is probably not to blame. All the information is there, I can "migrate" it here and close it from there if you want.

NVIDIA/TensorRT-RTX#35

[JESUSROYETH]

Hi @kar-dim 👋

I tried to reproduce this from the mainline TensorRT side, in order to help answer your open question ("it seems to be TRT specific and TRT-RTX is probably not to blame"). On my end mainline TRT does not crash.

I built the same tiny network you describe — Conv 3→64, Conv 64→64, Conv 64→3, ReLU — as an FP8 (E4M3) Q/DQ ONNX and ran it as a strongly-typed engine on TensorRT 11.1.0.106, Linux, RTX 4060 (Ada, sm 8.9). The engine builds, deserialises and runs inference fine, no STATUS_INTEGER_DIVIDE_BY_ZERO / SIGFPE, and the output stays finite. I ran it across the whole profile range you use, from 64x64 to 512x512:

64x64:   ok=True finite=True
128x128: ok=True finite=True
256x256: ok=True finite=True
512x512: ok=True finite=True

The verbose build log confirms the Q/DQ pairs keep the E4M3 type on both weights and activations, so it really is going through the FP8 path, not silently falling back to FP16. In a strongly-typed network TRT has to honour the graph precision, so the conv runs in FP8 too.

So it looks like the bug is in the TRT-RTX runtime rather than the shared engine core, or it was a core bug already fixed somewhere between the v10.5 / b114 lineage your TRT-RTX reports and 11.1. But we need to be honest about two limits here: I tested 11.1, not the exact 10.5 core you are on, and TRT-RTX JIT-compiles its kernels at runtime, which is a different path from the mainline AOT engine — a divide-by-zero there would not necessarily be in the shared code. One thing that would help narrow it down: running the same FP8 engine through mainline trtexec on a 10.5 build, if that one stays clean too, the core never had it.

Small note on the repro: I could not use modelopt here, so I added the FP8 Q/DQ by hand with per-tensor scales instead of per-channel. Since the crash is at kernel execution and you already confirmed it is data-independent (random calibration still crashes), the scale values shouldn't change much. I'm happy to share the full script if it's useful ..

[kar-dim]

Hello @JESUSROYETH

You are right! It builds and DOES inference correctly on the mainline TensorRT 11.1.0.106. So in the end it must be a TRT 10.5 issue (or TRT-RTX bug?). I guess I will wait until TRT-RTX is updated to the latest TRT 11 version, because I really like the JIT idea and the much smaller binary size. Thank you for the feedback!