Skip to content

[Core] Fix numeric_limits constants for low-precision types (issue #3508) - #3620

Open
XFDG wants to merge 1 commit into
NVIDIA:mainfrom
XFDG:fix/numeric-limits-3508
Open

XFDG wants to merge 1 commit into
NVIDIA:mainfrom
XFDG:fix/numeric-limits-3508

Conversation

@XFDG

@XFDG XFDG commented Sep 13, 2026

Copy link
Copy Markdown

Summary

Fix three numeric_limits constants whose encoded values do not match the represented low-precision types:

  • float_ue8m0_t::lowest(): 0xfe -> 0x00 in both the std and CUTLASS platform specializations
  • tfloat32_t::epsilon(): 0x00001000 -> 0x3a800000 (2^-10)
  • half_t::epsilon(): 0x1800 -> 0x1400 (2^-10) in both specializations

The existing core tests now pin each corrected encoding.

Rationale

float_ue8m0_t is unsigned, so its lowest finite value is also its minimum encoded value, 0x00 (2^-127), rather than 0xfe (2^127). TF32 and FP16 each carry 10 explicit fraction bits for numeric_limits purposes, so their machine epsilon is 2^-10.

Validation

Built the same standalone CUDA probe against current main before and after this change using CUDA 13.1 and -arch=sm_100a, then ran it on an NVIDIA B200.

Before:

  • both UE8M0 lowest() specializations returned 0xfe
  • TF32 epsilon() returned 0x00001000
  • both FP16 epsilon() specializations returned 0x1800 on the host and device
  • 7/7 checks failed

After:

  • all 7/7 host/device encoded-value checks pass
  • the probe exits with failures=0
  • git diff --check passes

Fixes #3508

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

numeric_limits constants wrong for float_ue8m0_t::lowest, tfloat32_t::epsilon and half_t::epsilon

1 participant