Skip to content

Round to_f64/to_f32 once, to the subnormal-aware precision#91

Open
gaoflow wants to merge 1 commit into
cmpute:masterfrom
gaoflow:fix-subnormal-double-rounding
Open

Round to_f64/to_f32 once, to the subnormal-aware precision#91
gaoflow wants to merge 1 commit into
cmpute:masterfrom
gaoflow:fix-subnormal-double-rounding

Conversation

@gaoflow

@gaoflow gaoflow commented Jul 18, 2026

Copy link
Copy Markdown

FBig/DBig → float conversion rounds the source to a fixed 53/24-bit intermediate and then lets f{32,64}::encode round again into the format's exponent-dependent subnormal grid. That second rounding is wrong in two ways. This is the subnormal counterpart of the normal-range double-rounding that was recently addressed.

Subnormal double-rounding (1 ULP). A subnormal has fewer than 53/24 significant bits — its spacing is fixed at 2^-1074 / 2^-149 — so the fixed intermediate keeps bits below the subnormal grid. For a value just past a subnormal halfway, the intermediate rounds to the halfway (dropping the excess that lifts it above) and encode then rounds half-to-even down, one ULP below the correctly-rounded value:

// value = (2m+1)·2^-1075 + 2^-1135, just above the halfway between subnormals m and m+1
let sig = (IBig::from(2 * m + 1) << 60) + IBig::ONE;
let v = FBig::<HalfEven, 2>::from_parts(sig, -1135);
v.to_f64(); // returned m·2^-1074 (rounded to even); correct is (m+1)·2^-1074

Debug panic / release double-round on high-precision inputs. to_f64/to_f32 feed the unrounded source significand into the base-changing division, whose debug_assert!(lhs.digits() <= self.precision + rhs.digits()) fires whenever the significand is wider than the target precision:

DBig::from_str("123456789012345678.9012345678901")?.to_f64(); // panics in debug builds

In release the assert is compiled out and the oversized quotient is re-rounded downstream — a silent double rounding.

Fix. Round the source once, straight to the target's precision at its own magnitude (min(53, e + 1075) bits for f64 at binary exponent e, analogously for f32), so encode re-rounds nothing. To keep that single rounding correct without an exact arbitrary-precision conversion, the source is first taken to a fixed, generous width with round-to-odd; rounding that down to the final width then reproduces the correctly-rounded result for every rounding mode. The division now rounds an over-wide quotient down to the requested precision instead of assuming the caller pre-bounded the dividend, so the assertion is removed and high-precision inputs convert without panicking (and without the release double-round).

Correctness holds to within the base conversion's working precision; a value closer to a boundary than the conversion resolves stays a pre-existing limitation of finite-precision base changing (shared with with_base). The base-2 / Repr paths are exact for all inputs.

Tests cover subnormal halfways nudged just past the boundary across the exponent range, in both base 2 and base 10, for f64 (oracle: float()) and f32 (oracle: exact Fraction → nearest 2^-149, not numpy.float32, which itself double-rounds), plus the high-precision decimals that previously panicked. cargo test --workspace --exclude dashu-python, cargo clippy --all-features --all-targets --workspace --exclude dashu-python -- -D warnings, and cargo fmt --all -- --check all pass.

The conversions rounded the source to a fixed 53/24-bit intermediate, then
the bit encoding rounded again into the exponent-dependent subnormal grid.
A value just past a subnormal halfway lands on the halfway after the first
rounding and the second rounds to even -- one ULP low. Round the source
once, straight to the target's precision at its magnitude (fewer than 53/24
bits for subnormals) through a round-to-odd base conversion, so the single
rounding is mode-correct and never double-rounds.

Also round an over-wide quotient down to the requested precision in the
base-changing division instead of requiring the caller to pre-bound the
dividend: high-precision decimals (e.g. "123456789012345678.9012345678901")
previously panicked in debug and double-rounded in release when converted.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant