Round to_f64/to_f32 once, to the subnormal-aware precision#91
Open
gaoflow wants to merge 1 commit into
Open
Conversation
The conversions rounded the source to a fixed 53/24-bit intermediate, then the bit encoding rounded again into the exponent-dependent subnormal grid. A value just past a subnormal halfway lands on the halfway after the first rounding and the second rounds to even -- one ULP low. Round the source once, straight to the target's precision at its magnitude (fewer than 53/24 bits for subnormals) through a round-to-odd base conversion, so the single rounding is mode-correct and never double-rounds. Also round an over-wide quotient down to the requested precision in the base-changing division instead of requiring the caller to pre-bound the dividend: high-precision decimals (e.g. "123456789012345678.9012345678901") previously panicked in debug and double-rounded in release when converted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FBig/DBig→ float conversion rounds the source to a fixed 53/24-bit intermediate and then letsf{32,64}::encoderound again into the format's exponent-dependent subnormal grid. That second rounding is wrong in two ways. This is the subnormal counterpart of the normal-range double-rounding that was recently addressed.Subnormal double-rounding (1 ULP). A subnormal has fewer than 53/24 significant bits — its spacing is fixed at
2^-1074/2^-149— so the fixed intermediate keeps bits below the subnormal grid. For a value just past a subnormal halfway, the intermediate rounds to the halfway (dropping the excess that lifts it above) andencodethen rounds half-to-even down, one ULP below the correctly-rounded value:Debug panic / release double-round on high-precision inputs.
to_f64/to_f32feed the unrounded source significand into the base-changing division, whosedebug_assert!(lhs.digits() <= self.precision + rhs.digits())fires whenever the significand is wider than the target precision:In release the assert is compiled out and the oversized quotient is re-rounded downstream — a silent double rounding.
Fix. Round the source once, straight to the target's precision at its own magnitude (
min(53, e + 1075)bits for f64 at binary exponente, analogously for f32), soencodere-rounds nothing. To keep that single rounding correct without an exact arbitrary-precision conversion, the source is first taken to a fixed, generous width with round-to-odd; rounding that down to the final width then reproduces the correctly-rounded result for every rounding mode. The division now rounds an over-wide quotient down to the requested precision instead of assuming the caller pre-bounded the dividend, so the assertion is removed and high-precision inputs convert without panicking (and without the release double-round).Correctness holds to within the base conversion's working precision; a value closer to a boundary than the conversion resolves stays a pre-existing limitation of finite-precision base changing (shared with
with_base). The base-2 /Reprpaths are exact for all inputs.Tests cover subnormal halfways nudged just past the boundary across the exponent range, in both base 2 and base 10, for f64 (oracle:
float()) and f32 (oracle: exactFraction→ nearest2^-149, notnumpy.float32, which itself double-rounds), plus the high-precision decimals that previously panicked.cargo test --workspace --exclude dashu-python,cargo clippy --all-features --all-targets --workspace --exclude dashu-python -- -D warnings, andcargo fmt --all -- --checkall pass.