Repository navigation
Round half floats up into the next power of two, and read small ones at their value - #334
Merged
Merged
Conversation
…at their value FloatToHalf rounded the mantissa up and then masked it to 10 bits, so a value that rounded past the largest mantissa lost the carry: 0.99995 became 0.5, 1.9999 became 1.0, and 65530 became 32768. A gradient's rotation is packed this way, so one at an angle near an axis pointed the wrong way. HalfToFloat read subnormal halves as half their value. - FloatToHalf adds the rounding to exponent and mantissa together: it carries into the exponent, and to infinity past the largest half. - HalfToFloat normalizes a subnormal with the right exponent. - Tests: rounding up carries into the exponent, rounding past the largest half is infinity, subnormals read as their value, and every finite half comes back as itself through a float.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FloatToHalfcould lose a carry when rounding, andHalfToFloatread the smallest values at half their size. The styles use both to pack a gradient's rotation into 16-bit floats, so a gradient at an angle near an axis pointed the wrong way.Cause
FloatToHalf: it rounded the mantissa up, then masked it to 10 bits. A value that rounded past the largest mantissa lost the carry instead of moving to the next power of two:HalfToFloat: it normalized a subnormal half with the exponent off by one, so every subnormal read as half its value.What changes
FloatToHalf: it now adds the rounding to the exponent and mantissa together. Rounding past the largest mantissa carries into the exponent, and rounding past the largest half gives infinity.HalfToFloat: it now normalizes a subnormal with the right exponent.Tests
HalfToFloatbug.Validation
gui_core_tests,gui_vulkan_tests,gui_gpu_tests,gui_d3d12_testsandgui_sdlgpu_testsall pass on Windows.