Fix normalized multiple-choice probability underflow - #1376
Open
YusefSyed wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NormalizedMultiChoiceProbabilityby subtracting the largestfinite log-probability before exponentiation
underflowed to zero
Why
The metric currently computes
exp(logprob)before dividing by the sum. Forfinite sequence log-likelihoods such as
[-1000.0, -1001.0], both exponentialsunderflow to zero. The metric then returns
0.0and emits an invalid-dividewarning, even though the gold choice has normalized probability
1 / (1 + exp(-1)), approximately0.7310586.Subtracting one shared finite maximum is algebraically neutral for the
probability ratios and prevents both underflow and unnecessary overflow.
Verification
Red-on-main control at
932e1f2f:Patched branch:
Checks:
pytest tests/test_unit_base_metrics.py -q -k 'mc_probability_metric'— 2 passedpytest tests/test_unit_base_metrics.py -q -k 'not f1_dynamic_metric and not exact_match_dynamic_metric'— 8 passed, 3 skippedPYTHONPATH=. pytest tests/unit/metrics/test_normalizations.py -q— 4 passedruff format --checkon changed files — passedruff checkon changed files — passedgit diff --check— passedThe two excluded tests require the optional
multilingualextra. A separatemetrics request run also reaches the repository's existing xxhash 4 string
compatibility failure tracked in #1330; neither failure touches this metric or
diff.
Closes #1375.