Skip to content

Fix normalized multiple-choice probability underflow - #1376

Open
YusefSyed wants to merge 1 commit into
huggingface:mainfrom
YusefSyed:fix/normalized-mc-prob-underflow
Open

Fix normalized multiple-choice probability underflow#1376
YusefSyed wants to merge 1 commit into
huggingface:mainfrom
YusefSyed:fix/normalized-mc-prob-underflow

Conversation

@YusefSyed

Copy link
Copy Markdown

Summary

  • stabilize NormalizedMultiChoiceProbability by subtracting the largest
    finite log-probability before exponentiation
  • preserve the existing behavior for non-finite and empty inputs
  • add a regression for long-sequence log-probabilities that previously
    underflowed to zero

Why

The metric currently computes exp(logprob) before dividing by the sum. For
finite sequence log-likelihoods such as [-1000.0, -1001.0], both exponentials
underflow to zero. The metric then returns 0.0 and emits an invalid-divide
warning, even though the gold choice has normalized probability
1 / (1 + exp(-1)), approximately 0.7310586.

Subtracting one shared finite maximum is algebraically neutral for the
probability ratios and prevents both underflow and unnecessary overflow.

Verification

Red-on-main control at 932e1f2f:

result=0.0
warning_count=1
invalid value encountered in divide

Patched branch:

result=0.7310585786300049
warning_count=0

Checks:

  • pytest tests/test_unit_base_metrics.py -q -k 'mc_probability_metric' — 2 passed
  • pytest tests/test_unit_base_metrics.py -q -k 'not f1_dynamic_metric and not exact_match_dynamic_metric' — 8 passed, 3 skipped
  • PYTHONPATH=. pytest tests/unit/metrics/test_normalizations.py -q — 4 passed
  • ruff format --check on changed files — passed
  • ruff check on changed files — passed
  • git diff --check — passed

The two excluded tests require the optional multilingual extra. A separate
metrics request run also reaches the repository's existing xxhash 4 string
compatibility failure tracked in #1330; neither failure touches this metric or
diff.

Closes #1375.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Normalized multiple-choice probability underflows for long sequence log-probabilities

1 participant