Skip to content

fix(metrics): share one guarded scoring loop between calculate_metrics and compute_metrics - #874

Open
Maarmapa wants to merge 1 commit into
DashAISoftware:developfrom
Maarmapa:fix/metric-scoring-guard
Open

fix(metrics): share one guarded scoring loop between calculate_metrics and compute_metrics#874
Maarmapa wants to merge 1 commit into
DashAISoftware:developfrom
Maarmapa:fix/metric-scoring-guard

Conversation

@Maarmapa

Copy link
Copy Markdown
Contributor

… metric

calculate_metrics and compute_metrics carried a copy each of the same scoring loop, and the copies drifted: only the first dropped non-finite scores. The second is the one the CV evaluation loop calls to build the objective of an HPO trial, so a metric too suspect to write to the database was still trusted to decide which hyperparameters won -- and every comparison against a NaN is False in both directions, so that trial could neither be beaten nor win.

Both now delegate to BaseModel._score_split, which owns the data loading, the prediction, the scoring and the math.isfinite guard. It returns None when there was nothing to score and a dict otherwise, which lets the two callers keep their different contracts: calculate_metrics persists an empty result (every metric was asked and none was usable) but skips when there was nothing to ask, while compute_metrics collapses both to {}.

Second half: cv.py indexed the fold's validation scores with the goal metric's name and trusted it to be there. It is not always -- the metric may not be among the run's validation metrics, and now that non-finite scores are dropped it can also be missing because it was undefined on that fold. Both are configuration errors, and both surfaced as a bare KeyError several frames below where the choice was made. It now raises with the metric asked for and the metrics actually scored.

RuntimeError and not ValueError on purpose: study.optimize runs with catch=UNFITTABLE_TRIAL_ERRORS, which includes ValueError, so a ValueError here would be swallowed into "all N trials failed, narrow the ranges and try again". The same reasoning is already written into optuna_optimizer.py, where an unsupported parameter type raises TypeError for this exact reason. A test asserts the exception is not one the optimizer catches.

Closes #843

Claude-Session: https://claude.ai/code/session_01BzZ6td4VJdRDe76UndWZY3

Summary

Short explanation of what changed and why.


Type of Change

Check all that apply like this [x]:

  • Backend change
  • Frontend change
  • CI / Workflow change
  • Build / Packaging change
  • Bug fix
  • Documentation

Changes (by file)

Briefly list the important modified files and what was done.

Example:

  • path/to/file.py: description
  • another/file.tsx: description

Testing (optional)

Only add if there's something reviewers should verify.


Notes (optional)

Additional context or considerations.

… metric

`calculate_metrics` and `compute_metrics` carried a copy each of the same
scoring loop, and the copies drifted: only the first dropped non-finite
scores. The second is the one the CV evaluation loop calls to build the
objective of an HPO trial, so a metric too suspect to write to the database
was still trusted to decide which hyperparameters won -- and every comparison
against a NaN is False in both directions, so that trial could neither be
beaten nor win.

Both now delegate to `BaseModel._score_split`, which owns the data loading,
the prediction, the scoring and the `math.isfinite` guard. It returns None
when there was nothing to score and a dict otherwise, which lets the two
callers keep their different contracts: `calculate_metrics` persists an
empty result (every metric was asked and none was usable) but skips when
there was nothing to ask, while `compute_metrics` collapses both to `{}`.

Second half: `cv.py` indexed the fold's validation scores with the goal
metric's name and trusted it to be there. It is not always -- the metric may
not be among the run's validation metrics, and now that non-finite scores are
dropped it can also be missing because it was undefined on that fold. Both
are configuration errors, and both surfaced as a bare KeyError several frames
below where the choice was made. It now raises with the metric asked for and
the metrics actually scored.

RuntimeError and not ValueError on purpose: `study.optimize` runs with
`catch=UNFITTABLE_TRIAL_ERRORS`, which includes ValueError, so a ValueError
here would be swallowed into "all N trials failed, narrow the ranges and try
again". The same reasoning is already written into optuna_optimizer.py, where
an unsupported parameter type raises TypeError for this exact reason. A test
asserts the exception is not one the optimizer catches.

Closes DashAISoftware#843

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzZ6td4VJdRDe76UndWZY3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant