Skip to content

Fix ECE confidence binary crash - #1215

Open
fbonc wants to merge 4 commits into
sunlabuiuc:masterfrom
fbonc:fix-ece-confidence-binary
Open

Fix ECE confidence binary crash#1215
fbonc wants to merge 4 commits into
sunlabuiuc:masterfrom
fbonc:fix-ece-confidence-binary

Conversation

@fbonc

@fbonc fbonc commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Issue

ece_confidence_binary indexed prob[:,0] and label[:,0] (calibration.py:150), requiring 2D arrays. Its only real caller, binary_metrics_fn (binary.py:91), passes 1D positive-class probabilities and 1D 0/1 labels, so ECE and ECE_adapt always raised IndexError for binary tasks via the documented API. As a secondary defect, even for 2D input the code used column 0 (class 0) despite the docstring stating the metric is computed on class 1.

Fix

Compute confidence from the positive class (prob[:,1] when 2D, else the 1D array) and use the 0/1 label as the accuracy target (label[:,1] when 2D one-hot, else the 1D array). This makes the function work with the 1D shapes the caller actually uses and corrects the class-0/class-1 indexing. ece_classwise is unaffected (it calls a separate internal helper, not this function).

Notes

Added new test file tests/core/test_calibration_binary_ece.py with two tests. No prior tests existed for this function.

fbonc added 3 commits August 27, 2026 17:26
The function indexed prob[:,0]/label[:,0], requiring 2D arrays, but its
only caller (binary_metrics_fn) passes 1D positive-class probs and 1D
0/1 labels, so ECE/ECE_adapt always raised IndexError. Use the positive
class (class 1) as confidence and the 0/1 label as target, tolerating
1D and 2D inputs. Also corrects the class-0 vs class-1 indexing.
@DarylOkeke

Copy link
Copy Markdown

I think this still crashes with PyHealth's normal binary model output. BinaryLabelProcessor.process() returns shape (1,), so after batching and inference both prob and label are (N, 1). Since ndim > 1, this code tries [:, 1], but column 1 does not exist. Could the shape handling distinguish flat (N,), single-column (N, 1) (use column 0), and two-column (N, 2) (use positive-class column 1), with a regression test for (N, 1)?

@fbonc

fbonc commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

I think this still crashes with PyHealth's normal binary model output. BinaryLabelProcessor.process() returns shape (1,), so after batching and inference both prob and label are (N, 1). Since ndim > 1, this code tries [:, 1], but column 1 does not exist. Could the shape handling distinguish flat (N,), single-column (N, 1) (use column 0), and two-column (N, 2) (use positive-class column 1), with a regression test for (N, 1)?

fixed!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants