Skip to content

Fix loo_compare() for subsampling results - #391

Open
florence-bockting wants to merge 5 commits into
masterfrom
fix-subsampling
Open

Fix loo_compare() for subsampling results#391
florence-bockting wants to merge 5 commits into
masterfrom
fix-subsampling

Conversation

@florence-bockting

@florence-bockting florence-bockting commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Fixes #390

Description

loo_compare for subsampling results provided a difference based on reference model - comparison model, while loo_compare for other loo objects returned the reverse difference leading to a flip in the sign of elpd_diff.

This PR fixes the sign flip, adds a test for the comparison direction, and updates the related outputs of the tutorial vignettes/loo2-large-data.Rmd.

Tests

test_loo_subsampling.R: [ FAIL 0 | WARN 0 | SKIP 0 | PASS 408 ]

Documentation

Updated the three printed tables in loo2-large-data.Rmd.

TODO

  • devtools::check()
  • Fill the PR number into NEWS.md

AI assistance

I used AI assistance while working on this PR. I have reviewed all changes.

@florence-bockting florence-bockting changed the title fix: correct elpd_diff sign and return a data.frame in loo_compare() … Fix loo_compare() for subsampling results Sep 2, 2026
@codecov-commenter

codecov-commenter commented Sep 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.86%. Comparing base (a59519c) to head (8635a88).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #391   +/-   ##
=======================================
  Coverage   92.86%   92.86%           
=======================================
  Files          31       31           
  Lines        3041     3041           
=======================================
  Hits         2824     2824           
  Misses        217      217           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 8635a88 is merged into master:

  • ✔️loo_function: 1.51s -> 1.5s [-1.61%, +0.23%]
  • ✔️loo_matrix: 1.5s -> 1.49s [-1.79%, +0.23%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

@florence-bockting
florence-bockting marked this pull request as ready for review September 3, 2026 06:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

loo_compare() returns elpd_diff with the opposite sign for psis_loo_ss objects

2 participants