Report attainable coverage on tournament leaderboards - #5124
Draft
cemreinanc wants to merge 3 commits into
Draft
Conversation
The leaderboard coverage column now divides total coverage by the total *attainable* coverage instead of by the raw question count. Questions that close early (e.g. resolve before their scheduled close time) have a maximum attainable coverage below 100%, so coverage is now measured against what was actually attainable. - Add Question.get_attainable_coverage() = (effective_close_time - open_time) / (scheduled_close_time - open_time). - LeaderboardSerializer.get_max_coverage now sums attainable coverage weighted by question weight over successfully resolved questions. - Expose attainable_coverage per contribution. - "My Score" section: the Coverage column becomes "Coverage (max)" showing your coverage and the max attainable in parentheses, and the totals now show total coverage, total attainable coverage, and effective coverage (which matches the leaderboard value). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NmzezEyuu7d2sX214hZWqZ
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughCoverage calculations now account for early question closure. Scoring data exposes attainable coverage, and the contributions UI displays total, attainable, effective, and maximum coverage with updated explanations. ChangesCoverage metrics
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Question
participant ScoringUtils
participant LeaderboardSerializer
participant ContributionsUI
Question->>ScoringUtils: Calculate attainable coverage
ScoringUtils->>LeaderboardSerializer: Include weighted coverage data
LeaderboardSerializer-->>ContributionsUI: Serialize contribution metrics
ContributionsUI->>ContributionsUI: Display total, attainable, effective, and maximum coverage
Possibly related issues
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
cemreinanc
marked this pull request as draft
August 8, 2026 07:28
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@front_end/messages/en.json`:
- Around line 927-929: Update the effectiveCoverageInfo translation to state
that both total coverage and total attainable coverage are weighted by each
question’s question_weight before division, while preserving the leaderboard
equivalence description.
In
`@front_end/src/app/`(main)/(leaderboards)/contributions/components/project_contributions.tsx:
- Around line 114-120: Update the attainable_coverage display branch in the
project contributions component to pass contribution.coverage directly to
formatPercent instead of coercing nullish values to 0. Preserve the existing
attainable coverage formatting and render null coverage as the formatter’s "-"
fallback.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 93321de6-2d38-40b5-bc0c-b2dec11b1b3b
📒 Files selected for processing (7)
front_end/messages/en.jsonfront_end/src/app/(main)/(leaderboards)/contributions/components/project_contributions.tsxfront_end/src/types/scoring.tsquestions/models.pyscoring/serializers.pyscoring/utils.pytests/unit/test_questions/test_models.py
Contributor
🚀 Preview EnvironmentYour preview environment is ready!
Details
ℹ️ Preview Environment InfoIsolation:
Limitations:
Cleanup:
|
The effectiveCoverageInfo help text described dividing unweighted sums, but both sums are weighted by question weight. The weighting is required for the value to match the tournament leaderboard, which computes coverage as sum(coverage * question_weight) / sum(attainable_coverage * question_weight). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NmzezEyuu7d2sX214hZWqZ
SylvainChevalier
temporarily deployed
to
testing_env
August 8, 2026 09:31 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
SylvainChevalier
temporarily deployed
to
testing_env
August 8, 2026 09:31 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
SylvainChevalier
temporarily deployed
to
Preview
August 8, 2026 09:40 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces the concept of "attainable coverage" to the scoring system, which represents the maximum coverage a forecaster could achieve on a question based on when it actually closed relative to its scheduled close time. This allows the system to fairly evaluate coverage on questions that resolve early.
Key Changes
New
get_attainable_coverage()method on Question model that calculates the fraction of the scheduled forecasting window during which a question was actually open:(effective_close_time - open_time) / (scheduled_close_time - open_time)Updated Contribution tracking to include
attainable_coveragefield alongside existingcoveragefield, populated only for successfully resolved questionsEnhanced coverage metrics in project contributions display:
"X% (Y%)"Updated leaderboard max_coverage calculation to use weighted attainable coverage instead of just question weights, ensuring coverage is measured against what was actually achievable
Added comprehensive test coverage for the new
get_attainable_coverage()method with multiple scenarios (full window, early close, exact scheduled close)Updated UI labels and help text in English translations to explain the new attainable coverage concept
Implementation Details
attainable_coverageis only populated for successfully resolved questions (excludes ambiguous/unknown resolutions)_get_attainable_coverage()ensures consistent logic across the codebasehttps://claude.ai/code/session_01NmzezEyuu7d2sX214hZWqZ
Summary by CodeRabbit