Testing/aggregation playground - #5044
Draft
lsabor wants to merge 11 commits into
Draft
Conversation
deafult year_performance type
…ction - Vectorize LearnedReputationWeighted.calculate_weights (single np.exp/power calls over arrays instead of per-element Python-loop calls) and switch ReputationWeighted.get_reputations to a bisect lookup instead of reversing and linear-scanning each user's reputation history. - Remove a dead duplicate calculate_weights override on PeerScoreReputationWeighted that shadowed the above fix. - Rewrite get_user_forecast_history as a single sweep over the timestep grid with add/remove events, replacing the previous per-forecast slice-and-copy loop; verified equivalent to the original via a dedicated equivalence test (random fuzzing + explicit edge cases) against the original algorithm kept as a reference oracle. - Add Reputation model typing hints (user_id, objects) for parity with the other scoring models. - Add TestAggregationSpeed/TestAggregationHeavyLoad perf tests exercising get_aggregation_history for Recency/Single/YearPerformance aggregations. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Thread a single np.asarray(forecast_set.forecasts_values) conversion through calculate_forecast_values/get_range_values/means/histogram in calculate_aggregation_entry instead of each independently re-converting the same list; audited every consumer for in-place mutation and added defensive .copy() only where needed (MeanAggregatorMixin mutates via a view into row 0). - get_reputations now returns a values array directly instead of a list of Reputation objects, dropping a redundant extraction loop in calculate_weights. - Cache ForecastSet.timesteps as epoch-second floats (computed once per forecast, not per snapshot) so LearnedReputationWeighted's decay-ratio computation is a single vectorized numpy op instead of per-element datetime/timedelta arithmetic. - ReputationWeighted can now precompute every forecaster's reputation value across a known forecast_history via one batched np.searchsorted call per user, instead of one live bisect call per (timestep, forecaster) pair; falls back to the live lookup when the history isn't known up front. Verified equivalent via a dedicated randomized equivalence test. - Increase TestAggregationHeavyLoad's intensity: 1500 users, 5000 forecasts, numeric question with a 201-value continuous_cdf (previously binary). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…istory Profiling the numeric-question heavy-load path showed np.asarray(forecast_ set.forecasts_values) in calculate_aggregation_entry dominating (~78% of total runtime, 12s/15.5s) for wide (201-value) continuous CDFs: every timestep re-converts every active forecaster's raw Python-float list from scratch, even though an individual forecaster's values don't change between consecutive timesteps they're active for. Store each forecast's get_prediction_values() as a pre-converted numpy array (computed once, when it becomes active in the sweep) instead of the raw list. forecasts_values stays a plain Python list of these arrays rather than one stacked 2D array, so `if forecast_set.forecasts_values:` elsewhere keeps working (a multi-row ndarray's truthiness is ambiguous and would raise). Converting a list of same-shape ndarrays is ~20x cheaper than converting a list of raw float lists, since it skips per-element boxing. Result: numeric/recency_weighted heavy-load benchmark (1000 users, 3000 forecasts, 201-value CDF) drops from 15.5s to 3.6s. Also reduce TestAggregationHeavyLoad back to 1000 users / 3000 forecasts (from the temporary 1500/5000 bump) while keeping the numeric question type, now that it isn't needed to make this bottleneck reproduce. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
scoring/fast_scoring.py bypasses evaluate_question/get_aggregation_history entirely for benchmark_aggregations: every forecast is reduced to a single scalar (its PMF value at the question's resolution bucket) once, and all aggregation/scoring math runs on compact (num_forecasters, num_timesteps) arrays instead of full PMF/CDF vectors at every timestep. - Mean-based methods (single_aggregation, year_performance) work on any question type; median-based methods (unweighted, recency_weighted) work when multiple_choice is excluded, since MC's median renormalization needs the full PMF - enforced via the new --exclude-question-type validation. - Peer and baseline scoring, both interval and a T=1 spot-scoring fast path. - Reputation-weighted methods reuse the existing PeerScoreReputationWeighted/ YearPerformanceReputationWeighted reputation histories, with an optional batch-level preload to avoid re-querying the same forecasters' full score history once per question. - Per-question data reduction is disk-cached (keyed by question id) so repeated runs over the same questions skip straight to aggregation/scoring (--rebuild-cache to force a refresh). Validated against evaluate_question's actual output across 150+ (question x method x score-type) combinations spanning binary/numeric/ multiple_choice questions, matching to floating-point precision. Also adds --a/--b flags for SingleAggregation-style reputation weighting parameters, laying groundwork for scipy.optimize-driven tuning. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
🚀 Preview EnvironmentYour preview environment is ready!
Details
ℹ️ Preview Environment InfoIsolation:
Limitations:
Cleanup:
|
…nWeighted from it Replaces PeerScoreReputationWeighted's live per-question Score query with a lookup against precomputed "average_peer_score" Reputation records, mirroring YearPerformanceReputationWeighted's approach. Both now share a PrecomputedReputationWeighted base. The migration backfills the new reputation type by replaying every user's peer scores on public questions in chronological order. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rkers, grid search fast_scoring.py: - Add support for medalists/silver_medalists/gold_medalists (RecencyWeighted combined with a medal-holder reputation multiplier) and metaculus_pros/ joined_before_date (RecencyWeighted combined with a static per-user filter), rounding out coverage of every aggregation registered in AGGREGATIONS. - Split REPUTATION_WEIGHTED_CLASSES into DECAYED_REPUTATION_CLASSES (mean-based decay/reputation blend) and RAW_REPUTATION_CLASSES (median-based raw 0/1 multiplier), fixing a bug where spot scoring would have applied the wrong weight formula to the new medal methods; spot scoring for those explicitly raises NotImplementedError instead. - Add subsample_timesteps for --sample-timesteps (approximate scoring on a reduced timestep grid). benchmark_aggregations.py: - --aggregation-method now accepts "method,param1,param2" for grid-searching a method's parameters (single_aggregation/year_performance's a,b; joined_before_date's cutoff date) across many specs in one run, each reported as its own labeled row. - --workers parallelizes the per-question scoring loop via ProcessPoolExecutor (auto-detects CPU count by default); --joined-before adds a global default cutoff for joined_before_date. - Results now print a settings summary (all options that shaped the run) and a rank/vs_best comparison table instead of a redundant per-row question count; progress reports elapsed/ETA in place. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…eputation preload OOM New aggregations: - spot_sensitive: for spot_peer-scored questions, uses a_spot=0/b_spot instead of the normal a/b for any timestep strictly before the question's spot scoring time (falls through to normal behavior otherwise). Wired into fast_scoring's vectorized path via _spot_sensitive_ab, with b_spot grid-searchable through benchmark_aggregations's comma-spec syntax. - peer_threshold_-20_coverage_50 / peer_continuous_with_coverage: two new Reputation types derived from average_peer_score's (r_i, c_i) history - a hard threshold (rho=-20, gamma=50: floor to MINIMUM_REPUTATION below either bound, else r_i - rho) and a softplus-smoothed continuous version (rho=0, gamma=50) of the same idea, addressing negative/low-coverage reputations swamping the aggregate while still gating out new accounts. Backfilled in scoring/migrations/0022_reputation.py by replaying the same peer-score history average_peer_score is built from. Corresponding PeerThresholdReputationWeighted/PeerContinuousReputationWeighted + Aggregation classes added to utils/the_math/aggregations.py. Fix: a benchmark_aggregations run requesting several reputation-weighted methods together (e.g. single_aggregation + spot_sensitive + the two new peer_* methods) could OOM-kill a worker process. Root cause was two-fold: (1) reputation preloading was keyed by method name, so methods sharing the exact same underlying Reputation records (single_aggregation and spot_sensitive both read average_peer_score) redundantly duplicated a multi-million-row preload; (2) preloaded histories were held as full lists of Django model instances, and forking --workers processes touches every object's refcount on read, forcing copy-on-write page duplication per worker even for read-only access. Fixed by deduplicating preloads by underlying reputation_type, and by converting each user's history to a compact (times, values) numpy-array pair immediately after fetching - verified this drops the pickled size of a full multi-type preload from an unusable size to ~58MB for the full forecaster population. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…gregation-playground
…caching Redesign --method into a composable key=value|... grammar so any predefined aggregation (recency_weighted, spot-sensitive variants, peer threshold/continuous, etc.) can be reproduced via composition instead of dedicated classes. Consolidate the reputation-weighting class family into a single DecayReputationWeighted, move reputation_type under each weight token (allowing multiple independently-typed reputation weights per spec), and make spot-sensitivity a composable boolean trait. Add disk caching (scoring/_reputation_cache/) for per-reputation-type history, mirroring the existing per-question fast-scoring cache, cutting multi-minute warmup before benchmark runs down to seconds on repeat/varied batches. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.