Skip to content

docs: ship the leakage chart renderer as a public script - #193

Merged
shivamlalakiya merged 1 commit into
mainfrom
docs/leakage-chart-script
Sep 6, 2026
Merged

docs: ship the leakage chart renderer as a public script#193
shivamlalakiya merged 1 commit into
mainfrom
docs/leakage-chart-script

Conversation

@shivamlalakiya

Copy link
Copy Markdown
Contributor

Why

r/dataisbeautiful requires a comment linking the tool and the data source. The renderer that produced the leakage figure lived in scripts/announce/, which is gitignored, so there was no public link to give.

What

  • scripts/render_leakage_chart.py, sitting next to the two leakage experiments it re-runs. It calls the same _panel / _clf helpers and the same FiscalYearGroupedSplitter walk-forward CV as leakage_experiment.py and real_data_leakage_experiment.py, but keeps the per-fold scores instead of averaging them, so the figure shows the shape of the backtest rather than one number per condition.
  • --cached re-plots from a saved score file (the real panel takes ~9 minutes to refit), --out picks the output directory.
  • .gitignore: scripts/announce/ and the other local working-note paths are now ignored explicitly rather than by accident, and the rendered figure files stay out of the tree.
  • CHANGELOG entry under ## [Unreleased].

No library code changes; nothing under philanthropy/ is touched.

Verification

Re-plotted from the committed run's score cache and diffed the numbers against docs/explanation/benchmarks.md and the preprint:

$ python scripts/render_leakage_chart.py --cached --out /tmp/chart
synthetic: as_of_mean 0.6248, whole_mean 0.7504
real:      as_of_mean 0.4820, whole_mean 0.8578

Matches the published 0.625 / 0.750 and 0.482 / 0.858.

r/dataisbeautiful requires a comment linking the tool and the data source. The renderer lived in scripts/announce/, which is gitignored, so there was nothing to link to.

Moves it to scripts/render_leakage_chart.py, next to the two leakage experiments it re-runs. Same walk-forward cross-validation, per-fold scores kept instead of averaged, so the figure shows the shape of the backtest rather than one number per condition. Reproduces the published means: synthetic 0.625 as-of against 0.750 whole-history, real 0.482 against 0.858.

Rendered output (PNG, SVG, the numbers JSON, the score cache) stays gitignored. scripts/announce/ and the other local working-note paths are ignored explicitly rather than by accident.
@shivamlalakiya
shivamlalakiya merged commit a226c1d into main Sep 6, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant