Skip to content

Add a multinomial fast path to the charge-diffusion sampler - #196

Merged
roytsmart merged 2 commits into
mainfrom
perf/diffuse-electrons-multinomial
Aug 17, 2026
Merged

Add a multinomial fast path to the charge-diffusion sampler#196
roytsmart merged 2 commits into
mainfrom
perf/diffuse-electrons-multinomial

Conversation

@roytsmart

Copy link
Copy Markdown
Collaborator

Summary

The inner loop of _electrons_measured_numba relocates each of a photon's electrons individually, which costs O(m) per photon. At short wavelengths a single photon liberates hundreds to thousands of pairs, and this loop dominates the Monte Carlo — it is a large part of why the signal() comparisons in #195 are so slow to iterate on.

Since the relocations are i.i.d. on integer pixel offsets, the counts landing at each offset are multinomial. The new _diffuse_electrons helper samples that multinomial directly when m is large: electrons are partitioned across a bounded window of x offsets via conditional binomials, then each occupied column across y offsets, with per-bin probabilities given by the Gaussian integrated over each pixel (an erf difference). This costs O(window) rather than O(m). Probability mass beyond the ±6σ window (~1e-9) folds into the edge bins, so charge is conserved exactly, and off-sensor columns/bins are dropped when wrap=False, matching the per-electron behavior.

The brute-force path is retained, as the small-m branch of the same function: per-photon, the sampler compares m against _factor_multinomial times the window area and takes whichever path is cheaper. Setting the module constant _factor_multinomial to inf forces per-electron sampling everywhere (the exact previous behavior), and 0 forces the multinomial path, so either can be pinned for testing.

No public API changes, no new files.

Benchmarks

32×32 grid, e2v-CCD97-like parameters (thickness_depletion=2 μm, thickness_substrate=14 μm, width_pixel=5 μm, wrap=True), wall time per image; auto is the default crossover (_factor_multinomial = 0.25, tuned from these measurements):

wavelength photons/px brute multinomial auto speedup
1.5 Å 200 34.9 s 11.0 s 11.0 s 3.2×
5.0 Å 200 11.6 s 7.1 s 7.1 s 1.6×
20 Å 1000 14.1 s 18.9 s 14.1 s 1.0×
100 Å 5000 14.5 s 56.1 s 14.5 s 1.0×
1000 Å 20000 8.4 s 75.7 s 8.4 s 1.0×

The crossover tracks the faster path at every wavelength (auto/best = 1.00 throughout), so long-wavelength workloads are unaffected.

Validation

  • Forced-path comparison over ~5×10⁷ electrons (3000 photons/trial at 5 Å): totals, spatial means, and spatial variances of the brute-force and multinomial paths agree within Monte Carlo noise (means to ~4×10⁻³ px, variances to ~2×10⁻³ relative).
  • New test_electrons_measured_factor_multinomial forces each path via monkeypatch and checks the measured spread against the analytic optika.sensors.charge_diffusion width, at a wavelength short enough (~680 pairs/photon) to exercise the large-m regime.
  • All 16 electrons_measured tests and the 40 downstream _materials_test.py tests touching signal/electrons_measured/uncertainty pass locally.

Follow-up worth knowing about

While profiling I found that probability_of_n_pairs takes ~1.6 s per call (for a 20-element PMF) and is recomputed on every electrons_measured call — for single-wavelength calls this fixed cost exceeds the entire electron loop. Caching it would speed up every Monte-Carlo-heavy workflow, including the #195 test loop, and is probably a bigger win than this PR for long-wavelength work. Left out of scope here since it touches evaluation caching rather than the sampler.

🤖 Generated with Claude Code

https://claude.ai/code/session_011q4461XE8hCcZCKsViMQC1

@codecov

codecov Bot commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.57%. Comparing base (64e05e6) to head (5f2727f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #196   +/-   ##
=======================================
  Coverage   99.57%   99.57%           
=======================================
  Files         118      118           
  Lines        6829     6857   +28     
=======================================
+ Hits         6800     6828   +28     
  Misses         29       29           
Flag Coverage Δ
unittests 99.57% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The inner loop of `_electrons_measured_numba` relocated each of a
photon's electrons individually, which costs O(m) per photon and
dominates the Monte Carlo at short wavelengths, where a single photon
can liberate thousands of pairs.

The relocations are i.i.d. on integer offsets, so the counts landing at
each offset are multinomial. `_diffuse_electrons` samples that
multinomial directly for large m: electrons are partitioned across a
bounded window of x offsets via conditional binomials, then each
occupied column across y offsets, with per-bin probabilities given by
the Gaussian integrated over each pixel (an erf difference). This costs
O(window) rather than O(m). Probability mass beyond the +/- 6 sigma
window (~1e-9) folds into the edge bins, so charge is still conserved
exactly.

The per-electron path is retained as the small-m branch of the same
function, selected by comparing m against `_factor_multinomial` times
the window size; setting that module constant to `inf` or `0` forces
either path, and a new test does so to check both against the analytic
charge-diffusion width.

Benchmarks (32x32 grid, e2v-CCD97-like parameters, wrap=True), where
`auto` is the default crossover:

  wavelength  photons/px    brute    multinomial   auto
      1.5 A          200   34.9 s       11.0 s    11.0 s  (3.2x)
      5.0 A          200   11.6 s        7.1 s     7.1 s  (1.6x)
     20.0 A         1000   14.1 s       18.9 s    14.1 s  (1.0x)
    100.0 A         5000   14.5 s       56.1 s    14.5 s  (1.0x)
   1000.0 A        20000    8.4 s       75.7 s     8.4 s  (1.0x)

The crossover tracks the faster path at every wavelength, and the
distributions agree: totals, spatial means, and spatial variances of
the two forced paths match within Monte Carlo noise over ~5e7
electrons.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011q4461XE8hCcZCKsViMQC1
@roytsmart
roytsmart merged commit 5943917 into main Aug 17, 2026
12 checks passed
@roytsmart
roytsmart deleted the perf/diffuse-electrons-multinomial branch August 17, 2026 18:53
roytsmart added a commit that referenced this pull request Aug 17, 2026
The multinomial fast path (#196) and PMF caching (#197) make the
Monte Carlo cheaper, and since the VMR is independent of the
illumination level, fewer photons per pixel buy more wavelength
samples at no statistical cost.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0192abyvBf2Zw5rq5CQ62JFb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant