Skip to content

v0.13.0: hold back the unvalidated features on develop - #181

Merged
kriboe90 merged 6 commits into
masterfrom
claude/version-0-13-0-cleanup-cea6d8
Nov 9, 2026
Merged

kriboe90 merged 6 commits into
masterfrom
claude/version-0-13-0-cleanup-cea6d8

Conversation

@kriboe90

@kriboe90 kriboe90 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

v0.13.0 is master without the recent features that are not validated yet. They are kept on the new develop branch: master before this PR, including #180 and #183, plus version 0.14.0.dev0 and CI on every push.

Please merge with "Create a merge commit", not squash or rebase, which would change the commit IDs below. Each removal is one commit, so a feature validated on develop comes back to master with git revert <commit>:

Commit Removes
cae109c the moving average num_moving_avg_images
bfc5c49 the SNR / layer weighting (snr_full_weight, min_layer_weight) and the read noise of the sums (noise_sum_col_val)
fae2ed7 the background corrections bgsub_sum_col_val, bgsub2_sum_col_val, sidebg_sum_col_val / sidebg_pedestal and the config.ini options sideband_distances, background_floor, sideband_exclusion_margin

The removals go in this order so that step 3 always writes a layout the reader already knows by its number of columns. Removing the background first would have written 7 columns, which the reader takes for the old bgsub layout.

Kept: the black level from the optical black, the unclipped data, average_images with n_summarize, the temperature correction, the nonlinear solver, the LED fit, the colour correction, several quantities per run, and the rest of v0.12.0 and of the PRs merged since.

Also in here

  • 9cf52b3: a config.ini / config_analysis.ini that still sets a removed option gets a note that the option is ignored. Without it, a v0.12.0 config_analysis.ini (snr_full_weight = 10.0) would lose its weighting silently.
  • 3302faf: fixes a bug master has as well. The linear solver is meant to leave out an LED whose I/I0 is not finite, but it zeroed that LED's row by multiplying with weight 0, and NaN × 0 is NaN. nnls then stopped with "array must not contain infs or NaNs".
  • 4f83091: version 0.13.0. The README now covers the dense-smoke limitation and where the removed features went, and gives the Python requirement as 3.9–3.12. The commented-out lambda_reg study in ExtinctionCoefficientsLinear.py, which had a hard-coded local path, is gone.

Rebased onto master after #180 and #183. Two places conflicted: the signature of _generate_led_analysis_data, next to the new find_fit_leds_for_restart of #183, and the -s3_fast row of the CLI table. The restart of #183 tells the fit mode from the header of the result files, which keeps fit_success in the 0.13.0 layout, and its tests pass here.

Changes for users of v0.12.0

  • Step 3 writes sum_col_val, mean_col_val and max_col_val: 5 columns, 19 with the fit, where v0.12.0 wrote 10 and 24. Result files of v0.11.0 and v0.12.0 are still read, including their background columns.
  • Without the weighting, the linear solver leaves out LEDs at I/I0 ≤ 0, as the stacked analysis already did in v0.12.0. Where the transmission reaches the read noise, k comes out too low: 14–31 % at 1.24 and 1.64 1/m in the Warsaw tests V008/V009 (Unclipped channel data, the read noise of a sum and the SNR weighting of the linear solver #170), where v0.12.0 returned NaN. At fog stages that still carry signal, the weighting had changed k by less than 1 %.

Test plan

  • Unit tests: 365 passed (434 on master; the removed features' tests are gone, the generic ones moved). Every commit of this PR passes them on its own.
  • Acceptance tests locally against this branch: 26 passed, 0 failed (run with PYTHONPATH set to the branch, as the local editable install points elsewhere).
  • Same synthetic acceptance images through master (c21f4f3) and this branch. Every result is identical: the step 3 columns sum/mean/max and the fit columns (except fit_time) of 16 result files, and the extinction coefficients of 9 runs: linear for sum_col_val, mean_col_val and max_col_val with -s3_fast and with -s3, average_images, nonlinear and the stacked analysis. Master ran with its default snr_full_weight = 10, which switches itself off for JPG images (no read noise), so this also shows that nothing else changed.
  • Docs build without warnings (locally without sphinx_copybutton, which isn't installed here).
  • RAW images: no RAW test data was available, so the RAW path is covered only by the unit tests with fake RAW images.

After merging

  • Merge master into develop and revert cae109c, bfc5c49 and fae2ed7 there, keeping its version 0.14.0.dev0 and dropping the README section on the features in validation, so develop keeps the features and later merges carry only fixes.
  • Tag v0.13.0 and publish the release (this uploads to PyPI).

🤖 Generated with Claude Code

kriboe90 added a commit that referenced this pull request Nov 9, 2026
Brings the fixes of the step 3 flags and the restart into develop:
#180 (-s3_fast with -r, -g or -b no longer
also runs the full step 3), #182 (a colour flag picks the channels of
-re instead of running step 3) and #183 (-re resumes in the fit mode
and in all channels of the interrupted run). They are merged before the
v0.13.0 cleanup (#181) lands on master, so that develop takes the fixes
without the removals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kriboe90 and others added 6 commits November 9, 2026 13:54
A config file keeps the options of the version that wrote it, and the
parser reads every option a file sets whether or not LEDSA uses it. An
option a release removes would therefore stop taking effect without a
word, and the result of an analysis would change unnoticed: a
config_analysis.ini of v0.12.0 sets snr_full_weight = 10.0, which
v0.13.0 no longer has.

report_removed_options names every such option once per process, with
what became of it. config.ini and config_analysis.ini each get a
REMOVED_OPTIONS registry, empty here; the commits that remove options
add them, so that reverting one of those commits takes its entries
with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
num_moving_avg_images smoothed the reference property of every LED with
a centred moving average over neighbouring images. It came with the SNR
weighting and has not been validated on the data of an experiment, so
v0.13.0 leaves it out; develop keeps it. average_images with n_summarize,
which summarizes consecutive images into one, stays.

A config_analysis.ini that still sets num_moving_avg_images is told
that the option is ignored. Reverting this commit brings the option
back as it was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The linear solver weighted every LED with min(1, SNR / snr_full_weight)
and set a layer to NaN when the path weighted mean weight of its LEDs
stayed below min_layer_weight. The SNR came from noise_sum_col_val, the
read noise of the sum, which step 3 estimated from the standard deviation
of the optical black pixels. Neither has been validated beyond the
Warsaw tests V008 and V009, so v0.13.0 leaves them out; develop keeps
them.

What stays from the same change is the unclipped data: the black level
of every Bayer color is still taken from the optical black, and the
linear solver still leaves out an LED whose relative intensity is at or
below zero. Without the weighting that rule alone decides, as it already
did in the stacked analysis. Its known limitation is documented with
the solver: where the transmission reaches the read noise, the LEDs that
are left are those whose noise happened to be positive, and the
coefficients come out low (14 to 31 % at 1.24 and 1.64 1/m in V008 and
V009, where the weighting gave NaN).

The read noise goes from the RAW reader as well: _optical_black_mean
and read_channel_data_and_mask_from_img return to the names and the
results they had before the noise was added, and ImageChannels builds
the data and the mask of a channel. Step 3 writes the layouts of
v0.11.0 again (10 names, 24 with the fit), so the reader names their
columns as before; the v0.12.0 layouts with noise_sum_col_val are still
read.

config_analysis.ini files that set snr_full_weight or min_layer_weight
are told that the options are ignored. test_led_weighting.py goes; the
rule for LEDs at or below zero moves to
test_extinction_coefficients_linear.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The linear solver marks an LED whose relative intensity is not finite as
not measured, but it zeroed the row of such an LED by multiplying it
with its weight of 0, and the optical depth of a NaN intensity is NaN,
which the product keeps. nnls then stopped with "array must not contain
infs or NaNs" instead of leaving the LED out. An LED whose reference
intensity is 0 gives such a value.

The optical depth of a left out LED is now set to 0 with np.where. For
finite intensities the system is the same as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 3 wrote three background-corrected sums besides the plain sum:
bgsub_sum_col_val (median of a border ring of the search area),
bgsub2_sum_col_val (mean of that ring) and sidebg_sum_col_val (mean of
side bands next to the LED strip, with sidebg_pedestal). They have not
been validated on the data of an experiment, so v0.13.0 leaves them out;
develop keeps them.

Step 3 writes sum_col_val, mean_col_val and max_col_val again, plus the
fit columns with -s3. The fast layout is the one of the versions before
v0.11.0; the fitted one has 20 names and is new, because those versions
glued max_col_val to led_center_x. The reader still knows every earlier
layout, so result files of v0.11.0 and v0.12.0 can be analysed,
including their background-corrected columns.

The search areas keep being stacked by their shape and calibrated once
for all channels; only the side band pixels leave the selection. The
options sideband_distances, background_floor and
sideband_exclusion_margin leave config.ini, and a config.ini that still
sets them is told that they are ignored.

The tests of the corrections go: test_background_subtraction.py,
test_ring_background_mean.py, test_sideband_background.py and the
acceptance suite 10. What they held beyond the corrections moves:
- the column layouts to test_file_handling.py, now with the layouts of
  v0.13.0 and v0.12.0,
- the Bayer channel mask to test_image_channels.py,
- the NaN check of the analysed quantity to test_analysis_quantities.py,
- the acceptance cases for several quantities per run (now sum_col_val
  and mean_col_val) and for fitted step 3 results to suite 08, renamed
  to 08_test_analysis_quantities.robot.
The README section and the documentation of the corrections go as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
v0.13.0 is v0.12.0 with the clock offset, the faster step 3 and the
removal of the features that are not validated yet: the background
corrections, the SNR weighting with the read noise and the moving
average. They stay on develop.

The README says where they went, and it describes what remains in dense
smoke without the weighting: the linear solver leaves out an LED at or
below zero, and the coefficients come out low where the transmission
reaches the read noise. Its Python requirement names the versions
pyproject.toml allows (3.9 to 3.12) instead of 3.8.

The commented-out study of lambda_reg goes from
ExtinctionCoefficientsLinear.py; it wrote to a directory of one machine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kriboe90
kriboe90 force-pushed the claude/version-0-13-0-cleanup-cea6d8 branch from db74f13 to 4f83091 Compare November 9, 2026 13:03
@kriboe90
kriboe90 marked this pull request as ready for review November 9, 2026 13:26
@kriboe90
kriboe90 merged commit 110743d into master Nov 9, 2026
5 checks passed
kriboe90 added a commit that referenced this pull request Nov 9, 2026
Brings v0.13.0 (#181) to develop. develop keeps its version 0.14.0.dev0; the features that #181
removed from master come back with the reverts that follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant