Repository navigation
Resume -re in the fit mode and in all channels of the interrupted run - #183
Merged
Merged
Conversation
A colour flag without a step runs step 3 with the LED fits. The guard against that checked -s3_fast only, so -re -r, -re -g and -re -rgb first ran the full step 3 on every image, which overwrote the partial results, and the restart then found nothing left to analyse. Plain -re resumes the red channel only, so an interrupted green, blue or RGB run could not be resumed at all. The guard now also leaves out -re, and the colour flags only select the channels of the restart. -re alone, an explicit -s3 next to -re and a colour flag next to -s1, -s2 or -coord behave as before. The CLI documentation listed --r, which argparse rejects as ambiguous, and no channel flags for -s3_fast and -re. The help of -re now says to give the channel flags of the interrupted run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart always analysed without the LED fits. After an interrupted -s3 run, the resumed images therefore got the -s3_fast columns, and the analysis refused the channel because its result files were written with different column layouts. -s3 is the slow run, so it is the one most likely to be interrupted. The restart now reads the mode from the headers of the result files of its channels: only a file written with fits names the fit columns, fit_success among them, in every version of the header. The number of columns is not used, because it changes between LEDSA versions. If no channel has a result file yet, the restart runs without fits as before. If the files of a channel, or those of different channels, were written in both modes, it stops and says how to go on instead of guessing, because one restart writes all its channels in one mode. Empty files, which an interruption can leave, are ignored. The help of -re and the CLI documentation describe the mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart took the images still to analyse from the first of its channels only. An image is analysed for one channel after the other, so an interrupted multi-channel run usually stops after an image was written for some channels but not for the others, and with several cores this happens for several images. The restart skipped them, and the analysis then left them out of the other channels. An image is now analysed again, for all channels of the restart, unless every one of them has its result file. The commented-out exit for restarts of several channels is removed. The images still to analyse are taken from image_infos_analysis.csv in its order, instead of looking up the name of every analysed image in that file, which read it once per image and wrote them in set order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kriboe90
added a commit
that referenced
this pull request
Nov 9, 2026
Brings the fixes of the step 3 flags and the restart into develop: #180 (-s3_fast with -r, -g or -b no longer also runs the full step 3), #182 (a colour flag picks the channels of -re instead of running step 3) and #183 (-re resumes in the fit mode and in all channels of the interrupted run). They are merged before the v0.13.0 cleanup (#181) lands on master, so that develop takes the fixes without the removals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kriboe90
added a commit
that referenced
this pull request
Nov 9, 2026
…-0-cleanup-cea6d8 Brings the fixes of the step 3 flags and the restart (#180, #182, #183) into the cleanup. Two conflicts, both from changes on adjacent lines: - doc/cli_documentation.rst: the -s3_fast row keeps the description without the background-corrected sums and takes the channel flags. - ledsa/data_extraction/step_3_functions.py: find_fit_leds_for_restart goes before _generate_led_analysis_data, which keeps the signature without the noise, side band and background floor parameters. The restart reads the fit mode from the header line of the column names, which still starts with id and lists fit_success only with the fits, so it works with the shorter header of the cleanup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
-renow resumes in the mode of the interrupted run: with the LED fits after-s3, without them after-s3_fast. #182 named this as a limitation it left unchanged. Until now the restart always analysed without fits.ledsa/data_extraction/step_3_functions.py:find_fit_leds_for_restart(channels)reads the header of every result file of the selected channels. Only a file written with fits names the fit columns in its column header.fit_successis one of them in every version of the header since 2020, including the shorter one of the v0.13.0 cleanup. The mode is read from the names, never from the number of columns.ValueErrorthat says how to go on. One restart writes all of its channels in one mode, so it cannot guess._find_result_fileslists the files for this and for_find_analysed_img_ids. It returns nothing for a channel without a directory, because the mode is read before theDataExtractorcreates the directories.ledsa/core/parser_arguments_run.py: the restart passes that mode to theDataExtractorasfit_leds, the same way-s3and-s3_fastchoose theirs.Images of a multi-channel restart (second commit):
setup_restarttook the images still to do fromself.channels[0]alone. Now an image is analysed again, in all channels of the restart, unless every one of them has its result file. The commented-out exit for multi-channel restarts is gone. The remaining images now come straight fromimage_infos_analysis.csv, in its order. Before, the name of every analysed image was looked up in that file, which read it once per image and wrote the list in set order.doc/cli_documentation.rstand the help of-redescribe both.Why
After an interrupted
-s3run, the resumed images got the-s3_fastcolumns, and the analysis then refused the channel:-s3is the slow run, so it is the one most likely to be interrupted. On the 100-LED test images it takes about 9 s per image, against about 8 ms for-s3_fast.The task offered the multi-channel image selection as optional. I made the change because each image is analysed one channel after the other. An interrupted
-rgbrun therefore almost always stops between the channels of an image, and with several cores this happens for up to one image per worker. With the first channel deciding, the restart skipped those images for the remaining channels, and the analysis left them out there ("No analysis results for the image IDs …"). Since #182 makes-re -rgbreachable, this hit nearly every interrupted RGB run.For the reviewer
run_data_extraction_argumentsand the same test file. Merge fix: honor fast step three for individual color channels #180 and Let a colour flag pick the channels of -re instead of running step 3 #182 first, and the diff here shows only this change.run_data_extraction_argumentsand passed to the constructor, rather than set insidesetup_restart. That is where-s3and-s3_fastchoose theirs. It also means disagreeing files stop the run before theDataExtractorasks for missing config values.-reresumed after an interrupted-s3run, or an interrupted rerun of step 3 in the other mode over complete results. In the second case every image already has a file, so the restart could not see which images are left anyway. Real output, on the channel the old restart produced in the end-to-end check below:-re -rgbafter a red-only run redoes red as well.fit_success, which_create_headerkeeps on that branch. The tests take their headers from_create_headeror write one with the base columns only, so they use none of the columns the cleanup removes.led_search_areas_with_coordinates.csv(no-coord) fails on NumPy 2, because_append_nans_to_paramsinfile_handling.pyusesnp.NaN.pyproject.tomlpinsnumpy ~= 1.26.4, so supported installs are not affected until that pin is lifted. It is the same on master and on the cleanup branch.Tests
test_parser_arguments_run.py, the restart gets thefit_ledsof the result files for-re,-re -gand-re -rgb, in both modes. The test files carry headers from_create_headerand from LEDSA before the background-corrected sums, which had fewer columns. Further tests cover taking the mode from the channels that have files, ignoring an empty file, and stopping when the files disagree within a channel or between channels. Against the dispatcher of Let a colour flag pick the channels of -re instead of running step 3 #182, the 8 cases that expect fits or a stop fail.test_restart.pychecks thatsetup_restartpicks every image that misses a channel, for an interruption inside an image and for images finished out of order by several cores. With the first-channel rule, 2 of its 3 tests fail.-s1 -s2 -coord -s3, delete the result files of images 3 and 4, then run-reand load channel 0 withcreate_binary_data. With Let a colour flag pick the channels of -re instead of running step 3 #182, images 3 and 4 are written without fits and loading fails with the error above. With this branch they are written with fits and 4 of 4 load.-s3_fastinstead, the resumed images are written without fits and 4 of 4 load.-s3_fast -rgb, delete the green and blue files of image 3 and all files of image 4.-re -rgbthen analyses images 3 and 4 in all channels, and each channel loads 4 of 4.03_test_s3.robotruns plain-reon an emptiedchannel0, which takes the path without result files.🤖 Generated with Claude Code