import_sofima_20260912 by trautmane · Pull Request #253 · saalfeldlab/render

trautmane · 2026-09-13T23:39:50Z

Added spark client for importing SOFIMA results and cleaned-up AlignmentPipelineStep interface.

… logic in the java ImportSofimaClient

…interface and add javadoc comment with reference information to each pipeline step implementation to document the mapping instead

…b-range of z layers from that client

Copilot

🟡 Changes recommended

There are a couple of concrete correctness/quality issues to address (misleading CLI parameter description and a driver-side collect() that can cause avoidable memory pressure), and the new pipeline step lacks direct test coverage.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR introduces a new Spark pipeline step for importing SOFIMA displacement fields into Render stacks, and simplifies the alignment pipeline step abstraction by removing per-client “default step id” plumbing in favor of enum-driven step creation.

Changes:

Added SOFIMA import support end-to-end: SofimaParameters, a Spark ImportSofimaClient, and a new IMPORT_SOFIMA pipeline step id.
Simplified AlignmentPipelineStep by removing getDefaultStepId() and updating all step clients/tests accordingly.
Minor housekeeping updates (Docker tag comment bump, additional db-dump stage options, small Java 21 stream/list modernizations).

File summaries

File	Description
render-ws-with-mongo-db/Dockerfile	Updates example image tags in comments (1.0.2 → 1.0.3).
render-ws-with-mongo-db/db-dump-google-collections.sh	Adds additional supported stage options including SOFIMA/IC3D.
render-ws-spark-client/src/test/java/org/janelia/render/client/spark/pipeline/AlignmentPipelineParametersTest.java	Adjusts tests for step-id/interface cleanup and Java 21 list APIs.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/zspacing/ZPositionCorrectionClient.java	Removes default-step-id method; updates step docstring.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/tile/TileIdHackClient.java	Removes default-step-id method; updates step docstring.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/tile/RenderTilesClient.java	Removes default-step-id method; updates step docstring.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/tile/MaskHackClient.java	Removes default-step-id method; updates step docstring.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/ScapeClient.java	Removes default-step-id method; minor Java 21 list usage.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/pipeline/AlignmentPipelineStepId.java	Adds `IMPORT_SOFIMA` step id mapped to the new Spark client.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/pipeline/AlignmentPipelineStep.java	Removes `getDefaultStepId()` from the step interface.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/pipeline/AlignmentPipelineParameters.java	Adds `SofimaParameters` to pipeline parameters and accessor.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/newsolver/DistributedIntensityCorrectionBlockSolverClient.java	Removes default-step-id method; updates step docstring.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/newsolver/DistributedAffineBlockSolverClient.java	Removes default-step-id method; updates step docstring.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/multisem/UnconnectedCrossMFOVClient.java	Removes default-step-id method; stream `.toList()` modernization.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/multisem/MultiSEMTileRemovalClient.java	Removes default-step-id method; updates step docstring.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/multisem/MFOVMontageMatchPatchClient.java	Removes default-step-id method; updates step docstring.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/multisem/MFOVAsTileClient.java	Removes default-step-id method; updates step docstring.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/multisem/MatchCollectionRenameClient.java	Removes default-step-id method; stream `.toList()` modernization.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/multisem/LayerAsTileClient.java	Removes default-step-id method; small string check and `.toList()` modernization.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/multisem/ImportSofimaClient.java	New Spark client implementing the `IMPORT_SOFIMA` pipeline step.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/multisem/CreepCorrectionSparkClient.java	Removes default-step-id method; updates step docstring.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/multisem/BeamCorrectionSparkClient.java	Removes default-step-id method; updates step docstring.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/MipmapClient.java	Removes default-step-id method; stream `.toList()` modernization.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/match/MultiStagePointMatchClient.java	Removes default-step-id method; updates step docstring.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/match/CopyMatchClient.java	Removes default-step-id method; updates step docstring.
render-ws-spark-client/src/main/java/org/janelia/render/client/spark/match/ClusterCountClient.java	Removes default-step-id method; updates step docstring.
render-ws-java-client/src/main/java/org/janelia/render/client/parameter/SofimaParameters.java	New shared parameter bean for SOFIMA import configuration.
render-ws-java-client/src/main/java/org/janelia/render/client/multisem/ImportSofimaClient.java	Refactors Java client to use `SofimaParameters` and derived target stacks.

Review details

Files reviewed: 28/28 changed files
Comments generated: 3
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    @Override
+    public void validatePipelineParameters(final AlignmentPipelineParameters pipelineParameters)
+            throws IllegalArgumentException {
+
+        final SofimaParameters sofima = pipelineParameters.getSofima();
+
+        AlignmentPipelineParameters.validateRequiredElementExists("sofima", sofima);
+
+        sofima.validate();
+    }


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

trautmane · 2026-09-14T00:00:07Z

@minnerbe - I went ahead and merged this, but please let me know if you think anything should be changed. Thanks!

trautmane · 2026-09-14T01:13:58Z

I needed to add this to get things working:
c89b171

trautmane · 2026-09-14T01:17:04Z

Also - the log from my first run contains many applyInPlace inversion warnings like these:

applyInPlace-inversion-convergence-warning

Please let me know if this is expected or if it identifies a problem.
Thanks!

minnerbe · 2026-09-14T14:09:27Z

Thanks, @trautmane ! The fix looks good to me. I'm a bit concerned about the warnings, especially the one that has a 33px residual. I think the screenshot has all information I need to investigate this.

minnerbe · 2026-09-15T14:58:16Z

I investigated the issue and it turned out that the convergence properties of the inversion of the displacement field weren't great. I implemented a Newton method, which should converge much more robustly (getting rid of the warnings above, which indicated that the inversion was wrong). This has been pushed directly to newsolver.

trautmane added 5 commits September 12, 2026 16:21

first draft of spark ImportSofimaClient implementation that uses core…

b3cded7

… logic in the java ImportSofimaClient

remove unused getDefaultStepId method from the AlignmentPipelineStep …

de5158c

…interface and add javadoc comment with reference information to each pipeline step implementation to document the mapping instead

use SofimaParameters in java client and remove option to process a su…

6904143

…b-range of z layers from that client

add new stages to db-dump-google-collections.sh

0bf7b7a

bump render-ws-with-mongodb version to 1.0.3

a4e24b5

trautmane requested review from minnerbe and a lite review from Copilot September 13, 2026 23:44

Copilot started reviewing on behalf of trautmane September 13, 2026 23:44 View session

Copilot AI reviewed Sep 13, 2026

View reviewed changes

trautmane and others added 2 commits September 13, 2026 19:53

Change logging to use count of imported stacks

ca211ec

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Update stack parameter description in ImportSofimaClient

f8359d1

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

trautmane merged commit 2905e9b into newsolver Sep 13, 2026
2 checks passed

trautmane deleted the import_sofima_20260912 branch September 13, 2026 23:59

Navigation Menu

import_sofima_20260912 - #253

import_sofima_20260912#253
trautmane merged 7 commits into
newsolverfrom
import_sofima_20260912

trautmane commented Sep 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

trautmane commented Sep 14, 2026 •

edited

Loading

Uh oh!

trautmane commented Sep 14, 2026

Uh oh!

trautmane commented Sep 14, 2026

Uh oh!

minnerbe commented Sep 14, 2026

Uh oh!

minnerbe commented Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

trautmane commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

trautmane commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trautmane commented Sep 14, 2026

Uh oh!

trautmane commented Sep 14, 2026

Uh oh!

minnerbe commented Sep 14, 2026

Uh oh!

minnerbe commented Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

trautmane commented Sep 13, 2026 •

edited

Loading

trautmane commented Sep 14, 2026 •

edited

Loading