Skip to content

NGFF Multiscales and Scene round-tripping - #1286

Draft
Tomaz-Vieira wants to merge 43 commits into
scverse:mainfrom
Tomaz-Vieira:ngff_io
Draft

NGFF Multiscales and Scene round-tripping#1286
Tomaz-Vieira wants to merge 43 commits into
scverse:mainfrom
Tomaz-Vieira:ngff_io

Conversation

@Tomaz-Vieira

@Tomaz-Vieira Tomaz-Vieira commented Oct 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Definitely not ready for review.

This PR makes a whole lot of assumptions about how the SpatialData and its internal graph is going to behave in the future. Some of those assumptions will be met by @ajkswamy next PR, some will be ripped out of this PR into smaller ones that can adjust SpatialData and the TransformationManager to fit its expectations.

I'm mostly pushing this here in case someone wants to take a look and in case I nuke my files accidentally again.

Main ideas ares:

Spaces will be replaced by TransformationManager

But:

  • I think the idea of having vertices NOT be coordinate systems, but rather a container-class that has the CS plus metadata has been very useful;
    • Importantly, this makes the CS very lightweight for use outside of the graph.
    • Also, serializing a CS needs "global" awareness anyway (e.g. to determine paths in CooridnateSystemIdentifier), so I think it is fine to keep the CS lean and move metadata onto the "global" scope that is the graph.
  • I also like having verts and edges typed, and having a class for Edge so it can have methods ;
  • The iterators over vertices (Verts) and edges (Edges ) are very convenient and ergonomic, so I'm hoping we can adopt those too.
  • The transformation graphs have to be "mergeable" for this implementation. This characteristic is useful because I don't have to make other classes to hold verts and edges as I am parsing, and I also don't clobber a good graph with a bad parse if it the parse fails midway through; I'll parse stuff into a brand new graph and once that is done merge the new graph with the current one in a single transaction.

FutureSpatialData will be replaced by SpatialData

SpatialData had some requirements on what needs to be in .attrs that we are going to abandon soon, so i didn't feel like I should persue compatibility with it.

ome-zarr-py needs some work

We need it to accept our own pyramids; I'll try to get the relevant PR pushed there.

I would also like for its typing to be a lot tighter than it is right now so we don't have to assert as much, but we'll see.

ajkswamy and others added 30 commits September 15, 2026 19:20
@ajkswamy
Transformation manager foundation (scverse#1164)
75e5a44 
* chore: added pandas-stubs as dev dependency; helps with type checking

* chore: added local hatch.toml to gitignore

* chore: Add .kilo and plans to .gitignore

* feat: added literals and types for element types and group names

* feat: added .coverage and htmlcov to gitignore

* feat: transformation manager at the root of spatialdata object

* fix: io_zarr.py mixup between ELEMENT_TYPE_VECTOR/RASTER

* fix: avoid exposing TransformationManager in spatialdata __init__.py

* fix: element associated to cs -> element belonging to cs

* refac: transformation_manager now uses nx.MultiDiGraph

* refac: tests to match new transformation manager implementation

* fix: restored unintentionally removed functions + others fixes

* feat: custom error messages for transformation graph elements

* feat: custom exceptions for transform manager

* refac: transform manager attributes private; added missing method

* feat: more custom error and warnings + fixes

* feat: more test coverage + fixes

* feat: support for transformation management in graph with mutliple edges

* fix: fixes in tests + renaming for readability

* fix: typing Affine Transform

* feat: TransformationManager return Sequence instead of list

* feat: made TransformationManager.graph type specific over node type

* fix: removed internal attribute access warnings

* refac: rename check_if... methods to assert_... + one additional method

* fix: node type spec for TransformationManager._graph

* fix: made edge key definition from transforms more robust

* refac: method rename

* refac: TransformationManager get/remove methods don't raise error if edges are missing

* feat: new test for TransformationManager.add_transformation

* fix: better error messages and edge case handling for when transformation path is ambiguous

* feat: TransformationManager edge (key) definition made stronger

* fix: simplified access to attributes of TransformationManager

* fix: TransformationManager.unset_element is now private

* fix: throw error a path has one node

* fix: simplified checking if coordinate system has associated transforms

* fix: improved Transformation Manager code quality with better typing

* fix: improved Transformation Manager exception naming and messaging

* fix: improved Transformation Manager documentation

* feat: added tests for Transformation Manager

* fix: used explicit syntax for type definitoin

* refac: typo

* fix: removed TransformationManager import from spatialdata __init__.py

* fix: TransformationManager, simplified code using custom error
@ajkswamy @Tomaz-Vieira
Transformation manager foundation (scverse#1164)
519a3e2 
* chore: added pandas-stubs as dev dependency; helps with type checking

* chore: added local hatch.toml to gitignore

* chore: Add .kilo and plans to .gitignore

* feat: added literals and types for element types and group names

* feat: added .coverage and htmlcov to gitignore

* feat: transformation manager at the root of spatialdata object

* fix: io_zarr.py mixup between ELEMENT_TYPE_VECTOR/RASTER

* fix: avoid exposing TransformationManager in spatialdata __init__.py

* fix: element associated to cs -> element belonging to cs

* refac: transformation_manager now uses nx.MultiDiGraph

* refac: tests to match new transformation manager implementation

* fix: restored unintentionally removed functions + others fixes

* feat: custom error messages for transformation graph elements

* feat: custom exceptions for transform manager

* refac: transform manager attributes private; added missing method

* feat: more custom error and warnings + fixes

* feat: more test coverage + fixes

* feat: support for transformation management in graph with mutliple edges

* fix: fixes in tests + renaming for readability

* fix: typing Affine Transform

* feat: TransformationManager return Sequence instead of list

* feat: made TransformationManager.graph type specific over node type

* fix: removed internal attribute access warnings

* refac: rename check_if... methods to assert_... + one additional method

* fix: node type spec for TransformationManager._graph

* fix: made edge key definition from transforms more robust

* refac: method rename

* refac: TransformationManager get/remove methods don't raise error if edges are missing

* feat: new test for TransformationManager.add_transformation

* fix: better error messages and edge case handling for when transformation path is ambiguous

* feat: TransformationManager edge (key) definition made stronger

* fix: simplified access to attributes of TransformationManager

* fix: TransformationManager.unset_element is now private

* fix: throw error a path has one node

* fix: simplified checking if coordinate system has associated transforms

* fix: improved Transformation Manager code quality with better typing

* fix: improved Transformation Manager exception naming and messaging

* fix: improved Transformation Manager documentation

* feat: added tests for Transformation Manager

* fix: used explicit syntax for type definitoin

* refac: typo

* fix: removed TransformationManager import from spatialdata __init__.py

* fix: TransformationManager, simplified code using custom error
@Tomaz-Vieira
Creates the graph module based on the Ngff* classes
3a47dd4 
Cleans up and modernizes the NGff classes so that they have stronger
invariants and guarantee that they always have input and output.

Reading and writing to zarr is done via ome-zarr-models-py.

Uses graph module as part of io_raster.py::try_read_ngff06_multiscale
to interpret Ngff trnasformations and produce an output that could be
added to the new graph implementation.
@Tomaz-Vieira
addresses some PR comments, simplifies MapAxis
e4ff532
@Tomaz-Vieira
Adds ProjectAxisEdge, makes inverse return optional
796d557
@Tomaz-Vieira
Moves exceptions to exceptions.py, edge names default to None
406375d
@Tomaz-Vieira
Use custom exceptions everywhere, adds Raises to docstrings
db250b4
@Tomaz-Vieira
Fix Affine.from_affine_matrix slicing]
28e13c9
@Tomaz-Vieira
Fix coords bug when reading from OMEZarrMultiscale. Some cleanup
d7da440
@Tomaz-Vieira
Adds test for parsing OMEZarrMultiscale
7384dd8
@Tomaz-Vieira @claude
Adds transformations tests, fixes some slicing bugs
9bf5b28 
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
@Tomaz-Vieira
Adds basic .sel test to parsed multiscale
14d5b71
@Tomaz-Vieira
Removes unused class
7b346a1
@Tomaz-Vieira
Adds more slicing tests to parsed multiscales
e3fbe7e
@Tomaz-Vieira
Adds/fixes comments in try_parse_ngff06_multiscales
ae02611
@Tomaz-Vieira
Removes to_model methods for now
7ca1476
@Tomaz-Vieira
Moves AxisParsingException to exceptions.py
13b2a3d
@Tomaz-Vieira
Moves AxisParsingException back into vert.py to prevent circular imports
ebb6f8a
@Tomaz-Vieira @claude
Fixes docstrings, list exceptions in "Raises"
bf2178f 
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
@Tomaz-Vieira
Incorporates parse_project_axis
fd74830
@Tomaz-Vieira @claude
Fixes parsing translatiion model getting bad name
1177ee4 
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
@Tomaz-Vieira @claude
Adds test for parsing *Edge transforms from ngff
4fe7f8a 
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
@Tomaz-Vieira
Fixes dependencies on ome-zarr and ome-zarr-models
dcc436b
@Tomaz-Vieira
Cleans up transform_points for rotation and affine
cec5085
@Tomaz-Vieira
Fixes renamed fields in ome-zarr-models
8327b55
@Tomaz-Vieira
Use frozen dataclasses for CoordSystems and Axis. Fix MapAxisEdge
25cfc73
@Tomaz-Vieira
Enforces Rotation matrix is orthonormal with det==1
45fb0ff
@Tomaz-Vieira
Bump xarray so we can use RangeIndex
09c8984
@Tomaz-Vieira
Removes try-from_model_or_defaul for now
9de2b15
@Tomaz-Vieira
Emits warning on AffineEdge.inverse when condition is big
155c78a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Tomaz-Vieira @ajkswamy