Skip to content

Testbed tool, conformance report table and reference datasets - #91

Draft
xulman wants to merge 17 commits into
mainfrom
testbed_datasets
Draft

Testbed tool, conformance report table and reference datasets#91
xulman wants to merge 17 commits into
mainfrom
testbed_datasets

Conversation

@xulman

@xulman xulman commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

This PR is currently a work-in-progress.

Its purpose is to have two entry test methods

  • A pure unit-test method that opens a given dataset using a given backend, shows an IJ window (to simulate a real user wanting to display an OME-Zarr dataset), and checks various parameters with asserts. This method is called iteratively over a collection of test data produced by testCasesFeeder().
  • Another method that iteratively calls the previous one, monitors its asserts, and builds up a conformance tests overview table, see below.

The first method is clearly the workhorse here. Additionally, the class TestDataset is crucial to the method's testing capability, as it contains a pointer to the source dataset and numerous dataset parameters, such as axis sizes, the number of resolution levels, pixel resolutions, axis order, etc. The workhorse method only verifies matches between the actual values (from the PyramidDataset object created at runtime) and the reference values (from the TestDataset object) over the said parameters.

The folder /testing currently holds aux tools to create a .csv table, see also #88, that feeds and arrives into TestDataset objects.

The second method reportSuccessRateAndPrintTable() can give an output like this:

+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
| Dataset                                                                        | Backend   | Result | Reason                                                                         |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                            file://...../2d_dataset_v5.ome.zarr (idr0106) v0.5: |        N5 |   PASS |                                                                                |
|                         size=(64 x 64 x 1 x 1 x 1) [XYZCT], axes='XY', pxType= |           |        |                                                                                |
|     multiscales=-1, wells=-1, fields=-1, resolution=(-1.0 x -1.0 x -1.0) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                            file://...../2d_dataset_v5.ome.zarr (idr0106) v0.5: | ZARR_JAVA |   PASS |                                                                                |
|                         size=(64 x 64 x 1 x 1 x 1) [XYZCT], axes='XY', pxType= |           |        |                                                                                |
|     multiscales=-1, wells=-1, fields=-1, resolution=(-1.0 x -1.0 x -1.0) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                 https://ome-z.....s.com/...../miranda.ome.zarr (miranda) v0.5: |        N5 |   PASS |                                                                                |
|        size=(1024 x 1024 x 1024 x -1 x -1) [XYZCT], axes='XYZ', pxType=float32 |           |        |                                                                                |
|         multiscales=4, wells=-1, fields=-1, resolution=(1.0 x 1.0 x 1.0) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                 https://ome-z.....s.com/...../miranda.ome.zarr (miranda) v0.5: | ZARR_JAVA |   FAIL | AssertionFailedError: Failed opening dataset 'miranda' with IJ. ==> Unexpected |
|        size=(1024 x 1024 x 1024 x -1 x -1) [XYZCT], axes='XYZ', pxType=float32 |           |        |                                  exception thrown: java.lang.RuntimeException: |
|         multiscales=4, wells=-1, fields=-1, resolution=(1.0 x 1.0 x 1.0) [XYZ] |           |        |                                       java.util.concurrent.ExecutionException: |
|                                                                                |           |        |                                       java.lang.ArrayIndexOutOfBoundsException |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|     https://ome-z.....s.com/...../marmoset_neurons.ome.zarr (marmoset_neurons) |        N5 |   PASS |                                                                                |
|                                                                          v0.5: |           |        |                                                                                |
|           size=(1024 x 1024 x 314 x -1 x -1) [XYZCT], axes='XYZ', pxType=uint8 |           |        |                                                                                |
|     multiscales=4, wells=-1, fields=-1, resolution=(0.497 x 0.497 x 1.5) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|     https://ome-z.....s.com/...../marmoset_neurons.ome.zarr (marmoset_neurons) | ZARR_JAVA |   PASS |                                                                                |
|                                                                          v0.5: |           |        |                                                                                |
|           size=(1024 x 1024 x 314 x -1 x -1) [XYZCT], axes='XYZ', pxType=uint8 |           |        |                                                                                |
|     multiscales=4, wells=-1, fields=-1, resolution=(0.497 x 0.497 x 1.5) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                     https://livin.....ac.uk/...../9798462.zarr (idr0073) v0.4: |        N5 |   PASS |                                                                                |
|                size=(21115 x 16433 x 1 x 3 x 1) [XYZCT], axes='XYZCT', pxType= |           |        |                                                                                |
|     multiscales=-1, wells=-1, fields=-1, resolution=(-1.0 x -1.0 x -1.0) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                     https://livin.....ac.uk/...../9798462.zarr (idr0073) v0.4: | ZARR_JAVA |   FAIL | AssertionFailedError: Failed opening dataset 'idr0073' with IJ. ==> Unexpected |
|                size=(21115 x 16433 x 1 x 3 x 1) [XYZCT], axes='XYZCT', pxType= |           |        |                                                              exception thrown: |
|     multiscales=-1, wells=-1, fields=-1, resolution=(-1.0 x -1.0 x -1.0) [XYZ] |           |        |   sc.fiji.ome.zarr.pyramid.exceptions.PyramidLevelAccessException: Cannot open |
|                                                                                |           |        |                                                         resolution level 0 of: |
|                                                                                |           |        |            https://livingobjects.ebi.ac.uk/idr/zarr/v0.4/idr0073A/9798462.zarr |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                        https://livin.....ac.uk/...../9512.zarr (idr0072) v0.4: |        N5 |   FAIL |      AssertionFailedError: After opening, no Dataset is available in Fiji. ==> |
|                    size=(1345 x 1017 x 1 x 2 x 1) [XYZCT], axes='XYC', pxType= |           |        |                                                     expected: <1> but was: <0> |
|     multiscales=-1, wells=72, fields=20, resolution=(-1.0 x -1.0 x -1.0) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                        https://livin.....ac.uk/...../9512.zarr (idr0072) v0.4: | ZARR_JAVA |   FAIL |      AssertionFailedError: After opening, no Dataset is available in Fiji. ==> |
|                    size=(1345 x 1017 x 1 x 2 x 1) [XYZCT], axes='XYC', pxType= |           |        |                                                     expected: <1> but was: <0> |
|     multiscales=-1, wells=72, fields=20, resolution=(-1.0 x -1.0 x -1.0) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+

Passed: 6 / 10

TODO

  • Fully populate the source .csv table, as it is currently missing values for some datasets. These are the '-1' or '' values.
  • Consider creating and including dummy OME-Zarr datasets that would be created "around" .json files from the official ngff spec repo.
  • Consider using our internal dataset (in the resources folder).
  • Move the .csv to the resources section of the package.

xulman added 17 commits June 5, 2026 09:25
@xulman
ADD: .gitignore
3fd7b07
@xulman
ADD: source data (listings), SciVis is already manually preprocessed
e6a5ce8
@xulman
ADD: convert_to_one_csv.py all-massaging script
2ce7cab
@xulman
CHG: convert_to_one_csv.py finalized the list of columns in the melte…
c8dcb6b 
…d final output table
@xulman
ADD: testbed_datasets.csv - 152 datasets with (nearly all) known values
f86b515
@xulman
ADD: ConformanceTest first complete implementation - should be functi…
dad814d 
…onal
@xulman
ADD,FIX: ConformanceTest reports on which test dataset it is currentl…
824e041 
…y operating
@xulman
FIX1/2: request to open the lowest res IJ window, because...
5e3a2d0 
...the display of it will trigger pixel loading, so reduce the
load to the least possible. But than the opened underlying Dataset
is typicaly not the one at the highest/best/base resolution...

However!  maxPreferred must a decent number because the openers
are complaining if no res. level _smaller_ than this number
is available.
@xulman
FIX2/2: use the finest resolution RAI for the testing
f897e23 
RAI = RandomAccessibleInterval, a representation of an image in Imglib2

the way we obtain/construct this RAI avoids loading of pixel data
@xulman
ADD: checkPixelTypes()
d72a28a
@xulman
ADD: C,T,axes,scales, numResLevels tests into the conformance suite
db9c23f
@xulman
FIX,COS: were switched the expected vs. actual params in asserts
90811c6
@xulman
FIX: tiny nicer report, introduced accuracy delta for checking the re…
f546755 
…solution of pixels
@xulman
ADD: reports table plus a wrapper that...
24cf96d 
...calls the original testOpenAndCheckDatasetParams(), which is
a standard unit test method, and monitors its exceptions to figure
out if the test passed or failed. This is logged into an ASCII table,
and X/Y passed is printed as the line of the report.

Thanks Claude.AI for the initial help.
@xulman
ADD: TestDataset can do shorter self-reports/self-description
9fb57b7
@xulman
CHG: asserts in testOpenAndCheckDatasetParams() do message reasons
b6d3d0a
@xulman
(BUG)FIX in axes order sub-test (toString() was giving full names for…
8e55ccb 
… Column, Time axes)
@xulman
xulman marked this pull request as draft June 9, 2026 14:14
@sonarqubecloud

sonarqubecloud Bot commented Jun 9, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@stefanhahmann stefanhahmann mentioned this pull request Aug 5, 2026
Closed
xulman pushed a commit that referenced this pull request Sep 2, 2026
@stefanhahmann @claude @xulman
Adds conformance testbed tool + reference datasets
a144d64 
Reconstructs PR #91 ("Testbed tool, conformance report table and reference
datasets") on top of the five-module split (PR #95). The original commits
predate that refactoring and no longer apply, so this is a fresh, equivalent
port rather than a git-rebase.

Changes vs. the original PR:
- ConformanceTest moved to the ome-zarr-fiji-ui test module and repackaged
  sc.fiji.ome.zarr.open -> ome.zarr.fijiui.open.
- Imports updated for the renamed packages
  (ome.zarr.fijiui.open.options.*, ome.zarr.fiji.PyramidalDataset).
- Adapted to the new data model: the removed rich PyramidalDataset API
  (numChannels/numTimepoints/numResolutions/voxelDimensions/asSources) is
  now reached via PyramidContents (getPyramidContents()):
    * base-resolution sizes read from contents.asImg(0) with axis lookup by
      name (axisIndex/hasAxis) instead of a fixed 0/1/2 BDV source order;
    * per-axis pixel scale read from contents.axesPerLevel[0];
    * pixel type read from contents.type.
- ZarrOpenActions is constructed with the 3-arg (uri, context, settings) constructor;
  the old 4-arg errorHandler variant is gone and the captured error was unused.
- ascii-table test dependency moved from the root pom to the ui module pom.
- Reference data moved from the repo-root testing/ folder to the shared test
  resources at test-shared/resources/ome/zarr/testdata/conformance_testing/
  (alongside the other testdata folders), and the .csv is now loaded from the
  test classpath via ZarrTestUtils.resourcePath(...) instead of a hardcoded
  absolute path.

Both tests remain @disabled (run manually).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@xulman xulman

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@xulman