AnnData obs/var string to categoricals

[ajkswamy]

Issue:

#1152

Summary:

• Writing AnnData table to zarr file now uses different methods depending on zarr format being written and anndata version installed
• A new custom Deprecation warning is thrown when writing any element to zarr format v2.
• A new hatch env config to help run all tests for combination of anndata/pandas versions

How to test:

• Use this notebook, modified from the one provided by LazDaria in Missing values in AnnData .uns DataFrame become string "nan" after SpatialData Zarr round trip #1152 to cover all cases there
• Use the hatch env added here and call hatch run test-anndata-pandas: test {args}. Check config before running, every script is run multiple times based on the matrix defined.

Related PRs

• Support sharding through config and raster_write_kwargs #1106 (comment)

[codecov]

Codecov Report

❌ Patch coverage is 86.66667% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.40%. Comparing base (4e19681) to head (d422160).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/spatialdata/_io/exceptions.py	60.00%	4 Missing ⚠️
src/spatialdata/_io/io_table.py	90.47%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1170      +/-   ##
==========================================
- Coverage   92.44%   92.40%   -0.04%     
==========================================
  Files          51       52       +1     
  Lines        7820     7858      +38     
==========================================
+ Hits         7229     7261      +32     
- Misses        591      597       +6     

Files with missing lines	Coverage Δ
src/spatialdata/_core/spatialdata.py	93.86% <ø> (ø)
src/spatialdata/_io/io_points.py	100.00% <100.00%> (ø)
src/spatialdata/_io/io_raster.py	89.35% <100.00%> (+0.30%)	⬆️
src/spatialdata/_io/io_shapes.py	96.34% <100.00%> (+0.18%)	⬆️
src/spatialdata/_io/io_table.py	89.28% <90.47%> (-1.20%)	⬇️
src/spatialdata/_io/exceptions.py	60.00% <60.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[LucaMarconato]

Here there is a problem: table.strings_to_categoricals() changes the dtype of the string column to a categorical. An in-place operation from a write operation is an unexpected side effect.

We need to find a different solution. I suggest to add an extra assert to test_sdata_with_nan_in_obs() to verify that the dtype of the obs columns of the input object does not change after write, and to run this against all the combinations of pandas and anndata via the hatch config you created.

[ajkswamy]

Yes, right, I added the extra assert statements to test_sdata_with_nan_in_obs() and indeed there is a side effect caused by strings_to_categoricals().

anndata.AnnData.write_zarr has an argument convert_strings_to_categorical which is set to True by default. Hence the default round trip for nullable stings columns works by default with anndata, but with this side effect. Here, I have added an argument to io_table.write_table along with all functions in the call chain from Spatialdata.write that allows users the same options to convert strings in AnnData objects to categorical before writing. The main difference that we set the default value to be false, i.e., we do not convert by default and users need to explicitly set this to be able to do round trip with nullable string/object columns with pandas 2.0.

[LucaMarconato]

Thanks for explaining! So if I understood correctly the side effect happens already with anndata? If so could you please report to the anndata team (GitHub issue) with a minimal reproducible example (only using anndata, not spatialdata)? The bug to show is that a write operation changes the input in-memory object silently.

[LucaMarconato]

One last thing, please rename the PR to a more descriptive text since upon release the automatically-generated release notes will use the PR title.