Skip to content

feat: support complex-valued dask-backed arrays in SVD decomposition - #252

Open
philipdkennedy wants to merge 3 commits into
xarray-contrib:mainfrom
philipdkennedy:feat/remove-dask-complex-limits
Open

philipdkennedy wants to merge 3 commits into
xarray-contrib:mainfrom
philipdkennedy:feat/remove-dask-complex-limits

Conversation

@philipdkennedy

@philipdkennedy philipdkennedy commented Oct 14, 2026 •

Copy link
Copy Markdown

Description

Complex-valued, dask-backed arrays can now be decomposed. Previously, the dispatch
matrix in the Decomposer had no branch for the complex + dask combination, so
Hilbert EOF/MCA on chunked data raised — the restriction existed because the code
path was never implemented, not because the numerics are unsound.

  • xeofs/linalg/decomposer.py and xeofs/linalg/_numpy/_svd.py: the dask branch
    now catches complex-valued input alongside real-valued input, for both the
    full (tsqr) and randomized (svd_compressed) solvers.
  • Model-level guards (assert_not_complex in GWPCA, SparsePCA, OPA, MultiCCA)
    are intentionally retained — those algorithms do not support complex input
    and should continue to reject it loudly.
  • Note: solver="full" on dask input requires tall-and-skinny chunking
    (single-column-of-blocks), as before; documented in the test fixtures.

Behavioural change

HilbertEOF.test_compute expectations inverted: fitting on dask input
previously raised NotImplementedError; it now succeeds and returns
complex-valued components and scores. This is the intended capability.

Tests

  • Fourth-quadrant test parity in tests/linalg/test_decomposer.py:
    test_fit_dask_full_complex, test_fit_dask_randomized_complex,
    test_dask_array_based_on_target_variance_complex, and a dask_complex
    case in the test_random_state parametrization.
  • Red-green verified: with the dispatcher files reverted to main, the
    complex-dask tests fail; with this branch, 29/29 pass in tests/linalg/
    and the full suite is green apart from two pre-existing test_mca.py
    failures that also occur on main.
  • New fixtures mock_complex_data_array / mock_dask_complex_data_array
    in tests/linalg/test_decomposer.py, mirroring the real-data fixtures.

Dependency note

Complex-valued dask input requires dask >= 2026.6.0: earlier versions
contained conjugation bugs in svd_compressed/tsqr for complex data
(dask/dask#7639, dask/dask#9421), producing silently incorrect
decompositions. Verified against 2026.6.0, where the numerics are correct
(round-trip tested in tests/linalg/).

Known issue

Complex-dask fits emit a cosmetic ComplexWarning from dask's meta
construction (hardcoded float64 meta in tsqr, upstream
dask/dask#10284). It is numerically inert since dask's complex conjugation
fixes. Suppressed locally in the affected tests with a comment, rather than
project-wide, so a future genuine cast is not hidden.

Agreed in advance with @nicrie to fix #197.

@philipdkennedy philipdkennedy changed the title feat: support complex-valued dask-backed arrays in SVD decomposition feat: support complex-valued dask-backed arrays in SVD decomposition to fix #197 Oct 14, 2026
@philipdkennedy philipdkennedy changed the title feat: support complex-valued dask-backed arrays in SVD decomposition to fix #197 feat: support complex-valued dask-backed arrays in SVD decomposition Oct 14, 2026
@nicrie

nicrie commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

@philipdkennedy thanks for the PR! I'll try to review it within the next day – please bear with me a little.

@nicrie

nicrie commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Sorry @philipdkennedy that I keep you waiting for so long. The CI requires some updates that I'm working on, then we can bring in your changes cleanly. I'll keep you updated.

@nicrie

nicrie commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Hi again @philipdkennedy – once #257 is merged, could you update your branch by pulling the latest changes. The CI should restart automatically and many (hopefully all?) failing tests will be gone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Missing dask support for models based on complex data

2 participants