Refactor dependencies to make more optional

[Sevans711]

Closes #1224

Waiting for #1541 before asking to merge (in order to move hvplot from required to "viz").

Overview

Addresses user feedback asking to handle dependencies in a more elegant manner. Many previously-required dependencies are now optional. Also may speed up "import uxarray" slightly, as now-optional dependencies are moved to no longer be at top-level / top of files; instead, they are imported the first time any relevant method gets called. Imports of some required dependencies were similarly moved, to possibly improve "import uxarray" timing.

Notes about all dependencies which were previously listed as required in pyproject.toml:

• antimeridian: important for geometry → keep as required. [EFF] Efficiency note: antimeridian was already never being imported at top-level.
• cartopy: used only in plotting → viz
• dask[dataframe]: keep as required for now. But, it is only used in core.aggrecation and core.zonal, so maybe could make optional? [EFF] already never at top-level.
• datashader: not used explicitly anywhere → removed
• geoviews: used only in examples, for plotting → viz
• holoviews: used only in examples, tests, and HoloviewsBackend for plotting → viz
• matplotlib: used only for plotting → viz
• matplotlib-inline: not used explicitly anywhere → removed
• netcdf4: not used explicitly, but everyone is reading .nc files → keep as required
• numba: keep as required, of course
• numpy: keep as required, of course
• pandas: important for grid methods, e.g. pd.Series and pd.IntervalIndex used explicitly → keep as required
• pyarrow: not used explicitly, and dask[dataframe] already depends on it → removed
• requests: not used explicitly anywhere in uxarray → removed
• scikit-learn: important for neighbor algorithms, e.g. KDTree → keep as required. [EFF] already not at top-level.
• scipy: important for delaunay triangulations and some io → keep as required. [EFF] removed from top-level.
• shapely: important for geometry → keep as required. [EFF] already not at top-level.
• spatialpandas: just for spatialpandas.GeoDataFrame conversions; should be optional → geo
• geopandas: just for geopandas.GeoDataFrame conversions; should be optional → geo
• xarray: keep as required, of course
• hvplot: used only for plotting → viz. Didn't move it yet though; waiting for Speed up import and load hvplot lazily #1541 to get merged
• healpix: just for healpix grids; should be optional → geo
• polars: used explicitly for some internal methods → keep as required
• pyproj: only used in uxarray.cross_sections.sample.sample_geodesic; should be optional → geo

Miscellaneous notes:

• removed pathlib; pathlib became a built-in part of the standard library in Python 3.4, and the pyproject.toml requires Python >= 3.10.
• added "all" as alias for "complete" because many libraries use "all"; see example usage below.
• added pooch to "dev" dependency group, because pooch was secretly required in order for all tests to pass. Comment in test/io/test_structure.py clarifies why pooch dependency exists despite not being imported explicitly anywhere in uxarray.

Expected Usage

Users can now install uxarray with optional dependency groups. Simple examples with pip interface:
Required dependencies only:

pip install uxarray

Also includes all optional dependencies related to geospatial grids (geopandas, healpix, pyproj, spatialpandas)

pip install "uxarray[geo]"

Also includes everything related to geospatial grids, plus everything related to visualization (cartopy, geoviews, holoviews, matplotlib)

pip install "uxarray[geo,viz]"

Includes all required plus all optional dependencies:

pip install "uxarray[complete]"

Equivalent to above: pip install "uxarray[all]"

PR Checklist

General

• An issue is linked created and linked
• Add appropriate labels
• Filled out Overview and Expected Usage (if applicable) sections

Testing

• Adequate tests are created if there is new functionality
• Tests cover all possible logical paths in your function
• Tests are not too basic (such as simply calling a function and nothing else)
• Question: should we add something to test suite to ensure you can at least import uxarray after doing pip install uxarray with minimal dependencies? We could maybe add a few more tests for different combinations of dependencies too?

[rajeeja]

One thing to flag: anyone doing pip install uxarray and then using plotting will now hit an ImportError until they install [viz]. This is intended per #1224, but it's a meaningful change to the default install contract, so I think we need a broader discussion before merging. @erogluorhan — can you weigh in on how we want to handle the default UX and messaging here?

[Sevans711]

One thing to flag: anyone doing pip install uxarray and then using plotting will now hit an ImportError until they install [viz]. This is intended per #1224, but it's a meaningful change to the default install contract, so I think we need a broader discussion before merging. @erogluorhan — can you weigh in on how we want to handle the default UX and messaging here?

I definitely agree, we should make sure to discuss more before merging. I am curious to know what people think.

One thought I had is: something that might help could be to provide hints during those crashes. Something like:

def function_that_uses_holoviews(*args, **kwargs):
    try:
        import holoviews as hv
    except ImportError as err:
        raise ImportError('holoviews import failed; to install all plotting dependencies, could do: pip install "uxarray[viz]".') from err
    # << code from function

As opposed to the code presently in the PR, which just does import holoviews as hv.

Although, this doesn't match xarray's style. In xarray, if you fail to install optional dependencies you will just see an ImportError crash once you try to use them, there's no hint to pip install xarray[viz] or anything like that.

Another slight downside is that this option might get a bit unwieldy due to the need to repeat it inside every relevant method. A partial workaround could be to handle these hints all in one place:

# --- inside new file: utils.imports.py --- #
def _import_with_hint_if_error(*modules):
    """import modules, but make nice error if crash, providing hint to pip install "uxarray[...]".
    E.g., if 'holoviews' in modules, and import holoviews fails, hint to use: pip install "uxarray[viz]".
    """
    if 'holoviews' in modules:
        try:
            import holoviews as hv
        except ImportError as err:
            raise ImportError('holoviews import failed; to install all plotting dependencies, could do: pip install "uxarray[viz]".') from err
    if 'geoviews' in modules:
        # similar
    # similar for all other optional dependencies.

Then functions with optional dependencies can look like:

from uxarray.utils.imports import _import_with_hint_if_error

def function_that_uses_holoviews_and_geoviews(*args, **kwargs):
    _import_with_hint_if_error('holoviews', 'geoviews')
    import holoviews as hv
    import geoviews as gv

I actually have a slight personal preference to do something like that; I think user experience is improved when common errors include debugging hints. But, I'm happy to go with whatever we decide together.

[Sevans711]

Quick note: #1563 ran into issue with pathlib still being in our dependencies and would also remove it, as mentioned in this comment. This PR could have fixed that issue before there was a need to spend efforts debugging it in that branch.

[erogluorhan]

Please resolve the merge conflicts so that I can see the test results before a larger review, but here are my initial thoughts about this work:

• Really appreciate the detailed description of this PR; it is extremely helpful!
• Per @rajeeja 's comment, I agree this is a major change and definitely requires being very careful; a few thoughts:
  • conda does not have built-in optional dependencies like pip. Mechanisms such as lazy imports and fail locally rather than fail on installation will be needed in place to make sure there is no divergence-as much as possible- between conda and pip
  • Per @Sevans711 's suggestion, clear guidance messages about the expected optional installs on both conda and pip will be needed moving forward.
• It is possible for some packages, e.g. datashader, that importing them will still be required even if their functionality is not explicitly called in the code (due to how the upstream, e.g. holoviz, dynamically uses them, for instance rasterization methods)
  • Removing such packages might not be viable, so extensive testing of these changes will be needed before a possible merge
• Maybe most importantly, due to the all mentioned above, I'd recommend to take some time on this, and maybe look into similar but smaller dependency-related tasks first.

[Sevans711]

@erogluorhan thank you for looking into this! Replying below with what I hope is a concrete action plan (see bottom; apologies this message ended up being very long!). Includes a few questions for your consideration, in bold; I will wait for your thoughts on those before I proceed. We can of course iterate more after it is implemented, but for now I hope to clarify the immediate next steps.

Minimize divergence between conda and pip:

Mechanisms such as lazy imports and fail locally rather than fail on installation will be needed in place to make sure there is no divergence-as much as possible- between conda and pip

I believe these mechanisms are already in place in this PR, but the relevant tests are not. I would propose: add these not-all-optional-dependencies pip builds to the autotests:

• (A) pip install uxarray
• (B) pip install "uxarray[viz]"
• (C) pip install "uxarray[geo]"
• (D) pip install "uxarray[viz,geo]"

(we don't need to add pip install "uxarray[complete]", because that's what the current autotest pipeline does already.)

And, run four test cases in each:

1. import uxarray - this should never crash
2. one simple plotting test that doesn't use geo dependencies - this should crash on (A) and (C) but not (B) and (D).
3. one simple test that uses geo dependencies but not plotting (maybe UxDataArray.to_geodataframe()) - this should crash on (A) and (B) but not (C) and (D)
4. one plotting test that also relies on geo dependencies - this should crash on (A), (B), and (C), but not (D).

Would that be a sufficient set of tests, or should we aim for more than that?

Clear guidance messages about optional installs

clear guidance messages about the expected optional installs on both conda and pip will be needed moving forward.

I am reading this as confirmation that you would maybe appreciate something like my suggested _import_with_hint_if_error so we can get clearer crash messages, even though this does not follow xarray's style. Is that correct? I'd be happy to implement this, it shouldn't take too long, but I wanted to confirm it is at least somewhat likely to be a worthwhile effort.

Beyond that, the PR currently includes updates to the uxarray readthedocs installation page, to help clarify how to do install with none/some/all optional dependencies.

Silent functionality changes if packages missing

It is possible for some packages, e.g. datashader, that importing them will still be required even if their functionality is not explicitly called in the code (due to how the upstream, e.g. holoviz, dynamically uses them, for instance rasterization methods). Removing such packages might not be viable, so extensive testing of these changes will be needed before a possible merge.

Thank you for raising these concerns. Considering more closely all the packages being removed:

• datashader: this was not being used anywhere explicitly in uxarray, which is why I initially suggested to remove it. However, I just spun up a new environment to figure out when this might be needed. It turns out, holoviews tries to use this as soon as you use the option rasterize=True. That option shows up in some of the uxarray example notebooks, and it can help when visualizing large datasets. Some of the examples also explicitly discuss the pros and cons of rasterizing. So:
  • I would now change the recommendation on datashader to: → viz; do not remove.
  • At least one test case should be added to our autotests which uses rasterize=True. Maybe the plotting test (2) above should use rasterize=True too.
  • A comment can be added on that test case, similar to the comment about pooch (see original message in this thread), explaining why the dependency exists despite not being imported anywhere explicitly.
• matplotlib-inline: this is not being used explicitly anywhere, and I am pretty sure its use is only for inline plots in places like jupyter/ipython. It should be jupyter/ipython's job to install matplotlib-inline, not uxarray's job to do it. For example, in my test environment with only pip install "uxarray[viz]" from this branch, and pip install ipykernel (to register this kernel so jupyter can recognize it), I was able to create inline matplotlib plots in jupyter without issue. I believe this should still be removed; I would be surprised (but please correct me if I'm wrong!) if any real workflow is relying on uxarray to get matplotlib-inline installed.
• pyarrow: will already be installed in the environment due to dask[dataframe] relying on it, and I believe that is our main (only?) use case for it? Since it isn't used explicitly anywhere and it gets installed anyways, I believe that is a good sign that we can remove it.
• requests: this also got installed automatically in my test environment, I suppose that means at least one of the required and/or viz dependencies already relies on it. Since it isn't used explicitly anywhere and it gets installed anyways, I believe that is a good sign that we can remove it.

I think this will reduce the amount of testing needed, as the "surface area" of possible effects of removing packages is decreased with these clarifications and change (moving datashader to viz instead of deleting it entirely).

Taking time to make sure it works

Maybe most importantly, due to the all mentioned above, I'd recommend to take some time on this, and maybe look into similar but smaller dependency-related tasks first.

Taking some time and being careful definitely makes sense.

It is a bit hard for me to come up with "similar but smaller dependency-related tasks." Maybe this would mean splitting this PR into multiple smaller PRs? I'm not really sure how to do that in a way that won't end up duplicating work or be confusing to follow. Do you have something in mind to that end already?

EDIT: ah, I just saw your comment on #1575. That is a good example.

Another option is to open another PR like "move should-be-optional dependency imports inside functions instead of at top of files" which could include all the changes to .py files here, but none of the changes to pyproject.toml and installation.rst. That would duplicate some effort, but it would allow the PR here to instead touch only the pyproject.toml, installation.rst, and yet-to-be-added optional-dependency-installation test files, which seem to be more contentious / needs caution than just relocating some imports. Do you think the duplicated effort is worth the ability to reduce the size of this PR?

Action plan / summary

Summary of changes discussed above / action plan:

1. add tests to ensure installation with optional dependencies is possible, and crashes only when we think it should crash
2. add something like _import_with_hint_if_error to make clearer errors when optional dependencies not installed.
3. actually do not remove datashader dependency entirely; but do move it to viz (it is currently required on main). Add comment explaining why it is still included despite no explicit import.
4. add a test somewhere using rasterize=True.
5. EDIT: (maybe, depending on your recommendation) move all changes to .py files into a separate PR, so that this PR can focus only on the changes to installation process and associated test files.