Fix convert_tdas_to_huss guard: submissions with 'ps' but no 'tdas' raise KeyError

[eastagiletracker]

This PR proposes a fix to the huss/tdas guard in evaluations/notebooks/aimip_data_utils.py, where the 'tdas' membership is never actually tested, so a submission that carries ps but neither tdas nor huss raises KeyError in the middle of every E1–E5 run instead of taking the fallback branch that was written for it. We include this PR work along with a full history of your repo at https://eastagiletracker.com/projects/227. You can sign in with your GitHub ID to claim ownership of the project.

What is wrong

convert_tdas_to_huss opens with

if 'tdas' and 'ps' in ds.data_vars and 'huss' not in ds.data_vars:

Python parses that as 'tdas' and ('ps' in ds.data_vars) and ('huss' not in ds.data_vars) — the non-empty string 'tdas' is simply truthy, so whether the dataset actually contains tdas is never checked. A dataset with ps, no tdas and no huss therefore enters the conversion branch, prints Converting 'tdas' to 'huss'., and dies on ds['tdas']. The elif branch right below it — No 'tdas' or 'ps' available to calculate 'huss'. — is unreachable for that case, even though it is exactly the case it describes.

That matters because all five evaluation notebooks apply this function to every submission they loaded, in the Checking {experiment_submission_name} for huss/tdas. loop (E5 runs it three times, once per experiment, and most of the notebooks also call it on the ERA5 dataset), and SUBMISSIONS.md already records that coverage differs per model — DLESyM is noted as having "only a small subset of variables available", and cBottle derives huss from surface pressure and dewpoint. A submission that reports ps but neither humidity variable aborts the notebook cell rather than being reported as missing and skipped.

Reproduction at current main (d376068)

From evaluations/notebooks, in the environment.yml environment:

$ python -c "import numpy as np, xarray as xr; from aimip_data_utils import convert_tdas_to_huss; convert_tdas_to_huss(xr.Dataset({'ps': ('time', np.array([1.0e5]))}))"
Converting 'tdas' to 'huss'.
...
KeyError: "No variable named 'tdas'. Variables on the dataset include ['ps']"

Driving it the way the notebooks do, with three submissions (one with tdas+ps, one with huss, one with ps only), the loop stops at the third:

Checking model-with-tdas for huss/tdas.
Converting 'tdas' to 'huss'.
Checking model-with-huss for huss/tdas.
'huss' already present in dataset.
Checking model-with-neither for huss/tdas.
Converting 'tdas' to 'huss'.
KeyError: "No variable named 'tdas'. Did you mean one of ('tas',)?"

The change

One condition: if 'tdas' in ds.data_vars and 'ps' in ds.data_vars and 'huss' not in ds.data_vars:. Every input that works today takes the same branch as before and produces the same values — only the previously-crashing case moves, from KeyError to the message the function already had for it. After the fix, the same loop finishes:

Checking model-with-neither for huss/tdas.
No 'tdas' or 'ps' available to calculate 'huss'.

Verification

evaluations/notebooks/test_aimip_data_utils.py is new and covers the three branches of convert_tdas_to_huss (converts when tdas+ps are present and drops tdas; leaves a dataset without tdas alone; keeps a submitted huss and leaves tdas in place), the tdas-without-ps edge, and the compute_huss_from_tdas conversion itself against an independent value (280 K dewpoint at 1000 hPa is about 6.19 g/kg). Run against main unchanged, test_convert_tdas_to_huss_without_tdas fails with the KeyError above and the other four pass; with the one-line change all five pass:

$ make test
cd notebooks && conda run -n aimip-evaluations pytest test_aimip_data_utils.py
test_aimip_data_utils.py .....                                           [100%]
5 passed

The repository's existing check is unaffected — make test-env prints All imports OK before and after, and no notebook, cached dataset or figure is touched. make test is added as a sibling target and pytest to environment.yml so the new test is runnable in the documented environment (make env needs re-running once to pick it up); happy to wire that target into a CI workflow in a follow-up if you would like one.

How this was managed

This work was tracked as a single story, Fix convert_tdas_to_huss guard, on a board at https://eastagiletracker.com/projects/227 built by importing this repository's own history — 21 stories from your 3 issues and 18 pull requests, with their labels.

If you'd rather not receive contributions like this, reply no-more-prs on this pull request and we won't open any further ones on your repositories.

---

Lawrence W. Sinclair
CEO / East Agile
linkedin.com/in/lwsinclair/
eastagile.com