Saving a solved FlowSystem spends ~80% of its time in zlib, and zstd is blocked by the netcdf4 pin

[FBumann]

Prompt: "Compare how long the saving/archiving and Loading of an archived solve takes"

Note

The following content was generated by AI.

Summary

Saving a solved FlowSystem spends ~80 % of its wall time inside zlib. netCDF4 supports zstd, which on the same bytes writes ~8× faster and reads ~1.7× faster at the same file size — but the filter is absent from every netcdf4 release in our pinned range except 1.7.4, and the library's own capability flag reports it as present anyway. So there is nothing to change today. Filing it so that whoever next moves the netcdf4 pin knows the win is sitting there.

Measured — where the time goes

FlowSystem.to_netcdf() / from_netcdf(), 8760 hourly steps, N sink/source pairs, HiGHS, min of 3 repeats.

solution values	in memory	save (compression=5, default)	load	file	save (compression=0)	load	file
0.85 M	6.9 MB	180 ms	75 ms	4.4 MB	44 ms	51 ms	8.8 MB
2.82 M	22.6 MB	553 ms	238 ms	13.4 MB	135 ms	151 ms	30.1 MB
5.45 M	43.7 MB	1043 ms	451 ms	25.4 MB	217 ms	279 ms	58.5 MB

Linear in solution values (~190 ns/value save, ~85 ns/value load at the default). Phase split at 5.45 M: to_dataset 129 ms + netCDF write 934 ms on the way out; netCDF read 243 ms + from_dataset 215 ms on the way back. The bytes are the cost, not the Python object graph — _stack_equal_vars is already doing its job.

The codec, on identical data

5 M float64 (40 MB raw), one variable, through netCDF4 directly:

encoding	write	read	file
{'zlib': True, 'complevel': 5} — today	1064 ms	119 ms	33.9 MB
{'compression': 'zstd', 'complevel': 5}	122 ms	83 ms	37.6 MB
{'compression': 'blosc_zstd', 'complevel': 5}	125 ms	68 ms	34.7 MB

Round-trip exact in all three. Same size, ~8× the write, ~1.7× the read.

Why it cannot be adopted yet

pyproject.toml pins netcdf4 >=1.6.1, <1.7.5. Writing a variable with compression='zstd' and reading it back:

netcdf4	libnetcdf	result
1.6.1 (our floor)	4.9.0	RuntimeError: NetCDF: Filter error: undefined filter encountered
1.6.5	4.9.3-dev	RuntimeError: NetCDF: Filter error: undefined filter encountered
1.7.0	4.9.2	NetCDF4MissingFeatureException
1.7.1	4.9.3-dev	NetCDF4MissingFeatureException
1.7.2	4.9.4-dev	NetCDF4MissingFeatureException
1.7.3	4.9.3	NetCDF4MissingFeatureException
1.7.4	4.9.3	works, round-trip exact

Two consequences:

1. Adopting it as the default would mean pinning >=1.7.4,<1.7.5 — a single release, and the one our own pin comment flags for missing wheels.
2. Files would stop being portable. A file written with zstd on 1.7.4 fails to open on 1.6.1 with NetCDF: Filter error: undefined filter encountered. For a format we hand between machines that is the whole cost, not a detail.

The capability flag cannot be used to feature-detect

>>> import netCDF4                      # netcdf4 1.6.1
>>> netCDF4.__has_zstandard_support__
1
>>> # ... and both writing and reading zstd then fail

It reports how libnetcdf was compiled, not whether the HDF5 filter plugin is reachable at runtime. A try zstd, fall back to zlib in save_dataset_to_netcdf would therefore have to probe with a real write rather than read the flag — and even then the on-disk format would depend on the writer's environment, which is worse for a format meant to be exchanged.

Also measured: the compression level is inert

On a real solved FlowSystem (3.7 M solution values, 29.7 MB in memory):

compression	write	read	file
0	69 ms	41 ms	30.1 MB
1	339 ms	120 ms	13.5 MB
2	352 ms	112 ms	13.5 MB
4	390 ms	115 ms	13.4 MB
5 (default)	358 ms	105 ms	13.4 MB
9	466 ms	102 ms	13.4 MB

Levels 1 through 9 produce the same file; the whole cost is zlib being invoked at all. Compression itself earns its keep here — 2.2× for ~290 ms — so the default of 5 is not wrong and should stay. Lowering it to 1 would buy ~5 % of the write for 0.7 % more bytes, which is inside the noise. Worth recording so nobody re-measures it.

Proposed action

None now. Revisit when the netcdf4 pin moves past 1.7.4 — at that point save_dataset_to_netcdf can take compression='blosc_zstd' in its encoding dict for an ~8× write, provided we are willing to say that files written by that version need a reader of that version.

Severity

Performance only; no incorrect results. Saving is ~2 % of a solve at these sizes (1.0 s against 166 s), so this matters for workflows that save many small-to-medium solutions, not for one large one.

Environment

• flixopt 65e80a5 (v9.0.0 + 1), Python 3.11.15
• netCDF4 1.7.4, libnetcdf 4.9.3, HDF5 1.14.6, HiGHS 1.15.1
• Intel Xeon @ 2.10 GHz, 4 cores, 15 GB, idle box, min of 3 repeats throughout
• The version matrix was taken in separate venvs, each with numpy<2 (netcdf4 < 1.7 does not import under numpy 2)
• A fourth rung (10.7 M variables) was OOM-killed at 13.9 GB RSS during the HiGHS solve, not during the save — unrelated to this issue, noted so the ladder's gap is not mistaken for a save failure