You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Split out of #269, which listed as an acceptance criterion: "The historical frame is released, or not held, across the forecast delivery phase."#269's shard-loading work landed in #271; this criterion did not, and it is filed here rather than closed over.
What it is
CRAFDPostProcessorManager / UNFAOPostProcessorManager set self._historical_frame in _read and hold it through _save — across the whole forecast delivery, which is where the memory peak is.
Why it is not urgent, measured
By its own declared dimensions — 64,742 cells x 438 months = 28,356,996 rows — the historical frame is about 108 MB at one float32 column.
For comparison, one forecast target's assembled frame at 64,742 cells x 36 months x ~1000 samples is 8.68 GB. The historical frame is therefore ≈1.2% of a single target's frame, and #271 just removed about 16.8 GB per target from the same phase.
Releasing it would be a real but small saving, and it is the kind of change that touches both partners' _read/_save sequencing — more risk than 108 MB justifies on its own.
The historical frame gaining columns — the 1.2% figure assumes a single float32 value column, and that assumption is the thing to re-check first.
Touching _read/_save for another reason anyway, at which point releasing the frame after the historical artifact is staged is nearly free.
Scope if picked up
Release self._historical_frame once _save_historical has staged its artifact, rather than holding it to the end of _save. Both partners, byte-identical (C-33). The coverage and provenance readers that use it (views_postprocessing/{unfao,crafd}/managers/*.py around :289, :369, :413, :424, :436) all run before that point — verify that, because it is the whole question.
Register: C-101 carries the measurement. Refs #269, #271.
Split out of #269, which listed as an acceptance criterion: "The historical frame is released, or not held, across the forecast delivery phase." #269's shard-loading work landed in #271; this criterion did not, and it is filed here rather than closed over.
What it is
CRAFDPostProcessorManager/UNFAOPostProcessorManagersetself._historical_framein_readand hold it through_save— across the whole forecast delivery, which is where the memory peak is.Why it is not urgent, measured
By its own declared dimensions — 64,742 cells x 438 months = 28,356,996 rows — the historical frame is about 108 MB at one float32 column.
For comparison, one forecast target's assembled frame at 64,742 cells x 36 months x ~1000 samples is 8.68 GB. The historical frame is therefore ≈1.2% of a single target's frame, and #271 just removed about 16.8 GB per target from the same phase.
Releasing it would be a real but small saving, and it is the kind of change that touches both partners'
_read/_savesequencing — more risk than 108 MB justifies on its own.What would make this worth doing
_read/_savefor another reason anyway, at which point releasing the frame after the historical artifact is staged is nearly free.Scope if picked up
Release
self._historical_frameonce_save_historicalhas staged its artifact, rather than holding it to the end of_save. Both partners, byte-identical (C-33). The coverage and provenance readers that use it (views_postprocessing/{unfao,crafd}/managers/*.pyaround:289,:369,:413,:424,:436) all run before that point — verify that, because it is the whole question.Register: C-101 carries the measurement. Refs #269, #271.