Skip to content

The manager holds _historical_frame across the forecast delivery — measured at ~1.2% of a target frame, so not urgent #273

Description

@Polichinel

Split out of #269, which listed as an acceptance criterion: "The historical frame is released, or not held, across the forecast delivery phase." #269's shard-loading work landed in #271; this criterion did not, and it is filed here rather than closed over.

What it is

CRAFDPostProcessorManager / UNFAOPostProcessorManager set self._historical_frame in _read and hold it through _save — across the whole forecast delivery, which is where the memory peak is.

Why it is not urgent, measured

By its own declared dimensions — 64,742 cells x 438 months = 28,356,996 rows — the historical frame is about 108 MB at one float32 column.

For comparison, one forecast target's assembled frame at 64,742 cells x 36 months x ~1000 samples is 8.68 GB. The historical frame is therefore ≈1.2% of a single target's frame, and #271 just removed about 16.8 GB per target from the same phase.

Releasing it would be a real but small saving, and it is the kind of change that touches both partners' _read/_save sequencing — more risk than 108 MB justifies on its own.

What would make this worth doing

  • A delivery that fails on memory after #269: assembling a target held three copies of it, not one — 3.06x peak to 1.13x #271, where the measurement points here rather than at the shard path.
  • The historical frame gaining columns — the 1.2% figure assumes a single float32 value column, and that assumption is the thing to re-check first.
  • Touching _read/_save for another reason anyway, at which point releasing the frame after the historical artifact is staged is nearly free.

Scope if picked up

Release self._historical_frame once _save_historical has staged its artifact, rather than holding it to the end of _save. Both partners, byte-identical (C-33). The coverage and provenance readers that use it (views_postprocessing/{unfao,crafd}/managers/*.py around :289, :369, :413, :424, :436) all run before that point — verify that, because it is the whole question.

Register: C-101 carries the measurement. Refs #269, #271.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions