Downstream consumer: GAUL codes regressed to float; and is historical append-only or revised?

[Polichinel]

Two questions from a downstream consumer of the parquet outputs we write for the UN-FAO delivery. I am drafting a reply and would like this repo's answer before I send it — please comment, or open a PR where the fix is clear.

Q1 — GAUL codes are floats again (regression)

The consumer reports that GAUL codes in the parquet outputs are floating-point numbers and asks for integers, "like the other ids".

This was reported once before and confirmed fixed in March 2026 — the response at the time was "we changed the code to store GAUL codes as integers". The recent global re-upload appears to have gone through a write path that does not carry that cast.

What I need:

• Where does the int cast live, and which path applies it?
• Why does the current path miss it — different function, different branch, or was the fix never merged to the path now in use?
• Is there a test that would have caught this? If not, that is arguably the more important fix.
• A realistic timing estimate, since I have to tell them something.

Please treat the "was it ever actually fixed" question as live. Saying it was fixed and shipping floats twice is worse than the float itself.

Q2 — is historical data append-only, or revised retroactively?

The consumer asks:

It would be very appreciated if you have any recommendations for a better way to update the historical data. Maybe we just need to append the last new month. But I don't know if the previous data are modified.

They are asking so they can decide between an incremental update and a full re-pull. This is an architecture decision on their side and I want to give them the true answer, not the convenient one.

What I need: do earlier months change on later runs — through UCDP revisions, reprocessing, boundary or coverage changes, anything? If history is ever rewritten, say so plainly and, if possible, say under what circumstances and how often. If it is genuinely append-only, say that too and I will tell them they can append.

---

Reply drafted, sending once these are answered. Comments on this issue are the fastest way in.

[Polichinel]

Answering Q1 from this repo's seat. It is not a regression, and the "was it ever actually fixed" question you flagged as live is the right one to have asked — because the honest answer is that it was fixed, and then the fix was deliberately reversed four months later.

Q1 — the floats are the contract, on purpose, since 2026-07-19

views_postprocessing/contract/gaul_schema.py:39-45 declares the wire dtype of every column once, and the three *_code columns are float64. ADR-013 §5 states it as a ruling:

the *_code columns always float64 (dtype ruling 2026-07-19, MINOR — matches the canonical fixture bytes; the earlier "int64 when complete" wording made the schema depend on the data, so identical content could ship two ways; one stable schema wins)

That earlier "int64 when complete" rule is almost certainly what was reported as fixed in March. It was not a different code path and it was not lost in a merge — a §5 falsification audit on 2026-07-19 found it and changed it, because a data-dependent dtype meant the same content could ship as int64 on one run and float64 on the next, purely according to whether that run happened to contain a missing code.

Missing codes are not hypothetical here. delivery/coverage.py:48 pins 76 GAUL-uncovered cells in the land region, preserved rather than dropped: ADR-013 §5 says "rows with missing GAUL codes are PRESERVED, never pre-dropped — missing geography is null (None) in the string columns and NaN in the float columns." An integer column cannot carry NaN, so under the old rule the schema flipped run to run. That is the whole reason for the change.

Is there a test that would have caught a genuine regression? Yes, and it is stronger than a dtype assertion: column order and dtypes are byte-pinned by the ADR-013 §10 golden fixture, checked by tests/test_wire_fixture.py and tests/test_falsify_adr013_s5.py. A silent revert to int64 would fail byte-parity in CI. So the guard exists and is working — what it is guarding is the float64 decision.

Timing, since you have to tell them something: there is nothing to fix here, and changing it is not a small ask. Making the codes integers again would be a wire change — it breaks the §10 byte-parity fixture by design, requires re-vendoring the fixture in the consumer repos that pin it, and views-faoapi hard-validates this schema (handlers.py, FAO_PGMDataset._METADATA_COLS). It would also need an answer for the 76 cells: nullable Int64 rather than plain int, which not every consumer reads cleanly.

If the consumer simply wants integers at their end, the cheap answer is that they cast on read and decide what to do with the NaN rows — which their own aggregation already drops under its legacy-parity rule.

If they want it changed on the wire, that is a contract amendment with three repos in it, and it should be a decision rather than a patch.

Q2 — what I can state, and what is not mine to state

What this repo does: the historical artifact is a full snapshot, rebuilt and re-uploaded every run — not an append. _read_historical_frame reads the whole actuals series (64,742 cells x 438 months = 28,356,996 rows) and _save writes the entire table each time. So from our side there is no incremental path today, and a consumer doing "append the last month" would be diverging from what we publish rather than tracking it.

What I will not claim: whether the values of earlier months change between runs. This repo does not own the actuals — it reads them from views-datafactory, which sources UCDP. Whether UCDP revises history, and how often, is a question for that chain, and answering it from here would be exactly the kind of convenient-rather-than-true answer you said you did not want.

My recommendation for the reply: tell them the delivery is a full snapshot, so a full re-pull is always correct and always cheap to reason about; and that whether an incremental append is safe depends on upstream revision behaviour we would need to confirm before promising it.

Happy to open a PR if you want any of this written into ADR-013 or the register — but nothing in this issue points at a defect in this repo's code.