util_RandomizeOverlapOrder: interleave across worker files (ILE was seeing only 4 of 25 workers)

[oshaughnessy-junior]

util_RandomizeOverlapOrder.py randomises within each worker file but not across them, so
the merged grid is [worker0][worker1]...[workerN]. Any consumer that reads the whole file is
unaffected. The nested ILE reads a prefix, so it has been evaluating only the first few workers'
proposals.

Measured on a live S240629by run (25 CIP workers x 800 points, merged to 20,000 rows):

	rows	contiguous same-worker runs	workers reaching rows 0-2999
before	20,000	25	4 of 25
after	20,000	19,248	25 of 25

ILE evaluated rows 0-2999 and nothing beyond, so 21 of 25 workers contributed nothing to the
likelihood evaluations while costing full CIP time. At the more common 6 workers x 3334 the entire
3000-row prefix sits inside worker 0 — a single worker's proposal drives the iteration.

This is the failure the file header already warns about:

Important when merging files from many workers, to avoid accidentally using only the output from one of them.

Why this is alignment, not new policy

The hyperpipeline branch of write_joingrids_sub already does this, concatenating every shard and
piping through shuf, with the comment "shuffle so spokes are interleaved". Only the XML path
was missing it. This brings the two merge paths into agreement.

The change

One line — permute P_list across files before writing. --preserve-block-order restores the old
behaviour exactly, for reproducing earlier runs.

Verified on 25 real worker files: row count unchanged, no duplicates introduced, and the legacy
flag reproduces the previous output structure exactly (25 runs, 4 workers in the prefix).

Impact on existing results

This changes which points get evaluated in every iteration of every run on the XML path, so results
will shift. That is the point — but it means in-flight runs should be pinned with
--preserve-block-order if they need to stay self-consistent, and any A/B should be against a
matched-ordering control.

🤖 Generated with Claude Code

[oshaughnessy-junior]

Follow-up measurement: beyond correctness, interleaving gives a measured accuracy gain at zero extra cost.

Arms differing only in this flag (no rescoring, DAG byte-identical), with training-set size matched (38,354 vs 38,355 rows) and scored on a third arm's evaluations so neither model sees points its own exploration chose. 5 RF seeds per cell:

held-out	training set	tail lnL RMSE (nats)	tail Spearman
arm B	block-ordered (4/25 workers)	11.03 ± 0.11	0.881 ± 0.003
arm B	interleaved (25/25)	9.02 ± 0.11	0.923 ± 0.003
arm A	block-ordered (4/25 workers)	10.54 ± 0.10	0.860 ± 0.007
arm A	interleaved (25/25)	6.93 ± 0.04	0.891 ± 0.001

RMSE improves by 2.0 and 3.6 nats against a seed scatter of ~0.1 (20–36σ); tail bias halves (+1.88 → +0.83 nats). Two independent held-out sets agree in direction and magnitude.

Interpretation: evaluating a prefix drawn from 4 of 25 workers trains a measurably worse fit than the same number of points drawn from all 25. Same GPU cost, same row count — only the diversity of what got evaluated differs.

Caveats: single event; a simplified coordinate set rather than CIP's implied coordinates, so absolute values are not CIP's (the between-arm comparison uses an identical feature map and is fair); and this does not move the transverse posterior width in the two sub-iterations measured so far — it improves the fit's ordering of tail candidates, which is what any targeted-followup stage would consume.