Phoenix case-optimization lane intermittently kills pre_process with SIGILL on every benchmark, independent of the branch

[sbryngelson]

The Case Opt | Georgia Tech | Phoenix (NVHPC) (gpu-omp) lane intermittently fails every benchmark at once, with
pre_process dying on exit code 132 (SIGILL) before it prints anything:

mfc: ERROR > :( .../build/install/gpu-mp-0e981924c0/bin/pre_process failed with exit code 132.
Failed cases: 5eq_rk3_weno3_hllc viscous_weno5_sgb_acoustic hypo_hll ibm igr

All five cases fail, nine seconds apart, which is the time it takes to launch and die rather than to run.

It is not the branch under test

The same signature appears on master, on a different runner:

run	branch	runner	result
34565571092	master	actions-runner-7	exit 132, all five cases
34565706693	master	—	success, two minutes later
34611616646	a PR touching one integer constant	actions-runner-5	exit 132, all five cases

Two master runs two minutes apart, one green and one red, is the clearest part: the lane fails independently of
what is being tested. The PR it blocked changes a single parameter in m_constants.fpp and nothing that any of
hypo_hll, ibm or igr reads.

Why it looks like a stale binary rather than a bad build

The install directory is gpu-mp-0e981924c0 in every one of these runs, green and red alike, because the name is
a hash of the build options rather than of the sources. The case-optimization lane runs with the binary already
built by a previous job, and the Phoenix runners share _work across jobs that are not pinned to a node. A
binary built with host-specific instruction selection on one node and executed on an older one dies exactly this
way: SIGILL, immediately, before any output, for every case equally.

That is a hypothesis, not a diagnosis -- I cannot see the Phoenix nodes. What is established is that the failure
is independent of the branch.

Suggestions

• Record the node name in the case-optimization lane the way test.sh already does for the ECC preflight, so a
  red run can be checked against the node that built the binary.
• Either pin the build and case-optimization jobs to the same node, or rebuild in the case-optimization job
  rather than inheriting build/install.
• If host-specific instruction selection is in play, the simplest fix is to target the oldest CPU in the
  partition for the shared build.

Until then this lane fails PRs for reasons unconnected to them, and the only remedy is to re-run it.