Ten non-Newtonian tests fail on amdflang gpu-omp (MI210): likely the #1588 bug class via static declare-target any_non_newtonian

[sbryngelson]

Summary

On master built alone (fbddfa9) with amdflang OpenMP offload on an AMD MI210, 10 non-Newtonian regression tests fail with tolerance mismatches. The pattern matches the bug class fixed for Re_size in #1588: a static declare target variable read inside device kernels in another translation unit. Here the variable is any_non_newtonian, which #1588's fix did not cover. The mechanism is a candidate and has not been tested; a discriminating test is proposed below.

Failing tests (upstream master alone, fbddfa9)

UUID	Test	First failure
B077CE88	1D -> 1 Fluid(s) -> Non-Newtonian -> tau0=0.001	var 32, cons.2, step 50: 6.814e-11 vs golden 1.47e-12 (rel 45)
122B0BF3	1D -> 1 Fluid(s) -> Non-Newtonian -> nn=0.5	var 256, cons.2, step 50: golden 0, reaches 1.80e-09 (band 1e-12)
EE8D38CF	2D -> 1 Fluid(s) -> Non-Newtonian -> nn=1.5	var 4, cons.3, step 50: 5.37996e-09 vs 5.38462e-09 (rel 8.65e-04)
BB0CAADB	2D -> 1 Fluid(s) -> Non-Newtonian -> tau0=0.001	var 6, cons.2, step 50: golden 0, reaches 8.61e-11 (band 4.58e-11)
E18624FD	2D -> 1 Fluid(s) -> Non-Newtonian -> nn=0.5	var 38, cons.2, step 50: golden 0, reaches 7.78e-11 (band 4.56e-11)
D6794F4C	2D -> 1 Fluid(s) -> Non-Newtonian -> IBM -> nn=0.5	var 38, cons.2, step 50: golden 0, reaches 1.89e-10 (band 1e-10)
D749AF0F	2D -> 2 Fluid(s) -> Non-Newtonian	var 1, cons.3, step 50: 1.00000000013622 vs 1.0 (band 1e-12)
00C20EE8	2D -> Example -> lid_driven_cavity_nn	cons.3, step 50: golden 0, value #106 reaches 1.15e-03 (band 1e-03)
5304E59F	2D -> Example -> poiseuille_thickening_nn	var 1, cons.2, step 1: 1.829e-03 vs 5.116e-04 (rel 2.58)
A582B270	2D -> Example -> herschel_bulkley_poiseuille_nn	var 32, cons.1, step 1: 0.998962 vs 1.0000000 (rel 1.04e-03, band 1e-03)

In the same run, 78EB6879 (1D -> 1 Fluid(s) -> Non-Newtonian -> nn=1.5) and three chemistry tests passed.

Environment

• AMD Instinct MI210 (gfx90a), one MPI rank per GPU, on the AMD AI & HPC Fund cluster
• amdflang from TheRock AFAR 23.2.1 (ROCm 7.13); ./mfc.sh build -j 16 --gpu mp --no-debug --no-single --mpi, then ./mfc.sh test --only <UUIDs>

What is established, and what is not

• Established: master alone (fbddfa9) fails these 10 on this GPU and compiler.
• Established: the same 10 also fail on a downstream branch on this GPU, and pass on CPU builds of that branch at the same commits.
• Not run: a CPU build of master alone.
• Consistent with codegen dependence: the Frontier (AMD) gpu-omp CI lane was green on master at dc0aec1, and one non-Newtonian test passes here. Fix silent viscosity loss on AMD flang GPU (host-capture Re_size in the Riemann solvers) #1588 described its Re_size failure as "unreliable and codegen-dependent", with some solver kernels affected and not others.

Candidate mechanism (not tested)

any_non_newtonian is a static declare target scalar (line numbers at master ec783a8):

• declared logical :: any_non_newtonian in src/simulation/m_global_parameters.fpp:165, under GPU_DECLARE(create=...) at :172;
• set at :848-851 and pushed with GPU_UPDATE(device=...) at :864, the conformant sync (see Harden GPU macros against silent map(to:) no-op on declare target statics (AMD path) #1613);
• read inside device code in other translation units: src/simulation/m_riemann_state.fpp :859, :888, :898, :908, :1069, :1103, :1114, :1125; src/simulation/m_hb_function.fpp:78; src/simulation/m_viscous.fpp :72, :1155; src/simulation/m_time_steppers.fpp:703.

This is the pattern #1588 fixed for Re_size. Its root cause there was a cross-translation-unit read of a static declare-target variable inside Riemann-solver kernels, and the upstream reports are ROCm/llvm-project#2890 and llvm/llvm-project#203711. The fix host-captured Re_size into Re_size_loc and passed it firstprivate; it covers Re_size only, and nothing captures any_non_newtonian. If a kernel reads it as .false., every non-Newtonian branch is skipped and the fluid integrates as Newtonian: wrong on GPU, correct on CPU. The per-fluid is_non_newtonian and hb_* arrays are allocatable, which is the kind #1589 does not flag.

Proposed discriminating test

Capture any_non_newtonian into a host-set local and pass it firstprivate in the kernels that read it, exactly as #1588 did for Re_size, then rerun the 10 tests on this platform.

• If they pass, the mechanism is confirmed, and the lint proposed in Harden AMD gpu-omp CI coverage + lint against silent declare-target regressions (follow-up to #1588) #1589 (item 3) should cover static declare-target scalars as well as arrays.
• If they still fail, the cause is elsewhere; the next check is whether is_non_newtonian and hb_* are current on the device where the kernels read them.

Related: #1588, #1589, #1613.