From 362525f2ff6ead9935aad16b7df590a74de82f26 Mon Sep 17 00:00:00 2001 From: Spencer Bryngelson Date: Wed, 16 Sep 2026 21:55:42 -0400 Subject: [PATCH] Refresh the device copy of num_gps when a moving immersed boundary is updated num_gps, the number of ghost points, is a declare-target module variable that bounds every device loop over the ghost points (s_ibm_correct_state, s_compute_image_points, s_compute_interpolation_coeffs). It is copied to the device once, in s_ibm_setup. s_update_mib recounts it on the host every RK stage for a moving body but never refreshed the device copy, so the kernels kept the setup-time count for the whole run: when the true count had dropped, the entries beyond it were stale ghost points from earlier stages -- cells that may now be fluid at the surface -- and s_ibm_correct_state wrote reconstructed wall states into them; when it had grown, the newest ghost cells were never corrected. Which stale entries survived depended on the atomic fill order of s_find_ghost_points, so results differed run to run. Measured on Frontier (CCE, OpenMP offload) with two flapping STL wings: a probe kernel reads the device copy, which differs from the host value once the body moves; the count on one rank swings between 68 and 1800 within a wingbeat against a setup value of 510. Two identical restarts differed in 15,000 cells after five steps; with this refresh they are bit-identical, and a mid-run restart matches the unrestarted run to 1e-13. On the same case the first-wingbeat mean lift changes by 30 % and a mirror-symmetric pair of wings becomes left/right symmetric to 0.3 % of peak (it was 30 %). Static bodies never enter s_update_mib and are unaffected. Fixes #1886. Made with Claude Code. --- src/simulation/m_ibm.fpp | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/simulation/m_ibm.fpp b/src/simulation/m_ibm.fpp index cbfeb88cd..f85ff723e 100644 --- a/src/simulation/m_ibm.fpp +++ b/src/simulation/m_ibm.fpp @@ -966,6 +966,12 @@ contains call nvtxStartRange("COMPUTE-GHOST-POINTS") ! recalculate the ghost point locations and coefficients call s_find_num_ghost_points(num_gps) + ! num_gps is a declare-target module variable and bounds every device loop over the ghost points; it is + ! copied to the device once in s_ibm_setup, so without this refresh the kernels keep the setup-time count + ! as a moving body's count changes: stale list entries beyond the current count are written as wall + ! states into cells that are now fluid, new ghost cells are left uncorrected, and which entries those are + ! depends on the atomic fill order -- nondeterministic results run to run (#1886). + $:GPU_UPDATE(device='[num_gps]') call s_find_ghost_points(ghost_points) call nvtxEndRange