Block-structured AMR with batched fine advance, load-balance and active-box infrastructure - #1628
Open
sbryngelson wants to merge 1013 commits into
Open
sbryngelson wants to merge 1013 commits into
sbryngelson wants to merge 1013 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in (“default-off”) family of performance/diagnostic features (load-weight and SFC partition diagnostics, weighted init-time decomposition, rank timing), plus major simulation capabilities (active-box RHS windowing and block-structured AMR) and corresponding post-processing support and documentation/validation updates.
Changes:
- Adds new runtime parameters and toolchain metadata/validation hooks for the experimental performance/AMR feature family.
- Extends the simulation code with new modules for active-box restriction, load-weight diagnostics, SFC partition reporting, rank timing, and AMR integration points (including restart/output plumbing).
- Updates post_process to read/write AMR fine-block overlays and adds/updates golden metadata plus documentation/indexing.
Reviewed changes
Copilot reviewed 82 out of 94 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/mfc/params/descriptions.py | Adds user-facing descriptions for new experimental/performance parameters. |
| toolchain/mfc/params/definitions.py | Registers new parameters (AMR, hybrid sensors, load-balance diagnostics) and target applicability. |
| toolchain/mfc/lint_docs.py | Treats new validator checks as non-physics doc checks. |
| tests/F980C769/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ECABA006/golden-metadata.txt | Adds golden metadata for active-box test coverage. |
| tests/DD4CD8F3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/CC4213FD/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BD21A5C0/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BCBA6E74/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ACE05393/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/987D9025/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/852CCB81/golden-metadata.txt | Adds golden metadata for AMR-related golden tests. |
| tests/65C375B4/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/4DADE04B/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/454C565F/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/3A474BEE/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/2FC423D3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/13945217/golden-metadata.txt | Adds golden metadata for moving-IB under AMR test coverage. |
| src/simulation/m_viscous.fpp | Clamps FD coefficient indexing to avoid ghost-region coefficient OOB in IB drag gradient evaluation. |
| src/simulation/m_time_steppers.fpp | Integrates active-box bounds into RK update loops and interleaves AMR fine-stage/subcycle operations. |
| src/simulation/m_start_up.fpp | Wires up new modules (rank timing, active-box, load-weight, SFC partition, AMR) into init/timestep/finalize and restart I/O. |
| src/simulation/m_sfc_partition.fpp | Adds analysis-only SFC tiling + weighted partition prediction and reporting. |
| src/simulation/m_rank_timing.fpp | Adds per-rank wall-time imbalance measurement helpers and reporting. |
| src/simulation/m_load_weight.fpp | Adds per-cell load-weight field construction and rank-level imbalance reporting. |
| src/simulation/m_hypoelastic.fpp | Refactors FD coefficient setup into a callable update routine (supporting AMR grid swaps). |
| src/simulation/m_global_parameters.fpp | Adds AMR working-state mirrors and slot selection helper plus defaults for new parameters. |
| src/simulation/m_data_output.fpp | Adds output/report hooks for load-weight, SFC partition, and rank-time diagnostics. |
| src/simulation/m_checker.fpp | Adds input validation/prohibits for active-box, hybrid sensors, load-balance, and AMR configurations. |
| src/simulation/m_active_box.fpp | Adds active-box initialization/growth and debug envelope checking. |
| src/simulation/m_acoustic_src.fpp | Adds AMR-aware handling of acoustic source support (bounding boxes and overlap abort). |
| src/post_process/m_start_up.fpp | Calls AMR fine-data reader and AMR overlay writer when amr is enabled. |
| src/post_process/m_global_parameters.fpp | Adds default-off amr flag for post_process overlay behavior. |
| src/post_process/m_data_output.fpp | Implements AMR fine-block overlay mesh/variables output (Silo/binary) and multimesh registration. |
| src/common/m_phase_change.fpp | Exposes per-cell Newton iteration count and threads it through relaxation to support load-weighting. |
| src/common/m_global_parameters_common.fpp | Adjusts start_idx lifecycle/allocation and makes load_weight_wrt visible to GPU macros. |
| src/common/m_derived_types.fpp | Introduces a simple t_box type used by new partitioning infrastructure. |
| src/common/m_box.fpp | Adds box/partition arithmetic helpers (equal/weighted splits, box-from-splits). |
| src/common/m_boundary_common.fpp | Skips BC buffer population during AMR fine advance to rely on coarse-driven ghost fill. |
| docs/module_categories.json | Registers new modules under documentation categories. |
| docs/documentation/readme.md | Adds AMR section link to the documentation index. |
| .typos.toml | Adds project-specific abbreviations to the spelling allowlist. |
| D = ((gs_min(lp) - 1.0_wp)*cvs(lp))/((gs_min(vp) - 1.0_wp)*cvs(vp)) | ||
|
|
||
| #ifdef MFC_SIMULATION | ||
| if (relax .and. load_weight_wrt) then |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1628 +/- ##
==========================================
+ Coverage 61.53% 62.60% +1.07%
==========================================
Files 84 107 +23
Lines 22341 27591 +5250
Branches 3271 4232 +961
==========================================
+ Hits 13747 17274 +3527
- Misses 6144 7067 +923
- Partials 2450 3250 +800 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
Author
|
Upstream latent gap found during the MHD+AMR investigation (independent of this PR): Also for the record: MHD+AMR was attempted and re-gated on measured evidence rather than assumption — the coarse/fine seam is a continuous O(1) div(B) source that cleaning spreads but cannot remove (details in the amr.md support matrix row and commit ac203b1). |
sbryngelson
changed the title
Member
Author
|
2 tasks
fahnab666
added a commit
to fahnab666/mfc-nabid
that referenced
this pull request
Aug 29, 2026
…isting lint debt Merges the 20 commits added to sbryngelson/up-mega since ee7758b: the CCE descriptor and GPU_DECLARE fixes for the Frontier AMR abort, the NVHPC declare-ordering fix, restart format v2 (per-block owner/extents), amr_blocking_factor defaulting to 4, and the level-order clustering reductions. The delta is confined to AMR: m_amr, m_amr_regrid, m_amr_restart, m_constants, m_global_parameters, cmake/GPU.cmake, 14 AMR goldens, and two AMR-only toolchain hunks. No JWL, IBM, Riemann, or EOS source file is touched and no non-AMR golden changes. Merge was conflict-free; the local static-AMR GPU allocation cap for IBM survives intact. Also clears lint debt that predates this merge and blocked any commit: formatter line-wrapping in six JWL/IBM files, the duplicated rxn_val PROHIBITs in pre_process/m_checker.fpp (already enforced in case_validator.check_stiffened_eos, as its own docstring states), the stale hyperelasticity row in case.md (removed upstream), the missing sec-mhd anchor, and m_jwl/m_jwl_sources missing from module_categories.json.
…pass; the three statements become the scorecard
…lement; the alternatives reproduce a map already rejected
…m 2.24x to 1.14x per doubling
…r stage over the plan's flat transfer list (amr_device_pack)
…se-grid halo is 39% of growth, batched advance saves 0.60 s/step, our geo-mean equals the bar rather than beating it
…the batched advance worth 0.38; the uniform denominator re-measured at 240 steps
…gather is 0.224 s/step of non-wait work -- 25% wait, not 12%, after review folded pgather into its own phase bracket
… with the fill waves The per-box s_amr_gather_coarse_patch / s_amr_gather_from_parent, the chunked rebuild gather (plan, walk order, post/send/consume phases, chunk pool) and the deferred ISEND pool are deleted. The level-1 and parent fill waves split into an exchange phase and a per-box consume, with a full-patch mode: the per-stage ghost fill keeps the ring-clipped shell, while init and the rebuild ship the whole patch and prolong from it level by level (one wave per level, so parents are built before their children read them). The prolong no longer pushes a host-filled patch; every patch is assembled on the device. Four per-box audit sites retired (30 remain). 61/61 CPU AMR set, 19/19 debug lane.
The regrid's old-block fine-state migration was the last bespoke exchange: its own need/send lists, spack/rpack pools with a device budget and host staging, per-box tags and ISEND/IRECV/WAITALL. It is now one wave (band 4) on m_amr_wave: per-peer aggregated messages, the shared wire pools, keyed tags and the debug identity headers, with the pack/unpack kernels addressing their pool slice like the fill packs. 61/61 CPU AMR set, 19/19 debug lane.
s_amr_swap_to_fine: the three-axis ghost-coordinate extension (and the level>=2 parent-boundary rebuild) is one s_amr_extend_ghost_coords per axis. s_l0_tiles_init: its copies of the block-pool and swap-buffer allocation call the s_amr_alloc_pool / s_amr_init_swap_buffers the AMR init uses (now in m_amr_store). s_read_amr_restart: a driver over s_amr_restart_read_serial / s_amr_restart_read_parallel with the shared s_amr_restart_install_blocks. 61/61 CPU AMR set (both restart cases), 19/19 debug lane.
… present here) into up/mega
…ry predicate amr_dim / amr_sidx / amr_ext replace the per-dimension guarded statements (some thirty sites become one merge each); the inlined copies of the region box, patch box, local extent, active-box clamp, box overlap and reflux-participation predicates call the one routine each (the four reflux-participation copies are s_amr_reflux_faces); the six buffered-bound scalars are three vectors; the two inits share the pool/swap-buffer allocation and the two finalizers share their teardown. The clusterer (signatures, bisection, Morton order, merge) is m_amr_cluster. Deleted: the constant amr_early_seam_post switch and its dead branch, the [amr-merge]/[amr-grideff]/[amr-mem] campaign reports and their counters, the store trip-wire, the seven-entry tag table (one tag used).
… Fypp-folded per-direction reflux applies s_amr_capture_batch captures into either register family from a per-slot geometry table filled by s_amr_capture_slot, and s_amr_capture_total is the single place that encodes the advective / viscous / chemistry total-flux policy (it used to live in four). The fine self-capture, the child creg capture and the coarse capture all go through it. The three per-direction blocks of s_amr_reflux_apply_faces and s_amr_apply_reflux, the twelve REG_GROW calls and the per-direction register allocations are Fypp loops over (d, ta, tb); rc/rf are vectors. f_amr_is_child replaces two inline copies of the child test. Goldens unchanged (61 AMR tests, 19-test multi-rank debug lane).
…C kernel; wave: one request routine per buffer kind s_l0_route_box moves one box of a tile between the L0 field and the tile store in either direction, locally or over MPI, and the seed, scatter, reflux copy-back and restrict copy-back are thin callers of it (the restrict local path is now a direct device copy instead of a pack/unpack round trip through a host buffer, which is the same values). The extrapolation and reflective face fills are one kernel with a reflect flag. s_amr_wave_req / s_amr_wave_req_raw take the direction (1 send, 2 receive) instead of four near-identical routines. Goldens unchanged (61 AMR tests, 19-test multi-rank debug lane).
…y for three targets, per-face register helpers Shell/ghost slab lists are sb(3,6)/se(3,6) instead of six parallel arrays, loaded into the device slab table by f_amr_slab_tab_load. The three restriction kernels (level-0 field, parent store, wire buffer) are one Fypp body that derives its own child stencil from rr. The reflux faces wave and the freg wave share s_amr_freg_faces_req/update/poison and s_amr_hdr_req/check instead of four inline copies of the per-face Fypp loops. s_amr_wave_add_slabs replaces four slab-add loops; box emptiness tests are any/all. Goldens unchanged (61 AMR tests, 19-test multi-rank debug lane).
…global cell-boundary builder The grid swap/restore, swap buffers, device sync, bridge face moves and per-slot coordinate arrays iterate one Fypp list over (x, y, z) instead of three hand-copied blocks. s_amr_build_global_cb takes the ghost pad, replacing s_l0_build_extended_global_cb; the tile slot geometry and the migration pack/unpack pair are single bodies. f_amr_boxes_overlap is a vector expression and the Lagrangian clearance check uses it. Goldens unchanged (61 AMR tests, 19-test multi-rank debug lane).
… L0 and exchange Box padding/clamping, the IB merge and cap tests, the rank decomposition and coarse-range accessors, the fine-geometry extents, the static multi-level checks, tile ranges and the edge-BC / halo dispatch use amr_dim masks and (x, y, z) Fypp lists instead of hand-written 1D/2D/3D branches. s_amr_body_bbox reuses s_amr_phys_to_cells (was s_lag_phys_to_cells). Goldens unchanged (61 AMR tests, 19-test multi-rank debug lane).
… export list The single-block seed is the batched loop's one-member case (own intersection origin, zero slab offset, the installed bounds), so the duplicated kernel goes. s_amr_loc_index_init and s_amr_st_finalize are module-private. Goldens unchanged (61 AMR tests, 19-test multi-rank debug lane).
…terer s_amr_sort_by_key (bottom-up stable merge sort of indices by int8 key) replaces three hand-copied merge sorts. Goldens unchanged (61 AMR tests, 19-test multi-rank debug lane).
…ll waves s_amr_fill_drain walks a block's received transfers (fused or per-transfer) for both fill waves, and s_amr_unpack_box_device takes the patch origin instead of having a zero-origin twin. Goldens unchanged (61 AMR tests, 19-test multi-rank debug lane).
Both callers copied a slot's whole buffered box, so the six bound arguments were noise. Goldens unchanged (61 AMR tests, 19-test multi-rank debug lane).
77c0bbe moved the rank subdomain to amr_sidx/amr_ext but left s_set_amr_fine_geometry's near-boundary test on its old locals, so amr_xchg_coarse_ghosts was decided from stack garbage. Found by review.
…docs match the rules A case-optimized build aborts when the runtime riemann_solver differs from the baked one instead of leaving every flux buffer stale; jac_old traffic moves to its own kernel, since m_igr allocates it for the Jacobi solver alone; chem_params%diffusion without chemistry is rejected (its consumers read below the source-flux band); the L0 tile rebalancer slices the block-indexed arrays to the tile count; s_amr_build_block_coords takes the global boundary array's lower bound instead of assuming -1; and an Open MPI launch that oversubscribes says so. check_amr's physics doc claimed phase change, 2D axisymmetric and stretched grids work under amr while the rules reject all three unconditionally, which also left four rules unreachable: the doc now states what is enforced and the dead rules are gone.
…es nothing The packer reads the simulation's ASCII dump under D/, which only the serial writer emits, so the three cases that keep parallel_io = T packed nothing and compared empty-against-empty: MPI-IO AMR restart, np=2 dynamic regrid and load_balance were all passing vacuously. A golden case whose pack is empty now fails with a message saying so. The load_balance case gets a probe at the material interface its weighted split is decided from, so its golden holds real numbers; the two MPI-IO restart cases are registered as run-only (kind="smoke"), which is what they always were - their value is the round trip completing with the reader's own header and extent checks armed - and their empty golden files are gone. A serial multi-level restart np=2 case carries the numbers the MPI-IO twin cannot.
…ative s_read_amr_restart pushed every owned block's host column to the device after both readers had run. The serial reader already pushes each block as it reads it - it must, because allocating the next slot can grow the store device-side - and by then the reconcile's compaction has moved columns on the device too, so the shared push copied an undefined host mirror over the state just restored. The push now sits at the end of the parallel reader, the only path whose host copy is authoritative when it finishes. CPU builds cannot see the difference (host == device), so the new 'store growth' case, which owns 13 blocks against the store's initial 8 slots at np=1, earns its place on the GPU gate.
The coarse stage applies body forces, synthetic turbulence, the continuum-damage clamp and the operator-split reaction/burn substeps to the base grid; the batched fine stage applied none of them, and s_amr_step_fold then overwrote the covered coarse cells with the unforced fine average. With bf_x = T and amr = T the refined region evolved with no gravity at all - and the AMR bodyforces golden, whose comment claims coarse and fine see one consistent forcing, encoded that. Body forces and the damage clamp now run per fine block through the same swap/load/store path the 6-equation relaxation uses (s_apply_bodyforces moves to m_body_forces so the AMR modules can reach it without a cycle). The rest have no fine-block form - synthetic turbulence and the spatial-support body force build their source on the coarse grid, and the reaction and burn substeps are whole-domain operator splits run after the fold - so they are rejected with amr instead of being silently skipped. The bodyforces golden is regenerated: the refined region now gains rho*g*t momentum (2.75e-2 over six steps at g = 10), which is the forcing it was missing.
nvfortran rejects a CYCLE that follows an associated loop construct in the same iteration body: 'CYCLE may only apply to the innermost associated loop'. Caught by the NVHPC 25.5 gate.
…eady does The rebuild seeds every new block by prolonging from its coarse side and then copies back, from the stashed old blocks, the detail that coarse side cannot carry. Level >= 2 skipped that second step, so an L2 block was rebuilt from its parent's covered cells - which hold nothing but the end-of-step restriction of that same block. The round trip restrict-then-prolong is a low-pass filter, so everything finer than an L1 cell was discarded every amr_regrid_int steps and the finest level's accuracy could never accumulate past one regrid interval. The L2 stash and its cross-rank migration were built and shipped for a reader that did not exist. The carry-forward now runs at every level. amr_isect_lo cannot be used for the shift (for level >= 2 it is a footprint in the PARENT's fine frame, not an L0 index); both stashes are anchored at their block's own region_lo and hold amr_ref_ratio**lev fine cells per L0 cell, so the frames differ by amr_ref_ratio**lev*(region_lo_new - region_lo_old). Three multi-level dynamic-regrid goldens are regenerated: the solution moves by up to 1.0e-2 relative (9.2e-4 absolute in 1D, 7.9e-3 in 2D, 1.5e-4 in 3D) - the L2 detail that was being thrown away - while total mass is unchanged to 8e-16, which is also what says the frame shift is right: a wrong one would not conserve.
…metry post_process's per-rank-file reader decided which blocks carried a data record from the geometric intersection with the local subdomain. Blocks are owned WHOLE by one rank under the SFC cut, and that rank need not be the one whose subdomain they sit in, so the writer emits a record only for the rank's own blocks and marks the rest m = -1. The moment a block crossed a rank seam the reader walked off the record boundary and the overlay aborted with 'writer and reader header layouts have drifted'. np = 1 never saw it (one rank owns and intersects everything), which is the configuration the suite ran. The reader now keys the stream on the file's flag, derives the refinement ratio from the block's own region rather than the intersection (a block may hang over the subdomain edge), keeps the intersection sub-box the way the MPI-IO branch already does, and skips level-0 tiles. What serial mode cannot do is render a block whose data sits in another rank's file, so it says so once instead of quietly writing that region coarse. The new np=2 case puts a 32-cell block across the seam of a 63-cell grid; it fails without this fix (post_process exits 1) and passes with it. Reaching it needed the IO-key honoring to cover the post_process branch too: that branch forces parallel_io = T, so no test could ask for the serial reader at all - which is why this survived.
… them With l0_ntile > 0 the tiles are the slot pool's fixed prefix and amr_num_blocks counts them, so the AMR restart writer emits them beside the fine blocks - and the reader rejected the very first record with 'block level outside 1..amr_max_level'. Every coexist run therefore wrote checkpoints it could not read back, with nothing in the validator to say so. A level-0 record is now read as what it is: the tile already exists (s_l0_tiles_init runs before the read), so the record is checked against it and its data consumed - the level-0 field the ordinary restart file restores is what reseeds the tiles anyway - and the fine-geometry rebuild skips the prefix, which s_amr_assign_block_owners already did. Both writers also record the block box from the replicated metadata rather than amr_slots(k)%region: that field is written only by s_set_amr_fine_geometry, which never runs for a tile, so a coexist file carried the slot struct's uninitialized box. The serial writer now marks a slot it does not own with -1 extents, the reader's data-presence flag, instead of reporting extents it never set. The coexist static np=1 case gains restart_check: it fails without this (the round trip aborts) and passes with it.
…ree lane-specific gaps
The case-optimization guard I added to s_initialize_riemann_solvers_module
expanded to an if/call/end-if BEFORE the routine's declarations, which is
not Fortran: every --case-optimization simulation build failed to compile.
Verified from the generated source (the executable statements sat at line
453 and the declaration at 463); the guard now follows the declarations.
None of the local gates build case-optimized, which is how it got past
them.
Three more, each invisible to the lane we gate on:
- post_process assigned to unallocated extents_dp, relying on F2003
allocate-on-assignment, which ifx/ifort disable by default
(-assume norealloc_lhs); both sites now allocate and free explicitly.
- q_T_sf's Newton-guess seed reached the host only: @:ALLOCATE creates
the device copy without copying and the ACC_SETUP copyin is Cray-only,
so every other offload build iterated on uninitialized device memory
over the part of the widened allocation no coarse conversion writes.
- the post AMR overlay allocated q_cons over post's sys_size but filled
only the simulation's, leaving the post-only Lagrange beta slot as heap
garbage for the conversion to read; the block is zeroed at allocation,
which is also the right value (fine blocks exclude the cloud).
Also: the restart round-trip no longer goes quiet when a non-smoke case
produces nothing to compare, and format.sh matches the token ffmt emits
('unmatched Fortran') instead of the bare word, with the temp file
removed on any exit.
…ms in the internals doc The AMR physics doc claims chemistry support 'single- and multi-rank' including species diffusion, but the diffusion case ran at np=1 only: the one part of that claim with no test was the multi-rank seam, which is exactly what the flux_src reflux of the species and energy diffusion fluxes exists for. The np=2 twin keeps the derived cap, so the block tiles to the rank extent and its coarse/fine boundary is also a rank seam. Its conserved totals match the np=1 run to 4e-16. Two statements in the internals doc had outgrown the code: store growth now stages on the device up to amr_grow_dev_bytes (the carry-forward and the stash became device kernels, so it no longer needs host == device), and the communication is no longer blocking with fixed tags - every exchange is a keyed-tag wave. In place of the second, the section now records what the coverage audit found: 49 of 60 AMR cases pin amr_max_grid_size and 30 regrid, while most physics couplings have one case each at np=1, so the seam and ownership paths that have carried the defects are the thin ones.
Four labelled paragraphs in a row (Exact conservation. / Free-stream preservation. / Element-exact multi-rank. / Accuracy posture.) is a template, not writing; they are two paragraphs of plain prose now. Both table sections opened by announcing the table the reader is already looking at. 'Ensuring', 'automatically', 'This correction is what makes the scheme exactly conservative' and a tricolon in the first sentence went the same way, as did a limitation that restated its own first clause and one that only pointed at the section above it. Also corrects regrid step 5, which said the new blocks are prolonged from the coarse solution and stopped there - since 1654204 an old block of the same level hands its own solution forward.
Same content, 543 words lighter (3455 -> 2912). The prose said things
twice (a paragraph that defined prolongation and then restated what it
guarantees), qualified what needed no qualifying ('dynamically
repositioned', 'performed on-device for every supported physics
configuration', 'a deterministic feasibility clamp ensures'), and buried
the point of a paragraph in its last clause. Long sentences carrying
three clauses are now two or three sentences. Nothing was dropped except
repetition.
Lines of Code
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Block-structured adaptive mesh refinement for MFC, with the load-balance and active-box infrastructure it was built on. All of it is default-off: a case without
amr = Truns the same code as master, and every golden file that exists on master is unchanged.src/simulation/m_amr*.fpp, 12.5k lines): a 2:1 refined block hierarchy of up toamr_max_levellevels over the base grid, Berger-Rigoutsos clustering on a density-gradient tag field, conservative restriction and conservative-linear prolongation with physics-specific closures, flux registers and refluxing, dynamic regrid on a fixed interval, restart. Blocks live in one device-resident flat store. Fine blocks advance in batches of equal shape (one RHS call per batch), every level in lock-step at the casedt. Every inter-rank transfer is a posted wave on one engine (m_amr_wave): keyed tags, per-peer aggregation, device-resident wire pools underrdma_mpi, and an identity-header audit underMFC_DEBUG.m_box,m_load_weight,m_sfc_partition,m_load_balance): partition arithmetic, a per-cell load-weight field, Morton space-filling-curve partitioning and tile migration for the base grid.m_active_box): restricts reconstruction, Riemann and RK windows to a light-cone-grown box around non-ambient flow.User documentation:
docs/documentation/amr.md. Internals:docs/documentation/amr_implementation.md. The performance campaign's records, design memos and what was learned:misc/amr_ledger/README.md.Physics support
The batched fine advance is the only fine advance. It runs on uniform Cartesian grids for the 5- and 6-equation models, MHD and relativistic MHD (1D), hypoelasticity and continuum damage, Lagrangian bubbles, chemistry (with and without species diffusion), IGR and static or prescribed-motion immersed bodies. Subcycling, stretched or cylindrical grids, QBMM, Euler bubbles, surface tension, phase change and moving particle clouds are rejected by the case validator with a named message, never silently. The per-block fine advance that served those (correct, golden-tested, about 1.2x slower per step) lives on
sbryngelson/MFC:amr-per-blockfor a follow-on PR.Correctness evidence
MFC_DEBUG, checked at unpack; a seeded-fault control proves the check can fail.upstream/masterbuilt the same way (non-Newtonian and 3D cylindrical viscous cases, 1e-10 off a 1e-12 golden band; nothing this PR touches); the remaining 18 were AMR cases that pass 62/62 on GPU in their own run. The multi-rank AMR cases also pass under--debugwith the exchange audit armed. NVHPC 25.5 OpenMP offload builds.Code shape (for reviewers of the AMR modules)
The AMR code is a chain of modules, lowest first, each using only the ones before it:
m_amr_state(all module-scope state,GPU_DECLAREs, leaf helpers),m_amr_distribution(rank decomposition, box arithmetic, SFC cut, ownership),m_amr_wave(the exchange engine: plan sides, pools, keyed tags, audit records, the wait),m_amr_store(flat-store slot machinery, bridge loaders),m_amr_exchange(the fill waves, seams, ghost fills),m_amr_frame(the grid-state swap),m_amr_transfer(prolong, restrict, reflux and their waves),m_amr_advance(the batched fine stage advance, IB/Lagrange fine services),m_amr_l0(level-0 tiling),m_amr_stage(the four hooks the RK stage loop calls);m_amris the init/finalize driver and re-exports them all.m_amr_regrid,m_amr_registers,m_amr_restartandm_amr_xchg_auditsit beside the chain;src/common/m_amr_restart_iois the restart layout shared with post_process.What the solver itself sees:
m_time_stepperscallss_amr_stage_begin/s_amr_stage_fine/s_amr_l0_stage_update/s_amr_step_fold;m_rhscarries the flux-register capture and the batched slab's reconstruction window;m_mpi_commongains the box-based equal split (byte-identical to the inline arithmetic) and a few reductions.m_riemann_solver_hllcshows +1408/-1371: that is the kernel restructuring the batched slab needed (per-cell bodies hoisted so one launch serves a slab of blocks), with every golden unchanged.The series of cleanups before review (each gated on the same goldens, CPU and GPU, byte-identical): one exchange engine for the seven waves; the fine advance reduced to the batched lock-step path; one gather mechanism (init and the regrid rebuild fill coarse patches with the same fill waves the step uses); a twelve-phase step budget under
rank_time_wrtin place of the campaign's instrumentation; the stage hooks pulled out ofm_time_steppers; the two largest routines split; dead machinery and stale comments removed; then a de-duplication pass (-1,300 lines) that gave the code one of each thing: one flux-register capture kernel for both register families, one L0 tile box router, one restriction body for its three targets, one stable key sort, one receive drain for the fill waves, one request routine per wire buffer kind, and anamr_dimdimension mask plus (x, y, z) Fypp lists in place of hand-copied 1D/2D/3D branches. Parameters are defined intoolchain/mfc/params/definitions.pyand constrained incase_validator.py; the batched-advance admissibility rules live there.Performance
Measured on one node of eight AMD MI250X GCDs, OpenMP offload, one rank per GCD, double precision in both codes. The comparable quantity between MFC and AMReX is each code's excess over its own uniform-grid run at the same number of advanced cells:
On a 400^3 base with two refinement levels, regrid every 20 steps, RK3, an advected density blob (AMReX:
Tests/GPU/CNSon the matched deck), same node, same session, three reps per arm, control and treatment interleaved:The instrument's floor is about 0.03 s/step (control-to-control on the same node). Relative to each code's own solver, MFC's AMR machinery inflates its step by 1.78x and AMReX's by 1.86x. MFC's AMR step is slower than AMReX's in absolute terms (0.96 vs 0.82 s/step) because its base scheme (WENO5 with mapped weights and monotonicity preservation, HLLC, the 5-equation model) costs 1.7x per cell, which is a scheme choice and not an AMR term. The ratio depends on the node: the same binary reads up to 1.9x on nodes with throttling GPUs or dead InfiniBand ports, so every figure above was read with its control on the same node.
Weak scaling from 8 to 32 GPUs at fixed work per rank is 1.28x per doubling against AMReX's 1.23x on matched cross-node rungs; the growth is the base grid's cross-node halo, not the regrid family.
Where the remaining 0.42 s/step goes: about a fifth is fine-solver inflation (ghost planes per block, batch padding, dispatch floor), about half is MPI wait set by the two heaviest ranks at roughly fourteen rendezvous per step, and the rest is regrid, restriction, gather and reflux work. The levers that were tried, what each bought, and what was found to be false are in
misc/amr_ledger/README.md.Known issues
amron this branch (validator-rejected); the per-block advance that supported all but surface tension is onamr-per-blockfor a follow-on PR.