Skip to content

Fix profiler and TAU harness checks - #181

Open
dannybaths wants to merge 1 commit into
amd:mainfrom
dannybaths:aac7-profiler-tau-harness-fixes
Open

Fix profiler and TAU harness checks#181
dannybaths wants to merge 1 commit into
amd:mainfrom
dannybaths:aac7-profiler-tau-harness-fixes

Conversation

@dannybaths

Copy link
Copy Markdown

Summary

This PR fixes several profiler and TAU test harness issues found while validating the AAC7 OpenMPI row.

  • Profiler tests now assert produced artifacts instead of matching fragile tool output, and the
    corresponding CTest pass expressions are updated to match.
  • TAU checks now fail on build failure and assert real profile/trace artifacts.
  • TAU and Jacobi MPI detection now query the active MPI wrapper instead of checking whether ompi_info
    exists somewhere on PATH.
  • Jacobi-based TAU launches avoid placing ranks on nodes that cannot see the node-local build
    directory.
  • The PyTorch rocprof-compute run checks the dependency pins that analyze enforces before profiling,
    and the rocprof-compute tests fail immediately, with the reason, when Python is older than 3.10.
    These are reported as failures rather than skips deliberately, so the failure count is not quietly
    reduced.

Why

Several tests could pass or fail for the wrong reason:

  • Some pass regexes matched diagnostic text printed on failure. profile.0 is a substring of
    Could not open profile.0.0.0, and proto appears in output the tool prints when it produces
    nothing.
  • One TAU script continued after make failed, so the final verdict depended only on regex output.
    Because PASS_REGULAR_EXPRESSION overrides the exit code, the script and the CMake criteria had to
    be fixed together.
  • MPI detection confused mixed environments where OpenMPI's ompi_info was visible but mpic++ had
    been replaced by MPICH wrappers.
  • The PyTorch profiling run spent about 25 minutes profiling before reaching an analyze step that
    could not run, turning a dependency problem into a timeout that named no cause.

Validation

Validated on AAC7 with the OpenMPI PrgEnv row: MI300A, ROCm 7.14.0, OpenMPI 5.0.10 over OFI/CXI.

  • The six tests whose pass criteria this PR rewrites were run against these exact changes and all
    pass: Rocprof-sys_ROCm_Stream_Overlap, Rocprof-compute_ROCm_Roofline_Check, TAU_Trace_Check,
    TAU_Profile_Check, TAU_Check_HIP_Profile and TAU_Check_MPI_Profile. Each now matches on a
    string only the success path can produce.
  • The rocprof-compute Python precondition was exercised in both directions. Under Python 3.9.21 the
    test fails in about three seconds, reporting the version it found and the version it needs. Under
    Python 3.12.12 it runs to completion and the roofline assertion finds a real roofline.csv. The
    failing case is deliberately a failure rather than a skip, so an environment that cannot run
    rocprof-compute is not quietly removed from the failure count.
  • A single-node GPU probe showed that the detection and pass-criteria fixes make the Jacobi-based TAU
    tests build and produce real profile.* / tautrace.* artifacts and pprof output containing MPI
    and HIP routines.
  • The full AAC7 two-node suite then exposed a separate launch-directory issue: the tests built under
    node-local /tmp, but rank 1 was launched on the second node where that directory did not exist. This
    PR fixes that launch assumption too, by keeping the ranks on the node that owns the build directory.
    Verified on two nodes with the checkout in node-local /tmp — the failure reproduces without the fix
    and both invocations pass with it — and then in the full two-node suite, where all four Jacobi TAU
    tests pass on real artifacts (319 pass / 27 fail / 15 skip / 1 timeout of 362, suspicious-pass 0).
  • Three TAU RCCL tests still fail for an environment/package reason: MPICH-built TAU cannot instrument
    the OpenMPI-built rccl-tests benchmark. This PR does not attempt to solve that packaging mismatch.
  • The rocprof-sys PyTorch thread explosion is tracked separately as a rocprof-sys issue, not as a test
    harness bug.

Make profiler and TAU tests assert real artifacts instead of matching
diagnostic text, fail TAU builds explicitly, detect MPI by querying the
wrapper that will actually be used, and check the rocprof-compute
dependency pins before a long profiling run rather than after it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant