Skip to content

Measure benchmark memory usage natively instead of via /usr/bin/time - #307

Merged
guitargeek merged 4 commits into
root-project:masterfrom
guitargeek:native-memory-measurement
Sep 3, 2026
Merged

Measure benchmark memory usage natively instead of via /usr/bin/time#307
guitargeek merged 4 commits into
root-project:masterfrom
guitargeek:native-memory-measurement

Conversation

@guitargeek

Copy link
Copy Markdown
Contributor

This supersedes #190 (itself a reincarnation of #95): same goal, but with a native implementation instead of shelling out to the time binary.

Motivation

The interpreter benchmarks obtained their RSS counter by prefixing the tutorial invocation with a hardcoded /usr/bin/time -v, parsing its output file with grep/awk, and passing the result through a tmp_mem_file in the working directory shared by all benchmarks. On systems without the time binary, the shell command failed before ever running the tutorial, so the benchmarks silently measured a failed shell invocation and reported RSS=0.

The PyROOT benchmarks were in even worse shape: no memory measurement at all (RSS was always 0), and the shell quoting of the TPython::Exec() call was broken, so the tutorials never actually ran — root only printed an interpreter parse error, which doesn't show up in its exit status either.

Changes

  • Add RB::RunCommandMeasuringRss() to RBSupport: runs a command like std::system() and reports the maximum resident set size of the child process tree via wait4(). This is the same value that /usr/bin/time -v reports as "Maximum resident set size", but with no external binary, no temporary files and no output parsing.
  • Use it in the interpreter and PyROOT benchmarks, and fail the benchmark with a clear error (SkipWithError) when a tutorial cannot be run, instead of silently reporting garbage timings.
  • Fix the PyROOT TPython::Exec() quoting and propagate failures of the interpreted code into the exit status via gSystem->Exit().
  • Fix the locations of the PyROOT tutorials after the tutorials reorganization in ROOT. Nine of them no longer exist and were dropped, as were geometry.py/na49view.py, which cannot run standalone (they need a geometry created by other macros — a failure the new error reporting surfaced).

Verification

Against a recent ROOT master build:

  • Test_nopch_mlp_higgs went from a bogus 18 ms / RSS=0 (on a machine without /usr/bin/time) to a real 4.2 s / RSS≈262M.
  • All six remaining PyROOT benchmarks now actually execute their tutorials (4.3–6.0 s each, RSS 444–490 MB), with failures reported as benchmark errors instead of fake numbers.

Note for the dashboards

Timings and RSS values for these benchmarks will jump discontinuously once this is merged: the old numbers were measuring a failing shell command, so the apparent "regression" on the graphs is the benchmarks starting to work.

RB::RunCommandMeasuringRss() runs a command like std::system() and
additionally reports the maximum resident set size of the child process
tree via wait4(). This is the same value that "/usr/bin/time -v"
reports as "Maximum resident set size", but without depending on an
external time binary, temporary files or output parsing.

This supersedes the approach of PR root-project#190, which kept shelling out to the
time binary.
The maximum resident set size was obtained by prefixing the tutorial
invocation with a hardcoded /usr/bin/time, parsing its output file with
grep and awk, and passing the result through a temporary file shared by
all benchmarks. On systems without the time binary, the invocation
failed before even running the tutorial, so the benchmark silently
measured a failed shell command and reported an RSS of zero.

Use RB::RunCommandMeasuringRss() instead, and fail the benchmark with a
clear error if the tutorial cannot be run.
The RSS counter of the PyROOT benchmarks was always reported as zero,
since no memory measurement was implemented at all. Worse, the shell
quoting of the TPython::Exec() invocation was broken, so the tutorials
never actually ran: root only reported an interpreter parse error,
which does not even show up in its exit status.

Fix the quoting, propagate a failure of the interpreted code into the
exit status via gSystem->Exit(), measure the memory usage with
RB::RunCommandMeasuringRss(), and fail the benchmark with a clear error
if the tutorial cannot be run.
The Python tutorials moved with the big tutorials reorganization in
ROOT, and several of them do not exist anymore. Update the paths of the
six remaining tutorials that still run standalone, following the same
convention as the interpreter benchmarks, and drop the others.
@guitargeek
guitargeek merged commit 178a71b into root-project:master Sep 3, 2026
1 check passed
@guitargeek
guitargeek deleted the native-memory-measurement branch September 3, 2026 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant