[ruiling-smartbear]

Change

Fixes #82. Each launcher records the meters it starts and stops them once its runs end (stop_meters_after): CL-Bench after its trainings, ServeLearnBench after training and tests, OfficeQA ReHarness and harness2 after the hermit command they exec (harness2's restart loop included).

A launch reuses a running meter only if it still answers three seconds later, so run && run eval never takes a meter being stopped. A meter another launch reuses stops only once that launch's runs have ended too: each launch on a port lists its processes in /tmp/hermit-wire-meter-<uid>-<port>.users. A meter restarted on the same ledger carries the earlier totals on (LEDGER_RESUME=1, set by the launchers).

Validation

• bash -n on the nine launchers.
• The helpers with a stand-in setsid (macOS) and the real wire_meter.py: meters stop after their runs; run && eval starts a fresh meter; a concurrent launch reuses a live one.
• Not run: the launchers on Linux with real trainings.

Review

• Tests and documentation cover the affected behavior, where applicable: no test covers the launchers; checked as above. CHANGELOG.
• Commands, defaults, and public interfaces are consistent: unchanged.
• Shared logs and examples contain no credentials or private data.
• Documentation-only: not applicable.
• Core changes. None.
• Independent review (post-PR audit) at 8489508: a meter could be reused while being stopped, and ServeLearnBench did not wait for the training; fixed in 6de34f0, which a second review found clean.

After review

Maintainer review fixes (66fd8a6, then main merged in):

• hermit/runtime/wire_meter.py: LEDGER_RESUME=1 makes a meter started on an existing ledger keep its totals. Without it, stopping meters made the next meter on the port (run && run eval, a ServeLearnBench test rerun, a CL-Bench tag refused because it exists) write an empty ledger over the earlier bill. Off by default, so lean_swarm is unchanged. Test: test_a_meter_carries_on_the_ledger_an_earlier_one_left.
• The nine launchers: the per-port users file above, so a meter reused by a concurrent launch is no longer stopped under it (checked with the real meter: two launches on one port, the meter outlives the first and stops after the second).
• OfficeQA ReHarness logs the meter's pid, not the watcher's.

[haizhongzheng]

Thanks for the fix. A review of 6de34f0 (read through, plus small shell/Python probes; not run with real trainings) found one issue worth fixing before merge and a few small ones.

Worth fixing

A restarted meter overwrites the previous ledger. Before this PR the meter stayed up, so a later launch on the same port reused it and kept adding to the same ledger. Now the meter stops when its run ends, and the next launch starts a new one. The new meter starts empty and writes to the same LEDGER path on its first flush (every 15 s), which replaces the earlier records:

• OfficeQA ReHarness, launch.sh run && launch.sh run eval (the documented Results setting): the eval's meter wipes the training's embedding bill from runs/wire_ledger-<port>.json. docs/recipes/officeqa.md:361 says that file "holds the whole embedding bill", which would no longer hold.
• ServeLearnBench: running launch.sh <scenario> test again later replaces $RUNS/$TAG/wire_ledger.json.
• CL-Bench: re-running a finished tag starts the meters before the REFUSING: ... exists check, so the refused launch still overwrites that tag's ledgers.

Reproduced: a ledger holding {"officeqa_reharness_run": {"calls": 42}} became "ledger": {} after a fresh meter on the same path flushed once.

Possible fixes: give each meter start its own ledger file (as harness2_stream already does with its timestamped wire_ledger-*.json) and document summing them, or have the meter load an existing ledger at startup. For CL-Bench, also run the "exists" checks before starting the meters.

Small

• examples/officeqa/reharness/launch.sh:104-105: stop_meters_after $$ puts the watcher in the background before the echo "wire meter pid $!" line, so the message prints the watcher's PID, not the meter's. Someone who stops "the meter" by that PID kills the cleanup instead. Printing ${METERS[-1]} fixes it.
• docker/offline.sh run: the watcher outlives the launcher and is not a meter, so the wrapper can list it as "left running" and wait one extra 20 s round.
• The watcher's kill -0 also succeeds on a zombie. If the caller doesn't reap the exec'd hermit (e.g. a Python Popen without wait()), the meter keeps running. A shell reaps at once, so this only affects unusual callers. Checking the state in /proc/$p/stat would cover it.

Noted, no change suggested

• A meter shared by two concurrent runs stops when the run that started it ends. This is already stated in the PR.
• The watcher signals PIDs by number. A recycled PID is possible in principle but unlikely with Linux's PID range.
• The meter records usage just after it returns the response, so a SIGTERM in that gap would lose the last call. The gap is far shorter than the watcher's 1 s poll.

Checked and fine: bash -n on all nine launchers; the helpers are identical across them; exit status, Ctrl-C and SIGTERM still pass through the exec (OfficeQA) and the EXIT trap (CL-Bench, ServeLearnBench); the process-group kill stops the harness2 restart loop with its meter. The core tests selected for this PR pass (455 passed, 30 skipped).

[haizhongzheng]

Thanks for fixing #82. We reviewed the PR in five rounds, with two independent reviews per round. Each finding was checked against the code, and most were reproduced with the real wire_meter. We then pushed one commit and a merge of main to this branch.

The stop itself works as described. On Linux, a meter stops about a second after its run ends. The group kill takes the harness2 restart loop with it. The meter writes its ledger on SIGTERM. ServeLearnBench's handoff from training to tests keeps the host meter up.

66fd8a6: review fixes

• Ledger overwritten on restart. A meter starts with an empty ledger and replaces the file when it flushes. Before this PR the meter never stopped, so this never came up. Once meters stop, the next meter on the same ledger wrote an empty ledger over the earlier bill. Three cases hit it:

  • OfficeQA's documented launch.sh run && launch.sh run eval lost the training's embedding bill;
  • running a ServeLearnBench test again lost the earlier host bill;
  • a CL-Bench tag refused because it already exists wiped that tag's ledgers, because its meters start before the check.

  With LEDGER_RESUME=1, the meter now loads the file's totals at start, after its bind. The eight launchers whose ledger outlives a meter set it. harness2 writes a new ledger per start, so it does not need it. Resuming is off by default, because lean_swarm reruns a crashed attempt in the same directory and expects an empty ledger. Test: test_a_meter_carries_on_the_ledger_an_earlier_one_left covers both settings and fails without the fix.

• Shared meter stopped under a running launch. This was the limit stated in the PR. OfficeQA ReHarness is built around one meter per port, with one bucket per config, so two configs running at once share it. Every launch on a port now lists its processes in /tmp/hermit-wire-meter-<uid>-<port>.users:

  • its own pid when it checks or starts a meter, and the processes it leaves running when it exits;
  • the launch that started the meter stops it once none of the listed processes is alive, and empties the file;
  • the file lives in /tmp rather than $TMPDIR, because clusters give each job its own $TMPDIR.

  Checked with the real meter: launch A starts the meter and launch B reuses it. The meter outlives A, stops within two seconds of B's end, and a later launch starts a fresh meter.

• Wrong pid in the log. OfficeQA ReHarness logged the watcher's pid as the meter's; it now logs the meter's.

We also merged main (#88) into the branch; the only conflict was the two CHANGELOG entries. The PR description now has an "After review" section, and the old limit is replaced.

Tests: tests/core/test_backends.py and tests/methods/reharness/test_reharness.py pass (31 tests), and all nine launchers pass bash -n.

Noted, no change

These are rare cases:

• Zombie process. A watched process that exits but is never reaped still answers kill -0, so its meter stays up.
• Last request's usage. A SIGTERM that lands in the milliseconds between a response's last byte and the usage record loses that request's usage.
• Three launches at once. If three launches hit one port within the same three seconds (one meter stopping, one launch listing itself, another restarting the meter), a launch can end up on the new meter without being listed.
• Another user's meter. When a different Unix user reuses a meter, it lists itself in its own users file, so the owner's watcher ignores it. Reusing another user's meter already sends your records into their ledger.
• Private /tmp. A per-job private /tmp, such as a container or Slurm job_container/tmpfs, does not share the users file.
• PID reuse. A watched pid that the system reuses within the one-second poll keeps the watcher waiting.
• setsid under job control. In a shell with job control on, setsid forks, so $! is not the meter's pid.

🤖 Generated with Claude Code