Sweep fixes: speaker eval shared DB, lab qa-bench exit codes, Sparkle smoke stale report

[r3dbars]

Three confirmed bugs from the Haiku sweep, all in internal tools. Test first each time. The first 3 commits are the original fixes; the next 2 fix review round 1, and the last one fixes round 2.

Speaker lab: shared speaker DB leaked between runs

meeting-series kept <work>/shared-db forever, so a --force rerun (or a fresh run over a work root an earlier run used) started with every voice the last run learned. fresh_db=false meetings then matched people they were supposed to meet for the first time, and lab_result.json / naming rows looked better than they were.

• Fix (MeetingLabSharedDB.swift): before the first meeting, prepare() reads a lab-shared-db.json marker in shared-db.
  • The marker records the set, work root, ordered shared meetings and DB file name. It also records every meeting-series flag except --only/--limit/--force (flag order doesn't matter), and sha256 of the --embedder-thresholds file and the TRANSCRIPTED_LAB_KNOBS_FILE knobs file.
  • Before each shared meeting can write voices, the marker gets inProgress: <id>. recordApplied() clears it and adds the meeting once its lab_result.json is written.
• How a run decides:
  • It resumes only when there's no --force, the marker matches this run, nothing is marked in progress, and the meetings in the DB are exactly the finished ones (first N in series order, no gap).
  • It starts empty on a brand-new run (nothing finished) or with --force. It only deletes after checking the path is under the run's own work root (refuses .. escapes and the root itself).
  • It stops and asks for --force, leaving the DB untouched, for any other mismatch while finished meetings exist. Examples: a meeting killed mid-run, a failed mid-series meeting retried after later ones finished, a different --work, changed flags (--calendar-naming, --separation, --sep-*, --speaker-hint, --embedder-*), edited thresholds or knobs files, or a DB with no marker. The error says why. It doesn't silently run the rest of the series on an empty DB.
  • Every run logs resumed or starting empty (<reason>).
• Behavior change to know: --force --only m3 now runs m3 against an empty shared DB, not one holding m1/m2's voices. To rerun one mid-series meeting properly, rerun the series from the start with --force.
• Test: runMeetingLabSharedDBSelfTests() in speaker-eval-harness autoeval-self-test. It drives prepare/beginMeeting/recordApplied/fingerprintArgs/fileHashes against real files in a temp work root, and lists every failed promise. The package has no XCTest target.
  • Each round went red first, then green. Round 2's red listed 14 broken promises against the round-1 code, including the interrupted meeting, the refusals without --force, --calendar-naming, and the thresholds/knobs hashes.
  • I also broke the config check, the in-order check, the in-progress check, the in-progress clear and the path guard on purpose; each went red.
• This self-test only runs by hand. The test-matrix command is BLOCKED by agent-check.py, which won't run a bare-path command (already true on main). SpeakerEvalHarness is also in TOOLS_WITHOUT_CI, so check.sh and CI don't cover it. I ran it by hand on this head: PASS. The runMeetingSeries call sites (prepare before the loop, beginMeeting before the pipeline, recordApplied after lab_result.json, die on refusal) aren't covered by any test, because they need Parakeet and the diarizer models.

Lab: qa-bench exit 1/3 never got scored

transcripted-qa-bench.sh's write_report writes qa-report.md and results.tsv, then returns 1 on any FAIL row and 3 on any SKIP or hard-WARN row. --skip-build always records a SKIP, so it always exits 3. LabExperimentRunner treated any nonzero exit as a dead process and returned before QAResultsAnalyzer ran, so the run showed as Failed with no breakdown.

• Fix: for the QA and transcription-corpus benches, an exit of 1 or 3 gets analyzed only when it came from write_report.
  • qa-report.md must exist. results.tsv alone proves nothing, because the script truncates it at startup and can exit 1 elsewhere (cd "$REPO_ROOT" || exit 1, set -u aborts).
  • An exit 1 also needs at least one FAIL row.
  • FAIL rows still become hard gates, so the run is still Failed, now with the breakdown. Exit 3 adds a warning, so it shows Warning, not Passed.
  • Timeouts, other exit codes, and anything that didn't reach write_report still fail like before.
• Tests (QABenchExitCodeTests, a fake repo with a fake qa-bench script):
  • testQABenchHoldExitStillScoresSkippedRun, testQABenchHoldExitIsNotReportedAsAPass, testQABenchFailExitKeepsStepBreakdown: red ("Experiment process exited with code 3.") then green.
  • Review round: testQABenchAbortAfterPassingStepsStillFails (was passed), testQABenchAbortWithEmptyResultsStillFails (was warning) and testQABenchFailExitWithoutFailRowsStillFails (was passed) went red, then green.
  • testQABenchCrashWithoutReportStillFails and testQABenchUnknownExitStillFailsEvenWithReport guard the failure side.

QA: Sparkle smoke passed on a stale report

runScenario never removed launch-ui-smoke.json from an earlier run, and the default --output is a fixed /tmp path. An app that crashed before reporting would decode the old report and PASS.

• Fix: delete the old report before launching.
• Tests: testScenarioDoesNotPassOnLaunchReportLeftByEarlierRun (fake app script that writes nothing; was PASS, now fails on launch-report) and testScenarioPassesOnLaunchReportWrittenByThisLaunch (a fresh report still counts). runScenario went from private to internal so the test can drive it without the native-smoke isolation gate. The fake app is a shell script, never the real app.

Checks

• swift test --package-path Tools/TranscriptedLab --jobs 4: 16 tests pass
• Both Lab product builds and Tools/TranscriptedLab/script/build_and_run.sh --verify (run by hand): pass
• swift test --package-path Tools/TranscriptedQA: 94 tests pass
• bash build-deps.sh --force, swift build --package-path Tools/SpeakerEvalHarness, speaker-eval-harness autoeval-self-test (run by hand): PASS
• bash check.sh: everything else passed. It reports two BLOCKED (invalid-command): build_and_run.sh --verify and speaker-eval-harness autoeval-self-test. That's agent-check.py refusing bare-path matrix commands, already true on main. Neither one runs in check.sh; I ran both by hand.

Independent review: changes requested (round 2), fixes pushed, re-review pending

🤖 Generated with Claude Code

[r3dbars]

Split into one draft PR per bug, at Justin's request. Same reviewed code (approved with nits after 3 review rounds here):

• Sweep fix: speaker lab shared DB no longer leaks voices between runs #2105 speaker lab shared DB no longer leaks voices between runs
• Sweep fix: Lab scores qa-bench runs that exit 1 or 3 from write_report #2106 Lab scores qa-bench runs that exit 1 or 3 from write_report
• Sweep fix: Sparkle update smoke ignores a stale launch report #2107 Sparkle update smoke ignores a stale launch report

Closing this one; branch claude/sweep-fix-tools is kept.