Repository navigation
Conversation
…d deviations, first_request fix Findings section 16, amended in place after recomputing every table from the raw session records (the quantile and ratio tables reproduce exactly): - Stock greedy outputs diverge between the runtimes in five cells (0.6b x3, 4bit/32, 4bit/512; 0/120 identical). Per plan section 3.3, their matched-generation ratios are rejected pending review; the timing stays recorded as stock-loop observations. - W1 now includes the session-A pair0 java 4bit/2048 worker (degraded warmups, measured +11.1%), which coincides with the deepest memfree trough. The worker-allocator-pool hypothesis for the memory consumer is recorded with the evidence for and against it. - Recorded deviations and open owner items: A2/A3 reduced pair counts and leader parity, the post-timeout A3 pair10, timeout vs incomplete, the memory-warning trigger, the stock 4bit/2048 exclusion bias, and the missing Python memory runs. Section 16.7 gives per-cell statuses. - Corrections: unreported controlled-python >10% shifts, the 4bit/2048 range basis, the degraded-pair counts, memory units (MB, not MiB), the pilot cell walls, the qualification host, and smaller numeric fixes. Launcher: BaselineBenchmark.finishFirstRequest re-created each runtime's first_request block per cell, keeping only the last cell; it now reuses the block. A two-cell regression test fails without the fix. The measured code remains cc0c603. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Appends section 16 (WP3) to
req/plans/phase8-2-reference-findings.md, plus one post-measurementlauncher fix. The measured code remains
cc0c603(the PR #90 merge); no golden, schema, pin orversion change.
Contents:
limits are unchanged) and the fixed-continuation qualification re-run (672/672 argmax match per
case/TF32 mode, no golden regenerated).
controlled 4bit/512 and 4bit/2048 close
unstable, with the host observations and the twocandidate causes of the memory consumer.
five cells, so their matched-generation ratios are rejected pending review (plan section 3.3).
blocks).
corrections and deviations.
Code (16.8):
BaselineBenchmark.finishFirstRequestkept only the last cell of each runtime insummary.first_request; it now reuses the runtime block. The new regression testBaselineBenchmarkTest.firstRequestSummaryKeepsEveryCellOfARuntimefails without the fix.:jmlx-benchmarks:checkpasses.🤖 Generated with Claude Code