perf(monitor): make Codex transcript ingestion incremental - #92
Merged
Conversation
|
Warning Review limit reachedNext included review available in 15 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (7)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Gavel summary
Totals: 0 passed · 0 failed · 0 skipped · - |
Gavel summary
Totals: 4373 passed · 0 failed · 12 skipped · 3m45s |
adityathebe
marked this pull request as ready for review
August 25, 2026 06:26
Live transcript updates reparse and normalize the complete Codex rollout after every append, but the production path had no scaling baseline. Add deterministic generated workloads for size and transcript-shape comparisons, plus phase benchmarks that isolate parsing, session construction, and final ingest mapping. Amp-Thread-ID: https://ampcode.com/threads/T-01a0343a-93d5-7085-bdc4-257d6f6a8e61
Whole-file Codex parsing allocated a new event receiver for every JSONL record. Decode each record into one caller-owned event, resetting it first so fields omitted by later records cannot leak across lines. The ingestion benchmark shows about 4.8% fewer allocated bytes and one fewer allocation per transcript line while preserving parser output. Amp-Thread-ID: https://ampcode.com/threads/T-01a034b2-a3f9-706d-bbb1-a2e8b6af58bb
The final chat record in a growing rollout can still receive a higher-priority twin on the next append. Treat that dedupe candidate as provisional, preserve its original source-line identity when merging, and keep synthetic EOF snapshots from closing the real record boundary. Amp-Thread-ID: https://ampcode.com/threads/T-01a034b2-a3f9-706d-bbb1-a2e8b6af58bb
Retain turn, call, reasoning, and twin-dedupe state across JSONL records while releasing settled rows immediately. Whole-file extraction now wraps the same parser used by future suffix reads, and EOF snapshots do not mutate its checkpoint. Amp-Thread-ID: https://ampcode.com/threads/T-01a034b2-a3f9-706d-bbb1-a2e8b6af58bb
Share Codex normalization between whole-file builds and a compact monitor checkpoint. The incremental path retains aggregate and per-turn state, emits only messages and turns touched by each append, and overlays provisional EOF rows without committing them. Amp-Thread-ID: https://ampcode.com/threads/T-01a034b2-a3f9-706d-bbb1-a2e8b6af58bb
Keep one parser and compact normalizer checkpoint per rollout path, seek from the last complete JSONL record, and bound reads to a same-descriptor size snapshot. Invalidate on rewrites, replacements, identity changes, or bookkeeping drift; publish the checkpoint only after database source bookkeeping succeeds and requeue growth observed during the write. Amp-Thread-ID: https://ampcode.com/threads/T-01a034b2-a3f9-706d-bbb1-a2e8b6af58bb
Keep the changed-file projection sorted as paths are first observed and order only the turns touched by the current append. Warm projections can now reuse historical aggregate slices instead of rebuilding them from every prior turn and file. Amp-Thread-ID: https://ampcode.com/threads/T-01a034b2-a3f9-706d-bbb1-a2e8b6af58bb
Seed the production parser and accumulator checkpoint behind 1k, 10k, and 25k-line rollouts, then measure a fixed nine-line append without database work. Keep the existing whole-file size series as the cold replay baseline. Amp-Thread-ID: https://ampcode.com/threads/T-01a034b2-a3f9-706d-bbb1-a2e8b6af58bb
adityathebe
force-pushed
the
feat/codex-ingestion-benchmarks
branch
from
August 25, 2026 06:26
829d4ca to
ded46b7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Captain reprocessed the complete Codex JSONL transcript after every append, so live-update latency and allocation grew with the full session history.
This keeps an in-memory parser and normalization checkpoint per transcript and resumes from the last complete newline. Partial records are reread on the next update, and checkpoints advance only after ingestion succeeds. Restart, parser-version changes, truncation, replacement, or source-bookkeeping drift safely fall back to a full replay.
Benchmark
benchstatcompares the PR's original benchmark commit (841a46c, whole-file refresh) with the incremental implementation (829d4ca, warm nine-line append). Each side used 10 samples and 10 iterations per sample. The benchmark labels were normalized toCodexLiveUpdatebecause the before and after measurements intentionally exercise different production entry points.Warm update time and allocation are now effectively flat as the transcript grows. A cold replay remains available for restart and invalidation cases.
Closes #91