Forward-merge release/0.9 into main - #1131
Merged
Merged
Conversation
#### Overview Anthropic compaction changes both the response stream and how later requests continue a conversation. Relay's optional response cache cannot yet reproduce those semantics faithfully, so this PR keeps compaction-related requests on the live provider path while leaving ordinary Anthropic Messages requests cacheable. It covers requests that enable compaction and follow-up requests that carry a previous compaction block. - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details - Recognize compaction configured through `context_management`, the top-level `compaction` field, or an `anthropic-beta` compaction token. - Recognize conversations that continue from an existing compaction content block. - Treat malformed or incomplete `context_management` declarations as unsafe for caching. - Keep ordinary Anthropic Messages requests and known non-compaction context-management edits cacheable. - Report `anthropic_compaction` as the response-cache bypass reason. - Add unit coverage for compaction, continuation, malformed, and non-compaction request shapes. - Add an integration test showing that repeated compaction requests both reach the provider and forward the native `compaction_delta` stream. - Document the cache limitation and bypass behavior. This complements NVIDIA#1035, which preserves compaction deltas when Relay assembles a live Anthropic stream. That collector fix does not make cached replay of the stream safe. Validation completed: - `just test-rust` - `just test-python` - `just test-node` - `just test-go` - `just docs` - `just docs-linkcheck` - `uv run pre-commit run --all-files` #### Where should the reviewer start? Start with `anthropic_compaction_requested` in `crates/adaptive/src/response_cache/key.rs`. It contains the cache-eligibility decision. The integration test `anthropic_compaction_streams_bypass_lookup_and_storage` then shows the resulting behavior across two identical requests. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Relates to NVIDIA#1034 ## Summary by CodeRabbit * **Bug Fixes** * Anthropic Messages requests using server-side compaction or compaction-related beta features now bypass response caching, preserving the native compaction lifecycle. * Compaction events are forwarded unchanged during streaming responses. * Cache-bypass telemetry now reports the `anthropic_compaction` reason. * **Documentation** * Updated response-cache documentation to describe compaction-related bypass behavior and observability. * Documented the known limitation for requests continuing from compaction blocks. Authors: - Maryam Najafian (https://github.com/mnajafian-nv) Approvers: - Will Killian (https://github.com/willkill07) URL: NVIDIA#1127
github-actions
Bot
added
size:M
|
Understand this PR’s impact Explore downstream dependencies and potential security impact with Blast Radius. No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository: NVIDIA/NeMo-Relay/.coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (6)
Included review availability: Your plan provides up to 12 included reviews per hour; 9 remain after this review. 📜 Recent review details⏰ Context from checks skipped due to timeout. (44)
🧰 Additional context used📓 Path-based instructions (7)Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.⚙️ CodeRabbit configuration file Files:
Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.⚙️ CodeRabbit configuration file Files:
Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.⚙️ CodeRabbit configuration file Files:
In MDX files, top-of-file comments must use JSX comment delimiters: `{/*` to open and `*/}` to close.📄 CodeRabbit inference engine (.agents/skills/contribute-docs/SKILL.md) Files:
Run `just docs` when the docs site changed; `./scripts/build-docs.sh html` remains the compatibility wrapper📄 CodeRabbit inference engine (.agents/skills/contribute-docs/SKILL.md) Files:
`crates/adaptive`📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md) Files:
Verify MDX files use JSX delimiters for top-of-file SPDX comments.📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md) Files:
🔇 Additional comments (7)
WalkthroughAnthropic compaction requests now bypass response caching. The change adds a dedicated bypass reason, preserves native compaction stream events, adds unit and integration coverage, and updates response-cache documentation. ChangesAnthropic compaction cache behavior
Priority: ⬇️ Low Estimated code review effort: 3 (Moderate) | ~25 minutes Change: Other Sequence Diagram(s)sequenceDiagram
participant Client
participant ResponseCache
participant AnthropicProvider
participant ResponseStream
Client->>ResponseCache: send compaction-enabled request
ResponseCache->>AnthropicProvider: execute request without cache lookup or storage
AnthropicProvider->>ResponseStream: emit compaction_delta
ResponseCache->>Client: forward compaction_delta and anthropic_compaction mark
Possibly related PRs
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1🧪 Generate unit tests (beta)
🛠️ Fix failing CI checks 💡
Comment |
willkill07
changed the title
github-actions
Bot
removed
the
Maintenance
ericevans-nv
approved these changes
Sep 21, 2026
Member
Author
|
/merge nosquash |
|
Could not determine original ForwardMerger PR from branch name. The branch name should follow the pattern |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Manually forward-merge
release/0.9intomainafter the automated forward-merge PR #1129 encountered a conflict in the versioned release-notes page. This branch was created from currentmainand mergedrelease/0.9locally, preservingmain's 0.10 release-note structure.Details
fix: bypass response cache for Anthropic compaction) fromrelease/0.9intomainwith a merge commit.main0.10 page; the 0.9-specific fixed-issue entry remains on the release branch.main.Where should the reviewer start?
Start with merge commit
4562a7b0cand its parents. The only conflict resolution isdocs/about-nemo-relay/release-notes/index.mdx, intentionally retained frommain; the #1127 changes are otherwise the release branch's commit.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Summary by CodeRabbit
New Features
anthropic_compaction.Documentation