Repository navigation
FIX: Reject unsupported system prompts at send under RAISE policy - #3874
Open
thelastbackspace wants to merge 2 commits into
Open
thelastbackspace wants to merge 2 commits into
thelastbackspace wants to merge 2 commits into
Conversation
PromptTarget._validate_request now raises when a system message remains in the normalized conversation and the target does not natively support system prompts. Previously, with SYSTEM_PROMPT set to RAISE, the message was forwarded to the provider. ADAPT and native support are unchanged. Fixes microsoft#3862
Contributor
Author
|
@microsoft-github-policy-service agree |
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the change to reject unsupported system prompts before provider dispatch.
No open issues 🎉
What changed in this PR
Fixes #3862 by rejecting unsupported system prompts at the target’s send boundary, before they reach the provider. Validation runs after normalization, preserving existing adaptation and native system-prompt support.
Changes:
- Rejects remaining system messages when the target lacks system-prompt support.
- Adds a regression test for explicit
RAISEhandling that verifies the provider client is never called.
| File | Description |
|---|---|
tests/unit/prompt_target/target/test_normalize_async_integration.py |
Tests rejection under RAISE and confirms no provider call occurs. |
pyrit/prompt_target/common/prompt_target.py |
Adds post-normalization system-role validation and updates its docstring. |
Review effort: Balanced
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
With
supports_system_prompt=FalseandSYSTEM_PROMPT: RAISE,set_system_prompt_asyncdocuments that the next send will raise, butsend_prompt_asyncforwarded the system message to the provider. The normalization pipeline correctly skips adaptation under RAISE and defers to validation, but_validate_requestonly checked multi-turn, piece count and data types, not leftover system roles.This adds that check to
_validate_request: if the normalized conversation still contains a system message and the target does not natively support system prompts, it raises before anything is sent. Under ADAPT the system message is already squashed into user content by then, so ADAPT and native support behave as before.Fixes #3862
Tests and Documentation
test_openai_chat_target_raises_for_unsupported_system_prompt_under_raise_policyintests/unit/prompt_target/target/test_normalize_async_integration.py. It fails on main and passes with the fix, and asserts the provider client is never called.uv run -m pytest -n 4 --dist=loadfile tests/unit, 34111 passed, 28 skipped, 4 xfailed.pre-commit(ruff, ty, etc.) passes on the changed files.set_system_prompt_asyncdocstring already describes this behavior.