Skip to content

FIX: Reject unsupported system prompts at send under RAISE policy - #3874

Open
thelastbackspace wants to merge 2 commits into
microsoft:mainfrom
thelastbackspace:fix/3862-raise-system-prompt
Open

thelastbackspace wants to merge 2 commits into
microsoft:mainfrom
thelastbackspace:fix/3862-raise-system-prompt

Conversation

@thelastbackspace

Copy link
Copy Markdown
Contributor

Description

With supports_system_prompt=False and SYSTEM_PROMPT: RAISE, set_system_prompt_async documents that the next send will raise, but send_prompt_async forwarded the system message to the provider. The normalization pipeline correctly skips adaptation under RAISE and defers to validation, but _validate_request only checked multi-turn, piece count and data types, not leftover system roles.

This adds that check to _validate_request: if the normalized conversation still contains a system message and the target does not natively support system prompts, it raises before anything is sent. Under ADAPT the system message is already squashed into user content by then, so ADAPT and native support behave as before.

Fixes #3862

Tests and Documentation

  • Added test_openai_chat_target_raises_for_unsupported_system_prompt_under_raise_policy in tests/unit/prompt_target/target/test_normalize_async_integration.py. It fails on main and passes with the fix, and asserts the provider client is never called.
  • Full unit suite: uv run -m pytest -n 4 --dist=loadfile tests/unit, 34111 passed, 28 skipped, 4 xfailed.
  • pre-commit (ruff, ty, etc.) passes on the changed files.
  • No doc changes needed; the existing set_system_prompt_async docstring already describes this behavior.

PromptTarget._validate_request now raises when a system message remains in the normalized conversation and the target does not natively support system prompts. Previously, with SYSTEM_PROMPT set to RAISE, the message was forwarded to the provider. ADAPT and native support are unchanged.

Fixes microsoft#3862
Copilot AI balanced review requested due to automatic review settings November 3, 2026 10:00
@thelastbackspace

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the change to reject unsupported system prompts before provider dispatch.

No open issues 🎉

What changed in this PR

Fixes #3862 by rejecting unsupported system prompts at the target’s send boundary, before they reach the provider. Validation runs after normalization, preserving existing adaptation and native system-prompt support.

Changes:

  • Rejects remaining system messages when the target lacks system-prompt support.
  • Adds a regression test for explicit RAISE handling that verifies the provider client is never called.
File Description
tests/​unit/​prompt_target/​target/​test_normalize_async_integration.py Tests rejection under RAISE and confirms no provider call occurs.
pyrit/​prompt_target/​common/​prompt_target.py Adds post-normalization system-role validation and updates its docstring.

Review effort: Balanced

@romanlutz Roman Lutz (romanlutz) self-assigned this Nov 5, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Target send forwards system prompts despite an explicit RAISE policy

3 participants