Skip to content

[BREAKING] FIX: Fail a backend scan when its default chat target isn't registered - #4127

Open
varunj-msft wants to merge 1 commit into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Require-Default-Chat-Targets
Open

varunj-msft wants to merge 1 commit into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Require-Default-Chat-Targets

Conversation

@varunj-msft

Copy link
Copy Markdown
Contributor

Description

When a backend scan needs a default adversarial or scorer chat target that the server didn't register as adversarial_chat or objective_scorer_chat, it now fails with that reason instead of quietly running against an OpenAIChatTarget built from the OPENAI_CHAT_* environment variables. Rich suggested this in the review of #3908 (point 3) as its own small fix on main: a misspelled name in a startup script should stop a scan, not swap in a different target.

  • require_registered_default_targets() in pyrit.scenario.core is a scope, like override_default_adversarial_target. Inside it, get_default_adversarial_target() and get_default_scorer_target() raise when the registry has no entry. Outside it, the library keeps its fallback, so notebooks and scripts that rely on OPENAI_CHAT_* work as before.
  • The scan service resolves every default inside that scope: the launch checks, preparing the scenario, and the run. Scoped and explicitly chosen targets are unaffected.
  • Starting a scan that needs a missing default returns 400 with the reason. The backend README says so.

Breaking for the backend: a server that only sets OPENAI_CHAT_* and relied on that fallback now needs to register the targets, for example through the target initializer's ADVERSARIAL_CHAT_* and OBJECTIVE_SCORER_CHAT_* variables, or choose an adversarial target for the scan.

…ad of falling back to one built from environment variables
Copilot AI balanced review requested due to automatic review settings November 10, 2026 03:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Valid scans with explicit or registered non-chat objective scorers are rejected because introspection and scorer-resolution paths unnecessarily require an unused objective_scorer_chat target.

2 open issues
What changed in this PR

This PR makes backend scans fail when a required default chat target is not registered, rather than silently using an environment-configured OpenAI target. It preserves fallback behavior for notebooks and scripts outside the strict scope.

Changes:

  • Adds and exports a scoped requirement for registered default targets.
  • Applies strict resolution to backend launch checks, preparation, and execution.
  • Adds regression tests and documents the backend configuration requirement.
File Description
tests/​unit/​scenario/​core/​test_scenario_target_defaults.py Tests strict resolution, fallback restoration, and target overrides.
tests/​unit/​backend/​test_scenario_run_service.py Tests launch failure when a required default is missing.
pyrit/​scenario/​core/​scenario_target_defaults.py Adds scoped enforcement of registered defaults.
pyrit/​scenario/​core/​__init__.py Exports the new context manager.
pyrit/​backend/​services/​scenario_run_service.py Enforces registered defaults throughout scan execution.
pyrit/​backend/​README.md Documents missing-default failures and registration requirements.

Review effort: Balanced

)
with override_default_adversarial_target(adversarial_target):
# The server's startup scripts register its defaults, so a missing one is an error rather than a fallback.
with override_default_adversarial_target(adversarial_target), require_registered_default_targets():
Comment on lines +175 to +179
if _registered_defaults_required.get():
raise ValueError(
f"No target is registered as '{preferred_target_key}', so there's no default to use. Register one "
"under that name, for example in a startup script, or choose a target explicitly."
)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants