Repository navigation
[BREAKING] FIX: Fail a backend scan when its default chat target isn't registered - #4127
Open
varunj-msft wants to merge 1 commit into
Open
varunj-msft wants to merge 1 commit into
varunj-msft wants to merge 1 commit into
Conversation
…ad of falling back to one built from environment variables
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Valid scans with explicit or registered non-chat objective scorers are rejected because introspection and scorer-resolution paths unnecessarily require an unused objective_scorer_chat target.
2 open issues
What changed in this PR
This PR makes backend scans fail when a required default chat target is not registered, rather than silently using an environment-configured OpenAI target. It preserves fallback behavior for notebooks and scripts outside the strict scope.
Changes:
- Adds and exports a scoped requirement for registered default targets.
- Applies strict resolution to backend launch checks, preparation, and execution.
- Adds regression tests and documents the backend configuration requirement.
| File | Description |
|---|---|
tests/unit/scenario/core/test_scenario_target_defaults.py |
Tests strict resolution, fallback restoration, and target overrides. |
tests/unit/backend/test_scenario_run_service.py |
Tests launch failure when a required default is missing. |
pyrit/scenario/core/scenario_target_defaults.py |
Adds scoped enforcement of registered defaults. |
pyrit/scenario/core/__init__.py |
Exports the new context manager. |
pyrit/backend/services/scenario_run_service.py |
Enforces registered defaults throughout scan execution. |
pyrit/backend/README.md |
Documents missing-default failures and registration requirements. |
Review effort: Balanced
| ) | ||
| with override_default_adversarial_target(adversarial_target): | ||
| # The server's startup scripts register its defaults, so a missing one is an error rather than a fallback. | ||
| with override_default_adversarial_target(adversarial_target), require_registered_default_targets(): |
Comment on lines
+175
to
+179
| if _registered_defaults_required.get(): | ||
| raise ValueError( | ||
| f"No target is registered as '{preferred_target_key}', so there's no default to use. Register one " | ||
| "under that name, for example in a startup script, or choose a target explicitly." | ||
| ) |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Description
When a backend scan needs a default adversarial or scorer chat target that the server didn't register as
adversarial_chatorobjective_scorer_chat, it now fails with that reason instead of quietly running against anOpenAIChatTargetbuilt from theOPENAI_CHAT_*environment variables. Rich suggested this in the review of #3908 (point 3) as its own small fix on main: a misspelled name in a startup script should stop a scan, not swap in a different target.require_registered_default_targets()inpyrit.scenario.coreis a scope, likeoverride_default_adversarial_target. Inside it,get_default_adversarial_target()andget_default_scorer_target()raise when the registry has no entry. Outside it, the library keeps its fallback, so notebooks and scripts that rely onOPENAI_CHAT_*work as before.Breaking for the backend: a server that only sets
OPENAI_CHAT_*and relied on that fallback now needs to register the targets, for example through the target initializer'sADVERSARIAL_CHAT_*andOBJECTIVE_SCORER_CHAT_*variables, or choose an adversarial target for the scan.