Repository navigation
FIX: Refuse saved targets that would authenticate with a key (part 7) - #4025
Open
varunj-msft wants to merge 137 commits into
Open
varunj-msft wants to merge 137 commits into
varunj-msft wants to merge 137 commits into
Conversation
…API input External construction now rejects a parameter whose type includes Path unless the calling service passes the name in allowed_path_params after making the value safe. The converter service allows its upload parameters once they are stored, and the target service allows the server-set upload folder and the GitHubCopilotTarget working directory. The scorer-only check is removed, and the "Valid parameters" list leaves out path parameters callers cannot set.
The target catalog now applies the path rule to required parameters too, so a target that needs a disallowed path is left out instead of listed without the field, matching the scorer catalog. allowed_path_params is typed frozenset[str] so a bare string cannot turn the allow-list into a substring match.
The target catalog still lists a target whose required path parameter is its server-set upload folder when the server has one, and leaves it out otherwise.
# Conflicts: # tests/unit/backend/test_converter_service.py
…benchmark's recipes folder
# Conflicts: # pyrit/backend/services/runtime_lifecycle.py # tests/unit/backend/test_runtime_lifecycle.py
# Conflicts: # pyrit/setup/initializers/tools.py # tests/unit/backend/test_scenario_run_service.py # tests/unit/backend/test_scenario_suite_execution_service.py # tests/unit/registry/test_attack_technique_registry.py # tests/unit/registry/test_instance_registry.py # tests/unit/registry/test_registry_recipe_restore.py
create_named_instance checks the name before it builds, and the technique registry checks names case-insensitively before it registers. Both raised the duplicate-name error without recording the clash, so a launch script that caught either one could still leave a saved entry in place of its own. The name check now records the clash itself, as register already did, and the technique check records it too.
# Conflicts: # tests/unit/registry/test_registry_recipe_restore.py
main now declares that TextTarget takes no credentials, with an empty supported_auth_modes. The saved-entry check read that as supporting no mode at all, so it refused saved TextTargets and left the saveability check for the target types with no mode to report. A class that declares no auth modes is now checked as the default api_key mode, which it has no key for.
varunj-msft
changed the title
FileDocumentStorage now raises DocumentStorageUnavailableError when the folder or container can't be used, and DocumentUnreadableError when one document exists but can't be read. The message names the location, with any SAS token hidden, and a short reason. One backend handler maps both to 503 and logs the location and reason, with the traceback at debug level, so the per-route storage helpers are gone. A single read, a conditional save and a delete now follow the listing's rules: a folder, pipe or broken link named like a document is unreadable rather than missing, and on Blob only a deleted blob counts as missing. Blob sign-in, permission, network and missing-container errors stop the listing instead of marking every file unreadable. Each unreadable report keeps the real file name and the original error, and the listing's documents are read-only. Initializer delete removes a script that can't be read or decoded, and registering one no longer turns a storage write failure into a 400. Tests patch module-level aliases instead of os.
List recipes escapes the OS text it reports, such as a file name that isn't UTF-8, and a recipe whose JSON holds an unpaired surrogate escape is reported as invalid, so the models no longer need a JSON-only serializer. Recipes may start with a byte-order mark. A document without schema_version reads as version 1, and each file is checked against the format version of the model that reads it. Validation summaries shorten each location part and the whole message, since a location can repeat a stored key. Recipes and StoredRecipe are no longer frozen, since their params and tags never were, and StoredRecipe checks that its recipe type matches its registry. The listing's recipes are read-only, and it keeps the error behind each recipe that couldn't be read, for the log.
Listing catches only storage errors, so a bug is raised instead of being reported as unreadable storage, and a bug while planning is a build error rather than an invalid recipe. Each failure is logged once, with the error behind it and URL credentials removed. The report shows the error's text with each URL's user name, query and fragment removed, any SAS signature hidden, and the length capped. A class that moved within PyRIT is restored; one now registered from outside PyRIT still fails. A technique whose name or tag clashes is a name clash, and anything else registration refuses is an invalid recipe. A recipe that sets a server file or folder is refused before it is built. Class metadata comes from the registry's cache. The class docstring lists the steps the API's create runs that restore skips.
The runtime becomes ready even when some saved entries fail; the failures stay in the runtime status. Each failed entry keeps its name in its registry: looking it up, or taking the name, raises SavedEntryUnavailableError, which the backend answers with 409, so nothing else stands in for it. Administrators see the full message and everyone else a fixed explanation of the reason. A folder that can't be listed holds every unregistered name of its kind, and each entry it left unbuilt is reported. While a saved technique is held, a launch or estimate that picks techniques by all, default or a tag is refused with 409 for scenarios whose techniques come from the registry, since the broken technique's tags may be unknown. Naming techniques still works; naming the held one fails with its reason. Route catch-alls let the error through instead of turning it into a 500.
A launch script that registers a name held for a saved entry that couldn't be restored is refused, and the refusal is recorded even when the script catches it, so the launch stops instead of running with the script's entry in the saved one's place. A suite entry that fails on a held entry, or on a broad technique selection, shows the reason without the error's details.
The OpenAI SDK adds that variable's headers to every request, so a saved OpenAI target would send whatever the server puts there, which can be a credential. While it's set, saving an OpenAI target is refused and a saved one isn't restored; the message names the variable, never its value. Targets declare such variables in server_header_variables.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is one of the changes for saving Registry entries across restarts. It builds on #3986. Until the earlier PRs merge, the diff here also shows their changes.
Problem. Restore refuses a recipe that stores a credential, but a target can use one without storing it. An OpenAI target with an empty
api_keyreads the key from this server's environment, an emptyendpointtakes this server's endpoint setting (which can hold a user name and password), and a URL can carry a SAS token or a key in its query. Saved targets are shared by everyone using the app, so one restored like that would send every user's requests with this server's keys.Change. Restore now runs every saved entry through one shared check,
check_saved_entry_params. On top of the rules it already had (no stored credential, no class that needs the server's upload folder), it refuses a target recipe that:server_fallback_parameters, with the value that turns the fallback off (for example, OpenAI'sheadersset to"{}"). Microsoft Entra ID ("identity") and no key ("none") are fine. Entra ID uses this server's Azure identity the same way targets created through the API do today, and an OpenAI target only mints that token for a recognized Azure endpoint.destination_parameters. One that would take this server's setting must be a full http(s) or ws(s) address. None may hold a user name, password, SAS token, fragment,;,&or=in the path, or a query other thanapi-version, even after percent-decoding and Unicode normalization, and a path can't have a query at all.not_saveable_reason:WebSocketCopilotTarget,HTTPTargetandHuggingFaceChatTarget. (HTTPXAPITargetwas already refused, since it needs the server's upload folder.)Each refusal is reported as a restore failure whose message names the class or parameter, never a value.
What stays the same. Converters and scorers get the same checks as before, since they don't declare fallbacks or destinations yet. Targets created without saving, and targets the startup scripts register, aren't checked, so a saved round-robin can still use startup targets, which keep their own sign-in.
server_credential_parameter(the admin check from #3247) is unchanged.Not covered. Environment variables the OpenAI SDK reads by itself (
OPENAI_CUSTOM_HEADERS,OPENAI_ORG_ID,OPENAI_PROJECT_ID) still apply to every OpenAI target on this server, saved ones included.Tests and Documentation
tests/unit/registry/test_registry_recipe_rules.py: Entra ID and no-key targets pass; an OpenAI target without a key or identity, an empty key the class reads from this server, identity withoutheaders, headers that can carry a credential, an auth mode the class doesn't support, and a class that can't be saved are refused; plain addresses and A2A agent card paths pass; URLs and paths that can carry a credential (user info, SAS, fragment, path parameters, other query keys, encoded or look-alike forms, and values that never finish decoding) are refused; an empty destination that would take this server's setting is refused, and one that isn't a full address gets its own message; text parameters that aren't destinations are left alone. Each URL rule also has a case only it catches, such as;or&alone in a path,sig=orSIG=in the host, a loneapi-key,subscription-keyorsvquery, anapi-versionvalue that isn't version-shaped, and an agent endpoint with another scheme, no host, or a value that can't be parsed.tests/unit/registry/test_registry_recipe_restore.py: restore reports a target that would use this server's credentials, or sign in with this server's account, instead of building it. The test that fed a server endpoint holding a credential into a build error is gone, since such a target is now refused before it's built; the tests that build errors hide credentials stay.uv run --frozen --extra all python -m pytest -n 8 --dist=loadfile tests/unit/registry tests/unit/backend tests/unit/prompt_target tests/unit/models tests/unit/setup tests/unit/converter tests/unit/score: 19,447 passed, 9 skipped, 4 xfaileddoc/getting_started/pyrit_conf.md: what a saved target can hold and use, which addresses must be full, the OpenAI SDK settings that still apply, and which target types can't be saved yet.No JupyText changes.