FEAT: Read saved registry recipes (part 2) by varunj-msft · Pull Request #3892 · microsoft/PyRIT

varunj-msft · 2026-11-03T17:28:29Z

Description

Part 2 of saving Registry entries across restarts. Stacked on part 1 (#3891); part 3 (#3908) builds on it. This PR's own changes: varunj-msft/PyRIT@varunj-msft/13178-Unreadable-Document-Listing...varunj-msft/13178-Read-Registry-Recipes

Design: 4.3 What a recipe holds, 4.2 Where recipes live. Where every part fits: PR map.

This is one of the changes for saving Registry entries across restarts. It builds on #3891. Until #3891 merges, the diff here also shows its changes.

Problem. There's no file format or storage for a saved target, converter, scorer or technique.

Change.

pyrit/models/catalog/registry_recipe.py: the recipe models, exported from pyrit.models.catalog.
- A target, converter or scorer recipe holds schema_version, class_module, class_name and params.
- A technique recipe holds the fields of the technique create request, without the name, with the same limits and tag rules. The library can't import the backend, so they're copied, and FEAT: Rebuild saved registry entries in dependency order (part 3) #3908 (part 3) adds a test that the two models accept and reject the same documents.
- Unknown fields are rejected, so a file with an extra field such as credentials is reported, not silently read.
- Also adds the failure and reference types.
pyrit/models/request_limits.py: adds MAX_TEXT_LENGTH for the technique description. FIX: Apply the request limits to request fields added since the last limits PR #3595 adds the same line, so the two merge cleanly; until it lands, backend/models/common.py keeps its own copy of the same value.
pyrit/registry/registry_recipe_storage.py: RegistryRecipeStorage, a FileDocumentStorage with one folder per registry (targets/, converters/, scorers/, techniques/) under one source.
- On Blob the folder goes on the URL path, so a SAS token stays in the query.
- list_recipes() uses the listing from the previous PR. It returns the parsed recipes with their versions (the SHA-256 of the bytes), a failure for each unreadable, invalid or newer-format file, and the ignored files.
- The format version is checked first, so a file from a newer PyRIT says so instead of failing on a field it doesn't know.
Report text that isn't valid UTF-8 is escaped in backslash form, such as a local file name with invalid bytes or an entry name written with a \udcff escape inside a recipe, so the report can always be sent as JSON. Valid text is unchanged.
The file name is the entry name, so only names that fit the document rule (up to 64 lowercase letters, digits and underscores, starting with a letter) can be saved. The PRs that save entries will refuse other names.

What stays the same. Nothing calls this yet; #3908 (part 3) rebuilds entries from these recipes. No API, config or behavior change.

Tests and Documentation

New tests/unit/registry/test_registry_recipe_storage.py: recipes and their versions, field limits and unknown fields, failures for unreadable, invalid and newer-format files, ignored names, Blob folders that keep a SAS token in the query, and report text that isn't UTF-8 (escaped only when sent as JSON; models keep the text they were given; valid text unchanged).
pytest tests/unit/registry tests/unit/models -n 4: 3,459 passed
pre-commit run --from-ref origin/main --to-ref HEAD: passed
No doc change: nothing calls this yet. JupyText: not run (no notebooks changed).

…sting

…ecipe models

…s JSON

Copilot

🟢 Approval recommended

No active review issues require changes to the registry recipe models or storage reader.

No open issues 🎉

What changed in this PR

Adds the models and storage reader needed to preserve registry entries across restarts. This second stacked PR builds on #3891’s document-listing support; entry rebuilding and startup restoration remain for later PRs.

Changes:

Defines versioned component and technique recipes, plus restore-report types with JSON-safe text.
Reads recipes from per-registry local or Azure Blob folders, preserving SAS queries and reporting invalid, unreadable, newer-format, and ignored files.
Adds unit coverage for parsing, versions, validation, storage failures, and report serialization without changing existing callers.

File	Description
`tests/unit/registry/test_registry_recipe_storage.py`	Tests recipe loading, validation, versions, Blob paths, and report serialization.
`tests/unit/registry/test_file_document_storage.py`	Tests issue-aware local and Blob listings and safe error messages.
`pyrit/registry/registry_recipe_storage.py`	Adds per-registry storage, recipe parsing, and failure reporting.
`pyrit/registry/file_document_storage.py`	Adds issue-aware listings while retaining legacy listing behavior.
`pyrit/models/request_limits.py`	Adds the shared technique-description text limit.
`pyrit/models/catalog/registry_recipe.py`	Defines recipe schemas, references, and restore-report models.
`pyrit/models/catalog/__init__.py`	Exposes the new models through lazy exports.

_{Review effort: Balanced}

…the listing

…s that can't be checked

…with a watchdog

…navailable storage

Richard Lundeen (richlundeen)

Requesting changes. My review on #3908 has the main comment about the direction for the restore work; this PR stays as it is apart from the inline comments below. Inline comments marked Blocking need to be addressed before merge; Non-blocking ones are judgment calls.

Richard Lundeen (richlundeen) · 2026-11-09T19:35:26Z

+
+    model_config = ConfigDict(extra="forbid", strict=True, allow_inf_nan=False, frozen=True)
+
+    schema_version: int = Field(


Blocking: default schema_version to 1, not to the current version.

When CURRENT_SCHEMA_VERSION moves to 2, a file without the field would read as version 2 while holding version 1 content, so a migration would skip exactly the files that need it. scenario_preset.py (lines 66-71) explains this and uses Field(1, ge=1), and scenario_suite.py does the same.

Pin the default with a test, as the preset and suite models do

The preset and suite models keep this rule from coming back with tests: test_schema_version_default_does_not_track_the_current_schema_version in test_scenario_preset.py and test_schema_version_default_stays_pinned_to_first_shape in test_scenario_suite.py. No recipe test does. test_reads_technique_recipes_as_technique_recipes reads a document without schema_version, but it compares the result with TechniqueRecipe.model_validate(document), which uses the same default as the code under test. With the fix, add a test that the schema_version default is 1 for ComponentRecipe and TechniqueRecipe. Then a later change cannot tie the default to the current version again without a test failure.

Richard Lundeen (richlundeen) · 2026-11-09T19:35:26Z

+        if not isinstance(payload, dict):
+            return self._failure(name=name, version=version, reason=invalid, message="Not a JSON object.")
+        schema_version = payload.get("schema_version")
+        current = ComponentRecipe.CURRENT_SCHEMA_VERSION


Blocking (small): this compares technique files against ComponentRecipe's version. Choose the model first (line 149) and use model.CURRENT_SCHEMA_VERSION.

Use one recipe format version instead of a per-model ClassVar

Choosing the model first fixes this lookup, but the ClassVar still cannot change per model. The schema_version default and le bound are set once, when _RegistryRecipe is defined. If TechniqueRecipe sets CURRENT_SCHEMA_VERSION = 2, its schema_version field keeps default 1 and Le(le=1). Then, with the suggested lookup, this check passes a version-2 technique file, and model_validate reports it as invalid_recipe ("Input should be less than or equal to 1"). No layer of this stack (#3908, #3909, #3965) overrides the ClassVar. Use one module constant, as PRESET_SCHEMA_VERSION does in scenario_preset.py: declare schema_version: int = Field(1, ge=1, le=RECIPE_SCHEMA_VERSION) and compare with that constant here. This also fixes the default in the schema_version thread on line 49 of registry_recipe.py.

Richard Lundeen (richlundeen) · 2026-11-09T19:35:26Z

+        version = self._compute_version(content)
+        invalid = RestoreFailureReason.INVALID_RECIPE
+        try:
+            payload = json.loads(content.decode("utf-8"))


Blocking (small): accept a UTF-8 byte-order mark.

Until saving lands, recipes are written by hand, and Windows tools often add a BOM; such a file is reported as "Not valid UTF-8 JSON." Decode with utf-8-sig.

Richard Lundeen (richlundeen) · 2026-11-09T19:35:26Z

+        return RestoreFailure(kind=self._kind, name=name, reason=reason, message=message, version=version)
+
+    @staticmethod
+    def _describe_validation_error(error: ValidationError) -> str:


Blocking (small): cap what this repeats.

pydantic's loc includes stored key names verbatim, such as an unknown top-level key or an over-long params key, with no length limit, although the docstring says stored values aren't echoed. Truncate each location part, cap the whole message, and adjust the docstring.

Also escape control characters, and apply the cap in _failure

Two additions to this cap. Key names keep control characters: an unknown key "\n\x1b[31mFAKE" goes into RestoreFailure.message unchanged, so a log or terminal that shows the message gets a new line and a color code. Escape non-printable characters in each location part as well as truncating it. The newer-version message on line 145 also repeats stored content: a 4,000-digit schema_version gives a 4,065-character message. Apply the cap in _failure, which builds every failure in this file, not only in _describe_validation_error, and add a test that pins the cap with an over-long key.

Richard Lundeen (richlundeen) · 2026-11-09T19:35:26Z

+
+
+# The model keeps the text it was given, so a name still matches the entry it names; only JSON output is escaped.
+_DisplayText = Annotated[str, PlainSerializer(_escape_invalid_text, return_type=str, when_used="json")]


Blocking (simplification): escape once at the edge instead of in the model.

Three commits went into making report text that isn't valid UTF-8 serializable. The only sources are file names from the OS and unpaired surrogate escapes such as \udcff inside a recipe. Escape file names once in list_recipes (line 120 of registry_recipe_storage.py), and reject a recipe whose text has unpaired surrogates as invalid (one check after json.loads). Then _DisplayText and its JSON-only serializer can go.

Check the whole payload, and escape the ignored-file message too

Two details for this change. Model validation rejects unpaired surrogates only in strings with length or pattern limits, such as class names, tags, converter names, params keys, and description. It keeps them in params values at any depth and in adversarial_system_prompt, adversarial_seed_prompt, and adversarial_prompt_template, which have no limits. At this head, a technique file with "adversarial_system_prompt": "p\udcff", or a params value "v\udcff", is returned as a valid StoredRecipe, and its model_dump_json() raises PydanticSerializationError. So the check after json.loads must walk the whole payload. Also, the ignored file's message repeats the raw name, because validate_registry_name builds "Invalid registry name 'bad\udcff' ...". Escape message as well as file_name in list_recipes; otherwise that message cannot be encoded after _DisplayText is removed.

Richard Lundeen (richlundeen) · 2026-11-09T19:35:26Z

+    )
+
+
+class TechniqueRecipe(_RegistryRecipe):


Non-blocking: shared model or copy.

TechniqueRecipe copies the API's CreateTechniqueRequest field for field, and #3908 adds three tests to keep them in step. A shared base model would remove the copy and the tests; the copy does force a deliberate decision about the stored format whenever the API changes. Either is defensible.

The import direction does not force the copy

One fact for this choice: the TechniqueRecipe docstring says the fields are copied because this library can't import the API's create request. A shared base does not need that import. Put the base in pyrit.models and let CreateTechniqueRequest extend it; pyrit/backend/models/techniques.py already imports MAX_ITEMS and TechniqueOverride from pyrit.models modules, and doc/code/framework.md (Models) asks authors to consider moving overlapping classes to pyrit.models. If you keep the copy, change the docstring to say that it is a deliberate copy of the stored format that the parity tests in #3908 keep in step.

Richard Lundeen (richlundeen) · 2026-11-09T19:35:26Z

+
+    CURRENT_SCHEMA_VERSION: ClassVar[int] = 1
+
+    model_config = ConfigDict(extra="forbid", strict=True, allow_inf_nan=False, frozen=True)


Non-blocking: frozen=True doesn't make params or tags immutable, and hashing an instance raises because of them. Fine unless someone caches or hashes these; otherwise consider dropping frozen so it doesn't promise more than it does.

Richard Lundeen (richlundeen) · 2026-11-09T19:35:26Z

+    name: _DisplayText
+
+
+class StoredRecipe(BaseModel):


Non-blocking: StoredRecipe accepts a technique ref with a ComponentRecipe, and the other way round. A small model validator would catch that misuse early.

FileDocumentStorage now raises DocumentStorageUnavailableError when the folder or container can't be used, and DocumentUnreadableError when one document exists but can't be read. The message names the location, with any SAS token hidden, and a short reason. One backend handler maps both to 503 and logs the location and reason, with the traceback at debug level, so the per-route storage helpers are gone. A single read, a conditional save and a delete now follow the listing's rules: a folder, pipe or broken link named like a document is unreadable rather than missing, and on Blob only a deleted blob counts as missing. Blob sign-in, permission, network and missing-container errors stop the listing instead of marking every file unreadable. Each unreadable report keeps the real file name and the original error, and the listing's documents are read-only. Initializer delete removes a script that can't be read or decoded, and registering one no longer turns a storage write failure into a 400. Tests patch module-level aliases instead of os.

List recipes escapes the OS text it reports, such as a file name that isn't UTF-8, and a recipe whose JSON holds an unpaired surrogate escape is reported as invalid, so the models no longer need a JSON-only serializer. Recipes may start with a byte-order mark. A document without schema_version reads as version 1, and each file is checked against the format version of the model that reads it. Validation summaries shorten each location part and the whole message, since a location can repeat a stored key. Recipes and StoredRecipe are no longer frozen, since their params and tags never were, and StoredRecipe checks that its recipe type matches its registry. The listing's recipes are read-only, and it keeps the error behind each recipe that couldn't be read, for the log.

Richard Lundeen (richlundeen) · 2026-11-09T23:31:40Z

+            return self._failure(
+                name=name, version=version, reason=invalid, message=self._describe_validation_error(error)
+            )
+        except RecursionError:


[P3] Report deep params nesting as too deep, and remove the RecursionError branch that cannot run

model_validate does not raise RecursionError for deep JSON. pydantic-core stops at about 100 nested levels and raises a ValidationError of type recursion_loop, so this handler never runs. A target file whose params nest 100 levels is reported as "Not a valid saved recipe: params.x.list.0.list.0...list.0: Recursion error - cyclic reference detected" (772 characters). A JSON file cannot hold a cycle, and the same problem at greater depth gets "Nested too deeply to read." from the json.loads branch. Map a ValidationError that has a recursion_loop error to "Nested too deeply to read." and remove this handler. test_reports_a_document_nested_too_deeply nests only a top-level list, so it reaches only the json.loads branch; add a case that nests about 150 levels in params and checks the message.

varunj-msft added 9 commits November 2, 2026 19:29

FEAT: Report the documents a storage listing cannot use

66fd155

Report non-files and missing containers instead of skipping them

2d8c465

Report an entry that can't be checked instead of failing the whole li…

5dec828

…sting

FEAT: Read saved registry recipes

bf662b7

Limit technique descriptions like the create request and export the r…

cfa4fc0

…ecipe models

Test that a description at the limit is accepted

e45865b

Escape text that isn't UTF-8 in restore reports so they can be sent a…

9a419b4

…s JSON

Escape entry names in restore reports too

274b4ce

Escape report text only when it is sent as JSON

c391b0d

Copilot AI balanced review requested due to automatic review settings November 3, 2026 17:28

Copilot started reviewing on behalf of varunj-msft November 3, 2026 17:29

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

varunj-msft added 6 commits November 3, 2026 21:03

Check the opened file before reading it and skip links deleted since …

507567b

…the listing

Merge the opened-file check

9d37ba3

Close a file that isn't regular before reporting it and report entrie…

a45ac23

…s that can't be checked

Merge the file handle fix

60477e8

Test a file replaced just before it's opened and stop a blocked read …

db245dc

…with a watchdog

Merge the opened-file tests

5d3d2d0

varunj-msft mentioned this pull request Nov 4, 2026

FEAT: Rebuild saved registry entries in dependency order (part 3) #3908

Open

varunj-msft requested review from hannahwestra25 and Richard Lundeen (richlundeen) November 4, 2026 15:32

hannahwestra25 self-assigned this Nov 5, 2026

varunj-msft added 8 commits November 5, 2026 21:25

List presets, suites, and initializers through the checked listing

d1bdd82

Merge the shared document listing

f5ed760

Report a preset, suite, or initializer folder that can't be read as u…

76dbf9c

…navailable storage

Merge the storage error mapping for unreadable folders

f823e4b

Log why storage is unavailable, without its path or URL

7d9ac2f

Merge the storage error logging

7b5bfeb

Merge origin/main

031dc61

Merge origin/main

7f8ab0d

Import the saved recipe models from their package, as main now requires

19205b1

varunj-msft mentioned this pull request Nov 9, 2026

FEAT: Report the documents a storage listing cannot use (part 1) #3891

Open

varunj-msft changed the title ~~FEAT: Read saved registry recipes~~ FEAT: Read saved registry recipes (part 2) Nov 9, 2026

Richard Lundeen (richlundeen) requested changes Nov 9, 2026

View reviewed changes

Richard Lundeen (richlundeen) self-assigned this Nov 9, 2026

varunj-msft added 3 commits November 9, 2026 23:02

Merge the single storage error

1cae0b9

Richard Lundeen (richlundeen) reviewed Nov 9, 2026

View reviewed changes

varunj-msft added 2 commits November 10, 2026 00:17

Merge origin/main

6ee84f0

Merge origin/main

d43a7e8


		model_config = ConfigDict(extra="forbid", strict=True, allow_inf_nan=False, frozen=True)

		schema_version: int = Field(



		# The model keeps the text it was given, so a name still matches the entry it names; only JSON output is escaped.
		_DisplayText = Annotated[str, PlainSerializer(_escape_invalid_text, return_type=str, when_used="json")]


		CURRENT_SCHEMA_VERSION: ClassVar[int] = 1

		model_config = ConfigDict(extra="forbid", strict=True, allow_inf_nan=False, frozen=True)

Conversation

varunj-msft commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Tests and Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Richard Lundeen (richlundeen) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

varunj-msft commented Nov 3, 2026 •

edited

Loading