DOC: initializer-rename doc updates and notebook/test fixes (post #2157)#2187
Open
romanlutz wants to merge 32 commits into
Open
DOC: initializer-rename doc updates and notebook/test fixes (post #2157)#2187romanlutz wants to merge 32 commits into
romanlutz wants to merge 32 commits into
Conversation
Keep scenario DTOs typed for CLI output, migrate the custom-target example to first_message, and remove empty per-objective attacks when reconstructing sampled scenario resumes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Repair Python E2E target propagation and notebook examples, correct the Responses vision payload, and isolate and harden frontend E2E validation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Repair scanner isolation, provider resource cleanup, Hugging Face downloads, Windows-safe output, and executable notebook examples discovered by exhaustive post-merge validation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Use the newly configured adversarial_chat target in the benchmark end-to-end smoke test after confirming registration and a clean 7/7 full scenario run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Remove the generic scorer lifecycle method and threshold-wrapper delegation while retaining Azure Content Safety's explicit resource cleanup. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Store the audited execution results for all runnable documentation notebooks so output cells can be reviewed directly in the branch changeset. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Retain Realtime WebSocket cleanup as a public API while moving ordinary OpenAI teardown behind a private hook and removing validation-only cleanup calls from documentation notebooks. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…lidation branch Resolved notebook conflicts in doc/code/converters/1_text_to_text_converters and doc/code/executor/1_single_turn by regenerating executed outputs from the merged .py sources. Verified affected tests (110 passed), full pre-commit, and notebook cell-output audit (0 errors). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Preserve refreshed executed outputs prior to removing split-owned code changes and merging latest main. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Merge latest origin/main (27d20ee). Resolutions: - 26 A-G split-owned code/test files reset to origin/main (now covered by PRs microsoft#2208-2214), so this branch carries no production-code duplication. - matrix_atomic_attack_builder(.py/test) taken from main (orphaned core refactor does not belong on a notebook-validation branch). - Executed notebook .ipynb outputs kept from this branch (regenerated next). Result vs main: notebooks + notebook-source hardening + docs + frontend e2e only. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
The first TextAdaptive() cell in 3_adaptive_scenarios.ipynb runs many objectives across several attempts and exceeds the default 600s per-cell timeout. Give that notebook a 1800s per-cell timeout via a per-notebook override, leaving other scenario notebooks at the 600s default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Sync validation branch with latest main. Resolved doc/scanner/airt.ipynb conflict (both sides changed source) by regenerating the notebook from the merged jupytext-paired airt.py. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Revert 40 notebooks whose cell SOURCE is byte-identical to origin/main (only re-executed LLM outputs / execution_count / metadata differed) plus 2 jupytext-paired .py files that differed only by a jupytext_version header stamp. pyrit/** is identical to origin/main, so these diffs cannot reflect any behavior change. Keeps only files with real source/doc/test changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
These 4 notebooks carried real .ipynb source edits (added Context Compliance section, reworded modality-feedback objective/scorer, heading numbering, garak encoding-doc trim) that were not reflected in their jupytext-paired .py. Regenerated each .py from its .ipynb so the pair is consistent, preserving each file's original jupytext_version stamp so the diff shows only the real source change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Main dropped numeric prefixes from notebook H1 titles. Sync the validation
branch:
- Un-number kept notebooks (real source edits remain):
3_image_converters ('# Image Converters'),
8_non_llm_targets ('# Azure Blob Storage Targets').
- After un-numbering, 1_loading_datasets and 2_openai_responses_target
differ from main only by re-executed output noise, so revert both
pairs (.ipynb + .py) to origin/main; they drop from the changeset.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Remove the DejaVu-Sans font_manager workaround (and the try/except unwrap)
from doc/code/converters/3_image_converters.{ipynb,py}, restoring main's
version. The underlying cause -- AddTextImageConverter defaulting to the
non-cross-platform "helvetica.ttf" -- is being fixed at the source (default
to None -> Pillow bundled default) in a separate session, after which this
notebook runs cleanly on its plain defaults with no font warning.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…nce executor)
The branch added a "Context Compliance" section importing
ContextComplianceAttack from pyrit.executor.attack, plus its supporting
adversarial-target setup and narrative. That class no longer exists in main
(context compliance was migrated out of executor/attack), so the section is
broken against main and the change is unnecessary. Restore main's version of
doc/code/executor/1_single_turn.{ipynb,py}.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…orkaround) 5_workflow.ipynb is explicitly CI-skipped (test_executor_notebooks.py skipped_files) because it needs a live Azure Storage Account + local AI-recruiter service. The branch had added runtime storage_available/recruiter_available guards and re-executed the notebook in an infra-less environment, which replaced main's preserved XPIAStatus.SUCCESS demonstration outputs with 'example not run: unavailable' skip messages. Main's version still resolves against current API, so restore it verbatim. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…fix) The validation branch's 8_modality_feedback carried a simplified 5-criteria objective (plus a text-only profile and max_turns=8) that predated and discarded microsoft#2188 "FIX: Preserve iterative image state in Crescendo". Restore the origin/main version so the notebook keeps the detailed albatross objective, 7-criteria scorer, and iterative-image-state fix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…aults doc) The validation branch's garak notebook trimmed the "Aggregate techniques: ALL / DEFAULT" markdown block that microsoft#2190 "Standardize garak.encoding defaults" added on main. Restore the origin/main version so the DEFAULT/ALL aggregate-technique documentation is preserved. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…soft#2222) The Playwright host-resolution change on this validation branch duplicates and conflicts with PR microsoft#2222 "FIX Stabilize Playwright frontend host resolution", which is the dedicated PR for playwright.config.ts. Revert to origin/main so microsoft#2222 owns that file. (The e2e timing-stabilization tweaks in api.spec.ts / flows.spec.ts are independent and left in place.) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…ion into PR microsoft#2222) api.spec.ts and flows.spec.ts carried e2e timing-stabilization tweaks (explicit /api/health timeouts, a toBeVisible wait, a beforeAll setTimeout). Per decision to consolidate all Playwright e2e stabilization into PR microsoft#2222, revert these to origin/main here; the tweaks are folded into microsoft#2222 alongside the playwright.config.ts host-resolution fix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…ch main) Undo the verbose TargetRegistry.get_registry_singleton().instances.get(...) pattern in round_robin_target, airt, and benchmark; restore main's canonical OpenAIChatTarget() usage. These registry rewrites (and the entangled validation-only bounding tweaks that referenced the registry vars) were scaffolding added during pre-release validation and are not wanted in docs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Undo the validation-only auth workaround (explicit endpoint/model + Entra get_azure_openai_auth fallback). Main's canonical bare OpenAITextEmbedding() resolves to Entra auth when no embedding key is set; the Entra path belongs in the environment (.env), not hardcoded in the doc notebook. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Undo validation-only scaffolding: main lists/looks up scenarios from import-time class registration (no init needed) and builds the target directly via OpenAIChatTarget(), initializing late with just LoadDefaultDatasets(). The added TargetInitializer/ScorerInitializer/ TechniqueInitializer are unused here and only add env-config fragility. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…meout) Restores the unbounded production-default adaptive example to match origin/main; keeps the 1800s per-cell override in test_notebooks_scenarios.py so the auto-run scenarios integration test has headroom instead of the default 600s. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
2_resiliency: drop unused Target/Scorer/Technique initializers (same scaffolding reverted in 1_class_registry). 8_non_llm_targets: restore AZURE_SQL backend and the unconditional Azure Blob example, dropping the storage-availability gating workaround. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
1_pyrit_scan: drop dynamic-port + explicit config-path scaffolding. 10_1_playwright_target: drop Windows sys.platform gating (skip-listed, not auto-run). 4_openai_video_target: drop unneeded scorer/temp-file cleanup + var extraction and the +10313-line re-executed output committed during validation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Integrates microsoft#2169 (ImageResizingConverter clamp) and microsoft#2186 (Scenarios/Technique Registry alignment). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Readers may not know what 'the initializer cleanup' refers to; frame it as the consolidation/rename leading up to v1.0.0 so pre-v1.0.0 configs know when the old names (simple/airt/scenario_technique/objective_list) were removed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
romanlutz
changed the title
romanlutz
marked this pull request as ready for review
July 20, 2026 02:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Small, independent documentation and test-robustness fixes surfaced during pre-release (v1.0.0) validation after #2157. No
pyrit/**production code is touched.Docs — initializer rename (v1.0.0):
doc/getting_started/pyrit_conf.md: add a migration note that the default initializers were consolidated and renamed leading up to v1.0.0, so pre-v1.0.0 configs know the removed names (simple,airt,scenario_technique,objective_list) must be replaced.doc/gui/0_gui.md: update a stale target auto-populate reference from the removedairtinitializer totarget.doc/getting_started/configuration.md: correct the in-memory setup blurb — it mentioned "converter/scorer config" although no converter initializer is configured; it now describes the targets and scorers registered for selection by name or tag.Docs — notebooks:
doc/code/setup/1_configuration: remove a malformed duplicate JupyText header block embedded mid-file and move theoutput_attack_asyncimport into the main imports cell.doc/code/targets/use_huggingface_chat_target: switch the demo model to the smallerHuggingFaceTB/SmolLM2-135M-Instruct(faster to download/run in the auto-executed integration test) and remove thetry/exceptthat swallowed model errors, so failures surface.Tests:
tests/integration/scenarios/test_notebooks_scenarios.py: add a per-notebook cell-timeout override (1800s) for3_adaptive_scenarios.ipynb, whose single cell runs many objectives across several attempts and exceeds the default 600s.Tests and Documentation
use_huggingface_chat_target.ipynbis auto-executed bytests/integration/targets/test_notebooks_targets.py; the smaller model keeps it within the 600s per-cell budget while still exercising a real HuggingFace target end-to-end.3_adaptive_scenarios.ipynbis auto-executed bytests/integration/scenarios/test_notebooks_scenarios.py; the new 1800s override prevents a false timeout on its long multi-attempt cell..py/.ipynbpairs are kept in sync per the repo convention.main.Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448