Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
c4f3f12
FIX: Repair integration notebook regressions
Copilot Jul 12, 2026
7ab6e47
FIX: Repair post-merge validation regressions
Copilot Jul 13, 2026
11b3a56
FIX: Repair notebook validation regressions
Copilot Jul 14, 2026
9ea93e1
TEST: Restore dedicated benchmark target
Copilot Jul 14, 2026
feb44f5
FIX: Keep scorer cleanup concrete
Copilot Jul 14, 2026
32cf9d0
DOC: Refresh executed notebook outputs
Copilot Jul 14, 2026
c446f64
FIX: Keep OpenAI target cleanup private
Copilot Jul 14, 2026
e78f544
Merge latest main into validation branch
Copilot Jul 14, 2026
e377ca8
Merge latest main (#2178 FlipAttack->flip technique) into validation …
Copilot Jul 14, 2026
8146a05
Merge remote-tracking branch 'origin/main' into romanlutz-validate-pr…
Copilot Jul 14, 2026
ed80c6c
DOC: Checkpoint executed notebook outputs before split cleanup
Copilot Jul 17, 2026
4e2a3d2
Merge origin/main into validation branch; drop split-owned code
Copilot Jul 17, 2026
35733e4
MAINT: Extend adaptive scenario notebook cell timeout
Copilot Jul 17, 2026
fb4fb5c
Merge origin/main into romanlutz-validate-pr-2157
Copilot Jul 18, 2026
56fddc9
Drop re-executed notebook output noise from validation branch
Copilot Jul 18, 2026
65e69bd
Sync paired .py sources for 4 notebooks with their .ipynb edits
Copilot Jul 18, 2026
78f6824
DOC: un-number notebook titles to match main
Copilot Jul 18, 2026
5710ca8
DOC: revert 3_image_converters font workaround to match main
Copilot Jul 19, 2026
70b53ed
DOC: revert 1_single_turn to match main (drop removed Context Complia…
Copilot Jul 19, 2026
a2bf536
DOC: revert 5_workflow to match main (drop unnecessary infra-gating w…
Copilot Jul 19, 2026
eb284d4
DOC: revert 8_modality_feedback to match main (preserve #2188 fix)
Copilot Jul 19, 2026
637377b
DOC: revert garak to match main (preserve #2190 encoding-defaults doc)
Copilot Jul 19, 2026
da68308
TEST: revert frontend/playwright.config.ts to main (owned by PR #2222)
Copilot Jul 19, 2026
09b57bd
TEST: revert frontend e2e specs to main (consolidating e2e stabilizat…
Copilot Jul 19, 2026
dd3693c
DOC: revert registry-target rewrites to plain OpenAIChatTarget() (mat…
Copilot Jul 19, 2026
9abfb8c
DOC: revert embeddings notebook to match main (bare OpenAITextEmbedding)
Copilot Jul 19, 2026
70c36d4
DOC: revert 1_class_registry to match main (drop unused initializers)
Copilot Jul 19, 2026
aac7301
DOC: revert 3_adaptive_scenarios notebook to main (keep 1800s test ti…
Copilot Jul 19, 2026
c4b7042
DOC: revert 2_resiliency and 8_non_llm_targets notebooks to main
Copilot Jul 20, 2026
56c1075
DOC: revert scanner, playwright, and video target notebooks to main
Copilot Jul 20, 2026
dd50163
Merge origin/main into romanlutz-validate-pr-2157
Copilot Jul 20, 2026
bea44f8
DOC: anchor initializer-rename note to the v1.0.0 release
Copilot Jul 20, 2026
ce33b0c
Update pyrit_conf.md
romanlutz Jul 20, 2026
7ce7747
DOC: remove stale simple initializer from scanner example (post #2157)
Copilot Jul 20, 2026
2ce48bf
DOC: replace stale `simple` initializer example in registry docstring…
Copilot Jul 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
287 changes: 148 additions & 139 deletions doc/code/setup/1_configuration.ipynb

Large diffs are not rendered by default.

13 changes: 1 addition & 12 deletions doc/code/setup/1_configuration.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,6 @@
# jupytext_version: 1.19.4
# ---

# %%
from pyrit.output import output_attack_async

# ---
# jupyter:
# jupytext:
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.1
# ---
# %% [markdown]
# # Configuration
#
Expand Down Expand Up @@ -148,6 +136,7 @@
AttackScoringConfig,
PromptSendingAttack,
)
from pyrit.output import output_attack_async
from pyrit.prompt_normalizer.converter_configuration import (
ConverterConfiguration,
)
Expand Down
105 changes: 42 additions & 63 deletions doc/code/targets/use_huggingface_chat_target.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -41,14 +41,6 @@
"id": "1",
Comment thread
romanlutz marked this conversation as resolved.
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"./git/copilot-worktrees/PyRIT/romanlutz-cautious-meme/.venv/Lib/site-packages/confusables/__init__.py:46: SyntaxWarning: \"\\*\" is an invalid escape sequence. Such sequences will not work in the future. Did you mean \"\\\\*\"? A raw string is also an option.\n",
" space_regex = \"[\\*_~|`\\-\\.]*\" if include_character_padding else ''\n"
]
},
{
"name": "stdout",
"output_type": "stream",
Expand All @@ -69,13 +61,13 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Running model: Qwen/Qwen2-0.5B-Instruct\n"
"Running model: HuggingFaceTB/SmolLM2-135M-Instruct\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2cb1ed2b5d6c4fa98456d1c217564010",
"model_id": "5ccc9dc2fe35480bbe25f5b8fd6bd50a",
"version_major": 2,
"version_minor": 0
},
Expand All @@ -89,12 +81,12 @@
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "76523dac8fe04b87921ad444853385c3",
"model_id": "3fc4b0633f704408aa0ca3a28d38bc1e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Fetching 10 files: 0%| | 0/10 [00:00<?, ?it/s]"
"Fetching 25 files: 0%| | 0/25 [00:00<?, ?it/s]"
]
},
"metadata": {},
Expand All @@ -103,12 +95,12 @@
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "650a316e795b4e84abdc46b06bc79cc1",
"model_id": "73eb775a3c3a419fbc79d3f3ce7d96ef",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Loading weights: 0%| | 0/290 [00:00<?, ?it/s]"
"Loading weights: 0%| | 0/272 [00:00<?, ?it/s]"
]
},
"metadata": {},
Expand All @@ -118,7 +110,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Average response time for Qwen/Qwen2-0.5B-Instruct: 7.43 seconds\n",
"Average response time for HuggingFaceTB/SmolLM2-135M-Instruct: 2.59 seconds\n",
"\n",
"\n",
"\u001b[33m════════════════════════════════════════════════════════════════════════════════════════════════════\u001b[0m\n",
Expand All @@ -130,11 +122,11 @@
"\u001b[1m 📋 Basic Information\u001b[0m\n",
"\u001b[36m • Objective: What is 3*3? Give me the solution.\u001b[0m\n",
"\u001b[36m • Attack Type: PromptSendingAttack\u001b[0m\n",
"\u001b[36m • Conversation ID: 3e01b421-e41b-4c47-a51b-ea192f7cdb24\u001b[0m\n",
"\u001b[36m • Conversation ID: 84048a6c-960d-483d-b54d-b19d8e0e3eb1\u001b[0m\n",
"\n",
"\u001b[1m ⚡ Execution Metrics\u001b[0m\n",
"\u001b[32m • Turns Executed: 1\u001b[0m\n",
"\u001b[32m • Execution Time: 12.20s\u001b[0m\n",
"\u001b[32m • Execution Time: 4.61s\u001b[0m\n",
"\n",
"\u001b[1m 🎯 Outcome\u001b[0m\n",
"\u001b[33m • Status: ❓ UNDETERMINED\u001b[0m\n",
Expand All @@ -151,14 +143,12 @@
"\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
"\u001b[1m\u001b[33m🔸 ASSISTANT\u001b[0m\n",
"\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
"\u001b[33m The answer to \"What is 3 * 3?\" is 9.\u001b[0m\n",
"\u001b[33m \u001b[0m\n",
"\u001b[33m This is because the product of two numbers, where the first number (\u001b[0m\n",
"\u001b[33m 3*3 = 9\u001b[0m\n",
"\n",
"\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
"\n",
"\u001b[2m\u001b[37m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
"\u001b[2m\u001b[37m Report generated at: 2026-07-16 05:18:39 UTC \u001b[0m\n",
"\u001b[2m\u001b[37m Report generated at: 2026-07-13 23:25:00 UTC \u001b[0m\n",
"\n",
"\u001b[33m════════════════════════════════════════════════════════════════════════════════════════════════════\u001b[0m\n",
"\u001b[1m\u001b[33m ❓ ATTACK RESULT: UNDETERMINED ❓ \u001b[0m\n",
Expand All @@ -169,11 +159,11 @@
"\u001b[1m 📋 Basic Information\u001b[0m\n",
"\u001b[36m • Objective: What is 4*4? Give me the solution.\u001b[0m\n",
"\u001b[36m • Attack Type: PromptSendingAttack\u001b[0m\n",
"\u001b[36m • Conversation ID: 76b845b6-0e72-4a80-a793-09652c4d7405\u001b[0m\n",
"\u001b[36m • Conversation ID: bc388b83-e4d3-441b-80f1-dc8d433bc314\u001b[0m\n",
"\n",
"\u001b[1m ⚡ Execution Metrics\u001b[0m\n",
"\u001b[32m • Turns Executed: 1\u001b[0m\n",
"\u001b[32m • Execution Time: 2.63s\u001b[0m\n",
"\u001b[32m • Execution Time: 557ms\u001b[0m\n",
"\n",
"\u001b[1m 🎯 Outcome\u001b[0m\n",
"\u001b[33m • Status: ❓ UNDETERMINED\u001b[0m\n",
Expand All @@ -190,17 +180,13 @@
"\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
"\u001b[1m\u001b[33m🔸 ASSISTANT\u001b[0m\n",
"\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
"\u001b[33m The result of multiplying 4 by itself four times is:\u001b[0m\n",
"\u001b[33m 256.\u001b[0m\n",
"\u001b[33m \u001b[0m\n",
"\u001b[33m Here's why:\u001b[0m\n",
"\u001b[33m First, we multiply 4 and\u001b[0m\n",
"\u001b[33m 4*4 = 16.\u001b[0m\n",
"\n",
"\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
"\n",
"\u001b[2m\u001b[37m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
"\u001b[2m\u001b[37m Report generated at: 2026-07-16 05:18:39 UTC \u001b[0m\n",
"Qwen/Qwen2-0.5B-Instruct: 7.43 seconds\n"
"\u001b[2m\u001b[37m Report generated at: 2026-07-13 23:25:00 UTC \u001b[0m\n",
"HuggingFaceTB/SmolLM2-135M-Instruct: 2.59 seconds\n"
]
}
],
Expand All @@ -218,7 +204,7 @@
"await initialize_pyrit_async(memory_db_type=IN_MEMORY) # type: ignore\n",
"\n",
"# models to test\n",
"model_id = \"Qwen/Qwen2-0.5B-Instruct\"\n",
"model_id = \"HuggingFaceTB/SmolLM2-135M-Instruct\"\n",
"\n",
"# List of prompts to send\n",
"prompt_list = [\"What is 3*3? Give me the solution.\", \"What is 4*4? Give me the solution.\"]\n",
Expand All @@ -228,51 +214,44 @@
"\n",
"print(f\"Running model: {model_id}\")\n",
"\n",
"try:\n",
" # Initialize HuggingFaceChatTarget with the current model\n",
" target = HuggingFaceChatTarget(model_id=model_id, use_cuda=False, tensor_format=\"pt\", max_new_tokens=30)\n",
"\n",
" # Initialize the attack\n",
" attack = PromptSendingAttack(objective_target=target)\n",
"# Initialize HuggingFaceChatTarget with the current model\n",
"target = HuggingFaceChatTarget(model_id=model_id, use_cuda=False, tensor_format=\"pt\", max_new_tokens=30)\n",
"\n",
" # Record start time\n",
" start_time = time.time()\n",
"# Initialize the attack\n",
"attack = PromptSendingAttack(objective_target=target)\n",
"\n",
" # Send prompts asynchronously\n",
" responses = await AttackExecutor().execute_attack_async( # type: ignore\n",
" attack=attack,\n",
" objectives=prompt_list,\n",
" )\n",
"# Record start time\n",
"start_time = time.time()\n",
"\n",
" # Record end time\n",
" end_time = time.time()\n",
"# Send prompts asynchronously\n",
"responses = await AttackExecutor().execute_attack_async( # type: ignore\n",
" attack=attack,\n",
" objectives=prompt_list,\n",
")\n",
"\n",
" # Calculate total and average response time\n",
" total_time = end_time - start_time\n",
" avg_time = total_time / len(prompt_list)\n",
" model_times[model_id] = avg_time\n",
"# Record end time\n",
"end_time = time.time()\n",
"\n",
" print(f\"Average response time for {model_id}: {avg_time:.2f} seconds\\n\")\n",
"# Calculate total and average response time\n",
"total_time = end_time - start_time\n",
"avg_time = total_time / len(prompt_list)\n",
"model_times[model_id] = avg_time\n",
"\n",
" # Print the conversations\n",
" for result in responses:\n",
" await output_attack_async(result)\n",
"print(f\"Average response time for {model_id}: {avg_time:.2f} seconds\\n\")\n",
"\n",
"except Exception as e:\n",
" print(f\"An error occurred with model {model_id}: {e}\\n\")\n",
" model_times[model_id] = None\n",
"# Print the conversations\n",
"for result in responses:\n",
" await output_attack_async(result)\n",
"\n",
"# Print the model average time\n",
"if model_times[model_id] is not None:\n",
" print(f\"{model_id}: {model_times[model_id]:.2f} seconds\")\n",
"else:\n",
" print(f\"{model_id}: Error occurred, no average time calculated.\")"
"print(f\"{model_id}: {model_times[model_id]:.2f} seconds\")"
]
}
],
"metadata": {
"jupytext": {
"cell_metadata_filter": "-all"
"cell_metadata_filter": "-all",
"main_language": "python"
},
"language_info": {
"codemirror_mode": {
Expand All @@ -284,7 +263,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.14.4"
"version": "3.12.12"
}
},
"nbformat": 4,
Expand Down
57 changes: 25 additions & 32 deletions doc/code/targets/use_huggingface_chat_target.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.0
# jupytext_version: 1.19.4
# ---

# %% [markdown]
# # HuggingFace Chat Target - optional
#
Expand Down Expand Up @@ -50,7 +51,7 @@
await initialize_pyrit_async(memory_db_type=IN_MEMORY) # type: ignore

# models to test
model_id = "Qwen/Qwen2-0.5B-Instruct"
model_id = "HuggingFaceTB/SmolLM2-135M-Instruct"

# List of prompts to send
prompt_list = ["What is 3*3? Give me the solution.", "What is 4*4? Give me the solution."]
Expand All @@ -60,42 +61,34 @@

print(f"Running model: {model_id}")

try:
# Initialize HuggingFaceChatTarget with the current model
target = HuggingFaceChatTarget(model_id=model_id, use_cuda=False, tensor_format="pt", max_new_tokens=30)

# Initialize the attack
attack = PromptSendingAttack(objective_target=target)
# Initialize HuggingFaceChatTarget with the current model
target = HuggingFaceChatTarget(model_id=model_id, use_cuda=False, tensor_format="pt", max_new_tokens=30)

# Record start time
start_time = time.time()
# Initialize the attack
attack = PromptSendingAttack(objective_target=target)

# Send prompts asynchronously
responses = await AttackExecutor().execute_attack_async( # type: ignore
attack=attack,
objectives=prompt_list,
)
# Record start time
start_time = time.time()

# Record end time
end_time = time.time()
# Send prompts asynchronously
responses = await AttackExecutor().execute_attack_async( # type: ignore
attack=attack,
objectives=prompt_list,
)

# Calculate total and average response time
total_time = end_time - start_time
avg_time = total_time / len(prompt_list)
model_times[model_id] = avg_time
# Record end time
end_time = time.time()

print(f"Average response time for {model_id}: {avg_time:.2f} seconds\n")
# Calculate total and average response time
total_time = end_time - start_time
avg_time = total_time / len(prompt_list)
model_times[model_id] = avg_time

# Print the conversations
for result in responses:
await output_attack_async(result)
print(f"Average response time for {model_id}: {avg_time:.2f} seconds\n")

except Exception as e:
print(f"An error occurred with model {model_id}: {e}\n")
model_times[model_id] = None
# Print the conversations
for result in responses:
await output_attack_async(result)

# Print the model average time
if model_times[model_id] is not None:
print(f"{model_id}: {model_times[model_id]:.2f} seconds")
else:
print(f"{model_id}: Error occurred, no average time calculated.")
print(f"{model_id}: {model_times[model_id]:.2f} seconds")
2 changes: 1 addition & 1 deletion doc/getting_started/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ from pyrit.setup.initializers import ScorerInitializer, TargetInitializer
await initialize_pyrit_async(memory_db_type="InMemory", initializers=[TargetInitializer(), ScorerInitializer()])
```

This gives you an in-memory database and default converter/scorer config — enough to run most notebooks and examples. Replace the endpoint/key/model for your provider (Azure, Ollama, Groq, HuggingFace, etc.).
This gives you an in-memory database with configured targets and scorers registered for selection by name or tag. Replace the endpoint/key/model for your provider (Azure, Ollama, Groq, HuggingFace, etc.).

## For Persistent Setup

Expand Down
4 changes: 4 additions & 0 deletions doc/getting_started/pyrit_conf.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,10 @@ Most users should enable the following initializers. These are what the `.pyrit_
**Execution order follows listing order.** Initializers execute in the order they appear in the config. Ensure dependencies are satisfied — for example, list `target` before `scorer` since scorers need targets to be registered first.
```

```{important}
The default initializers were consolidated and renamed as part of the cleanup leading up to the **v1.0.0** release. Configurations created before v1.0.0 may still reference the removed names `simple`, `airt`, `scenario_technique`, or `scenario_objective_list` — replace them with the current initializers listed above.
```

The recommended config:

```yaml
Expand Down
2 changes: 1 addition & 1 deletion doc/gui/0_gui.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ For `AzureMLChatTarget`, additional fields are available: **Max New Tokens**, **

#### Auto-Populating Targets

Targets can also be auto-populated by adding an initializer (e.g., `airt`) to your `~/.pyrit/.pyrit_conf` file. This reads endpoints from your `.env` and `.env.local` files. See [.pyrit_conf_example](https://github.com/microsoft/PyRIT/blob/main/.pyrit_conf_example) for details.
Targets can also be auto-populated by adding the `target` initializer to your `~/.pyrit/.pyrit_conf` file. This reads endpoints from your `.env` and `.env.local` files. See [.pyrit_conf_example](https://github.com/microsoft/PyRIT/blob/main/.pyrit_conf_example) for details.

---

Expand Down
2 changes: 1 addition & 1 deletion doc/scanner/1_pyrit_scan.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -610,7 +610,7 @@
"\n",
"PyRITInitializers are how you can configure the CLI scanner. PyRIT includes several built-in initializers you can use with the `--initializers` flag.\n",
"\n",
"The `--list-initializers` command shows all available initializers. Initializers are referenced by their filename (e.g., `target`, `scorer`, `simple`) regardless of which subdirectory they're in.\n",
"The `--list-initializers` command shows all available initializers. Initializers are referenced by their filename (e.g., `target`, `scorer`) regardless of which subdirectory they're in.\n",
"\n",
"List the available initializers using the --list-initializers flag."
]
Expand Down
2 changes: 1 addition & 1 deletion doc/scanner/1_pyrit_scan.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@
#
# PyRITInitializers are how you can configure the CLI scanner. PyRIT includes several built-in initializers you can use with the `--initializers` flag.
#
# The `--list-initializers` command shows all available initializers. Initializers are referenced by their filename (e.g., `target`, `scorer`, `simple`) regardless of which subdirectory they're in.
# The `--list-initializers` command shows all available initializers. Initializers are referenced by their filename (e.g., `target`, `scorer`) regardless of which subdirectory they're in.
#
# List the available initializers using the --list-initializers flag.

Expand Down
2 changes: 1 addition & 1 deletion pyrit/registry/components/initializer_registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ class InitializerRegistry(ParamBagRegistry["PyRITInitializer", InitializerMetada
``pyrit/setup/initializers`` directory structure via a filesystem scan (so
``_discover`` is overridden rather than supplying ``_base_type`` /
``_discovery_package``). Initializers are identified by their suffix-stripped
snake_case class name (e.g., ``"objective_target"``, ``"simple"``); the
snake_case class name (e.g., ``"objective_target"``, ``"technique"``); the
directory structure is used for organization but not exposed to users.
"""

Expand Down
Loading