Skip to content

FEAT: Add Garak exploitation scenario (Jinja template injection + SQL injection echo) - #2576

Open
Manohar Paturi (ManoharPaturi) wants to merge 1 commit into
microsoft:mainfrom
ManoharPaturi:feat/garak-exploitation-scenario
Open

FEAT: Add Garak exploitation scenario (Jinja template injection + SQL injection echo)#2576
Manohar Paturi (ManoharPaturi) wants to merge 1 commit into
microsoft:mainfrom
ManoharPaturi:feat/garak-exploitation-scenario

Conversation

@ManoharPaturi

Copy link
Copy Markdown

Body:

Fixes #2534 (part of #511)

Summary

Adds a PyRIT-native Exploitation scenario packaging garak's two active
server-side exploitation checks — JinjaTemplatePythonInjection and
SQLInjectionEcho from
garak/probes/exploitation.py
as techniques jinja_template_python_injection and sql_injection_echo,
following the same shape as the other garak scenarios (and the open api_key
PR #2538): a _ExploitationDatasetConfiguration plus one standard
PromptSendingAttack per technique, prompt_cap (default 20,
deterministic round-robin), BaselineAttackPolicy.Forbidden, run-size
estimate override, and resume support.

Datasets

Payloads are stored locally under pyrit/datasets/seed_datasets/local/garak/
with per-payload seed metadata (payload, exploit family,
expected_emitted_text, pinned NVIDIA/garak@<commit> provenance; verified
byte-for-byte against upstream commits 560c75a5 / 61ed0b4e) and a
THIRD_PARTY_NOTICE.md mirroring the other garak datasets' Apache-2.0
attribution:

  • exploitation_sql_injection.prompt — 10 SQL payloads
  • exploitation_python_code_execution.prompt — 9 Python payloads (one exact
    upstream duplicate is deduped from the population — AtomicAttack requires
    unique objective hashes, same as Encoding/WebInjection — the dataset keeps
    all 9)
  • exploitation_templates.prompt — echo wrapper + Jinja {{<payload>}}
    wrapper (stored inside a {% raw %} block so seed rendering yields the
    literal {{...}} wrapper)

Success contract

Per the issue, each technique scores with a TrueFalseCompositeScorer (OR)
of payload-specific SubStringScorers — positive when the target emits the
expected payload material. The existing SSTIOutputScorer /
SQLInjectionOutputScorer are wired as opt-in extended_checks=True
auxiliary scorers only, unmodified. Objectives and docs describe emitted
exploit material
, never downstream exploitation.

Docs

doc/scanner/garak.{py,ipynb} gain an Exploitation section with CLI
examples; doc/code/datasets/1_loading_datasets.{py,ipynb} list the new
datasets.

Testing

  • New: tests/unit/scenario/garak/test_exploitation.py (17) +
    tests/unit/datasets/test_garak_exploitation_dataset.py (8) — 25 passed
  • Targeted run (tests/unit/scenario/garak/ + dataset loader/schema/metadata
    • docs-consistency): 270 passed, no network
  • ruff check and ruff format --check clean on all touched files
  • End-to-end smoke with the real local datasets (in-memory DB): prompts
    render ECHO: {{...}} / ECHO: <sql>, default estimate 18 (8+10), SQL
    scorer True on echo / False on refusal, Jinja scorer False on bare payload

Note: the notebook section's saved output is a plausible snapshot — happy to
regenerate or adjust anything the maintainers prefer.

Copilot AI balanced review requested due to automatic review settings September 4, 2026 18:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@ManoharPaturi

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FEAT: Add Garak exploitation scenario

2 participants