Skip to content

FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader#2175

Open
3nesdeniz wants to merge 6 commits into
microsoft:mainfrom
3nesdeniz:feat/agentic-prompt-injection-boundary-dataset
Open

FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader#2175
3nesdeniz wants to merge 6 commits into
microsoft:mainfrom
3nesdeniz:feat/agentic-prompt-injection-boundary-dataset

Conversation

@3nesdeniz

@3nesdeniz 3nesdeniz commented Jul 13, 2026

Copy link
Copy Markdown

Description

Closes #2174.

This PR adds a remote loader for the public CC BY 4.0 Agentic Prompt-Injection Boundary Pairs dataset:

  • 1,200 English records organized as 600 controlled benign/attack pairs;
  • 50 scenario-isolated enterprise and agentic workflows;
  • 12 prompt-injection and trust-boundary attack families;
  • train, validation, test, or combined loading;
  • attack-only default plus benign and full-pair modes;
  • typed filters for label, split, attack family, and source context;
  • pair, scenario, source, boundary, expected-action, version, and revision provenance on each seed.

The loader maps each row to a literal SeedPrompt and pins the immutable v1.0.0 Hugging Face revision. Before filtering, it validates required fields, label-dependent values, allowed families and source contexts, split metadata, duplicate IDs, and complete pair integrity.

Family filtering uses pair_family so full-pair mode preserves both sides of a controlled pair. Agent-security decision labels and attack families remain in metadata rather than being misclassified as PyRIT harm categories. Pair members remain independent evaluation cases; pair_id preserves the relationship without collapsing two prompts into one SeedGroup.

Source and release pipeline: https://github.com/3nesdeniz/agentic-prompt-injection-boundary-pairs

I am the dataset author and maintain the tagged source release. The dataset is synthetic, manually reviewed, and contains no customer or private data.

Validation

Revalidated after merging the current upstream main into this branch:

  • focused loader tests: 26 passed;
  • full unit suite: 11,100 passed, 5 skipped;
  • all targeted pre-commit hooks passed, including Ruff formatting/linting, notebook checks, documentation validation, secret detection, and type checking;
  • strict Jupytext source/notebook comparison passed (only local tool-version metadata was ignored);
  • live loading from the pinned Hugging Face revision returned 1,200 records, 600 complete pairs, balanced labels (600 attack / 600 benign), and the published split counts (840 train / 120 validation / 240 test), with no incomplete pairs;
  • git diff --check passed.

The loader is registered in dataset discovery and documented in the loading guide, paired notebook, and project bibliography.

@3nesdeniz 3nesdeniz changed the title [DRAFT] FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader Jul 16, 2026
@3nesdeniz
3nesdeniz marked this pull request as ready for review July 16, 2026 13:23
@3nesdeniz

Copy link
Copy Markdown
Author

This branch is now synchronized with current main at merge commit 787627f2 and is conflict-free.

Fresh validation on the merged head:

  • focused loader tests: 26 passed;
  • full unit suite: 11,100 passed, 5 skipped;
  • targeted pre-commit suite: all hooks passed;
  • live pinned-release check: 1,200 records, 600 complete pairs, balanced labels, published split counts, no incomplete pairs;
  • Jupytext pair and diff checks passed.

The five GitHub Actions runs are currently action_required, so a maintainer needs to approve the workflows before hosted CI can report. When convenient, could you approve those runs and review the PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader

1 participant