Skip to content

Add XL-SafetyBench datasets#1791

Merged
romanlutz merged 30 commits into
microsoft:mainfrom
romanlutz:romanlutz/explore-benchmark-integration
Jul 20, 2026
Merged

Add XL-SafetyBench datasets#1791
romanlutz merged 30 commits into
microsoft:mainfrom
romanlutz:romanlutz/explore-benchmark-integration

Conversation

@romanlutz

@romanlutz romanlutz commented May 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the XL-SafetyBench seed datasets to PyRIT. The benchmark contains 5,500 native-validated prompts across 10 country-language pairs and covers country-grounded jailbreak safety plus cultural sensitivity.

This PR is intentionally dataset-only. The benchmark-specific judges have been moved to a separate stacked follow-up.

What's in this PR

Three logical providers under pyrit/datasets/seed_datasets/remote/xl_safety_bench_dataset.py:

Provider Emits Size Description
_XLSafetyBenchJailbreakDataset SeedPrompt 4,500 Country-grounded adversarial attack prompts in five harm categories
_XLSafetyBenchJailbreakObjectivesDataset SeedObjective ~1,500 Deduplicated harmful goals behind the attack prompts, for use with PyRIT attack strategies
_XLSafetyBenchCulturalDataset SeedPrompt 1,000 Innocuous scenarios embedding local taboos or sensitivities in six categories

Each loader accepts countries: list[XLSafetyBenchCountry] | None; None loads all 10 countries, while callers can provide one or several enum values to select only those country files. This avoids separate per-country provider classes while retaining single-country and multi-country loading.

Shared enums cover the 10 countries, five jailbreak categories, six cultural categories, and language mode. Per-row provenance is preserved in metadata, including country/language display information, base queries, localized scenarios, and hidden violations.

Documentation

doc/code/datasets/1_loading_datasets cites XL-SafetyBench and its executed output lists the three logical benchmark providers.

Tests

tests/unit/datasets/test_xl_safety_bench_dataset.py covers all-country defaults, single- and multi-country filtering, country deduplication, objectives, category filtering, language selection, metadata propagation, schema validation, UTF-8 BOM normalization, and the three-name registry contract.

Notes for reviewers

  • The upstream CSVs include a UTF-8 BOM on the id column; rows are normalized before field access so IDs and prompt names are retained.
  • The jailbreak prompts loader has the default tag. Objectives and cultural loaders intentionally omit it to avoid duplicate/default sweep coverage.
  • Cultural scenarios default to the local language and can also be loaded in English.
  • CC-BY-4.0 attribution and the paper citation are preserved.
  • The benchmark carries a content warning, consistent with similar safety datasets in PyRIT.

Deferred follow-ups

  • XL-SafetyBench jailbreak and cultural judge scorers (separate stacked session)
  • Benchmark scenarios
  • Example notebook

Verification

  • 79 focused dataset and registry metadata tests pass
  • The loading-datasets notebook executes without errors, renders exactly the three logical XL-SafetyBench providers, and passes strict Jupytext round-trip validation
  • Repository pre-commit hooks pass, including notebook sanitizers, Ruff, ty, and documentation checks

romanlutz and others added 8 commits May 22, 2026 22:26
Implements Phase 1 (seed datasets) and Phase 2 (judge scorers) of
XL-SafetyBench (arXiv:2605.05662), a country-grounded cross-cultural
benchmark for LLM safety and cultural sensitivity (CC-BY-4.0,
AIM-Intelligence/XL-SafetyBench on HuggingFace).

Datasets (pyrit/datasets/seed_datasets/remote/xl_safety_bench_dataset.py):
- _XLSafetyBenchJailbreakDataset (4,500 prompts, 5 categories, 10 countries)
- _XLSafetyBenchCulturalDataset (1,000 prompts, 6 categories, 10 countries)
  - language_mode='local'|'english' selects scenario language
  - hidden_violation preserved in metadata for the cultural judge
- Shared enums: XLSafetyBenchCountry, XLSafetyBenchJailbreakCategory,
  XLSafetyBenchCulturalCategory

Judges (pyrit/score/true_false/):
- XLSafetyBenchJailbreakJudgeScorer: 4-criterion judge (H/NC/S/R),
  produces a label (Critical/Direct/Soft Unsafe, Ideal/Neutral Safe)
  with the booleans in score_metadata for downstream ASR/NSR aggregation
- XLSafetyBenchCulturalJudgeScorer: relevance + cultural recognition;
  CSR truthy when the response flags the embedded sensitivity
- Judge prompts ported verbatim from the paper's GitHub repo and stored
  as companion .txt files for readability

Tests: 34 new unit tests (17 dataset + 17 scorer) covering filters,
metadata propagation, language modes, malformed-JSON retries, and the
full jailbreak label / cultural matrix.

Phase 3 (Scenarios) and Phase 4 (example notebook) deferred to follow-ups.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Following the sibling-loaders pattern (each distinct upstream artifact
gets its own registered SeedDatasetProvider subclass so the daily e2e
matrix exercises every URL), split both XL-SafetyBench tracks into
country-pinned subclasses:

- 10 jailbreak siblings (one per country, pinning `countries=[X]`,
  `size="medium"`, inheriting the parent's "default" tag)
- 10 cultural siblings (same shape, `language_mode` remains a kwarg
  since it selects a column in the same CSV)

The parent loaders are unchanged; the kwarg-based API still works.

Also fixes the cultural parent's size bucket: 1,000 prompts falls in
the "large" (500-4999) range, not "medium".

Added 22 unit tests covering dataset_name, country pinning, tag
inheritance, and kwarg forwarding (categories + language_mode) for the
siblings.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@romanlutz
romanlutz marked this pull request as ready for review May 28, 2026 05:21
romanlutz and others added 15 commits May 30, 2026 18:47
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…vely

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…chmark-integration

# Conflicts:
#	doc/bibliography.md
…ch loader

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…precated shim)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…chmark-integration

# Conflicts:
#	doc/bibliography.md
#	pyrit/score/__init__.py
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 68ca6a05-cfaf-4967-839a-907baa2b937f
Keep PR microsoft#1791 focused on dataset loaders and move the scorer layer to a stacked follow-up session.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 68ca6a05-cfaf-4967-839a-907baa2b937f
@romanlutz romanlutz changed the title Add XL-SafetyBench datasets and judges Add XL-SafetyBench datasets Jul 17, 2026
Copilot AI added 5 commits July 17, 2026 07:07
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 68ca6a05-cfaf-4967-839a-907baa2b937f
Add the XL-SafetyBench citation and regenerate executed output so all benchmark dataset providers appear in the rendered list.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 68ca6a05-cfaf-4967-839a-907baa2b937f
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 68ca6a05-cfaf-4967-839a-907baa2b937f
Use the existing countries argument on each logical dataset instead of registering 20 country-specific sibling classes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 68ca6a05-cfaf-4967-839a-907baa2b937f
Resolve dataset registry, bibliography, and loading notebook conflicts while preserving WildGuardMix and XL-SafetyBench entries.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 68ca6a05-cfaf-4967-839a-907baa2b937f
@hannahwestra25 hannahwestra25 self-assigned this Jul 20, 2026
Comment thread doc/references.bib Outdated

@hannahwestra25 hannahwestra25 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor comment related to the title but lgtm otherwise!

Copilot AI and others added 2 commits July 20, 2026 15:35
@romanlutz
romanlutz enabled auto-merge July 20, 2026 22:46
@romanlutz
romanlutz disabled auto-merge July 20, 2026 23:03
@romanlutz
romanlutz enabled auto-merge July 20, 2026 23:03
@romanlutz
romanlutz added this pull request to the merge queue Jul 20, 2026
Merged via the queue into microsoft:main with commit 8bae485 Jul 20, 2026
53 checks passed
@romanlutz
romanlutz deleted the romanlutz/explore-benchmark-integration branch July 20, 2026 23:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants