Add a PUZZLED converter: word-puzzle jailbreak (arXiv:2508.01306)

[shashank03-dev]

Is your feature request related to a problem? Please describe.

PyRIT has 87 converters today, but none of them hide a harmful request inside a word puzzle that the model has to solve first. That is a real gap, because a recent paper ("PUZZLED", arXiv:2508.01306, August 2025) reports this approach clearly beating CodeChameleon, which PyRIT already ships as a converter.

The idea is simple. You take the sensitive words out of a prompt, turn them into a word search, an anagram, or a crossword, and ask the model to solve the puzzle and then answer the reconstructed request. The harmful words never appear in plain text, so the usual keyword-style checks have nothing to catch. The model does the reassembly itself.

What makes it worth adding: the attack gets stronger as models get smarter, because a better reasoner solves the puzzle more reliably. That is the opposite of most encoding tricks, which fade as models improve. For a red-teaming tool meant to stay useful against future models, that property matters.

Describe the solution you'd like

Add PUZZLED to PyRIT. In plain terms, the converter would:

1. Pick the sensitive words in the prompt to hide (rule-based, using part-of-speech tagging).
2. Replace them with placeholders like [WORD1], [WORD2].
3. Encode the hidden words as one of three puzzles: word search, anagram, or crossword.
4. Add a short clue for each hidden word (length, part of speech, and a plain-language hint).
5. Assemble the final prompt: the puzzle, the clues, and instructions to solve it, rebuild the original sentence, and answer it.

One design question I'd like your steer on before I build anything, since your contributing guide asks to discuss research first (doc/contributing/2_incorporating_research.md):

• Converter or attack strategy? The puzzle transform fits pyrit/converter/ cleanly. But the paper also generates the semantic clue for each word using a helper model (GPT-4o in the paper). Following your guidance that attacks should stay generic, I'd make that clue step use a generic PromptChatTarget rather than hardcoding a model, and I'd make it optional so there's also a fully offline path (length and part-of-speech clues only). If you'd rather see this as an attack strategy in pyrit/executor/attack/, or split into a converter plus a clue helper, I'll follow your call.

I'm offering to implement it fully: the converter, the three puzzle builders, unit tests to your diff-coverage bar, and docs.

Describe alternatives you've considered, if relevant

• WordGame (arXiv:2405.14023) and QueryAttack (arXiv:2502.09723) are related "disguise the request" techniques, but neither reports results as strong as PUZZLED, and neither has the "improves with model capability" property.
• DrAttack (arXiv:2402.16914) is already in PyRIT as decomposition_converter. It splits a prompt by meaning; PUZZLED hides words as puzzles. Different mechanism, so this isn't a duplicate.

Additional context

Paper: PUZZLED: Jailbreaking LLMs through Word-Based Puzzles, Yelim Ahn and Jaejin Lee, arXiv:2508.01306 (https://arxiv.org/abs/2508.01306).

Reported results (attack success rate on AdvBench, judged by GPT-4o at a score of 7 or higher):

Model	ASR
GPT-4.1	96.5%
Gemini 2.0 Flash	94.4%
Claude 3.7 Sonnet	92.3%
GPT-4o	86.7%
LLaMA 3.1 8B	74.2%
Average	88.8%

For comparison, the paper reports CodeChameleon (already a converter in PyRIT) at 44.2% average under the same setup. PUZZLED nearly doubles it.

Two honest caveats:

• The clue step uses a helper model, so the technique is not fully offline. Clues are cached and reused, so the calls are limited, and the puzzle building itself is deterministic and easy to test.
• The authors did not release code, so I'd implement from the paper. The exact prompt wording lives in the paper's appendix figures, which I'd follow closely.

Happy to adjust scope based on your feedback.