fix(ai-red-teaming): normalize agent tool_calls in generated targets - #151
Merged
Merged
Conversation
Generated agent targets returned tool_calls verbatim. Agents that emit
{"tool": ..., "arguments": {dict}} (a common non-OpenAI shape) produced empty
name fields and dict arguments, so the SDK scorers' _extract_tool_calls
returned [] and every agentic tool scorer (any_tool_invoked,
tool_selection_safety, dangerous_tool_args) silently saw no tool use against
agents that were in fact calling dangerous tools. Emit a _normalize_tool_calls
helper into the generated script and apply it in both agent-target builders so
tool_calls are always [{name, arguments:str}]. Bump 1.17.0 -> 1.17.1.
Found via a TUI coverage sweep across all 10 agentic attack categories.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Generated agent targets (
_build_agent_target_codeand the ATLAS multi-agenttarget builder) returned
tool_callsverbatim. Agents that emit{"tool": ..., "arguments": {dict}}(a common non-OpenAI shape) produced emptynamefields and dictarguments, so the SDK scorers'_extract_tool_callsreturned
[]. Result:any_tool_invoked,tool_selection_safety, anddangerous_tool_argssilently reported no tool use against agents that werein fact calling
execute_command/send_emailon nearly every probe.Found via a TUI coverage sweep across all 10 agentic attack categories: 10/10
ran end to end, but every trace showed
tool_calls: []despite confirmed RCEand exfil.
Fix
_normalize_tool_callshelper into the generated workflow script.tool_callsis always[{name, arguments:str}], accepting thetool/argsaliases and OpenAInested
functionshape, and serializing dict arguments to JSON.1.17.0 -> 1.17.1(published versions are immutable).This pairs with the SDK-side hardening in dreadnode-tiger#2565 (scorer
_extract_tool_callstolerance); normalizing at the target makes agenticscoring correct regardless of the bundled SDK version.
Validation
TestAgentToolCallNormalization(3 tests) -> pass.test_attack_runner.py: 158 passed, 1 pre-existing unrelated failure(
TestGenerateMultimodalAttack::test_generates_compiling_script_with_image,fails identically on clean
main).