Repository navigation
feat: add PI114-PI118 multilingual evasion patterns - #197
hermanngeorge15 merged 7 commits into
Conversation
hermanngeorge15
left a comment
There was a problem hiding this comment.
This is a strong PR and the bookkeeping is unusually complete — catalogue regenerated, a clean-corpus specimen added, relaxed_pattern present on all five (required for PI050+), and every number moved in step: 79 -> 84 patterns, multilingual 4 -> 9, recall 8/10 -> 10/10, total 103/115 -> 105/115 (91.3%). I checked that arithmetic and it holds.
The part I most want to keep is the caveat you added to the README: that the recall row measures only the original Czech/German corpus and is not eleven-language recall. That is exactly how CAT-03 reports its number, and it is the right instinct.
I also mutation-checked the false-positive controls on PI114, PI116 and PI117 — in each case the shipped pattern misses its counter_example while the relaxed form catches it, so GATE-05 is load-bearing here rather than decorative.
Two things to fix before this merges.
1. (blocking) PI118's description makes the scanner flag its own catalogue, at HIGH.
description: "Multilingual claim that developer mode is enabled"That English sentence is itself a PI030 match. In isolation:
$ echo 'Multilingual claim that developer mode is enabled' | injection-scanner check -
[{"pattern_id":"PI030","severity":"HIGH"}]
It renders into docs/PATTERN-CATALOGUE.md:1919, and the whole-repo self-scan goes from 2 findings on main to 3 on this branch — the new one being PI030 at that line. This is the "the scanner flags its own documentation" regression called out in .claude/skills/pattern-library, and HIGH is the tier install-hook blocks commits at, so it can block a contributor's commit.
Rewording the description is enough — e.g. "Multilingual claim that a developer mode has been switched on" — or put the phrase in a code span.
2. (blocking) .github/code-scanning-baseline.json needs regenerating.
This PR changes patterns/ and adds payloads to examples/multilingual-attack.md. Both .github/workflows/code-scanning.yml (lines 82-86) and the pattern-library checklist require the baseline to be regenerated when either changes. It will not fail CI — that job emits an alert list rather than a pass/fail gate — but it will raise unreviewed alerts in the security tab.
cargo run --release -- check . --exclude '.planning/**' \
--write-baseline .github/code-scanning-baseline.jsonNon-blocking notes:
- The
relaxed_patterns strip down to a bare verb alternation (PI114's is(?:ignora|ignore|ignoriere|...)). That proves some narrowing exists, but not that the intended discriminator — the object clause — is the thing doing the work. Relaxing only the object would make the control sharper. README.mdanddocs/DETECTION-BACKLOG.mdadd payloads in double quotes inside markdown tables. They stay silent on the table confidence downgrade rather than on correctness, andDETECTION-BACKLOG.mdis the file that caused this exact problem twice before. Backticks would be safer.- The
(?x)plus YAML>-folding combination is genuinely clever — folded newlines become spaces, which(?x)then ignores — and it is not obvious to a future reader. Worth one comment line so nobody "simplifies" it into a broken regex. - On "native-speaker review is welcome": agreed, and I cannot provide it for Spanish, Arabic, Hindi, Korean or Japanese. I verified the regexes mechanically — that they compile, match their examples, and miss their counter-examples — but not that these are the phrasings an attacker would actually write. I will open a separate issue asking for that review rather than holding this PR on it.
CI is red on PERF-01 only, which is a pre-existing flake on main and not caused by this PR. See #188 — I am fixing the gate first.
Note for sequencing: this PR and #191 both touch tests/recall_test.rs, tests/pattern_test.rs and tests/markdown_context_test.rs. I plan to merge #191 first, so this will need a rebase — and tests/multilingual_test.rs should then use #191's common::scanner() rather than building its own.
|
Status update so the red badge on this PR isn't misleading you. The PERF-01 gate is fixed on Your PR's current FAILURE predates that fix. Rebase onto Also merged in the meantime, both of which touch files this PR touches:
The suite is now ~3 minutes locally, down from ~11, so your own iteration loop should be much faster. The two blocking items from my review are unchanged: PI118's |
|
Updated PR #197 with both blockers fixed: PI118’s description no longer triggers PI030, and the code-scanning baseline is regenerated. The branch now includes current main, uses common::scanner(), and addresses the relaxed-control and documentation notes. Added regression checks; local validation passes apart from the known Windows-only assertion. CI is rerunning. |
|
Re-reviewed — both requested changes verified, by re-running the gate rather than reading the diff. Your write-up made that fast to check, and every number in it held. (a) The PI118 self-match is fixed at the source. (b) The baseline is correctly regenerated. 360 → 376 entries, exactly the +16 you describe. The specific thing I checked is that you didn't take the easy route: the catalogue's baseline entries are identical to Also confirmed independently:
On the skipped assertion: This is accepted on correctness. I am holding the merge briefly behind a release announcement that quotes the current 79-pattern count — purely a publishing-sequence thing on my side, with nothing further needed from you. I am leaving the formal "changes requested" state as it is only so that nothing merges ahead of that; please don't read it as another round of work. Thanks for the care on this one, and for the CI-flake diagnosis on #188 that sized the new budget. |
hermanngeorge15
left a comment
There was a problem hiding this comment.
Both requested changes verified by running the gates, not reading the diff: the PI118 description no longer self-matches PI030 (catalogue raw scan is back to the two standing findings, PI001 and PI031), and the baseline was regenerated 360 -> 376 without baselining away the catalogue finding. Baselined self-scan is 0 findings, CI is green, and the branch is current with main. Thank you for a careful, well-evidenced contribution.
Summary
Closes #39. Adds PI114-PI118 for five bounded attack shapes across Spanish, French, German, Portuguese, Italian, Russian, simplified Chinese, Japanese, Korean, Arabic and Hindi, while preserving Czech PI110-PI113 and leaving PI119 unallocated.
Rules carry
language:<ISO 639-1 code>tags through the existing schema. These describe supported regex branches, not per-finding language detection; filtering a multi-language rule does not isolate a branch. No engine, schema or dependency change.Review fixes (2026-10-21)
edec71c. Merged that main into this branch without rewriting the published commits; test: cache scanners per test binary to cut CI runtime #191, fix: localize gate03 sweep helper variables #192 and ci: raise the PERF-01 budget 200 -> 400ms for the 79-pattern library #198 are included. The CI workflow has no PR-specific changes relative to current main..github/code-scanning-baseline.json: all 360 existing entries remain, with 16 additional intentional findings in examples, patterns and the existing attack corpus (376 entries total). No new documentation finding was accepted. Native Windows generator paths were normalized to the repository's existing slash-separated convention; fingerprints are unchanged. Added a test proving the multilingual example findings are covered.tests/multilingual_test.rstocommon::scanner(). Only the deliberately unfiltered and relaxed control scanners retain their own OnceLock cache; the synthetic case-folding probe remains separate.>-folding works with(?x).Measured recall and limits
The original corpus payloads are unchanged: multilingual development recall is 10/10, previously 8/10; overall development recall is 105/115 (91.3%), previously 103/115. The CAT-03 held-out results are unchanged.
The multilingual recall row measures only the original Czech/German corpus, not eleven-language recall. The eleven-language matrix is synthetic regression coverage: 55 primary shape/language cases plus a German disclosure variant, uppercase variants, 55 near-misses, and Chinese/Japanese no-whitespace-boundary cases. Complete reports are compared with and without prefiltering. Phrases are deliberately bounded; other translations, inflections and paraphrases remain gaps. Native-speaker review is welcome.
Current verification
cargo test --locked --no-fail-fast -- --skip findings_name_the_repository_path_not_the_staging_copy: 475 passed, one filtered out. Run outside the sandbox so temporary Git repositories can be initialized. The single skipped assertion is the previously confirmed Windows path-format mismatch; neither that test nor its implementation changes here.cargo clippy --locked --all-targets -- -D warnings,cargo fmt --all -- --check, andgit diff --check: pass.cargo bench --locked -- --test: all four benchmark shapes pass.No ADR is needed: these are pattern metadata, test-control, baseline and documentation changes within the existing formats. Hosted CI is rerunning on the pushed update; local results above are not a claim that hosted checks are already green.