Skip to content

feat: add PI114-PI118 multilingual evasion patterns - #197

Merged
hermanngeorge15 merged 7 commits into
UnityInFlow:mainfrom
BasilZafar11:codex/multilingual-patterns-39
Oct 23, 2026
Merged

hermanngeorge15 merged 7 commits into
UnityInFlow:mainfrom
BasilZafar11:codex/multilingual-patterns-39

Conversation

@BasilZafar11

@BasilZafar11 BasilZafar11 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes #39. Adds PI114-PI118 for five bounded attack shapes across Spanish, French, German, Portuguese, Italian, Russian, simplified Chinese, Japanese, Korean, Arabic and Hindi, while preserving Czech PI110-PI113 and leaving PI119 unallocated.

Rules carry language:<ISO 639-1 code> tags through the existing schema. These describe supported regex branches, not per-finding language detection; filtering a multi-language rule does not isolate a branch. No engine, schema or dependency change.

Review fixes (2026-10-21)

  • Fetched the fork and upstream first: both main branches matched at edec71c. Merged that main into this branch without rewriting the published commits; test: cache scanners per test binary to cut CI runtime #191, fix: localize gate03 sweep helper variables #192 and ci: raise the PERF-01 budget 200 -> 400ms for the 79-pattern library #198 are included. The CI workflow has no PR-specific changes relative to current main.
  • Reworded PI118's description so it no longer triggers PI030 at HIGH, regenerated the catalogue, and added a regression test covering all multilingual descriptions.
  • Regenerated .github/code-scanning-baseline.json: all 360 existing entries remain, with 16 additional intentional findings in examples, patterns and the existing attack corpus (376 entries total). No new documentation finding was accepted. Native Windows generator paths were normalized to the repository's existing slash-separated convention; fingerprints are unchanged. Added a test proving the multilingual example findings are covered.
  • Changed the shipped scanner in tests/multilingual_test.rs to common::scanner(). Only the deliberately unfiltered and relaxed control scanners retain their own OnceLock cache; the synthetic case-folding probe remains separate.
  • Sharpened PI114-PI117's relaxed controls: retain the override/priorness, forget-imperative, second-person role reassignment, or disclosure framing instead of matching a bare verb. The complete payload/uppercase/near-miss matrix still passes, and another test checks that unrelated framing does not satisfy the relaxed forms. PI118 still relaxes the enabled-status discriminator.
  • Put the new README and detection-backlog payload specimens in code spans and documented how YAML >- folding works with (?x).

Measured recall and limits

The original corpus payloads are unchanged: multilingual development recall is 10/10, previously 8/10; overall development recall is 105/115 (91.3%), previously 103/115. The CAT-03 held-out results are unchanged.

The multilingual recall row measures only the original Czech/German corpus, not eleven-language recall. The eleven-language matrix is synthetic regression coverage: 55 primary shape/language cases plus a German disclosure variant, uppercase variants, 55 near-misses, and Chinese/Japanese no-whitespace-boundary cases. Complete reports are compared with and without prefiltering. Phrases are deliberately bounded; other translations, inflections and paraphrases remain gaps. Native-speaker review is welcome.

Current verification

  • cargo test --locked --no-fail-fast -- --skip findings_name_the_repository_path_not_the_staging_copy: 475 passed, one filtered out. Run outside the sandbox so temporary Git repositories can be initialized. The single skipped assertion is the previously confirmed Windows path-format mismatch; neither that test nor its implementation changes here.
  • Final focused run after adding the baseline guard: 11 multilingual tests and 3 catalogue tests pass.
  • cargo clippy --locked --all-targets -- -D warnings, cargo fmt --all -- --check, and git diff --check: pass.
  • cargo bench --locked -- --test: all four benchmark shapes pass.
  • Release 500-file smoke measurement on Windows, after warm-up: 308/251/236ms, best 236ms, against main's inherited 400ms CI budget. Hosted CI remains the authoritative runner measurement.
  • Whole-repo self-scan outside examples/patterns/tests/tools reports exactly the two pre-existing catalogue findings (PI001 at line 77 and PI031 at line 890); the additional PI030 finding is gone. Applying the regenerated native baseline leaves zero unreviewed findings (exit 0).
  • Catalogue regenerated; existing recall pins and all strict clean-corpus tests pass.

No ADR is needed: these are pattern metadata, test-control, baseline and documentation changes within the existing formats. Hosted CI is rerunning on the pushed update; local results above are not a claim that hosted checks are already green.

@hermanngeorge15 hermanngeorge15 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a strong PR and the bookkeeping is unusually complete — catalogue regenerated, a clean-corpus specimen added, relaxed_pattern present on all five (required for PI050+), and every number moved in step: 79 -> 84 patterns, multilingual 4 -> 9, recall 8/10 -> 10/10, total 103/115 -> 105/115 (91.3%). I checked that arithmetic and it holds.

The part I most want to keep is the caveat you added to the README: that the recall row measures only the original Czech/German corpus and is not eleven-language recall. That is exactly how CAT-03 reports its number, and it is the right instinct.

I also mutation-checked the false-positive controls on PI114, PI116 and PI117 — in each case the shipped pattern misses its counter_example while the relaxed form catches it, so GATE-05 is load-bearing here rather than decorative.

Two things to fix before this merges.


1. (blocking) PI118's description makes the scanner flag its own catalogue, at HIGH.

description: "Multilingual claim that developer mode is enabled"

That English sentence is itself a PI030 match. In isolation:

$ echo 'Multilingual claim that developer mode is enabled' | injection-scanner check -
[{"pattern_id":"PI030","severity":"HIGH"}]

It renders into docs/PATTERN-CATALOGUE.md:1919, and the whole-repo self-scan goes from 2 findings on main to 3 on this branch — the new one being PI030 at that line. This is the "the scanner flags its own documentation" regression called out in .claude/skills/pattern-library, and HIGH is the tier install-hook blocks commits at, so it can block a contributor's commit.

Rewording the description is enough — e.g. "Multilingual claim that a developer mode has been switched on" — or put the phrase in a code span.

2. (blocking) .github/code-scanning-baseline.json needs regenerating.

This PR changes patterns/ and adds payloads to examples/multilingual-attack.md. Both .github/workflows/code-scanning.yml (lines 82-86) and the pattern-library checklist require the baseline to be regenerated when either changes. It will not fail CI — that job emits an alert list rather than a pass/fail gate — but it will raise unreviewed alerts in the security tab.

cargo run --release -- check . --exclude '.planning/**' \
  --write-baseline .github/code-scanning-baseline.json

Non-blocking notes:

  • The relaxed_patterns strip down to a bare verb alternation (PI114's is (?:ignora|ignore|ignoriere|...)). That proves some narrowing exists, but not that the intended discriminator — the object clause — is the thing doing the work. Relaxing only the object would make the control sharper.
  • README.md and docs/DETECTION-BACKLOG.md add payloads in double quotes inside markdown tables. They stay silent on the table confidence downgrade rather than on correctness, and DETECTION-BACKLOG.md is the file that caused this exact problem twice before. Backticks would be safer.
  • The (?x) plus YAML >- folding combination is genuinely clever — folded newlines become spaces, which (?x) then ignores — and it is not obvious to a future reader. Worth one comment line so nobody "simplifies" it into a broken regex.
  • On "native-speaker review is welcome": agreed, and I cannot provide it for Spanish, Arabic, Hindi, Korean or Japanese. I verified the regexes mechanically — that they compile, match their examples, and miss their counter-examples — but not that these are the phrasings an attacker would actually write. I will open a separate issue asking for that review rather than holding this PR on it.

CI is red on PERF-01 only, which is a pre-existing flake on main and not caused by this PR. See #188 — I am fixing the gate first.

Note for sequencing: this PR and #191 both touch tests/recall_test.rs, tests/pattern_test.rs and tests/markdown_context_test.rs. I plan to merge #191 first, so this will need a rebase — and tests/multilingual_test.rs should then use #191's common::scanner() rather than building its own.

@hermanngeorge15

Copy link
Copy Markdown
Contributor

Status update so the red badge on this PR isn't misleading you.

The PERF-01 gate is fixed on main (#198, merged). The budget went 200 -> 400ms, re-derived for the 79-pattern library — your 213ms measurement on #188 is what sized it. main is green again for the first time since 2026-10-10.

Your PR's current FAILURE predates that fix. Rebase onto main and the gate should stop failing you. I deliberately did not run update-branch myself, in case you have local work in flight — your call when to pull.

Also merged in the meantime, both of which touch files this PR touches:

The suite is now ~3 minutes locally, down from ~11, so your own iteration loop should be much faster.

The two blocking items from my review are unchanged: PI118's description firing PI030 at HIGH on the generated catalogue, and the code-scanning baseline. Everything else in the PR stood up to review.

Thanks for the three PRs — #191 and #192 are both merged.

@BasilZafar11

Copy link
Copy Markdown
Contributor Author

Updated PR #197 with both blockers fixed: PI118’s description no longer triggers PI030, and the code-scanning baseline is regenerated. The branch now includes current main, uses common::scanner(), and addresses the relaxed-control and documentation notes. Added regression checks; local validation passes apart from the known Windows-only assertion. CI is rerunning.

@hermanngeorge15

Copy link
Copy Markdown
Contributor

Re-reviewed — both requested changes verified, by re-running the gate rather than reading the diff. Your write-up made that fast to check, and every number in it held.

(a) The PI118 self-match is fixed at the source. docs/PATTERN-CATALOGUE.md now scans to the same 2 raw findings as main — PI001 at :77 and PI031 at :890, the standing pair tracked in #171. The PI030 HIGH at :1919 is gone. Rewording the description to "a developer mode has been switched on" is the right shape of fix: it holds at the raw severity tier, so install-hook won't block a contributor's commit, which is what the original objection was actually about.

(b) The baseline is correctly regenerated. 360 → 376 entries, exactly the +16 you describe. The specific thing I checked is that you didn't take the easy route: the catalogue's baseline entries are identical to main's (PI001, PI031 and nothing else), so the PI030 finding was eliminated, not suppressed behind a baseline entry. Normalising the native Windows backslash paths to the repo's slash convention was the right call too.

Also confirmed independently:

  • Baselined self-scan (the CI gate's own command): 0 findings on both main and this branch.
  • fmt · clippy · test · build: passing.
  • The branch is 0 commits behind main, so the ruleset's strict up-to-date requirement is already satisfied — no rebase needed.
  • Merged, this takes the library to 84 patterns across 9 categories.

On the skipped assertion: findings_name_the_repository_path_not_the_staging_copy is a pre-existing Windows path-format mismatch, and nothing here touches it or its implementation, so skipping it is correct. Worth saying plainly though that CI has no Windows leg at all — ci.yml runs only ubuntu-latest — so that assertion has never been covered in CI either way. Your Windows runs are currently the only signal this project has on it. That is a gap in our CI, not in your PR, and I would rather state it than let it look like your environment is the problem.

This is accepted on correctness. I am holding the merge briefly behind a release announcement that quotes the current 79-pattern count — purely a publishing-sequence thing on my side, with nothing further needed from you. I am leaving the formal "changes requested" state as it is only so that nothing merges ahead of that; please don't read it as another round of work.

Thanks for the care on this one, and for the CI-flake diagnosis on #188 that sized the new budget.

@hermanngeorge15 hermanngeorge15 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both requested changes verified by running the gates, not reading the diff: the PI118 description no longer self-matches PI030 (catalogue raw scan is back to the two standing findings, PI001 and PI031), and the baseline was regenerated 360 -> 376 without baselining away the catalogue finding. Baselined self-scan is 0 findings, CI is green, and the branch is current with main. Thank you for a careful, well-evidenced contribution.

@hermanngeorge15
hermanngeorge15 merged commit 6df7a7b into UnityInFlow:main Oct 23, 2026
1 check passed
@BasilZafar11
BasilZafar11 deleted the codex/multilingual-patterns-39 branch October 24, 2026 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

patterns: PI110-PI119 multilingual evasion — every pattern is English-only today

2 participants