Repository navigation
Conversation
Contributor
Author
|
Bump — scopes PII redaction to free-text attributes + requires Luhn-valid card numbers. Mergeable. |
PII scanning ran on every attribute value, and the credit-card pattern matched any bare 13-19 digit run — corrupting session.id UUIDs, user.id, and numeric metadata fields (future-agi#195). - mask() now scans only keys that carry free text: input.value, output.value, message content/content-text, document content, tool-call arguments/results, retrieval/reranker queries - credit-card candidates must be card-grouped (4-4-4-4 / 4-6-5 / contiguous) and pass the Luhn checksum - phone pattern gains a leading word boundary so it cannot claim the tail of a longer digit run Generated with Devin Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
jjcav84
force-pushed
the
fix/pii-redaction-scope
branch
from
November 6, 2026 00:56
64f4fff to
6843391
Compare
nik13
pushed a commit
that referenced
this pull request
Nov 9, 2026
… card shape (#244) Fixes `FI_PII_REDACTION` corrupting correlation identifiers and over-redacting numeric content (#195). Two changes: 1. **Key scoping.** `TraceConfig.mask()` ran the PII scanner on *every* attribute value. It now scans only attributes that carry free text — `input.value`/`output.value`, message `content`/`contents.*.text`, document `content`, tool-call arguments/results, retrieval/reranker queries. Structured identifiers (`session.id`, `user.id`, `metadata`) and typed scalars pass through untouched. 2. **Card-like card detection.** The credit-card pattern `\b(?:\d[ \-]*?){13,19}\b` matched any bare digit run — ~2% of random UUIDs, timestamps, request ids. Candidates must now be card-shaped (contiguous, or 4-6-digit groups like Visa `4-4-4-4` / Amex `4-6-5`) **and** pass the Luhn checksum. Also fixed: the phone pattern gains a leading `\b` so it can no longer claim the tail of a longer digit run after the card check clears it, and card checking is restored to its original ordering before the phone pattern (phone would otherwise eat card digits first). Issue repro, verbatim: ```python cfg.mask(SESSION_ID, "73630065-0794-4450-a1f9-8cc987a02b09") # now: unchanged cfg.mask(USER_ID, "1234567890123") # now: unchanged cfg.mask(METADATA, '{"request_id": "20240115103045123"}') # now: unchanged cfg.mask(INPUT_VALUE, "card 4111 1111 1111 1111") # still: <CREDIT_CARD> ``` **Deliberate tradeoff (please sanity-check):** keys outside the free-text set are never scanned — a stray custom attribute carrying an email won't be redacted. This favors identifier correctness over coverage, matching the issue's expected behavior. Widening the allowlist later is one line.
nik13
added a commit
that referenced
this pull request
Nov 9, 2026
#244 stops the phone pattern from matching digits inside a longer identifier, so parallel.search_id is no longer rewritten to search_<PHONE_NUMBER>. Update the stale expectation and the README note.
Contributor
|
Thanks! This change is now on |
nik13
added a commit
that referenced
this pull request
Nov 9, 2026
#244 exempts correlation identifiers and metadata (session.id, user.id, conversation ids, metadata) from PII redaction and stops the phone pattern from matching digits inside a longer identifier. Hide flags and the image cap still apply. Agno's context-privacy tests now pin that scope and keep checking redaction of free text; discoveryengine derives the project-number expectation from the installed core and documents both behaviours.
nik13
added a commit
that referenced
this pull request
Nov 9, 2026
… fields fi-security ruling on the #244 scope: session.id, user.id and conversation ids stay exempt from PII redaction (#195), but free-form metadata, where opted-in users put emails and phone numbers, is redacted again when FI_PII_REDACTION is on. #244's pattern fixes (no phone match inside longer ids, Luhn-valid cards) are unchanged. Changelog documents the #195 scope.
nik13
added a commit
that referenced
this pull request
Nov 9, 2026
#244's (?<!\w) lookbehind keeps digits inside longer identifiers intact, but a JSON-escaped newline, tab or return before a phone number (for example in serialized metadata) put a word character in front of it and the number was exported. Also start a match after a backslash escape; long identifiers after an escape are still not phone numbers.
nik13
added a commit
that referenced
this pull request
Nov 10, 2026
…14 and Java 1.0.3 Core 1.3.1 also ships the PII-redaction scoping fixes (#195/#244) that sat under Unreleased, and the 1.3.0 heading gets its release date. OpenAI 0.1.14 also ships the Azure metadata (#154), Baseten (#541), Azure Government (#502), OpenRouter host (#547), full multi-message input (#245) and exhausted-stream changes. Java 1.0.3 also ships #543 and #605. Notes the remaining blank OTEL-env Authorization limitation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13
added a commit
that referenced
this pull request
Nov 10, 2026
… traceAI-langchain 0.2.2, Java 1.0.3 (#636) * chore(release): bump core, OpenAI, LangChain and Java patch versions Synchronize runtime versions, reactor parent references and release changelogs for #155, #348, #397, #393 and #425. Dependency floors are unchanged. Co-Authored-By: Claude Code <noreply@anthropic.com> * chore(release): list every change shipping in core 1.3.1, OpenAI 0.1.14 and Java 1.0.3 Core 1.3.1 also ships the PII-redaction scoping fixes (#195/#244) that sat under Unreleased, and the 1.3.0 heading gets its release date. OpenAI 0.1.14 also ships the Azure metadata (#154), Baseten (#541), Azure Government (#502), OpenRouter host (#547), full multi-message input (#245) and exhausted-stream changes. Java 1.0.3 also ships #543 and #605. Notes the remaining blank OTEL-env Authorization limitation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes
FI_PII_REDACTIONcorrupting correlation identifiers and over-redacting numeric content (#195). Two changes:Key scoping.
TraceConfig.mask()ran the PII scanner on every attribute value. It now scans only attributes that carry free text —input.value/output.value, messagecontent/contents.*.text, documentcontent, tool-call arguments/results, retrieval/reranker queries. Structured identifiers (session.id,user.id,metadata) and typed scalars pass through untouched.Card-like card detection. The credit-card pattern
\b(?:\d[ \-]*?){13,19}\bmatched any bare digit run — ~2% of random UUIDs, timestamps, request ids. Candidates must now be card-shaped (contiguous, or 4-6-digit groups like Visa4-4-4-4/ Amex4-6-5) and pass the Luhn checksum. Also fixed: the phone pattern gains a leading\bso it can no longer claim the tail of a longer digit run after the card check clears it, and card checking is restored to its original ordering before the phone pattern (phone would otherwise eat card digits first).Issue repro, verbatim:
Deliberate tradeoff (please sanity-check): keys outside the free-text set are never scanned — a stray custom attribute carrying an email won't be redacted. This favors identifier correctness over coverage, matching the issue's expected behavior. Widening the allowlist later is one line.
Why?
Closes #195 — silent data corruption: session/user correlation ids mangled by a PII token, invisible in the UI.
How was it tested?
TestPiiRedactionScopeinpython/tests/test_config.py(9 cases: issue repros, Luhn-valid cards still redact, non-Luhn runs survive, nested message-content keys still scanned, identifier keys never scanned)pytest tests/test_config.py— 29/29 pass; baseline failures intest_otel.py/test_context_attributes.pyexist on cleanmain(unrelated, pre-existing)FI_PII_REDACTIONsemantics narrowed only to correct attributesChecklist
mainfix(pii): …)Notes for reviewers
The phone-regex
\baddition is load-bearing — without it20240115103045123(17 digits, not a card) still loses its last 10 digits to<PHONE_NUMBER>. The card↔phone ordering is restored to the pre-existing convention (card first).