fix(ai): send stable, non-sequential user identifiers to AI providers - #3856
rasadregmi wants to merge 1 commit into
Conversation
|
@404oops can you review? |
404oops
left a comment
There was a problem hiding this comment.
I don't think this is ready to merge. The fix creates several new problems, mostly because it reuses the same flawed pattern in multiple places instead of centralizing how the safety identifier is built.
When a request is authenticated with an access token instead of a direct user session, the actor record does not put the issuing app in actor.app, but in actor.effectiveApp.
The current code reads actor.app?.uid, so for access-token requests, the provider still gets the user ID, but no app ID. This means you can't tell whether abuse is coming from a third-party app, or from the user directly.
It also leaks our internal sequential user IDs to external AI vendors. That lets them correlate Puter users across different services, and estimate rough signup order and total user count.
This is especially bad, because the old code accidentally avoided sending the raw ID at all due to a precedence bug. Azure and OpenAI previously sent :undefined; the others sent an empty string because numeric ID plus undefined became NaN.
With this PR, OpenAI, Azure, and xAI start sending the raw integer for the first time. Meta and ZAI already receive it as puter-<id>. We should send a UUID or a hash instead, but that should be up to us to decide.
The PR description says it mirrors MetaProvider, but that's inaccurate. MetaProvider and ZAIProvider have the same actor.app?.uid bug and the same raw integer problem, so copying them just spreads it across 8 places, instead of fixing it.
Also, Meta and ZAI use the format puter-<id>-<appUid>, while this code uses <id>:<appUid>. This should've been one shared helper that reads actor.effectiveApp ?? actor.app and returns a non-sequential identifier.
The ?? actor.app fallback is safe because makeActor derives effectiveApp from app, so a null effectiveApp always means app is null too. The fallback is only needed because the new tests construct actor objects directly without using makeActor, leaving effectiveApp undefined. Either keep the fallback or update the tests to use makeActor and only read actor.effectiveApp.
Test gaps
-
The existing Azure test titled "sends
safety_identifierfor OpenAI deployments" now runs under the system actor. With the new code, both relevant fields areundefined, and the assertion passes only because{ safety_identifier: undefined }still creates anundefinedvalue. The test title no longer matches what it actually verifies. It should either run under a user actor or be renamed to assert that the value isundefined. The same applies to thesafety_identifierassertion at the end of the Azure Responses request-shape test, which assignsundefineddirectly and likewise passes for the wrong reason. -
The two Azure tests and the two image tests do not include a system-actor case, but the OpenAI chat tests do.
-
There is no access-token test anywhere. Adding one that expects
42:app-abcwould fail against the current code. -
The Grok "strip
safety_identifier" test is still valid, because the identifier is removed before the request goes out.
To be clear, the precedence fix itself is correct, and the actor?.user?.id chaining removes the old TypeError. That part is fine. But the rest of the change is a quick fix that introduces bigger problems than it solves.
3376353 to
012ef65
Compare
|
@404oops One shared helper. All eight providers now build their identifier via src/backend/drivers/util/aiUserIdentifier.ts, and Meta/ZAI use it too: export const aiUserIdentifier = (actor, maxLength = 64) => {
Test updates.
Verification. Nine affected suites pass (203 tests); npm run typecheck has no new errors. Full npm run test:backend is green for everything touched by this change, the only intermittent failures are pre-existing env-dependent ones (PostgresDatabaseClient.integration 180s timeouts and the share-email suite: they pass in isolation and flake run-to-run regardless of this diff, and this run had a different subset fail than the last). |
There was a problem hiding this comment.
I have more notes on this.
-
src/backend/drivers/util/aiUserIdentifier.tsat line48: The identifier this produces for a real user and a real app does not survive the cap on this line intact. The tests do not catch it. -
src/backend/drivers/ai-image/providers/xai/XAIImageProvider.tsat line183: The edit path does not do what the generation path at line130does. -
src/backend/drivers/ai-chat/providers/azure/AzureChatProvider.test.tsat line468: This test would still pass ifuserwere removed from the Grok branch it covers. -
OpenAiChatCompletionsProvider.tsat line184andAzureChatProvider.tsat line211: Switchinguserfrom per-app to per-user has a side effect on OpenAI's end that the PR does not address or mention. -
aiUserIdentifier.tsat lines from23to39: The exported constant, its comment, and themaxLengthparameter do not hold up when checked against their callers and against the OpenAI SDK docs. -
aiUserIdentifier.tsat line44: The fallback on this line is not reachable by any actor the codebase actually builds, and the test that covers it shows why. -
ZAIProvider.test.tsat line409: Comment is stale. -
The commit subject contradicts the commit body.
-
OpenAiImageProvider.test.tsat line254is in the wrong describe block. -
The four-actor test matrix is pasted into six files.
Please, take a look at the code, attack it from all sides and try to do extensive reviews. I think your agent did a shallow analysis, which lead to these problems and holes.
I'm not saying this PR is bad at all, this is a good PR, but some things are simply overlooked and that's a BIG issue when you're attempting to fix a problem.
Do fix those issues, and if you (or your agent) find more within the scope of what you were trying to solve, don't hesitate to fix them as well.
012ef65 to
075891d
Compare
075891d to
b41ac36
Compare
Addresses review feedback on PR HeyPuter#3856 (send stable, non-sequential user identifiers to AI providers): - aiUserIdentifier: drop the app suffix entirely when the remaining budget is too small to meaningfully distinguish app uids, instead of emitting a short, collision-prone truncation. - XAIImageProvider#edit: switch to a named-options parameter so the new `user` arg can't be silently transposed with the adjacent same-typed `aspectRatio` arg in a future edit. - MetaProvider: drop the duplicate SAFETY_IDENTIFIER_MAX_LENGTH constant and rely on aiUserIdentifier's own default, which already matches it. - ZAIProvider.test: fix a comment that incorrectly claimed SYSTEM_ACTOR has no uuid; it does, and is excluded via isSystemActor() instead. - integrationTestUtil: extract assertActorMatrixIdentifiers() and use it across the six provider tests that were duplicating the same makeActorMatrix() assertion block. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Addresses review feedback on PR HeyPuter#3856 (send stable, non-sequential user identifiers to AI providers): - aiUserIdentifier: drop the app suffix entirely when the remaining budget is too small to meaningfully distinguish app uids, instead of emitting a short, collision-prone truncation. - XAIImageProvider#edit: switch to a named-options parameter so the new `user` arg can't be silently transposed with the adjacent same-typed `aspectRatio` arg in a future edit. - MetaProvider: drop the duplicate SAFETY_IDENTIFIER_MAX_LENGTH constant and rely on aiUserIdentifier's own default, which already matches it. - ZAIProvider.test: fix a comment that incorrectly claimed SYSTEM_ACTOR has no uuid; it does, and is excluded via isSystemActor() instead. - integrationTestUtil: extract assertActorMatrixIdentifiers() and use it across the six provider tests that were duplicating the same makeActorMatrix() assertion block.
aa4fadc to
94b1c07
Compare
…ed caps Addresses the remaining two open points from the second round of review on PR HeyPuter#3856 (the other eight — cap truncation, xAI edit parity, the vacuous Grok test, the effectiveApp-only fallback, the stale ZAI comment, the misplaced describe block, and the duplicated test matrix — were already fixed in earlier commits on this branch): - `user` is deprecated by OpenAI in favor of `safety_identifier` (abuse detection) and `prompt_cache_key` (cache-hit bucketing); continuing to only send `user` drops the caching benefit it used to provide. All four OpenAI/Azure chat providers and MetaProvider now also send `prompt_cache_key`, defaulting to the same per-user identifier when the caller doesn't supply one — verified against the OpenAI SDK's own `@deprecated` annotation on `user` ("Use prompt_cache_key instead to maintain caching optimizations"). Azure's Grok branch drops it alongside `safety_identifier`, matching its existing "unknown args 400" handling. - Rewrote the comment above AI_USER_IDENTIFIER_MAX_LENGTH: OpenAI's 64-char cap on `safety_identifier` is directly confirmed in the OpenAI SDK types for both Chat Completions and Responses; Z.AI's 128 is confirmed against Z.AI's own docs (6-128 chars). Meta's and xAI's own APIs don't document a limit for this field, so the comment no longer claims one is verified for them. Tests: extended the shared actor-matrix assertion helper to check any number of aliased fields (was hardcoded to one), added prompt_cache_key coverage to the matrix test in all four chat providers plus a dedicated caller-override test, and a defaulting test for MetaProvider. Verified against the actual OpenAI SDK type declarations (openai package) and via WebFetch against OpenAI's and Z.AI's own API docs, not from memory. 212 provider-suite tests pass; full test:backend passes twice in a row (8461/8461) after one run showed two unrelated flakes (a Postgres migration-integration timeout and this same MetaProvider test) that did not reproduce on rerun. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A precedence bug in the AI providers' identifier expression made every request send `user: ":undefined"` (the ternary bound the app-uid suffix to the whole `actor.user.id + actor.app?.uid` sum instead of just the suffix), or read `actor.user.id` on a missing user. The same expression also shipped the sequential internal user id, letting AI vendors correlate a single account across apps and sessions. All eight OpenAI-, Azure-, xAI-, Meta- and ZAI-style providers now build the identifier through one shared helper, `aiUserIdentifier()`: - `puter-<user-uuid>[-<app-token>]`: the random user UUID is always preserved in full; `maxLength` constrains only the app-bearing form - app attribution reads `effectiveApp`, so access-token requests name the issuing app instead of looking like direct user traffic - the app token is truncated to fit the budget, and omitted entirely when the remaining budget is below 8 chars, where a truncation could collide with another app's uid - nothing is sent for the system actor - Meta and ZAI keep a caller-supplied `safety_identifier` / `user_id` override, applied before the helper result `user` is deprecated by OpenAI; the SDK types direct callers to `safety_identifier` (abuse detection) and `prompt_cache_key` (cache-hit bucketing). The four OpenAI/Azure chat providers and MetaProvider now send `prompt_cache_key` as well, defaulting it to the same per-user identifier unless the caller supplies one; Azure's Grok branch drops both fields, matching its rejection of unknown args. The cap comment cites only verified limits: OpenAI's 64 for `safety_identifier` (from the SDK types) and Z.AI's 6-128 for `user_id` (from Z.AI's docs); Meta and xAI document none, so none is claimed. The xAI image `#edit` path now carries the identifier like generation, and takes a named-options param so `user` cannot be transposed with the adjacent same-typed `aspectRatio`. Tests share a four-actor matrix (`user` / `user+app` / `access token` / `system`) with `assertActorMatrixIdentifiers()` across the six OpenAI-style suites; the helper has exact-string and boundary coverage (size caps, zero-budget and sub-base cases, no dangling separator, UUID never truncated, collision guard); the Azure Grok assertions run under a real user actor so they cannot pass vacuously. 212 provider-suite tests pass; typecheck and ESLint are clean.
0baf1af to
7b67ae7
Compare
Thanks for the second pass. This revision is squashed to a single commit and addresses every point:
Verification: 212 provider-suite tests pass; npm run typecheck has no new errors; ESLint clean on all touched files (the 5 pre-existing no-explicit-any warnings in OpenAiChatResponsesProvider.ts are unrelated and untouched). |
An operator-precedence bug in the AI providers' identifier expression — the ternary bound the app-uid suffix to the whole actor.user.id + actor.app?.uid sum instead of just the suffix — meant providers received user: ":undefined" (or a raw sequential id). All eight OpenAI-, Azure-, xAI-, Meta- and ZAI-style providers now build the identifier through one shared helper emitting puter-[-]. 212 provider-suite tests pass; typecheck and ESLint are clean.
Why
What changed
Testing
Verification