Skip to content

fix(ci): align release contracts and preserve audio cache counts - #620

Merged
nik13 merged 3 commits into
devfrom
fix/dev-ci-contract-tests
Nov 8, 2026
Merged

nik13 merged 3 commits into
devfrom
fix/dev-ci-contract-tests

Conversation

@nik13

@nik13 nik13 commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Restore the six failing TypeScript contract workflows after the zero-code/semantic-conventions release. Keep the stronger privacy contract; fix the confirmed audio-cache-count runtime regression rather than changing its expected count.

Failure analysis and intended behavior

All six baseline runs below failed on dev 2c3a6e8700a0e76b6c8c0c6b820017cffe0c4804 before this branch.

Workflow / failing dev run Cause and decision Change and verification
OpenAI SDK contracts 78ae25ed (#605) defines time-to-first-token semantics; 6a3cc768 adds its OpenAI stream emission. The same zero-code commit / fi-core CHANGELOG 1.2.0 forbids exporting base64 images. Both changes are intentional. Keep exact snapshots with asymmetric numeric timing matchers; expect [image-redacted] in all three image-bearing aliases. The request fixture still contains its original image. Run the six-version OpenAI workflow, including packed consumers and the complete OpenAI/core unit suites.
OpenAI Vercel gateway tests Same five shared OpenAI snapshot failures, not a separate gateway defect. Apply the same narrow snapshot corrections; no gateway code changes. Run all three SDK versions.
Legacy Google Generative AI compatibility 6a3cc768 introduced shared export privacy and fail-closed redactRequestMetadata: content hiding in either direction projects request/tool metadata to canonical safe fields. Old output-hiding tests expected provider-specific topP, content-bearing stopSequences, tool descriptions and schemas verbatim. Expect explicit redaction markers while retaining temperature and tool identity. Keep request non-mutation, unhidden raw input, finish reasons and output secrecy assertions. No privacy allowlist widening. Run SDK 0.1.0 and 0.24.1 against real loopback SDK/OTLP fixtures.
Google GenAI SDK compatibility Runtime regression in 6a3cc768: the shared structural-media classifier treats gen_ai.usage.cache_read_tokens.audio = 2 as audio payload because this exact metric key is absent from its scalar catalog. This contradicts the existing modality cost-input contract. Add only this exact key as a nonnegative-safe-integer metric. Reject strings, booleans, arrays and invalid counts; retain registered-secret, media-ancestry and exact-path precedence. Assert the count before and after OTLP export, for generation/SSE and hiding on/off. Run the eight-job Node 20/22/24 + SDK matrix, with five loopback repetitions per job.
LangChain TypeScript contracts 6a3cc768 removed the project-name override of OTel service.name. register() still sets project_name; service identity belongs to OTel resource configuration. Assert project_name and a deliberately distinct OTEL_SERVICE_NAME through the exact packaged README recipe. Run all six core versions.
Integration tests fi-core CHANGELOG 1.2.0 and 6a3cc768 explicitly export only typed setSession identities; raw session-named attributes remain masked. Codex writes a raw thread-derived attribute. The aggregate failed because this wire assertion failed. Assert the raw-session mask and add a real OTLP positive control proving a typed setSession identity exports. Keep prompt secrecy, usage, parent-child trace identity and HTTP assertions. Do not weaken privacy to re-expose untyped thread IDs. Run the full planned TypeScript integration matrix and aggregate.

Latent native-message contract failures

Once the OpenAI snapshots passed, its final fi-core/test/genai-privacy.test.cjs command exposed three additional stale expectations from before 6a3cc768:

  • Declared audio payload bytes are always omitted by shared media privacy, even when only image hiding is enabled. Preserve the audio part's structure, video URI and ordinary text; assert the exact omission marker for its payload.
  • The process-wide credential registry outranks disabled/opposite-direction hide flags. Use a distinct public-text control and explicitly register the private control so this test also passes in isolation; never reset or bypass the registry.
  • The shared inspection budget fails closed on the 1,000-message fixture. Keep that large fixture and assert no private data reaches span attributes or the sampler. Add a 100-message case to prove an in-budget batch retains its complete shape.

An E2B run of the unchanged dev runtime and original test reproduced exactly these three failures (85 passing / 3 failing). With only the revised contract test, the same dev runtime passes all 89 cases, and the direction/credential test also passes on its own.

Runtime and release impact

  • @traceai/fi-core 1.2.1 -> 1.2.2, synchronized in package metadata, release-please manifest and CHANGELOG. fi-core has no separate hard-coded runtime version constant.
  • Keep the authoritative shipped scalar catalog and both bundled copies synchronized (scalar catalog v6). Rebase the performance-equivalence test's catalog fingerprint explicitly; leave its pinned runtime hashes and comparisons intact.
  • Google GenAI's previous workflow installed only published shared dependencies, so it could not test an unpublished core fix. Continue compiling the adapter against published dependencies, then install the candidate core/semconv tarballs into the clean consumer with explicit overrides (including the legacy coexistence adapter). npm ls records the resolved tree. SDK matrix, five repetitions, OS-level network isolation and all assertions remain enabled.
  • No public API, auth, tenant boundary, migration or persistence change. The only runtime behavior change is retaining this one valid numeric metric; invalid/media/secret inputs remain fail-closed. Rollback is a normal revert and restores the prior omission behavior.
  • No package has been published. Existing npm consumers need the eventual fi-core 1.2.2 release; this PR does not claim published 1.2.0 is repaired.

Validation receipts

CI for head 9734891d90533f55d2a21fc0a5f5d297671e83bc:

Cloud-only causal controls:

  • E2B fu-dev-ci-audio-count-1108-r1: new focused tests pass 15/15. Replacing only the candidate scalar catalog with dev's catalog yields exactly two failures (numeric preservation with hiding off/on), with the other 13 privacy controls still passing.
  • E2B fu-dev-ci-genai-privacy-1108-r1: original dev native-message contracts fail 3/88; revised contracts pass 89/89 on the same dev runtime; isolated direction/credential test passes 1/1.
  • Both E2B sandboxes were terminated and deletion verified. Only synthetic fixtures/loopback endpoints were used. No local builds/tests; local checks were syntax inspection, source/diff review and git diff --check.

Review

Self-reviewed the complete final diff against internal minimal-code, typing, failure-mode, PR-review, scenario-testing and API-contract standards, including exact-key admission, secret/media precedence, bounded inputs, repeated export, package resolution and version consistency. No disabled tests or weakened privacy assertions. No unrelated refactors. Human approval and merge remain outstanding; this is deliberately a draft, and no merge or administrative override is requested by this change.

Generated with Claude Code

nik13 and others added 3 commits November 8, 2026 02:51
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
@nik13
nik13 marked this pull request as ready for review November 8, 2026 11:33
@nik13
nik13 merged commit 3e96029 into dev Nov 8, 2026
182 of 184 checks passed
@nik13
nik13 deleted the fix/dev-ci-contract-tests branch November 8, 2026 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant