Skip to content

feat(typesafe): TypeSafe (Jev) System One instrumentor (TH-8322) - #264

Merged
nik13 merged 2 commits into
devfrom
feat/th-8322-typesafe-jev
Oct 26, 2026
Merged

nik13 merged 2 commits into
devfrom
feat/th-8322-typesafe-jev

Conversation

@nik13

@nik13 nik13 commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Status 2026-10-21: merged dev into the branch (3dc3f3d, no conflicts), so dev's Python tests workflow ran on this PR: Plan, fi_instrumentation, package (typesafe), Python tests all green. Local run: 41 passed (Python 3.11, typesafe-sdk 0.7.5). traceAI-typesafe 0.1.0 is not on PyPI yet. After this merges, publish 0.1.0, then merge docs#977. docs#977 already dropped the git install Note, and its TRACEAI_TYPESAFE default is now traceAI-typesafe==0.1.0, so the BLOCKED-ON line at the end is done. Marked ready for review.


Summary

New package traceAI-typesafe (python/frameworks/typesafe, import traceai_typesafe, Apache-2.0) for TH-8322 (parent TH-8103): traces calls a customer's own code makes to TypeSafe's System One decision API (Jev) with the official Python client typesafe-sdk.

  • TypeSafeInstrumentor wraps typesafe_sdk.TypeSafeClient.system_one and AsyncTypeSafeClient.system_one (the async client exists in the pinned SDK, so both are wrapped). One EVALUATOR span (typesafe.system_one) per call, however many questions; questions are attributes, not child spans.
  • Per question typesafe.question.<name>.type plus noul (number), choice (label) + confidence, score (number) + confidence, box.{top,bottom,left,right} (+ confidence only when not null). noul never gains a confidence; nothing is mapped to Pass/Fail or boolean; no probability maps, no legends, no explanation attribute.
  • gen_ai.provider.name=typesafe, gen_ai.request.model (as asked: model=, extra_body["model"], or the client default), gen_ai.response.model + typesafe.response.model only when it differs, gen_ai.usage.{input,output,total}_tokens, typesafe.request_id, server.address, typesafe.question_count, typesafe.idempotency_key (only when inputs are shown). No cost.
  • Errors: ERROR status <Class> (HTTP <status>), error.type, http.response.status_code, typesafe.error_type (TypeSafe's detail.error_type), typesafe.retry_after_ms, request id; an exception event without a stack trace. The server's message is appended only while content is captured. Client-side TypeSafeError text (can name a local image path) is never exported.
  • Never sets base_url or a model, never calls the API, does not wrap client.models.list() (GET /v1/models) or any HTTP library, no eval-registry/gateway import (TH-8183 untouched), no README catalog row, no other package touched.

Docs page + runnable example: future-agi/docs PR #977 (/docs/integrations/traceai/typesafe).

Privacy (traceAI TraceConfig convention, per the 2026-10-16 coordinator clarification)

Content on by default: input.value = {state, questions} (images as <image:…> references only), output.value = decisions, choice labels. FI_HIDE_INPUTS/FI_HIDE_INPUT_TEXT drop the input side (and the idempotency key); FI_HIDE_OUTPUTS/FI_HIDE_OUTPUT_TEXT drop output.value and choice labels (numbers stay); FI_PII_REDACTION also masks the error message. No package-specific switch. API key, stack traces, local paths never exported in either mode. Question names are attribute keys (documented).

Versions tested

  • typesafe-sdk 0.7.5 (latest on PyPI, released 2026-10-14; wheel sha256 967a8b61…6d2) on Python 3.12.10: unit tests + E2E.
  • typesafe-sdk 0.7.4 (the scout's pin) on Python 3.13 and 0.7.0 (declared floor: first pydantic release with response_model) on Python 3.10: unit tests. Range typesafe-sdk >=0.7.0,<1; Python >=3.10,<3.14 (fi-instrumentation-otel 1.1.0 caps at <3.14). Box questions and image assets exist from 0.7.5; their tests skip on older SDKs.
  • fi-instrumentation-otel 1.1.0, opentelemetry-sdk 1.45.1, wrapt 1.17.3.
python 3.12 typesafe-sdk 0.7.5: 41 passed
python 3.13 typesafe-sdk 0.7.4: 39 passed, 2 skipped (image assets exist from 0.7.5)
python 3.10 typesafe-sdk 0.7.0: 39 passed, 2 skipped (image assets exist from 0.7.5)

How to run

cd python/frameworks/typesafe
pip install -e . pytest pytest-asyncio
pytest                      # real typesafe-sdk clients over httpx2.MockTransport, shaped like the published OpenAPI; no network

E2E (docs repo, examples/traceai/typesafe): FI_API_KEY=… FI_SECRET_KEY=… FI_BASE_URL=… PRIVACY=1 REAL_AUTH_PROBE=1 ./run_e2e.sh (installs this package from this branch's commit).

E2E evidence (real run against local Future AGI, 2026-10-16)

run_e2e.sh exactly as the docs page tells customers: fresh venv, pip install typesafe-sdk 0.7.5 + fi-instrumentation-otel 1.1.0 + this package from GitHub at e6dd3c0e16e7, register() + TypeSafeInstrumentor().instrument(), FI_BASE_URL=http://localhost:14338, project th8103-rest-typesafe-jev.

STANDIN: no TypeSafe key exists (TH-8183 recorded the same), so the success path ran the real SDK against standin/systemone_stub.py on 127.0.0.1 (shaped from https://api.typesafe.ai/openapi.json v0.2.0 and the real API's error bodies; numbers are hash-derived, not Jev). The 401 failure case additionally ran against the real api.typesafe.ai with a deliberately invalid key (no account, nothing billed).

scenario trace id result
ok (sync 3 questions + async 1 question, models.list()) 3c5a790f-fb46-4ada-b4b9-5723b9be141b chain → 2 evaluator spans, OK; no span for GET /v1/models
fail-422 (choice criteria as a list; API-side validation) 218ad3e0-9438-4ec6-afa2-c7a388ee430b ERROR TypeSafeUnprocessableEntityError (HTTP 422): questions.tone.choice.criteria: …
fail-auth (stand-in) a8e9792c-ed82-4850-9839-dd7248ae08d3 ERROR TypeSafeAuthenticationError (HTTP 401)
fail-asset (missing local image, client-side) dd455b3f-cb65-40f9-8b62-ba82d0395df6 ERROR TypeSafeError, no path anywhere
real api.typesafe.ai, invalid key 642fa55a-28d9-418e-8bf5-1e9036b5afc0 ERROR TypeSafeAuthenticationError (HTTP 401): Cannot authenticate with the server. …, typesafe.request_id=req_01a1451997917a7290acf25521236f10, server.address=api.typesafe.ai, typesafe.error_type=authentication_error

fi_verify.py --project th8103-rest-typesafe-jev --wait 120 --min-spans 7 --expect-type evaluator --expect-type chain → PASS:

traces=5 spans=11
trace 3c5a790f-fb46-4ada-b4b9-5723b9be141b (3 spans)
- [chain] triage_ticket in=115B out=7B (92ms)
  - [evaluator] typesafe.system_one model=jev-latest tok=131/6 in=534B out=185B (4ms)
  - [evaluator] typesafe.system_one model=jev-latest tok=70/2 in=272B out=46B (16ms)
trace 218ad3e0-… - [evaluator] typesafe.system_one model=jev-latest status=ERROR:TypeSafeUnprocessableEntityError (HTTP 422): questions.tone.
trace 642fa55a-… - [evaluator] typesafe.system_one model=jev-latest status=ERROR:TypeSafeAuthenticationError (HTTP 401): Cannot authenticate
PASS

Stored row (ClickHouse spans): observation_type=evaluator, provider=typesafe, model=jev-latest (request model kept; gen_ai.response.model=jev-1.13.0), prompt_tokens=131, completion_tokens=6, cost=0, numeric answers in attrs_number. Observe UI shows Type evaluator / Provider typesafe / tokens / input+output JSON and the numeric attributes (screenshots kept locally).

Privacy proof

PRIVACY=1: unique marker per run in the ticket text, an email address built from it, the question instructions and a criteria description; run ok, fail-422, fail-asset with FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true into …-hidden and with defaults into …-default (two rounds each). Then a scan of every stored row and column for each project (9–10 ClickHouse tables incl. rollups/sessions/traces; all 46 spans columns incl. resource_attrs, span_events, attributes_extra, status_message; Postgres tracer_*):

hidden  (traces 4ee962be…, 4012e134…, fc90cdd5…, 85f7d350…, 55bee686…, 9ee01ccb…): rows=50
  MARK_HIDDEN_1/2, EMAIL_HIDDEN_2, ticket text, state key, instruction text, criteria text,
  choice labels calm/angry/excited: rows_hit=0
default (traces 1ee22341…, 03aaa1da…, 165c4444…, 1edb555e…, 4fd95491…, 3a0d6ea0…): rows=56
  MARK_DEFAULT_1/2 rows_hit=9 each, EMAIL_DEFAULT_2 rows_hit=9 (columns input, attributes_extra)   <- content on by default
all three projects: FI_API_KEY / FI_SECRET_KEY / stand-in key / invalid TypeSafe key / "Bearer " /
  "uthorization" / hostname / username / $HOME / temp dir / missing-image path / Traceback /
  stacktrace / site-packages / probabilities / legend: rows_hit=0

Live vendor verification (2026-10-17)

TYPESAFE_API_KEY set, no TYPESAFE_BASE_URL: the SDK called the real https://api.typesafe.ai (jev-latest resolved to jev-1.13.0), traces to local Future AGI project th8103-rest-typesafe-jev-live. typesafe-sdk 0.7.5, traceAI-typesafe 0.1.0 (e6dd3c0e16e7), fi-instrumentation-otel 1.1.0, opentelemetry-sdk 1.45.1, Python 3.12.10. Vendor spend: two decisions (397/73 and 318/22 tokens) plus two more in the privacy runs, well under $1.

scenario trace id result
ok (sync noul+choice+score, async noul) 5e0d9155-271c-4183-924f-8f9411b53701 chain -> 2 evaluator spans, OK, tokens 397/73 and 318/22, request ids req_...
fail-422 1d89de2c-3560-411f-a21f-71c4b7eda3f2 ERROR TypeSafeUnprocessableEntityError (HTTP 422)
fail-auth (invalid key) 11a1e3da-31e5-49d6-8709-ca1462bfc54b ERROR TypeSafeAuthenticationError (HTTP 401)
fail-asset d45fd1ab-0cb3-4609-948b-aa9413f60937 ERROR TypeSafeError, no path
fi_verify --project th8103-rest-typesafe-jev-live --expect-type evaluator --expect-type chain
traces=5 spans=11
trace 5e0d9155-... (3 spans)
- [chain] triage_ticket in=115B out=7B (616ms)
  - [evaluator] typesafe.system_one model=jev-latest tok=397/73 in=534B out=172B (179ms)
  - [evaluator] typesafe.system_one model=jev-latest tok=318/22 in=272B out=44B (245ms)
PASS

Span attributes seen live: gen_ai.request.model=jev-latest, gen_ai.response.model=typesafe.response.model=jev-1.13.0, server.address=api.typesafe.ai, typesafe.question.billing.type=noul, ...tone.choice=angry, ...urgency.type=score.

Privacy against the live API (marker + email + ticket text + instructions, hide flags on vs default; all 46 spans columns and every ClickHouse table of the project): hidden -> marker, email, ticket, instruction/criteria text, choice label, TypeSafe key, FI keys, Bearer , user, host, home, Traceback, probabilities, legend all rows_hit=0 (28 rows); default -> marker/ticket in 9 rows (input, attributes_extra), keys 0.

Still not seen live (unit tests and STANDIN only): HTTP 429 with retry_after_ms, box answers, image assets, SDK retries.

Limitations / unsupported

  • Live Jev success path now verified (2026-10-17, section above). The 2026-10-16 evidence above used the STANDIN server for System One answers; the 401 trace of that run was already real.
  • The SDK's own RetryPolicy (default 2 retries on 408/429/5xx) runs inside one span: 429 then success = one OK span. The PRD J2 wording "one span per attempt" holds only for calls the customer repeats. Flagging for review.
  • PRD listed typesafe.usage.*; this uses the standard gen_ai.usage.* keys so Future AGI's token columns fill (verified). Request model stays in the Model column.
  • Custom response_model without an answers field: span keeps model/usage, no per-question attributes. Probability maps, legends, X-System-One-* headers, JS SDK: not supported (documented).
  • Agent Graph in the Observe UI draws the evaluator node with a generic "?" icon (UI cosmetic, not this package).
  • No registry key added to python/traceai/__init__.py (optional per family architecture).

Nav suggestion

Docs page: group Integrations → traceAI, title TypeSafe (Jev), path /docs/integrations/traceai/typesafe.

BLOCKED-ON: none. Merge order unchanged; drop the git install Note and the TRACEAI_TYPESAFE default once traceAI-typesafe is on PyPI.

🤖 Generated with Claude Code

New package python/frameworks/typesafe (distribution traceAI-typesafe,
import traceai_typesafe, Apache-2.0). TypeSafeInstrumentor wraps
typesafe_sdk.TypeSafeClient.system_one and AsyncTypeSafeClient.system_one
(typesafe-sdk >=0.7.0,<1): one EVALUATOR span per decision call with
typesafe.question.<name>.{type,noul,choice,score,confidence,box.*},
request/response model, token usage, request id and server host.

- No probability maps, legends, explanation or Pass/Fail mapping.
- noul and null box confidence stay absent, never zero.
- Content follows the traceAI TraceConfig convention (on by default,
  FI_HIDE_INPUTS / FI_HIDE_OUTPUTS off); API key, stack traces and
  client-side error text (local paths) are never exported.
- Never sets base_url or a model; client.models.list() is not traced.
- Tests run the real SDK over httpx2.MockTransport, shaped like the
  published OpenAPI document; no call to api.typesafe.ai.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nik13
nik13 marked this pull request as ready for review October 21, 2026 23:41
@nik13
nik13 merged commit 087c488 into dev Oct 26, 2026
8 checks passed
@nik13
nik13 deleted the feat/th-8322-typesafe-jev branch October 26, 2026 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant