Repository navigation
feat(exa): traceAI-exa instrumentor for TH-8324 - #205
Merged
Merged
Conversation
The Node OTLP exporter streams with Transfer-Encoding: chunked and sends no Content-Length. The Receiver read Content-Length only, so every Node export came back 400 with zero spans; two TH-8103 children had to put a de-chunking relay in front of it. Receiver now accepts both framings. It also records one entry per accepted export (path, lower-cased headers, flattened resource attributes) via requests(), so a contract test can assert X-Api-Key/X-Secret-Key and project_name through the shared harness instead of a private recorder. Verified: 6 harness tests pass, and the TanStack example's real Node exporter delivered 4 chunked exports (4 spans, collector path, both auth headers, project_name/project_type) with no relay. Refs: TH-8339, TH-8103
…/th-8324-exa Refs: TH-8103
The real exa-py 2.25.0 client calls a loopback fake of the Exa API (Exa(base_url=...)): search, answer, and a search that gets HTTP 401. Spans leave through the real fi_instrumentation register() exporter into harness.Receiver (3eaadc8). Asserts the collector path /tracer/v1/traces, X-Api-Key/X-Secret-Key, project_name/project_type=observe, RETRIEVER kind, query and document count, ERROR status plus exception event on the 401, and that neither the Exa key nor any response text is exported. Mutation check: turning the error status into OK fails the contract. 6 passed (4 unit + 2 contract). approved: Nikhil 2026-10-03 blanket Refs: TH-8324
exa-py keeps the API key only in headers["x-api-key"]; the old guard read a non-existent api_key attribute, so a key pasted into a query was exported. Read the header first and fall back to api_key. Adds a loopback Exa fake (JSON and SSE routes) for real-SDK tests and covers Exa and AsyncExa. approved: Nikhil 2026-10-03 blanket Refs: TH-8324
The spec caps the query at 1 KB, but slicing 1024 code points let a non-ASCII query reach 3-4 KB. Cut the UTF-8 encoding at 1024 bytes and drop a partial trailing character. approved: Nikhil 2026-10-03 blanket Refs: TH-8324
The backend input panel reads input.value, which the spans never set. Write the same redacted, capped query there and keep fi.retrieval.query. approved: Nikhil 2026-10-03 blanket Refs: TH-8324
get_contents exported the requested URLs, joined, as fi.retrieval.query. URLs can carry tokens or personal data, and the spec asks for a URL count. Record fi.retrieval.url_count by default and no query/input.value for get_contents. instrument(capture_urls=True) adds fi.retrieval.urls: at most 20 URLs, each with the Exa key redacted and capped at 1 KB. Attribute extraction can no longer raise into the caller. approved: Nikhil 2026-10-03 blanket Refs: TH-8324
fi.retrieval.document_count was 0 on every answer span (AnswerResponse has citations, not results) and on every stream span (the count never saw the chunks). Count AnswerResponse.citations, sum StreamChunk.citations over a fully consumed stream, and omit the attribute when the count is unknown (error, cancellation, unrecognised result) instead of writing 0. approved: Nikhil 2026-10-03 blanket Refs: TH-8324
Spans were started but never made current, so an HTTP client span from the vendor call became a sibling instead of a child. Run each vendor call inside trace.use_span(span, end_on_exit=False). Stream proxies are now wrapt.ObjectProxy subclasses, so isinstance(stream, StreamSearchResponse) and the async twins still hold. approved: Nikhil 2026-10-03 blanket Refs: TH-8324
Cancellation recorded a synthetic RuntimeError, and an asyncio cancellation was reported as a CancelledError exception. Closing or garbage-collecting a stream before its end, and cancelling an AsyncExa call or stream, now set exa.cancelled=true with ERROR status "cancelled", no exception event and no document count. The fake can hold a request or a stream open so the tests cancel mid-call. approved: Nikhil 2026-10-03 blanket Refs: TH-8324
On an async stream, the vendor's close() went straight through the proxy, so the span stayed open until garbage collection; aclose() never closed the httpx response because exa-py has no aclose(). close() now ends the span as cancelled and then delegates (the vendor's own outcome is unchanged), and aclose() ends the span and awaits _raw_response.aclose(). The sync close() also ends the span before delegating. approved: Nikhil 2026-10-03 blanket Refs: TH-8324
Unit tests replaced the vendor methods with stubs, so the async wrappers and stream proxies never ran. Against the loopback fake (JSON + SSE routes): AsyncExa search/get_contents/answer/search_and_contents give the same spans as Exa; sync and async streams complete as one OK span, non-200 streams end ERROR and re-raise, and no span is ended twice (SDK warnings checked); uninstrument() restores all 12 methods by identity and no spans follow. approved: Nikhil 2026-10-03 blanket Refs: TH-8324
The exact ==2.25.0 pin forced the user's exa-py version, and BaseInstrumentor turns instrument() into a logged no-op on any other one. Declare >=2.25.0,<3 in pyproject and _instruments; tests and CI keep exa-py==2.25.0. approved: Nikhil 2026-10-03 blanket Refs: TH-8324
The >=0.1.11 floor (copied from pinecone) admits 0.1.11/0.1.12, which export to the old /tracer/observation-span/create_otel_span/ path, and the published 1.0.0, which fails while building its headers. 1.1.0 posts to /tracer/v1/traces (otel.py:719 in the PyPI wheel). approved: Nikhil 2026-10-03 blanket Refs: TH-8324
The architecture's contract test is harness run() on the shipped example; the old test re-implemented the journey inline, so the example never ran. Run it as a subprocess with FI_BASE_URL at harness.Receiver and EXA_BASE_URL at the loopback fake, and assert one OK exa.search span on the collector path. The fake now honours numResults. approved: Nikhil 2026-10-03 blanket Refs: TH-8324
The contract assertion accepted UNSET or a missing status although the wrapper sets OK. Require OK for the search and answer spans. register()'s processor also promotes UNSET to OK on export, so the wrapper's own OK stays pinned by the in-memory tests, which fail if it is removed. approved: Nikhil 2026-10-03 blanket Refs: TH-8324
Classifiers claimed 3.9-3.13; the suite runs on 3.10, 3.11 and 3.13 (PRD matrix 3.10/3.13 plus 3.11). List those three. approved: Nikhil 2026-10-03 blanket Refs: TH-8324
"Content capture is off by default" implied a switch that does not exist. Describe what each span records and never records, the get_contents URL count and capture_urls opt-in, key redaction, the 1 KB UTF-8 cap, error and cancellation status, and the tested versions. Document, and pin with a test, that a missing API key raises in Exa()/AsyncExa() before any traced call, so no span is recorded (PRD J5 assumed an ERROR span). approved: Nikhil 2026-10-03 blanket Refs: TH-8324
Add the Exa row to the root README's Python "Tools and Libraries" table and support matrix, and to python/README.md's "Tools & Integrations" table, next to the sibling tool integrations. approved: Nikhil 2026-10-03 blanket Refs: TH-8324
…apped error text Review r2 N2: spans now come from FITracer with a TraceConfig (default from FI_* env), so hide_inputs, pii_redaction and using_* context attributes apply. With hidden inputs, input.value is __REDACTED__, fi.retrieval.query and fi.retrieval.urls are not recorded, and counts stay. Review r2 N5: exa-py copies the server body into its ValueError. The status description and exception event now have the Exa key (and a hidden query) removed and the message capped at 1 KB; the stack trace is rebuilt from the frames plus that message. The caller's exception is unchanged. Also covers the reviewer's missing tests: mid-stream vendor errors (sync and async) end ERROR, not cancelled; context restored after an error; capture_urls with Result objects.
Review r2 N3: fi-instrumentation-otel 1.1.0 requires >3.9 and 3.9 is not tested. Also pins the README note that exa-py's async streams have no aclose() (r2 N1).
Review r2 N1 (aclose is added by traceAI-exa), N2 (TraceConfig and FI_HIDE_INPUTS), N4 (register with ProjectType.OBSERVE) and N5 (what error text is recorded).
Verify r2 V3 (PRD AC-09): a stream still referenced at exit was finalised after register()'s atexit provider shutdown, so its span was dropped (batch) or exported without the cancelled status (simple). Open stream states are tracked in a WeakSet and ended as cancelled from an atexit hook that instrument() (re-)registers, so it runs before the provider shutdown.
…in error text Verify r3 V4: span.record_exception() calls str(error) unguarded, so an exception whose __str__ raises skipped set_status/end and replaced the caller's exception. The event is now built with add_event from the already-guarded safe text, and end() runs in a finally. Verify r3 V5: with hidden inputs, a get_contents error body that echoes the requested URLs is scrubbed like the query (verbatim, longest first).
Verify r4 V6 (docs only): the verbatim-match limit now covers the requested URLs added in fd1c109 and escaped/normalised forms.
The suite passes on Python 3.12 (80 tests, fresh venv, published fi-instrumentation-otel 1.1.0), so list it alongside 3.10, 3.11 and 3.13. Add CHANGELOG.md as the root changelog asks of each package, and drop the package.py comment that referred to a CI job this repo does not have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ping - Record the query in gen_ai.retrieval.query (the repo constant the backend reads) and counts in exa.result_count / exa.url_count / exa.urls instead of the unmapped fi.retrieval.* keys. - gen_ai.span.kind: RETRIEVER for search, answer and the streams; TOOL for get_contents, which fetches named pages rather than searching. - Wrap only methods defined on each class (vars), so AsyncExa can never wrap an inherited Exa method twice and uninstrument() restores exactly. - A tracer that fails to start a span leaves the call untraced. - pii_redaction also covers the error status and exception event. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The catalog rows for all TH-8103 integrations now live in one PR (docs/th-8103-integration-catalog), so this PR no longer conflicts with its siblings in README.md and can merge in any order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The exception event carried scrubbed traceback frames, which include local file paths. It now carries exception.type and exception.message only. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…milar untraced Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d README true (TH-8324) exa-py 2.26.0 (2026-10-19) gives AsyncStreamSearchResponse and AsyncStreamAnswerResponse an aclose(), so test_vendor_async_streams_have_no_aclose failed on the version CI resolves. The traced aclose() already delegates to the vendor one. The test now pins the README's claim to the exa-py version, and the README, a docstring and the changelog say which version has what. Suite passes on exa-py 2.25.0 and 2.26.0 (86 each). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status 2026-10-21: merged
devinto the branch (9afda66, no conflicts). exa-py 2.26.0 (released 2026-10-19) adds its own asyncaclose(), which broketest_vendor_async_streams_have_no_aclose. c5ea631 pins that README claim to the exa-py version, and the README, a docstring and the CHANGELOG now say which version has what. The tracedaclose()already delegates to the vendor's. 27d9bb1 updates the README's tested-versions line. Suite: 86 passed on exa-py 2.25.0 and on 2.26.0. CI: Plan, fi_instrumentation, package (exa), Python tests all green. The live run happened on 2026-10-17 (docs#996), so the 'Not verified / BLOCKED-ON an Exa API key' line below is superseded.traceAI-exa0.1.0 is not on PyPI yet. After this merges, publish 0.1.0, then merge docs#996.Summary
python/frameworks/exa/, a new instrumentortraceAI-exafor the Exa client (exa-py2.25.0). It wrapssearch,get_contents,answer, the deprecatedsearch_and_contentsalias, the async twins, and the two streaming methods.input.valueand the query attribute, cut at 1024 UTF-8 bytes on a character boundary. It also records the document count; answer spans count citations, and streams add them up across chunks.x-api-keyheader, falling back toapi_key, and is replaced by[redacted]in queries, URLs and error text.get_contentsrecordsfi.retrieval.url_countby default.instrument(capture_urls=True)adds the URLs: at most 20, 1 KB each, with the key redacted.FITracerwith aTraceConfig(config=, default fromFI_*env).hide_inputs/FI_HIDE_INPUTS=truesetsinput.valueto__REDACTED__and drops the query and URL attributes (counts stay);pii_redactionandusing_session/using_user/using_metadata/using_tagsapply.str()fails.cancelledwithexa.cancelled=true. A stream still open at interpreter exit is ended as cancelled beforeregister()'s exit hook shuts the exporter down.wrapt.ObjectProxy, soisinstanceholds.close()/aclose()end the span before closing the vendor stream.exa-py>=2.25.0,<3(2.25.0 tested),fi-instrumentation-otel>=1.1.0,python>=3.10,<3.14. Classifiers list Python 3.10, 3.11, 3.12 and 3.13, all tested.traceai_exa/withpackage.py,version.py,py.typed; poetrypyproject.toml; README, CHANGELOG, example, tests) and is listed in the root andpython/README tables.Depends on #203
The contract tests use the shared OTLP harness from #203 (
python/tests/harness/). That branch (3eaadc8) is merged into this one, so this PR's diff includes the two harness files until #203 lands; the Exa package does not modify them. Merging #203 first keeps the history clean, but either order works.Spec: https://github.com/future-agi/company-brain/pull/62 (TH-8324).
Tests
On the current head, from a fresh venv per Python version (
pip install -e python/frameworks/exa pytest pytest-asyncio, which resolves the publishedfi-instrumentation-otel1.1.0 andexa-py2.25.0), run from the package directory withpython -m pytest tests -q:Earlier heads were also run with the in-repo
fi_instrumentationonPYTHONPATH(80 passed on 3.10, 3.11 and 3.13). This repo has no CI job for the Python packages, so these local runs are the evidence.exa-pyclient makes real HTTP calls (JSON and SSE) to a loopback fake of the Exa API. Spans go out through the realregister()exporter to the shared harnessReceiver, on/tracer/v1/traces, with both auth headers andproject_type=observe.str()raises (sync, async, stream): re-raised unchanged, span endedTraceConfig(hide_inputs=True)andFI_HIDE_INPUTS=true, incl. streams, URLs and error text;using_sessionResultobjects)aclose()uninstrument()restoring all 12 methodsregister()turns an UNSET status into OK before export, so the wrapper's own OK status is asserted by in-memory tests, not the contract test.Review
Four review rounds by separate reviewer and verifier agents. Round 1 requested changes (15 findings: citation counts, real-SDK tests, URL count and opt-in URLs, header-held key, stream close order, version ranges, example run, cancellation, current span and
ObjectProxy, docs, contract OK status,input.value, missing-key docs, UTF-8 cap). All were fixed test-first, and rounds 2 to 4 approved/verified with only non-blocking findings (TraceConfig support, Python floor, key-safe error text, exit-time stream export,str()-raising exceptions, URL scrubbing with hidden inputs, README wording), each also fixed test-first. For each round the new tests fail against the previous head's code.Video demo
Narrated real-terminal walkthrough, 8:20, 12 chapters, recorded at
338560d. The package code and behaviour are unchanged since; the only later commit adds the 3.12 classifier, the package CHANGELOG and a comment fix. The video is private to Future AGI and attached to Linear issue TH-8324 with captions, transcript and chapter list. It runs the realexa-py2.25.0 client against the package's loopback fake Exa API, with placeholder keys only.Limits
python/onPYTHONPATH, so it imports the in-repofi_instrumentationrather than the published one (the other 79 tests use the published 1.1.0).fi.retrieval.*was not checked. Whether live Exa repeats citations across stream chunks is unverified.instrument()followsregister(), as in the README.exais not added to thetraceaimeta-package'sauto_instrumentregistry (python/traceai/__init__.py); that list already omits most non-LLM instrumentors.test_instrumentor.pyis kept.AI use: implementation, review fixes and tests written with AI coding agents under Rick (Future AGI's engineering agent) and reviewed by separate reviewer/verifier agents.
Verification sweep (2026-10-12)
dev, and the spans were checked in ClickHouse and in the Observe UI (hierarchy, span types, input/output, model and tokens, sessions, error status, and a secret scan of everything stored).gen_ai.retrieval.query; counts areexa.result_count/exa.url_count(exa.urlswithcapture_urls);get_contentsis a TOOL span; the kind key isgen_ai.span.kind.Re-verification (2026-10-16)
exa-py2.25.0 (PyPI latest on 2026-10-16, also the pinned version, so one leg);fi-instrumentation-otel1.1.0; Python 3.10, 3.12, 3.13 (86 passed on each); local Future AGI.exa-py2.25.0 client (ExaandAsyncExa) calling the package's loopback fake of api.exa.ai (no Exa key exists;api.exa.aiwas not called),register()exporting uncompressed OTLP to local Future AGI, projectth8103-rick-exa. One agent span withsearch,get_contents,answer,stream_searchand a 401 failure case as children, plus anAsyncExa.searchroot trace.629b817b-bd88-4500-8c66-dab0bf490a33(6 spans):[agent] research-agent>[retriever] exa.search,[tool] exa.get_contents,[retriever] exa.answer,[retriever] exa.search(stream),[retriever] exa.search status=ERROR: ValueError: Request failed with status code 401. Trace94968785-2ad7-47ac-82e0-b541353009a0(asyncexa.search). No model or tokens (Exa returns none; none invented). UI screenshot checked: tree and agent graph show the hierarchy and the one error.TraceConfig). WithFI_HIDE_INPUTS=true(projectth8103-rick-exa-hidden), the marker appears in 0 of 7 rows across all stored columns (incl. resource attrs and span events). The fake key, the fake's response-content markers,/Users/, user name,TracebackandFile "appear in 0 rows in both modes.exceptionevent carriedexception.stacktracebuilt from traceback frames, i.e. local file paths (user name, host). Future AGI storage drops event attributes, but other OTLP receivers would see them. Commitb44e450removes it: the event now hasexception.type,exception.message(key-scrubbed, 1 KB cap) only. Tests updated; a new test checks that noTraceback,File ",site-packages, user name or host name is on the wire; 8 tests fail on the previous code, 86 pass now.main, no navigation.ts), pagesrc/pages/docs/integrations/traceai/exa.mdx; suggested nav group "traceAI" (Python frameworks), title "Exa".Live vendor verification (2026-10-17)
api.exa.ai, no base-URL override),exa-py2.25.0 (PyPI latest),fi-instrumentation-otel1.1.0,traceAI-exa0.1.0, Python 3.12, run exactly as the README shows, exporting to local Future AGI (projectth8103-rick-exa-live). Spend: a handful of search/contents/answer calls, well under $0.50.fa0ce113-0c73-48a7-bbd8-6852d6062ddb(6 spans):[agent] research-agent>[retriever] exa.search,[tool] exa.get_contents,[retriever] exa.answer,[retriever] exa.search(stream_search),[retriever] exa.answer(stream_answer).exa.result_count1, 1 (+exa.url_count1), 8, 0, 8. Traceedad9ed7-1183-4554-9b15-9aa2e3b20a38:AsyncExa.search.14aa18fd-a5e5-4a5f-a019-3ba6faf7cd7b, invalid key:[retriever] exa.search status=ERROR: ValueError: Request failed with status code 401: ... INVALID_API_KEY; exception event has no stack trace.FI_HIDE_INPUTS=true(traces967cfabf-72c6-4813-86bc-9b13de89bfc2,2fc3dcb4-bb86-4a8d-8bb4-ec8c50f247dc): marker in 0 of 7 rows,input.value=__REDACTED__. The Exa key,/Users/, user name, host name,TracebackandFile "appear in 0 rows in both modes.stream_answerdoes not repeat citations across chunks (span counted 8, same as the non-stream answer). Observed:stream_search(num_results=1)returned no chunks live, so its span hasexa.result_count0. Exa returns no model/tokens; none recorded.capture_urls=Truelive, Exa error shapes other than 401,find_similar*(not traced, documented).🤖 Generated with Claude Code