Repository navigation
feat(replicate): traceAI-replicate native client tracing (TH-8320) - #219
Merged
Merged
Conversation
The Node OTLP exporter streams with Transfer-Encoding: chunked and sends no Content-Length. The Receiver read Content-Length only, so every Node export came back 400 with zero spans; two TH-8103 children had to put a de-chunking relay in front of it. Receiver now accepts both framings. It also records one entry per accepted export (path, lower-cased headers, flattened resource attributes) via requests(), so a contract test can assert X-Api-Key/X-Secret-Key and project_name through the shared harness instead of a private recorder. Verified: 6 harness tests pass, and the TanStack example's real Node exporter delivered 4 chunked exports (4 spans, collector path, both auth headers, project_name/project_type) with no relay. Refs: TH-8339, TH-8103
Real replicate 1.x client calls against FakeReplicate (create via version, official-model and deployment routes, get, cancel, SSE stream, versions, trainings) through a recording httpx transport that refuses any non-loopback host. Covers run/async_run, create+wait (one span), create without wait, cancel, streams and iterators, token redaction, TraceConfig hide flags, failure isolation, uninstrument identity restore, packaging ranges, the TH-8339 Receiver contract and the example script. RED: traceai_replicate does not exist yet. approved: Nikhil 2026-10-03 blanket Refs: TH-8320
…client ReplicateInstrumentor wraps replicate 1.x (>=1.0.0,<2): - Client.run/async_run -> replicate.run; Client.stream/async_stream -> replicate.stream (one span per stream, open until it ends, is closed or is dropped; wrapt.ObjectProxy keeps the vendor generator type). - Predictions/ModelsPredictions/DeploymentsPredictions/DeploymentPredictions create (+async) -> replicate.predictions.create. A non-terminal prediction keeps the span open on the returned object so wait/cancel end it (one span, AC-03); otherwise it ends with the create-time status and end time when the object is released, at uninstrument() or at exit (AC-02, J2.4). - Prediction.wait/async_wait and cancel/async_cancel, Predictions.cancel (+async): continue an open create span or open one span; the client's poll loop never adds spans. trainings, get, list and reload are untouched. - replicate.run & co. (bound to default_client at import) are rebound through the wrappers and restored by identity. Spans are current during the vendor call so HTTP spans nest; nested wrapped calls are suppressed with a ContextVar. Attributes: provider, model, prediction id/status/version, predict_time, output type, input/output via TraceConfig; no token usage or cost. The API token is redacted wherever the client stored it; the environment is never read. Caller cancellation -> ERROR "cancelled" + replicate.cancelled; a canceled prediction is a terminal state, not an error (J3.3, AC-04). Test corrections found while running against the real SDK: replicate 1.0.x drops wait= when predictions.create delegates model=, so the Prefer-wait tests use the version route; pickling/copying are compared with a plain Prediction (the client's httpx transport is not picklable); uninstrument checks use identity (a FunctionWrapper == its function). Added coverage: chained async create().async_wait(), a missing vendor method at instrument time, a no-environment-read source check, and a subprocess exit-without-wait export through the Receiver. approved: Nikhil 2026-10-03 blanket Refs: TH-8320
…1.0.0 floor - A file handle in the prediction input is recorded as "<BytesIO>"; the client still reads and inlines it (file_encoding_strategy="base64"). - The example test accepts replicate 1.0.0, which sends no Prefer: wait by default and so polls; 1.0.7 does not. The span contract is unchanged. approved: Nikhil 2026-10-03 blanket Refs: TH-8320
Documents the span per call (run, stream, create held open for wait/cancel, wait, cancel), every attribute, TraceConfig content control, token redaction, file/data-URI handling, webhook presence-only recording, error/cancellation semantics, and the known limits (Prediction.stream and output_iterator not wrapped, no Cancel-After option in replicate 1.x, env-only token before the first request). approved: Nikhil 2026-10-03 blanket Refs: TH-8320
One row in the root README's Python "LLM Providers" table and one in python/README.md's "LLM Providers" table. approved: Nikhil 2026-10-03 blanket Refs: TH-8320
…lock Runs in a subprocess through harness.run: a PendingPrediction released while its thread holds the pending-span registry lock (as the cyclic GC can do) must end its span without deadlocking. RED on the non-reentrant lock (the subprocess times out at 30 s). approved: Nikhil 2026-10-03 blanket Refs: TH-8320
A PendingPrediction finalised by the cyclic GC inside a registry operation ends its span and calls discard() on the same thread; with a plain Lock that deadlocks. approved: Nikhil 2026-10-03 blanket Refs: TH-8320
…shuts down register() installs SIGTERM/SIGINT handlers that shut the tracer provider down and then sys.exit(0). Held create spans were ended only from an atexit hook, which runs after the processors have stopped, so every span held for a prediction nobody waited on was dropped (batch) or swept up without its prediction attributes as gen_ai.span.leaked (simple). instrument() now replaces force_flush and shutdown on the provider instance it was given with a hook that ends every held create span (as of create time, with the create-time status) and then calls the original. uninstrument() puts back exactly what was on the instance before (by identity), leaves a hook someone layered on top in place as a pass-through, and providers without these methods (no-op, proxy) or with read-only attributes (__slots__) are left untouched without raising. The atexit hook stays as the fallback. A span still ends exactly once. Tests: test_signals.py runs register() + the real OTLP exporter into the harness Receiver in a subprocess, holds a create, and sends SIGTERM or SIGINT (batch and simple processors). test_provider_hooks.py covers force_flush/shutdown export without release, a later wait() getting its own span, and identity restore. approved: Nikhil 2026-10-03 blanket Refs: TH-8320
…nds) Rick accepted the held-open create span (one span for create + wait, AC-03) as a deviation from spec J2.4, with a cap on how long delivery can be deferred. A create span that no wait()/cancel() continued is now held for at most max_pending_seconds (default 600). An older one is ended, with its create-time status and create-time end timestamp, the next time the registry is touched: any traced call (sync or async), force_flush or shutdown. A later wait() on that prediction gets its own replicate.prediction.wait span. The registry keeps entries in hold order so the check stops at the first young entry. instrument(max_pending_seconds=...) takes a positive, finite int or float; anything else raises TypeError (type) or ValueError (value) before anything is patched. The README states the J2.4 deviation and why (AC-02 vs AC-03), the cap and the flush/shutdown behaviour. approved: Nikhil 2026-10-03 blanket Refs: TH-8320
PendingPrediction, TracedIterator and TracedAsyncIterator ended their span inside __del__. The cyclic GC can run __del__ on any thread at any allocation, including while that thread holds a non-reentrant lock on the export path (register(batch=False) exports inline through the HTTP pool), so a finaliser that runs the processors can deadlock. __del__ now only appends (call, cancelled, release time) to a module-level deque (deque.append is atomic). The queue is drained, and the spans ended, at a safe point: on every traced call's entry, on registry operations (add/get/touch/expire_all), in the hooked provider force_flush/shutdown, at uninstrument() and at exit (the instrumentor's atexit hook, plus a module-level drain that stays registered for a stream released after uninstrument). Spans still end exactly once and keep their original end timestamps: create time for a held prediction, drop time for an abandoned stream or iterator. Tests: test_release.py records span.end calls and shows __del__ makes none (RED before: ['MainThread'] == []) for a held prediction, a sync stream, a run iterator and an async stream, then drains through the next traced call, force_flush, uninstrument and shutdown. test_exit.py and test_signals.py gain released variants (drain at exit / at the signal). Two existing GC tests now drain explicitly with drain_released(): test_predictions.py::test_create_without_wait_is_one_span_ended_at_create_and_never_polled and test_streams.py::test_dropping_a_stream_mid_iteration_ends_its_span_as_cancelled. approved: Nikhil 2026-10-03 blanket Refs: TH-8320
…e 1.0.0/1.0.7
Rick ran the full suite on every cell of {3.10, 3.11, 3.12, 3.13} x
{replicate 1.0.0, 1.0.7} (79 passed at 370f7ef; rerun at the final head of
this round). The README said 3.10, 3.11 and 3.13 and the classifiers left
out 3.12, which poetry-core would add anyway from python = ">=3.10,<3.14".
README and classifiers now name 3.10, 3.11, 3.12 and 3.13 with replicate
1.0.0 and 1.0.7. test_packaging.py pins the classifier list to that matrix
and checks that the README sentence states it.
approved: Nikhil 2026-10-03 blanket
Refs: TH-8320
….cancelled R5: with Prefer: wait (the default wait=True in replicate 1.0.7, an explicit wait= in 1.0.0) the client's run() returns without polling once the create response is no longer 'starting', and the server can answer 'processing' for a finished prediction (vendor run.py). The status row now says a replicate.run span can show 'processing' on an OK span with output. V1: the replicate.cancelled row said 'true when the caller cancelled', which an explicit cancel() does not set. It now reads: true when the caller abandoned the call (closed/dropped an iterator, cancelled a task); an explicit cancel() is recorded as status canceled. Docs only; no code change. approved: Nikhil 2026-10-03 blanket Refs: TH-8320
Verifier finding V2 (t_053f4104): LifecycleWrapper found the held create span with registry.get(), which left it registered with its create-time held_since. While wait() polled, a traced call on another thread or task (the max_pending_seconds cap) or provider.force_flush() ended the span as of create time with status 'starting'; when the wait returned, finish() was a no-op and the final status, output and duration were lost. wait()/cancel() now claim the span: registry.claim() pops it from the held set under the lock and records it as claimed, so the cap, a flush and a release of the prediction object leave it to that call, which ends it with its result. Provider shutdown and interpreter exit (end_all) still end a claimed span as a best effort, as of that moment with the last polled status, so it is exported before the processors stop; the call's later finish() ends nothing. force_flush and uninstrument() keep ending only held spans. finish() is an atomic non-blocking test-and-set, so a shutdown run from register()'s signal handler while this thread is inside another finish() returns instead of deadlocking. Tests first (tests/test_inflight_claim.py, 18 cases): cap and flush during a running wait and cancel (sync, async, by id), a release while a cancel by id runs, shutdown during a running wait (sync, async) and SIGTERM/exit under register() with batch and simple processors. Against 0bcc3f5: 18 failed (13 V2 early ends, 5 shutdown ended as of create with 'starting'). Full suite after: 136 passed (Python 3.11, replicate 1.0.7). approved: Nikhil 2026-10-03 blanket Refs: TH-8320
README: while wait()/cancel() runs, the cap, a flush and a release leave its create span alone; shutdown and exit end it as a best effort with the last polled status. New 'Which provider' section: without tracer_provider= the hooks go on the global provider at instrument() time, so call register() before instrument() (the placeholder provider has no force_flush/shutdown to hook). Verify-r2 t_053f4104 P3 follow-ups. approved: Nikhil 2026-10-03 blanket Refs: TH-8320
Replicate's language models report metrics.input_token_count and metrics.output_token_count. LLM spans were showing 0 tokens; record them as gen_ai.usage.input_tokens / output_tokens / total_tokens when present (never estimated, no cost). This reverses the earlier no-token rule, which assumed Replicate reports none. Also apply pii_redaction to the error status and the exception event, and add a CHANGELOG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The catalog rows for all TH-8103 integrations now live in one PR (docs/th-8103-integration-catalog), so this PR no longer conflicts with its siblings in README.md and can merge in any order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… name) span.record_exception() always adds a Python traceback, which carries the caller's file paths, user name and virtualenv. traceAI never exports those, so the exception event is now built with add_event and keeps only exception.type (module.QualName, bare for builtins, as the SDK writes it) and exception.message (token-redacted, PII-redacted, __REDACTED__ when outputs are hidden). Found in the 2026-10-16 re-verification: the OTLP body sent to Future AGI carried /Users/<name>/.../site-packages frames. tests/test_no_stacktrace.py fails on 3e6d005 (4 failed) and passes now: ModelError (default and hidden), ReplicateError, ValueError and a stream RuntimeError export no stacktrace, no Traceback, no site-packages path, no test-file path and no user name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cation notes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nal status and usage e7e16c4's title announced dropping the placeholder version "hidden", but its diff changed README.md only: the code kept recording it, and the rows stored in the 2026-10-20 live run carry replicate.prediction.version="hidden". This commit makes the change in code: - real_version() drops "hidden" (and empty or non-string values) wherever a version is read: the prediction a call returns (run, async_run, stream, the held create snapshot, wait, cancel), a "owner/name:hidden" ref, a ref tuple, and a Version object. A real version id is still recorded. Streams: the SSE stream carries no status or metrics, so stream spans had no status, predict_time or token counts. After a completed stream (StopIteration after "done") the prediction is now read once with predictions.get / async_get (GET /v1/predictions/{id}: free, creates nothing), and once more poll_interval later if it is not finished yet. New option stream_reload_seconds (default 2.0, 0 = off) bounds it: the sync read runs on a daemon thread joined for the budget (the client's RetryTransport honours an unbounded Retry-After), the async read uses asyncio.wait_for. The span keeps the stream's end time. No read after an error, close() or a drop. _write keeps a terminal status on a stream span and still drops the create-time one. Tests: tests/test_live_shapes.py (29 tests; 21 fail on e7e16c4) and an official-model route in the stand-in that answers version "hidden" and lets a stream's prediction finish at "done", as the live API did. README / CHANGELOG: the option, the stream row, the version row, the CHAIN kind of a failed call, 429 behaviour, and the live-verification section restated to the stored evidence (second live session 2026-10-20 14:59 UTC: async_run, stream with the read after done, and HTTP 422 stored). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e is completion
The read after a completed stream (GET /v1/predictions/{id}) ran inside the
caller's last next(): up to stream_reload_seconds of added latency, a
caller's asyncio.timeout tripped by it, RetryTransport retries with an
uncapped Retry-After on a thread left running past the join, an HTTP child
span starting after the parent's end, and a KeyboardInterrupt in join()
leaking the span. A caller that broke at done and closed the stream got an
ERROR "cancelled" span with no output and no read.
- A stream completes at its done event (or the end of iteration). From then
a close(), a drop or a later exception leaves the span OK with its output.
- complete_stream() fixes the end time and returns at once; a finisher
thread starts a reader and ends the span when the read returns or the
budget runs out, as of the stream's end. Async streams use the same
threads.
- The read: one GET, one more poll_interval later only when the prediction
is not finished; sent to the transport under replicate's RetryTransport
(no retry, Retry-After ignored); every phase given at most the time left;
body capped at 2 MiB and dropped at the deadline; OTel instrumentation
suppressed, so no HTTP span; only id/status/version/model/metrics/error
kept. At most 256 reads in flight; past that a stream ends without one.
- force_flush/shutdown/uninstrument/exit (atexit) wait for reads in flight
at most until their deadline, then end the spans; a BaseException while
waiting ends them first.
- Tests (test_stream_read_offpath.py, 22): 21 fail on 18f7f8b. The stand-in
transport now honours instrumentation suppression like a real HTTP
instrumentation; the fake can send Retry-After on a refused read.
- README/CHANGELOG: the read's cost and rate-limit note, how to turn it off,
the third live session, and the stray "before the fix below" wording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round-2 review of traceAI#219 (ab5f5c4). The read after a stream moves to a new module, traceai_replicate/_stream_read.py: - Completing a stream is one SimpleQueue.put: no thread starts on the caller's thread or loop (only the finisher, once per process), so the next() that returns done no longer pays two Thread.start handshakes. - With replicate's own transport (no transport=/mounts=), the read goes through a package-owned httpx pool (16 connections) on a package event loop, with the client's base URL, headers and proxy options. It never borrows the caller's pool and never builds a sync httpx client on an async-only Client. The budget is an absolute deadline (an anyio cancel scope: a plain asyncio cancel was absorbed inside httpcore's own scopes under 100 concurrent trickling reads), and name lookups run on at most 2 daemon resolver threads. - Through a caller transport: at most 4 live reads on daemon reader threads (or a task on the caller's loop for an async-only transport, which is now read instead of silently skipped), none on an httpx pool under 8 connections. The cap counts reads until their I/O returns. - No read for a non-recording span, after provider shutdown, or without a provider hook; calls under the caller's suppress-instrumentation context pass through untraced. - force_flush(timeout_millis) waits at most half its timeout, cuts what is left (span ends with what is known) and passes the remaining time to the provider's flush; shutdown/exit wait at most the budget; uninstrument() does not wait. stream_reload_seconds is capped at 10. - replicate.stream.read / done_reason / truncated; done reason canceled records status canceled; aborted is a final status. - Accept-Encoding: identity and a bounded decompressor; a closed client is not read; a forked child never ends its parent's spans; a stream that is never iterated has no span (the span starts at first next()). Tests: test_stream_read_bounds.py ports the verifiers' probes (28 of 31 fail on ab5f5c4); FakeReplicate gains trickled reads and truncated, canceled and aborted streams. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… pool Fourth live session (one paid prediction, meta/meta-llama-3-8b-instruct, max_tokens 24) on 22832fd: client.async_stream on a client with replicate's default transport. The __anext__ that returned done took 0.42 ms; the read went through the package's own connection pool and the span ended on the finisher thread 104 ms after the loop exited; no sync httpx client was built on the Client. Stored span: LLM, OK, succeeded, predict_time, 39/17 tokens, the only span in its trace (trace b3858eab-adff-4d03-8720-3d56d459f054). Eleven predictions since 2026-10-20 in the account's list; 137 stored rows scanned, the token appears 0 times. async_stream moves from stand-in only to live-verified; the read through a caller's transport is stand-in only in this release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hare name lookups Review round 3 of the read after a stream (TH-8320): - A cancel from the caller's loop (asyncio.run() ending) is never swallowed: the read ends there (timeout, or skipped if it never ran) and no second GET is sent. A plain httpx.HTTPTransport / AsyncHTTPTransport (no proxy) passed to replicate.Client is read through the package's own pool with that transport's TLS context, HTTP versions, local address, socket options and connect retries, so asyncio.run() does not wait and the caller's pool is never lent. Package clients dropped from the cache close only once idle. - force_flush() no longer waits for reads in flight: their spans end at once (replicate.stream.read = flushed) and that flush exports them. The new stream_flush_wait_seconds (0 to 10, default 0) opts into a wait, never more than half the flush's timeout. shutdown() and exit still wait up to the budget. The time passed to the provider's own flush is rounded up (a 1 ms flush is no longer cut to 0) and replaces the caller's timeout whatever the original's signature; the hook keeps the original's name and signature. - Name lookups wait for a free resolver within the read's deadline instead of failing when more than two overlap; reads of one name share one lookup, kept 5 s. A new lookup fails at once only while both resolvers have been stuck for 5 s. - Completing a stream is one SimpleQueue.put (no GIL release). The worker threads start in instrument(); a dispatcher hands reads to the read loop in batches, and the finisher only ends spans, so an inline exporter no longer delays the start of reads. Admitted spans not yet ended count against the 64 cap, the own pool has 64 connections (one per admitted read), and the caller-transport cap is 4 per transport (16 in all). - A KeyboardInterrupt while the read is planned ends the span and frees its slot. A fork waits for the finisher to finish exporting a span. An ended span no longer stays referenced until its read's deadline. tests/test_stream_read_round3.py ports the verifiers' probes (27 tests; run against ee9cb13's package: 23 fail, 20 of them for the behaviour and 3 only because they use the new option or function; 4 are guards that pass on both). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README and CHANGELOG: force_flush() does not wait for reads by default (replicate.stream.read = flushed; stream_flush_wait_seconds opts in), a plain httpx transport is read through the package's pool, an async-only transport's read ends with its loop, name lookups are shared and wait for a resolver, 64 stream spans at most wait at once (one connection each), and 4 caller-transport reads per transport (16 in all). The caller-cost figure names its setup and load. The flush promise is qualified for inline exporters; the 2 MiB cap for transports that read whole bodies. Live section: five sessions, twelve predictions. The third and fourth sessions are labelled with the code they ran (ab5f5c4, 22832fd); the Accept-Encoding sentence no longer reads as a live observation. Session 5 (02:37 UTC on 2026-10-21, code b5df956): async_stream inside asyncio.run() on httpx.AsyncHTTPTransport(retries=2), read through the package's pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ds only through the package's pool Round 4 of the TH-8320 review. stream_reload_seconds now defaults to 0: a stream span ends at done (or the end of iteration) on the caller's thread, OK, with what the stream gave and no status or token counts. No request is made after a stream, no thread is started, and force_flush, shutdown, uninstrument, exit and fork wait for nothing. With the opt-in (stream_reload_seconds > 0): - the read runs only through the package's own pool, for replicate's own transport or a plain httpx.HTTPTransport / AsyncHTTPTransport without a proxy; any other caller transport is never called for it (replicate.stream.read=skipped, replicate.stream.skip_reason= custom_transport). The reader-thread and caller-loop routes are gone, so no read runs on the caller's thread or event loop; - shutdown() and interpreter exit no longer wait: pending stream spans end at once (replicate.stream.read=shutdown); stream_shutdown_wait_seconds (0 to 10, default 0) opts into a bounded wait; - uninstrument() ends pending stream spans at once (uninstrumented), so a shutdown right after it exports them; a stream completing after it makes no read (after_uninstrument); - the name-lookup queue is bounded at 64; - every skipped span says why (replicate.stream.skip_reason). Fork: only the finisher thread is coordinated; docs no longer imply span processors are fork-safe. tests/test_stream_read_round4.py ports the round-3 verifiers' probes (mock transport, uninstrument then shutdown, asyncio.run, dns queue) and checks the default literally (threads before and after, in a subprocess, against an uninstrumented run). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er is exporting With the opt-in read, force_flush(), shutdown(), exit and uninstrument() end pending stream spans at once (d610ede). A span the finisher thread was exporting at that moment was not waited for, so a process exit right after a flush could cut it off: the round-4 fork probe lost 2 of 100 child spans that way. That one export (the span has already ended) is now waited for, at most 5 s. With the read off there is no finisher and nothing waits. test_with_the_read_on_a_span_the_finisher_is_exporting_is_not_cut_off (shutdown, force_flush, uninstrument) fails without the fix. The per-call asyncio.run() test now uses a 2.5 s read and checks the median, so machine load cannot be mistaken for a wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (default settings) README: the opt-in read's caller cost, burst and slow-exporter figures, re-measured with the round-3 verifiers' probes on a git archive of f1866eb (load average 11 to 33; outputs in livefill/TH-8320/ev/r4); with an inline exporter, shutdown and exit still take the export time of the spans waiting (at most 64), which is export, not a wait for a read. A pointer near the top says stream spans carry no status or token counts by default. Live, session 6 (05:15 UTC on 2026-10-21, code f1866eb): sync replicate.stream with the default settings. The span ended on the caller's thread inside the next() that returned done (end time 0.27 ms before it returned), no request after the stream (counted at httpx's transport classes), no thread started, shutdown() 0.15 ms; the stored span is OK with no status, tokens or replicate.stream.* attribute, alone in its trace. Thirteen predictions in all; 145 stored rows scanned. CHANGELOG: the inline-export note for shutdown and exit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…HTTP mock or a changed transport Default path: - A text chunk that starts with https://, http:// or data: (or the FileOutput replicate 1.0.7 makes of it) stays in output.value, in order, for streams, run iterators and list outputs. Output is files only when every chunk is a URL or a file; a text stream reports no file count. - A KeyboardInterrupt while the span's attributes are written, or while the done event's data is read, still ends the span. - send()/throw() and asend()/athrow() step a stream through the same span logic as next()/__anext__(). Opt-in read (stream_reload_seconds > 0): - No read while httpx's or httpcore's send path is replaced on the class (respx, pytest-httpx, VCR.py, mock.patch): skip_reason patched_http, checked when the stream completes and again before each request. opentelemetry-instrumentation-httpx's wrapt wrappers are allowed. - Only replicate's own RetryTransport class is looked through, not a subclass; a transport or pool with handle_request set on the instance, and a client with proxy=, get custom_transport. - respx is a dev dependency for the new tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…igures and a seventh live session - CHANGELOG: the default-off bullet leads Added, then the opt-in and its limits (patched_http, the stricter custom_transport rule, the exit bound). - README: the opt-in's skip rules and HTTP-mock check; the exit cost with an inline exporter stated as up to 64 times the per-span export time, with figures re-measured at 4767825 (and the slow-exporter rate with each skip's reason); a stream dropped after uninstrument(); a flush's inline export of dropped streams; URL-like text chunks; send()/throw(). - Live session 7 (code 4767825): async_stream with the default settings; requests counted through shutdown() and after it. The opt-in read as it is now is marked as stand-in only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it memo; subclass resolver skip - A stream or run iterator that stops before it completes (close, drop, cancelled task, Ctrl-C, an exception, or still open at shutdown/exit) keeps the output it had yielded (subject to TraceConfig), so a text stream stays LLM; status unchanged. Follows the OTel GenAI conventions on partial streamed output. - KeyboardInterrupt / SystemExit inside next() / __anext__() is recorded as the caller stopping the stream (replicate.cancelled), as one in the loop body is. - A fork whose wait for the finisher's export or the read loop ran out does not make later forks wait again until that thread has moved on. - _read_routing never calls a _transport_for_url overridden by a subclass of httpx's client: no read, custom_transport. - Tests for each (failing first on e288580), for Sentry's HttpxIntegration (patched_http), truststore.inject_into_ssl (patched_socket) and instrument_client (custom_transport); the 100-thread trickling-read test gives each thread its own client (httpcore 1.0.9 pool race on a shared client, not the read). - README/CHANGELOG: uninstrument() waits for no read but, like flush, shutdown and exit, up to 5 s for the finisher's export; skip-reason table and stub-server note; no evidence paths or worktree figures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mns privacy scan The eighth session streamed one prediction through replicate.stream with the read off, importing the package from a git archive of f0376ad: span ended at done on the caller's thread, nothing sent after the stream, no package thread, LLM/OK with no status or tokens. README only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ller stops alike everywhere - A traced call still running (run, async_run, a create, wait or cancel) at provider shutdown, including register()'s SIGTERM/SIGINT handler, or at interpreter exit now ends then as cancelled, with what it knows, so it is exported before the processors stop. An in-flight run()/async_run() or create span was lost there before, and a running wait() ended OK. - Ctrl-C/SystemExit during run(), async_run(), a create, wait() or cancel(), a trio cancellation, and throw(GeneratorExit) are recorded as the caller stopping the call (replicate.cancelled, no exception event), as inside a stream's next(). - A close() the generator refuses because another thread is inside next() no longer ends the span; that stream ends it. - Tests: test_review_round8 (21, 20 fail on 55c5672 for the reason each targets; the stream-under-SIGINT case is a control). The 200-thread tail bound is now every tail below the read budget (2 s) and the median below a quarter of it, after a 0.54 s outlier on CI run 39587088349. - Dev dependencies: opentelemetry-instrumentation-httpx (CI skipped those tests) and trio. - README/CHANGELOG: fork() ends no span; dead docs link replaced by the PR links; the privacy scan listed views and dictionaries, did not read them; the h3 cause is worded as most likely. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… after it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README said in one place that such a stream ends at the next traced call or at exit, and in another that a dropped one ends only at exit. Both were wrong in one case (probe on e29fcc6): a held stream is not ended by the next traced call, and a dropped one is, once instrument() runs again. Both passages now say: it ends when closed or completed; once dropped, at the next traced call (after instrument() again) or at exit; force_flush() and shutdown() no longer end it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stream spans carry no status,
predict_timeor token counts by default. Withstream_reload_seconds=0(the default), areplicate.streamspan ends on the caller's thread atdone(or the end of iteration) with the prediction id and the stream's output, and withoutreplicate.prediction.status,replicate.metrics.predict_timeorgen_ai.usage.*. No request is made after the stream and no package thread is ever started, soforce_flush(),shutdown(),uninstrument(), interpreter exit andos.fork()wait for no read and no package thread (a fork ends no span).force_flush(),shutdown(),uninstrument()and exit still end, inside the call, the spans that are due (held create spans and the spans of streams dropped since the last traced call); with an inline exporter those are exported there, even byforce_flush(timeout_millis=0).A call still running at shutdown or exit is exported, as cancelled. A traced call whose vendor code is still running (
run,async_run, a create,wait()orcancel()), or a stream or run iterator still open, when the provider shuts down (including the SIGTERM/SIGINT handlerregister()installs) or the interpreter exits, ends then as ERRORcancelledwithreplicate.cancelled=trueand what it knows, so it reaches the processors before they stop.The caller stopping a call is recorded the same way wherever it lands. Ctrl-C or
SystemExitduringrun(), a create,wait()orcancel(), or inside a stream'snext(), a cancelled asyncio task or trio cancel scope,throw(GeneratorExit), and closing or dropping a stream beforedone: ERRORcancelled,replicate.cancelled=true, no exception event. A stream or run iterator that stops early keeps the output it had received (subject toTraceConfig), so a text stream staysLLM.Opt-in read.
stream_reload_secondsabove 0 (up to 10) readsGET /v1/predictions/{id}once (twice for a non-final answer) for status,predict_timeand tokens. It is a real request to the client'sbase_url, so a stub server in tests (pytest-httpserver, WireMock, Prism) receives it; keep0in those tests. It runs only through the package's own pool for a plain httpx transport picked by httpx's own_transport_for_url, and is skipped with a recordedreplicate.stream.skip_reasonotherwise:custom_transport(custom/subclassed transports, instance changes includingHTTPXClientInstrumentor.instrument_client(), a subclass's_transport_for_url, custom network backends, proxies),patched_http(respx, pytest-httpx, VCR.py,mock.patch, Sentry'sHttpxIntegration),patched_socket(mocket, httpretty, gevent, eventlet,truststore.inject_into_ssl()),at_capacity(64 waiting spans or live reads), and the lifecycle reasons. By default a flush, shutdown or exit does not wait for a read (stream_flush_wait_secondsopts in for flush,stream_shutdown_wait_secondsfor shutdown and also interpreter exit).uninstrument()never waits for a read; like flush, shutdown and exit, it waits up to 5 s for the one span the finisher is exporting. All four cancel reads in flight. The opt-in in its current form has not been run against the live API.Round 8 (commits
c5b9505,e29fcc6,a8cc5e3)a8cc5e3(README only): one rule for a stream still open atuninstrument(). It ends when closed or completed; once dropped, at the next traced call (afterinstrument()again) or at exit;force_flush()/shutdown()no longer end it. The README had contradicted itself, each passage wrong in one case (probe one29fcc6).Code, with tests in
tests/test_review_round8.pythat fail on the previous head55c5672for the reason each targets:register()'s SIGINT/SIGTERM handler the span of an in-flightrun()/async_run()was never exported (and neither was a create in flight), while a stream at the same moment was exported as cancelled and a runningwait()as OK. Each traced call whose vendor code runs is now tracked, as open iterators already were, until its span ends or a registry holds it, andend_open_iterators()(provider shutdown and exit) ends it as cancelled with the prediction it polled. Await()/cancel()that claimed a held create span is also ended as cancelled there (it was OK with the last polled status). A create that returns after a shutdown ended its span returns the client's own prediction and holds nothing. The copy of the tracked set is one C-levellist()of its weak references.KeyboardInterrupt/SystemExitduringrun(),async_run(), a create,wait()orcancel()is now the caller stopping the call (ERRORcancelled,replicate.cancelled, no exception event), as it already was inside a stream'snext(). Before, run/create/wait recorded ERRORKeyboardInterruptwith an exception event.trio.Cancelled(what anyio raises under trio) is a caller stop in the async iterator and the async wrappers, found throughsys.modules(trio is never imported by the package). Other libraries' cancellations are recorded as errors; documented.throw(GeneratorExit)/athrow(GeneratorExit)is a caller stop. Aclose()that the generator refuses because another thread is inside itsnext()(ValueError: generator already executing;RuntimeError: ... already runningfor an async generator) no longer ends the span: the stream goes on in that thread and ends its span there. The docs now saythrow()/athrow()end it with that error except forGeneratorExit,KeyboardInterrupt,SystemExitand asyncio/trio cancellations.CI and tests:
55c5672(major). Run 39587088349 (the one started by the round-7 description edit) failed one assertion:max(tails) < 0.5intest_200_concurrent_streams_on_threads_end_every_span_and_leave_no_read_runningmeasured 0.538 s on the GitHub runner; the run before it at the same SHA (39584469188) passed. The bound now follows from what the test proves: every read outlives its budget, so a loop that waited for its read would exit at least the budget after its last event. The budget there is now 2 s (the read still outlives budget + slack), every tail must be below it and the median below a quarter of it. CI status at the new head is under Verification.opentelemetry-instrumentation-httpxis now a dev dependency (CI showed 5 skipped, among them the m5instrument_clienttest), and so istriofor the new test.Docs and claims (README, CHANGELOG, docstrings; the docs page in future-agi/docs#1015):
git archive e29fcc6, naming the quoted blocks. The README still describes the ninth live stream's measured timings; those are that session's observations, not benchmark/fork figures.uninstrument()do not wait for a read by default", and its troubleshooting entry "uninstrument()never waits for a read". The README, CHANGELOG, docs page and this description were searched again for "never waits", "do not wait", "does not wait" and "waits for nothing": every remaining one is qualified (for a read; with the read off; or the memo after a timed-out wait: "the next one does not wait until the finisher has ended another span", "later forks do not wait for that thread again").force_flush(),shutdown(),uninstrument()and exit only; a fork ends no span.Verification
e29fcc6(369 before this round, plus 21 intest_review_round8.py). Fromgit archives,test_review_round8.pyand the updated in-flight tests run against the package code of55c5672: 25 fail and 15 pass. The 20 round-8 tests that fail do so for the reason each targets: ERRORKeyboardInterrupt/GeneratorExit/Cancelled: cancelled due to deadlineinstead ofcancelled(Ctrl-C in run, async_run, create and wait; trio;throw/athrow); no span exported for run, async_run and create in flight at shutdown or SIGTERM/SIGINT/exit underregister(); status OK for a wait at SIGINT; and the refused cross-thread close ending the span. The one round-8 test that passes there is a control: a stream under SIGINT was already exported as cancelled. The 5 changedtest_inflight_claim.pytests fail there because a wait running at shutdown or exit ended OK; they now expect cancelled. The 200-thread test passes on both; only its bound changed.e29fcc6. Freshgit archiveexport and freshuvvenvs, Python 3.10.17 / 3.11.12 / 3.12.10 / 3.13.15 × replicate 1.0.0 / 1.0.7: all 8 cells 390 passed, 0 failed, 0 skipped (httpx 0.28.1, httpcore 1.0.9, anyio 4.15.1, OpenTelemetry SDK 1.45.1, opentelemetry-instrumentation-httpx 0.66b1, respx 0.23.1, mocket 3.15.0, sentry-sdk 2.72.0, truststore 0.10.5, trio 0.34.0). The same matrix atc5b9505, before the live run, was 8/8 with 390 passed each.e29fcc6. Run39597696665, started by the code push, and run39598488802, started by the PR-description edit;Plan,fi_instrumentation,package (replicate)andPython testspassed in both. The prior red run on55c5672was39587088349; its failure is described above.meta/meta-llama-3-8b-instruct,max_tokens24) through the module functionreplicate.stream(sync),instrument(tracer_provider=...)with no other option, the package imported from agit archiveofc5b9505(e29fcc6changes only the README), Python 3.12.10, replicate 1.0.7, exported to the local projectth8103-livefill-th8320; the same recipe as session 8. Eventsoutputx5 anddone; the span ended on the caller's thread inside thenext()that returneddone(0.36 ms including the read of the event; end time 0.26 ms before it returned); the loop exited 0.41 ms afterdone. Counted at httpx's transport classes: the create and the event stream only, nothing after the stream, duringshutdown()or in the 0.5 s after it; thread names unchanged byinstrument(), the stream and the flush;shutdown()0.14 ms. A dry run against loopback stand-ins came first, and a ledger stops a second paid call.fi_verify(wait 240, min spans 1) PASS: the new trace holds onereplicate.streamspan, LLM, OK, with the prediction id and the joined text, tokens stored as 0 and no status,predict_timeorreplicate.stream.*attribute. The account's prediction list since 2026-10-20 went from 15 to 16.system.tables/system.columns(49 read by project id, trace id, or whole when small, no table skipped; spans with and withoutFINAL; 17 views, materialized views and dictionaries listed, not read), plus every Postgrestracer_*table (by project id,tracer_projectby id, else trace id, else whole). 99 sources read, 10 with rows, 153 rows. Counts: API token 0, its last 12 characters 0,r8_0,Bearer0,authorization0,/Users/0,/private/0, login 0, host 0,Traceback0,File "0,site-packages0,exception.stacktrace0,api.replicate.com0; the session marker 3 (spans, spansFINAL, traces). The token is counted client-side, never sent in a query. Token counts over this round's evidence files, this description, the docs PR's description, the README, the CHANGELOG and the docs page: 0 (whole token and last 12 characters; token counts were run after this description and both PR bodies were written).git archiveofe29fcc6(Python 3.12.10, replicate 1.0.7, httpx 0.28.1, httpcore 1.0.9, anyio 4.15.1, OTel SDK 1.45.1; loopback stand-in; 10-core laptop shared with other work, 1-minute load average 13.35–22.02, 5-minute 16.07–17.74, 15-minute 13.14–13.49; each output block headed with the SHA, the package tree884786e, time, load and versions). Quoted blocks:asyncio.run()per call, 10 calls, 0.2 s read, read on: p50 9 ms, max 12 ms, 10/10 readssucceeded(uninstrumented in the same block p50 6 ms, max 50 ms). [A2] read off: p50 7 ms, max 10 ms, 0 reads (uninstrumented p50 7 ms, max 44 ms).complete_stream()on the caller's thread, 500 threads × 5 runs, read on (2 s): p50 0.107–0.133 ms, p99 0.308–0.545 ms, max 0.63–3.45 ms per run; 2,500 spans OK, 320 with read data, 2,180 skipped (500 streams complete at once against the cap of 64), so these figures mostly measure the inline end of a skipped stream.done→loop end: uninstrumented 10.7–16.0 ms, read off 11.7–18.8 ms, read on 21.1–26.6 ms. [G2] a client per stream, 2 runs: 10.9–11.3 ms, 15.0–17.7 ms, 32.7–89.9 ms.at_capacity, peak 64 pending; caller loop p50 3 ms, p99 108 ms; last span exported at 15.2 s.fork()3.3 ms in the parent; the child exited 0, started with no package thread and nothing pending, AnyIO's asyncio backend had been imported before the fork, its own async httpx call finished within 8 s, its first stream took 12 ms and its read endedsucceeded3.02 s later on the child's own finisher; the parent's 10 spans endedsucceeded. [F2] Finisher stuck in an export, four forks: 5.002 s, then 0.001, 0.002 and 0.001 s.no_proxy: on macOS a forked child that looks up system proxies after the parent did (CPython's_scproxy, used by everyhttpxclient withtrust_env) crashes in the Objective-C runtime with or without the package (reproduced in round 7 with plainurllib.request.getproxies()and no httpx, replicate or traceai). Documented on the docs page.e29fcc6only in the README, amended before the push) are kept with the evidence and not quoted; the first lacked [D1] becausepsutilwas missing from the probe venv. The probe scripts, the fork probe among them, are kept with the evidence.httpcore.ReadError: [Errno 9] Bad file descriptoron the caller's create POST (round 7). Most likely an httpcore 1.0.9 pool race in the caller's own sync pool, not the read machinery; the evidence is circumstantial. The test that hit it passed 50 of 50 reruns, each in its own pytest process, from agit archiveofe288580in a fresh 3.12 / replicate 1.0.7 venv. In httpcore 1.0.9's_sync/connection_pool.py,_assign_requests_to_connectionsdecides under the pool lock to close an IDLE connection whenlen([connection.is_idle() for connection in self._connections])(which counts every connection) exceedsmax_keepalive_connections(20), and closes it after releasing the lock; a connection just handed to another thread's request is still IDLE until that thread marks it ACTIVE, so it can be closed while that request uses it. The package does no fd-level work and never uses the caller's pool. A plain-httpx reproducer, with no traceai or replicate imported (200 threads sharing onehttpx.Client, a keep-alive POST then aConnection: closeGET), hitReadError: [Errno 9] Bad file descriptorin 2 of 12 widened runs (2 of the 10 at 200 threads; one run with 1 such error, on the POST; one with 11 among 15 errors), 0 of the 2 widened runs at 100 and 150 threads; one of the two hits was an exploratory run whose console output was copied by hand, not captured by the script. Both stock runs (10 rounds each at 100 and 200 threads, 3,000 calls) had 0 errors. The strongest evidence that the read is not the cause: on the original shared-client form of the test ate29fcc6, the error appeared with the read OFF (3 of 100 runs, no package thread or pool at all), not with the read on (0 of 100), and not with replicate alone, uninstrumented (0 of 200). 3/100 vs 0/200 is only weakly significant (one-sided p ≈ 0.04), so this supports 'not the read machinery' but not 'independent of the instrumentation's timing'. The test now gives each thread its own client, so it measures the reads it is about.bf6bb646):BUILD_EXIT=0,AUDIT_EXIT=0, 0 broken nav links, 0 broken content links, 30 orphan pages: 29 on docs main plus this new page, which is not in the navigation yet (reachable by URL). The build script's output (last 25 build lines and the audit summary) is kept with the round's evidence; a full 3,726-line build + audit run atbf6bb646also exited 0 with no page-specific warnings.Background for m1: partial output
open-telemetry/semantic-conventions-genaiat8538971,docs/gen-ai/client-inference.md,gen_ai.output.messages): "When an operation ends in an error, this attribute SHOULD only be set when the operation produced output, including partial streamed output. Instrumentation SHOULD NOT synthesize output content for a failed operation." Spans end "when the response is fully received or the operation is terminated due to an error or cancellation" (gen-ai-spans.md).opentelemetry-instrumentation-openai-v2(opentelemetry-python-contribatde43f81,patch.py):BaseStreamWrapper.close()and an exception in__next__callcleanup(), which emits the text received so far ("finish_reason": choice.finish_reason or "error") before ending the span (withhandle_span_exceptionon error).Arize-ai/openinferenceatca3ed46,_stream.py/_utils.py): an exception in__next__calls_finish_tracing(status=ERROR), which writes the response accumulator's attributes (the output so far).traceloop/openllmetryat8d9c93f,chat_wrappers.py):_ensure_cleanup()on an exception,__exit__or__del__runs_record_partial_metrics(), which sets the completions "for partial streams (parity with happy path)".This package keeps the received text with the same status rules, synthesizes nothing, and leaves output empty when nothing was received (kind
CHAINthen, as for any call without output).Limits and remaining behaviour
register()handler in subprocesses, not live.provider.shutdown()) is recorded as cancelled although the call itself goes on; its later result ends no second span.uninstrument()with the read on, andtimeout_millisdoes not shorten it. With an inline exporter, ending pending spans still costs up to 64 per-span exports.uninstrument(), worker threads stay idle until exit. A stream still open afteruninstrument()ends at drop, the next traced call or exit; if the provider was shut down before that, its span is not exported.BatchSpanProcessorthreads are not coordinated. On macOS, see the_scproxynote above.KeyboardInterruptexactly insideOutputCollector.add()can leave that one chunk unrecorded (the span then ends as cancelled at drop or exit, with the chunks before it).History (condensed)
Earlier rounds moved the read after a stream off the caller's path, then made it opt-in (round 4), restricted it to the package's own pool for plain transports and skipped HTTP mocks (round 5), and added socket-mock detection, AnyIO pre-import, fork coordination and cancellation of reads in flight (round 6). Round 7 kept a stopped stream's output, recorded Ctrl-C inside
next()as a cancel, added the fork wait memo and the subclass-resolver skip, and moved every quoted figure to a measured head. The round-6 description said "12 regressions"; precisely, round 6 added 12 tests that all failed onaea2604, 10 on the behaviour they target and 2 only because a module attribute they patch or call (importlib,socket_layer_patched) did not exist there. Live sessions 1–9 are described in the README; sessions 2–5 ran earlier designs of the read, 6–7 ran with the read off on code from before rounds 6 and 7, 8 on the round-7 code, and 9 (the default path, read off) on the round-8 code.🤖 Generated with Claude Code