Skip to content

feat: record time to first token on the OpenTelemetry chat span - #447

Open
cpsievert wants to merge 1 commit into
mainfrom
feat/otel-time-to-first-token
Open

cpsievert wants to merge 1 commit into
mainfrom
feat/otel-time-to-first-token

Conversation

@cpsievert

Copy link
Copy Markdown
Collaborator

Ports tidyverse/ellmer#1142.

Summary

Streamed responses now record the time to first token (in seconds) as the gen_ai.response.time_to_first_chunk attribute on the OpenTelemetry chat <model> span.

  • chatlas/_otel.py: record_chat_ttft_attr() sets the attribute on recording spans, given a start timestamp.
  • chatlas/_chat.py: both sync and async streaming loops capture stream_start just before chat_perform() (which blocks until headers arrive, so TTFT is measured from request issuance) and record the attribute once, at the first chunk carrying non-empty text or thinking content (via the existing display_text() helper).
  • Tests: offline mocked test asserting TTFT is recorded at the first non-empty text token (not merely the first chunk), plus TTFT assertions in the VCR streaming-span test and the async stream-iteration test.

Two small adaptations from the R implementation: time.monotonic() is used instead of wall-clock time since this measures an elapsed interval, and the mocked empty first delta uses ContentText.model_construct(text="") (as real providers do) because ContentText.__init__ normalizes empty text to "[empty string]".

Verification

pytest tests/test_otel.py — all 24 tests pass, including the new test_ttft_recorded_at_first_non_empty_text_token and the new TTFT assertions on sync/async streamed chat spans.

Streamed responses now record the time to first token (in seconds) as
the `gen_ai.response.time_to_first_chunk` attribute on the `chat` span.
It is measured from request issuance (`chat_perform()` blocks until
headers arrive) and recorded once, at the first chunk carrying non-empty
text or thinking content.

Ports tidyverse/ellmer#1142.
@cpsievert
cpsievert marked this pull request as ready for review October 7, 2026 15:04

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant