[nflechner]

Why

On Oct 19 a dwh-llm-forecaster run cost $1,148 for zero forecasts. A VPN closed idle connections; the synchronous responses.create call sits silent for the minutes the model works, so every call was billed server-side but never returned. The retry wrapper then re-ran each task up to 5 times (times the SDK's own 3), and nothing held a response id, so nothing could be cancelled or fetched afterwards. This PR makes an OpenAI call something the caller holds a handle to from the first second.

Paired with dwh-llm-forecaster#24, which consumes it. forecastbench pins 843ae21 and is unaffected until it bumps; when it does, it gets background mode automatically with no call-site change.

What

OpenAI provider: Responses API background mode. responses.create(background=True) returns at once; the provider polls retrieve (2 s backing off to 10 s) until the response leaves queued/in_progress. No connection is idle while the model works. The response id is handed to the caller the moment the request is accepted. A failed poll is polled again for as long as the budget lasts; the generation it watches is already paid for and is never started again because a short status request failed. MetaProvider keeps the synchronous call since the Meta route has no background mode.

Budget and cancellation. A run's timeout option is now the budget for the whole background response (default 30 min when absent); when it runs out the response is cancelled server-side and the call fails without retry. A caller's cancel event stops the call at the next checkpoint; with cancel_server_side (the default) the response is cancelled, without it the response keeps running so it can be fetched later. fetch_response(id) returns a finished response, wait_for_response(id) polls a still-running one to completion; neither ever creates a new generation.

Call metadata. New get_response_with_metadata(prompt, context=..., **options) on ModelRun, model_registry and the providers returns an LLMResponse: text, provider response id, status, LLMUsage (input/output/reasoning tokens, tool-call count), attempts, duration. get_response keeps returning the text, so existing callers are unchanged. Anthropic fills the same metadata from its streamed final message. CallContext (utils/llm/types.py) carries the cancel event, the cancel_server_side flag and the response-id hook.

Retries. Attempts are per provider: 5 for OpenAI (in background mode a retry only follows a failed create or a response that ended failed, neither billed), 2 for every other provider (a dropped stream is a billed response lost). A response that ends incomplete raises LLMResponseIncomplete and is never retried: the same request would hit the same cap and be billed again. The wait between attempts ends early on cancel. Failed attempts are logged at WARNING so they show under a caller's default logging.

Docs. README section on metadata, cancellation, fetching by id and background mode. AGENTS.md records that transport mechanics (Anthropic streaming, OpenAI background mode) are provider behaviour, not run options, so they stay out of option fingerprints and immutability, and that an OpenAI run's timeout is the whole-response budget.

Semantic changes to published runs

• The timeout: 900 on the gpt-5.6-sol variants was an SDK per-request timeout (a call cut at 15 min, retried up to 5 times). It is now the budget for the whole background response, after which the response is cancelled and the call fails without retry. Runs without timeout get 30 min (before: the SDK default of 10 min, then retries). The create/retrieve requests themselves keep the SDK default timeout.
• Background mode needs store=true, the API default; responses are retained 30 days.
• Anthropic streams cannot be cancelled mid-call; the cancel event is honoured between retries only.

Commits

• 650127e feat(llm): background-mode OpenAI calls, call metadata, cancellation, two attempts
• ab3c7cd fix(llm): retry failed polls in place; wait_for_response for a running response
• de47c57 fix(llm): poll through failures, keep 5 attempts, never retry incomplete
• fe4adce fix(llm): five attempts for OpenAI, two for every other provider
• 7380e71 fix(llm): log failed attempts at WARNING

Tests

pytest tests/unit: 185 pass. isort, black, flake8, pydocstyle clean. Pre-existing and untouched: tests/unit/test_archiving.py fails on Python 3.13 (tarfile DeprecationWarning under filterwarnings=error).

New tests cover: polling to completion with id and usage extraction; cancel after accept (server-side cancel, no retry, no further poll) and cancel without server-side cancel; budget overrun; a failed poll retried in place and the generation created once; a failed response retried, an incomplete one not; fetch_response and wait_for_response never calling create; Anthropic id and usage from the final message; per-provider attempt counts; cancel before and during the retry wait; WARNING-level attempt logs.

Live, through dwh-llm-forecaster against the real APIs (Oct 21): four runs on the LEAP survey that failed on Oct 19, three with 3 question groups and one full survey with both gpt-6-astra and claude-opus-5-5 (16 calls, 92 rows). Every OpenAI response id was logged 1–4 s after the run started, before any result existed; one create and one completion per call, no retries, no duplicate generations; usage on every call; the dashboards show exactly the number of requests made. Cancellation, budget expiry and wait_for_response have unit coverage only so far.

🤖 Generated with Claude Code

https://claude.ai/code/session_013tLtyvBGmSaTwPwS81PuTB