feat: request throttling — bounded concurrency and per-provider pacing

[JNK234]

Summary

Adds proactive request throttling: a configurable cap on in-flight requests per provider, plus optional pacing between request starts. Both are off by default, so existing models are unaffected until a modeler opts in.

retry_max_retries (#37) recovers from a rate limit after it happens. This is the proactive counterpart — not tripping one in the first place. ask turtles [ llm:chat-async ... ] currently sends one request per turtle with nothing between the agents and the socket, so a 200-turtle model opens 200 connections on the first tick.

Addresses #48.

Scope and remaining work

Issue #48 lists five design requirements. This PR delivers two of them and deliberately does not close the issue:

• Delivered here: bounded concurrency (max_concurrent_requests) and optional per-provider pacing (min_request_interval_ms).
• Already present: jitter on retry backoff, which landed with Add retry with backoff for rate limit errors (HTTP 429) #37 — no change needed.
• Remaining, as separate follow-up work: broadening retry beyond HTTP 429 to 5xx and timeouts, and per-agent failure isolation so one agent's failed call cannot abort a tick. Neither is touched here — the retry path still handles 429 only.

Design and behavior

RequestThrottle is a non-blocking FIFO gate wrapped around the request path in BaseHttpProvider.executeWithRetry.

• Never blocks a thread. A caller that cannot get a permit receives an incomplete Future; its continuation runs when a permit is handed over. A Semaphore.acquire() would park a thread per queued agent and can starve the global pool.
• FIFO admission. A free permit is taken only when nobody is already waiting, so a later arrival cannot overtake a waiter. With agents calling every tick, a barging gate can pass over one agent for an entire run.
• One permit per logical request, held across retries. Reacquiring per attempt would let a retrying request re-enter behind fresh arrivals and push real in-flight work above the cap.
• Permits are released on every exit path — success, HTTP failure, synchronous throw, exhausted retries, and a failed or rejected pacing delay. A stranded permit is worse than a failed request: capacity never returns, so with a cap of 1 the gate would be dead.
• Shared per provider + endpoint. The gate is keyed process-wide, so it survives LLMExtension rebuilding its provider (which set-provider, load-config, and set-thinking all do). Different providers never share a cap — a Gemini free tier will not throttle a paid OpenAI key. Endpoint spellings differing only by trailing slash, host case, or default port resolve to one gate, so a trailing slash cannot silently double the configured cap.
• API keys are never part of the throttle identity. It is rebuilt from scheme, host, port and path only, discarding credentials in a URL's userinfo or query string (which is how Gemini passes its key) so a secret cannot surface in a diagnostic.

All chat traffic routes through this gate: every provider extends BaseHttpProvider and none overrides chat. Ollama's /api/tags health-check and model-list calls are deliberately outside it — they are not per-agent fan-out and hit a local server with no quota.

Configuration

# At most 4 requests in flight, started at least 250ms apart
max_concurrent_requests=4
min_request_interval_ms=250

• max_concurrent_requests — whole number of simultaneous requests. Unset or 0 disables (the default, unbounded as before).
• min_request_interval_ms — milliseconds between request starts. Unset or 0 disables. A concurrency cap bounds simultaneity, not rate, so a small cap recycling quickly can still exceed an RPM quota; pacing covers that.
• Negative or unparseable values are reported on stderr once (not once per request) and treated as disabled. A typo must not stall a model, but it must not pass in silence either.

Documented in docs/CONFIGURATION.md and docs/SETUP.md, including how to pick values from a provider's published limits.

Compatibility

• Throttling is off unless configured; the unthrottled path is byte-for-byte the prior behavior, and a test asserts fan-out is still unbounded when unset.
• No public API or primitive changes. No changes to existing config key semantics.
• Additive only — one new file plus a wrapper at one call site.

One documented limitation: queue time is deliberately not added to the timeout_seconds await bound. Queue depth is set by how many agents call in a tick, not by the cap, so no fixed formula could size it honestly. A model whose fan-out greatly exceeds its cap can therefore still exceed timeout_seconds while queued — modelers setting a small cap for a large population should raise timeout_seconds to match. This is called out in both the code and SETUP.md. Unthrottled models are unaffected.

Test plan

Added 18 tests across two suites, all verified locally.

RequestThrottleSpec (16 tests) — unit level, using a manual clock so pacing is asserted on the delays the throttle requests rather than on wall-clock timing (which would make the suite a scheduler-noise detector):

• cap enforcement with an observed peak, FIFO admission order, and no-thread-occupancy while queued
• permit release on success, failure, synchronous throw, failed pacing delay, and a pacing delay that throws
• pacing interval arithmetic, and zero-interval scheduling nothing at all
• config parsing: unset/empty/0 disable; negative and unparseable warn and disable; warn-once across 50 resolutions
• identity: shared per provider+endpoint, split across providers and hosts, normalized across trailing slash/case/whitespace, and never containing a credential

ThrottleIntegrationSpec (4 tests) — through BaseHttpProvider's real request path with a latch-gated stub backend so admitted requests genuinely overlap:

• a 10-request burst is capped at 2 and all 10 still complete
• an unthrottled provider fans out to 5 concurrent sends exactly as before
• a retrying request holds one permit across all attempts and releases it once
• a request that exhausts its retries releases its permit

Verified results

Command	Result
sbt test	117 succeeded, 0 failed, 6 suites
sbt test with pool pinned to 4 threads (matches a 4-vCPU CI runner)	117 succeeded, 0 failed
sbt "testOnly ...RequestThrottleSpec ...ThrottleIntegrationSpec"	20 succeeded, 0 failed
Same focused suites, 5 consecutive runs at 4 threads	20/20 each run, no flakes
sbt assembly (from a removed jar, not cached)	success, 21s, throttle classes present in llm.jar
git diff --check	clean
CI (sbt test, ubuntu-latest, Java 17)	117 succeeded, 0 failed, 6 suites, 0 aborted

Test counts go from 115 to 117 because the two permit-release regression tests below were added; the pre-existing 115 continue to pass.

Two defects found and fixed during review

Both were caught by an independent review pass and fixed test-first (failing test demonstrated first, then the minimal fix):

1. Stranded permit on pacing failure (production defect, RequestThrottle.scala). The permit was taken before pacing ran, but withPermit could only register its release once acquire() completed — so a pacing delay that failed, or a scheduler that rejected it by throwing, lost the permit permanently. With a cap of 1 the gate deadlocked for the rest of the run. The failing test showed the second request timing out because the first leaked. Release is now attached inside acquire(), covering both the failed-Future and synchronous-throw paths.

2. Test-only pool starvation (test defect). The integration stub blocked in CountDownLatch.await() inside a Future without scala.concurrent.blocking. The unthrottled case needs five sends open simultaneously; on a 4-vCPU runner the fork-join pool spawns no compensation thread, so the fifth never starts. Reproduced by pinning the pool to 4 threads — this would have failed CI — and fixed with blocking. Production code was correct; only the test was at fault.

3. Order-dependent provider registry tests (pre-existing test defect, surfaced by this branch). CI aborted ClaudeRequestSpec with Unknown provider: anthropic. The cause is unrelated to throttling: ClaudeRequestSpec builds a ClaudeProvider in its constructor, which reads the global provider registry, but nothing in that suite installed it — while ProviderDefaultsSpec cleared the same global registry from its constructor. Which suite won depended on the order ScalaTest built them in, and adding two suites to this branch changed that order. Fixed by having ClaudeRequestSpec register what it needs and by dropping the reset() that emptied shared state out from under already-constructed suites. registerAll() inserts by name, so re-registering suffices. Both suites now pass in isolation, which is what makes them order-independent. CI is green.