feat: request throttling — bounded concurrency and per-provider pacing

docs/CONFIGURATION.md

@@ -28,6 +28,7 @@ Starting from this version, configuration is validated immediately:
 - No quotes required; avoid trailing spaces around `=`.
 - **Supported keys**:
   - Common: `provider`, `model`, `temperature`, `max_tokens`, `timeout_seconds`
+  - Rate limiting: `retry_max_retries`, `retry_max_elapsed_seconds`, `max_concurrent_requests`, `min_request_interval_ms`
   - Provider-specific API keys: `openai_api_key`, `anthropic_api_key`, `gemini_api_key`, `openrouter_api_key`, `together_api_key`
   - Provider-specific base URLs: `openai_base_url`, `anthropic_base_url`, `gemini_base_url`, `ollama_base_url`, `openrouter_base_url`, `together_base_url`
   - Legacy (still supported): `api_key`, `base_url` (applies to current provider)
@@ -124,6 +125,34 @@ model=gpt-4o-mini
 # llm:set-model "claude-3-5-sonnet-20241022"
 ```
 
+### Request Throttling (staying inside a rate limit)
+A model that calls the LLM once per agent per tick sends one request per agent
+simultaneously. On a free tier that exceeds the quota on the first tick.
+
+```
+# At most 4 requests in flight; start them at least 250ms apart
+max_concurrent_requests=4
+min_request_interval_ms=250
+```
+
+- `max_concurrent_requests` — whole number of simultaneous requests. Unset or `0` means
+  **disabled** (unbounded, the default). Negative or unparseable values are reported on
+  stderr and treated as disabled.
+- `min_request_interval_ms` — **milliseconds** between request starts. Unset or `0`
+  means **disabled**. Use this for a requests-per-minute quota; a concurrency cap alone
+  limits simultaneity, not rate.
+
+Excess requests queue rather than fail, occupy no thread while waiting, and are admitted
+in arrival order. The cap applies per provider and endpoint, using the provider-specific
+base URL key (`openai_base_url`, `gemini_base_url`, and so on) — the same value used to
+build the request — so different providers never share a cap. Spellings of one endpoint
+that differ only by trailing slash or letter case count as the same endpoint.
+
+Queue time is not added to `timeout_seconds`, so a cap set well below your agent
+population may require raising `timeout_seconds` too.
+
+See [SETUP.md](SETUP.md#request-throttling) for how to pick values.
+
 ## Ollama Quick Start (No API Key)
 Use Ollama to run models locally without any cloud credentials.
 

docs/SETUP.md

@@ -191,9 +191,66 @@ Browse the full catalog: [console.groq.com/docs/models](https://console.groq.com
 | `timeout_seconds` | Request timeout | No | 30 |
 | `retry_max_retries` | Retry attempts after a rate-limit (HTTP 429) response | No | 6 |
 | `retry_max_elapsed_seconds` | Total time allowed waiting out rate limits | No | 65 |
+| `max_concurrent_requests` | Maximum requests in flight at once, per provider | No | 0 (off) |
+| `min_request_interval_ms` | Minimum milliseconds between request starts | No | 0 (off) |
 
 *Not required for Ollama
 
+### Request throttling
+
+`retry_max_retries` recovers from a rate limit *after* it happens. Throttling exists
+to avoid tripping one in the first place.
+
+`ask turtles [ llm:chat-async ... ]` sends one request per turtle with nothing between
+the agents and the network, so 200 turtles open 200 connections on the first tick.
+`max_concurrent_requests` caps how many may be in flight at once; the rest queue and
+run as capacity frees up. No request is dropped — they are deferred, not discarded.
+
+```
+# At most 4 requests in flight, started at least 250ms apart
+max_concurrent_requests=4
+min_request_interval_ms=250
+```
+
+**Units and semantics:**
+
+- `max_concurrent_requests` — a whole number of requests. **Unset or `0` disables**
+  throttling, giving the unbounded behavior of earlier versions. Throttling is off by
+  default, so existing models are unaffected until you set it.
+- `min_request_interval_ms` — **milliseconds** between the starts of successive
+  requests. Unset or `0` disables pacing. Use it for a requests-per-minute quota: a
+  concurrency cap bounds how many run *at once*, not how many run *per minute*, so a
+  small cap recycling quickly can still exceed an RPM limit. For a 20 RPM quota, an
+  interval of `3000` keeps you inside it.
+- A **negative or unparseable** value is a mistake rather than a choice, so it is
+  reported on stderr and then treated as disabled. A typo will not stall your model,
+  but it will not pass silently either. Use `0` when you mean to switch throttling off.
+
+The limit is shared per provider and endpoint. Every request to the same provider and
+provider-specific base URL key (`openai_base_url`, `gemini_base_url`, and so on — the
+same value used to build the request) draws on one cap, no matter how many times the
+extension rebuilds its provider internally, while a different provider gets its own —
+a Gemini free tier will not throttle a paid OpenAI key running alongside it. Spellings
+of one endpoint that differ only by trailing slash or letter case count as the same
+endpoint. API keys are never part of that identity: it is rebuilt from scheme, host,
+port and path only, so a credential in a URL's userinfo or query string is discarded.
+
+Queued requests hold no thread while they wait, and are admitted in arrival order, so
+no agent can be starved by later arrivals.
+
+**Throttling and `timeout_seconds`:** queue time is *not* added to the timeout budget.
+How long a request waits depends on how many agents call in a tick, not on the cap, so
+there is no formula that could size an allowance for it honestly. This means a model
+whose fan-out greatly exceeds its cap can still exceed `timeout_seconds` while queued —
+if you set a small cap for a large population, raise `timeout_seconds` to match. An
+unthrottled model's timeout behavior is exactly as it was.
+
+Choosing a value: start from your provider's documented limits. A free tier allowing
+5 requests/minute suits `max_concurrent_requests=1` with `min_request_interval_ms=12000`.
+A paid key with generous limits may not need throttling at all. If a run still hits 429s
+with throttling on, the cap is above the quota — lower it, or reduce how often the model
+calls the LLM.
+
 ### Rate limits and retries
 
 When a provider returns HTTP 429, the extension waits and retries with jittered

src/main/LLMExtension.scala

@@ -196,6 +196,18 @@ class LLMExtension extends DefaultClassManager {
    * trip a 30s request timeout. The await bound is therefore the request timeout plus
    * the retry budget, so `timeout_seconds` keeps its meaning and modelers who lower it
    * do not thereby lose the ability to recover from a rate limit.
+   *
+   * Queue time behind a concurrency cap is NOT added to this bound, and deliberately
+   * so. Queue depth is set by how many agents call in a tick, not by the cap: with 200
+   * turtles and a cap of 4, the last request waits out ~50 waves. No fixed formula
+   * covers that, and one that pretended to would fail precisely where fan-out is
+   * widest — the case throttling exists for.
+   *
+   * So a heavily throttled model can still exceed this bound while queued, and that is
+   * a documented, intentional limitation rather than a solved problem: a modeler who
+   * sets a cap far below their fan-out should raise `timeout_seconds` accordingly. The
+   * bound is unchanged from before throttling existed, so an unthrottled model — the
+   * default — behaves exactly as it did.
    */
   private def getAwaitTimeout: FiniteDuration = {
     val retryBudget = configStore.get(RetryPolicy.MAX_ELAPSED_SECONDS)

src/main/providers/BaseHttpProvider.scala

@@ -103,6 +103,22 @@ abstract class BaseHttpProvider(implicit ec: ExecutionContext) extends LLMProvid
   /** Random source for jitter. Overridable so tests can make backoff deterministic. */
   protected def retryRandom: () => Double = () => scala.util.Random.nextDouble()
 
+  /**
+   * Proactive throttle for this provider, or None when it is switched off.
+   *
+   * Resolved per request so a config change takes effect without recreating the
+   * provider, matching how retryPolicy is handled. The gate is shared across
+   * every instance addressing the same provider and endpoint — see
+   * RequestThrottle.identity for why that is the right granularity.
+   */
+  protected def requestThrottle: Option[RequestThrottle] =
+    RequestThrottle.forProvider(
+      providerName,
+      configStore.get(baseUrlConfigKey).getOrElse(defaultBaseUrl),
+      configStore.get(RequestThrottle.MAX_CONCURRENT_REQUESTS),
+      configStore.get(RequestThrottle.MIN_REQUEST_INTERVAL_MS)
+    )
+
   /**
    * Reports a rate-limit wait to the modeler. A silent multi-second stall inside
    * `go` is indistinguishable from a hang, so long waits are announced on stderr.
@@ -243,6 +259,11 @@ abstract class BaseHttpProvider(implicit ec: ExecutionContext) extends LLMProvid
    * because retrying before a quota window reopens is guaranteed to fail. Waiting
    * is bounded by the policy's total elapsed budget rather than a fixed per-sleep
    * cap, so a quota window longer than the old 10s ceiling can actually be cleared.
+   *
+   * When a throttle is configured, the whole attempt sequence runs under a single
+   * permit. Holding it across retries rather than reacquiring per attempt is what
+   * keeps a retrying request from re-entering behind fresh arrivals and pushing
+   * the real in-flight count over the cap.
    */
   protected def executeWithRetry(
     httpRequest: Request[Either[String, String]],
@@ -280,7 +301,11 @@ abstract class BaseHttpProvider(implicit ec: ExecutionContext) extends LLMProvid
         }
       }
 
-    attempt(0, 0L)
+    requestThrottle match {
+      // One permit covers the complete logical request, including all retries.
+      case Some(throttle) => throttle.withPermit(attempt(0, 0L))
+      case None           => attempt(0, 0L)
+    }
   }
 
   override def setConfig(key: String, value: String): Unit = {