Skip to content

feat(http): auth, rate_limit, max_in_flight and retries on the http connection - #356

Merged
kvlonge merged 2 commits into
mainfrom
http-connection-limits
Nov 19, 2026
Merged

kvlonge merged 2 commits into
mainfrom
http-connection-limits

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

An http connection now holds everything every request to its API shares in a run: how requests authenticate (auth), the limits they stay under together (rate_limit and the new max_in_flight), and how often a failed one is retried (retries).

[connections.shop]
kind = "http"
base_url = "https://api.example.com/v1"
token = "${ENV:SHOP_TOKEN}"
auth = { kind = "bearer" }
rate_limit = "10/s"
max_in_flight = 4
retries = 5

Before, rate_limit, retries and auth were arguments of each sl.http(...) call, and the limiter was shared only by jobs with the same rate text. So two tables on one API with "10/s" and "600/m" got two limiters and could send twice the allowed rate, and a table with no rate_limit wasn't limited at all. Nothing capped open requests, which reached concurrency × read_ahead.

Now:

  • There is one limiter and one in-flight cap per connection per process, keyed by the connection's name. Every table, page, read_ahead request, fan-out key and retry on the connection shares them. A request takes its permit, then its OAuth2 token if it has one, then its turn, so no turn passes unused while it waits. It gives the permit back once its response is read, not during a retry's backoff. A 429 retry takes a permit and a turn again.
  • auth is a table: bearer, api_key with header or query, basic, and oauth2_client_credentials, oauth2_refresh_token or oauth2_password with token_url, scope and client_auth. Credentials stay in their own keys. A key written beside an auth's kind is refused, not dropped. So is auth written as a string, and that error doesn't show the value.
  • The settings are checked when the project is read: the rate, max_in_flight from 1 to 10,000, retries up to 100, the token_url's scheme, and api_key taking one of header or query. Keys that don't apply to a connection's kind are refused, for example rate_limit on a snowflake connection.
  • plan starts with a line per http connection, and never shows its credentials:
    http  shop  rate_limit 10/s, max_in_flight 4, retries 5, auth bearer
    
  • sqlload debug says which settings the credentials come from, and fails a connection that lacks one its auth needs:
    ok    connection shop  auth bearer: token from the connection
    fail  connection erp  auth oauth2_client_credentials needs client_secret on the connection
    
  • rate_limited load events still name the connection and its rate.

Separate processes don't share the limits, so the docs say to load one API's tables in one sqlload load, which in an orchestrator means one task per API.

Breaking: sl.http() takes only headers and connection. Passing auth, rate_limit, max_in_flight or retries fails with, for example, sl.http no longer takes rate_limit: set rate_limit on [connections.shop] in sqlload.toml. sl.bearer, sl.api_key, sl.basic and the sl.oauth2_* helpers are removed.

Docs: configuration (http), the Python API reference (sl.http, auth, read_ahead, multi-level fan-out, OAuth2), the Python guide, Coming from dlt, Prefect and Dagster, plan, debug and the landing page.

Tested, against a local stub API that records each request's arrival and the most requests open at once:

  • Two tables on one connection with rate_limit = "20/s" loading at once together stay under 20 a second. On 0.79.0, two tables set to 20/s and 1200/m sent 34 requests in 0.80 s, where one shared limit needs at least 1.65 s.
  • With max_in_flight = 3, four tables at read_ahead=20 never have more than 3 requests open. On 0.79.0 there were 21 open at once.
  • With every third request answered 429 and Retry-After: 0, the retries stay under both limits.
  • plan shows the line and no secret, and the old sl.http arguments fail with the message above.
  • Existing HTTP tests, including the OAuth2 grants, API keys, redirects and fan-outs, now set auth and retries on the connection. There are parse tests for each refused setting, and unit tests for debug's credential line.

BEGIN_COMMIT_OVERRIDE
feat(http): auth, rate_limit, max_in_flight and retries on the http connection

BREAKING CHANGE: sl.http() no longer takes auth, rate_limit or retries; passing one fails, naming the key to set on the connection in sqlload.toml. sl.bearer, sl.api_key, sl.basic and the sl.oauth2_* helpers are removed: set auth = { kind = "..." } on the connection instead.
END_COMMIT_OVERRIDE

…onnection

An http connection now holds everything every request to its API shares in
a run: `auth` (bearer, api_key in a header or query parameter, basic, and
the OAuth2 client credentials, refresh token and password grants),
`rate_limit`, the new `max_in_flight` and `retries`. One limiter and one
in-flight cap per connection, keyed by its name alone, hold across every
table, page, read-ahead request, fan-out key and retry in the run, so two
tables reading one API at once stay under one limit together, and open
requests no longer grow to concurrency x read_ahead. A 429 retry takes a
turn and a permit again.

`plan` shows each http connection's limits, retries and auth scheme, never
its credentials, and `sqlload debug` says which settings the credentials
come from. Connection settings are checked when the project is read.

BREAKING CHANGE: `sl.http()` no longer takes `auth`, `rate_limit` or
`retries`; passing one fails, naming the key to set on the connection in
sqlload.toml. `sl.bearer`, `sl.api_key`, `sl.basic` and the
`sl.oauth2_*` helpers are removed: set `auth = { kind = "..." }` on the
connection instead.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Deploying sqlload with  Cloudflare Pages  Cloudflare Pages

Latest commit: dd3879d
Status: ✅  Deploy successful!
Preview URL: https://f12c3ee7.sqlload.pages.dev
Branch Preview URL: https://http-connection-limits.sqlload.pages.dev

View logs

The stub API counted a request closed only after writing its answer, so
the client could send its next request before the handler thread ran
again, and the stub saw one request more open than the client held.
@kvlonge
kvlonge merged commit 04c5b2b into main Nov 19, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant