Repository navigation
feat(http): auth, rate_limit, max_in_flight and retries on the http connection - #356
Merged
Merged
Conversation
…onnection
An http connection now holds everything every request to its API shares in
a run: `auth` (bearer, api_key in a header or query parameter, basic, and
the OAuth2 client credentials, refresh token and password grants),
`rate_limit`, the new `max_in_flight` and `retries`. One limiter and one
in-flight cap per connection, keyed by its name alone, hold across every
table, page, read-ahead request, fan-out key and retry in the run, so two
tables reading one API at once stay under one limit together, and open
requests no longer grow to concurrency x read_ahead. A 429 retry takes a
turn and a permit again.
`plan` shows each http connection's limits, retries and auth scheme, never
its credentials, and `sqlload debug` says which settings the credentials
come from. Connection settings are checked when the project is read.
BREAKING CHANGE: `sl.http()` no longer takes `auth`, `rate_limit` or
`retries`; passing one fails, naming the key to set on the connection in
sqlload.toml. `sl.bearer`, `sl.api_key`, `sl.basic` and the
`sl.oauth2_*` helpers are removed: set `auth = { kind = "..." }` on the
connection instead.
Deploying sqlload with
|
| Latest commit: |
dd3879d
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://f12c3ee7.sqlload.pages.dev |
| Branch Preview URL: | https://http-connection-limits.sqlload.pages.dev |
The stub API counted a request closed only after writing its answer, so the client could send its next request before the handler thread ran again, and the stub saw one request more open than the client held.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An
httpconnection now holds everything every request to its API shares in a run: how requests authenticate (auth), the limits they stay under together (rate_limitand the newmax_in_flight), and how often a failed one is retried (retries).Before,
rate_limit,retriesandauthwere arguments of eachsl.http(...)call, and the limiter was shared only by jobs with the same rate text. So two tables on one API with"10/s"and"600/m"got two limiters and could send twice the allowed rate, and a table with norate_limitwasn't limited at all. Nothing capped open requests, which reachedconcurrency×read_ahead.Now:
read_aheadrequest, fan-out key and retry on the connection shares them. A request takes its permit, then its OAuth2 token if it has one, then its turn, so no turn passes unused while it waits. It gives the permit back once its response is read, not during a retry's backoff. A 429 retry takes a permit and a turn again.authis a table:bearer,api_keywithheaderorquery,basic, andoauth2_client_credentials,oauth2_refresh_tokenoroauth2_passwordwithtoken_url,scopeandclient_auth. Credentials stay in their own keys. A key written beside anauth'skindis refused, not dropped. So isauthwritten as a string, and that error doesn't show the value.max_in_flightfrom 1 to 10,000,retriesup to 100, thetoken_url's scheme, andapi_keytaking one ofheaderorquery. Keys that don't apply to a connection's kind are refused, for examplerate_limiton asnowflakeconnection.planstarts with a line perhttpconnection, and never shows its credentials:sqlload debugsays which settings the credentials come from, and fails a connection that lacks one itsauthneeds:rate_limitedload events still name the connection and its rate.Separate processes don't share the limits, so the docs say to load one API's tables in one
sqlload load, which in an orchestrator means one task per API.Breaking:
sl.http()takes onlyheadersandconnection. Passingauth,rate_limit,max_in_flightorretriesfails with, for example,sl.http no longer takes rate_limit: set rate_limit on [connections.shop] in sqlload.toml.sl.bearer,sl.api_key,sl.basicand thesl.oauth2_*helpers are removed.Docs: configuration (
http), the Python API reference (sl.http, auth,read_ahead, multi-level fan-out, OAuth2), the Python guide, Coming from dlt, Prefect and Dagster,plan,debugand the landing page.Tested, against a local stub API that records each request's arrival and the most requests open at once:
rate_limit = "20/s"loading at once together stay under 20 a second. On 0.79.0, two tables set to20/sand1200/msent 34 requests in 0.80 s, where one shared limit needs at least 1.65 s.max_in_flight = 3, four tables atread_ahead=20never have more than 3 requests open. On 0.79.0 there were 21 open at once.Retry-After: 0, the retries stay under both limits.planshows the line and no secret, and the oldsl.httparguments fail with the message above.authandretrieson the connection. There are parse tests for each refused setting, and unit tests fordebug's credential line.BEGIN_COMMIT_OVERRIDE
feat(http): auth, rate_limit, max_in_flight and retries on the http connection
BREAKING CHANGE:
sl.http()no longer takesauth,rate_limitorretries; passing one fails, naming the key to set on the connection in sqlload.toml.sl.bearer,sl.api_key,sl.basicand thesl.oauth2_*helpers are removed: setauth = { kind = "..." }on the connection instead.END_COMMIT_OVERRIDE