fix: back off tinybird requests when query latency rises

[epipav]

Summary

On Sep 18, between 00:00 and 04:30 UTC, Insights app traffic to Tinybird roughly doubled (5-6k distinct projects per hour, crawler-like). The AdaptiveSemaphore sat at its limit of 35 the whole time, which kept 25-31 queries in flight and 20-25 Tinybird cores busy (normal is 8-10 in flight). That load shares the cluster with our copy pipes, which ran 1.3-1.8x slower, backed up past Tinybird's 12-job cap, and failed 77 times. Two daily copies lost their only run.

The semaphore only backs off on HTTP 429. Tinybird returned zero 429s across 6.87M requests from Sep 17 onward and answered the overload with slower queries, so the limit never moved.

This PR adds latency-based backoff to @lfx-insights/tinybird-client:

• Each throttled request reports Tinybird's own query time (statistics.elapsed), so network and event-loop lag stay out of the signal.
• LatencyMonitor takes the p90 of every 10s window, smooths it over about 60s, and compares it with a baseline that learns over 24h (a running mean while the process is young).
• Above 1.4x baseline, the limit shrinks by a quarter per window, down to half of maxConcurrent (17 in production). Below 1.2x, it grows back one slot per window.
• The baseline stops learning while backoff is active, so a long overload keeps its protection.
• The limit in force is the lower of the 429 ceiling and the latency ceiling. When it rises, queued requests are served right away; previously a raised limit was only used by new arrivals.
• tinybird_queue_status logs now include latencyLimit, latencyP90Ms and latencyBaselineP90Ms, and limit changes log latency_backoff / latency_recovery events.
• NUXT_TINYBIRD_LATENCY_BACKOFF=false turns it off without a deploy.

How the thresholds were chosen

I replayed 3.5 days of production latency (10s windows from tinybird.pipe_stats_rt, Sep 15-18) through the built client:

Period	Mean limit	Windows throttled
Normal day (Sep 17)	34-35	5-8%
Surge, 00:00-02:00	22	75%
Surge, 02:00-04:30	17	100%
Pod started 2h before the surge, 02:00-04:30	18	97%

Normal traffic uses 8-10 slots, so the occasional dip on a normal day doesn't affect requests. The replay has no feedback loop; in production, backing off also lowers the latency it reacts to.

A pod that starts in the middle of a surge learns that surge as its baseline and doesn't back off until latency returns to normal. Pods that are already running keep their protection.

Test plan

• New latency-monitor.test.ts: warm-up, steady state, p90 ignores a few slow queries, reaction to a sustained slowdown, smoothing of a single slow window, frozen baseline through a 6h overload, skipped sparse windows, running-mean start
• adaptive-semaphore.test.ts: shrink to floor, one-slot recovery, hold band, queued requests served when the limit grows, lower of the 429 and latency ceilings, disabled mode
• throttle.test.ts: statistics.elapsed reported in ms for throttled successes only
• libs + api CI steps on Node 24: format, lint, tsc, build, tests (tinybird-client 63, api 67)
• frontend format:check, lint, tsc-check, and vitest with the CI env (284 tests)

[Copilot]

🟡 Changes recommended

The p90 calculation and default floor do not consistently implement the documented backoff semantics.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds Tinybird latency-based concurrency backoff to protect shared cluster capacity during load surges.

Changes:

• Introduces p90 latency monitoring with smoothing and baseline learning.
• Combines latency and HTTP 429 concurrency ceilings.
• Adds configuration, telemetry, and automated tests.

File summaries

File	Description
libs/tinybird-client/src/adaptive-semaphore.ts	Adds latency-driven concurrency control.
libs/tinybird-client/src/client.ts	Reports Tinybird query latency.
libs/tinybird-client/src/index.ts	Exports backoff configuration type.
libs/tinybird-client/src/latency-monitor.ts	Implements latency monitoring.
libs/tinybird-client/src/types.ts	Defines latency backoff options.
libs/tinybird-client/tests/adaptive-semaphore.test.ts	Tests concurrency adjustment behavior.
libs/tinybird-client/tests/latency-monitor.test.ts	Tests latency signal processing.
libs/tinybird-client/tests/throttle.test.ts	Tests latency reporting.
frontend/server/data/tinybird/tinybird.ts	Adds the environment-controlled kill switch.

Review details

• Files reviewed: 9/9 changed files
• Comments generated: 3
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

The implementation appears sound, but production traffic-control thresholds and feedback behavior warrant final human operational review.

Review details

• Files reviewed: 9/9 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟢 Approval recommended

The implementation and edge cases are well tested; the remaining comment is non-blocking documentation cleanup.

Review details

• Files reviewed: 9/9 changed files
• Comments generated: 1
• Review effort level: Balanced

[gaspergrom]

Traced the ceiling math, the queue wakeup on limit increase, and the baseline freeze during backoff. All three hold up, and the tests pin down specific numeric behavior instead of just running the code. The two earlier Copilot catches got fixed already, both with regression tests covering the exact cases.

[themarolt]

lgtm - traced the ceiling math and the queue wakeup, both hold. one question on the floor and what queues up behind it, not blocking