You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
HTTP service web pods use ~1.25 GB since v0.0.655, keeping the deployment at its maximum of 30 pods #7586
Since v0.0.655-prod-1 was deployed on November 13, the HTTP service's web pods use ~1.25 GB of memory instead of ~900 MB. That's above the HPA's memory target (50% of the 2 Gi request), so the web deployment scaled from 5 pods to its 30-pod maximum and stays there, at 0.1 to 3.7 req/s. Fresh pods start around 650-750 MB and step up to ~1.25 GB within 20-30 minutes, so every redeploy scales down and back up.
The increase is anonymous (non-evictable) memory in the bugbug-web container; nginx and file cache didn't change. The deployment's command, env and resources were the same before and after: only the image changed (v0.0.648-prod-1 → v0.0.655-prod-1).
Running both production images locally with the production command and traffic mix: v0.0.655's container goes from ~420 MB to ~890 MB within a minute or two, with all four gunicorn processes (master and the 3 workers) ending up with ~170 MB of private memory each, which were shared on v0.0.648 (~440 MB, flat). The workers share the preloaded modules with the master (--preload) until they write to them: this looks like full garbage collections unsharing them. It only happens with Sentry enabled, but not because of the sentry-sdk upgrade (2.66.1 behaves the same).
Activity
marco-c commentedon Nov 19, 2026
#7583 and #7585 should fix this.
marco-c commentedon Nov 19, 2026
This should be fixed now.