Skip to content

HTTP service web pods use ~1.25 GB since v0.0.655, keeping the deployment at its maximum of 30 pods #7586

Closed

Description

@marco-c

Since v0.0.655-prod-1 was deployed on November 13, the HTTP service's web pods use ~1.25 GB of memory instead of ~900 MB. That's above the HPA's memory target (50% of the 2 Gi request), so the web deployment scaled from 5 pods to its 30-pod maximum and stays there, at 0.1 to 3.7 req/s. Fresh pods start around 650-750 MB and step up to ~1.25 GB within 20-30 minutes, so every redeploy scales down and back up.

The increase is anonymous (non-evictable) memory in the bugbug-web container; nginx and file cache didn't change. The deployment's command, env and resources were the same before and after: only the image changed (v0.0.648-prod-1 → v0.0.655-prod-1).

Running both production images locally with the production command and traffic mix: v0.0.655's container goes from ~420 MB to ~890 MB within a minute or two, with all four gunicorn processes (master and the 3 workers) ending up with ~170 MB of private memory each, which were shared on v0.0.648 (~440 MB, flat). The workers share the preloaded modules with the master (--preload) until they write to them: this looks like full garbage collections unsharing them. It only happens with Sentry enabled, but not because of the sentry-sdk upgrade (2.66.1 behaves the same).

Activity

  1. marco-c commented on Nov 19, 2026

    @marco-c
    CollaboratorAuthor

    #7583 and #7585 should fix this.

  2. self-assigned this
    on Nov 19, 2026
  3. marco-c commented on Nov 19, 2026

    @marco-c
    CollaboratorAuthor

    This should be fixed now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions