Skip to content

Freeze the objects allocated before forking the gunicorn workers - #7584

Merged
marco-c merged 1 commit into
mozilla:masterfrom
marco-c:web-gc-freeze
Nov 19, 2026
Merged

marco-c merged 1 commit into
mozilla:masterfrom
marco-c:web-gc-freeze

Conversation

@marco-c

@marco-c marco-c commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

With --preload, the workers share the master's memory until they write to it, and a full garbage collection writes to every object it traverses. When the preloaded heap is large, this unshares it in every process. Running the v0.0.655 production image locally with the production traffic mix, each of the four gunicorn processes got its own copy of ~170 MB of preloaded memory within a minute or two (~420 MB → ~890 MB). That matches the step-up of the production pods in #7586.

gc.freeze() in gunicorn's pre_fork hook keeps collections from traversing the objects allocated before forking. With the same v0.0.655 image and traffic, memory stays at ~405 MB.

As recommended by the gc.freeze() documentation, collections are also disabled while the app is preloaded, so they don't leave holes in the shared pages for the workers to fill. They're re-enabled right after freezing, in the master too, as it keeps running with background threads.

#7583 and #7585 remove most of that heap from the web processes. With them, preload alone stays flat (~122 MB under the same traffic) and the freeze makes no measurable difference. This is insurance against the problem returning if heavy imports creep back into the web. Dropping --preload instead would double memory (~246 MB, as each worker keeps a full private copy) and make worker restarts re-import the app instead of forking.

With --preload, the workers share the master's memory until they write to it,
and a full garbage collection writes to every object it traverses: when the
preloaded heap is large, this unshares it in every process. With the v0.0.655
production image and traffic, each of the four gunicorn processes got its own
copy of ~170 MB of preloaded memory within a minute or two (~420 MB -> ~890 MB
for the container), which gc.freeze() in the pre_fork hook prevents (~405 MB).

As recommended by the gc.freeze() documentation, collections are also disabled
while the app is preloaded, so that they don't leave holes in the shared pages
that the workers would fill. They are enabled again after freezing, in the
master too, as it keeps running.

Now that the web processes don't import the ML and AI libraries anymore, there
is little left to unshare and the freeze makes no measurable difference: this
is insurance against heavy imports creeping back into them.
@marco-c
marco-c marked this pull request as ready for review November 19, 2026 21:43
@marco-c
marco-c requested a review from a team as a code owner November 19, 2026 21:43
@marco-c
marco-c merged commit cb5ffd0 into mozilla:master Nov 19, 2026
13 checks passed
@marco-c
marco-c deleted the web-gc-freeze branch November 19, 2026 21:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant