Skip to content

Don't import bugbug_http.models and the ML libraries in the HTTP service's web processes - #7583

Merged
marco-c merged 1 commit into
mozilla:masterfrom
marco-c:web-no-ml-imports
Nov 19, 2026
Merged

marco-c merged 1 commit into
mozilla:masterfrom
marco-c:web-no-ml-imports

Conversation

@marco-c

@marco-c marco-c commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

The web processes imported bugbug_http.models for the job functions, which they only enqueue: the functions run in the RQ workers (bugbug-worker). Through it and bugbug.utils, every web process (gunicorn master + 3 workers) loaded numpy, pandas, scipy, scikit-learn, xgboost and imbalanced-learn, ~260 MB per process.

  • The web refers to job functions by name, with the constants of bugbug_http.jobs: JobInfo enqueues bugbug_http.models.<name>, which RQ imports in the workers, and imported by name a misspelt one fails like a misspelt function would (undefined name for ruff, or ImportError). The Redis keys and the stored job paths are unchanged.
  • The model lists the web needs move to bugbug_http.model_names; bugbug_http.models keeps its normal top-level imports.
  • The scikit-learn transformers move from bugbug.utils to the new bugbug.sklearn_utils (with their test, in tests/test_sklearn_utils.py), since the web does use bugbug.utils.

Importing bugbug_http.app now takes ~95 MB. Tests check that it doesn't import bugbug_http.models or the ML libraries, and that the job names match the functions in bugbug_http.models; the tests' queue mock also checks that every enqueued job function exists.

Fixes #7587

@marco-c marco-c changed the title Don't import the ML libraries in the HTTP service's web processes Don't import bugbug_http.models and the ML libraries in the HTTP service's web processes Nov 19, 2026
Comment thread tests/test_utils.py Outdated
from sklearn.feature_extraction.text import CountVectorizer

from bugbug import utils
from bugbug import sklearn_utils, utils

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of doing this, split the tests into a new test_sklearn_utils.py

…ice's web processes

The web processes imported bugbug_http.models for the job functions, which
they only enqueue: they run in the RQ workers. Through it and bugbug.utils,
each web process (the gunicorn master and its 3 workers) loaded numpy, pandas,
scipy, scikit-learn, xgboost and imbalanced-learn, ~260 MB.

The web processes now refer to the job functions by name, with the constants
of bugbug_http.jobs (RQ imports the functions in the workers from their path):
imported by name, a misspelled one fails like a misspelled function would, as
an undefined name for ruff or an ImportError. The lists of models they need
move to bugbug_http.model_names. The scikit-learn transformers move from
bugbug.utils to bugbug.sklearn_utils, as the web processes use bugbug.utils.

Importing bugbug_http.app now takes ~95 MB. Tests check that it doesn't import
bugbug_http.models or the ML libraries, and that the job names match the
functions of bugbug_http.models.
@marco-c
marco-c marked this pull request as ready for review November 19, 2026 21:45
@marco-c
marco-c requested a review from a team as a code owner November 19, 2026 21:45
@marco-c
marco-c merged commit d31860d into mozilla:master Nov 19, 2026
12 of 13 checks passed
@marco-c
marco-c deleted the web-no-ml-imports branch November 19, 2026 21:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The HTTP service's web processes import the ML libraries, which only the workers need

1 participant