Repository navigation
Don't import bugbug_http.models and the ML libraries in the HTTP service's web processes - #7583
Merged
Merged
Conversation
This was referenced Nov 19, 2026
marco-c
force-pushed
the
web-no-ml-imports
branch
from
November 19, 2026 20:59
2161927 to
986b3e5
Compare
marco-c
commented
Nov 19, 2026
| from sklearn.feature_extraction.text import CountVectorizer | ||
|
|
||
| from bugbug import utils | ||
| from bugbug import sklearn_utils, utils |
Collaborator
Author
There was a problem hiding this comment.
instead of doing this, split the tests into a new test_sklearn_utils.py
…ice's web processes The web processes imported bugbug_http.models for the job functions, which they only enqueue: they run in the RQ workers. Through it and bugbug.utils, each web process (the gunicorn master and its 3 workers) loaded numpy, pandas, scipy, scikit-learn, xgboost and imbalanced-learn, ~260 MB. The web processes now refer to the job functions by name, with the constants of bugbug_http.jobs (RQ imports the functions in the workers from their path): imported by name, a misspelled one fails like a misspelled function would, as an undefined name for ruff or an ImportError. The lists of models they need move to bugbug_http.model_names. The scikit-learn transformers move from bugbug.utils to bugbug.sklearn_utils, as the web processes use bugbug.utils. Importing bugbug_http.app now takes ~95 MB. Tests check that it doesn't import bugbug_http.models or the ML libraries, and that the job names match the functions of bugbug_http.models.
marco-c
force-pushed
the
web-no-ml-imports
branch
from
November 19, 2026 21:40
986b3e5 to
225b109
Compare
marco-c
marked this pull request as ready for review
November 19, 2026 21:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The web processes imported
bugbug_http.modelsfor the job functions, which they only enqueue: the functions run in the RQ workers (bugbug-worker). Through it andbugbug.utils, every web process (gunicorn master + 3 workers) loaded numpy, pandas, scipy, scikit-learn, xgboost and imbalanced-learn, ~260 MB per process.bugbug_http.jobs:JobInfoenqueuesbugbug_http.models.<name>, which RQ imports in the workers, and imported by name a misspelt one fails like a misspelt function would (undefined name for ruff, orImportError). The Redis keys and the stored job paths are unchanged.bugbug_http.model_names;bugbug_http.modelskeeps its normal top-level imports.bugbug.utilsto the newbugbug.sklearn_utils(with their test, intests/test_sklearn_utils.py), since the web does usebugbug.utils.Importing
bugbug_http.appnow takes ~95 MB. Tests check that it doesn't importbugbug_http.modelsor the ML libraries, and that the job names match the functions inbugbug_http.models; the tests' queue mock also checks that every enqueued job function exists.Fixes #7587