Skip to content

Fix unnecessary deepcopy in validate_psm_list causing OOM on large PSM lists#24

Merged
ArthurDeclercq merged 1 commit into
mainfrom
oom-memory-error
Jul 15, 2026
Merged

Fix unnecessary deepcopy in validate_psm_list causing OOM on large PSM lists#24
ArthurDeclercq merged 1 commit into
mainfrom
oom-memory-error

Conversation

@ArthurDeclercq

@ArthurDeclercq ArthurDeclercq commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Problem

validate_psm_list() unconditionally called copy.deepcopy() on the filtered PSMList on every call to predict(), calibrate(), and predict_and_calibrate(). This was flagged by a # TODO: Is deepcopy really necessary or can it be avoided? comment at the time it was introduced (d7a016a), but never resolved.

This surfaced as an OOM crash when running IM2Deep through MS²Rescore on a large timsTOF immunopeptidomics dataset (PSM list expanded by Mumble's mass-shift candidate search). The process was killed by earlyoom at ~78 GB RSS after several minutes of steadily ramping memory, while the equivalent DeepLC step on the same PSM list completed quickly.

Investigation

n PSMs deepcopy time extra RSS
50,000 3.0 s +251 MB
200,000 12.7 s +1,006 MB
446,902 27.5 s +2,094 MB

Roughly linear, ~4.7 KB/PSM. On a Mumble-exploded PSM list in the millions, this alone accounts for tens of GB and the multi-minute "slow ramp" observed before the kill.

  • Confirmed deeplc._parse_psms (DeepLC's equivalent entry point) does no copying at all when given a PSMList — this was the actual asymmetry, not the shared feature-encoding path.

Fix

The deepcopy is only structurally necessary because the needs_target=True branch mutates PSMs in place (psm.metadata["CCS"] = ...), and boolean-mask PSMList indexing shares PSM instances with the input rather than cloning them (verified directly). For the plain predict() path (needs_target=False, what MS²Rescore's feature generator uses), nothing downstream mutates any PSM, so the copy was pure overhead.

Made the deepcopy conditional on needs_target, matching what it actually protects against.

Caveat for reviewers

predict_and_calibrate() calls validate_psm_list(psm_list) with the default needs_target=False for its primary psm_list argument, then later does:

for idx, psm in enumerate(psm_list):
    ...
    psm.metadata["predicted_CCS_uncalibrated"] = predicted_ccs[idx]

Previously this ran against an isolated deep copy, so the mutation never reached the caller's original object. With this fix, predict_and_calibrate() will now mutate the caller-supplied psm_list in place (attach predicted_CCS_uncalibrated to metadata). This has no impact on MS²Rescore (it only calls predict(), never predict_and_calibrate()), but it is a visible behavior change for any other direct consumer of this function. Flagging so you can decide whether that's acceptable or whether predict_and_calibrate() needs its own explicit copy of psm_list restored.

@ArthurDeclercq ArthurDeclercq changed the title deepcopy fix in helper function to reduce memory consumption when not… Fix unnecessary deepcopy in validate_psm_list causing OOM on large PSM lists Jul 15, 2026
@ArthurDeclercq
ArthurDeclercq merged commit 745c3ba into main Jul 15, 2026
6 checks passed
@ArthurDeclercq
ArthurDeclercq deleted the oom-memory-error branch July 15, 2026 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant