fix: bound memory used by resize chunk cleanup - #4344
Open
2160039878-cyber wants to merge 1 commit into
Open
Conversation
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4344 +/- ##
=======================================
Coverage 94.34% 94.34%
=======================================
Files 92 92
Lines 12935 12950 +15
=======================================
+ Hits 12203 12218 +15
Misses 732 732
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses the remaining shrinking and mixed growing/shrinking cases in #3650.
Resize cleanup currently materializes both complete chunk grids and schedules every deletion at once. This generates only the out-of-bounds coordinates and deletes them in bounded batches, so auxiliary memory depends on the number of dimensions and the batch size rather than the total grid size.
The coordinate generator partitions the removed region by its first out-of-bounds axis. Integer iteration avoids
itertools.productcaching very large axes. Batches reuseconcurrent_map, respectasync.concurrency, and contain at most 1000 deletions when that setting isNone. The existing default fordelete_outside_chunksis preserved.For reviewers
I would particularly appreciate a review of the disjoint coordinate regions for mixed resizing and the batching behavior with unlimited configured concurrency. The tests compare coordinates against set difference, check for duplicates and pending-task bounds, and verify retained chunk bytes across v2/v3, regular/rectilinear, and sharded grids. Deletion failures still propagate before metadata is updated; already-deleted chunks are not rolled back.
Validation on commit
3a8ab604fca937a35de30be6cdcb4b3d4041e606:(2**40, 2)chunk grid while issuing only two deletions. Local statement coverage is 100% for the coordinate generator and_resize.Verification limits: GPU tests require the upstream runner. Codecov uploads on the fork were rejected because a token was required. The non-blocking docs type-link check reports the same 83 unlinked types across 44 pages on this commit and the unmodified baseline.
Author attestation
AI assistance: OpenAI Codex was used for the implementation, tests, and drafting this description. I have personally reviewed and understood the code changes.
TODO
docs/user-guide/*.md— not applicable; this is a cleanup-memory bug fix.changes/