Download: add a 'resume' option - #105
Open
fingolfin wants to merge 1 commit into
Open
Conversation
A download interrupted halfway currently starts again from zero, which for a data set of any size is the difference between a retry and an afternoon. With 'resume' set and a target given, the curl method continues the partial file via '-C -', and a failure keeps that file so the next attempt can carry on from it. Methods that cannot resume decline the request rather than proceeding: wget's '-c' does not resume when combined with '-O', and SingleHTTPRequest has no range support, so both would truncate or delete the very file the curl method needs. They run first, so without this a resumed download would never see its partial file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## mh/download-target-cleanup #105 +/- ##
==============================================================
+ Coverage 86.23% 86.38% +0.14%
==============================================================
Files 32 32
Lines 1918 1939 +21
==============================================================
+ Hits 1654 1675 +21
Misses 264 264
🚀 New features to boost your workflow:
|
Collaborator
|
@fingolfin Be sure to tell me when these four (or more?) PRs are ready to be merged and a release made. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An interrupted download starts again from zero. For a data set of any size
that is the difference between a retry and an afternoon.
With
resume := trueandtargetgiven, thevia curlmethod continues thepartial file (
-C -), and a failure keeps that file so the next attempt cancarry on.
Methods that cannot resume decline rather than proceed, because they run
before
via curland would destroy the file it needs: wget's-cdoes notresume when combined with
-O— the output file is always created anew — andSingleHTTPRequesthas no range support. Measured before adding the decline:If no resuming method is available the download fails and the caller can
retry without
resume.The test server answers a Range request with the remainder in upper case, so
the test distinguishes a resumed download from a restarted one; removing the
-C -makes it fail withdid not resume: abcdefghijklmnopqrst.Based on #103, which introduced the "no target file after a failure"
guarantee that
resumehas to except.Written with Claude Opus 5 via Claude Code; reviewed by me.
CC @ThomasBreuer