fix: verify the size of a finished download - #291
Merged
Conversation
mkb79
force-pushed
the
fix/verify-download-size
branch
from
August 14, 2026 12:55
529602e to
9d015bf
Compare
The download size was never checked. `check_download_size` existed but nothing called it: the postprocessing list held `check_status_code` twice, once in the slot the size check belonged in. So an audiobook that arrived at a different length than announced was renamed to its final name and counted as a success. Three defects kept each other hidden: - The check was missing from `Downloader._postprocessing`, replaced by a duplicate of `check_status_code`. - `check_download_size` returned `DownloadSizeMismatch` whenever both sizes were known, equal or not — its `return` sat outside the comparison. Wiring the check up without fixing this would have failed every download. - `_check_and_return_download_result` never passed `head_response`, which is the only argument the size check actually needs. Wiring it up would have raised TypeError instead of checking anything. The old `utils.Downloader` does check the size, so covers and PDFs were covered while the audiobooks were not. What it catches: a download whose final length differs from the one the preceding request announced. Not a mid-transfer cut — httpx already rejects a response that ends early against its own Content-Length. An encoded transfer is left alone, whichever scheme it uses. Content-Length may then describe the encoded bytes while httpx may hand over decoded ones, so the two are not reliably comparable and the comparison could reject a sound download. The guard asks whether the transfer was plain rather than listing schemes, so gzip, deflate, br, zstd, chained values such as "gzip, br" and anything added later all skip the comparison; only an absent header and `identity` reach it. Deliberately out of scope, for the downloader work rather than a bug fix: parsing Content-Range, insisting on 206 for a resumed request, and tying the two requests together through If-Range. Size alone does not prove the content is right; it catches the length going wrong. The check runs last. A text response is reported through its own message first, so `DownloadIndividualParts` still reaches the caller instead of being reported as a size mismatch.
mkb79
force-pushed
the
fix/verify-download-size
branch
from
August 14, 2026 20:57
9d015bf to
5bdb2f0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The download size was never checked.
check_download_sizeexisted but nothing called it — the postprocessing list heldcheck_status_codetwice, once in the slot the size check belonged in:So an audiobook that arrived at a different length than announced was renamed to its final name and counted as a success.
Three defects, each hiding the next
check_download_sizemissing from_postprocessingcheck_status_codemade the list look fullreturn Status.DownloadSizeMismatchsat outside the comparison, firing even when the sizes matched_check_and_return_download_resultnever passedhead_response— the one argument the check needsTypeError, not checked anythingFixing any one alone does nothing useful. Fixing 1 without 2 would have failed every download.
The old
utils.Downloaderdoes check the size (utils.py:279), so covers and PDFs were covered while the audiobooks — what the tool exists for — were not.What it catches, and what it does not
Catches: a download whose final length differs from the one the preceding request announced.
Not a mid-transfer cut. httpx already rejects a response that ends early against its own
Content-Length, raisingRemoteProtocolErrorbefore postprocessing runs. Verified.Encoded transfers are skipped, whichever scheme.
Content-Lengthmay then describe the encoded bytes while what reaches the file may have been decoded on the way. The guard asks whether the transfer was plain rather than listing schemes, sogzip,deflate,br,zstd, chained values likegzip, brand anything added later all skip the comparison; only an absent header andidentityreach it.Deliberately out of scope
Parsing
Content-Range, insisting on206for a resumed request, and tying the two requests together withIf-Range. Those harden resume itself and belong with the downloader rework. Size alone does not prove the content is right; it catches the length going wrong.Verification
24 new tests, 101 → 125. Each defect and the encoding guard was reintroduced individually to confirm the tests fail. The wiring is proven through behaviour rather than by reading source:
_postprocessingruns over a prepared temp file and must returnDownloadSizeMismatchand leave the target absent.Encoding matrix: 9 schemes skip the comparison, 3 spellings of
identitystill reach it, and an encoding on either the probe or the real response is enough to skip.Smoke tested against the real API in a temp directory:
Success, no false positive, no leftoversSuccess38.9M/250M,Success, MP4 box chainftyp → moov → mdatends exactly flush with the file sizeSuccess, comparison correctly skipped