Skip to content

fix(reader): stray bytes that read as an element no longer pass for a cut or cost media - #34

Open
gravity-zero wants to merge 9 commits into
masterfrom
fix/head-junk-read-as-element
Open

gravity-zero wants to merge 9 commits into
masterfrom
fix/head-junk-read-as-element

Conversation

@gravity-zero

@gravity-zero gravity-zero commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Stray bytes that happen to decode as an element header made a complete file read as cut short, or lose media without a word. Found on a real file: 6 bytes of an old tag string left ahead of the first Cluster by an in-place edit from before 0.56.0 (fixed there by 990a7cb; reproduced here with a binary built just before it).

What changes, one commit each:

  • d5ba131 - the block walk resynchronizes past head bytes that read as an element, as Read and ReadMeta already did. analyze, ingest -analyze and validate no longer say "the source stops before its declared end" on a complete file, nor lose the first Clusters with exit 0.
  • 5f5a5fb - a Void or CRC-32 declaring a few bytes too many no longer swallows the element behind it; reindex --resync and salvage keep the Info and Tracks behind such bytes (the output had no track, exit 0); keyframes -walk keeps its keyframes.
  • cdd2211 - up to 64 stray bytes between the last Cluster and the Cues or Tags closing the Segment end the walk as the end of the media. A longer run is still refused: it may hide a Cluster.
  • 0da6d98 - diagnose walks the head to the first Cluster (it answered healthy when the Cues and Tags sat ahead of the bytes); the strict refusals of reindex, reindex-inplace and retime name the offset and mkvgo reindex --resync.
  • 30ee97d - bytes that read as a Cluster too small to hold a block are passed over; analyze warns when it read past undecodable bytes.

Then an audit of the branch and of the bitmap-subtitle decoders, never audited (two read-only reviewers, every finding reproduced by a failing test before its fix):

  • 2031d2e - padding declaring too much ahead of a Cluster that is not the first resumes on that Cluster (the last one was dropped with no error, a defect of cdd2211); a Cluster whose size was damaged but that opens with a Timestamp is not taken for stray bytes; padding counts for its excess, not its whole length; matroska.WithWholeHead.
  • 1039fc9 - a block's composites share a two-plane budget (381 MiB allocated for a 200-byte block before); a PGS object's buffer grows with its data (400 MiB for 1.4 KB before); a nil SubtitleIndex refuses MarshalBinary.
  • f0e7345 - a PGS block hands back two planes at most (256 MiB for 115 bytes before); a VobSub control sequence with no end command stops where the next one starts; extract-subtitle -format png encodes a picture shown again once.
  • 2a551ce - to-mp4, to-webm, to-hls, demux, split and remove-track warn when the Matroska source was read past undecodable bytes, and exit 3 under --exit-damaged.

Added: Diagnosis.ResyncedBytes (resynced_bytes) and reader.WithWholeHead. gorelease asks for v0.71.0.

Checks, on 2a551ce:

  • make preflight RELEASE=v0.71.0 exit 0 in a container (gorelease, linux/windows/darwin/wasm/arm builds, lint, tests, 386, poisoned buffers, wasm smoke).
  • go test -race on mkv/reader, mkv/ops, mkv/subtitle and the commands, exit 0; the fuzz targets of mkv/reader and of the subtitle decoders, 20 s each, exit 0.
  • A matrix of 53 damaged variants of two sound files (9 byte shapes at 6 places, Voids declared too long) against about 20 commands, compared with the sound file by content fingerprint: no command exits 0 with a different content. The 28 variants with the bytes in the head, ahead of the first Cluster or after the last one give the sound file's content on every command; bytes between two Clusters or inside one are still refused by the strict walks, by name, and reindex --resync recovers them.
  • ffprobe reads the same packet counts from the MP4 and the resynced MKV written from damaged variants as from the sound file, and decodes them with no error.
  • The downstream consumer's full suite against this branch and against v0.70.0: 12599 unit tests and 97 integration tests, no difference test by test. Its bitmap-subtitle path on real files (18 PGS tracks of 8 films, 21 VobSub tracks, 5 DVB captures, a 2059-cue .sup): every track's cues byte for byte identical to v0.70.0.

Not covered: -race on the whole module (CI runs it).

Known and left as is:

  • the strict walks of reindex and retime report the offset where the read breaks, not where the stray bytes start;
  • the commands that write warn of stray bytes in the head only: those ahead of the closing elements are passed over without a word.

…lement, so a complete file is not reported cut short nor loses its first Clusters
…ement behind it, and the resyncing rewrite and the keyframe walk keep what sits behind head bytes that read as an element
…nts closing the Segment end the block walk instead of reading as a cut
…ed bytes are a field, and the strict refusals of bytes read as an element name the offset and the command that works
…re passed over, and analyze says what it read past
… it, a Cluster with a shrunk size is not stray, and padding counts for its excess
…bject's buffer grows with its data, and a nil subtitle index refuses to marshal
…quence ends where the next starts, and a picture shown again is PNG-encoded once
…was read past undecodable bytes, and exit 3 under --exit-damaged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant