Skip to content

perf: planning a png or a gif stops coding the picture twice #58

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
donislawdev merged 2 commits into from
Sep 6, 2026
Merged

perf: planning a png or a gif stops coding the picture twice #58

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9694f11
perf: planning a png or a gif stops coding the picture twice
donislawdev Sep 6, 2026
945629c
Merge branch 'main' into perf/plan-without-encoding
donislawdev Sep 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,25 @@ because it turns other people's test suites red.

### Changed

- **Producing `.png` and `.gif` files is about twice as cheap.** Working out
what a file will contain used to draw the whole picture and compress it, only
to throw the result away and do it again when the file was actually written.
It now does that once.

Measured on 300 files of 200 kB: `.png` takes **2.0 times less processor time
and 1.8 times less wall clock**. For `.gif`, 1.8 and 1.6.

**The files are byte for byte identical.** This changes only how the work is
ordered, and it was checked that way - across sizes either side of every step
in the picture ladder, for several seeds, with the label on and off.

A preview (`--dry-run`) of a large run gets the bigger share of this, since
previewing was almost entirely the work now removed.

`.jpg` is unchanged and cannot get the same treatment: it writes its padding
in front of the picture, so it has to know how large the picture is before it
starts.

- **`verify` and `cleanup` read the files over several threads, so checking a
large run is several times faster.** Nothing about what they report changes -
the same differences, in the same order, with the same exit codes.
Expand Down
61 changes: 56 additions & 5 deletions internal/format/gif/gif.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,11 @@ type memo struct {
label string
// body is the encoded picture up to but not including the trailer.
body int64
// bodyKnown says whether planning worked that out. For a request far
// above what the largest rung encodes to, the answer cannot change which
// picture is chosen, so planning skips the encoding and the writer fills
// this in. See ladderCeiling.
bodyKnown bool
// payload is how many bytes of filler the comment carries, and blocks how
// many sub blocks carry them. Both zero means no comment at all.
payload int64
Expand Down Expand Up @@ -209,8 +214,14 @@ func (generator) Plan(r format.Request) (format.Plan, error) {
},
}

if err := settlePadding(&m, r.Bytes, bare); err != nil {
return format.Plan{}, err
// With the body unknown the padding cannot be settled yet, and it does not
// need to be: the fast path in chooseSize already established there is room
// for a comment carrying whatever is left. The writer settles it once it
// has encoded, which it has to do anyway.
if m.bodyKnown {
if err := settlePadding(&m, r.Bytes, bare); err != nil {
return format.Plan{}, err
}
}

labelled := r.Label && imagelabel.Fits(w, len(label))
Expand Down Expand Up @@ -312,7 +323,17 @@ func (generator) Write(ctx context.Context, w io.Writer, p format.Plan) error {
if err := encode(holder, m); err != nil {
return err
}
if holder.written != m.body {
if !m.bodyKnown {
// Planning skipped the encoding, so this is where the exact size
// arrives and the padding gets settled - the same arithmetic planning
// would have done with the same number.
m.body = holder.written
if err := settlePadding(&m, p.Bytes, m.body+trailerSize); err != nil {
// Unreachable unless ladderCeiling is wrong, and then saying so
// beats writing a file of the wrong length.
return fmt.Errorf("gif: %w - ladderCeiling is wrong", err)
}
} else if holder.written != m.body {
return fmt.Errorf("gif: the picture encoded to %d B where planning said %d B", holder.written, m.body)
}
if holder.tail[0] != 0x3B {
Expand Down Expand Up @@ -368,6 +389,21 @@ func writeComment(ctx context.Context, w io.Writer, seed uint64, blocks, payload
return err
}

// ladderCeiling is the most the largest rung has ever been seen to encode to,
// with room to spare. It only ever decides that a request is far enough above
// the ladder that no search is needed, so being generous costs a few sizes
// their fast path and being wrong costs nothing silently - the writer refuses
// rather than producing a file of the wrong length.
//
// Measured 2026-09-06 at 640x480 over three seeds, the label both on and off,
// and one, three, ten and sixty frames: 54518 B at the smallest and 64020 B at
// the largest. Frames barely move it, about 150 B each, which is why this is
// one number rather than a function of the frame count.
//
// TestTheLadderCeilingIsAboveEveryPictureTheTopRungMakes sweeps it rather than
// trusting this comment.
const ladderCeiling = 98304

// sizeLadder is tried from the largest down when the recipe names no picture
// size, exactly as PNG does. The first rung that leaves a reachable remainder
// wins, so a small file gets a small picture instead of being refused.
Expand Down Expand Up @@ -402,18 +438,33 @@ func chooseSize(r format.Request, label string) (memo, error) {
if err != nil {
return memo{}, err
}
m.body = body
m.body, m.bodyKnown = body, true
return m, nil
}

// Planning does not have to encode the picture to know which rung wins.
// The ladder is walked largest first, so for a request comfortably above
// what the largest rung encodes to, that rung is the answer and encoding
// only confirms it - at the cost of a whole encode thrown away so the
// writer can do it again (P7 in the 2026-09-05 performance review).
//
// A fast path, not a change of answer: it fires only where the rung is
// already settled, and the margin also guarantees the comment can carry
// whatever is left, so none of the refusals in settlePadding are reachable
// from here.
if r.Bytes >= ladderCeiling+trailerSize+smallestCarryingComment {
rung := sizeLadder[0]
return memo{width: rung[0], height: rung[1], frames: frames, seed: r.Seed, label: label}, nil
}

var smallest memo
for _, rung := range sizeLadder {
m := memo{width: rung[0], height: rung[1], frames: frames, seed: r.Seed, label: label}
body, err := encodedBodySize(m)
if err != nil {
return memo{}, err
}
m.body = body
m.body, m.bodyKnown = body, true
smallest = m

bare := body + trailerSize
Expand Down
83 changes: 76 additions & 7 deletions internal/format/png/png.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,10 +118,16 @@ type memo struct {
width, height int
seed uint64
label string
// body is the exact number of bytes the encoded picture takes before the
// closing chunk. Worked out during planning so that a size this format
// cannot reach is refused before any file exists.
body int64
// body is the number of bytes the encoded picture takes before the closing
// chunk. Worked out during planning so that a size this format cannot
// reach is refused before any file exists.
//
// bodyKnown says whether it was worked out at all. For a request far above
// what the largest rung can encode to, the answer cannot change which
// picture is chosen, so planning skips the encoding and the writer - which
// has to encode anyway - fills both fields in. See ladderCeiling.
body int64
bodyKnown bool
// padData is how many bytes of padding the chunk carries. A negative
// value means no chunk at all, which happens when the picture lands
// exactly on the requested size.
Expand Down Expand Up @@ -184,6 +190,14 @@ func (generator) Plan(r format.Request) (format.Plan, error) {
bare := body + iendSize

switch {
case !m.bodyKnown:
// The fast path in chooseSize already established that this request is
// far above the largest rung and that one chunk can carry the padding,
// so all three refusals below are unreachable and the only number still
// missing is how much padding there is. The writer settles that once it
// has encoded, which it has to do anyway.
m.withPad = true

case r.Bytes == bare:
// The picture lands exactly on the requested size. No padding chunk.
m.withPad = false
Expand Down Expand Up @@ -263,7 +277,21 @@ func (generator) Write(ctx context.Context, w io.Writer, p format.Plan) error {
return err
}

if holder.written != m.body {
if !m.bodyKnown {
// Planning skipped the encoding because the request was far above the
// largest rung, so this is where the exact size arrives. The padding is
// whatever is left, which is the same arithmetic planning would have
// done with the same number.
m.body = holder.written
m.padData = p.Bytes - m.body - iendSize - chunkOverhead
if m.padData < 0 || m.padData > maxChunkData {
// Unreachable unless ladderCeiling is wrong, and then it is better
// to say so than to write a file of the wrong length.
return fmt.Errorf(
"png: the picture encoded to %d B, which leaves %d B of padding for a %d B file - ladderCeiling is wrong",
m.body, m.padData, p.Bytes)
}
} else if holder.written != m.body {
return fmt.Errorf("png: the picture encoded to %d B where planning said %d B", holder.written, m.body)
}
if string(holder.tail[4:8]) != "IEND" {
Expand All @@ -280,6 +308,22 @@ func (generator) Write(ctx context.Context, w io.Writer, p format.Plan) error {
return err
}

// ladderCeiling is the most the largest rung has ever been seen to encode to,
// with room to spare. It is only ever used to decide that a request is far
// enough above the ladder that no search is needed, so being generous costs a
// few sizes their fast path and being wrong costs nothing silently - a picture
// larger than this simply leaves less padding, and the writer would refuse
// rather than produce a wrong file.
//
// Measured 2026-09-06 over ten seeds with the label both on and off: 5456 B at
// the smallest and 5808 B at the largest, a spread of 352 B. The gradient
// compresses about 210 to 1, so the number is nowhere near the 1229280 B that
// an incompressible 640x480 picture would take.
//
// TestTheLadderCeilingIsAboveEveryPictureTheTopRungMakes sweeps it rather than
// trusting this comment.
const ladderCeiling = 16384

// sizeLadder is tried from the largest down when the recipe names no picture
// size. The first rung that leaves room for the padding chunk wins, so a
// small file gets a small picture instead of being refused.
Expand Down Expand Up @@ -321,18 +365,43 @@ func chooseSize(r format.Request, label string) (memo, error) {
if err != nil {
return memo{}, err
}
m.body = body
m.body, m.bodyKnown = body, true
return m, nil
}

// Planning does not have to encode the picture to know which rung wins.
//
// The ladder is walked from the largest rung down and the first one that
// fits is taken, so for any request comfortably above what the largest rung
// encodes to, the answer is the largest rung and encoding only confirms it.
// That confirmation was 38 to 53% of a PNG run - a whole encode, thrown
// away, so that the writer could do it again (P7 in the 2026-09-05
// performance review).
//
// This is a fast path and NOT a change of answer. It fires only where the
// rung is already settled, so the bytes are the ones the slow path below
// produces. Everything near a rung boundary still encodes and still gets
// the exact number.
//
// The second condition keeps the refusal above the chunk limit exact.
// Padding is r.Bytes minus the picture and the overheads, so it is largest
// when the picture is smallest, and a picture is never smaller than
// nothing. Bounding it that way costs a fallback to the slow path for a
// sliver of sizes just under two gigabytes and keeps the refusal honest.
if r.Bytes >= ladderCeiling+iendSize+chunkOverhead &&
r.Bytes-iendSize-chunkOverhead <= maxChunkData {
rung := sizeLadder[0]
return memo{width: rung[0], height: rung[1], seed: r.Seed, label: label}, nil
}

var smallest memo
for _, rung := range sizeLadder {
m := memo{width: rung[0], height: rung[1], seed: r.Seed, label: label}
body, err := encodedBodySize(m)
if err != nil {
return memo{}, err
}
m.body = body
m.body, m.bodyKnown = body, true
smallest = m

bare := body + iendSize
Expand Down
Loading
Loading