Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 35 additions & 25 deletions .github/race-weights
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,14 @@
# heaviest first, and sends every test not named here by the checksum of its
# name, counted at the "unlisted" weight.
#
# Taken on 2026-11-03 from eight parts of two runs, each test at the longest
# time any of them took. The race job runs on whatever processor the runner
# is given, and the same part took 689 s on one and 1202 s on another, every
# test of it slower by the same factor. The longest time is the slowest
# processor's, so the parts are even where they are closest to the timeout.
# Taken on 2026-11-03 from eight parts of two runs, and on 2026-11-12 from the
# four parts of one more, each test at the longest time any of them took. The
# race job runs on whatever processor the runner is given, and the same part
# took 689 s on one and 1202 s on another, every test of it slower by the same
# factor. The longest time is the slowest processor's, so the parts are even
# where they are closest to the timeout. A time from a faster processor is
# scaled up by its part: on 2026-11-12 the listed tests of two parts on an AMD
# EPYC 9V74 took 1/1.20 and 1/1.15 of their weights.
# Listed: every test that took 10 s or more. "unlisted" is the mean of the rest.
#
# A test that is not here and takes 60 s or more fails its part with the line
Expand All @@ -16,50 +19,57 @@ unlisted 0.58
348.3 TestAFormatRefusesASizeItCannotDescribe
343.3 TestAFilmStoppedHalfWayLeavesNoHelperRunning
329.9 TestEveryFormatHitsTheOrderedSizeExactly
167.5 TestACompressedZipIsEitherWrittenAtItsSizeOrRefusedInWords
162.6 TestEveryScreenStillDrawsItsStoredPicture
155.5 TestACompressedZipIsEitherWrittenAtItsSizeOrRefusedInWords
142.2 TestEveryTileOfAFilmIsGav1dsCodingOfItsPixelsAndDecodesAsItDoesAlone
134.0 TestEveryAvifRungStillFitsInsideItsDeclaredCeilings
130.0 TestTheMinimumThisToolPrintsStandsWhateverTheSeedIs
117.9 TestFreeBoxesCarryPaddingOfAnyLength
124.3 TestFreeBoxesCarryPaddingOfAnyLength
99.9 TestEveryBase64TextSizeIsExact
93.8 TestAFilmPictureLargerThanAnyAV1LevelIsRefusedNotBroken
73.6 TestEveryAvifRungStillFitsInsideItsDeclaredCeilings
68.4 TestEveryFormatsMapIsTheOneItsStructureReadsBack
67.9 TestADeclaredDefaultIsTheOneTheFormatUses
84.3 TestEveryFormatsMapIsTheOneItsStructureReadsBack
68.2 TestADeclaredDefaultIsTheOneTheFormatUses
65.9 TestNoGeneratorHoldsTheWholeFileInMemory
65.6 TestADamagedFileIsExactlyTheSizeAskedFor
64.3 TestNoTrackedFileCarriesPrivateContent
63.8 TestNoGeneratorHoldsTheWholeFileInMemory
60.8 TestEveryDamageIsRefusedByTheStructuralCheckInEveryFormat
64.6 TestNoTrackedFileCarriesPrivateContent
62.2 TestEveryDamageIsRefusedByTheStructuralCheckInEveryFormat
57.3 TestAFilmSaysItCarriesItsLabelOnlyWhenAPictureShowsItWhole
55.6 TestEveryJxlRungStillFitsInsideItsDeclaredCeiling
55.6 TestTheArchiveScanLooksIntoTheArchivesInside
55.3 TestAFilmOfANamedSizeKeepsEveryPictureInsideItsReserve
53.4 TestTheArchiveScanLooksIntoTheArchivesInside
38.5 TestAContainerStopsBetweenTheFilesItPlans
52.2 TestChoosingAToolLaysTheScreenOutAsIfItOpenedWithIt
38.6 TestAContainerStopsBetweenTheFilesItPlans
37.5 TestEveryPictureNamedAfterARefusalShowsOne
36.7 TestJxlStillFitsWhenTheQualityIsNotTheOneTheCeilingsWereMeasuredAt
36.7 TestTheArchiveScanReadsTheListArchiveZipReads
35.9 TestTheArchiveScanFindsEveryTrapItNames
35.5 TestJxlStillFitsWhenTheQualityIsNotTheOneTheCeilingsWereMeasuredAt
30.8 TestOurOwnGeneratorsHaveNotDrifted
34.0 TestOurOwnGeneratorsHaveNotDrifted
29.9 TestTheSameSeedGivesTheSameBytes
26.8 TestEveryJxlRungStillFitsInsideItsDeclaredCeiling
26.6 TestTheMinimumStandsWhateverTheSeedIsUnderEveryDeclaredChoice
27.9 TestTheMinimumStandsWhateverTheSeedIsUnderEveryDeclaredChoice
25.9 TestEveryLinkOnTheSiteLeadsSomewhere
24.9 TestEveryEMLSettingSurvivesItsReferenceTool
24.2 TestEveryLinkOnTheSiteLeadsSomewhere
24.2 TestAPictureSaysItCarriesItsLabelOnlyWhenItsTextIsWhole
23.2 TestAnAVIFLargerThanOneAV1TileSurvivesItsReferenceTool
23.1 TestACutTakesTheShareOfTheContentItIsAskedFor
22.4 TestTheLadderCeilingIsAboveEveryPictureTheTopRungMakes
22.2 TestTheSiteSaysWhatTheToolSays
20.9 TestAFilmsProgressMovesWithItsPicturesNotItsPadding
20.7 TestAFilmHoldsTheFramesTheManifestDeclares
20.7 TestTheWindowBinaryStartsWithoutAConsole
19.9 TestAFilmWhoseSquareCoversItsLabelSaysSoInItsPlan
19.9 TestTheSiteSaysWhatTheToolSays
19.8 TestChoosingAToolLaysTheScreenOutAsIfItOpenedWithIt
20.3 TestAFilmWhoseSquareCoversItsLabelSaysSoInItsPlan
19.8 TestTheArchiveScanHoldsNoMoreOfANameThanTheName
17.4 TestThePresetScreenKeepsTheNameAFileIsSentUnder
17.0 TestTheSmallestDamagedFileIsTakenAndOneByteLessIsNot
16.9 TestEveryFormatSurvivesItsReferenceTool
16.7 TestEveryLanguageDescribesEverythingTheProgramCanProduce
16.6 TestEveryLanguageDescribesEveryDamage
16.3 TestTheSameWorkDoneAgainLeavesTheWindowAsItWas
16.2 TestABuildWithNoWindowInItSaysSoAndKeepsStandardOutputEmpty
16.0 TestEveryLanguageDescribesEverythingTheProgramCanProduce
15.4 TestAvifStillFitsWhenTheQualityIsNotTheOneTheCeilingsWereMeasuredAt
15.1 TestABuildWithoutTheBuildTagsRefusesAndSaysWhy
14.9 TestAShortcutCannotStartASecondRunDuringTheFirst
14.6 TestTheLicenceCommandListsWhatTheBinaryLinks
14.4 TestEveryFormatSurvivesItsReferenceTool
12.2 TestTheMemoryFlagReachesThePlanOnEveryCommandThatPlans
12.1 TestTheManifestOfARunPastTheOldCeilingIsVerifiedAndCleanedUp
12.0 TestTheUploadSetSendsFourFilesUnderNamesThatLeaveTheDirectory
11.2 TestTheCommandLineHoldsTheWholeRunToItsMemorySetting
14 changes: 8 additions & 6 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1116,14 +1116,16 @@ jobs:
github.event_name == 'workflow_dispatch' ||
needs.touched.outputs.concurrency == 'true'
runs-on: ubuntu-latest
# Four parts since 2026-09-24, the owner's decision. The whole package under
# Five parts, the owner's decision. In one process the whole package under
# the detector took 32 to 42 minutes, and six of the twelve runs on
# 2026-09-23 and 2026-09-24 were killed at the ceiling with no data race in
# their logs. Both ceilings had been raised once already, on 2026-08-31 -
# the job's from 30 to 45 minutes and Go's from 25 to 40 - for the same
# reason, and a problem that keeps coming back is the wrong shape rather
# than a missing notch. The guards run one after another - none calls
# t.Parallel - so four processes share the time between them, and every
# than a missing notch. Four parts weighed 1074 s each by the weights of
# 2026-11-12, past 85% of their twenty minutes, and two of them took 1055
# and 1067 s that day. The guards run one after another - none calls
# t.Parallel - so the processes share the time between them, and every
# test still runs under the detector once, in one of them.
#
# Every part reports. One red part does not cancel the others, whose tests
Expand All @@ -1139,7 +1141,7 @@ jobs:
strategy:
fail-fast: false
matrix:
part: [0, 1, 2, 3]
part: [0, 1, 2, 3, 4]
timeout-minutes: 35
env:
# The one thing in this project that needs a C toolchain. Linux runners
Expand Down Expand Up @@ -1227,8 +1229,8 @@ jobs:
# and stays below the job's ceiling, so a slow part fails as a test with
# a name rather than as a killed job with no output. Go's ten minutes
# killed this job on 2026-08-25, before a timeout was stated here.
# By the weights the heaviest part is 961 s on the slowest processor
# measured, and twenty minutes is 1.25 times that.
# By the weights the heaviest part is 861 s on the slowest processor
# measured, and twenty minutes is 1.39 times that.
#
# The processor's counters and the clock are read just before the
# tests, for the step after this one (O298).
Expand Down
20 changes: 13 additions & 7 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,13 +73,19 @@ because it turns other people's test suites red.
outcome is `unspecified`, offers both answers in the new `policy_options`
field and says why. `tfg damage <id>` lists those formats.
- **A file cut short says where the cut fell and how much of its content is
left.** In many formats a large file is mostly padding after its content,
so `--damage truncate:missing=90` on a 1 MB PNG still leaves the whole
picture. The `truncate` entry in the manifest's `damage` now carries `cut`:
`in` (`content`, `padding`, `structure` or `end`), `content_bytes` and
`content_kept_bytes`. A cut in the padding also gets the note
`cut_in_padding`. In a compressed tar.gz a cut in the padding is reported as
a cut in the content.
left.** In some formats a large file is mostly padding beside its content,
so a cut can take all of the content or none of it. The `truncate` entry in
the manifest's `damage` now carries `cut`: `in` (`content`, `padding`,
`structure` or `end`) and `content_kept_bytes`, and `content_bytes` where
the length of the content is known. A cut in the padding also gets the
note `cut_in_padding`. In a compressed tar.gz a cut in the padding is
reported as a cut in the content.
- **`truncate` cuts a PNG's picture by the percentage you set.**
`--damage truncate:missing=90` on a 1 MB PNG used to cut only padding and
leave the whole picture. Now it removes 90% of the picture's content as
well, so a viewer that shows what is left shows only the top of the
picture, the way an interrupted upload leaves it. Files nothing cuts are
unchanged.

- **tfg is on npm, for tests in Node.**
`npm install --save-dev testing-files-generator` gives Playwright, Vitest
Expand Down
13 changes: 8 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -522,11 +522,14 @@ how many more there are, and the manifest lists every one. The label is still
written before the damage, so the damage can overwrite it, and the manifest does
not promise it is there.

A cut falls on whatever the file holds at that point. A large picture, sound,
document or archive is mostly padding after its content, so `truncate` can leave
the content whole. The manifest says where each cut fell: the `truncate` entry
carries `cut`, with `in` (`content`, `padding`, `structure` or `end`),
`content_bytes` and `content_kept_bytes`. A cut in the padding also gets the note
In most formats `truncate` removes the same percentage of the content as of the
file: `missing=50` leaves about the top half of a PNG's picture, half of a WAV's sound
or half of a text's lines, the way an interrupted upload does. In some formats a
large file is mostly padding beside its content, so a cut there can take all of the
content or none of it. The manifest says where each cut fell: the `truncate` entry
carries `cut`, with `in` (`content`, `padding`, `structure` or `end`) and
`content_kept_bytes`, and with `content_bytes`, how long the content of the whole
file is, where that is known. A cut in the padding also gets the note
`cut_in_padding`.

The smallest file a damage can be given follows its settings, so the column is
Expand Down
16 changes: 10 additions & 6 deletions internal/damage/truncate.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,14 +48,18 @@ func init() {
// anyone says it, and the sentence below names the percent.
Min: fewestMissing, Max: mostMissing,
Default: strconv.Itoa(defaultMissing),
// The last sentence is the limit a person cannot see from the
// outside: in 13 formats of 32 the padding stands after the
// content, so a large file cut at any share keeps its whole
// picture or sound (docs/TRUNCATE-2026-11-11.md). The manifest
// says per file where the cut fell.
// The last two sentences are what a person cannot see from the
// outside. In most formats the cut removes the same percentage
// of the picture, sound or text, png among them since it lays
// its padding out for the cut. In the rest the padding stands
// only on one side of the content, so a large file cut at any
// share keeps all of it or none (docs/TRUNCATE-2026-11-11.md
// section 7.1, and notYetCutAtTheShare in internal/guard for
// which). The manifest says per file where the cut fell.
Detail: "How much of the whole file is gone, as a percentage. " +
"At least this much is cut - a few bytes more where the format cannot end exactly there. " +
"A large file is often mostly padding after its content, so the content can stay whole - " +
"In most formats the cut removes the same percentage of the content. " +
"In some, a large file is mostly padding beside its content, so the cut can take all of the content or none - " +
"the manifest says what each cut left.",
},
},
Expand Down
8 changes: 7 additions & 1 deletion internal/engine/damagedformat.go
Original file line number Diff line number Diff line change
Expand Up @@ -100,9 +100,15 @@ func (g *damagedGenerator) PlanContext(ctx context.Context, r format.Request) (f
// size bytes on disk, and answers the request it asked with.
//
// The label inside it states the file on disk, which is size - not the size
// the generator writes, nor one stretch moves it to.
// the generator writes, nor one stretch moves it to. Where a damage cuts the
// tail, the request says where, so a format can lay its content out for the
// cut. A size stretch moves up keeps the same cut, and so a smaller share of
// the content, which is the "at least" missing promises.
func (g *damagedGenerator) whole(ctx context.Context, r format.Request, want, size int64) (format.Plan, format.Request, error) {
r.Bytes, r.OnDisk = want, size
if _, kept, ok := g.chain.Cut(size); ok {
r.CutAt = kept
}
p, err := g.plan(ctx, r)
if err != nil && g.chain.Stretches() {
p, err = g.stretch(ctx, r, err)
Expand Down
12 changes: 12 additions & 0 deletions internal/format/format.go
Original file line number Diff line number Diff line change
Expand Up @@ -680,6 +680,18 @@ type Request struct {
// asks the generator for more than it keeps. Zero means Bytes is the size
// on disk. Read for the label alone, like Damage.
OnDisk int64
// CutAt is where a damage will cut the file this request writes, as the
// number of its first bytes that stay, or zero when nothing cuts it.
//
// A format whose padding can stand on both sides of its content splits
// the padding so that the cut keeps the same share of the content as of
// the whole file, CutAt of Bytes. Otherwise a large file is mostly padding
// after its content, and a cut of any share leaves the content whole.
// Bytes is still the whole file and the plan still writes all of it, so
// the bytes after the cut are the ones the generator never gets to. A
// format that leaves this alone writes as it always did, and the cut the
// manifest reports says so (docs/TRUNCATE-2026-11-11.md section 7.2).
CutAt int64
// Properties are format specific settings, straight from the recipe.
Properties map[string]string
}
Expand Down
Loading
Loading