Skip to content

fix: verify says whose the other files in the directory are #80

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
donislawdev merged 1 commit into from
Sep 7, 2026
Merged

fix: verify says whose the other files in the directory are #80

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,31 @@ because it turns other people's test suites red.

### Fixed

- **`tfg verify` no longer calls another run's files "extra".** A directory is
allowed to hold more than one run - that is what `output.manifest` is for -
and verifying one of them reported every file the other had written as a file
nobody asked for, then called the directory a mismatch. Measured with two runs
into one directory whose file names do not collide, both ending `0`: three
differences against one manifest and four against the other, every one of them
the neighbour's work.

They are reported as `another-run` now, each one naming the record that lists
it, and they no longer make the directory a mismatch. `tfg verify` on a shared
directory ends `0`, and `matched` in `--json` is `true`. A real disagreement
is unaffected - a missing or changed file is still a mismatch and still exits
`7`.

**Nothing is hidden.** Every file is still in the report: one entry each in
`--json`, and in the prose one line per neighbouring record rather than one
per file. A directory holding a neighbour's ten thousand files used to print
ten thousand and one lines and exit `7`. It now prints one line and exits `0`.

Two limits worth knowing. A neighbour's record is recognised only when its
name ends in `.json`, because opening every unlisted file was measured and was
too expensive - a record under another name is reported the way it was before.
And a file that no manifest in the directory lists is still `extra`, so
leaving a manifest in a directory does not account for everything in it.

- **Two runs writing into one directory can no longer write over each other's
files.** A run holds the directory it is writing into for as long as it is
writing. A second run that starts meanwhile is refused before it writes a
Expand Down
114 changes: 88 additions & 26 deletions internal/audit/audit.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,29 @@ const (
// authority over what may be removed and a file that never finished never
// reached it. So the only useful thing is to say plainly what it is.
Leftover Kind = "leftover"
// AnotherRun is a file in the directory that a DIFFERENT run's manifest,
// sitting in the same directory, claims - or that manifest itself.
//
// Reported apart from Extra for the reason Leftover and Respelled are, and
// it is the third time that reason has come up. Extra means somebody else
// put it here and the question is whose. This one has an answer to that
// question, written down in the same directory: the neighbouring record
// names it, and Want carries the name of that record.
//
// A directory is allowed to hold more than one run. output.manifest exists
// so that a second run can record itself beside the first rather than being
// refused, and calling the result a mismatch made that unusable in the
// place it is for - a CI job cannot have a check that is red whenever it
// worked. Measured on 2026-09-07: two runs into one directory with names
// that do not collide, both ending 0, and verify then reported three
// differences against one manifest and four against the other, every one of
// them the other run's work.
//
// It does not make the directory a mismatch, and it is still printed. That
// pair is the whole design: attribution rather than suppression, so a
// manifest somebody drops into a directory can claim a file out loud and
// cannot hide one.
AnotherRun Kind = "another-run"
// Respelled is the file this manifest describes, stored under a spelling
// the filesystem treats as the same name.
//
Expand Down Expand Up @@ -124,6 +147,21 @@ func (d Difference) String() string {
"Nothing described by this manifest is missing because of it. "+
"cleanup will not remove it, because it removes only what the manifest lists - delete it by hand",
d.Path)
case AnotherRun:
// Two sentences, because the file is either the neighbour's record or
// one of the files it lists, and what a reader does about them differs.
// Naming the record in the second is the point: "somebody else's" is
// only useful when it says which somebody.
if d.Want == "" {
return fmt.Sprintf(
"another-run %s\n the record of another run that wrote into this directory. Nothing "+
"described by this manifest is affected by it. Verify it on its own to check the files it lists",
d.Path)
}
return fmt.Sprintf(
"another-run %s\n written by the run that %s records, not by this one. Nothing described by "+
"this manifest is missing because of it, and cleanup will not remove it - run cleanup on %s to remove it",
d.Path, d.Want, d.Want)
case Respelled:
return fmt.Sprintf(
"respelled %s\n the manifest calls this file %s. The letters differ only in a way this "+
Expand Down Expand Up @@ -262,6 +300,7 @@ func Verify(ctx context.Context, dir string, m *manifest.Manifest, skip string)
// above asks before it, and Verify ends on ctx.Err() - so a check would be a
// branch no test could ever redden, which this project removes rather than
// keeps.
var unclaimed []string
for _, p := range present {
// Not normalised on this side, and that was measured rather than
// decided. walk builds these with filepath.Rel, which returns a clean
Expand All @@ -270,32 +309,13 @@ func Verify(ctx context.Context, dir string, m *manifest.Manifest, skip string)
if seen[p] || filepath.Base(p) == skip {
continue
}
// Ours or somebody else's, and the reader needs to be told which.
kind := Extra
want := ""
switch {
case core.IsPartialName(filepath.Base(p)),
core.IsWritingName(filepath.Base(p)),
core.IsRunLockName(filepath.Base(p)):
// All three markers, because each names something this tool put
// here and did not take away. Only the first was recognised until
// 2026-09-06, so a half written manifest was reported as "extra" -
// the word that means somebody else put it here - and the third
// arrived with the run lock on 2026-09-07. They get different
// sentences in String, because what a reader should do about them
// differs, and about the lock it differs most: it is the only one
// that may belong to a run that is still going.
kind = Leftover
default:
// One file under two spellings reads as a polluted directory
// otherwise, and on a filesystem that ignores the difference it
// arrives on its own, without the "missing" that would give it
// away - os.Stat found the entry under the name the manifest
// gives. Measured on 2026-08-27.
if claimedAs, ok := folded[core.FoldName(p)]; ok {
kind, want = Respelled, claimedAs
}
}
unclaimed = append(unclaimed, p)
}
// Read before the loop rather than inside it, because a file the FIRST
// neighbour lists may sit before that neighbour's own record in the walk.
neighbours := findNeighbours(ctx, dir, unclaimed)
for _, p := range unclaimed {
kind, want := nameFor(p, folded, neighbours)
diffs = append(diffs, Difference{Kind: kind, Path: p, Want: want})
}

Expand All @@ -308,6 +328,48 @@ func Verify(ctx context.Context, dir string, m *manifest.Manifest, skip string)
return diffs, ctx.Err()
}

// nameFor says what a file the manifest does not claim actually is.
//
// Ours or somebody else's, and the reader needs to be told which. Extra is the
// last answer rather than the first, and three of the four ahead of it were put
// there by a report that had used the word about a file that was not anybody
// else's.
//
// Lifted out of the walk on 2026-09-07 when the fourth answer arrived. The loop
// it came from had a switch inside it and a chain inside that, and one more
// branch would have made the question harder to read than the answer.
func nameFor(p string, folded map[string]string, neighbours neighbourClaims) (Kind, string) {
base := filepath.Base(p)
// All three markers, because each names something this tool put here and
// did not take away. Only the first was recognised until 2026-09-06, so a
// half written manifest was reported as "extra" - the word that means
// somebody else put it here - and the third arrived with the run lock on
// 2026-09-07. They get different sentences in String, because what a reader
// should do about them differs, and about the lock it differs most: it is
// the only one that may belong to a run that is still going.
if core.IsPartialName(base) || core.IsWritingName(base) || core.IsRunLockName(base) {
return Leftover, ""
}
// One file under two spellings reads as a polluted directory otherwise, and
// on a filesystem that ignores the difference it arrives on its own,
// without the "missing" that would give it away - os.Stat found the entry
// under the name the manifest gives. Measured on 2026-08-27.
//
// Asked before the neighbours are, because this one is about a file THIS
// manifest describes. A neighbour that happens to list the same name does
// not make the spelling somebody else's problem.
if claimedAs, ok := folded[core.FoldName(p)]; ok {
return Respelled, claimedAs
}
if neighbours.records[p] {
return AnotherRun, ""
}
if by, ok := neighbours.claimedBy[p]; ok {
return AnotherRun, by
}
return Extra, ""
}

// comparablePath is the spelling two paths are matched under when one comes
// from a manifest and the other from the disk.
//
Expand Down
140 changes: 140 additions & 0 deletions internal/audit/neighbours.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
// Part of package audit. See audit.go.
package audit

import (
"context"
"path/filepath"
"strings"

"github.com/donislawdev/TestingFilesGenerator/internal/manifest"
)

// What the OTHER runs recorded in a directory say they wrote.
//
// A directory is allowed to hold more than one run. output.manifest exists for
// exactly that, so that a second run records itself beside the first instead of
// being refused, and people use it - a set of fixtures per test suite, one
// directory. What verify did with it was call every one of the neighbour's
// files "extra", which is the word for a file nobody asked for, and the report
// then read as a directory somebody had polluted.
//
// Measured on 2026-09-07, two runs one after another into one directory with
// name templates that do not collide, both ending 0:
//
// verify manifest-alpha.json 3 differences, exit 7
// verify manifest-beta.json 4 differences, exit 7
//
// Every one of those differences was the other run's work, and which run had
// written it was recorded in the same directory the whole time.
//
// The rule this follows is ATTRIBUTION, NOT SUPPRESSION, and the difference
// matters more than the repair. Every file stays in the report. What changes is
// the word it is given and whether it makes the directory a mismatch. So
// untouchable rule 6 is kept literally rather than on trust, and a manifest
// somebody drops into a directory cannot hide a file - at most it can claim
// one, out loud, with its own name printed beside it.

// neighbourClaims is what the other runs in this directory account for.
type neighbourClaims struct {
// records is the path of every file that is itself another run's manifest.
records map[string]bool
// claimedBy maps a file to the base name of the record listing it, which
// is what a reader needs in order to know which run to ask.
claimedBy map[string]string
}

// findNeighbours reads the manifests of other runs sitting in this directory.
//
// candidates are the files this manifest does not claim, which is the only set
// worth looking at: anything our own manifest lists has already been compared
// against it, and a neighbour listing it too cannot change that answer.
//
// A candidate that cannot be read, or that is not a manifest, or that is a
// manifest this build refuses, is simply not a neighbour. Nothing is reported
// about it here and it keeps whatever verify would have called it. Guessing on
// a file we could not read is how a tool starts accounting for files nobody
// wrote.
func findNeighbours(ctx context.Context, dir string, candidates []string) neighbourClaims {
found := neighbourClaims{
records: map[string]bool{},
claimedBy: map[string]string{},
}
for _, rel := range candidates {
if ctx.Err() != nil {
// A cancelled pass stops looking. The caller reports what it
// compared and calls nothing sound.
return found
}
if !couldBeNamedLikeARecord(rel) {
continue
}
full := filepath.Join(dir, filepath.FromSlash(rel))
m, err := manifest.Load(full)
if err != nil {
continue
}
found.records[rel] = true
base := filepath.Base(rel)
found.take(m, base)
}
return found
}

// take records what one neighbour says it wrote.
//
// Its own function rather than a loop inside a loop, which is the ceiling on
// how deeply this project nests talking. It is also the better shape to read:
// the caller decides WHICH files are records, and this decides what a record
// accounts for.
func (n neighbourClaims) take(m *manifest.Manifest, base string) {
for _, f := range Claimed(m) {
// First record wins, and the walk that produced the candidates is in a
// fixed order, so two neighbours claiming one file name the same one of
// themselves every time rather than whichever was read first on the day.
//
// The path is put through the same spelling both sides of the
// comparison use. Without it a neighbour writing "./a.txt" would claim
// nothing, which is the fault comparablePath was written for.
key := comparablePath(f.Path)
if _, taken := n.claimedBy[key]; !taken {
n.claimedBy[key] = base
}
}
}

// couldBeNamedLikeARecord is the sieve that costs nothing, and it had to exist.
//
// Reading the first bytes of every file the manifest does not claim was the
// first design and it was measured out of existence on 2026-09-07: on a
// directory holding ten thousand unclaimed files it made verify several times
// slower, because opening a file on Windows is not free - a scanner sees every
// one of them. The exact factor is refused, because the canary runs of the
// unchanged binary disagreed with each other by a factor of seven while it was
// taken. What is not in doubt is that it was large enough to change the design.
//
// So a file is only opened when its name could be a record at all. This is a
// NARROWING rather than a guess about the world, and the difference is what
// makes it safe: a neighbour's manifest under some other extension is reported
// exactly as it was reported before any of this existed, as extra. The cost of
// the sieve being wrong is yesterday's answer rather than a wrong one.
//
// The extension is the one this tool writes and the one it documents.
// DefaultManifestName is manifest.json, the help for verify says
// "tfg verify <manifest.json>", and every preset and every example writes one.
// Written down in docs/SHARED-DIRECTORY-2026-09-07.md section 2.3 as a limit
// rather than left to be discovered.
//
// It is the only sieve, and a second one was written and taken out again the
// same day. That one read the first half kilobyte of every candidate and looked
// for the first key of a manifest, to keep a large JSON document that is not one
// from being read in full. Nothing could make it fail: a document that got past
// it was refused by manifest.Load's schema check anyway, so removing it changed
// no answer and no test could tell. A defence nothing can redden is not a
// defence, and this project takes those out rather than keeping them.
//
// What is left is the name, and then manifest.Load, which has a ceiling of its
// own - so the worst an unclaimed JSON document can cost is one read of at most
// that ceiling.
func couldBeNamedLikeARecord(rel string) bool {
return strings.EqualFold(filepath.Ext(rel), ".json")
}
Loading
Loading