From 6150e3a292e48aa133c2035b9dc8b49c56e57568 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Fri, 20 Nov 2026 15:30:35 +0000 Subject: [PATCH 1/7] fix(api): refuse a branch or tag at a commit whose history the packs cannot walk MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit POST /repos/branches/create and POST /repos/tags wrote a ref at any commit that a registered pack holds. A push can leave a commit in a registered pack whose parent, tree or blob no pack holds, because receive-pack checks the commits that a push's refs reach and no other commit of its pack. A ref at such a commit hydrates, and every clone, fetch and archive that walks the history then fails, on every node. A ref write now asks the registered packs for the commit's whole history, as receive-pack does for the new tips of a push: git rev-list --objects --not --all over a view of the registered packs, with --missing=print so that the answer is the set of objects that git cannot read and never git's error text. The view has no commit-graph, because a commit-graph can describe a commit that no registered pack holds. A commit that a ref reaches ends the walk at once, and the oldest commit of 500,000 linear commits costs about two seconds. A walk of more than 30 seconds answers 503. A commit whose history the packs cannot walk answers 404, like a commit that does not exist, falls back to preferred_base, and writes an operator line to the API log. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_ --- docs/how-to/deploy-production.md | 2 + docs/reference/http-api.md | 14 ++ internal/api/handlers_write_refs.go | 58 ++++--- internal/api/registry_history_test.go | 177 ++++++++++++++++++++ internal/api/registry_target_test.go | 23 ++- internal/gitread/gitread.go | 79 +++++++++ internal/gitread/registered_closure_test.go | 157 +++++++++++++++++ 7 files changed, 479 insertions(+), 31 deletions(-) create mode 100644 internal/api/registry_history_test.go create mode 100644 internal/gitread/registered_closure_test.go diff --git a/docs/how-to/deploy-production.md b/docs/how-to/deploy-production.md index 0730fa47..374c8b37 100644 --- a/docs/how-to/deploy-production.md +++ b/docs/how-to/deploy-production.md @@ -735,6 +735,7 @@ webhook events. | Postgres availability + advisory-lock waits at boot | The database is the single authority; boot hangs show in `pg_locks` | | `verify token: key store lookup failed` / `verify token: a registered key cannot be used` log lines | A git, LFS or REST request reached the key store or a registered key, and it failed. The line holds the cause (a cause of more than 300 runes loses its middle). A git or LFS client reads a text that does not hold the cause. For each kind, the git and LFS routes together write at most one line per minute in each `serve` process, the REST routes write at most one more, and a line says how many earlier lines of its kind were held back when some were | | `receive-pack : hook: … git cannot walk the history between two commits, so the update is refused` log line | A push named two commits whose history git cannot read, so the hook refused that update: as non-fast-forward under `no-force-push`, and with `is not a readable commit in this repository` under `verify-sig`. The line has the fields `git`, `old` and `new`. A pusher can cause it with a commit whose parent is missing. A `` that repeats it for ordinary pushes from several clients has a damaged history: check a fresh copy with `git fsck` | +| `a ref write names a commit that the registered packs cannot walk` log line | A `POST /repos/branches/create` or `POST /repos/tags` named a commit that no registered pack holds, or whose parent, tree or blob no registered pack holds, and answered `404`. The line has the fields `repo` and `sha`. A pusher can leave such a commit in a pack (git checks only the commits that the refs of a push reach). One line is a refused request. A `repo` that repeats it with the same `sha` for ordinary branch creates has a repository whose registry lost a pack: compare `openstorage list-packs` with the blob store | | `lane scan: requeued a lane whose work had no job` log line | A sync or push request lost its job and the [lane scan](#background-maintenance-in-a-fleet) enqueued it again. A `repo=` that repeats needs a look: its runs keep crashing | The opt-in [metrics listener](../reference/metrics.md) covers some of these signals; @@ -767,6 +768,7 @@ line can be up to 1 MiB. | Postgres availability and advisory-lock waits (What to monitor) | **log only** | Observe PostgreSQL itself (`pg_locks`) | | `verify token: …` lines (What to monitor) | **log only** | The two texts shown, at most one line for each kind per minute for the git and LFS routes together, and one more for the REST routes, in each `serve` process | | `git cannot walk the history between two commits` (What to monitor) | **log only** | A hook line, so metrics do not count it. Fields `git`, `old`, `new` and `cause` | +| `a ref write names a commit that the registered packs cannot walk` (What to monitor) | **log only** | An API line, so metrics do not count it. Fields `repo` and `sha` | | `lane scan: requeued a lane whose work had no job` (What to monitor) | **log only** | Fields `org`, `repo` and `lane` | | S3WAL `… conditional-write contention retries exhausted` (S3WAL signals) | `openstorage_s3wal_cas_contention_exhausted_total` by `scope` | Serve-process mutations only; push-side failures reach the client and hook stderr | | `WARN s3wal manifest approaching size cap` (S3WAL signals) | `openstorage_s3wal_manifest_size_guard_total{result="warn"}` | Counts when the once-per-repository warning is logged | diff --git a/docs/reference/http-api.md b/docs/reference/http-api.md index fe305182..a572a21e 100644 --- a/docs/reference/http-api.md +++ b/docs/reference/http-api.md @@ -753,6 +753,20 @@ for a tag that exists or for a create that leaves the branch at its tip (with `f `expected_target_sha`). So a request that the policy denies answers `403` even when its commit is only in the cache. +The registered packs must also hold what git needs to read the commit's history: its ancestors, +down to the commits that a ref of the repository reaches, and their trees and blobs. A push can +leave a commit in a registered pack whose parent, tree or blob no pack holds, because +`receive-pack` checks the commits that a push's refs reach and no other commit of its pack. A ref +at such a commit would hydrate, and every clone, fetch and archive that walks the history would +fail. A commit whose history the registered packs cannot walk answers `404`, like a commit that +does not exist. The check is the walk that `receive-pack` runs for the new tips of a push (`git +rev-list --objects --not --all`) over the registered packs, without a commit-graph, because a +commit-graph can describe a commit that no registered pack holds. A commit that a ref reaches ends +the walk at once, so a base near the tip of a branch costs milliseconds. The oldest commit of +500,000 linear commits costs about two seconds, and a commit that no ref reaches costs the walk of +the commits since the ones that a ref reaches, with their trees. A walk of more than 30 seconds +answers `503`. + #### Guarded branch create and move `expected_target_sha` on `POST /repos/branches/create` (both path styles) guards the diff --git a/internal/api/handlers_write_refs.go b/internal/api/handlers_write_refs.go index c439d872..a67e507d 100644 --- a/internal/api/handlers_write_refs.go +++ b/internal/api/handlers_write_refs.go @@ -386,24 +386,33 @@ func wrefValidateCreateBranch(req *wrefCreateBranchRequest) (target string, allo return target, allocate, guard, errs.err() } -// wrefHeldCommit reports whether a pack that the registry lists for the -// repository holds the commit sha. A write that stores the commit in a ref -// must ask: the node's cache can hold objects that no registered pack holds (a -// cache that was copied or edited out of band, or a registry that was rolled -// back), and a ref at such an object wedges every other node, and this node -// after its cache is gone, because they hydrate from the registered packs -// alone. -func (s *server) wrefHeldCommit(ctx context.Context, repoID, repoPath, sha string) (bool, error) { - kinds, err := gitread.RegisteredKinds(ctx, s.refStore, s.blobStore, repoPath, repoID, []string{sha}) - if err != nil { - return false, err +// wrefUsableCommit reports whether the registered packs hold the commit sha and +// what git needs to walk its history: its parents down to the commits that a +// ref reaches, and their trees and blobs (gitread.RegisteredClosure). A write +// that stores the commit in a ref must ask, for two reasons. +// +// The node's cache can hold objects that no registered pack holds (a cache that +// was copied or edited out of band, or a registry that was rolled back), and a +// ref at such an object wedges every other node, and this node after its cache +// is gone, because they hydrate from the registered packs alone. +// +// A pack can also hold a commit whose parent no pack holds. receive-pack checks +// the commits that a push's refs reach and no other commit of its pack, so a +// push can leave such a commit in the registry. A ref at it hydrates, and +// every clone, fetch and archive that walks the history then fails. +func (s *server) wrefUsableCommit(ctx context.Context, repoID, repoPath, sha string) (bool, error) { + usable, err := gitread.RegisteredClosure(ctx, s.refStore, s.blobStore, repoPath, repoID, sha) + if err == nil && !usable { + s.logger.Warn("a ref write names a commit that the registered packs cannot walk: a pack lacks the commit or one of its ancestors, trees or blobs", + "repo", repoID, "sha", sha) } - return kinds[sha] == "commit", nil + return usable, err } // handleCreateBranch resolves the base to a commit (branch, tag — peeled — // or hex sha via gitread.ResolveRef; preferred_base is the fallback when -// the primary base is absent or names no ref, and a base whose commit git +// the primary base is absent, names no ref or names a commit whose history +// the registered packs cannot walk, and a base whose commit git // cannot read is a 500, with no fallback), policy-checks the new ref, // and CAS creates it. With force, an existing target is CAS-recreated at // the base — no-force-push policies still deny non-fast-forward moves via a @@ -450,13 +459,14 @@ func (s *server) handleCreateBranch(w http.ResponseWriter, r *http.Request) { } ctx := r.Context() // create runs the creation at one resolved base. It returns false, with no - // response written, when no registered pack holds the base commit: the - // node's cache can hold objects that no registered pack holds (a cache that - // was copied or edited out of band, or a registry that was rolled back), and - // a ref at such an object wedges every other node, and this node after its - // cache is gone, because they hydrate from the registered packs alone. The - // registry is read only for a request that is about to write: a denied - // request, and one that leaves the branch where it is, never read it. + // response written, when the registered packs lack the base commit or part + // of its history (wrefUsableCommit): the node's cache can hold objects that + // no registered pack holds (a cache that was copied or edited out of band, + // or a registry that was rolled back), and a ref at such an object wedges + // every other node, and this node after its cache is gone, because they + // hydrate from the registered packs alone. The registry is read only for a + // request that is about to write: a denied request, and one that leaves the + // branch where it is, never read it. create := func(baseSHA string) bool { held := false attempts := 1 @@ -515,9 +525,9 @@ func (s *server) handleCreateBranch(w http.ResponseWriter, r *http.Request) { return true } if !held { - ok, err := s.wrefHeldCommit(ctx, repo.RepoID, repoPath, baseSHA) + ok, err := s.wrefUsableCommit(ctx, repo.RepoID, repoPath, baseSHA) if err != nil { - s.internalErr(w, r, "check that a registered pack holds the base", err) + s.internalErr(w, r, "check that the registered packs hold the base and its history", err) return true } if !ok { @@ -808,10 +818,10 @@ func (s *server) handleCreateTag(w http.ResponseWriter, r *http.Request) { writeErr(w, http.StatusConflict, "tag already exists") return } - held, err := s.wrefHeldCommit(r.Context(), repo.RepoID, repoPath, sha) + held, err := s.wrefUsableCommit(r.Context(), repo.RepoID, repoPath, sha) switch { case err != nil: - s.internalErr(w, r, "check that a registered pack holds the tag target", err) + s.internalErr(w, r, "check that the registered packs hold the tag target and its history", err) return case !held: writeProblem(w, http.StatusNotFound, codeRefNotFound, "ref does not resolve to a commit") diff --git a/internal/api/registry_history_test.go b/internal/api/registry_history_test.go new file mode 100644 index 00000000..4ee2bf51 --- /dev/null +++ b/internal/api/registry_history_test.go @@ -0,0 +1,177 @@ +package api + +// A ref write must not store a commit whose history the registered packs +// cannot walk. receive-pack checks the commits that a push's refs reach and no +// other commit of the pack, so a push can leave a commit in a registered pack +// whose parent, tree or blob no pack holds. A ref at it hydrates, and every +// clone, fetch and archive that walks the history fails (#896 held the commit +// itself and left this "closure variant" open). + +import ( + "context" + "net/http" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "testing" + + "github.com/coder/openstorage/internal/blob" + "github.com/coder/openstorage/internal/gitread" + "github.com/coder/openstorage/internal/ref" +) + +const absentID = "1111111111111111111111111111111111111111" + +// scratchGit runs git in dir with stdin and no GIT_ variable of the host. +func scratchGit(t *testing.T, dir, stdin string, args ...string) string { + t.Helper() + cmd := exec.Command("git", args...) + cmd.Dir = dir + for _, kv := range os.Environ() { + if !strings.HasPrefix(kv, "GIT_") { + cmd.Env = append(cmd.Env, kv) + } + } + cmd.Env = append(cmd.Env, "GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_SYSTEM=/dev/null") + cmd.Stdin = strings.NewReader(stdin) + out, err := cmd.Output() + if err != nil { + t.Fatalf("git %v: %v", args, err) + } + return strings.TrimSpace(string(out)) +} + +// commitText is the raw text of a commit with the given tree and parents. +func commitText(tree string, parents ...string) string { + var b strings.Builder + b.WriteString("tree " + tree + "\n") + for _, p := range parents { + b.WriteString("parent " + p + "\n") + } + b.WriteString("author a 1 +0000\ncommitter a 1 +0000\n\nplanted\n") + return b.String() +} + +// registerLooseObjects packs exactly the listed objects of dir (no walk), uploads +// the pack and registers it for the repository, as a push that carries objects +// that its refs do not reach would. +func registerLooseObjects(t *testing.T, e *testEnv, repoID, dir string, ids ...string) { + t.Helper() + ctx := context.Background() + packBase := filepath.Join(t.TempDir(), "p") + id := scratchGit(t, dir, strings.Join(ids, "\n")+"\n", "pack-objects", "--threads=1", "-q", packBase) + key := blob.PackKey(repoID, id) + if err := e.blobStore.UploadFile(ctx, key, packBase+"-"+id+".pack"); err != nil { + t.Fatalf("upload pack %s: %v", key, err) + } + if err := e.refStore.RegisterPacks(ctx, repoID, []ref.Pack{{ID: id, Key: key}}); err != nil { + t.Fatalf("register pack %s: %v", id, err) + } +} + +// plantHistory registers a pack with a commit, and returns the commit, for each +// way that the pack can lack part of what the commit names. c2's tree and +// commits are in the repository's first pack. +func plantHistory(t *testing.T, e *testEnv, fx wrefFixture, shape string) string { + t.Helper() + dir := t.TempDir() + scratchGit(t, dir, "", "init", "-q", "-b", "main", ".") + hash := func(kind, text string) string { + return scratchGit(t, dir, text, "hash-object", "-t", kind, "-w", "--stdin", "--literally") + } + tree := func(entries string) string { + return scratchGit(t, dir, entries, "mktree", "--missing") + } + switch shape { + case "parent": + // The tree is the empty tree: it is the commit's own parent that is absent. + empty := tree("") + c := hash("commit", commitText(empty, absentID)) + registerLooseObjects(t, e, fx.repoID, dir, c, empty) + return c + case "grandparent": + empty := tree("") + p := hash("commit", commitText(empty, absentID)) + c := hash("commit", commitText(empty, p)) + registerLooseObjects(t, e, fx.repoID, dir, c, p, empty) + return c + case "tree": + c := hash("commit", commitText(absentID, fx.c2)) + registerLooseObjects(t, e, fx.repoID, dir, c) + return c + case "blob": + tr := tree("100644 blob " + absentID + "\tf.txt\n") + c := hash("commit", commitText(tr, fx.c2)) + registerLooseObjects(t, e, fx.repoID, dir, c, tr) + return c + case "complete": + // A commit that no ref reaches, with everything that it names in the pack. + blobID := hash("blob", "complete\n") + tr := tree("100644 blob " + blobID + "\tf.txt\n") + c := hash("commit", commitText(tr, fx.c2)) + registerLooseObjects(t, e, fx.repoID, dir, c, tr, blobID) + return c + } + t.Fatalf("unknown shape %q", shape) + return "" +} + +func TestRefWritesRefuseACommitWhoseHistoryThePacksLack(t *testing.T) { + runBackends(t, func(t *testing.T, e *testEnv) { + for _, history := range []string{"parent", "grandparent", "tree", "blob"} { + for _, tc := range refWriteShapes() { + t.Run(history+"/"+tc.name, func(t *testing.T) { + fx := wrefSeedFixture(t, e) + planted := plantHistory(t, e, fx, history) + srv := newWrefServer(t, e) + before := wrefRefs(t, e, fx.repoID) + + resp, body := wrefDo(t, srv, http.MethodPost, tc.path, fx.token, tc.body(planted)) + if resp.StatusCode != http.StatusNotFound { + t.Fatalf("a commit whose %s the packs lack: status %d, want 404, body %v", history, resp.StatusCode, body) + } + after := wrefRefs(t, e, fx.repoID) + if len(after) != len(before) || after[tc.ref] != before[tc.ref] { + t.Fatalf("refs changed after the refused request: before %v, after %v", before, after) + } + + // A commit that the packs hold completely still works. + resp, body = wrefDo(t, srv, http.MethodPost, tc.path, fx.token, tc.body(fx.c1)) + if resp.StatusCode != http.StatusOK { + t.Fatalf("a commit that the packs hold: status %d, want 200, body %v", resp.StatusCode, body) + } + }) + } + } + }) +} + +// A commit that no ref reaches is no reason for a refusal when the packs hold +// everything that it names, and a commit that a ref reaches costs no walk. +func TestRefWritesAcceptACommitWhoseHistoryThePacksHold(t *testing.T) { + runBackends(t, func(t *testing.T, e *testEnv) { + fx := wrefSeedFixture(t, e) + complete := plantHistory(t, e, fx, "complete") + srv := newWrefServer(t, e) + for i, tc := range []struct{ name, sha string }{{"unreferenced and complete", complete}, {"the root", fx.c1}, {"the tip", fx.c2}} { + t.Run(tc.name, func(t *testing.T) { + resp, body := wrefDo(t, srv, http.MethodPost, "/api/v1/repos/branches/create", fx.token, + map[string]any{"target_branch": "ok" + strconv.Itoa(i), "base_ref": tc.sha}) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status %d, want 200, body %v", resp.StatusCode, body) + } + if body["commit_sha"] != tc.sha { + t.Fatalf("body %v, want commit_sha %s", body, tc.sha) + } + }) + } + // Every other node hydrates the repository and reads the history. + repoPath, err := gitread.Materialize(context.Background(), e.refStore, e.blobStore, t.TempDir(), fx.repoID, "main") + if err != nil { + t.Fatal(err) + } + scratchGit(t, repoPath, "", "rev-list", "--objects", "--all") + }) +} diff --git a/internal/api/registry_target_test.go b/internal/api/registry_target_test.go index 50a58708..cb88718a 100644 --- a/internal/api/registry_target_test.go +++ b/internal/api/registry_target_test.go @@ -47,13 +47,18 @@ func plantCacheOnlyCommit(t *testing.T, repoPath, parent string) string { return run("", "commit-tree", "-p", parent, "-m", "cache only", tree) } -func TestRefWritesRefuseACommitThatOnlyTheCacheHolds(t *testing.T) { - type shape struct { - name, path string - body func(sha string) map[string]any - ref string // the ref that the request would create - } - shapes := []shape{ +// refWriteShape is one request that stores a commit of the body in a ref. +type refWriteShape struct { + name, path string + body func(sha string) map[string]any + ref string // the ref that the request would create +} + +// refWriteShapes lists the requests that store a commit that the request names +// in a ref: the base of a branch (named, as a fallback, forced, ephemeral) and +// the target of a tag. +func refWriteShapes() []refWriteShape { + return []refWriteShape{ {"branch base_ref", "/api/v1/repos/branches/create", func(sha string) map[string]any { return map[string]any{"target_branch": "co", "base_ref": sha} }, "refs/heads/co"}, @@ -70,6 +75,10 @@ func TestRefWritesRefuseACommitThatOnlyTheCacheHolds(t *testing.T) { return map[string]any{"name": "co", "ref": sha} }, "refs/tags/co"}, } +} + +func TestRefWritesRefuseACommitThatOnlyTheCacheHolds(t *testing.T) { + shapes := refWriteShapes() runBackends(t, func(t *testing.T, e *testEnv) { for _, tc := range shapes { t.Run(tc.name, func(t *testing.T) { diff --git a/internal/gitread/gitread.go b/internal/gitread/gitread.go index eb3b037a..50a4ba98 100644 --- a/internal/gitread/gitread.go +++ b/internal/gitread/gitread.go @@ -22,6 +22,7 @@ import ( "os/exec" "regexp" "strings" + "time" "github.com/coder/openstorage/internal/auth" "github.com/coder/openstorage/internal/blob" @@ -299,3 +300,81 @@ func RegisteredKinds(ctx context.Context, st ref.RefStore, bl blob.BlobStore, re return githttp.ParseObjectKinds(ids, out) }) } + +// registeredWalkTimeout bounds the history walk of RegisteredClosure. The walk +// reads the commits between sha and the commits that refs reach: a base that a +// ref reaches ends it at once, and the oldest commit of 500,000 linear commits +// costs about two seconds without a commit-graph. A write endpoint has no +// request budget, so the walk carries its own. +var registeredWalkTimeout = 30 * time.Second + +// missingObjectScan reads the output of `git rev-list --missing=print --quiet`, +// which holds one "?" line for each object that the walk could not read and +// nothing else. The first byte decides: git is stopped there, because one +// missing object answers the question. +type missingObjectScan struct { + missing bool + err error +} + +func (m *missingObjectScan) Write(p []byte) (int, error) { + if len(p) == 0 { + return 0, nil + } + if p[0] != '?' { + m.err = errors.New("rev-list --missing=print --quiet printed something other than a missing object") + return 0, ErrStop + } + m.missing = true + return 0, ErrStop +} + +// RegisteredClosure reports whether the registered packs hold what git needs to +// read sha and everything that sha names that no ref of the node's cache +// reaches: the commits down to the ones that a ref reaches, and their trees and +// blobs. A ref may be created at sha only then. A pack can hold a commit whose +// parent no registered pack holds (a push can carry commits that its refs do +// not reach, and git checks the commits that the refs reach alone), and a ref +// at such a commit lets git read the ref, yet every clone that walks the +// history fails. +// +// The walk is git's own connectivity check, the one that receive-pack runs for +// the new tips of a push: `rev-list --objects --not --all` over a view of the +// registered packs (githttp.RegisteredPackView). The answer rests on which +// objects git can read, never on git's error text: --missing=print makes git +// print the id of each object that it cannot read and exit 0, and any other +// outcome of git is an error. The view has no commit-graph on purpose, because a +// commit-graph can describe commits that no registered pack holds and git then +// walks past them. So the cost is the same with and without a commit-graph in +// the cache. +// +// A registered pack that the cache lacks makes a "missing" answer an error that +// wraps githttp.ErrViewIncomplete, as it does for RegisteredKinds. A walk that +// outlasts registeredWalkTimeout is an error that wraps ErrTimeBudget. +func RegisteredClosure(ctx context.Context, st ref.RefStore, bl blob.BlobStore, repoPath, repo, sha string) (bool, error) { + assertf(ref.ValidSHA(sha), "RegisteredClosure: malformed sha %q", sha) + kinds, err := githttp.AskRegisteredPacks(ctx, st, repoPath, repo, func(ctx context.Context) error { + return githttp.HydratePacks(ctx, st, bl, repoPath, repo) + }, func(view *githttp.RegisteredPackView) (map[string]string, error) { + walkCtx, cancel := context.WithTimeout(ctx, registeredWalkTimeout) + defer cancel() + scan := &missingObjectScan{} + _, err := Run(walkCtx, Cmd{Class: Heavy, Dir: repoPath, Env: view.Env(os.Environ()), Stdout: scan, + Args: []string{"-c", "core.commitGraph=false", "rev-list", "--objects", "--missing=print", "--quiet", sha, "--not", "--all"}}) + switch { + case ctx.Err() == nil && errors.Is(walkCtx.Err(), context.DeadlineExceeded): + return nil, fmt.Errorf("walk the history of %s: %w", sha, ErrTimeBudget) + case err != nil: + return nil, err + case scan.err != nil: + return nil, scan.err + case scan.missing: + return map[string]string{sha: "missing"}, nil + } + return map[string]string{sha: "commit"}, nil + }) + if err != nil { + return false, err + } + return kinds[sha] != "missing", nil +} diff --git a/internal/gitread/registered_closure_test.go b/internal/gitread/registered_closure_test.go new file mode 100644 index 00000000..a7c21cab --- /dev/null +++ b/internal/gitread/registered_closure_test.go @@ -0,0 +1,157 @@ +package gitread + +import ( + "context" + "errors" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/coder/openstorage/internal/ref" +) + +// RegisteredClosure decides from the objects that git can read, which +// `rev-list --missing=print` prints as "?" lines, and from nothing else: +// any other outcome of git is an error, never a refusal. A stand-in git shows +// each outcome. +func TestRegisteredClosureAnswersFromTheMissingObjectsAlone(t *testing.T) { + sha := strings.Repeat("a", 40) + st, err := ref.Open(t.Context(), "sqlite://"+filepath.Join(t.TempDir(), "refs.sqlite")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { st.Close() }) + if err := st.InitSchema(t.Context()); err != nil { + t.Fatal(err) + } + repoPath := filepath.Join(t.TempDir(), "r.git") + if err := os.MkdirAll(filepath.Join(repoPath, "objects", "pack"), 0o755); err != nil { + t.Fatal(err) + } + for _, tc := range []struct { + name, script string + want bool + wantErr string // text of the error, "" for none + }{ + {"nothing missing", "exit 0", true, ""}, + {"an object is missing", "echo '?" + sha + "'; exit 0", false, ""}, + {"a missing object and a failed walk", "echo '?" + sha + "'; exit 128", false, ""}, + {"output that is no missing object", "echo 'deadbeef'; exit 0", false, "something other than a missing object"}, + {"git fails", "echo 'fatal: out of memory' >&2; exit 128", false, "out of memory"}, + {"git is killed", "kill -9 $$", false, "killed"}, + } { + t.Run(tc.name, func(t *testing.T) { + bin := t.TempDir() + // rev-list is the walk. The stand-in answers the other git commands with exit 0. + script := "#!/bin/sh\ncase \"$*\" in *rev-list*) " + tc.script + " ;; esac\nexit 0\n" + if err := os.WriteFile(filepath.Join(bin, "git"), []byte(script), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) + got, err := RegisteredClosure(t.Context(), st, nil, repoPath, "repo_x", sha) + if tc.wantErr == "" && err != nil || tc.wantErr != "" && (err == nil || !strings.Contains(err.Error(), tc.wantErr)) { + t.Fatalf("RegisteredClosure error = %v, want %q", err, tc.wantErr) + } + if err == nil && got != tc.want { + t.Fatalf("RegisteredClosure = %v, want %v", got, tc.want) + } + if tc.name == "a missing object and a failed walk" && err != nil { + t.Fatalf("a missing object was printed before the walk failed: %v", err) + } + }) + } +} + +// A walk that outlasts its budget is ErrTimeBudget, which the handlers answer +// with 503, and never a refusal. A walk that the caller gave up is the caller's +// own error. +func TestRegisteredClosureWalkBudget(t *testing.T) { + sha := strings.Repeat("a", 40) + st, err := ref.Open(t.Context(), "sqlite://"+filepath.Join(t.TempDir(), "refs.sqlite")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { st.Close() }) + if err := st.InitSchema(t.Context()); err != nil { + t.Fatal(err) + } + repoPath := filepath.Join(t.TempDir(), "r.git") + if err := os.MkdirAll(filepath.Join(repoPath, "objects", "pack"), 0o755); err != nil { + t.Fatal(err) + } + bin := t.TempDir() + if err := os.WriteFile(filepath.Join(bin, "git"), []byte("#!/bin/sh\ncase \"$*\" in *rev-list*) sleep 30 ;; esac\nexit 0\n"), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) + prev := registeredWalkTimeout + registeredWalkTimeout = 200 * time.Millisecond + t.Cleanup(func() { registeredWalkTimeout = prev }) + + start := time.Now() + if _, err := RegisteredClosure(t.Context(), st, nil, repoPath, "repo_x", sha); !errors.Is(err, ErrTimeBudget) { + t.Fatalf("a walk past its budget: %v, want ErrTimeBudget", err) + } + if d := time.Since(start); d > 10*time.Second { + t.Fatalf("the walk was not stopped at its budget: %v", d) + } + ctx, cancel := context.WithCancel(t.Context()) + cancel() + if _, err := RegisteredClosure(ctx, st, nil, repoPath, "repo_x", sha); errors.Is(err, ErrTimeBudget) || err == nil { + t.Fatalf("a caller that gave up: %v, want its own error, not ErrTimeBudget", err) + } +} + +// The walk is git's connectivity check, over the registered packs alone and +// with no commit-graph: a commit-graph can describe a commit that no registered +// pack holds, and git then walks past it. +func TestRegisteredClosureRunsGitsConnectivityCheckOverTheView(t *testing.T) { + sha := strings.Repeat("b", 40) + st, err := ref.Open(t.Context(), "sqlite://"+filepath.Join(t.TempDir(), "refs.sqlite")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { st.Close() }) + if err := st.InitSchema(t.Context()); err != nil { + t.Fatal(err) + } + repoPath := filepath.Join(t.TempDir(), "r.git") + if err := os.MkdirAll(filepath.Join(repoPath, "objects", "pack"), 0o755); err != nil { + t.Fatal(err) + } + bin := t.TempDir() + log := filepath.Join(bin, "log") + script := "#!/bin/sh\ncase \"$*\" in *rev-list*) echo \"$*\" >> " + log + "\n" + + "echo \"GIT_DIR=$GIT_DIR\" >> " + log + "\necho \"OBJECTS=$GIT_OBJECT_DIRECTORY\" >> " + log + "\n" + + "echo \"ALT=$GIT_ALTERNATE_OBJECT_DIRECTORIES\" >> " + log + " ;; esac\nexit 0\n" + if err := os.WriteFile(filepath.Join(bin, "git"), []byte(script), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) + t.Setenv("GIT_ALTERNATE_OBJECT_DIRECTORIES", "/elsewhere") + if ok, err := RegisteredClosure(t.Context(), st, nil, repoPath, "repo_x", sha); err != nil || !ok { + t.Fatalf("RegisteredClosure = %v, %v, want true", ok, err) + } + b, err := os.ReadFile(log) + if err != nil { + t.Fatal(err) + } + lines := strings.Split(strings.TrimSpace(string(b)), "\n") + if len(lines) != 4 { + t.Fatalf("the stand-in logged %q, want the walk once", lines) + } + if want := "-c core.commitGraph=false rev-list --objects --missing=print --quiet " + sha + " --not --all"; lines[0] != want { + t.Fatalf("git ran %q, want %q", lines[0], want) + } + if lines[1] != "GIT_DIR="+repoPath { + t.Fatalf("the walk reads the refs of %q, want %q", lines[1], repoPath) + } + if objects := strings.TrimPrefix(lines[2], "OBJECTS="); objects == "" || objects == filepath.Join(repoPath, "objects") { + t.Fatalf("the walk reads objects from %q, want a view of the registered packs, not the cache's own", objects) + } + if lines[3] != "ALT=" { + t.Fatalf("the walk keeps the alternates %q, want none", lines[3]) + } +} From 095115fd3b1ca18658e3148c1de2df5c9fba8434 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Fri, 20 Nov 2026 16:23:18 +0000 Subject: [PATCH 2/7] fix(api): run the registered history walk under the git admission limits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review of the first head found that the walk had no admission slot, no heap cap and no request budget of its own, that git 2.43 dies on a missing start object, that a plain create of an existing branch paid for a walk before it lost its swap, and that a merge with an expected_source_sha could store a commit that no registered pack holds. - The walk runs under the slots, the 30-second budget and the 256 MiB heap cap of the git:read routes (503 with Retry-After, 503, 422). - A cat-file asks for the start object first, and exit status 128 of the walk reads as a history that cannot be walked, so git 2.39 and 2.43 answer as git 2.55 does. - A plain create of a branch that exists answers 409 before the walk. - POST /repos/merge checks an expected_source_sha that is not the tip of the source branch like a base. - Tests pin the log line, the default budget and the merge pin. Docs state the measured cost, the dependence on commit dates and the git version note. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_ --- docs/how-to/deploy-production.md | 2 +- docs/reference/http-api.md | 31 ++++++--- internal/api/handlers_write_merge.go | 19 ++++++ internal/api/handlers_write_refs.go | 20 +++++- internal/api/registry_history_test.go | 71 ++++++++++++++++++++- internal/api/registry_target_test.go | 6 +- internal/api/write_merge_source_pin_test.go | 57 +++++++++++++++++ internal/api/write_refs_test.go | 9 ++- internal/gitread/gitread.go | 57 ++++++++++++----- internal/gitread/registered_closure_test.go | 61 ++++++++++++++++-- 10 files changed, 295 insertions(+), 38 deletions(-) diff --git a/docs/how-to/deploy-production.md b/docs/how-to/deploy-production.md index 374c8b37..826bd43c 100644 --- a/docs/how-to/deploy-production.md +++ b/docs/how-to/deploy-production.md @@ -735,7 +735,7 @@ webhook events. | Postgres availability + advisory-lock waits at boot | The database is the single authority; boot hangs show in `pg_locks` | | `verify token: key store lookup failed` / `verify token: a registered key cannot be used` log lines | A git, LFS or REST request reached the key store or a registered key, and it failed. The line holds the cause (a cause of more than 300 runes loses its middle). A git or LFS client reads a text that does not hold the cause. For each kind, the git and LFS routes together write at most one line per minute in each `serve` process, the REST routes write at most one more, and a line says how many earlier lines of its kind were held back when some were | | `receive-pack : hook: … git cannot walk the history between two commits, so the update is refused` log line | A push named two commits whose history git cannot read, so the hook refused that update: as non-fast-forward under `no-force-push`, and with `is not a readable commit in this repository` under `verify-sig`. The line has the fields `git`, `old` and `new`. A pusher can cause it with a commit whose parent is missing. A `` that repeats it for ordinary pushes from several clients has a damaged history: check a fresh copy with `git fsck` | -| `a ref write names a commit that the registered packs cannot walk` log line | A `POST /repos/branches/create` or `POST /repos/tags` named a commit that no registered pack holds, or whose parent, tree or blob no registered pack holds, and answered `404`. The line has the fields `repo` and `sha`. A pusher can leave such a commit in a pack (git checks only the commits that the refs of a push reach). One line is a refused request. A `repo` that repeats it with the same `sha` for ordinary branch creates has a repository whose registry lost a pack: compare `openstorage list-packs` with the blob store | +| `a ref write names a commit that the registered packs cannot walk` log line | A `POST /repos/branches/create`, `POST /repos/tags` or `POST /repos/merge` (with `expected_source_sha`) named a commit that no registered pack holds, or whose ancestor, tree or blob no registered pack holds. The line has the fields `repo` and `sha`. A branch create logs one line for each base that it refuses (`base_ref` and `preferred_base` can both be refused). It answers `200` at `preferred_base` when that base is usable and `404` otherwise, a tag answers `404` and a merge answers `409` (`ref_guard_failed`). A pusher can leave such a commit in a pack (git checks only the commits that the refs of a push reach). A `repo` that logs the line for ordinary creates, with different `sha` values that name healthy bases, has a ref at an object that no registered pack holds: compare `openstorage list-packs` with the blob store, and list the ref with `openstorage repair-refs` | | `lane scan: requeued a lane whose work had no job` log line | A sync or push request lost its job and the [lane scan](#background-maintenance-in-a-fleet) enqueued it again. A `repo=` that repeats needs a look: its runs keep crashing | The opt-in [metrics listener](../reference/metrics.md) covers some of these signals; diff --git a/docs/reference/http-api.md b/docs/reference/http-api.md index a572a21e..f1be7311 100644 --- a/docs/reference/http-api.md +++ b/docs/reference/http-api.md @@ -685,7 +685,11 @@ moved on since. - The source branch must still contain the commit: the commit is the tip or one of its ancestors. Otherwise the guard fails (`detail` "source branch moved: the source - branch does not contain commit …"). + branch does not contain commit …"). The same answer covers a commit that the source + branch reaches but the registered packs cannot walk, the way the [branch and tag + create routes](#which-commit-the-base-and-the-tag-target-can-name) refuse such a commit + (a push can leave one behind the tip of a branch): a pin that is not the tip is checked + like a base, and the server logs the refusal. - A source branch that does not exist fails the guard too, with `current_sha` 40 zeros. Without `expected_source_sha`, a missing source branch is `404`. - `source.sha` in the response is the pinned commit. @@ -749,8 +753,8 @@ The server checks the registered packs, not the node's cache repository, because ref at any other object. A commit that only a cache holds answers `404`, like a commit that does not exist, and a `base_ref` that names no usable commit still falls back to `preferred_base`. The server reads the registry only for a request that is about to write: after the ref policy, and not -for a tag that exists or for a create that leaves the branch at its tip (with `force` or -`expected_target_sha`). So a request that the policy denies +for a tag that exists, for a plain create of a branch that exists (`409`) or for a create that +leaves the branch at its tip (with `force` or `expected_target_sha`). So a request that the policy denies answers `403` even when its commit is only in the cache. The registered packs must also hold what git needs to read the commit's history: its ancestors, @@ -761,11 +765,22 @@ at such a commit would hydrate, and every clone, fetch and archive that walks th fail. A commit whose history the registered packs cannot walk answers `404`, like a commit that does not exist. The check is the walk that `receive-pack` runs for the new tips of a push (`git rev-list --objects --not --all`) over the registered packs, without a commit-graph, because a -commit-graph can describe a commit that no registered pack holds. A commit that a ref reaches ends -the walk at once, so a base near the tip of a branch costs milliseconds. The oldest commit of -500,000 linear commits costs about two seconds, and a commit that no ref reaches costs the walk of -the commits since the ones that a ref reaches, with their trees. A walk of more than 30 seconds -answers `503`. +commit-graph can describe a commit that no registered pack holds. + +The cost is the walk from the commits that the refs reach down to the commit, so it grows with the +distance between the two and with the commit dates that the pusher chose. Measured with git 2.55 +on 500,000 linear commits: the tip of a branch takes 5 ms, a commit 1,000 back 15 ms, 100,000 back +0.3 s and the root commit 1.6 s. A commit that no ref reaches costs the walk of its own new +commits and trees, and also the walk of the history of the refs down to its commit date (1,000 new +commits dated after the tip took 14 ms, and the same commits dated 1970 took 1.7 s). A commit +whose parent no pack holds always costs the whole history (1.6 s). A separate unreferenced history +costs more: 100,000 commits took 2.8 s. The walk runs under the admission slots, the 30-second +time budget and the 256 MiB heap cap of the git:read routes ([Resource limits on git +reads](#resource-limits-on-git-reads)): a request that finds no free slot answers `503` with +`Retry-After`, a walk that outlasts its budget answers `503`, and a walk that needs more memory +(an unreferenced history of 500,000 commits) answers `422`. git 2.39 exits with status 128 on a +missing parent commit, and git 2.43 on a ref at a missing object, instead of printing the object: +the check reads that exit status as a history that cannot be walked. #### Guarded branch create and move diff --git a/internal/api/handlers_write_merge.go b/internal/api/handlers_write_merge.go index 96fc8605..05caf482 100644 --- a/internal/api/handlers_write_merge.go +++ b/internal/api/handlers_write_merge.go @@ -684,6 +684,25 @@ func (s *server) handleMerge(w http.ResponseWriter, r *http.Request) { writeRefGuardFailed(w, "expected_source_sha", sourceSHA, sourceMovedMsg) return } + // The source branch reaches the pin, but the pin can still be a commit + // that only this node's cache holds (a push can leave a commit behind + // a ref tip that no registered pack holds). A ref or a merge commit at + // it would wedge every other node, so it answers like a pin that the + // source branch does not contain. The tip needs no check: a ref is at it. + if req.sourcePin != sourceSHA { + usable, err := s.wrefUsableCommit(ctx, repo.RepoID, repoPath, req.sourcePin) + if err != nil { + const op = "check that the registered packs hold expected_source_sha and its history" + if !s.writeGitLimitErr(w, r, op, err) { + s.wmInternal(w, r, op, err) + } + return + } + if !usable { + writeRefGuardFailed(w, "expected_source_sha", sourceSHA, sourceMovedMsg) + return + } + } sourceSHA = req.sourcePin } diff --git a/internal/api/handlers_write_refs.go b/internal/api/handlers_write_refs.go index a67e507d..00cea144 100644 --- a/internal/api/handlers_write_refs.go +++ b/internal/api/handlers_write_refs.go @@ -400,8 +400,14 @@ func wrefValidateCreateBranch(req *wrefCreateBranchRequest) (target string, allo // the commits that a push's refs reach and no other commit of its pack, so a // push can leave such a commit in the registry. A ref at it hydrates, and // every clone, fetch and archive that walks the history then fails. +// +// The walk runs under the same admission slots, time budget and heap cap as a +// git:read route (gitReads): a write endpoint has no request budget of its own, +// and a writer must not be able to start any number of walks at once. func (s *server) wrefUsableCommit(ctx context.Context, repoID, repoPath, sha string) (bool, error) { - usable, err := gitread.RegisteredClosure(ctx, s.refStore, s.blobStore, repoPath, repoID, sha) + b := gitread.NewBudget(gitReads.slots, gitlimit.RepoKey(repoID), gitReads.timeout, gitReads.outputBytes).WithLimits(gitReads.limits) + defer b.Close() + usable, err := gitread.RegisteredClosure(gitread.WithBudget(ctx, b), s.refStore, s.blobStore, repoPath, repoID, sha) if err == nil && !usable { s.logger.Warn("a ref write names a commit that the registered packs cannot walk: a pack lacks the commit or one of its ancestors, trees or blobs", "repo", repoID, "sha", sha) @@ -525,6 +531,18 @@ func (s *server) handleCreateBranch(w http.ResponseWriter, r *http.Request) { return true } if !held { + // A create that cannot win is answered before the walk: a plain + // create of a branch that exists loses its swap, whatever the base. + if oldSHA == ref.ZeroSHA && !allocate { + now, ok := s.wrefTip(w, r, repo.RepoID, targetRef) + if !ok { + return true + } + if now != ref.ZeroSHA { + writeErr(w, http.StatusConflict, "branch already exists") + return true + } + } ok, err := s.wrefUsableCommit(ctx, repo.RepoID, repoPath, baseSHA) if err != nil { s.internalErr(w, r, "check that the registered packs hold the base and its history", err) diff --git a/internal/api/registry_history_test.go b/internal/api/registry_history_test.go index 4ee2bf51..5a5d1015 100644 --- a/internal/api/registry_history_test.go +++ b/internal/api/registry_history_test.go @@ -9,6 +9,7 @@ package api import ( "context" + "log/slog" "net/http" "os" "os/exec" @@ -16,14 +17,36 @@ import ( "strconv" "strings" "testing" + "time" "github.com/coder/openstorage/internal/blob" + "github.com/coder/openstorage/internal/gitlimit" "github.com/coder/openstorage/internal/gitread" "github.com/coder/openstorage/internal/ref" ) const absentID = "1111111111111111111111111111111111111111" +// refusedWalkLine is the log line of a ref write that names a commit whose +// history the registered packs cannot walk. The operator's only sign of such a +// request: the response is the same 404 as for a commit that does not exist. +const refusedWalkLine = "a ref write names a commit that the registered packs cannot walk" + +// logLines returns the lines of the log that hold every one of the substrings. +func logLines(l *lockedLog, substrings ...string) []string { + var out []string +next: + for _, line := range strings.Split(l.String(), "\n") { + for _, sub := range substrings { + if !strings.Contains(line, sub) { + continue next + } + } + out = append(out, line) + } + return out +} + // scratchGit runs git in dir with stdin and no GIT_ variable of the host. func scratchGit(t *testing.T, dir, stdin string, args ...string) string { t.Helper() @@ -125,23 +148,32 @@ func TestRefWritesRefuseACommitWhoseHistoryThePacksLack(t *testing.T) { t.Run(history+"/"+tc.name, func(t *testing.T) { fx := wrefSeedFixture(t, e) planted := plantHistory(t, e, fx, history) - srv := newWrefServer(t, e) + logs := &lockedLog{} + srv := newWrefServerLogging(t, e, slog.New(slog.NewTextHandler(logs, nil))) before := wrefRefs(t, e, fx.repoID) resp, body := wrefDo(t, srv, http.MethodPost, tc.path, fx.token, tc.body(planted)) if resp.StatusCode != http.StatusNotFound { t.Fatalf("a commit whose %s the packs lack: status %d, want 404, body %v", history, resp.StatusCode, body) } + // The operator hears of the refusal: one line per refused commit, with the repository and the commit. + if got := logLines(logs, refusedWalkLine, "level=WARN", "repo="+fx.repoID, "sha="+planted); len(got) == 0 { + t.Fatalf("no %q line for %s in the log:\n%s", refusedWalkLine, planted, logs.String()) + } after := wrefRefs(t, e, fx.repoID) if len(after) != len(before) || after[tc.ref] != before[tc.ref] { t.Fatalf("refs changed after the refused request: before %v, after %v", before, after) } // A commit that the packs hold completely still works. + refused := len(logLines(logs, refusedWalkLine)) resp, body = wrefDo(t, srv, http.MethodPost, tc.path, fx.token, tc.body(fx.c1)) if resp.StatusCode != http.StatusOK { t.Fatalf("a commit that the packs hold: status %d, want 200, body %v", resp.StatusCode, body) } + if got := len(logLines(logs, refusedWalkLine)); got != refused { + t.Fatalf("a commit that the packs hold added %d %q lines", got-refused, refusedWalkLine) + } }) } } @@ -175,3 +207,40 @@ func TestRefWritesAcceptACommitWhoseHistoryThePacksHold(t *testing.T) { scratchGit(t, repoPath, "", "rev-list", "--objects", "--all") }) } + +// The walk is git work that a writer starts, so it runs under the admission +// slots, the time budget and the heap cap of the git:read routes. With every +// slot of the repository taken, a create answers 503 with Retry-After and +// writes nothing, and it frees its own slot when it is done. +func TestRefWritesWalkUnderTheGitAdmissionSlots(t *testing.T) { + runBackends(t, func(t *testing.T, e *testEnv) { + for _, tc := range refWriteShapes() { + t.Run(tc.name, func(t *testing.T) { + fx := wrefSeedFixture(t, e) + p := withGitReadPolicy(t, testGitReadPolicy(1, time.Minute, gitread.DefaultOutputBytes, gitread.DefaultLimits())) + srv := newWrefServer(t, e) + before := wrefRefs(t, e, fx.repoID) + + // c2 is the tip of main: a base that every shape has to walk to. + key := gitlimit.RepoKey(fx.repoID) + if err := p.slots.Acquire(t.Context(), key); err != nil { + t.Fatal(err) + } + resp, body := wrefDo(t, srv, http.MethodPost, tc.path, fx.token, tc.body(fx.c2)) + if resp.StatusCode != http.StatusServiceUnavailable || resp.Header.Get("Retry-After") == "" { + t.Fatalf("with no free slot: status %d, Retry-After %q, body %v, want 503 with Retry-After", resp.StatusCode, resp.Header.Get("Retry-After"), body) + } + if after := wrefRefs(t, e, fx.repoID); len(after) != len(before) || after[tc.ref] != before[tc.ref] { + t.Fatalf("refs changed while no slot was free: before %v, after %v", before, after) + } + p.slots.Release(key) + + resp, body = wrefDo(t, srv, http.MethodPost, tc.path, fx.token, tc.body(fx.c2)) + if resp.StatusCode != http.StatusOK { + t.Fatalf("with a free slot: status %d, body %v, want 200", resp.StatusCode, body) + } + requireSlotsFree(t, p) + }) + } + }) +} diff --git a/internal/api/registry_target_test.go b/internal/api/registry_target_test.go index cb88718a..0d112812 100644 --- a/internal/api/registry_target_test.go +++ b/internal/api/registry_target_test.go @@ -130,8 +130,9 @@ func (c *countsPackListings) ListPacks(ctx context.Context, repo string) ([]ref. // The registry is read only for a request that is about to write a ref. A // writer with a token can send the same request again and again, and a request -// that the ref policy denies, that names a tag that exists, or that leaves a -// branch where it is, must not cost a registry read and a view of the cache. +// that the ref policy denies, that names a tag or a branch that exists, or that +// leaves a branch where it is, must not cost a registry read, a view of the +// cache and a walk of its history. func TestRefWritesReadTheRegistryOnlyBeforeAWrite(t *testing.T) { runBackends(t, func(t *testing.T, e *testEnv) { fx := wrefSeedFixture(t, e) @@ -155,6 +156,7 @@ func TestRefWritesReadTheRegistryOnlyBeforeAWrite(t *testing.T) { {"policy denies a tag", "/api/v1/repos/tags", denyAll, map[string]any{"name": "denied", "ref": fx.c1}, http.StatusForbidden}, {"the tag exists", "/api/v1/repos/tags", fx.token, map[string]any{"name": "annot", "ref": fx.c1}, http.StatusConflict}, {"the branch is already at the base", "/api/v1/repos/branches/create", fx.token, map[string]any{"target_branch": "feature", "base_ref": fx.c1, "force": true}, http.StatusOK}, + {"a plain create of a branch that exists", "/api/v1/repos/branches/create", fx.token, map[string]any{"target_branch": "feature", "base_ref": fx.c2}, http.StatusConflict}, } { before := counter.n.Load() resp, body := wrefDo(t, srv, http.MethodPost, tc.path, tc.token, tc.body) diff --git a/internal/api/write_merge_source_pin_test.go b/internal/api/write_merge_source_pin_test.go index b3f7bb8c..91d85178 100644 --- a/internal/api/write_merge_source_pin_test.go +++ b/internal/api/write_merge_source_pin_test.go @@ -7,6 +7,9 @@ import ( "reflect" "strings" "testing" + + "github.com/coder/openstorage/internal/gitread" + "github.com/coder/openstorage/internal/ref" ) // TestMergeExpectedSourceSHA pins the source commit with @@ -147,3 +150,57 @@ func TestMergeExpectedSourceSHA(t *testing.T) { }) }) } + +// The source branch can reach a pin that no registered pack holds: a push can +// leave a commit behind a ref tip while the node's cache holds it. A target ref +// or a merge commit at that pin would wedge every other node, which hydrates +// from the registered packs alone, so the pin answers like a commit that the +// source branch does not contain. The tip of the source branch is a ref already +// and needs no check. +func TestMergeExpectedSourceSHARefusesAPinThatThePacksLack(t *testing.T) { + runBackends(t, func(t *testing.T, e *testEnv) { + srv := wmServer(t, e) + fx := wmSeedFixture(t, e) + ctx := t.Context() + + // pin is in the cache only. plantsrc is a branch whose tip is a child of + // pin, in a registered pack that lacks pin: the push that left it there + // was checked against a cache that held pin. + repoPath, err := gitread.Materialize(ctx, e.refStore, e.blobStore, e.cacheDir, fx.repoID, "main") + if err != nil { + t.Fatal(err) + } + pin := plantCacheOnlyCommit(t, repoPath, fx.c2) + dir := t.TempDir() + scratchGit(t, dir, "", "init", "-q", "-b", "main", ".") + tip := scratchGit(t, dir, commitText(coreRevParse(t, fx.work, fx.c2+"^{tree}"), pin), "hash-object", "-t", "commit", "-w", "--stdin", "--literally") + registerLooseObjects(t, e, fx.repoID, dir, tip) + for name, sha := range map[string]string{"plantsrc": tip, "t-plant-ff": fx.c2, "t-plant-merge": fx.c2, "t-plant-ok": fx.c2} { + if won, err := e.refStore.CASRef(ctx, fx.repoID, "refs/heads/"+name, ref.ZeroSHA, sha); err != nil || !won { + t.Fatalf("create %s: won %v, err %v", name, won, err) + } + } + + for _, tc := range []struct{ target, strategy string }{{"t-plant-ff", "ff_only"}, {"t-plant-merge", "merge"}} { + t.Run(tc.strategy, func(t *testing.T) { + before := wmRefsSnapshot(t, e, fx.repoID) + resp, raw := wmDo(t, srv, "POST", "/api/v1/repos/merge", fx.token, + wmMergeBody("plantsrc", tc.target, tc.strategy, map[string]any{"author": wmTestAuthor, "expected_source_sha": pin})) + wmRequireRefGuard(t, resp, raw, "expected_source_sha", tip, + "source branch moved: the source branch does not contain commit "+pin) + if after := wmRefsSnapshot(t, e, fx.repoID); !reflect.DeepEqual(before, after) { + t.Fatalf("refs changed on a refused merge: before %v, after %v", before, after) + } + }) + } + + // The tip itself is a ref already: pinning it is no walk and no refusal. + t.Run("the tip", func(t *testing.T) { + resp, raw := wmDo(t, srv, "POST", "/api/v1/repos/merge", fx.token, + wmMergeBody("plantsrc", "t-plant-ok", "ff_only", map[string]any{"expected_source_sha": tip})) + if body := wmDecode(t, raw); resp.StatusCode != http.StatusOK || body["commit_sha"] != tip { + t.Fatalf("status %d body %s, want fast_forward to the tip %s", resp.StatusCode, raw, tip) + } + }) + }) +} diff --git a/internal/api/write_refs_test.go b/internal/api/write_refs_test.go index 6428e2a0..785aa3cc 100644 --- a/internal/api/write_refs_test.go +++ b/internal/api/write_refs_test.go @@ -18,6 +18,7 @@ import ( "encoding/json" "fmt" "io" + "log/slog" "maps" "net/http" "net/http/httptest" @@ -34,11 +35,17 @@ import ( // newWrefServer builds an httptest server exposing only the W3 write routes // with the production auth middleware, backed by the lane's stores. func newWrefServer(t *testing.T, e *testEnv) *httptest.Server { + t.Helper() + return newWrefServerLogging(t, e, discardLogger()) +} + +// newWrefServerLogging is newWrefServer with the server's log lines going to logger. +func newWrefServerLogging(t *testing.T, e *testEnv, logger *slog.Logger) *httptest.Server { t.Helper() s := &server{ store: e.store, httpBaseURL: testHTTPBaseURL, - logger: discardLogger(), + logger: logger, refStore: e.refStore, blobStore: e.blobStore, cacheDir: e.cacheDir, diff --git a/internal/gitread/gitread.go b/internal/gitread/gitread.go index 50a4ba98..8b34f4cb 100644 --- a/internal/gitread/gitread.go +++ b/internal/gitread/gitread.go @@ -341,16 +341,22 @@ func (m *missingObjectScan) Write(p []byte) (int, error) { // The walk is git's own connectivity check, the one that receive-pack runs for // the new tips of a push: `rev-list --objects --not --all` over a view of the // registered packs (githttp.RegisteredPackView). The answer rests on which -// objects git can read, never on git's error text: --missing=print makes git -// print the id of each object that it cannot read and exit 0, and any other -// outcome of git is an error. The view has no commit-graph on purpose, because a -// commit-graph can describe commits that no registered pack holds and git then -// walks past them. So the cost is the same with and without a commit-graph in -// the cache. +// objects git can read, never on git's error text. A cat-file first asks for +// sha itself, because git before 2.45 dies on a start object that it cannot +// read, whatever --missing says. Then --missing=print makes git print the id of +// each object that it cannot read and exit 0. git before 2.43 also dies on a +// missing parent commit, and git 2.43 dies on a ref at a missing object: exit +// status 128 means that git could not walk the history, so it is the same +// answer, as it is for the ancestry check of a push (isAncestorUncached). Any +// other outcome of git is an error. The view has no commit-graph on purpose, +// because a commit-graph can describe commits that no registered pack holds and +// git then walks past them. So the cost is the same with and without a +// commit-graph in the cache. // // A registered pack that the cache lacks makes a "missing" answer an error that // wraps githttp.ErrViewIncomplete, as it does for RegisteredKinds. A walk that -// outlasts registeredWalkTimeout is an error that wraps ErrTimeBudget. +// outlasts registeredWalkTimeout, or the request budget in ctx, is an error that +// wraps ErrTimeBudget. func RegisteredClosure(ctx context.Context, st ref.RefStore, bl blob.BlobStore, repoPath, repo, sha string) (bool, error) { assertf(ref.ValidSHA(sha), "RegisteredClosure: malformed sha %q", sha) kinds, err := githttp.AskRegisteredPacks(ctx, st, repoPath, repo, func(ctx context.Context) error { @@ -358,20 +364,37 @@ func RegisteredClosure(ctx context.Context, st ref.RefStore, bl blob.BlobStore, }, func(view *githttp.RegisteredPackView) (map[string]string, error) { walkCtx, cancel := context.WithTimeout(ctx, registeredWalkTimeout) defer cancel() - scan := &missingObjectScan{} - _, err := Run(walkCtx, Cmd{Class: Heavy, Dir: repoPath, Env: view.Env(os.Environ()), Stdout: scan, - Args: []string{"-c", "core.commitGraph=false", "rev-list", "--objects", "--missing=print", "--quiet", sha, "--not", "--all"}}) + env := view.Env(os.Environ()) + out, err := Run(walkCtx, Cmd{Class: Light, Dir: repoPath, Env: env, Stdin: strings.NewReader(sha + "\n"), + Args: []string{"cat-file", "--batch-check=%(objecttype)"}}) + if err == nil { + var start map[string]string + if start, err = githttp.ParseObjectKinds([]string{sha}, out); err == nil && start[sha] == "missing" { + return start, nil + } + } + if err == nil { + scan := &missingObjectScan{} + _, err = Run(walkCtx, Cmd{Class: Heavy, Dir: repoPath, Env: env, Stdout: scan, + Args: []string{"-c", "core.commitGraph=false", "rev-list", "--objects", "--missing=print", "--quiet", sha, "--not", "--all"}}) + switch { + case err == nil && scan.err != nil: + err = scan.err + case err == nil && scan.missing: + return map[string]string{sha: "missing"}, nil + case err == nil: + return map[string]string{sha: "commit"}, nil + } + } switch { case ctx.Err() == nil && errors.Is(walkCtx.Err(), context.DeadlineExceeded): return nil, fmt.Errorf("walk the history of %s: %w", sha, ErrTimeBudget) - case err != nil: - return nil, err - case scan.err != nil: - return nil, scan.err - case scan.missing: - return map[string]string{sha: "missing"}, nil + default: + if _, unreadable := gitStderr(err, 128); unreadable { + return map[string]string{sha: "missing"}, nil + } } - return map[string]string{sha: "commit"}, nil + return nil, err }) if err != nil { return false, err diff --git a/internal/gitread/registered_closure_test.go b/internal/gitread/registered_closure_test.go index a7c21cab..1ec6d819 100644 --- a/internal/gitread/registered_closure_test.go +++ b/internal/gitread/registered_closure_test.go @@ -13,9 +13,10 @@ import ( ) // RegisteredClosure decides from the objects that git can read, which -// `rev-list --missing=print` prints as "?" lines, and from nothing else: -// any other outcome of git is an error, never a refusal. A stand-in git shows -// each outcome. +// `rev-list --missing=print` prints as "?" lines, and from the exit status +// 128 with which older git dies on a commit that it cannot read, and from +// nothing else: any other outcome of git is an error, never a refusal. A +// stand-in git shows each outcome. func TestRegisteredClosureAnswersFromTheMissingObjectsAlone(t *testing.T) { sha := strings.Repeat("a", 40) st, err := ref.Open(t.Context(), "sqlite://"+filepath.Join(t.TempDir(), "refs.sqlite")) @@ -39,13 +40,14 @@ func TestRegisteredClosureAnswersFromTheMissingObjectsAlone(t *testing.T) { {"an object is missing", "echo '?" + sha + "'; exit 0", false, ""}, {"a missing object and a failed walk", "echo '?" + sha + "'; exit 128", false, ""}, {"output that is no missing object", "echo 'deadbeef'; exit 0", false, "something other than a missing object"}, - {"git fails", "echo 'fatal: out of memory' >&2; exit 128", false, "out of memory"}, + {"git cannot walk the history (exit 128)", "echo 'fatal: ...' >&2; exit 128", false, ""}, + {"git fails another way (exit 129)", "echo 'usage: git rev-list' >&2; exit 129", false, "exit status 129"}, {"git is killed", "kill -9 $$", false, "killed"}, } { t.Run(tc.name, func(t *testing.T) { bin := t.TempDir() // rev-list is the walk. The stand-in answers the other git commands with exit 0. - script := "#!/bin/sh\ncase \"$*\" in *rev-list*) " + tc.script + " ;; esac\nexit 0\n" + script := "#!/bin/sh\ncase \"$*\" in *cat-file*) echo commit ;; *rev-list*) " + tc.script + " ;; esac\nexit 0\n" if err := os.WriteFile(filepath.Join(bin, "git"), []byte(script), 0o755); err != nil { t.Fatal(err) } @@ -82,7 +84,7 @@ func TestRegisteredClosureWalkBudget(t *testing.T) { t.Fatal(err) } bin := t.TempDir() - if err := os.WriteFile(filepath.Join(bin, "git"), []byte("#!/bin/sh\ncase \"$*\" in *rev-list*) sleep 30 ;; esac\nexit 0\n"), 0o755); err != nil { + if err := os.WriteFile(filepath.Join(bin, "git"), []byte("#!/bin/sh\ncase \"$*\" in *cat-file*) echo commit ;; *rev-list*) sleep 30 ;; esac\nexit 0\n"), 0o755); err != nil { t.Fatal(err) } t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) @@ -104,6 +106,15 @@ func TestRegisteredClosureWalkBudget(t *testing.T) { } } +// A walk that is left to its own budget must end within the time that the +// documentation promises (docs/reference/http-api.md, "Which commit the base and +// the tag target can name"). +func TestRegisteredWalkTimeoutIsThirtySeconds(t *testing.T) { + if registeredWalkTimeout != 30*time.Second { + t.Fatalf("registeredWalkTimeout = %v, want 30s as documented", registeredWalkTimeout) + } +} + // The walk is git's connectivity check, over the registered packs alone and // with no commit-graph: a commit-graph can describe a commit that no registered // pack holds, and git then walks past it. @@ -123,7 +134,7 @@ func TestRegisteredClosureRunsGitsConnectivityCheckOverTheView(t *testing.T) { } bin := t.TempDir() log := filepath.Join(bin, "log") - script := "#!/bin/sh\ncase \"$*\" in *rev-list*) echo \"$*\" >> " + log + "\n" + + script := "#!/bin/sh\ncase \"$*\" in *cat-file*) echo commit ;; *rev-list*) echo \"$*\" >> " + log + "\n" + "echo \"GIT_DIR=$GIT_DIR\" >> " + log + "\necho \"OBJECTS=$GIT_OBJECT_DIRECTORY\" >> " + log + "\n" + "echo \"ALT=$GIT_ALTERNATE_OBJECT_DIRECTORIES\" >> " + log + " ;; esac\nexit 0\n" if err := os.WriteFile(filepath.Join(bin, "git"), []byte(script), 0o755); err != nil { @@ -155,3 +166,39 @@ func TestRegisteredClosureRunsGitsConnectivityCheckOverTheView(t *testing.T) { t.Fatalf("the walk keeps the alternates %q, want none", lines[3]) } } + +// git before 2.45 dies on a start object that it cannot read, whatever +// --missing says, so the walk asks for the object first: an object that no +// registered pack holds is missing without a walk. +func TestRegisteredClosureAsksForTheObjectBeforeTheWalk(t *testing.T) { + sha := strings.Repeat("c", 40) + st, err := ref.Open(t.Context(), "sqlite://"+filepath.Join(t.TempDir(), "refs.sqlite")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { st.Close() }) + if err := st.InitSchema(t.Context()); err != nil { + t.Fatal(err) + } + repoPath := filepath.Join(t.TempDir(), "r.git") + if err := os.MkdirAll(filepath.Join(repoPath, "objects", "pack"), 0o755); err != nil { + t.Fatal(err) + } + bin := t.TempDir() + log := filepath.Join(bin, "log") + script := "#!/bin/sh\necho \"$1\" >> " + log + "\ncase \"$*\" in *cat-file*) cat >/dev/null; echo '" + sha + " missing' ;; *rev-list*) echo 'fatal: bad object' >&2; exit 128 ;; esac\nexit 0\n" + if err := os.WriteFile(filepath.Join(bin, "git"), []byte(script), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) + if ok, err := RegisteredClosure(t.Context(), st, nil, repoPath, "repo_x", sha); err != nil || ok { + t.Fatalf("RegisteredClosure = %v, %v, want false and no error", ok, err) + } + b, err := os.ReadFile(log) + if err != nil { + t.Fatal(err) + } + if got := strings.Fields(string(b)); len(got) != 1 || got[0] != "cat-file" { + t.Fatalf("git ran %q, want one cat-file and no walk", got) + } +} From feb41d2e5b4af8d99d44fc80aee5c447d21b0232 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Fri, 20 Nov 2026 16:48:48 +0000 Subject: [PATCH 3/7] fix(api): state the cost and the failure modes of the registered history walk as measured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The second review of the walk found that exit status 128 also covers a damaged cache (a corrupt pack), that a separate unreferenced history of 100,000 commits already answers 422 under the heap cap, and that a merge answering a busy pool as 500 would not fail a test. - The log line, the monitoring row and http-api.md name both causes of exit status 128 (a missing object on older git, a damaged cache) and say that the answer is 404 for both, as the ancestry check of a push does. - The cost paragraph gives the measured heap threshold of a separate history (50,000 commits fit, 100,000 do not) and that a create with a refused base_ref walks preferred_base under a budget of its own. - A merge with an expected_source_sha answers 503 with Retry-After when no slot is free and frees its slot (test: no free slot). --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_ --- docs/how-to/deploy-production.md | 2 +- docs/reference/http-api.md | 13 +++++---- internal/api/handlers_write_refs.go | 2 +- internal/api/write_merge_source_pin_test.go | 32 +++++++++++++++++++++ 4 files changed, 42 insertions(+), 7 deletions(-) diff --git a/docs/how-to/deploy-production.md b/docs/how-to/deploy-production.md index 826bd43c..92e9d80b 100644 --- a/docs/how-to/deploy-production.md +++ b/docs/how-to/deploy-production.md @@ -735,7 +735,7 @@ webhook events. | Postgres availability + advisory-lock waits at boot | The database is the single authority; boot hangs show in `pg_locks` | | `verify token: key store lookup failed` / `verify token: a registered key cannot be used` log lines | A git, LFS or REST request reached the key store or a registered key, and it failed. The line holds the cause (a cause of more than 300 runes loses its middle). A git or LFS client reads a text that does not hold the cause. For each kind, the git and LFS routes together write at most one line per minute in each `serve` process, the REST routes write at most one more, and a line says how many earlier lines of its kind were held back when some were | | `receive-pack : hook: … git cannot walk the history between two commits, so the update is refused` log line | A push named two commits whose history git cannot read, so the hook refused that update: as non-fast-forward under `no-force-push`, and with `is not a readable commit in this repository` under `verify-sig`. The line has the fields `git`, `old` and `new`. A pusher can cause it with a commit whose parent is missing. A `` that repeats it for ordinary pushes from several clients has a damaged history: check a fresh copy with `git fsck` | -| `a ref write names a commit that the registered packs cannot walk` log line | A `POST /repos/branches/create`, `POST /repos/tags` or `POST /repos/merge` (with `expected_source_sha`) named a commit that no registered pack holds, or whose ancestor, tree or blob no registered pack holds. The line has the fields `repo` and `sha`. A branch create logs one line for each base that it refuses (`base_ref` and `preferred_base` can both be refused). It answers `200` at `preferred_base` when that base is usable and `404` otherwise, a tag answers `404` and a merge answers `409` (`ref_guard_failed`). A pusher can leave such a commit in a pack (git checks only the commits that the refs of a push reach). A `repo` that logs the line for ordinary creates, with different `sha` values that name healthy bases, has a ref at an object that no registered pack holds: compare `openstorage list-packs` with the blob store, and list the ref with `openstorage repair-refs` | +| `a ref write names a commit that the registered packs cannot walk` log line | A `POST /repos/branches/create`, `POST /repos/tags` or `POST /repos/merge` (with `expected_source_sha`) named a commit that no registered pack holds, or whose ancestor, tree or blob no registered pack holds. The line has the fields `repo` and `sha`. A branch create logs one line for each base that it refuses (`base_ref` and `preferred_base` can both be refused). It answers `200` at `preferred_base` when that base is usable and `404` otherwise, a tag answers `404` and a merge answers `409` (`ref_guard_failed`). A pusher can leave such a commit in a pack (git checks only the commits that the refs of a push reach). A `repo` that logs the line for ordinary creates, with different `sha` values that name healthy bases, has a ref at an object that no registered pack holds (compare `openstorage list-packs` with the blob store, and list the ref with `openstorage repair-refs`) or a damaged cache on that node (git exits with status 128 on a corrupt pack, and the check reads that as a history that cannot be walked) | | `lane scan: requeued a lane whose work had no job` log line | A sync or push request lost its job and the [lane scan](#background-maintenance-in-a-fleet) enqueued it again. A `repo=` that repeats needs a look: its runs keep crashing | The opt-in [metrics listener](../reference/metrics.md) covers some of these signals; diff --git a/docs/reference/http-api.md b/docs/reference/http-api.md index f1be7311..dadf6a20 100644 --- a/docs/reference/http-api.md +++ b/docs/reference/http-api.md @@ -774,13 +774,16 @@ on 500,000 linear commits: the tip of a branch takes 5 ms, a commit 1,000 back 1 commits and trees, and also the walk of the history of the refs down to its commit date (1,000 new commits dated after the tip took 14 ms, and the same commits dated 1970 took 1.7 s). A commit whose parent no pack holds always costs the whole history (1.6 s). A separate unreferenced history -costs more: 100,000 commits took 2.8 s. The walk runs under the admission slots, the 30-second -time budget and the 256 MiB heap cap of the git:read routes ([Resource limits on git +costs more, because git also holds its trees: 50,000 commits took 2.4 s, and 100,000 commits need +more than the heap cap. The walk runs under the admission slots, the 30-second time budget and the +256 MiB heap cap (Linux) of the git:read routes ([Resource limits on git reads](#resource-limits-on-git-reads)): a request that finds no free slot answers `503` with `Retry-After`, a walk that outlasts its budget answers `503`, and a walk that needs more memory -(an unreferenced history of 500,000 commits) answers `422`. git 2.39 exits with status 128 on a -missing parent commit, and git 2.43 on a ref at a missing object, instead of printing the object: -the check reads that exit status as a history that cannot be walked. +answers `422`. A create whose `base_ref` is refused walks `preferred_base` under a budget of its +own. git 2.39 exits with status 128 on a missing parent commit, and git 2.43 on a ref at a missing +object, instead of printing the object. The check reads exit status 128 as a history that cannot be +walked, as the ancestry check of a push does. That status is also git's answer to a damaged cache +(a corrupt pack), which therefore answers `404` too, and the log line names both causes. #### Guarded branch create and move diff --git a/internal/api/handlers_write_refs.go b/internal/api/handlers_write_refs.go index 00cea144..d97580ee 100644 --- a/internal/api/handlers_write_refs.go +++ b/internal/api/handlers_write_refs.go @@ -409,7 +409,7 @@ func (s *server) wrefUsableCommit(ctx context.Context, repoID, repoPath, sha str defer b.Close() usable, err := gitread.RegisteredClosure(gitread.WithBudget(ctx, b), s.refStore, s.blobStore, repoPath, repoID, sha) if err == nil && !usable { - s.logger.Warn("a ref write names a commit that the registered packs cannot walk: a pack lacks the commit or one of its ancestors, trees or blobs", + s.logger.Warn("a ref write names a commit that the registered packs cannot walk: a pack lacks the commit or one of its ancestors, trees or blobs, or the cache is damaged", "repo", repoID, "sha", sha) } return usable, err diff --git a/internal/api/write_merge_source_pin_test.go b/internal/api/write_merge_source_pin_test.go index 91d85178..25606abd 100644 --- a/internal/api/write_merge_source_pin_test.go +++ b/internal/api/write_merge_source_pin_test.go @@ -7,7 +7,9 @@ import ( "reflect" "strings" "testing" + "time" + "github.com/coder/openstorage/internal/gitlimit" "github.com/coder/openstorage/internal/gitread" "github.com/coder/openstorage/internal/ref" ) @@ -194,6 +196,36 @@ func TestMergeExpectedSourceSHARefusesAPinThatThePacksLack(t *testing.T) { }) } + // The check is git work that the merge starts, so it runs under the same + // admission slots as a create: with every slot taken the merge answers + // 503 with Retry-After (not 500) and writes nothing, and it frees its + // slot when it is done. ffMid is held completely and reached by ffsrc. + t.Run("no free slot", func(t *testing.T) { + p := withGitReadPolicy(t, testGitReadPolicy(1, time.Minute, gitread.DefaultOutputBytes, gitread.DefaultLimits())) + if won, err := e.refStore.CASRef(ctx, fx.repoID, "refs/heads/t-plant-busy", ref.ZeroSHA, fx.c2); err != nil || !won { + t.Fatalf("create t-plant-busy: won %v, err %v", won, err) + } + body := wmMergeBody("ffsrc", "t-plant-busy", "ff_only", map[string]any{"expected_source_sha": fx.ffMid}) + before := wmRefsSnapshot(t, e, fx.repoID) + key := gitlimit.RepoKey(fx.repoID) + if err := p.slots.Acquire(ctx, key); err != nil { + t.Fatal(err) + } + resp, raw := wmDo(t, srv, "POST", "/api/v1/repos/merge", fx.token, body) + if resp.StatusCode != http.StatusServiceUnavailable || resp.Header.Get("Retry-After") == "" { + t.Fatalf("with no free slot: status %d, Retry-After %q, body %s, want 503 with Retry-After", resp.StatusCode, resp.Header.Get("Retry-After"), raw) + } + if after := wmRefsSnapshot(t, e, fx.repoID); !reflect.DeepEqual(before, after) { + t.Fatalf("refs changed while no slot was free: before %v, after %v", before, after) + } + p.slots.Release(key) + resp, raw = wmDo(t, srv, "POST", "/api/v1/repos/merge", fx.token, body) + if got := wmDecode(t, raw); resp.StatusCode != http.StatusOK || got["commit_sha"] != fx.ffMid { + t.Fatalf("with a free slot: status %d body %s, want fast_forward to %s", resp.StatusCode, raw, fx.ffMid) + } + requireSlotsFree(t, p) + }) + // The tip itself is a ref already: pinning it is no walk and no refusal. t.Run("the tip", func(t *testing.T) { resp, raw := wmDo(t, srv, "POST", "/api/v1/repos/merge", fx.token, From ec8a23e3e383e223547f97b3fcab5fdf58f7963c Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Fri, 20 Nov 2026 16:51:19 +0000 Subject: [PATCH 4/7] fix(api): draw the walks of one ref write from one git budget MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A branch create with a refused base_ref walks preferred_base next, and each walk made a budget of its own, so one request could take two slots in turn and two time budgets. The handler now makes one budget for the request and both walks draw on it, and the slot is freed before the ref is written. TestBranchCreateWalksBothBasesUnderOneBudget runs a fake git that takes a second for each walk under a budget of 1.6 s: the request answers 503 and frees its slot, and with a budget for each walk it answers 404. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_ --- docs/reference/http-api.md | 4 +-- internal/api/handlers_write_merge.go | 4 ++- internal/api/handlers_write_refs.go | 31 +++++++++++++++++----- internal/api/registry_history_test.go | 37 +++++++++++++++++++++++++++ 4 files changed, 66 insertions(+), 10 deletions(-) diff --git a/docs/reference/http-api.md b/docs/reference/http-api.md index dadf6a20..b2e4f1ea 100644 --- a/docs/reference/http-api.md +++ b/docs/reference/http-api.md @@ -779,8 +779,8 @@ more than the heap cap. The walk runs under the admission slots, the 30-second t 256 MiB heap cap (Linux) of the git:read routes ([Resource limits on git reads](#resource-limits-on-git-reads)): a request that finds no free slot answers `503` with `Retry-After`, a walk that outlasts its budget answers `503`, and a walk that needs more memory -answers `422`. A create whose `base_ref` is refused walks `preferred_base` under a budget of its -own. git 2.39 exits with status 128 on a missing parent commit, and git 2.43 on a ref at a missing +answers `422`. A create whose `base_ref` is refused walks `preferred_base` under the same slot and the +same time budget. git 2.39 exits with status 128 on a missing parent commit, and git 2.43 on a ref at a missing object, instead of printing the object. The check reads exit status 128 as a history that cannot be walked, as the ancestry check of a push does. That status is also git's answer to a damaged cache (a corrupt pack), which therefore answers `404` too, and the log line names both causes. diff --git a/internal/api/handlers_write_merge.go b/internal/api/handlers_write_merge.go index 05caf482..0ec8324d 100644 --- a/internal/api/handlers_write_merge.go +++ b/internal/api/handlers_write_merge.go @@ -690,7 +690,9 @@ func (s *server) handleMerge(w http.ResponseWriter, r *http.Request) { // it would wedge every other node, so it answers like a pin that the // source branch does not contain. The tip needs no check: a ref is at it. if req.sourcePin != sourceSHA { - usable, err := s.wrefUsableCommit(ctx, repo.RepoID, repoPath, req.sourcePin) + walk := wrefWalkBudget(repo.RepoID) + usable, err := s.wrefUsableCommit(ctx, walk, repo.RepoID, repoPath, req.sourcePin) + walk.Close() if err != nil { const op = "check that the registered packs hold expected_source_sha and its history" if !s.writeGitLimitErr(w, r, op, err) { diff --git a/internal/api/handlers_write_refs.go b/internal/api/handlers_write_refs.go index d97580ee..3c094cb8 100644 --- a/internal/api/handlers_write_refs.go +++ b/internal/api/handlers_write_refs.go @@ -401,12 +401,14 @@ func wrefValidateCreateBranch(req *wrefCreateBranchRequest) (target string, allo // push can leave such a commit in the registry. A ref at it hydrates, and // every clone, fetch and archive that walks the history then fails. // -// The walk runs under the same admission slots, time budget and heap cap as a +// The walk runs under the same admission slot, time budget and heap cap as a // git:read route (gitReads): a write endpoint has no request budget of its own, -// and a writer must not be able to start any number of walks at once. -func (s *server) wrefUsableCommit(ctx context.Context, repoID, repoPath, sha string) (bool, error) { - b := gitread.NewBudget(gitReads.slots, gitlimit.RepoKey(repoID), gitReads.timeout, gitReads.outputBytes).WithLimits(gitReads.limits) - defer b.Close() +// and a writer must not be able to start any number of walks at once. b is the +// budget of the request, so that the walks of one request (a base, then its +// preferred_base) share one slot and one time budget, as the git calls of a +// read route do. The caller closes b as soon as its walks are over, so that the +// slot is not held while the ref is written. +func (s *server) wrefUsableCommit(ctx context.Context, b *gitread.Budget, repoID, repoPath, sha string) (bool, error) { usable, err := gitread.RegisteredClosure(gitread.WithBudget(ctx, b), s.refStore, s.blobStore, repoPath, repoID, sha) if err == nil && !usable { s.logger.Warn("a ref write names a commit that the registered packs cannot walk: a pack lacks the commit or one of its ancestors, trees or blobs, or the cache is damaged", @@ -415,6 +417,13 @@ func (s *server) wrefUsableCommit(ctx context.Context, repoID, repoPath, sha str return usable, err } +// wrefWalkBudget returns the git budget of one ref write request: the slots, +// time budget and heap cap of a git:read route, drawn for the repository. It +// takes its slot when the first walk starts. The caller closes it. +func wrefWalkBudget(repoID string) *gitread.Budget { + return gitread.NewBudget(gitReads.slots, gitlimit.RepoKey(repoID), gitReads.timeout, gitReads.outputBytes).WithLimits(gitReads.limits) +} + // handleCreateBranch resolves the base to a commit (branch, tag — peeled — // or hex sha via gitread.ResolveRef; preferred_base is the fallback when // the primary base is absent, names no ref or names a commit whose history @@ -464,6 +473,10 @@ func (s *server) handleCreateBranch(w http.ResponseWriter, r *http.Request) { return } ctx := r.Context() + // Both bases of a request (base_ref, then preferred_base) walk under one + // budget: one slot and one time budget for the request. + walks := wrefWalkBudget(repo.RepoID) + defer walks.Close() // create runs the creation at one resolved base. It returns false, with no // response written, when the registered packs lack the base commit or part // of its history (wrefUsableCommit): the node's cache can hold objects that @@ -543,7 +556,7 @@ func (s *server) handleCreateBranch(w http.ResponseWriter, r *http.Request) { return true } } - ok, err := s.wrefUsableCommit(ctx, repo.RepoID, repoPath, baseSHA) + ok, err := s.wrefUsableCommit(ctx, walks, repo.RepoID, repoPath, baseSHA) if err != nil { s.internalErr(w, r, "check that the registered packs hold the base and its history", err) return true @@ -552,6 +565,8 @@ func (s *server) handleCreateBranch(w http.ResponseWriter, r *http.Request) { return false } held = true + // No walk follows: free the slot before the ref is written. + walks.Close() } won, err := s.refStore.CASRef(ctx, repo.RepoID, targetRef, oldSHA, baseSHA) if err != nil { @@ -836,7 +851,9 @@ func (s *server) handleCreateTag(w http.ResponseWriter, r *http.Request) { writeErr(w, http.StatusConflict, "tag already exists") return } - held, err := s.wrefUsableCommit(r.Context(), repo.RepoID, repoPath, sha) + walk := wrefWalkBudget(repo.RepoID) + held, err := s.wrefUsableCommit(r.Context(), walk, repo.RepoID, repoPath, sha) + walk.Close() switch { case err != nil: s.internalErr(w, r, "check that the registered packs hold the tag target and its history", err) diff --git a/internal/api/registry_history_test.go b/internal/api/registry_history_test.go index 5a5d1015..4f8fa336 100644 --- a/internal/api/registry_history_test.go +++ b/internal/api/registry_history_test.go @@ -244,3 +244,40 @@ func TestRefWritesWalkUnderTheGitAdmissionSlots(t *testing.T) { } }) } + +// A create that names two bases walks both under the budget of the request: one +// slot, and one time budget that the second walk draws on. A fake git makes each +// walk take about a second, and the budget lasts for 1.6 s: the first walk fits +// and the second does not. With a budget for each walk both would fit, and the +// request would take two budgets of git time. +func TestBranchCreateWalksBothBasesUnderOneBudget(t *testing.T) { + realGit, err := exec.LookPath("git") + if err != nil { + t.Fatal(err) + } + runBackends(t, func(t *testing.T, e *testEnv) { + fx := wrefSeedFixture(t, e) + first := plantHistory(t, e, fx, "parent") + second := plantHistory(t, e, fx, "tree") + srv := newWrefServer(t, e) + before := wrefRefs(t, e, fx.repoID) + + bin := t.TempDir() + script := "#!/bin/sh\nfor a in \"$@\"; do [ \"$a\" = rev-list ] && sleep 1; done\nexec " + realGit + " \"$@\"\n" + if err := os.WriteFile(filepath.Join(bin, "git"), []byte(script), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) + p := withGitReadPolicy(t, testGitReadPolicy(1, 1600*time.Millisecond, gitread.DefaultOutputBytes, gitread.DefaultLimits())) + + resp, body := wrefDo(t, srv, http.MethodPost, "/api/v1/repos/branches/create", fx.token, + map[string]any{"target_branch": "both", "base_ref": first, "preferred_base": second}) + if resp.StatusCode != http.StatusServiceUnavailable { + t.Fatalf("two bases of a second each under a budget of 1.6 s: status %d, body %v, want 503 (the time budget of the request)", resp.StatusCode, body) + } + if after := wrefRefs(t, e, fx.repoID); len(after) != len(before) { + t.Fatalf("refs changed after the request that ran out of time: before %v, after %v", before, after) + } + requireSlotsFree(t, p) + }) +} From 9f30a09a234e36bf293e1658443962c942e89ab1 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Fri, 20 Nov 2026 17:30:19 +0000 Subject: [PATCH 5/7] fix(api): free the slot of a ref write check on a panic and pin that it is free at the write MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The third review found that the tag handler and the merge pin check closed their budget after the walk without a defer, so a panic between the first git child and the close (what the recovery middleware is for) kept the slot until restart, and that nothing pinned that a branch create frees its slot before the ref is written. - wrefUsableCommitAlone owns its budget and frees it with a defer. The tag handler and the merge pin check use it. - TestRefWritesFreeTheirSlotAfterAPanic and the merge subtest panic at the registry read that follows a "missing" answer and check that no slot is held. - TestRefWritesFreeTheirSlotBeforeTheRefIsWritten records the slots in use at each ref write of a tag, a branch, an allocated name and a fallback. - The http-api.md and PR wording of the damaged cache and of the two walks of a create are made exact. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_ --- docs/reference/http-api.md | 11 +- internal/api/handlers_write_merge.go | 4 +- internal/api/handlers_write_refs.go | 14 ++- internal/api/registry_history_test.go | 115 ++++++++++++++++++++ internal/api/write_merge_source_pin_test.go | 20 ++++ 5 files changed, 153 insertions(+), 11 deletions(-) diff --git a/docs/reference/http-api.md b/docs/reference/http-api.md index b2e4f1ea..d7c72ac3 100644 --- a/docs/reference/http-api.md +++ b/docs/reference/http-api.md @@ -779,11 +779,12 @@ more than the heap cap. The walk runs under the admission slots, the 30-second t 256 MiB heap cap (Linux) of the git:read routes ([Resource limits on git reads](#resource-limits-on-git-reads)): a request that finds no free slot answers `503` with `Retry-After`, a walk that outlasts its budget answers `503`, and a walk that needs more memory -answers `422`. A create whose `base_ref` is refused walks `preferred_base` under the same slot and the -same time budget. git 2.39 exits with status 128 on a missing parent commit, and git 2.43 on a ref at a missing -object, instead of printing the object. The check reads exit status 128 as a history that cannot be -walked, as the ancestry check of a push does. That status is also git's answer to a damaged cache -(a corrupt pack), which therefore answers `404` too, and the log line names both causes. +answers `422`. A create whose `base_ref` is refused walks `preferred_base` under the same slot +and the same time budget. git 2.39 exits with status 128 on a missing parent commit, and git 2.43 +on a ref at a missing object, instead of printing the object. The check reads exit status 128 as a +history that cannot be walked, as the ancestry check of a push does. That status is git's answer to +any fatal failure, so a damaged cache (a corrupt pack, for one) answers `404` too, and the log +line names both causes. #### Guarded branch create and move diff --git a/internal/api/handlers_write_merge.go b/internal/api/handlers_write_merge.go index 0ec8324d..c27e8db6 100644 --- a/internal/api/handlers_write_merge.go +++ b/internal/api/handlers_write_merge.go @@ -690,9 +690,7 @@ func (s *server) handleMerge(w http.ResponseWriter, r *http.Request) { // it would wedge every other node, so it answers like a pin that the // source branch does not contain. The tip needs no check: a ref is at it. if req.sourcePin != sourceSHA { - walk := wrefWalkBudget(repo.RepoID) - usable, err := s.wrefUsableCommit(ctx, walk, repo.RepoID, repoPath, req.sourcePin) - walk.Close() + usable, err := s.wrefUsableCommitAlone(ctx, repo.RepoID, repoPath, req.sourcePin) if err != nil { const op = "check that the registered packs hold expected_source_sha and its history" if !s.writeGitLimitErr(w, r, op, err) { diff --git a/internal/api/handlers_write_refs.go b/internal/api/handlers_write_refs.go index 3c094cb8..abb776d4 100644 --- a/internal/api/handlers_write_refs.go +++ b/internal/api/handlers_write_refs.go @@ -417,6 +417,16 @@ func (s *server) wrefUsableCommit(ctx context.Context, b *gitread.Budget, repoID return usable, err } +// wrefUsableCommitAlone is wrefUsableCommit for a request that checks one +// commit: the budget is its own, and it is freed when the check ends, whatever +// ends it (a panic included), so that the slot is not held while the ref is +// written. +func (s *server) wrefUsableCommitAlone(ctx context.Context, repoID, repoPath, sha string) (bool, error) { + b := wrefWalkBudget(repoID) + defer b.Close() + return s.wrefUsableCommit(ctx, b, repoID, repoPath, sha) +} + // wrefWalkBudget returns the git budget of one ref write request: the slots, // time budget and heap cap of a git:read route, drawn for the repository. It // takes its slot when the first walk starts. The caller closes it. @@ -851,9 +861,7 @@ func (s *server) handleCreateTag(w http.ResponseWriter, r *http.Request) { writeErr(w, http.StatusConflict, "tag already exists") return } - walk := wrefWalkBudget(repo.RepoID) - held, err := s.wrefUsableCommit(r.Context(), walk, repo.RepoID, repoPath, sha) - walk.Close() + held, err := s.wrefUsableCommitAlone(r.Context(), repo.RepoID, repoPath, sha) switch { case err != nil: s.internalErr(w, r, "check that the registered packs hold the tag target and its history", err) diff --git a/internal/api/registry_history_test.go b/internal/api/registry_history_test.go index 4f8fa336..fc6d488f 100644 --- a/internal/api/registry_history_test.go +++ b/internal/api/registry_history_test.go @@ -16,6 +16,8 @@ import ( "path/filepath" "strconv" "strings" + "sync" + "sync/atomic" "testing" "time" @@ -281,3 +283,116 @@ func TestBranchCreateWalksBothBasesUnderOneBudget(t *testing.T) { requireSlotsFree(t, p) }) } + +// slotsAtCAS records how many git read slots are held at each ref write. +type slotsAtCAS struct { + ref.Store + p *gitReadPolicy + + mu sync.Mutex + inUse []int +} + +func (s *slotsAtCAS) CASRef(ctx context.Context, repo, name, old, new string) (bool, error) { + s.mu.Lock() + s.inUse = append(s.inUse, s.p.slots.InUse()) + s.mu.Unlock() + return s.Store.CASRef(ctx, repo, name, old, new) +} + +// A walk holds its slot while git runs and no longer: a ref write waits for a +// concurrent push's hold on its ref for up to ten seconds, and the process has +// eight slots for every git read. +func TestRefWritesFreeTheirSlotBeforeTheRefIsWritten(t *testing.T) { + runBackends(t, func(t *testing.T, e *testEnv) { + type shape struct { + name, path string + body func(fx wrefFixture, planted string) map[string]any + } + shapes := []shape{ + {"a tag", "/api/v1/repos/tags", func(fx wrefFixture, _ string) map[string]any { + return map[string]any{"name": "slot-tag", "ref": fx.c2} + }}, + {"a branch", "/api/v1/repos/branches/create", func(fx wrefFixture, _ string) map[string]any { + return map[string]any{"target_branch": "slot-branch", "base_ref": fx.c2} + }}, + {"an allocated branch name", "/api/v1/repos/branches/create", func(fx wrefFixture, _ string) map[string]any { + return map[string]any{"base_ref": fx.c2} + }}, + {"a branch after a refused base_ref", "/api/v1/repos/branches/create", func(fx wrefFixture, planted string) map[string]any { + return map[string]any{"target_branch": "slot-fallback", "base_ref": planted, "preferred_base": fx.c2} + }}, + } + for _, tc := range shapes { + t.Run(tc.name, func(t *testing.T) { + fx := wrefSeedFixture(t, e) + planted := plantHistory(t, e, fx, "parent") + p := withGitReadPolicy(t, testGitReadPolicy(1, time.Minute, gitread.DefaultOutputBytes, gitread.DefaultLimits())) + probe := &slotsAtCAS{Store: e.refStore, p: p} + env := *e + env.refStore = probe + srv := newWrefServer(t, &env) + + resp, body := wrefDo(t, srv, http.MethodPost, tc.path, fx.token, tc.body(fx, planted)) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status %d, body %v, want 200", resp.StatusCode, body) + } + probe.mu.Lock() + defer probe.mu.Unlock() + if len(probe.inUse) == 0 { + t.Fatal("the request wrote no ref") + } + for _, n := range probe.inUse { + if n != 0 { + t.Fatalf("%d git read slots held while the ref was written, want 0 (slots held at each write: %v)", n, probe.inUse) + } + } + requireSlotsFree(t, p) + }) + } + }) +} + +// panicsWhileASlotIsHeld panics at the first registry read that happens while a +// git read slot is held, as a bug in the walk would. A walk reads the registry +// again after git answered "missing", so a commit that the packs lack reaches it. +type panicsWhileASlotIsHeld struct { + ref.Store + p *gitReadPolicy + armed atomic.Bool +} + +func (s *panicsWhileASlotIsHeld) ListPacks(ctx context.Context, repo string) ([]ref.Pack, error) { + if s.armed.Load() && s.p.slots.InUse() > 0 { + panic("a bug in the walk") + } + return s.Store.ListPacks(ctx, repo) +} + +// A panic between the first git child and the end of the check is what the +// recovery middleware is for. It must not take the slot with it: the process has +// eight, and two lost slots of one repository end every git read of it. +func TestRefWritesFreeTheirSlotAfterAPanic(t *testing.T) { + runBackends(t, func(t *testing.T, e *testEnv) { + for _, tc := range refWriteShapes() { + t.Run(tc.name, func(t *testing.T) { + fx := wrefSeedFixture(t, e) + // A commit that the packs lack: only a "missing" answer reads the + // registry again, and it does so while the walk holds its slot. + planted := plantHistory(t, e, fx, "parent") + p := withGitReadPolicy(t, testGitReadPolicy(1, time.Minute, gitread.DefaultOutputBytes, gitread.DefaultLimits())) + probe := &panicsWhileASlotIsHeld{Store: e.refStore, p: p} + env := *e + env.refStore = probe + srv := newWrefServer(t, &env) + probe.armed.Store(true) + resp, body := wrefDo(t, srv, http.MethodPost, tc.path, fx.token, tc.body(planted)) + probe.armed.Store(false) + if resp.StatusCode != http.StatusInternalServerError { + t.Fatalf("status %d, body %v, want the 500 of a panic", resp.StatusCode, body) + } + requireSlotsFree(t, p) + }) + } + }) +} diff --git a/internal/api/write_merge_source_pin_test.go b/internal/api/write_merge_source_pin_test.go index 25606abd..e9c3ea12 100644 --- a/internal/api/write_merge_source_pin_test.go +++ b/internal/api/write_merge_source_pin_test.go @@ -226,6 +226,26 @@ func TestMergeExpectedSourceSHARefusesAPinThatThePacksLack(t *testing.T) { requireSlotsFree(t, p) }) + // A panic after the first git child must not take the slot with it. + t.Run("a panic after the walk", func(t *testing.T) { + p := withGitReadPolicy(t, testGitReadPolicy(1, time.Minute, gitread.DefaultOutputBytes, gitread.DefaultLimits())) + if won, err := e.refStore.CASRef(ctx, fx.repoID, "refs/heads/t-plant-panic", ref.ZeroSHA, fx.c2); err != nil || !won { + t.Fatalf("create t-plant-panic: won %v, err %v", won, err) + } + probe := &panicsWhileASlotIsHeld{Store: e.refStore, p: p} + env := *e + env.refStore = probe + panicSrv := wmServer(t, &env) + probe.armed.Store(true) + resp, raw := wmDo(t, panicSrv, "POST", "/api/v1/repos/merge", fx.token, + wmMergeBody("plantsrc", "t-plant-panic", "ff_only", map[string]any{"expected_source_sha": pin})) + probe.armed.Store(false) + if resp.StatusCode != http.StatusInternalServerError { + t.Fatalf("status %d, body %s, want the 500 of a panic", resp.StatusCode, raw) + } + requireSlotsFree(t, p) + }) + // The tip itself is a ref already: pinning it is no walk and no refusal. t.Run("the tip", func(t *testing.T) { resp, raw := wmDo(t, srv, "POST", "/api/v1/repos/merge", fx.token, From 411bbc06830d1cf19f3a4fde53fbda346b7a9603 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Fri, 20 Nov 2026 18:00:10 +0000 Subject: [PATCH 6/7] docs(api): say that the registered history walk trusts the refs of the repository MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The walk stops at a commit that a ref reaches (rev-list --not --all) and reads no tree or blob below it. A ref that a push left at a commit whose history the packs cannot walk therefore lets a branch or tag at the same commit, or at one that it reaches, pass. A walk without the boundary reads the whole history with its trees for every request, which a repository of 500,000 commits does not fit under the heap cap. http-api.md now states the limit and its reason. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_ --- docs/reference/http-api.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/reference/http-api.md b/docs/reference/http-api.md index d7c72ac3..b67713d1 100644 --- a/docs/reference/http-api.md +++ b/docs/reference/http-api.md @@ -786,6 +786,13 @@ history that cannot be walked, as the ancestry check of a push does. That status any fatal failure, so a damaged cache (a corrupt pack, for one) answers `404` too, and the log line names both causes. +The walk trusts the refs of the repository to be complete: it stops at a commit that a ref reaches +and reads no tree or blob below it. A ref that a push left at a commit whose history the packs +cannot walk (pushes keep that limit) therefore lets a branch or tag at the same commit, or at a +commit that it reaches, pass. The check cannot tell that ref from a healthy one without the walk +of the whole history for every request, which a repository of 500,000 commits does not fit under +the heap cap. + #### Guarded branch create and move `expected_target_sha` on `POST /repos/branches/create` (both path styles) guards the From e06cac768a67a319f5f54a1c3abe4fdab70f7c76 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Fri, 20 Nov 2026 18:15:05 +0000 Subject: [PATCH 7/7] docs(api): state exactly what the trust in the refs of a repository lets through MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The walk does read trees below the commits that a ref reaches (git marks their trees as uninteresting and skips a missing one without reporting it), so the paragraph now says that it reports no missing object below them. A complete commit built on a ref that sits at a broken history also passes, which the paragraph left out. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_ --- docs/reference/http-api.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/reference/http-api.md b/docs/reference/http-api.md index b67713d1..f926e7e3 100644 --- a/docs/reference/http-api.md +++ b/docs/reference/http-api.md @@ -786,12 +786,12 @@ history that cannot be walked, as the ancestry check of a push does. That status any fatal failure, so a damaged cache (a corrupt pack, for one) answers `404` too, and the log line names both causes. -The walk trusts the refs of the repository to be complete: it stops at a commit that a ref reaches -and reads no tree or blob below it. A ref that a push left at a commit whose history the packs -cannot walk (pushes keep that limit) therefore lets a branch or tag at the same commit, or at a -commit that it reaches, pass. The check cannot tell that ref from a healthy one without the walk -of the whole history for every request, which a repository of 500,000 commits does not fit under -the heap cap. +The walk trusts the refs of the repository to be complete: it stops at the commits that a ref +reaches and reports no missing object below them. A ref that a push left at a commit whose history +the packs cannot walk (pushes keep that limit) therefore lets a branch or tag at the same commit, +at a commit that it reaches, or at a complete commit built on top of it, pass. The check cannot +tell that ref from a healthy one without the walk of the whole history for every request, which a +repository of 500,000 commits does not fit under the heap cap. #### Guarded branch create and move