diff --git a/docs/how-to/deploy-production.md b/docs/how-to/deploy-production.md index bce9303a..f2f04564 100644 --- a/docs/how-to/deploy-production.md +++ b/docs/how-to/deploy-production.md @@ -738,6 +738,7 @@ webhook events. | Postgres availability + advisory-lock waits at boot | The database is the single authority; boot hangs show in `pg_locks` | | `verify token: key store lookup failed` / `verify token: a registered key cannot be used` log lines | A git, LFS or REST request reached the key store or a registered key, and it failed. The line holds the cause (a cause of more than 300 runes loses its middle). A git or LFS client reads a text that does not hold the cause. For each kind, the git and LFS routes together write at most one line per minute in each `serve` process, the REST routes write at most one more, and a line says how many earlier lines of its kind were held back when some were | | `receive-pack : hook: … git cannot walk the history between two commits, so the update is refused` log line | A push named two commits whose history git cannot read, so the hook refused that update: as non-fast-forward under `no-force-push`, and with `is not a readable commit in this repository` under `verify-sig`. The line has the fields `git`, `old` and `new`. A pusher can cause it with a commit whose parent is missing. A `` that repeats it for ordinary pushes from several clients has a damaged history: check a fresh copy with `git fsck` | +| `a ref write names a commit that the registered packs cannot walk` log line | A `POST /repos/branches/create`, `POST /repos/tags` or `POST /repos/merge` (with `expected_source_sha`) named a commit that no registered pack holds, or whose ancestor, tree or blob no registered pack holds. The line has the fields `repo` and `sha`. A branch create logs one line for each base that it refuses (`base_ref` and `preferred_base` can both be refused). It answers `200` at `preferred_base` when that base is usable and `404` otherwise, a tag answers `404` and a merge answers `409` (`ref_guard_failed`). A pusher can leave such a commit in a pack (git checks only the commits that the refs of a push reach). A `repo` that logs the line for ordinary creates, with different `sha` values that name healthy bases, has a ref at an object that no registered pack holds (compare `openstorage list-packs` with the blob store, and list the ref with `openstorage repair-refs`) or a damaged cache on that node (git exits with status 128 on a corrupt pack, and the check reads that as a history that cannot be walked) | | `receive-pack : hook: … a push deletes a ref that its replica does not list, and the ref store holds it, so the update is refused` log line | A client sent `git push :` for a ref that its replica's cache does not list, and the ref store holds the ref. The pusher read ` exists at in the ref store, but this replica does not list it`. The line has the fields `ref`, `sha` and `cause`. One line after a ref was created is normal: the client fetches and pushes again. A `` that repeats it has a replica whose cache stays behind: look for a `hydrate :` error on that replica | | `lane scan: requeued a lane whose work had no job` log line | A sync or push request lost its job and the [lane scan](#background-maintenance-in-a-fleet) enqueued it again. A `repo=` that repeats needs a look: its runs keep crashing | @@ -771,6 +772,7 @@ line can be up to 1 MiB. | Postgres availability and advisory-lock waits (What to monitor) | **log only** | Observe PostgreSQL itself (`pg_locks`) | | `verify token: …` lines (What to monitor) | **log only** | The two texts shown, at most one line for each kind per minute for the git and LFS routes together, and one more for the REST routes, in each `serve` process | | `git cannot walk the history between two commits` (What to monitor) | **log only** | A hook line, so metrics do not count it. Fields `git`, `old`, `new` and `cause` | +| `a ref write names a commit that the registered packs cannot walk` (What to monitor) | **log only** | An API line, so metrics do not count it. Fields `repo` and `sha` | | `a push deletes a ref that its replica does not list` (What to monitor) | **log only** | A hook line, so metrics do not count it. Fields `ref`, `sha` and `cause` | | `lane scan: requeued a lane whose work had no job` (What to monitor) | **log only** | Fields `org`, `repo` and `lane` | | S3WAL `… conditional-write contention retries exhausted` (S3WAL signals) | `openstorage_s3wal_cas_contention_exhausted_total` by `scope` | Serve-process mutations only; push-side failures reach the client and hook stderr | diff --git a/docs/reference/http-api.md b/docs/reference/http-api.md index fe305182..f926e7e3 100644 --- a/docs/reference/http-api.md +++ b/docs/reference/http-api.md @@ -685,7 +685,11 @@ moved on since. - The source branch must still contain the commit: the commit is the tip or one of its ancestors. Otherwise the guard fails (`detail` "source branch moved: the source - branch does not contain commit …"). + branch does not contain commit …"). The same answer covers a commit that the source + branch reaches but the registered packs cannot walk, the way the [branch and tag + create routes](#which-commit-the-base-and-the-tag-target-can-name) refuse such a commit + (a push can leave one behind the tip of a branch): a pin that is not the tip is checked + like a base, and the server logs the refusal. - A source branch that does not exist fails the guard too, with `current_sha` 40 zeros. Without `expected_source_sha`, a missing source branch is `404`. - `source.sha` in the response is the pinned commit. @@ -749,10 +753,46 @@ The server checks the registered packs, not the node's cache repository, because ref at any other object. A commit that only a cache holds answers `404`, like a commit that does not exist, and a `base_ref` that names no usable commit still falls back to `preferred_base`. The server reads the registry only for a request that is about to write: after the ref policy, and not -for a tag that exists or for a create that leaves the branch at its tip (with `force` or -`expected_target_sha`). So a request that the policy denies +for a tag that exists, for a plain create of a branch that exists (`409`) or for a create that +leaves the branch at its tip (with `force` or `expected_target_sha`). So a request that the policy denies answers `403` even when its commit is only in the cache. +The registered packs must also hold what git needs to read the commit's history: its ancestors, +down to the commits that a ref of the repository reaches, and their trees and blobs. A push can +leave a commit in a registered pack whose parent, tree or blob no pack holds, because +`receive-pack` checks the commits that a push's refs reach and no other commit of its pack. A ref +at such a commit would hydrate, and every clone, fetch and archive that walks the history would +fail. A commit whose history the registered packs cannot walk answers `404`, like a commit that +does not exist. The check is the walk that `receive-pack` runs for the new tips of a push (`git +rev-list --objects --not --all`) over the registered packs, without a commit-graph, because a +commit-graph can describe a commit that no registered pack holds. + +The cost is the walk from the commits that the refs reach down to the commit, so it grows with the +distance between the two and with the commit dates that the pusher chose. Measured with git 2.55 +on 500,000 linear commits: the tip of a branch takes 5 ms, a commit 1,000 back 15 ms, 100,000 back +0.3 s and the root commit 1.6 s. A commit that no ref reaches costs the walk of its own new +commits and trees, and also the walk of the history of the refs down to its commit date (1,000 new +commits dated after the tip took 14 ms, and the same commits dated 1970 took 1.7 s). A commit +whose parent no pack holds always costs the whole history (1.6 s). A separate unreferenced history +costs more, because git also holds its trees: 50,000 commits took 2.4 s, and 100,000 commits need +more than the heap cap. The walk runs under the admission slots, the 30-second time budget and the +256 MiB heap cap (Linux) of the git:read routes ([Resource limits on git +reads](#resource-limits-on-git-reads)): a request that finds no free slot answers `503` with +`Retry-After`, a walk that outlasts its budget answers `503`, and a walk that needs more memory +answers `422`. A create whose `base_ref` is refused walks `preferred_base` under the same slot +and the same time budget. git 2.39 exits with status 128 on a missing parent commit, and git 2.43 +on a ref at a missing object, instead of printing the object. The check reads exit status 128 as a +history that cannot be walked, as the ancestry check of a push does. That status is git's answer to +any fatal failure, so a damaged cache (a corrupt pack, for one) answers `404` too, and the log +line names both causes. + +The walk trusts the refs of the repository to be complete: it stops at the commits that a ref +reaches and reports no missing object below them. A ref that a push left at a commit whose history +the packs cannot walk (pushes keep that limit) therefore lets a branch or tag at the same commit, +at a commit that it reaches, or at a complete commit built on top of it, pass. The check cannot +tell that ref from a healthy one without the walk of the whole history for every request, which a +repository of 500,000 commits does not fit under the heap cap. + #### Guarded branch create and move `expected_target_sha` on `POST /repos/branches/create` (both path styles) guards the diff --git a/internal/api/handlers_write_merge.go b/internal/api/handlers_write_merge.go index 96fc8605..c27e8db6 100644 --- a/internal/api/handlers_write_merge.go +++ b/internal/api/handlers_write_merge.go @@ -684,6 +684,25 @@ func (s *server) handleMerge(w http.ResponseWriter, r *http.Request) { writeRefGuardFailed(w, "expected_source_sha", sourceSHA, sourceMovedMsg) return } + // The source branch reaches the pin, but the pin can still be a commit + // that only this node's cache holds (a push can leave a commit behind + // a ref tip that no registered pack holds). A ref or a merge commit at + // it would wedge every other node, so it answers like a pin that the + // source branch does not contain. The tip needs no check: a ref is at it. + if req.sourcePin != sourceSHA { + usable, err := s.wrefUsableCommitAlone(ctx, repo.RepoID, repoPath, req.sourcePin) + if err != nil { + const op = "check that the registered packs hold expected_source_sha and its history" + if !s.writeGitLimitErr(w, r, op, err) { + s.wmInternal(w, r, op, err) + } + return + } + if !usable { + writeRefGuardFailed(w, "expected_source_sha", sourceSHA, sourceMovedMsg) + return + } + } sourceSHA = req.sourcePin } diff --git a/internal/api/handlers_write_refs.go b/internal/api/handlers_write_refs.go index c439d872..abb776d4 100644 --- a/internal/api/handlers_write_refs.go +++ b/internal/api/handlers_write_refs.go @@ -386,24 +386,58 @@ func wrefValidateCreateBranch(req *wrefCreateBranchRequest) (target string, allo return target, allocate, guard, errs.err() } -// wrefHeldCommit reports whether a pack that the registry lists for the -// repository holds the commit sha. A write that stores the commit in a ref -// must ask: the node's cache can hold objects that no registered pack holds (a -// cache that was copied or edited out of band, or a registry that was rolled -// back), and a ref at such an object wedges every other node, and this node -// after its cache is gone, because they hydrate from the registered packs -// alone. -func (s *server) wrefHeldCommit(ctx context.Context, repoID, repoPath, sha string) (bool, error) { - kinds, err := gitread.RegisteredKinds(ctx, s.refStore, s.blobStore, repoPath, repoID, []string{sha}) - if err != nil { - return false, err - } - return kinds[sha] == "commit", nil +// wrefUsableCommit reports whether the registered packs hold the commit sha and +// what git needs to walk its history: its parents down to the commits that a +// ref reaches, and their trees and blobs (gitread.RegisteredClosure). A write +// that stores the commit in a ref must ask, for two reasons. +// +// The node's cache can hold objects that no registered pack holds (a cache that +// was copied or edited out of band, or a registry that was rolled back), and a +// ref at such an object wedges every other node, and this node after its cache +// is gone, because they hydrate from the registered packs alone. +// +// A pack can also hold a commit whose parent no pack holds. receive-pack checks +// the commits that a push's refs reach and no other commit of its pack, so a +// push can leave such a commit in the registry. A ref at it hydrates, and +// every clone, fetch and archive that walks the history then fails. +// +// The walk runs under the same admission slot, time budget and heap cap as a +// git:read route (gitReads): a write endpoint has no request budget of its own, +// and a writer must not be able to start any number of walks at once. b is the +// budget of the request, so that the walks of one request (a base, then its +// preferred_base) share one slot and one time budget, as the git calls of a +// read route do. The caller closes b as soon as its walks are over, so that the +// slot is not held while the ref is written. +func (s *server) wrefUsableCommit(ctx context.Context, b *gitread.Budget, repoID, repoPath, sha string) (bool, error) { + usable, err := gitread.RegisteredClosure(gitread.WithBudget(ctx, b), s.refStore, s.blobStore, repoPath, repoID, sha) + if err == nil && !usable { + s.logger.Warn("a ref write names a commit that the registered packs cannot walk: a pack lacks the commit or one of its ancestors, trees or blobs, or the cache is damaged", + "repo", repoID, "sha", sha) + } + return usable, err +} + +// wrefUsableCommitAlone is wrefUsableCommit for a request that checks one +// commit: the budget is its own, and it is freed when the check ends, whatever +// ends it (a panic included), so that the slot is not held while the ref is +// written. +func (s *server) wrefUsableCommitAlone(ctx context.Context, repoID, repoPath, sha string) (bool, error) { + b := wrefWalkBudget(repoID) + defer b.Close() + return s.wrefUsableCommit(ctx, b, repoID, repoPath, sha) +} + +// wrefWalkBudget returns the git budget of one ref write request: the slots, +// time budget and heap cap of a git:read route, drawn for the repository. It +// takes its slot when the first walk starts. The caller closes it. +func wrefWalkBudget(repoID string) *gitread.Budget { + return gitread.NewBudget(gitReads.slots, gitlimit.RepoKey(repoID), gitReads.timeout, gitReads.outputBytes).WithLimits(gitReads.limits) } // handleCreateBranch resolves the base to a commit (branch, tag — peeled — // or hex sha via gitread.ResolveRef; preferred_base is the fallback when -// the primary base is absent or names no ref, and a base whose commit git +// the primary base is absent, names no ref or names a commit whose history +// the registered packs cannot walk, and a base whose commit git // cannot read is a 500, with no fallback), policy-checks the new ref, // and CAS creates it. With force, an existing target is CAS-recreated at // the base — no-force-push policies still deny non-fast-forward moves via a @@ -449,14 +483,19 @@ func (s *server) handleCreateBranch(w http.ResponseWriter, r *http.Request) { return } ctx := r.Context() + // Both bases of a request (base_ref, then preferred_base) walk under one + // budget: one slot and one time budget for the request. + walks := wrefWalkBudget(repo.RepoID) + defer walks.Close() // create runs the creation at one resolved base. It returns false, with no - // response written, when no registered pack holds the base commit: the - // node's cache can hold objects that no registered pack holds (a cache that - // was copied or edited out of band, or a registry that was rolled back), and - // a ref at such an object wedges every other node, and this node after its - // cache is gone, because they hydrate from the registered packs alone. The - // registry is read only for a request that is about to write: a denied - // request, and one that leaves the branch where it is, never read it. + // response written, when the registered packs lack the base commit or part + // of its history (wrefUsableCommit): the node's cache can hold objects that + // no registered pack holds (a cache that was copied or edited out of band, + // or a registry that was rolled back), and a ref at such an object wedges + // every other node, and this node after its cache is gone, because they + // hydrate from the registered packs alone. The registry is read only for a + // request that is about to write: a denied request, and one that leaves the + // branch where it is, never read it. create := func(baseSHA string) bool { held := false attempts := 1 @@ -515,15 +554,29 @@ func (s *server) handleCreateBranch(w http.ResponseWriter, r *http.Request) { return true } if !held { - ok, err := s.wrefHeldCommit(ctx, repo.RepoID, repoPath, baseSHA) + // A create that cannot win is answered before the walk: a plain + // create of a branch that exists loses its swap, whatever the base. + if oldSHA == ref.ZeroSHA && !allocate { + now, ok := s.wrefTip(w, r, repo.RepoID, targetRef) + if !ok { + return true + } + if now != ref.ZeroSHA { + writeErr(w, http.StatusConflict, "branch already exists") + return true + } + } + ok, err := s.wrefUsableCommit(ctx, walks, repo.RepoID, repoPath, baseSHA) if err != nil { - s.internalErr(w, r, "check that a registered pack holds the base", err) + s.internalErr(w, r, "check that the registered packs hold the base and its history", err) return true } if !ok { return false } held = true + // No walk follows: free the slot before the ref is written. + walks.Close() } won, err := s.refStore.CASRef(ctx, repo.RepoID, targetRef, oldSHA, baseSHA) if err != nil { @@ -808,10 +861,10 @@ func (s *server) handleCreateTag(w http.ResponseWriter, r *http.Request) { writeErr(w, http.StatusConflict, "tag already exists") return } - held, err := s.wrefHeldCommit(r.Context(), repo.RepoID, repoPath, sha) + held, err := s.wrefUsableCommitAlone(r.Context(), repo.RepoID, repoPath, sha) switch { case err != nil: - s.internalErr(w, r, "check that a registered pack holds the tag target", err) + s.internalErr(w, r, "check that the registered packs hold the tag target and its history", err) return case !held: writeProblem(w, http.StatusNotFound, codeRefNotFound, "ref does not resolve to a commit") diff --git a/internal/api/registry_history_test.go b/internal/api/registry_history_test.go new file mode 100644 index 00000000..fc6d488f --- /dev/null +++ b/internal/api/registry_history_test.go @@ -0,0 +1,398 @@ +package api + +// A ref write must not store a commit whose history the registered packs +// cannot walk. receive-pack checks the commits that a push's refs reach and no +// other commit of the pack, so a push can leave a commit in a registered pack +// whose parent, tree or blob no pack holds. A ref at it hydrates, and every +// clone, fetch and archive that walks the history fails (#896 held the commit +// itself and left this "closure variant" open). + +import ( + "context" + "log/slog" + "net/http" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/coder/openstorage/internal/blob" + "github.com/coder/openstorage/internal/gitlimit" + "github.com/coder/openstorage/internal/gitread" + "github.com/coder/openstorage/internal/ref" +) + +const absentID = "1111111111111111111111111111111111111111" + +// refusedWalkLine is the log line of a ref write that names a commit whose +// history the registered packs cannot walk. The operator's only sign of such a +// request: the response is the same 404 as for a commit that does not exist. +const refusedWalkLine = "a ref write names a commit that the registered packs cannot walk" + +// logLines returns the lines of the log that hold every one of the substrings. +func logLines(l *lockedLog, substrings ...string) []string { + var out []string +next: + for _, line := range strings.Split(l.String(), "\n") { + for _, sub := range substrings { + if !strings.Contains(line, sub) { + continue next + } + } + out = append(out, line) + } + return out +} + +// scratchGit runs git in dir with stdin and no GIT_ variable of the host. +func scratchGit(t *testing.T, dir, stdin string, args ...string) string { + t.Helper() + cmd := exec.Command("git", args...) + cmd.Dir = dir + for _, kv := range os.Environ() { + if !strings.HasPrefix(kv, "GIT_") { + cmd.Env = append(cmd.Env, kv) + } + } + cmd.Env = append(cmd.Env, "GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_SYSTEM=/dev/null") + cmd.Stdin = strings.NewReader(stdin) + out, err := cmd.Output() + if err != nil { + t.Fatalf("git %v: %v", args, err) + } + return strings.TrimSpace(string(out)) +} + +// commitText is the raw text of a commit with the given tree and parents. +func commitText(tree string, parents ...string) string { + var b strings.Builder + b.WriteString("tree " + tree + "\n") + for _, p := range parents { + b.WriteString("parent " + p + "\n") + } + b.WriteString("author a 1 +0000\ncommitter a 1 +0000\n\nplanted\n") + return b.String() +} + +// registerLooseObjects packs exactly the listed objects of dir (no walk), uploads +// the pack and registers it for the repository, as a push that carries objects +// that its refs do not reach would. +func registerLooseObjects(t *testing.T, e *testEnv, repoID, dir string, ids ...string) { + t.Helper() + ctx := context.Background() + packBase := filepath.Join(t.TempDir(), "p") + id := scratchGit(t, dir, strings.Join(ids, "\n")+"\n", "pack-objects", "--threads=1", "-q", packBase) + key := blob.PackKey(repoID, id) + if err := e.blobStore.UploadFile(ctx, key, packBase+"-"+id+".pack"); err != nil { + t.Fatalf("upload pack %s: %v", key, err) + } + if err := e.refStore.RegisterPacks(ctx, repoID, []ref.Pack{{ID: id, Key: key}}); err != nil { + t.Fatalf("register pack %s: %v", id, err) + } +} + +// plantHistory registers a pack with a commit, and returns the commit, for each +// way that the pack can lack part of what the commit names. c2's tree and +// commits are in the repository's first pack. +func plantHistory(t *testing.T, e *testEnv, fx wrefFixture, shape string) string { + t.Helper() + dir := t.TempDir() + scratchGit(t, dir, "", "init", "-q", "-b", "main", ".") + hash := func(kind, text string) string { + return scratchGit(t, dir, text, "hash-object", "-t", kind, "-w", "--stdin", "--literally") + } + tree := func(entries string) string { + return scratchGit(t, dir, entries, "mktree", "--missing") + } + switch shape { + case "parent": + // The tree is the empty tree: it is the commit's own parent that is absent. + empty := tree("") + c := hash("commit", commitText(empty, absentID)) + registerLooseObjects(t, e, fx.repoID, dir, c, empty) + return c + case "grandparent": + empty := tree("") + p := hash("commit", commitText(empty, absentID)) + c := hash("commit", commitText(empty, p)) + registerLooseObjects(t, e, fx.repoID, dir, c, p, empty) + return c + case "tree": + c := hash("commit", commitText(absentID, fx.c2)) + registerLooseObjects(t, e, fx.repoID, dir, c) + return c + case "blob": + tr := tree("100644 blob " + absentID + "\tf.txt\n") + c := hash("commit", commitText(tr, fx.c2)) + registerLooseObjects(t, e, fx.repoID, dir, c, tr) + return c + case "complete": + // A commit that no ref reaches, with everything that it names in the pack. + blobID := hash("blob", "complete\n") + tr := tree("100644 blob " + blobID + "\tf.txt\n") + c := hash("commit", commitText(tr, fx.c2)) + registerLooseObjects(t, e, fx.repoID, dir, c, tr, blobID) + return c + } + t.Fatalf("unknown shape %q", shape) + return "" +} + +func TestRefWritesRefuseACommitWhoseHistoryThePacksLack(t *testing.T) { + runBackends(t, func(t *testing.T, e *testEnv) { + for _, history := range []string{"parent", "grandparent", "tree", "blob"} { + for _, tc := range refWriteShapes() { + t.Run(history+"/"+tc.name, func(t *testing.T) { + fx := wrefSeedFixture(t, e) + planted := plantHistory(t, e, fx, history) + logs := &lockedLog{} + srv := newWrefServerLogging(t, e, slog.New(slog.NewTextHandler(logs, nil))) + before := wrefRefs(t, e, fx.repoID) + + resp, body := wrefDo(t, srv, http.MethodPost, tc.path, fx.token, tc.body(planted)) + if resp.StatusCode != http.StatusNotFound { + t.Fatalf("a commit whose %s the packs lack: status %d, want 404, body %v", history, resp.StatusCode, body) + } + // The operator hears of the refusal: one line per refused commit, with the repository and the commit. + if got := logLines(logs, refusedWalkLine, "level=WARN", "repo="+fx.repoID, "sha="+planted); len(got) == 0 { + t.Fatalf("no %q line for %s in the log:\n%s", refusedWalkLine, planted, logs.String()) + } + after := wrefRefs(t, e, fx.repoID) + if len(after) != len(before) || after[tc.ref] != before[tc.ref] { + t.Fatalf("refs changed after the refused request: before %v, after %v", before, after) + } + + // A commit that the packs hold completely still works. + refused := len(logLines(logs, refusedWalkLine)) + resp, body = wrefDo(t, srv, http.MethodPost, tc.path, fx.token, tc.body(fx.c1)) + if resp.StatusCode != http.StatusOK { + t.Fatalf("a commit that the packs hold: status %d, want 200, body %v", resp.StatusCode, body) + } + if got := len(logLines(logs, refusedWalkLine)); got != refused { + t.Fatalf("a commit that the packs hold added %d %q lines", got-refused, refusedWalkLine) + } + }) + } + } + }) +} + +// A commit that no ref reaches is no reason for a refusal when the packs hold +// everything that it names, and a commit that a ref reaches costs no walk. +func TestRefWritesAcceptACommitWhoseHistoryThePacksHold(t *testing.T) { + runBackends(t, func(t *testing.T, e *testEnv) { + fx := wrefSeedFixture(t, e) + complete := plantHistory(t, e, fx, "complete") + srv := newWrefServer(t, e) + for i, tc := range []struct{ name, sha string }{{"unreferenced and complete", complete}, {"the root", fx.c1}, {"the tip", fx.c2}} { + t.Run(tc.name, func(t *testing.T) { + resp, body := wrefDo(t, srv, http.MethodPost, "/api/v1/repos/branches/create", fx.token, + map[string]any{"target_branch": "ok" + strconv.Itoa(i), "base_ref": tc.sha}) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status %d, want 200, body %v", resp.StatusCode, body) + } + if body["commit_sha"] != tc.sha { + t.Fatalf("body %v, want commit_sha %s", body, tc.sha) + } + }) + } + // Every other node hydrates the repository and reads the history. + repoPath, err := gitread.Materialize(context.Background(), e.refStore, e.blobStore, t.TempDir(), fx.repoID, "main") + if err != nil { + t.Fatal(err) + } + scratchGit(t, repoPath, "", "rev-list", "--objects", "--all") + }) +} + +// The walk is git work that a writer starts, so it runs under the admission +// slots, the time budget and the heap cap of the git:read routes. With every +// slot of the repository taken, a create answers 503 with Retry-After and +// writes nothing, and it frees its own slot when it is done. +func TestRefWritesWalkUnderTheGitAdmissionSlots(t *testing.T) { + runBackends(t, func(t *testing.T, e *testEnv) { + for _, tc := range refWriteShapes() { + t.Run(tc.name, func(t *testing.T) { + fx := wrefSeedFixture(t, e) + p := withGitReadPolicy(t, testGitReadPolicy(1, time.Minute, gitread.DefaultOutputBytes, gitread.DefaultLimits())) + srv := newWrefServer(t, e) + before := wrefRefs(t, e, fx.repoID) + + // c2 is the tip of main: a base that every shape has to walk to. + key := gitlimit.RepoKey(fx.repoID) + if err := p.slots.Acquire(t.Context(), key); err != nil { + t.Fatal(err) + } + resp, body := wrefDo(t, srv, http.MethodPost, tc.path, fx.token, tc.body(fx.c2)) + if resp.StatusCode != http.StatusServiceUnavailable || resp.Header.Get("Retry-After") == "" { + t.Fatalf("with no free slot: status %d, Retry-After %q, body %v, want 503 with Retry-After", resp.StatusCode, resp.Header.Get("Retry-After"), body) + } + if after := wrefRefs(t, e, fx.repoID); len(after) != len(before) || after[tc.ref] != before[tc.ref] { + t.Fatalf("refs changed while no slot was free: before %v, after %v", before, after) + } + p.slots.Release(key) + + resp, body = wrefDo(t, srv, http.MethodPost, tc.path, fx.token, tc.body(fx.c2)) + if resp.StatusCode != http.StatusOK { + t.Fatalf("with a free slot: status %d, body %v, want 200", resp.StatusCode, body) + } + requireSlotsFree(t, p) + }) + } + }) +} + +// A create that names two bases walks both under the budget of the request: one +// slot, and one time budget that the second walk draws on. A fake git makes each +// walk take about a second, and the budget lasts for 1.6 s: the first walk fits +// and the second does not. With a budget for each walk both would fit, and the +// request would take two budgets of git time. +func TestBranchCreateWalksBothBasesUnderOneBudget(t *testing.T) { + realGit, err := exec.LookPath("git") + if err != nil { + t.Fatal(err) + } + runBackends(t, func(t *testing.T, e *testEnv) { + fx := wrefSeedFixture(t, e) + first := plantHistory(t, e, fx, "parent") + second := plantHistory(t, e, fx, "tree") + srv := newWrefServer(t, e) + before := wrefRefs(t, e, fx.repoID) + + bin := t.TempDir() + script := "#!/bin/sh\nfor a in \"$@\"; do [ \"$a\" = rev-list ] && sleep 1; done\nexec " + realGit + " \"$@\"\n" + if err := os.WriteFile(filepath.Join(bin, "git"), []byte(script), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) + p := withGitReadPolicy(t, testGitReadPolicy(1, 1600*time.Millisecond, gitread.DefaultOutputBytes, gitread.DefaultLimits())) + + resp, body := wrefDo(t, srv, http.MethodPost, "/api/v1/repos/branches/create", fx.token, + map[string]any{"target_branch": "both", "base_ref": first, "preferred_base": second}) + if resp.StatusCode != http.StatusServiceUnavailable { + t.Fatalf("two bases of a second each under a budget of 1.6 s: status %d, body %v, want 503 (the time budget of the request)", resp.StatusCode, body) + } + if after := wrefRefs(t, e, fx.repoID); len(after) != len(before) { + t.Fatalf("refs changed after the request that ran out of time: before %v, after %v", before, after) + } + requireSlotsFree(t, p) + }) +} + +// slotsAtCAS records how many git read slots are held at each ref write. +type slotsAtCAS struct { + ref.Store + p *gitReadPolicy + + mu sync.Mutex + inUse []int +} + +func (s *slotsAtCAS) CASRef(ctx context.Context, repo, name, old, new string) (bool, error) { + s.mu.Lock() + s.inUse = append(s.inUse, s.p.slots.InUse()) + s.mu.Unlock() + return s.Store.CASRef(ctx, repo, name, old, new) +} + +// A walk holds its slot while git runs and no longer: a ref write waits for a +// concurrent push's hold on its ref for up to ten seconds, and the process has +// eight slots for every git read. +func TestRefWritesFreeTheirSlotBeforeTheRefIsWritten(t *testing.T) { + runBackends(t, func(t *testing.T, e *testEnv) { + type shape struct { + name, path string + body func(fx wrefFixture, planted string) map[string]any + } + shapes := []shape{ + {"a tag", "/api/v1/repos/tags", func(fx wrefFixture, _ string) map[string]any { + return map[string]any{"name": "slot-tag", "ref": fx.c2} + }}, + {"a branch", "/api/v1/repos/branches/create", func(fx wrefFixture, _ string) map[string]any { + return map[string]any{"target_branch": "slot-branch", "base_ref": fx.c2} + }}, + {"an allocated branch name", "/api/v1/repos/branches/create", func(fx wrefFixture, _ string) map[string]any { + return map[string]any{"base_ref": fx.c2} + }}, + {"a branch after a refused base_ref", "/api/v1/repos/branches/create", func(fx wrefFixture, planted string) map[string]any { + return map[string]any{"target_branch": "slot-fallback", "base_ref": planted, "preferred_base": fx.c2} + }}, + } + for _, tc := range shapes { + t.Run(tc.name, func(t *testing.T) { + fx := wrefSeedFixture(t, e) + planted := plantHistory(t, e, fx, "parent") + p := withGitReadPolicy(t, testGitReadPolicy(1, time.Minute, gitread.DefaultOutputBytes, gitread.DefaultLimits())) + probe := &slotsAtCAS{Store: e.refStore, p: p} + env := *e + env.refStore = probe + srv := newWrefServer(t, &env) + + resp, body := wrefDo(t, srv, http.MethodPost, tc.path, fx.token, tc.body(fx, planted)) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status %d, body %v, want 200", resp.StatusCode, body) + } + probe.mu.Lock() + defer probe.mu.Unlock() + if len(probe.inUse) == 0 { + t.Fatal("the request wrote no ref") + } + for _, n := range probe.inUse { + if n != 0 { + t.Fatalf("%d git read slots held while the ref was written, want 0 (slots held at each write: %v)", n, probe.inUse) + } + } + requireSlotsFree(t, p) + }) + } + }) +} + +// panicsWhileASlotIsHeld panics at the first registry read that happens while a +// git read slot is held, as a bug in the walk would. A walk reads the registry +// again after git answered "missing", so a commit that the packs lack reaches it. +type panicsWhileASlotIsHeld struct { + ref.Store + p *gitReadPolicy + armed atomic.Bool +} + +func (s *panicsWhileASlotIsHeld) ListPacks(ctx context.Context, repo string) ([]ref.Pack, error) { + if s.armed.Load() && s.p.slots.InUse() > 0 { + panic("a bug in the walk") + } + return s.Store.ListPacks(ctx, repo) +} + +// A panic between the first git child and the end of the check is what the +// recovery middleware is for. It must not take the slot with it: the process has +// eight, and two lost slots of one repository end every git read of it. +func TestRefWritesFreeTheirSlotAfterAPanic(t *testing.T) { + runBackends(t, func(t *testing.T, e *testEnv) { + for _, tc := range refWriteShapes() { + t.Run(tc.name, func(t *testing.T) { + fx := wrefSeedFixture(t, e) + // A commit that the packs lack: only a "missing" answer reads the + // registry again, and it does so while the walk holds its slot. + planted := plantHistory(t, e, fx, "parent") + p := withGitReadPolicy(t, testGitReadPolicy(1, time.Minute, gitread.DefaultOutputBytes, gitread.DefaultLimits())) + probe := &panicsWhileASlotIsHeld{Store: e.refStore, p: p} + env := *e + env.refStore = probe + srv := newWrefServer(t, &env) + probe.armed.Store(true) + resp, body := wrefDo(t, srv, http.MethodPost, tc.path, fx.token, tc.body(planted)) + probe.armed.Store(false) + if resp.StatusCode != http.StatusInternalServerError { + t.Fatalf("status %d, body %v, want the 500 of a panic", resp.StatusCode, body) + } + requireSlotsFree(t, p) + }) + } + }) +} diff --git a/internal/api/registry_target_test.go b/internal/api/registry_target_test.go index 50a58708..0d112812 100644 --- a/internal/api/registry_target_test.go +++ b/internal/api/registry_target_test.go @@ -47,13 +47,18 @@ func plantCacheOnlyCommit(t *testing.T, repoPath, parent string) string { return run("", "commit-tree", "-p", parent, "-m", "cache only", tree) } -func TestRefWritesRefuseACommitThatOnlyTheCacheHolds(t *testing.T) { - type shape struct { - name, path string - body func(sha string) map[string]any - ref string // the ref that the request would create - } - shapes := []shape{ +// refWriteShape is one request that stores a commit of the body in a ref. +type refWriteShape struct { + name, path string + body func(sha string) map[string]any + ref string // the ref that the request would create +} + +// refWriteShapes lists the requests that store a commit that the request names +// in a ref: the base of a branch (named, as a fallback, forced, ephemeral) and +// the target of a tag. +func refWriteShapes() []refWriteShape { + return []refWriteShape{ {"branch base_ref", "/api/v1/repos/branches/create", func(sha string) map[string]any { return map[string]any{"target_branch": "co", "base_ref": sha} }, "refs/heads/co"}, @@ -70,6 +75,10 @@ func TestRefWritesRefuseACommitThatOnlyTheCacheHolds(t *testing.T) { return map[string]any{"name": "co", "ref": sha} }, "refs/tags/co"}, } +} + +func TestRefWritesRefuseACommitThatOnlyTheCacheHolds(t *testing.T) { + shapes := refWriteShapes() runBackends(t, func(t *testing.T, e *testEnv) { for _, tc := range shapes { t.Run(tc.name, func(t *testing.T) { @@ -121,8 +130,9 @@ func (c *countsPackListings) ListPacks(ctx context.Context, repo string) ([]ref. // The registry is read only for a request that is about to write a ref. A // writer with a token can send the same request again and again, and a request -// that the ref policy denies, that names a tag that exists, or that leaves a -// branch where it is, must not cost a registry read and a view of the cache. +// that the ref policy denies, that names a tag or a branch that exists, or that +// leaves a branch where it is, must not cost a registry read, a view of the +// cache and a walk of its history. func TestRefWritesReadTheRegistryOnlyBeforeAWrite(t *testing.T) { runBackends(t, func(t *testing.T, e *testEnv) { fx := wrefSeedFixture(t, e) @@ -146,6 +156,7 @@ func TestRefWritesReadTheRegistryOnlyBeforeAWrite(t *testing.T) { {"policy denies a tag", "/api/v1/repos/tags", denyAll, map[string]any{"name": "denied", "ref": fx.c1}, http.StatusForbidden}, {"the tag exists", "/api/v1/repos/tags", fx.token, map[string]any{"name": "annot", "ref": fx.c1}, http.StatusConflict}, {"the branch is already at the base", "/api/v1/repos/branches/create", fx.token, map[string]any{"target_branch": "feature", "base_ref": fx.c1, "force": true}, http.StatusOK}, + {"a plain create of a branch that exists", "/api/v1/repos/branches/create", fx.token, map[string]any{"target_branch": "feature", "base_ref": fx.c2}, http.StatusConflict}, } { before := counter.n.Load() resp, body := wrefDo(t, srv, http.MethodPost, tc.path, tc.token, tc.body) diff --git a/internal/api/write_merge_source_pin_test.go b/internal/api/write_merge_source_pin_test.go index b3f7bb8c..e9c3ea12 100644 --- a/internal/api/write_merge_source_pin_test.go +++ b/internal/api/write_merge_source_pin_test.go @@ -7,6 +7,11 @@ import ( "reflect" "strings" "testing" + "time" + + "github.com/coder/openstorage/internal/gitlimit" + "github.com/coder/openstorage/internal/gitread" + "github.com/coder/openstorage/internal/ref" ) // TestMergeExpectedSourceSHA pins the source commit with @@ -147,3 +152,107 @@ func TestMergeExpectedSourceSHA(t *testing.T) { }) }) } + +// The source branch can reach a pin that no registered pack holds: a push can +// leave a commit behind a ref tip while the node's cache holds it. A target ref +// or a merge commit at that pin would wedge every other node, which hydrates +// from the registered packs alone, so the pin answers like a commit that the +// source branch does not contain. The tip of the source branch is a ref already +// and needs no check. +func TestMergeExpectedSourceSHARefusesAPinThatThePacksLack(t *testing.T) { + runBackends(t, func(t *testing.T, e *testEnv) { + srv := wmServer(t, e) + fx := wmSeedFixture(t, e) + ctx := t.Context() + + // pin is in the cache only. plantsrc is a branch whose tip is a child of + // pin, in a registered pack that lacks pin: the push that left it there + // was checked against a cache that held pin. + repoPath, err := gitread.Materialize(ctx, e.refStore, e.blobStore, e.cacheDir, fx.repoID, "main") + if err != nil { + t.Fatal(err) + } + pin := plantCacheOnlyCommit(t, repoPath, fx.c2) + dir := t.TempDir() + scratchGit(t, dir, "", "init", "-q", "-b", "main", ".") + tip := scratchGit(t, dir, commitText(coreRevParse(t, fx.work, fx.c2+"^{tree}"), pin), "hash-object", "-t", "commit", "-w", "--stdin", "--literally") + registerLooseObjects(t, e, fx.repoID, dir, tip) + for name, sha := range map[string]string{"plantsrc": tip, "t-plant-ff": fx.c2, "t-plant-merge": fx.c2, "t-plant-ok": fx.c2} { + if won, err := e.refStore.CASRef(ctx, fx.repoID, "refs/heads/"+name, ref.ZeroSHA, sha); err != nil || !won { + t.Fatalf("create %s: won %v, err %v", name, won, err) + } + } + + for _, tc := range []struct{ target, strategy string }{{"t-plant-ff", "ff_only"}, {"t-plant-merge", "merge"}} { + t.Run(tc.strategy, func(t *testing.T) { + before := wmRefsSnapshot(t, e, fx.repoID) + resp, raw := wmDo(t, srv, "POST", "/api/v1/repos/merge", fx.token, + wmMergeBody("plantsrc", tc.target, tc.strategy, map[string]any{"author": wmTestAuthor, "expected_source_sha": pin})) + wmRequireRefGuard(t, resp, raw, "expected_source_sha", tip, + "source branch moved: the source branch does not contain commit "+pin) + if after := wmRefsSnapshot(t, e, fx.repoID); !reflect.DeepEqual(before, after) { + t.Fatalf("refs changed on a refused merge: before %v, after %v", before, after) + } + }) + } + + // The check is git work that the merge starts, so it runs under the same + // admission slots as a create: with every slot taken the merge answers + // 503 with Retry-After (not 500) and writes nothing, and it frees its + // slot when it is done. ffMid is held completely and reached by ffsrc. + t.Run("no free slot", func(t *testing.T) { + p := withGitReadPolicy(t, testGitReadPolicy(1, time.Minute, gitread.DefaultOutputBytes, gitread.DefaultLimits())) + if won, err := e.refStore.CASRef(ctx, fx.repoID, "refs/heads/t-plant-busy", ref.ZeroSHA, fx.c2); err != nil || !won { + t.Fatalf("create t-plant-busy: won %v, err %v", won, err) + } + body := wmMergeBody("ffsrc", "t-plant-busy", "ff_only", map[string]any{"expected_source_sha": fx.ffMid}) + before := wmRefsSnapshot(t, e, fx.repoID) + key := gitlimit.RepoKey(fx.repoID) + if err := p.slots.Acquire(ctx, key); err != nil { + t.Fatal(err) + } + resp, raw := wmDo(t, srv, "POST", "/api/v1/repos/merge", fx.token, body) + if resp.StatusCode != http.StatusServiceUnavailable || resp.Header.Get("Retry-After") == "" { + t.Fatalf("with no free slot: status %d, Retry-After %q, body %s, want 503 with Retry-After", resp.StatusCode, resp.Header.Get("Retry-After"), raw) + } + if after := wmRefsSnapshot(t, e, fx.repoID); !reflect.DeepEqual(before, after) { + t.Fatalf("refs changed while no slot was free: before %v, after %v", before, after) + } + p.slots.Release(key) + resp, raw = wmDo(t, srv, "POST", "/api/v1/repos/merge", fx.token, body) + if got := wmDecode(t, raw); resp.StatusCode != http.StatusOK || got["commit_sha"] != fx.ffMid { + t.Fatalf("with a free slot: status %d body %s, want fast_forward to %s", resp.StatusCode, raw, fx.ffMid) + } + requireSlotsFree(t, p) + }) + + // A panic after the first git child must not take the slot with it. + t.Run("a panic after the walk", func(t *testing.T) { + p := withGitReadPolicy(t, testGitReadPolicy(1, time.Minute, gitread.DefaultOutputBytes, gitread.DefaultLimits())) + if won, err := e.refStore.CASRef(ctx, fx.repoID, "refs/heads/t-plant-panic", ref.ZeroSHA, fx.c2); err != nil || !won { + t.Fatalf("create t-plant-panic: won %v, err %v", won, err) + } + probe := &panicsWhileASlotIsHeld{Store: e.refStore, p: p} + env := *e + env.refStore = probe + panicSrv := wmServer(t, &env) + probe.armed.Store(true) + resp, raw := wmDo(t, panicSrv, "POST", "/api/v1/repos/merge", fx.token, + wmMergeBody("plantsrc", "t-plant-panic", "ff_only", map[string]any{"expected_source_sha": pin})) + probe.armed.Store(false) + if resp.StatusCode != http.StatusInternalServerError { + t.Fatalf("status %d, body %s, want the 500 of a panic", resp.StatusCode, raw) + } + requireSlotsFree(t, p) + }) + + // The tip itself is a ref already: pinning it is no walk and no refusal. + t.Run("the tip", func(t *testing.T) { + resp, raw := wmDo(t, srv, "POST", "/api/v1/repos/merge", fx.token, + wmMergeBody("plantsrc", "t-plant-ok", "ff_only", map[string]any{"expected_source_sha": tip})) + if body := wmDecode(t, raw); resp.StatusCode != http.StatusOK || body["commit_sha"] != tip { + t.Fatalf("status %d body %s, want fast_forward to the tip %s", resp.StatusCode, raw, tip) + } + }) + }) +} diff --git a/internal/api/write_refs_test.go b/internal/api/write_refs_test.go index 6428e2a0..785aa3cc 100644 --- a/internal/api/write_refs_test.go +++ b/internal/api/write_refs_test.go @@ -18,6 +18,7 @@ import ( "encoding/json" "fmt" "io" + "log/slog" "maps" "net/http" "net/http/httptest" @@ -34,11 +35,17 @@ import ( // newWrefServer builds an httptest server exposing only the W3 write routes // with the production auth middleware, backed by the lane's stores. func newWrefServer(t *testing.T, e *testEnv) *httptest.Server { + t.Helper() + return newWrefServerLogging(t, e, discardLogger()) +} + +// newWrefServerLogging is newWrefServer with the server's log lines going to logger. +func newWrefServerLogging(t *testing.T, e *testEnv, logger *slog.Logger) *httptest.Server { t.Helper() s := &server{ store: e.store, httpBaseURL: testHTTPBaseURL, - logger: discardLogger(), + logger: logger, refStore: e.refStore, blobStore: e.blobStore, cacheDir: e.cacheDir, diff --git a/internal/gitread/gitread.go b/internal/gitread/gitread.go index eb3b037a..8b34f4cb 100644 --- a/internal/gitread/gitread.go +++ b/internal/gitread/gitread.go @@ -22,6 +22,7 @@ import ( "os/exec" "regexp" "strings" + "time" "github.com/coder/openstorage/internal/auth" "github.com/coder/openstorage/internal/blob" @@ -299,3 +300,104 @@ func RegisteredKinds(ctx context.Context, st ref.RefStore, bl blob.BlobStore, re return githttp.ParseObjectKinds(ids, out) }) } + +// registeredWalkTimeout bounds the history walk of RegisteredClosure. The walk +// reads the commits between sha and the commits that refs reach: a base that a +// ref reaches ends it at once, and the oldest commit of 500,000 linear commits +// costs about two seconds without a commit-graph. A write endpoint has no +// request budget, so the walk carries its own. +var registeredWalkTimeout = 30 * time.Second + +// missingObjectScan reads the output of `git rev-list --missing=print --quiet`, +// which holds one "?" line for each object that the walk could not read and +// nothing else. The first byte decides: git is stopped there, because one +// missing object answers the question. +type missingObjectScan struct { + missing bool + err error +} + +func (m *missingObjectScan) Write(p []byte) (int, error) { + if len(p) == 0 { + return 0, nil + } + if p[0] != '?' { + m.err = errors.New("rev-list --missing=print --quiet printed something other than a missing object") + return 0, ErrStop + } + m.missing = true + return 0, ErrStop +} + +// RegisteredClosure reports whether the registered packs hold what git needs to +// read sha and everything that sha names that no ref of the node's cache +// reaches: the commits down to the ones that a ref reaches, and their trees and +// blobs. A ref may be created at sha only then. A pack can hold a commit whose +// parent no registered pack holds (a push can carry commits that its refs do +// not reach, and git checks the commits that the refs reach alone), and a ref +// at such a commit lets git read the ref, yet every clone that walks the +// history fails. +// +// The walk is git's own connectivity check, the one that receive-pack runs for +// the new tips of a push: `rev-list --objects --not --all` over a view of the +// registered packs (githttp.RegisteredPackView). The answer rests on which +// objects git can read, never on git's error text. A cat-file first asks for +// sha itself, because git before 2.45 dies on a start object that it cannot +// read, whatever --missing says. Then --missing=print makes git print the id of +// each object that it cannot read and exit 0. git before 2.43 also dies on a +// missing parent commit, and git 2.43 dies on a ref at a missing object: exit +// status 128 means that git could not walk the history, so it is the same +// answer, as it is for the ancestry check of a push (isAncestorUncached). Any +// other outcome of git is an error. The view has no commit-graph on purpose, +// because a commit-graph can describe commits that no registered pack holds and +// git then walks past them. So the cost is the same with and without a +// commit-graph in the cache. +// +// A registered pack that the cache lacks makes a "missing" answer an error that +// wraps githttp.ErrViewIncomplete, as it does for RegisteredKinds. A walk that +// outlasts registeredWalkTimeout, or the request budget in ctx, is an error that +// wraps ErrTimeBudget. +func RegisteredClosure(ctx context.Context, st ref.RefStore, bl blob.BlobStore, repoPath, repo, sha string) (bool, error) { + assertf(ref.ValidSHA(sha), "RegisteredClosure: malformed sha %q", sha) + kinds, err := githttp.AskRegisteredPacks(ctx, st, repoPath, repo, func(ctx context.Context) error { + return githttp.HydratePacks(ctx, st, bl, repoPath, repo) + }, func(view *githttp.RegisteredPackView) (map[string]string, error) { + walkCtx, cancel := context.WithTimeout(ctx, registeredWalkTimeout) + defer cancel() + env := view.Env(os.Environ()) + out, err := Run(walkCtx, Cmd{Class: Light, Dir: repoPath, Env: env, Stdin: strings.NewReader(sha + "\n"), + Args: []string{"cat-file", "--batch-check=%(objecttype)"}}) + if err == nil { + var start map[string]string + if start, err = githttp.ParseObjectKinds([]string{sha}, out); err == nil && start[sha] == "missing" { + return start, nil + } + } + if err == nil { + scan := &missingObjectScan{} + _, err = Run(walkCtx, Cmd{Class: Heavy, Dir: repoPath, Env: env, Stdout: scan, + Args: []string{"-c", "core.commitGraph=false", "rev-list", "--objects", "--missing=print", "--quiet", sha, "--not", "--all"}}) + switch { + case err == nil && scan.err != nil: + err = scan.err + case err == nil && scan.missing: + return map[string]string{sha: "missing"}, nil + case err == nil: + return map[string]string{sha: "commit"}, nil + } + } + switch { + case ctx.Err() == nil && errors.Is(walkCtx.Err(), context.DeadlineExceeded): + return nil, fmt.Errorf("walk the history of %s: %w", sha, ErrTimeBudget) + default: + if _, unreadable := gitStderr(err, 128); unreadable { + return map[string]string{sha: "missing"}, nil + } + } + return nil, err + }) + if err != nil { + return false, err + } + return kinds[sha] != "missing", nil +} diff --git a/internal/gitread/registered_closure_test.go b/internal/gitread/registered_closure_test.go new file mode 100644 index 00000000..1ec6d819 --- /dev/null +++ b/internal/gitread/registered_closure_test.go @@ -0,0 +1,204 @@ +package gitread + +import ( + "context" + "errors" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/coder/openstorage/internal/ref" +) + +// RegisteredClosure decides from the objects that git can read, which +// `rev-list --missing=print` prints as "?" lines, and from the exit status +// 128 with which older git dies on a commit that it cannot read, and from +// nothing else: any other outcome of git is an error, never a refusal. A +// stand-in git shows each outcome. +func TestRegisteredClosureAnswersFromTheMissingObjectsAlone(t *testing.T) { + sha := strings.Repeat("a", 40) + st, err := ref.Open(t.Context(), "sqlite://"+filepath.Join(t.TempDir(), "refs.sqlite")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { st.Close() }) + if err := st.InitSchema(t.Context()); err != nil { + t.Fatal(err) + } + repoPath := filepath.Join(t.TempDir(), "r.git") + if err := os.MkdirAll(filepath.Join(repoPath, "objects", "pack"), 0o755); err != nil { + t.Fatal(err) + } + for _, tc := range []struct { + name, script string + want bool + wantErr string // text of the error, "" for none + }{ + {"nothing missing", "exit 0", true, ""}, + {"an object is missing", "echo '?" + sha + "'; exit 0", false, ""}, + {"a missing object and a failed walk", "echo '?" + sha + "'; exit 128", false, ""}, + {"output that is no missing object", "echo 'deadbeef'; exit 0", false, "something other than a missing object"}, + {"git cannot walk the history (exit 128)", "echo 'fatal: ...' >&2; exit 128", false, ""}, + {"git fails another way (exit 129)", "echo 'usage: git rev-list' >&2; exit 129", false, "exit status 129"}, + {"git is killed", "kill -9 $$", false, "killed"}, + } { + t.Run(tc.name, func(t *testing.T) { + bin := t.TempDir() + // rev-list is the walk. The stand-in answers the other git commands with exit 0. + script := "#!/bin/sh\ncase \"$*\" in *cat-file*) echo commit ;; *rev-list*) " + tc.script + " ;; esac\nexit 0\n" + if err := os.WriteFile(filepath.Join(bin, "git"), []byte(script), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) + got, err := RegisteredClosure(t.Context(), st, nil, repoPath, "repo_x", sha) + if tc.wantErr == "" && err != nil || tc.wantErr != "" && (err == nil || !strings.Contains(err.Error(), tc.wantErr)) { + t.Fatalf("RegisteredClosure error = %v, want %q", err, tc.wantErr) + } + if err == nil && got != tc.want { + t.Fatalf("RegisteredClosure = %v, want %v", got, tc.want) + } + if tc.name == "a missing object and a failed walk" && err != nil { + t.Fatalf("a missing object was printed before the walk failed: %v", err) + } + }) + } +} + +// A walk that outlasts its budget is ErrTimeBudget, which the handlers answer +// with 503, and never a refusal. A walk that the caller gave up is the caller's +// own error. +func TestRegisteredClosureWalkBudget(t *testing.T) { + sha := strings.Repeat("a", 40) + st, err := ref.Open(t.Context(), "sqlite://"+filepath.Join(t.TempDir(), "refs.sqlite")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { st.Close() }) + if err := st.InitSchema(t.Context()); err != nil { + t.Fatal(err) + } + repoPath := filepath.Join(t.TempDir(), "r.git") + if err := os.MkdirAll(filepath.Join(repoPath, "objects", "pack"), 0o755); err != nil { + t.Fatal(err) + } + bin := t.TempDir() + if err := os.WriteFile(filepath.Join(bin, "git"), []byte("#!/bin/sh\ncase \"$*\" in *cat-file*) echo commit ;; *rev-list*) sleep 30 ;; esac\nexit 0\n"), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) + prev := registeredWalkTimeout + registeredWalkTimeout = 200 * time.Millisecond + t.Cleanup(func() { registeredWalkTimeout = prev }) + + start := time.Now() + if _, err := RegisteredClosure(t.Context(), st, nil, repoPath, "repo_x", sha); !errors.Is(err, ErrTimeBudget) { + t.Fatalf("a walk past its budget: %v, want ErrTimeBudget", err) + } + if d := time.Since(start); d > 10*time.Second { + t.Fatalf("the walk was not stopped at its budget: %v", d) + } + ctx, cancel := context.WithCancel(t.Context()) + cancel() + if _, err := RegisteredClosure(ctx, st, nil, repoPath, "repo_x", sha); errors.Is(err, ErrTimeBudget) || err == nil { + t.Fatalf("a caller that gave up: %v, want its own error, not ErrTimeBudget", err) + } +} + +// A walk that is left to its own budget must end within the time that the +// documentation promises (docs/reference/http-api.md, "Which commit the base and +// the tag target can name"). +func TestRegisteredWalkTimeoutIsThirtySeconds(t *testing.T) { + if registeredWalkTimeout != 30*time.Second { + t.Fatalf("registeredWalkTimeout = %v, want 30s as documented", registeredWalkTimeout) + } +} + +// The walk is git's connectivity check, over the registered packs alone and +// with no commit-graph: a commit-graph can describe a commit that no registered +// pack holds, and git then walks past it. +func TestRegisteredClosureRunsGitsConnectivityCheckOverTheView(t *testing.T) { + sha := strings.Repeat("b", 40) + st, err := ref.Open(t.Context(), "sqlite://"+filepath.Join(t.TempDir(), "refs.sqlite")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { st.Close() }) + if err := st.InitSchema(t.Context()); err != nil { + t.Fatal(err) + } + repoPath := filepath.Join(t.TempDir(), "r.git") + if err := os.MkdirAll(filepath.Join(repoPath, "objects", "pack"), 0o755); err != nil { + t.Fatal(err) + } + bin := t.TempDir() + log := filepath.Join(bin, "log") + script := "#!/bin/sh\ncase \"$*\" in *cat-file*) echo commit ;; *rev-list*) echo \"$*\" >> " + log + "\n" + + "echo \"GIT_DIR=$GIT_DIR\" >> " + log + "\necho \"OBJECTS=$GIT_OBJECT_DIRECTORY\" >> " + log + "\n" + + "echo \"ALT=$GIT_ALTERNATE_OBJECT_DIRECTORIES\" >> " + log + " ;; esac\nexit 0\n" + if err := os.WriteFile(filepath.Join(bin, "git"), []byte(script), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) + t.Setenv("GIT_ALTERNATE_OBJECT_DIRECTORIES", "/elsewhere") + if ok, err := RegisteredClosure(t.Context(), st, nil, repoPath, "repo_x", sha); err != nil || !ok { + t.Fatalf("RegisteredClosure = %v, %v, want true", ok, err) + } + b, err := os.ReadFile(log) + if err != nil { + t.Fatal(err) + } + lines := strings.Split(strings.TrimSpace(string(b)), "\n") + if len(lines) != 4 { + t.Fatalf("the stand-in logged %q, want the walk once", lines) + } + if want := "-c core.commitGraph=false rev-list --objects --missing=print --quiet " + sha + " --not --all"; lines[0] != want { + t.Fatalf("git ran %q, want %q", lines[0], want) + } + if lines[1] != "GIT_DIR="+repoPath { + t.Fatalf("the walk reads the refs of %q, want %q", lines[1], repoPath) + } + if objects := strings.TrimPrefix(lines[2], "OBJECTS="); objects == "" || objects == filepath.Join(repoPath, "objects") { + t.Fatalf("the walk reads objects from %q, want a view of the registered packs, not the cache's own", objects) + } + if lines[3] != "ALT=" { + t.Fatalf("the walk keeps the alternates %q, want none", lines[3]) + } +} + +// git before 2.45 dies on a start object that it cannot read, whatever +// --missing says, so the walk asks for the object first: an object that no +// registered pack holds is missing without a walk. +func TestRegisteredClosureAsksForTheObjectBeforeTheWalk(t *testing.T) { + sha := strings.Repeat("c", 40) + st, err := ref.Open(t.Context(), "sqlite://"+filepath.Join(t.TempDir(), "refs.sqlite")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { st.Close() }) + if err := st.InitSchema(t.Context()); err != nil { + t.Fatal(err) + } + repoPath := filepath.Join(t.TempDir(), "r.git") + if err := os.MkdirAll(filepath.Join(repoPath, "objects", "pack"), 0o755); err != nil { + t.Fatal(err) + } + bin := t.TempDir() + log := filepath.Join(bin, "log") + script := "#!/bin/sh\necho \"$1\" >> " + log + "\ncase \"$*\" in *cat-file*) cat >/dev/null; echo '" + sha + " missing' ;; *rev-list*) echo 'fatal: bad object' >&2; exit 128 ;; esac\nexit 0\n" + if err := os.WriteFile(filepath.Join(bin, "git"), []byte(script), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) + if ok, err := RegisteredClosure(t.Context(), st, nil, repoPath, "repo_x", sha); err != nil || ok { + t.Fatalf("RegisteredClosure = %v, %v, want false and no error", ok, err) + } + b, err := os.ReadFile(log) + if err != nil { + t.Fatal(err) + } + if got := strings.Fields(string(b)); len(got) != 1 || got[0] != "cat-file" { + t.Fatalf("git ran %q, want one cat-file and no walk", got) + } +}