Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/how-to/deploy-production.md
Original file line number Diff line number Diff line change
Expand Up @@ -738,6 +738,7 @@ webhook events.
| Postgres availability + advisory-lock waits at boot | The database is the single authority; boot hangs show in `pg_locks` |
| `verify token: key store lookup failed` / `verify token: a registered key cannot be used` log lines | A git, LFS or REST request reached the key store or a registered key, and it failed. The line holds the cause (a cause of more than 300 runes loses its middle). A git or LFS client reads a text that does not hold the cause. For each kind, the git and LFS routes together write at most one line per minute in each `serve` process, the REST routes write at most one more, and a line says how many earlier lines of its kind were held back when some were |
| `receive-pack <repo>: hook: … git cannot walk the history between two commits, so the update is refused` log line | A push named two commits whose history git cannot read, so the hook refused that update: as non-fast-forward under `no-force-push`, and with `is not a readable commit in this repository` under `verify-sig`. The line has the fields `git`, `old` and `new`. A pusher can cause it with a commit whose parent is missing. A `<repo>` that repeats it for ordinary pushes from several clients has a damaged history: check a fresh copy with `git fsck` |
| `a ref write names a commit that the registered packs cannot walk` log line | A `POST /repos/branches/create`, `POST /repos/tags` or `POST /repos/merge` (with `expected_source_sha`) named a commit that no registered pack holds, or whose ancestor, tree or blob no registered pack holds. The line has the fields `repo` and `sha`. A branch create logs one line for each base that it refuses (`base_ref` and `preferred_base` can both be refused). It answers `200` at `preferred_base` when that base is usable and `404` otherwise, a tag answers `404` and a merge answers `409` (`ref_guard_failed`). A pusher can leave such a commit in a pack (git checks only the commits that the refs of a push reach). A `repo` that logs the line for ordinary creates, with different `sha` values that name healthy bases, has a ref at an object that no registered pack holds (compare `openstorage list-packs` with the blob store, and list the ref with `openstorage repair-refs`) or a damaged cache on that node (git exits with status 128 on a corrupt pack, and the check reads that as a history that cannot be walked) |
| `receive-pack <repo>: hook: … a push deletes a ref that its replica does not list, and the ref store holds it, so the update is refused` log line | A client sent `git push <remote> :<ref>` for a ref that its replica's cache does not list, and the ref store holds the ref. The pusher read `<ref> exists at <sha> in the ref store, but this replica does not list it`. The line has the fields `ref`, `sha` and `cause`. One line after a ref was created is normal: the client fetches and pushes again. A `<repo>` that repeats it has a replica whose cache stays behind: look for a `hydrate <repo>:` error on that replica |
| `lane scan: requeued a lane whose work had no job` log line | A sync or push request lost its job and the [lane scan](#background-maintenance-in-a-fleet) enqueued it again. A `repo=` that repeats needs a look: its runs keep crashing |

Expand Down Expand Up @@ -771,6 +772,7 @@ line can be up to 1 MiB.
| Postgres availability and advisory-lock waits (What to monitor) | **log only** | Observe PostgreSQL itself (`pg_locks`) |
| `verify token: …` lines (What to monitor) | **log only** | The two texts shown, at most one line for each kind per minute for the git and LFS routes together, and one more for the REST routes, in each `serve` process |
| `git cannot walk the history between two commits` (What to monitor) | **log only** | A hook line, so metrics do not count it. Fields `git`, `old`, `new` and `cause` |
| `a ref write names a commit that the registered packs cannot walk` (What to monitor) | **log only** | An API line, so metrics do not count it. Fields `repo` and `sha` |
| `a push deletes a ref that its replica does not list` (What to monitor) | **log only** | A hook line, so metrics do not count it. Fields `ref`, `sha` and `cause` |
| `lane scan: requeued a lane whose work had no job` (What to monitor) | **log only** | Fields `org`, `repo` and `lane` |
| S3WAL `… conditional-write contention retries exhausted` (S3WAL signals) | `openstorage_s3wal_cas_contention_exhausted_total` by `scope` | Serve-process mutations only; push-side failures reach the client and hook stderr |
Expand Down
46 changes: 43 additions & 3 deletions docs/reference/http-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -685,7 +685,11 @@ moved on since.

- The source branch must still contain the commit: the commit is the tip or one of its
ancestors. Otherwise the guard fails (`detail` "source branch moved: the source
branch does not contain commit …").
branch does not contain commit …"). The same answer covers a commit that the source
branch reaches but the registered packs cannot walk, the way the [branch and tag
create routes](#which-commit-the-base-and-the-tag-target-can-name) refuse such a commit
(a push can leave one behind the tip of a branch): a pin that is not the tip is checked
like a base, and the server logs the refusal.
- A source branch that does not exist fails the guard too, with `current_sha` 40
zeros. Without `expected_source_sha`, a missing source branch is `404`.
- `source.sha` in the response is the pinned commit.
Expand Down Expand Up @@ -749,10 +753,46 @@ The server checks the registered packs, not the node's cache repository, because
ref at any other object. A commit that only a cache holds answers `404`, like a commit that does
not exist, and a `base_ref` that names no usable commit still falls back to `preferred_base`. The
server reads the registry only for a request that is about to write: after the ref policy, and not
for a tag that exists or for a create that leaves the branch at its tip (with `force` or
`expected_target_sha`). So a request that the policy denies
for a tag that exists, for a plain create of a branch that exists (`409`) or for a create that
leaves the branch at its tip (with `force` or `expected_target_sha`). So a request that the policy denies
answers `403` even when its commit is only in the cache.

The registered packs must also hold what git needs to read the commit's history: its ancestors,
down to the commits that a ref of the repository reaches, and their trees and blobs. A push can
leave a commit in a registered pack whose parent, tree or blob no pack holds, because
`receive-pack` checks the commits that a push's refs reach and no other commit of its pack. A ref
at such a commit would hydrate, and every clone, fetch and archive that walks the history would
fail. A commit whose history the registered packs cannot walk answers `404`, like a commit that
does not exist. The check is the walk that `receive-pack` runs for the new tips of a push (`git
rev-list --objects --not --all`) over the registered packs, without a commit-graph, because a
commit-graph can describe a commit that no registered pack holds.

The cost is the walk from the commits that the refs reach down to the commit, so it grows with the
distance between the two and with the commit dates that the pusher chose. Measured with git 2.55
on 500,000 linear commits: the tip of a branch takes 5 ms, a commit 1,000 back 15 ms, 100,000 back
0.3 s and the root commit 1.6 s. A commit that no ref reaches costs the walk of its own new
commits and trees, and also the walk of the history of the refs down to its commit date (1,000 new
commits dated after the tip took 14 ms, and the same commits dated 1970 took 1.7 s). A commit
whose parent no pack holds always costs the whole history (1.6 s). A separate unreferenced history
costs more, because git also holds its trees: 50,000 commits took 2.4 s, and 100,000 commits need
more than the heap cap. The walk runs under the admission slots, the 30-second time budget and the
256 MiB heap cap (Linux) of the git:read routes ([Resource limits on git
reads](#resource-limits-on-git-reads)): a request that finds no free slot answers `503` with
`Retry-After`, a walk that outlasts its budget answers `503`, and a walk that needs more memory
answers `422`. A create whose `base_ref` is refused walks `preferred_base` under the same slot
and the same time budget. git 2.39 exits with status 128 on a missing parent commit, and git 2.43
on a ref at a missing object, instead of printing the object. The check reads exit status 128 as a
history that cannot be walked, as the ancestry check of a push does. That status is git's answer to
any fatal failure, so a damaged cache (a corrupt pack, for one) answers `404` too, and the log
line names both causes.

The walk trusts the refs of the repository to be complete: it stops at the commits that a ref
reaches and reports no missing object below them. A ref that a push left at a commit whose history
the packs cannot walk (pushes keep that limit) therefore lets a branch or tag at the same commit,
at a commit that it reaches, or at a complete commit built on top of it, pass. The check cannot
tell that ref from a healthy one without the walk of the whole history for every request, which a
repository of 500,000 commits does not fit under the heap cap.

#### Guarded branch create and move

`expected_target_sha` on `POST /repos/branches/create` (both path styles) guards the
Expand Down
19 changes: 19 additions & 0 deletions internal/api/handlers_write_merge.go
Original file line number Diff line number Diff line change
Expand Up @@ -684,6 +684,25 @@ func (s *server) handleMerge(w http.ResponseWriter, r *http.Request) {
writeRefGuardFailed(w, "expected_source_sha", sourceSHA, sourceMovedMsg)
return
}
// The source branch reaches the pin, but the pin can still be a commit
// that only this node's cache holds (a push can leave a commit behind
// a ref tip that no registered pack holds). A ref or a merge commit at
// it would wedge every other node, so it answers like a pin that the
// source branch does not contain. The tip needs no check: a ref is at it.
if req.sourcePin != sourceSHA {
usable, err := s.wrefUsableCommitAlone(ctx, repo.RepoID, repoPath, req.sourcePin)
if err != nil {
const op = "check that the registered packs hold expected_source_sha and its history"
if !s.writeGitLimitErr(w, r, op, err) {
s.wmInternal(w, r, op, err)
}
return
}
if !usable {
writeRefGuardFailed(w, "expected_source_sha", sourceSHA, sourceMovedMsg)
return
}
}
sourceSHA = req.sourcePin
}

Expand Down
103 changes: 78 additions & 25 deletions internal/api/handlers_write_refs.go
Original file line number Diff line number Diff line change
Expand Up @@ -386,24 +386,58 @@ func wrefValidateCreateBranch(req *wrefCreateBranchRequest) (target string, allo
return target, allocate, guard, errs.err()
}

// wrefHeldCommit reports whether a pack that the registry lists for the
// repository holds the commit sha. A write that stores the commit in a ref
// must ask: the node's cache can hold objects that no registered pack holds (a
// cache that was copied or edited out of band, or a registry that was rolled
// back), and a ref at such an object wedges every other node, and this node
// after its cache is gone, because they hydrate from the registered packs
// alone.
func (s *server) wrefHeldCommit(ctx context.Context, repoID, repoPath, sha string) (bool, error) {
kinds, err := gitread.RegisteredKinds(ctx, s.refStore, s.blobStore, repoPath, repoID, []string{sha})
if err != nil {
return false, err
}
return kinds[sha] == "commit", nil
// wrefUsableCommit reports whether the registered packs hold the commit sha and
// what git needs to walk its history: its parents down to the commits that a
// ref reaches, and their trees and blobs (gitread.RegisteredClosure). A write
// that stores the commit in a ref must ask, for two reasons.
//
// The node's cache can hold objects that no registered pack holds (a cache that
// was copied or edited out of band, or a registry that was rolled back), and a
// ref at such an object wedges every other node, and this node after its cache
// is gone, because they hydrate from the registered packs alone.
//
// A pack can also hold a commit whose parent no pack holds. receive-pack checks
// the commits that a push's refs reach and no other commit of its pack, so a
// push can leave such a commit in the registry. A ref at it hydrates, and
// every clone, fetch and archive that walks the history then fails.
//
// The walk runs under the same admission slot, time budget and heap cap as a
// git:read route (gitReads): a write endpoint has no request budget of its own,
// and a writer must not be able to start any number of walks at once. b is the
// budget of the request, so that the walks of one request (a base, then its
// preferred_base) share one slot and one time budget, as the git calls of a
// read route do. The caller closes b as soon as its walks are over, so that the
// slot is not held while the ref is written.
func (s *server) wrefUsableCommit(ctx context.Context, b *gitread.Budget, repoID, repoPath, sha string) (bool, error) {
usable, err := gitread.RegisteredClosure(gitread.WithBudget(ctx, b), s.refStore, s.blobStore, repoPath, repoID, sha)
if err == nil && !usable {
s.logger.Warn("a ref write names a commit that the registered packs cannot walk: a pack lacks the commit or one of its ancestors, trees or blobs, or the cache is damaged",
"repo", repoID, "sha", sha)
}
return usable, err
}

// wrefUsableCommitAlone is wrefUsableCommit for a request that checks one
// commit: the budget is its own, and it is freed when the check ends, whatever
// ends it (a panic included), so that the slot is not held while the ref is
// written.
func (s *server) wrefUsableCommitAlone(ctx context.Context, repoID, repoPath, sha string) (bool, error) {
b := wrefWalkBudget(repoID)
defer b.Close()
return s.wrefUsableCommit(ctx, b, repoID, repoPath, sha)
}

// wrefWalkBudget returns the git budget of one ref write request: the slots,
// time budget and heap cap of a git:read route, drawn for the repository. It
// takes its slot when the first walk starts. The caller closes it.
func wrefWalkBudget(repoID string) *gitread.Budget {
return gitread.NewBudget(gitReads.slots, gitlimit.RepoKey(repoID), gitReads.timeout, gitReads.outputBytes).WithLimits(gitReads.limits)
}

// handleCreateBranch resolves the base to a commit (branch, tag — peeled —
// or hex sha via gitread.ResolveRef; preferred_base is the fallback when
// the primary base is absent or names no ref, and a base whose commit git
// the primary base is absent, names no ref or names a commit whose history
// the registered packs cannot walk, and a base whose commit git
// cannot read is a 500, with no fallback), policy-checks the new ref,
// and CAS creates it. With force, an existing target is CAS-recreated at
// the base — no-force-push policies still deny non-fast-forward moves via a
Expand Down Expand Up @@ -449,14 +483,19 @@ func (s *server) handleCreateBranch(w http.ResponseWriter, r *http.Request) {
return
}
ctx := r.Context()
// Both bases of a request (base_ref, then preferred_base) walk under one
// budget: one slot and one time budget for the request.
walks := wrefWalkBudget(repo.RepoID)
defer walks.Close()
// create runs the creation at one resolved base. It returns false, with no
// response written, when no registered pack holds the base commit: the
// node's cache can hold objects that no registered pack holds (a cache that
// was copied or edited out of band, or a registry that was rolled back), and
// a ref at such an object wedges every other node, and this node after its
// cache is gone, because they hydrate from the registered packs alone. The
// registry is read only for a request that is about to write: a denied
// request, and one that leaves the branch where it is, never read it.
// response written, when the registered packs lack the base commit or part
// of its history (wrefUsableCommit): the node's cache can hold objects that
// no registered pack holds (a cache that was copied or edited out of band,
// or a registry that was rolled back), and a ref at such an object wedges
// every other node, and this node after its cache is gone, because they
// hydrate from the registered packs alone. The registry is read only for a
// request that is about to write: a denied request, and one that leaves the
// branch where it is, never read it.
create := func(baseSHA string) bool {
held := false
attempts := 1
Expand Down Expand Up @@ -515,15 +554,29 @@ func (s *server) handleCreateBranch(w http.ResponseWriter, r *http.Request) {
return true
}
if !held {
ok, err := s.wrefHeldCommit(ctx, repo.RepoID, repoPath, baseSHA)
// A create that cannot win is answered before the walk: a plain
// create of a branch that exists loses its swap, whatever the base.
if oldSHA == ref.ZeroSHA && !allocate {
now, ok := s.wrefTip(w, r, repo.RepoID, targetRef)
if !ok {
return true
}
if now != ref.ZeroSHA {
writeErr(w, http.StatusConflict, "branch already exists")
return true
}
}
ok, err := s.wrefUsableCommit(ctx, walks, repo.RepoID, repoPath, baseSHA)
if err != nil {
s.internalErr(w, r, "check that a registered pack holds the base", err)
s.internalErr(w, r, "check that the registered packs hold the base and its history", err)
return true
}
if !ok {
return false
}
held = true
// No walk follows: free the slot before the ref is written.
walks.Close()
}
won, err := s.refStore.CASRef(ctx, repo.RepoID, targetRef, oldSHA, baseSHA)
if err != nil {
Expand Down Expand Up @@ -808,10 +861,10 @@ func (s *server) handleCreateTag(w http.ResponseWriter, r *http.Request) {
writeErr(w, http.StatusConflict, "tag already exists")
return
}
held, err := s.wrefHeldCommit(r.Context(), repo.RepoID, repoPath, sha)
held, err := s.wrefUsableCommitAlone(r.Context(), repo.RepoID, repoPath, sha)
switch {
case err != nil:
s.internalErr(w, r, "check that a registered pack holds the tag target", err)
s.internalErr(w, r, "check that the registered packs hold the tag target and its history", err)
return
case !held:
writeProblem(w, http.StatusNotFound, codeRefNotFound, "ref does not resolve to a commit")
Expand Down
Loading
Loading