Skip to content

fix(githttp): answer a delete of a ref that the replica does not list - #928

Merged
ThomasK33 merged 6 commits into
mainfrom
delete-absent-ref-refusal
Nov 20, 2026
Merged

ThomasK33 merged 6 commits into
mainfrom
delete-absent-ref-refusal

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Why

git push <remote> :<ref> through a replica whose cache does not list the ref reaches the update hook as a command from the zero id to the zero id. Stock git sends exactly that: git push o :refs/notes/a, for a ref that the remote does not advertise, makes the client send 0000... 0000... refs/notes/a, and stock receive-pack answers it with ok and the warning deleting a non-existent ref (checked with git 2.53). A client can also send it for any ref by hand.

The hook gave the command to the ref store, where a swap from nothing to nothing is an assertion in every store (casRef). The hook panicked. The pusher read only the generic update failed; push again later (the server log has the cause) and git's hook declined, and the log held a Go trace. Found while verifying #917, whose runbook says that this push fails.

What changed

  • updateHook (internal/githttp/hook.go): a command with old and new both zero is answered by the new answerDeleteOfAbsentRef, after the ref policy, the hold and the sync fence, and before the swap. It reads the ref once from the ref store (GetRef, a point read that every store has):
    • The store holds no such ref: the hook answers ok, as stock git does. Nothing is written, so no generation bump, webhook or sync poke follows.
    • The store holds the ref: the hook refuses the command with a per-ref ng (hook declined) whose remote: text names the ref and the first 8 hex digits of its SHA (update: <ref> exists at <sha> in the ref store, but this replica does not list it: fetch, then push the delete again) and writes a WARN line to the hook-errors file, which serve logs (ref, sha, cause). The ref stays. Stock git would delete an existing ref here, because it reads a zero old value as "no value to check". The hook cannot do that: the ref policy ran with old = zero, so a policy that decides on the value that a delete removes (no-force-push, for one) has not seen it.
    • The read fails: a hook failure, with the generic line for the pusher and the cause in the hook-errors file, like any other store error.
  • Docs: the runbook step for a pair of refs in another namespace said that the push fails. It now says what the replica answers. git-protocol-and-namespaces.md lists the answer, and the monitoring table and the metrics mapping in deploy-production.md list the operator line.
  • The lane-schedule fingerprints are re-recorded (updateHook is covered, answerDeleteOfAbsentRef is new). The model does not describe the zero-to-zero command.

Tests

  • TestDeleteOfAnAbsentRefIsAnsweredNotCrashed (e2e, runs on every lane): a zero-to-zero command for a tag, a branch and a notes ref that no store holds answers ok and writes nothing, the same command for a notes ref that the store holds answers ng with the ref and SHA in the pusher's text and the operator line (ref, sha, cause) in serve's log and leaves the ref, and a no-push policy still refuses. It is red on main on all three lanes: the first head of this PR held only this test, and CI failed test (PostgreSQL), test-local and test-s3wal on its four subtests (a tag that no store holds, a branch ..., a notes ref that no store holds, a notes ref that the ref store holds).

  • TestUpdateHookAnswersAZeroToZeroCommand runs the real update hook on a SQLite store (red on main: update failed; push again later), and TestAnswerDeleteOfAbsentRefOnEachStore runs the answer on SQLite and on S3-WAL with an in-memory conditional store, including a failed read. PostgreSQL runs in the e2e test.

  • TestUpdateHookSwapsEveryCommandThatIsNotZeroToZero runs a create, a delete and zero-to-zero commands through the real hook and counts the swaps: only zero-to-zero is answered without one.

  • Mutants that each fail a test: the absent answer refused, the held answer ok, a read error ignored, no operator line, the line without sha and cause, the condition widened to old == zero only or new == zero only, the call moved before the policy check, and the branch removed (main's behavior).

  • Evidence at the head (404ac51): CI is green (test, test-local, test-s3wal, lint, both fingerprint gates). go test -race ./internal/githttp on the local backend passed at 2dbe03d (857 s), and the commits since change a test and docs only. Codex code and security reviews of the head found nothing. A read-only verifier that I did not brief on the fix ran its own probes and answered VERDICT #928 404ac51a PASS. Its earlier round on 2dbe03d found wording and test gaps, which 37c86c0 closes.

Limits

  • A client that sent zero to zero for a ref that exists gets a refusal, not a delete, as described above. Through a replica that lists the ref the client sends the real old value, and the push deletes it as before.
  • Under --atomic, pre-receive refuses the whole push first, as for any command whose old value the store contradicts (<ref> already exists (created concurrently); fetch first), with no SHA and no operator line.
  • The refusal tells the pusher to fetch. On a repository that cannot hydrate (the pair of fix(cmd): repair-refs lists an uncertain delete in its recovery warning #917) the fetch fails, so the delete has to go through a replica whose cache lists the ref. The runbook says so.

Generated with xum • Model: anthropic:claude-sonnet-5-5 • Thinking: xhigh

…oes not list

git sends `git push <remote> :<ref>` as a command from the zero id to the
zero id when the replica's cache does not list the ref. Stock git's
receive-pack answers it with ok and the warning "deleting a non-existent
ref". The update hook swaps nothing to nothing in the ref store, which is an
assertion there, and the pusher reads only "hook declined". The test runs
the command against a ref that no store holds and against one that the store
holds, on every lane (PostgreSQL, local and S3-WAL), and fails on main.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
<!-- xum-attribution: model=anthropic:claude-sonnet-5-5 thinking=xhigh -->
git sends `git push <remote> :<ref>` as a command from the zero id to the
zero id when the replica's cache does not list the ref, and a client can
send it for any ref. The update hook passed it to the ref store, where a
swap from nothing to nothing is an assertion, so the push ended in "hook
declined" and a Go panic in the log.

The hook now answers the command from the ref store. When the store holds no
such ref, the command is a delete of a ref that does not exist, which stock
git answers with ok and a warning, and so does the hook. When the store holds
the ref, the pusher's belief that it is absent is wrong (the replica's cache
lags, or the ref was created since the client read the refs), so the hook
refuses the command with a per-ref ng that names the ref and its SHA, and
writes an operator line to the hook-errors file. A failed read is a hook
failure that the pusher reads as a generic line.

The runbook step for a ref pair in another namespace said that the push
fails. It now says what the replica answers. The git protocol page and the
monitoring table name the new line.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
<!-- xum-attribution: model=anthropic:claude-sonnet-5-5 thinking=xhigh -->
… absent ref

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
<!-- xum-attribution: model=anthropic:claude-sonnet-5-5 thinking=xhigh -->
…usal

Change-Id: I83d590f52987284a0c6d29125bef2bf78bd837fd
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33
ThomasK33 marked this pull request as ready for review November 20, 2026 15:43
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T16:11:52.732056Z 404ac51 Manual request
🔒 Security Review ✅ Completed 2026-11-20T16:18:23.276465Z 404ac51 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 2dbe03de95

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 2dbe03de95

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…o answer

A verifier found that the operator line could lose its sha and cause fields
and that the zero-to-zero condition could be widened to one of its two
halves, and that no test of the pull request failed on either. A test now
runs a create, a delete and zero-to-zero commands through the real update
hook and counts the swaps, and the operator line test names every field.
The git protocol page says that an atomic push with such a command is
refused whole by pre-receive, and that the remote text names the ref and the
first 8 hex digits of the SHA.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
<!-- xum-attribution: model=anthropic:claude-sonnet-5-5 thinking=xhigh -->
…usal

Change-Id: Ia4de684eee91d47f9cc81fbec16de1ebb41c9bc1
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 404ac51a0a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 404ac51a0a

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ThomasK33
ThomasK33 added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 7211578 Nov 20, 2026
16 of 17 checks passed
@ThomasK33
ThomasK33 deleted the delete-absent-ref-refusal branch November 20, 2026 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant