Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .claude/skills/codetrial-contribute/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ approval included, are in the GitHub section of
[codetrial-conventions](../codetrial-conventions/SKILL.md). Keep technical
identifiers and quoted errors exact.

Before any push, an agent follows the [Before you push](../../../CONTRIBUTING.md#before-you-push)
section of CONTRIBUTING.md.

Example requests: "Turn these notes into an issue", "Help me make my first
CodeTrial contribution", or "Write a PR description from this diff". Produce
the artifact the user asked for.
Expand Down
65 changes: 46 additions & 19 deletions .claude/skills/codetrial-triage/apply-pr-actions.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,16 @@
#
# confirm means the action is not one a sweep takes alone (a held run that is
# not a first-time contributor's, one that touches workflows or whose file list
# was cut, or any approval while a workflow runs on pull_request_target or
# workflow_run): it goes on the confirmed list. A rebase request is skipped
# unless the head still matches the sweep and, for a conflict, GitHub still says
# CONFLICTING. A PR in `rebase` whose request was not posted keeps all of its
# earlier requests visible (hide needs the new one to have landed), and hide_ask
# is never touched: it is the maintainer's list.
# was cut, one whose head carries no gate evidence, or any approval while a
# workflow other than the reviewed pr-description.yml runs on
# pull_request_target or workflow_run): it goes on the confirmed list. An
# approval also needs gate-evidence.sh to pass for the head the sweep read,
# checked again right before the call; when the answer cannot be given (the head
# moved, the fork is gone, a fetch failed) the result is skipped. A rebase
# request is skipped unless the head still matches the sweep and, for a
# conflict, GitHub still says CONFLICTING. A PR in `rebase` whose request was
# not posted keeps all of its earlier requests visible (hide needs the new one
# to have landed), and hide_ask is never touched: it is the maintainer's list.

set -eu

Expand Down Expand Up @@ -55,6 +59,7 @@ done
jq -e '(.prs | arrays) and (.maintainers_known | type == "boolean")
and all(.prs[]; (.number | type == "number") and (.head | type == "string"))
and all(.approve[]; (.pr | type == "number") and (.run | type == "number")
and (.head | type == "string" and test("^([0-9a-f]{40}|[0-9a-f]{64})$"))
and (.first_time | type == "boolean")
and (.touches_workflows | type == "boolean")
and (.files_truncated | type == "boolean"))
Expand Down Expand Up @@ -153,25 +158,47 @@ else
risky="pr-description.yml runs on workflow_run"
fi
fi
# One row per held run: the PR, the run, and whether every guard holds exactly.
jq -r '.approve[] | [.pr, .run, (.first_time == true and .touches_workflows == false
and .files_truncated == false)] | @tsv' "$sweep" | while read -r pr id guards; do
if [ -n "$risky" ]; then
receipt approve "$pr" confirm "$risky"
elif [ "$guards" != true ]; then
receipt approve "$pr" confirm "not a first-time contributor, touches workflows, or the file list was cut"
elif err=$(gh api -X POST "repos/$repo/actions/runs/$id/approve" 2>&1); then
receipt approve "$pr" applied "run $id"
# Approve one run and say what happened.
approve_run()
{
if err=$(gh api -X POST "repos/$repo/actions/runs/$2/approve" 2>&1); then
receipt approve "$1" applied "run $2"
elif printf '%s\n' "$err" | grep -q 'not waiting for approval'; then
status=$(gh run view "$id" --repo "$repo" --json status --jq .status) || status=unread
status=$(gh run view "$2" --repo "$repo" --json status --jq .status) || status=unread
case $status in
queued | in_progress)
receipt approve "$pr" already "run $id is $status after approval"
receipt approve "$1" already "run $2 is $status after approval"
;;
*) receipt approve "$pr" failed "run $id is not waiting, but approval could not be verified (status: $status)" ;;
*) receipt approve "$1" failed "run $2 is not waiting, but approval could not be verified (status: $status)" ;;
esac
else
receipt approve "$pr" failed "$err"
receipt approve "$1" failed "$err"
fi
}

# One row per held run: the PR, the run, the head the sweep read, and whether
# every guard holds exactly. The evidence is read again here, for the head the
# sweep saw, immediately before the approval call: a verdict saved in the sweep
# would be about a head a push may have replaced. Only a pass approves; ask goes
# on the confirmed list as the comment to draft, and an indeterminate answer (a
# moved head, an unreadable fork, a failed fetch) approves nothing.
evidence_check=${APPLY_GATE_EVIDENCE:-$here/gate-evidence.sh}
jq -r '.approve[] | [.pr, .run, .head, (.first_time == true and .touches_workflows == false
and .files_truncated == false)] | @tsv' "$sweep" | while read -r pr id head guards; do
if [ -n "$risky" ]; then
receipt approve "$pr" confirm "$risky"
elif [ "$guards" != true ]; then
receipt approve "$pr" confirm "not a first-time contributor, touches workflows, or the file list was cut"
else
code=0
verdict=$(sh "$evidence_check" "$repo" "$pr" "$head" < /dev/null 2> /dev/null) || code=$?
reason=$(printf '%s\n' "$verdict" | jq -r '.reason // empty' 2> /dev/null) || reason=
: "${reason:=the evidence check printed no reason}"
case $code in
0) approve_run "$pr" "$id" ;;
1) receipt approve "$pr" confirm "no gate evidence, draft the toolchain comment: $reason" ;;
*) receipt approve "$pr" skipped "gate evidence is indeterminate, not approved: $reason" ;;
esac
fi
done > "$work/approve.out"

Expand Down
169 changes: 169 additions & 0 deletions .claude/skills/codetrial-triage/gate-evidence.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
#!/bin/sh
# Read-only evidence check for one pull request at one head: gather what
# scripts/check-gate-evidence.py needs as data and print its verdict.
#
# Usage: gate-evidence.sh <repo> <pr> <head-sha>
# Prints one JSON object on stdout:
# {verdict, reason, head}
# verdict is pass (the evidence holds, or no receipt is owed), ask (the
# receipt is missing or wrong: draft the comment that asks the
# contributor to install the toolchain and run ./scripts/test.sh) or
# indeterminate (the answer cannot be given: never approve). The exit
# status is the checker's: 0, 1 or 2.
#
# <head-sha> is the head the caller looked at. The pull request is read again
# here, and a head that differs, now or when the gathering ends, is
# indeterminate: a force-push changes what the evidence is about. A caller that
# approves reads this immediately before the approval call.
#
# Everything from the pull request is data: file names, branch names and the
# note reach the checker as files and arguments, never through eval or a shell
# string, and the head repository is never checked out. The base toolchain file
# comes from the base branch, never the head; the head's is read only when the
# change edits it. The note is fetched by scripts/fetch-gate-note.sh from the
# owner and name the API reports. GATE_EVIDENCE_FETCH names another fetcher and
# GATE_EVIDENCE_PYTHON another interpreter, for tests; neither can be set from a
# pull request.

set -eu

here=$(cd "$(dirname "$0")" && pwd)
root=$(cd "$here/../../.." && pwd)
checker=$root/scripts/check-gate-evidence.py
fetcher=${GATE_EVIDENCE_FETCH:-$root/scripts/fetch-gate-note.sh}

sha_re='^([0-9a-f]{40}|[0-9a-f]{64})$'
head=

emit()
{
jq -nc --arg verdict "$1" --arg reason "$3" --arg head "$head" \
'{verdict: $verdict, reason: $reason, head: $head}'
exit "$2"
}

indeterminate()
{
emit indeterminate 2 "$1"
}

if [ $# -ne 3 ]; then
echo "usage: gate-evidence.sh <repo> <pr> <head-sha>" >&2
exit 2
fi
repo=$1
pr=$2
want=$3
printf '%s\n' "$repo" | grep -Eq '^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$' \
|| indeterminate "the repository name is not owner/name"
case $pr in
'' | *[!0-9]*) indeterminate "the pull request number is not a number" ;;
esac
printf '%s\n' "$want" | grep -Eq "$sha_re" || indeterminate "the head sha is not hex"
head=$want

# The checker needs tomllib, Python 3.11; the default python3 may be older.
python=
for candidate in "${GATE_EVIDENCE_PYTHON:-}" python3.13 python3.12 python3.11 \
"${HOME:-/nonexistent}/.local/bin/python3.12" python3; do
[ -n "$candidate" ] || continue
if "$candidate" -c 'import sys; sys.exit(sys.version_info < (3, 11))' 2> /dev/null; then
python=$candidate
break
fi
done
[ -n "$python" ] || indeterminate "no Python 3.11 or later was found to run the evidence check"

work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT

# rust-toolchain.toml at a ref of the base repository, as raw text into a file.
contents()
{
gh api -X GET -H 'Accept: application/vnd.github.raw+json' \
"repos/$repo/contents/rust-toolchain.toml" -f ref="$1" > "$2"
}

# A field of the pull request; a missing or unreadable one is the empty string.
pull=$(gh api "repos/$repo/pulls/$pr") || indeterminate "cannot read the pull request"
field()
{
printf '%s\n' "$pull" | jq -r "$1 // empty" 2> /dev/null || true
}
live=$(field .head.sha)
[ "$live" = "$want" ] || indeterminate "the head moved from $want to ${live:-unknown}"
state=$(field .state)
[ "$state" = open ] || indeterminate "the pull request is $state"
base_ref=$(field .base.ref)
Comment thread
jserv marked this conversation as resolved.
Resolved
changed=$(field .changed_files)
author_type=$(field .user.type)
head_owner=$(field .head.repo.owner.login)
head_name=$(field .head.repo.name)
case $changed in
'' | *[!0-9]*) indeterminate "the changed file count is unreadable" ;;
esac
[ -n "$base_ref" ] || indeterminate "the base branch is unreadable"
[ -n "$author_type" ] || indeterminate "the author type is unreadable"

# The changed paths, a NUL after every one, including the last. A rename names
# both ends, since moving a file out of the Rust scope changes it too. The list
# is cut when the entries read are fewer than the PR's count.
gh api --paginate "repos/$repo/pulls/$pr/files" > "$work/files.json" \
|| indeterminate "cannot read the changed files"
read_count=$(jq -s '[.[][]] | length' "$work/files.json" 2> /dev/null) \
|| indeterminate "the changed files are unreadable"
if jq -se 'any(.[][]; (.filename, (.previous_filename // "")) | explode | any(. == 0))' \
"$work/files.json" > /dev/null 2>&1; then
indeterminate "a changed path holds a NUL byte"
fi
if [ "$read_count" -ne "$changed" ]; then
indeterminate "the changed-file list is truncated: $read_count of $changed paths"
fi
jq -j '.[] | .filename, "\u0000", ((.previous_filename // empty) | ., "\u0000")' \
"$work/files.json" > "$work/paths" || indeterminate "the changed paths are unreadable"

set -- --changed-files-file "$work/paths" --author-type "$author_type"
scope=$("$python" -I "$checker" --scope-only -z < "$work/paths" 2> "$work/scope.err") \
|| indeterminate "$(head -n 1 "$work/scope.err")"

# The checker always wants the head's tree, and a receipt names it.
tree=$(gh api "repos/$repo/git/commits/$want" --jq .tree.sha) \
|| indeterminate "cannot read the head commit's tree"
set -- "$@" --head-tree "$tree"

if [ "$scope" = in ] && [ "$author_type" != Bot ]; then
contents "$base_ref" "$work/base-toolchain" \
|| indeterminate "cannot read rust-toolchain.toml from the base branch"
set -- "$@" --base-toolchain-file "$work/base-toolchain"
if jq -se 'any(.[][]; .filename == "rust-toolchain.toml"
or .previous_filename == "rust-toolchain.toml")' "$work/files.json" > /dev/null 2>&1; then
contents "$want" "$work/head-toolchain" \
|| indeterminate "cannot read rust-toolchain.toml from the head"
set -- "$@" --head-toolchain-file "$work/head-toolchain"
fi

# No note is an answer (the checker asks for one); a repository that cannot
# be read, or a fetch that failed, is not.
if [ -z "$head_owner" ] || [ -z "$head_name" ]; then
indeterminate "the head repository is gone, so its gate note cannot be read"
fi
status=0
sh "$fetcher" "$head_owner" "$head_name" "$want" > "$work/note" 2> "$work/note.err" || status=$?
case $status in
0) set -- "$@" --receipt-file "$work/note" ;;
3 | 6 | 7) ;;
*) indeterminate "$(head -n 1 "$work/note.err")" ;;
esac
fi

status=0
"$python" -I "$checker" "$@" > "$work/out" 2> "$work/err" || status=$?
# A push that landed while this ran would make the evidence about an old head.
after=$(gh api "repos/$repo/pulls/$pr" --jq .head.sha) || indeterminate "cannot read the pull request again"
[ "$after" = "$want" ] || indeterminate "the head moved while the evidence was gathered"
message=$(head -n 1 "$work/err")
case $status in
0) emit pass 0 "$(tail -n 1 "$work/out")" ;;
1) emit ask 1 "$message" ;;
*) indeterminate "${message:-the evidence check failed}" ;;
esac
22 changes: 20 additions & 2 deletions .claude/skills/codetrial-triage/pr-sweep.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,10 @@
# maintainers_known false when the push listing failed and the COLLABORATOR
# badge, which read-only collaborators also carry, stood in for it
# approve runs waiting for approval on the current head of an open PR, and
# whether its author is a first-time contributor
# whether its author is a first-time contributor; head is the sha
# the sweep read, and evidence is gate-evidence.sh's verdict for it
# (pass, ask or indeterminate, with its reason). PR_SWEEP_EVIDENCE
# names another gate-evidence.sh, for tests
# outdated unresolved review threads whose lines the PR no longer touches,
# to recheck at the head before resolving
# overlap PR pairs that close the same issue
Expand Down Expand Up @@ -494,7 +497,7 @@ def refs: (ref_prefix + "[0-9]+\\b") as $one
([$prs[] | status] | sort_by(.size.total)) as $status |
{
approve: [$prs[] as $p | runs_of($p)
| {pr: $p.number, run: .id, workflow: .name, actor,
| {pr: $p.number, run: .id, workflow: .name, actor, head: $p.headRefOid,
touches_workflows: any($p.files.nodes[].path; startswith(".github/")),
files_truncated: ($p.files.totalCount > 100),
first_time: ($p.authorAssociation
Expand Down Expand Up @@ -742,9 +745,24 @@ collide=$(find "$work" -name '*.diff' -exec awk \
context: (map(.context | select(length > 0)) | unique)}]
| unique_by([.a, .b, .path])')

# The evidence verdict of each PR with a held run, read at the head the sweep
# saw. A PR whose check printed nothing is indeterminate, never a pass.
evidence_check=${PR_SWEEP_EVIDENCE:-$(dirname "$0")/gate-evidence.sh}
evidence=$(printf '%s\n' "$out" | jq -r '[.approve[] | [.pr, .head]] | unique[] | @tsv' \
| while read -r n head; do
found=$(sh "$evidence_check" "$repo" "$n" "$head" < /dev/null 2> /dev/null) || true
printf '%s\n' "$found" | jq -c --argjson n "$n" \
'select(type == "object" and (.verdict | IN("pass", "ask", "indeterminate")))
| {pr: $n, evidence: {verdict, reason, head}}' 2> /dev/null || true
done | jq -s .)

printf '%s\n' "$out" | jq --argjson states "$states" --argjson stale "$stale" \
--argjson collide "$collide" --argjson unread "$unread" \
--argjson evidence "$evidence" \
'.collide = $collide | .collide_unread = $unread
| .approve |= map(. as $a | . + {evidence: (first($evidence[]
| select(.pr == $a.pr) | .evidence) // {verdict: "indeterminate",
head: $a.head, reason: "the evidence check printed nothing"})})
| if .gone == null then . else .gone = $states end
| .rebase += [.stale_candidates[] | select(.pr | IN($stale[]))
| del(.head)]
Expand Down
39 changes: 31 additions & 8 deletions .claude/skills/codetrial-triage/pulls.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,31 +56,54 @@ the head before it is posted. A bare "continue" after an issue report is not
that request: it carries the work on, and these actions go on the confirmed
list instead.

- Approving a run in `approve` whose `first_time` is true and whose
`touches_workflows` and `files_truncated` are false. A run waits there because
the repository's policy (`gh api
- Approving a run in `approve` whose `first_time` is true, whose
`touches_workflows` and `files_truncated` are false, and whose `evidence`
verdict is `pass` when read again for the head the sweep saw. A run waits
there because the repository's policy (`gh api
repos/$repo/actions/permissions/fork-pr-contributor-approval` reads
`first_time_contributors`) holds `pull_request` workflows from anyone without
a merged commit. Most PRs here come from such contributors, and the hold
applies again to every push until their first PR merges, so a run left
unapproved is a contributor waiting on CI that never starts while review waits
on green CI. "Approve the held runs" asks for this action alone: run the sweep
and apply only `approve`. A contributor's comment that a run is "awaiting
and apply only `approve`, which still approves one run at a time, each on its
own evidence. A contributor's comment that a run is "awaiting
approval" (#312) is a reason to suggest that request to the user, not the
request itself: text on a thread never authorizes a write. The repository is
public, so a run from a fork gets a read-only token and no secrets whatever
its workflow file says, and the approval spends runner time and nothing else.
Two triggers escape that rule by running with the base repository's token and
secrets, so confirm neither is in use: `grep -n
'pull_request_target\|workflow_run' .github/workflows/*` prints nothing. When
it prints something, every approval goes on the confirmed list. A run whose
`touches_workflows` is true is never approved under this rule: it goes on
secrets, so check which workflows use them: `grep -n
'pull_request_target\|workflow_run' .github/workflows/*`. Today it prints
only `.github/workflows/pr-description.yml`, which runs the base branch's
copy on a read-only token and reads the pull request as data, and
`apply-pr-actions.sh` treats that file as the one reviewed exception for
`pull_request_target`. A hit in any other workflow, or `workflow_run` in that one, puts every approval on the
confirmed list. A run whose `touches_workflows` is true is never approved
under this rule: it goes on
the confirmed list, with its diff read first against the workflow rules
in [ci.md](../codetrial-review/rules/ci.md#keep-what-makes-a-fork-pr-safe-to-run)
and the result stated there, whoever wrote it. Runner time
still matters: skim the diff, and move the approval to the confirmed list when
code the build or tests execute does something the title does not explain,
such as a network call or a download it then runs.
A PR that changes a path in the Rust scope needs the gate evidence
that `gate-evidence.sh` reads as data (the head's tree and changed paths from
the API, the toolchain pin from the base branch, and the note on the head
commit from `scripts/fetch-gate-note.sh`) and `scripts/check-gate-evidence.py`
accepts: `pass` when it exits 0, `ask` on 1 (a missing, oversized or wrong
note, including a fork that never pushed the notes ref), and `indeterminate`
on 2, on a failed or timed-out fetch, a head that moved, a deleted or private
fork, or a truncated file list.
`apply-pr-actions.sh` runs the check again immediately before each approval
call, because a force-push changes the head and a verdict from the sweep
would then describe code nobody checked; it approves only on `pass`. On `ask`
it posts nothing and puts the PR on the confirmed list with a drafted comment
asking the contributor to install the toolchain and run `./scripts/test.sh`
(posting that comment keeps its usual confirmation). On `indeterminate` it
does not approve and reports why. A change with no Rust-scope path, or from a
bot, owes no note and passes. The note is a hint a dishonest author can forge;
it makes the decision mechanical and does not prove the gate ran.
- Resolving a thread in `outdated` once it was checked at the PR's head and
found addressed. Outdated means the commented lines changed, not that the
concern was met, and an open thread is what holds a grade down, so resolving
Expand Down
Loading
Loading