From 6fd7ad5830a410e1f03b552d63bfb735baa1b346 Mon Sep 17 00:00:00 2001 From: AlexeyShalaev <75322386+AlexeyShalaev@users.noreply.github.com> Date: Thu, 8 Oct 2026 04:47:34 +0300 Subject: [PATCH] docs: no GitHub sandbox; the bedrock-python hub is the live check No sandbox organization will run the live GitHub scenarios. e2e.md now lists them as what only a live GitHub confirms, says which of them the bedrock-python hub takes on github.com (the positive paths of items 15 to 17 and 21), and drops the sandbox's setup and the harness to build. The threat model, CONTRIBUTING and the code comments that waited for the sandbox now say unverified, or name the hub. Also stale since the releases: the batch read is used by plan and distribute; the publish workflow has run (v0.1.0 and v0.2.0); the template pins a release, so the after-release steps pin the next one instead of replacing placeholders; and there is no GitLab mirror of the template. --- CONTRIBUTING.md | 3 ++- docs/project/e2e.md | 20 ++++++++++---------- docs/project/release.md | 8 ++++---- docs/project/threat-model.md | 12 ++++++------ internal/platform/github/client.go | 4 ++-- internal/platform/github/ghfake/doc.go | 8 ++++---- internal/platform/github/ghfake/rest.go | 2 +- internal/platform/github/ghfake/setup.go | 3 +-- internal/platform/github/github.go | 4 ++-- internal/platform/github/probe.go | 3 +-- internal/platform/github/rules.go | 4 ++-- 11 files changed, 35 insertions(+), 36 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 868284b..51551fa 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -124,7 +124,8 @@ bash scripts/e2e/gitlab.sh all # every supported GitLab with a runner; about CI runs the Gitea and Forgejo tests on every pull request; a maintainer adds the label `e2e-gitlab` to run the GitLab ones. A change to the image, the Action or what the hub template's CI files rely on also gets a run of the template's pipelines (`--template`). -GitHub's tests run against a fake of its API; the live ones need a sandbox organisation. +GitHub's tests run against a fake of its API; on github.com touchmark runs for real in the +bedrock-python hub. [End-to-end tests](https://bedrock-python.github.io/touchmark/project/e2e/) has the details. diff --git a/docs/project/e2e.md b/docs/project/e2e.md index fab9821..0d0ebba 100644 --- a/docs/project/e2e.md +++ b/docs/project/e2e.md @@ -1,6 +1,6 @@ # e2e tests: Gitea, Forgejo, GitLab and GitHub -`internal/e2e` tests touchmark against real Gitea and Forgejo servers in Docker. The build tag `e2e` keeps them out of the ordinary `go test ./...`; `scripts/e2e/gitea.sh` starts a server, seeds it and runs them. `internal/e2e/gitlab` does the same for GitLab with `scripts/e2e/gitlab.sh`: see [GitLab](#gitlab) below. `internal/e2e/github` runs the GitHub driver against an HTTP fake of GitHub, in the ordinary `go test ./...`, and lists what only a live GitHub sandbox can confirm: see [GitHub](#github). With `--template`, both scripts also run the hub template's own CI files, and a dry run checks its GitHub workflow: see [The hub template](#the-hub-template). +`internal/e2e` tests touchmark against real Gitea and Forgejo servers in Docker. The build tag `e2e` keeps them out of the ordinary `go test ./...`; `scripts/e2e/gitea.sh` starts a server, seeds it and runs them. `internal/e2e/gitlab` does the same for GitLab with `scripts/e2e/gitlab.sh`: see [GitLab](#gitlab) below. `internal/e2e/github` runs the GitHub driver against an HTTP fake of GitHub, in the ordinary `go test ./...`, and lists what only a live GitHub can confirm: see [GitHub](#github). With `--template`, both scripts also run the hub template's own CI files, and a dry run checks its GitHub workflow: see [The hub template](#the-hub-template). ## Run @@ -309,11 +309,11 @@ The cleanup removes the runner, the registry, the forwarder and the image tags, What held, on GitLab CE 18.11.12 with the runner 18.11.4 and on Gitea 1.27.3 with its runner 4.1.0: the image as uid 65532 reads a checkout another user made (`safe.directory`) and writes the report files; GitLab's Docker executor runs the job script with the image's `sh` (`entrypoint: [""]`) and its helper uploads the artifacts; `GIT_STRATEGY: none` for the probe; the merge request pipeline gets the reader's variable but not the protected, environment-scoped write key (the probe passes); `distribute` on the protected default branch in the environment `touchmark-distribute`; the schedules told apart by description. Gitea's runner keeps the job container alive with `sleep`, gives the workspace to the image's user (`chown -R 65532:65532`), runs `run:` steps with the image's `sh`, resolves the YAML anchor of the clone step, and lets the job token (Basic auth, any user name) fetch the hub and comment on the pull request. -Not covered: GitLab's "Run pipeline" (source `web`, which the API cannot start: `POST /projects/:id/pipeline` is source `api`, which the template's `workflow:rules` filter out), Forgejo's runner, the GitHub workflow on GitHub itself (the sandbox, scenario 17 below), and the step summary on Gitea. +Not covered: GitLab's "Run pipeline" (source `web`, which the API cannot start: `POST /projects/:id/pipeline` is source `api`, which the template's `workflow:rules` filter out), Forgejo's runner, and the step summary on Gitea. The GitHub workflow runs on GitHub itself in the bedrock-python hub; its negative checks (item 17 below) are not covered. ## GitHub -There is no live GitHub sandbox yet. Until there is, `internal/e2e/github` runs the GitHub driver (`internal/platform/github`) end to end against `ghfake` (`internal/platform/github/ghfake`), an HTTP fake of GitHub's REST API, GraphQL API and git smart HTTP on one local server, backed by real bare repositories. The package has no build tag and needs no network: it is part of the ordinary `go test ./...`. What the fake does and where each behavior comes from (docs, read-only observation, or assumed) is the table in `ghfake`'s package documentation. A test that depends on an assumed behavior proves only the fake; the [live sandbox scenarios](#live-sandbox-scenarios) below are what confirms them. +There is no live GitHub sandbox: `internal/e2e/github` runs the GitHub driver (`internal/platform/github`) end to end against `ghfake` (`internal/platform/github/ghfake`), an HTTP fake of GitHub's REST API, GraphQL API and git smart HTTP on one local server, backed by real bare repositories. The package has no build tag and needs no network: it is part of the ordinary `go test ./...`. What the fake does and where each behavior comes from (docs, read-only observation, or assumed) is the table in `ghfake`'s package documentation. A test that depends on an assumed behavior proves only the fake; [what only a live GitHub confirms](#what-only-a-live-github-confirms) lists those behaviors and what the bedrock-python hub confirms on github.com. ### Run @@ -365,7 +365,7 @@ CI's job `contracts` does both on every pull request and push, and fails when a - a person's pull request from a fork on the sync branch's name, with a copy of the marker, is never touched; - after every run, a second one writes nothing (by the report and by the fake's state); every per-target token was minted for one repository and revoked, and only those of the two targets whose writes change workflows carry workflows write; every reading token of the writer is revoked when its run ends; the journal is empty (no token used on another repository, no write with a revoked token, no branch of an open pull request deleted or moved onto its base); no secret (App keys, the person's token, every token the fake minted) appears in any output, report, pull request, comment or commit message. - **`TestDistributeUnsigned`**: the same hub on a GitHub Enterprise Server without web commit signing: the API commit comes back unsigned, the target is `blocked:cannot-sign`, the branch never moves to it, no pull request opens, and the stage ref is deleted. The next run tries again (two writes: the stage push and the API commit), since the platform may have started signing. -- **`TestBatchReader`**: the driver's GraphQL batch read of one path in many repositories agrees with `ReadFile` for a regular file, an executable, a binary file, a symlink, a submodule, a directory, a missing file, a file over the limit, an empty repository and a repository that does not exist. The core does not use the batch read yet. +- **`TestBatchReader`**: the driver's GraphQL batch read of one path in many repositories agrees with `ReadFile` for a regular file, an executable, a binary file, a symlink, a submodule, a directory, a missing file, a file over the limit, an empty repository and a repository that does not exist. `plan` and `distribute` read the opt-in files of GitHub targets with it. - **`TestShapes`**: `testdata/shapes.json` holds the shapes of github.com's answers to read-only requests on public repositories (2026-09-29), trimmed to the fields the driver reads and reduced to JSON types, with no value, login, name or id: a repository and a listing item, a pull request from a fork, a tree with one entry of each mode, a blob, contents of a file, a symlink and a submodule, `rules/branches` for an existing and a missing branch, `hash-algorithm`, a bot and a user, a label, a 404, a commit GitHub signed, `/meta`, the `Link` header of a pull request listing, `git/ref/heads/{branch}` of a branch (one ref) and of a missing one whose name starts another's (404, not a list of matches), and GraphQL's errors with HTTP 200 (a missing repository alias, a missing `Commit.file` path, an unknown field, an unused variable), `Commit.file` modes, an open pull request's node with every field the driver reads, and a bot's `ClosedEvent`. The test asks the fake the same and compares fields, types, pinned values and fields GitHub leaves out. - **`TestGraphQLContracts`** (`internal/platform/github`): every fixed GraphQL document the driver sends validates against the schemas of github.com and GHES 3.19 (the fake's subsets, which `TestSchemaSubset` checks against GitHub's). `TestSchemaSubset` runs in CI (job `contracts`) with both schemas downloaded. - **`TestRESTContracts`** (`internal/platform/github`): every REST route the driver's unit tests see it call is in its table of routes, and, with `GHFAKE_REST_DESCRIPTION` and `GHFAKE_REST_DESCRIPTION_GHES` naming downloaded copies of github/rest-api-description (`api.github.com.json`, `ghes-3.19.json`), every route of the table exists there with its method and every query parameter the driver sends is declared (CI job `contracts`). @@ -398,11 +398,11 @@ gh api -i "repos///pulls?state=all&per_page=2&page=2" | grep -i '^l gh api graphql -f query='query { a: repository(owner: "", name: "") { databaseId } b: repository(owner: "", name: "") { databaseId } }' ``` -### Live sandbox scenarios +### What only a live GitHub confirms -What only a live GitHub confirms, above all which writes count against GitHub's content-creation limits and when a push needs the Workflows permission. Each item says how to run it and what to record; the findings correct the fake (`ghfake`), and the driver where it assumed wrong. +What `ghfake` assumes and only a live GitHub settles, above all which writes count against GitHub's content-creation limits and when a push needs the Workflows permission. No sandbox organization runs these items, and none is planned. touchmark runs on github.com for real in the bedrock-python hub ([bedrock-python/engineering-assets](https://github.com/bedrock-python/engineering-assets)), whose runs since 2026-10-07 take the positive paths of items 15 to 17 and 21: per-target tokens, the App's bot as the author of the sync commits, the environment `touchmark-distribute` that only `master` may use, with `probe` and `doctor` reading it, the step summary, and the targets listed through the installation. The rest stays assumed. Each item says what to observe and record; a finding corrects the fake, and the driver where it assumed wrong. -The sandbox: an organization on the Free plan with public repositories (and one private repository, for drafts), at most 10 targets, writes at most once a second, and three GitHub Apps installed on it: +The items name three GitHub Apps: | App | Permissions | Used by | |---|---|---| @@ -410,7 +410,7 @@ The sandbox: an organization on the Free plan with public repositories (and one | WRITE | contents, pull requests, workflows: write (metadata read); installable without workflows | distribute, the writer | | FIXTURE | administration, contents, pull requests, workflows: write | setup only: repositories, rulesets, people's actions through a person's token | -plus a person's account with a fine-grained personal access token on the sandbox, and a second organization for forks. The harness to build: a live mode of `internal/e2e/github` behind the build tag `e2e` and `TOUCHMARK_E2E_GITHUB_*` variables (the organization, the Apps' ids and keys, the person's token), whose fixture implements the same `conformance.Fixture` and scenario as the fake's, as `internal/e2e/gitlab` does for GitLab, printing `FINDING` lines. Tokens reach it through the environment only. The measurements below that need bursts of writes are separate scripts with `gh api` or `curl`, run by hand. +and some need a person's account with a fine-grained personal access token, and a second organization for forks. 1. **Content-creation accounting**. GitHub limits content creation to 80 a minute and 500 an hour and does not say which writes count. With WRITE's per-target token on a scratch repository, make 81 writes of one kind within a minute, one kind per run, and see whether the 81st is refused ("You have exceeded a secondary rate limit", 403 or 429, `retry-after`): create a pull request, edit its body, close and reopen it, comment, create a label, add labels (`POST issues/{n}/labels`), `POST /git/commits`, the `updateRefs` mutation, `POST`/`PATCH`/`DELETE git/refs`, a git push of a new branch, a force push, a push to `refs/touchmark/…`, a token mint and `DELETE /installation/token`. Record which kinds count, and correct the fake's `limits.go` (its counted kinds are partly assumed) and the write estimates of `plan`. 2. **When Workflows is needed**. WRITE installed without workflows, pushing with its per-target token: (a) a new branch that adds `.github/workflows/x.yml`; (b) a new branch without workflow changes from a default branch that has workflows; (c) a push to `refs/touchmark//stage` whose tree changes a workflow; (d) a force push that moves a sync branch onto a new base carrying a person's workflow change (a rebuild across others' changes; public reports say it needs Workflows); (e) the branch deleted and pushed afresh from that base (the path that recreates a sync branch, unverified); (f) `POST /git/commits` with a tree that changes a workflow, then `updateRefs` moving the branch to it. Record allowed or refused and the exact message (git's stderr, GraphQL's error type) for each; repeat with workflows granted (all allowed). The fake judges a ref write by its diff from the old tip (from the default branch for a new ref): it refuses (a), (c), (d) and the `updateRefs` of (f), and allows (b) and (e) (assumed). @@ -428,7 +428,7 @@ plus a person's account with a fine-grained personal access token on the sandbox 14. **Body limit**. Bodies of 65 536 and 65 537 ASCII characters, and of 65 536 two-byte characters: accepted or refused, and the message; whether GitHub counts code points, UTF-16 units or bytes. touchmark's budget is 58 000 bytes. 15. **Tokens**. `DELETE /installation/token` → 204, and the token then fails at once on REST, GraphQL and git; a per-target token (`repository_ids=[id]`) on another repository: REST, GraphQL and git answers; the length of github.com's installation tokens (about 520 characters since 2026-04-27) and that every output masks them; the mint's 422 messages for a permission the installation lacks and for a repository outside it (the driver maps them to permission and not-found); whether an installation token of an organization sees its public repositories outside the selection. 16. **The App's identity**. `GET /app` (the slug), `GET /users/%5Bbot%5D` (the id), the commit email `+[bot]@users.noreply.github.com` attributes pushed commits to the bot; `GET /user` with an installation token answers 403 "Resource not accessible by integration" (`Self` refuses such a token with that message); on GHE.com and GHES the noreply domain (assumed `users.noreply.`). -17. **The hub in GitHub Actions**. A hub repository in the sandbox with the environment `touchmark-distribute` restricted to the default branch and the write key as its secret: a job of another branch naming the environment is refused; `touchmark probe` fails with exit 2 when the key is also a repository secret; the hub channel reads the environment and its deployment branch policies with `GITHUB_TOKEN` and `permissions: actions: read` (the shapes were checked anonymously on public repositories only); the step summary and annotations; a public hub skips private targets. distribute's environment check fails closed: an environment set to "Protected branches only", or one the token cannot read, stops it with exit 2; confirm that a hub protecting its default branch only with a ruleset lets every branch deploy to a "Protected branches only" environment (the docs say so for a repository without branch protection rules). +17. **The hub in GitHub Actions**. A hub repository with the environment `touchmark-distribute` restricted to the default branch and the write key as its secret: a job of another branch naming the environment is refused; `touchmark probe` fails with exit 2 when the key is also a repository secret; the hub channel reads the environment and its deployment branch policies with `GITHUB_TOKEN` and `permissions: actions: read` (the shapes were checked anonymously on public repositories only); the step summary and annotations; a public hub skips private targets. distribute's environment check fails closed: an environment set to "Protected branches only", or one the token cannot read, stops it with exit 2; confirm that a hub protecting its default branch only with a ruleset lets every branch deploy to a "Protected branches only" environment (the docs say so for a repository without branch protection rules). 18. **git history**. `git fetch --shallow-since` (`deepen-since`) against github.com with a per-target token: the history the core reads to classify the sync branch. 19. **Renames and transfers.** A renamed and a transferred target: GET follows the 301 to `/repositories/` with an installation token; git on the old path. A target transferred from an owner without the App to one with it: `GET /repos/{old owner}/{repo}/installation` with the JWT follows the redirect (the driver's fallback when the owner has no installation). An organization renamed while `targets.yml` names `repo: /`: API requests under the old name answer 404 (docs), the target is `target-missing`, and the sweep must leave its pull request, listed under the new name, open (`decide.Sweep` skips repositories named like an unresolved target). 20. **Pull request edge cases the fake assumes**: reopening after the head branch was deleted or force-pushed (the message); `PATCH /pulls/{n}` with title, body, state and base when the base is invalid (all or nothing?); the head moved behind its base; the base deleted (retargeted or closed); labels added by name that do not exist yet (created?). @@ -438,4 +438,4 @@ plus a person's account with a fine-grained personal access token on the sandbox 24. **The scenarios every platform runs**, with the live fixture, as `internal/e2e/gitlab` runs them for GitLab: a fork's pull request with the branch's name and a copy of the marker is not touchmark's (`TestDistribute` covers it on the fake); someone else's open pull request on the sync branch gives `branch-in-use`; a merge of a branch other than the base into the sync branch gives `blocked:edited`; a hub rolled back to a declined version after a merge opens a pull request; a change of the hub's `id` keeps the pull request and the memory (the old branch as an alias); an `ignore` edit in the opt-in file lifts a decline and a comment edit does not; a title edit or taking a pull request out of draft causes no write; a new engine version does not rewrite bodies. On the fake these run in the core's tests (`internal/distribute`, property and sequence tests) and, for GitHub, only partly in `TestDistribute`. 25. **Message fidelity of `POST /git/commits`**. The message of the commit GitHub makes must come back byte for byte (the trailing newline, the blank line before the trailers, the trailers): after an answer lost, the core compares the branch's head message with the one it sent (`commitApplied`), and a normalized message would turn every lost-answer reconcile into a conflict. Record the message of `GET /git/commits/{sha}` and of `git cat-file` for a message with and without a trailing newline, with CRLF, and with trailing blank lines. 26. **The writes an API commit costs**. Count the content-creating requests per target on the fake's and the live journal (`POST /git/commits` twice when the first answer is unsigned, `updateRefs`, the split path's second `updateRefs` and `DELETE /git/refs`), and compare with the report's `writes` and the estimate `writesAPICommit` (3). -27. **`touchmark setup github`**, with an owner of the sandbox organization in a real browser: GitHub's "new App" page takes the manifest from setup's page and redirects to `http://127.0.0.1:/callback` with the code and the state (the fake assumes it accepts a loopback `redirect_url` and an inactive webhook whose URL is the hub's); the App gets exactly `default_permissions` (metadata read included, nothing more); the conversion's answer has `id`, `slug`, `pem`, `client_id`, `client_secret` and `webhook_secret`. Then, with the organization owner's token: `PUT …/environments/touchmark-distribute` with custom branch policies on a public hub, and what a private hub on the Free plan answers (the fake answers 422, assumed); whether a `PUT` without `reviewers` and `wait_timer` clears them (the fake clears them, so setup sends them back); a duplicate deployment branch policy (303 in the docs, 422 in the fake); `POST …/actions/variables` for an existing name (409 in the docs); `POST …/rulesets` on a Free private hub (the fake answers 403 "Upgrade to GitHub Pro…", from reports); `gh secret set … --env touchmark-distribute < writer.pem` and the secrets listing setup reads afterwards; and `doctor --hub-token` grading the result ok. +27. **`touchmark setup github`**, with an organization owner in a real browser: GitHub's "new App" page takes the manifest from setup's page and redirects to `http://127.0.0.1:/callback` with the code and the state (the fake assumes it accepts a loopback `redirect_url` and an inactive webhook whose URL is the hub's); the App gets exactly `default_permissions` (metadata read included, nothing more); the conversion's answer has `id`, `slug`, `pem`, `client_id`, `client_secret` and `webhook_secret`. Then, with the organization owner's token: `PUT …/environments/touchmark-distribute` with custom branch policies on a public hub, and what a private hub on the Free plan answers (the fake answers 422, assumed); whether a `PUT` without `reviewers` and `wait_timer` clears them (the fake clears them, so setup sends them back); a duplicate deployment branch policy (303 in the docs, 422 in the fake); `POST …/actions/variables` for an existing name (409 in the docs); `POST …/rulesets` on a Free private hub (the fake answers 403 "Upgrade to GitHub Pro…", from reports); `gh secret set … --env touchmark-distribute < writer.pem` and the secrets listing setup reads afterwards; and `doctor --hub-token` grading the result ok. diff --git a/docs/project/release.md b/docs/project/release.md index 142c8c7..db8c842 100644 --- a/docs/project/release.md +++ b/docs/project/release.md @@ -130,10 +130,10 @@ When something fails: ## After a release: the template and the hubs -The hub template (`bedrock-python/engineering-assets-template`) and the bedrock-python hub name touchmark by commit and image digest. Until the first release they carry placeholders marked `TODO(release)`: `bedrock-python/touchmark@0000… # v0.1.0` in `.github/workflows`, and `ghcr.io/bedrock-python/touchmark:0.1.0@sha256:0000…` in `.gitlab-ci.yml` and `.gitea/workflows`. A hub created from a template with placeholders fails at the Action's step or at the image pull. +The hub template (`bedrock-python/engineering-assets-template`) and the bedrock-python hub name touchmark by commit and image digest: the Action in `.github/workflows` by the commit of its tag, with `# vX.Y.Z` after it, and in the template the image as `ghcr.io/bedrock-python/touchmark:X.Y.Z@` in `.gitlab-ci.yml` and `.gitea/workflows`. Dependabot proposes the next release of the Action and Renovate the image's, after their cooldown; a release of our own can be pinned at once: -1. In each of them, replace every placeholder with the release: the tag's commit (`git rev-parse vX.Y.Z^{commit}`, with `# vX.Y.Z` after it) and the image `ghcr.io/bedrock-python/touchmark:X.Y.Z@`, the digest from `docker buildx imagetools inspect ghcr.io/bedrock-python/touchmark:X.Y.Z --format '{{ .Manifest.Digest }}'`, and delete the `TODO(release)` lines. Afterwards Dependabot moves the Action and Renovate the image. -2. Run `bash scripts/validate.sh --release` there: it fails while a placeholder is left. +1. In each of them, pin the release: the tag's commit (`git rev-parse vX.Y.Z^{commit}`) and the image's digest from `docker buildx imagetools inspect ghcr.io/bedrock-python/touchmark:X.Y.Z --format '{{ .Manifest.Digest }}'`. +2. Run `bash scripts/validate.sh --release` there: it checks that touchmark is pinned to a release everywhere. 3. Run the template's pipelines against the published image ([e2e.md](e2e.md), "The hub template"): `scripts/e2e/gitea.sh` and `scripts/e2e/gitlab.sh` with `--template