From 32bcba490c3f5e5435afaf2d1c252fb4c821336a Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 14:27:50 -0700 Subject: [PATCH 1/7] docs(claude): rewrite CLAUDE.md in STE, link the Writing rules C77: CLAUDE.md in ASD-STE100. Every constraint, must-know and check keeps its fact; the Tick step list now names prepare as well. C75: the Constraints list gains one line that points to the Writing section of the org CONTRIBUTING, so an agent reads it before it writes. --- CLAUDE.md | 135 ++++++++++++++++++++++++++++++++---------------------- 1 file changed, 79 insertions(+), 56 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index dddfcd7..f994de6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,76 +1,99 @@ # dispatch -Starts the katoptra mirrors' workflows on UTC slots, from a systemd timer on a NixOS host. -`README.md` is for users; this file is the design. +This repository is the scheduler. It starts the workflows of the katoptra mirrors at UTC +slots. A systemd timer on a NixOS host starts it. `README.md` is for users. This file is +the design. ## The files -- `schedules/` -- `Slot` (a bitmask of UTC hours, each at `HH:42`), the named aliases, - `Latest`, `Job`, and one file per katoptra repo registering its jobs. Routine changes - touch only this directory. -- `tick.go` -- the state file, `Plan`, and `Tick`: lock, load, plan, write ahead, dispatch. -- `github.go` -- App JWT, the org's installation, installation token, `workflow_dispatch`, - retries. No SDK. -- `main.go` -- flags, credentials from `$CREDENTIALS_DIRECTORY`, the healthcheck ping, the - exit code. -- `module.nix` -- the timer and the hardened oneshot. `flake.nix` -- package, module, checks. -- `Taskfile.yml`, `Dockerfile` -- every Go command runs in the toolbox image. +- `schedules/`: `Slot` (a bitmask of UTC hours, each at `HH:42`), the slot names (`Hourly`, + `Evening`, `Overnight`, `Morning`, `Afternoon`), `Latest`, `Job`, and one file for each + katoptra repository. Each of these files adds the jobs of its repository. A usual change + is only in this directory. +- `tick.go`: the state file, `Plan`, and `Tick`. `Tick` does these steps in this sequence: + lock, load, plan, prepare, record (the write before the first POST), dispatch. +- `github.go`: the App JWT, the installation of the App on the organization, the + installation token, `workflow_dispatch`, and the retries. It uses no SDK. +- `main.go`: the flags, the credentials from `$CREDENTIALS_DIRECTORY`, the healthcheck ping, + and the exit code. +- `module.nix`: the timer, and the oneshot service with its systemd protections. +- `flake.nix`: the package, the module, and the checks. +- `Taskfile.yml` and `Dockerfile`: each Go command operates in the toolbox image. ## Constraints -- It dispatches. It does not run work or watch outcomes; each mirror pings its own - healthcheck. -- katoptra repositories only; the App is installed on the `katoptra` org alone. -- Standard library only (`vendorHash = null`). Go 1.26, matching nixos-26.05's - `buildGoModule`; the toolbox image pins the same. -- UTC throughout. No DST handling anywhere. -- No secrets in the repo, the store, or the state file. The module takes file paths. +- The scheduler only dispatches. It does not do the work of a run, and it does not monitor + the result of a run. Each mirror sends pings to its healthcheck. +- It dispatches workflows only in katoptra repositories. The App is installed only on the + `katoptra` organization. +- It uses only the Go standard library (`vendorHash = null`). +- It uses Go 1.26, the same version as the `buildGoModule` of nixos-26.05. The toolbox image + also uses Go 1.26: its `FROM` line holds that version and a digest. +- All times are in UTC. No part of the code adjusts a time for DST. +- The repository, the Nix store and the state file contain no secrets. The module gets file + paths, not secrets. +- Writing: obey ASD-STE100 and the rules in the + [Writing section of the org CONTRIBUTING](https://github.com/katoptra/.github/blob/main/CONTRIBUTING.md#writing). + Read that section before you write. ## Must knows -- **At most once, by writing ahead.** `Tick` records every due slot in `state.json` before - the first POST. A crash or a failed dispatch loses that slot; nothing retries it, the next - slot is the retry. Moving the write after the dispatch turns this into at-least-once, and - `TestSlotIsRecordedBeforeTheDispatch` fails. -- **The token comes before the write.** `Dispatcher.Prepare` mints the installation token - before anything is recorded, since minting starts no run. A GitHub or network outage there - leaves the slots unrecorded and the next tick tries again, instead of losing a daily - mirror's day. `TestPrepareFailureRecordsNothing` holds it. -- **Catch-up is `Plan`, not systemd.** The timer has no `Persistent=`. Every tick compares - each job's latest slot with its recorded one, so any number of missed slots fire once. - A timer per slot with `Persistent=true` would fire each missed slot instead. -- **A corrupt `state.json` stops everything.** It is never treated as empty: empty state - fires every job for slots already sent. Repair or delete it by hand; deleting fires each - job once. -- **A new job fires at the next tick**, for its latest slot, because it has no entry. -- **The timer is `*:02/5 UTC`.** It starts at :02 so `:42` is a tick; `UTC` because the - jgrid.net hosts run in `America/Los_Angeles`. Changing `schedules.Minute` means changing - the timer too. -- **Every dispatch is `ref: main`, empty inputs.** A repo whose default branch is not - `main` answers 404, which is fatal. -- **Retries (10 s, 20 s, 40 s) depend on whether a repeat is harmless.** A 4xx other than - 408/429 is never retried. The two mint requests retry any 5xx, 408, 429 or network error. - A dispatch POST retries only 408, 429 and a connection that never opened: a 5xx or a - timeout can follow a dispatch GitHub already accepted, and a retry would be a second run. -- **GitHub gets two minutes a tick** (`githubBudget`), retries included; a retry that would - pass the deadline is not made. The rest of `TimeoutStartSec=4min` is the ping's, so a - failing tick still reaches `/fail`. -- **The binary is renamed in `postInstall`.** `buildGoModule` names it after the module - path's last element, `dispatch`; the unit runs `katoptra-dispatch`. The `module` check - fails if `ExecStart` is not an executable. -- **Journal priorities** (`<3>`, `<4>`) are written only when `JOURNAL_STREAM` is set, so - a hand run prints plain lines. +- **At-most-once, because the write comes first.** Before the first POST, `Tick` records in + `state.json` the slot of each job that it will dispatch. After a crash or an error in a + workflow start, that slot gets no run. The scheduler does not try that slot again: the + next slot is the retry. If you move the write after the POST, a slot can get two runs + (at-least-once). `TestSlotIsRecordedBeforeTheDispatch` finds that error. +- **The token comes before the write.** `Dispatcher.Prepare` gets the installation token + before `Tick` records a slot, because a token request starts no run. If GitHub or the + network is not available at that step, `Tick` records no slot, and the next tick tries + again. Thus, a daily mirror gets its run for that day. + `TestPrepareFailureRecordsNothing` makes sure that this sequence does not change. +- **`Plan`, not systemd, finds the slots that had no run.** The timer has no + `Persistent=`. Each tick compares the latest slot of each job with the slot that the + state records for it. Thus, a job that had no run for one or more slots gets one run. If + each slot has its timer with `Persistent=true`, systemd starts a run for each of these + slots. +- **A corrupted `state.json` stops all jobs.** `LoadState` gives an error for it, not empty + state, because empty state starts each job again for slots that had a run. Repair or + delete the file manually. If you delete it, each job starts one time. +- **A new job starts at the next tick**, for its latest slot, because the state has no entry + for it. +- **The timer is `*:02/5 UTC`.** The timer starts at :02. Thus, `:42` is a tick. The + calendar has `UTC` because the time zone of the jgrid.net hosts is `America/Los_Angeles`. + If you change `schedules.Minute`, also change the timer. +- **The scheduler dispatches on `ref: main`, with empty inputs.** If the default branch of + a repository is not `main`, GitHub sends a 404. The scheduler does not try again, and + that slot gets no run. +- **A retry (after 10 s, 20 s, 40 s) occurs only when a second request is safe.** + - A 4xx other than 408 and 429 gets no retry. + - The two token requests try again after a 5xx, 408, 429, or network error. + - A workflow start tries again only after a 408, a 429, or a connection that did not + open. A 5xx or a timeout can come after GitHub accepted the workflow start, and a retry + then starts a second run. +- **GitHub gets two minutes in each tick** (`githubBudget`), with the retries included. If + a retry cannot start in the two minutes, the scheduler does not make it. The other two + minutes of `TimeoutStartSec=4min` are for the ping. Thus, a tick with errors can send its + ping to `/fail`. +- **`postInstall` changes the name of the binary.** `buildGoModule` gives the binary the + name of the last part of the module path: `dispatch`. The unit starts + `katoptra-dispatch`. If `ExecStart` is not an executable, the `module` check stops with + an error. +- **`Log` writes the journal priorities** (`<3>`, `<4>`) only when `JOURNAL_STREAM` is set. + Thus, a manual start of the binary prints its lines without these prefixes. ## Verifying a change -`task check` is what CI runs: gofmt, go vet, go test, in the toolbox image. +CI uses `task check`: gofmt, go vet and go test, in the toolbox image. -`nix flake check` builds the package (its tests run in `checkPhase`) and renders the two -units from a minimal system. Without nix on the laptop: +`nix flake check` builds the package and makes the two units from a small NixOS system +that has the module. The tests of the package operate in `checkPhase`. If the laptop does +not have nix, use this command: container run --rm --memory 4G -v "$PWD":/work -w /work nixos/nix sh -c \ 'git config --global --add safe.directory /work && NIX_CONFIG="experimental-features = nix-command flakes" nix flake check' -Files must be `git add`ed for the flake to see them. Only a real tick proves the GitHub App -contract: on the host, `sudo systemctl start katoptra-dispatch` and read the journal. +The flake uses only the files in the git index. Use `git add` on each new file first. + +Only a tick on the host can show that the GitHub App contract is correct. On the host, use +`sudo systemctl start katoptra-dispatch`. Then read the journal. From d312bebb8c9fb6f39048248cc4a7b4fc73d23720 Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 14:27:50 -0700 Subject: [PATCH 2/7] docs(readme): rewrite the README in STE C77: the README in ASD-STE100. "How it works" and "When something goes wrong" keep their headings, because other repos link to them. "Running it" and "Run your own" become "Operating it" and "Want your own?", the names the mirror READMEs use; nothing links to the old anchors. The secrets note becomes a caution. Code blocks keep their commands; only the comments in the task block change. --- README.md | 169 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 93 insertions(+), 76 deletions(-) diff --git a/README.md b/README.md index e275886..8ea1d45 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@

dispatch

-

Runs the katoptra mirrors on a schedule.

+

The scheduler that starts the katoptra mirrors.

check @@ -17,48 +17,53 @@ tick

-Each katoptra mirror is a GitHub Actions workflow with no schedule of its own. This repo -starts them. +Each katoptra mirror is a GitHub Actions workflow that has no schedule. The scheduler in +this repository starts these workflows. -- A systemd timer on a NixOS host runs it every five minutes. -- Each run starts any workflow that is due, then pings a healthcheck. -- If the host goes down, each job that missed a run fires once when it comes back. -- A job never fires twice for the same time slot. -- It is one Go binary with no dependencies outside the standard library. +- A systemd timer on a NixOS host starts a tick at intervals of five minutes. +- Each tick finds the jobs to start and starts their workflows. Then the tick sends a ping + to a healthcheck. +- If the host stops, each job that had no run for its slots gets one run when the host + starts again. +- A job does not get two runs for the same slot. +- The scheduler is one Go binary, and it uses only the Go standard library. ## Adding a job -Each repo gets one file in [`schedules/`](schedules), named after the repo: +Each repository has one file in [`schedules/`](schedules). The file has the name of the +repository: ```go // schedules/ctan.go var _ = register(Job{Repo: "katoptra/ctan", File: "sync.yml", Slots: Hourly}) ``` -Every slot runs at 42 minutes past the hour, UTC: +Each slot is at minute 42 of the hour, UTC: | Slot | UTC | Pacific (winter) | |---|---|---| -| `Hourly` | every hour at :42 | every hour at :42 | +| `Hourly` | each hour at :42 | each hour at :42 | | `Evening` | 05:42 | 21:42 | | `Overnight` | 11:42 | 03:42 | | `Morning` | 17:42 | 09:42 | | `Afternoon` | 23:42 | 15:42 | -- `S0` to `S23` name each hour. The four daily names are shortcuts for `S5`, `S11`, `S17` - and `S23`. -- Combine slots with `|`, like `Morning | Evening`. -- A misspelled slot won't compile. +- `S0` to `S23` are the names of the hours. The four daily names are the same slots as + `S5`, `S11`, `S17` and `S23`. +- To put slots together, use `|`, for example `Morning | Evening`. +- An incorrect slot name causes a compile error. -The workflow needs three things that this repo can't check for you: +This repository cannot examine the workflow. Make sure that the workflow has these three +items: -- `workflow_dispatch:` under `on:`. -- A `concurrency` group with `cancel-in-progress: false`, so a second start waits for the - first run to finish. -- Its own healthcheck ping. The scheduler starts runs but never sees whether they pass. +- It has `workflow_dispatch:` in `on:`. +- It has a `concurrency` group with `cancel-in-progress: false`. Thus, a second start waits + for the end of the first run. +- It sends a ping to its healthcheck. The scheduler starts runs, but it does not know the + result of a run. -A change goes live when the host's flake lock moves to the new commit. A new job fires on -the next tick. +The host uses a change when the host's flake lock moves to the new commit. A new job starts +at the next tick. ## How it works @@ -70,41 +75,46 @@ flowchart LR token -. "GitHub down, nothing saved" .-> ping ``` -The timer fires at :02, :07 and every five minutes after that. Each tick: +The timer starts a tick at :02, at :07, and at intervals of five minutes after that. Each +tick does these steps: -1. Loads `state.json`, which holds the last slot each job fired for. -2. Works out which jobs are due. A job that missed several slots is due once, for the - latest one. -3. Gets a token for the GitHub App. -4. Saves the new state. -5. Starts each due workflow on `main`. -6. Pings the healthcheck: the plain URL if everything worked, or `/fail` with the errors. +1. It loads `state.json`, which holds the last dispatched slot of each job. +2. It finds the jobs to start. If a job had no run for some slots, it starts one time, for + the latest slot. +3. It gets a token for the GitHub App. +4. It saves the new state. +5. It starts the workflow of each of these jobs on `main`. +6. It sends a ping to the healthcheck: to the URL if the tick has no errors, or to `/fail` + with the errors. -The state is saved before any workflow starts. If a tick crashes after that, the run for -that slot is lost and the job runs again at its next slot. No run ever starts twice. If -GitHub is down when the token is requested, nothing is saved and the next tick tries again. +The tick saves the state before it starts a workflow. If a crash stops the tick after that +step, the job gets no run for that slot. The job starts again at its next slot. Thus, a run +does not start two times. If GitHub is not available when the tick sends the token request, +the tick does not save the state. Then the next tick tries again. -Retries wait 10, 20 and then 40 seconds: +Before each retry, the scheduler waits 10, 20, and then 40 seconds. The retry rules are: -- Token requests retry on any 5xx, 408, 429 or network error. -- Workflow starts retry only on 408, 429 or a connection that never opened. GitHub can - return a 5xx after it has already started the run, so a retry could start it twice. -- GitHub gets two minutes per tick, which leaves time to ping the healthcheck. +- A token request tries again after a 5xx, 408, 429, or network error. +- A workflow start tries again only after a 408, a 429, or a connection that did not open. + GitHub can send a 5xx after it started the run. Then a retry can start a second run. +- GitHub gets two minutes in each tick. Thus, the tick has time to send its ping to the + healthcheck. -The design and the reasons behind it are in [`CLAUDE.md`](CLAUDE.md). +[`CLAUDE.md`](CLAUDE.md) gives the design and the cause of each rule. -## Running it +## Operating it -`task` on its own prints the menu. From a laptop: +`task` without a task name prints the menu. Use these commands on a laptop: ```sh -task check # gofmt, go vet and go test in the toolbox image, same as CI -task targets # every job and when it runs -task runs # recent runs of each job on GitHub (needs gh logged in) -task runs LIMIT=10 # more of them +task check # gofmt, go vet and go test in the toolbox image, the same as CI +task targets # each job and its slot times, in UTC +task runs # the last 3 runs of each job on GitHub (gh must have a login) +task runs LIMIT=10 # more runs for each job ``` -Runs started by this repo show up as `workflow_dispatch`. On the host: +On GitHub, a run that the scheduler starts has the event `workflow_dispatch`. Use these +commands on the host: ```sh systemctl list-timers katoptra-dispatch.timer @@ -114,30 +124,34 @@ sudo cat /var/lib/private/katoptra-dispatch/state.json sudo STATE_DIRECTORY=/var/lib/private/katoptra-dispatch katoptra-dispatch --dry-run ``` -`--dry-run` shows what is due and why, without saving, starting or pinging anything. +`--dry-run` shows the jobs to start, with the slot of each and its last dispatched slot. It +does not save the state, start a workflow, or send a ping. ### When something goes wrong -- **The healthcheck goes quiet.** The host or its timer is down. When it comes back, each - job that missed a slot fires once. -- **The healthcheck gets a `/fail` ping.** The error is in the ping body and in - `journalctl -u katoptra-dispatch -p err`. That slot is skipped, and the job runs again - at its next slot. -- **A workflow start returns 404.** The repo's default branch isn't `main`, or the workflow - file isn't there. -- **Every tick fails on `state.json`.** The file is corrupt, and nothing runs until it is - fixed. Repair it by hand or delete it. Deleting it makes every job fire once. - -## Run your own - -1. **Fork this repo** and replace the files in `schedules/` with your own jobs. The tests - only accept `katoptra/` repos, so change that prefix to your organization's. -2. **Create a GitHub App** owned by your organization, with one permission: Actions, read - and write. It needs no webhook. Install it on every repo in the organization so new - repos are covered automatically. Note the App ID and generate a private key. The key - works as GitHub issues it. -3. **Create a healthchecks.io check** with a 5 minute period and a 10 minute grace. -4. **Add the flake to your NixOS host** and give the module three files: +- **The healthcheck gets no pings.** The host or its timer does not operate. When the host + and the timer operate again, each job that had no run for a slot gets one run. +- **The healthcheck gets a `/fail` ping.** The body of the ping and + `journalctl -u katoptra-dispatch -p err` show the error. The job gets no run for that + slot. It starts again at its next slot. +- **A workflow start gets a 404.** The default branch of the repository is not `main`, or + the repository does not have the workflow file. +- **Each tick stops with an error about `state.json`.** The file is corrupted. No job starts + until you repair the file. Repair or delete the file manually. If you delete it, each job + starts one time. + +## Want your own? + +1. **Fork this repository.** Replace the files in `schedules/` with files for your jobs. + The tests accept only `katoptra/` repositories. Change that prefix in the tests to the + name of your organization. +2. **Make a GitHub App in your organization.** Give it one permission: Actions, read and + write. A webhook is not necessary. Install the App on all repositories in the + organization. Then each new repository also gets the App automatically. Record the App + ID. Make a private key. You can use the key in the format that GitHub gives. +3. **Make a healthchecks.io check** with a period of 5 minutes and a grace time of 10 + minutes. +4. **Add the flake to your NixOS host.** Give the module the paths of three files: ```nix { @@ -151,15 +165,18 @@ sudo STATE_DIRECTORY=/var/lib/private/katoptra-dispatch katoptra-dispatch --dry- } ``` - Keep the paths as strings. A Nix path would copy the secrets into the Nix store, where - anyone on the host can read them. The files are loaded with `LoadCredential=`, so - root-owned files with mode 0400 work. -5. **Check it.** On a laptop with go-task and Docker or Apple `container`, run `task check` - and `task targets`. `nix flake check` builds the package and the systemd units; - [`CLAUDE.md`](CLAUDE.md) shows how to run it without nix installed. The only real test - of the App is a live tick: run `sudo systemctl start katoptra-dispatch` on the host and - read the journal. + **Caution:** Write each path as a string, not as a Nix path. A Nix path puts a copy of the + secret in the Nix store. All users on the host can read the Nix store. + + systemd loads the files with `LoadCredential=`. Thus, the files can have the owner `root` + and the mode 0400. +5. **Make sure that the scheduler operates.** On a laptop with go-task and Docker or Apple + `container`, use `task check` and `task targets`. `nix flake check` builds the package + and the systemd units. [`CLAUDE.md`](CLAUDE.md) shows how to use it without nix. Only a + tick on the host is a full test of the App. On the host, use + `sudo systemctl start katoptra-dispatch`. Then read the journal. ## License -MIT. Built by [Josh Vaughen](https://ijosh.com). Pull requests are welcome. +The license is MIT. [Josh Vaughen](https://ijosh.com) made this repository. You can send a +pull request. From 47661a17867df82d0f954678b9a2fe22cb19490f Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 14:27:50 -0700 Subject: [PATCH 3/7] docs(comments): rewrite code comments in STE C78: every comment in the Go files, Taskfile.yml, Dockerfile, module.nix and flake.nix in ASD-STE100. Each comment keeps its place and its job, and Go doc comments still open with their names. No identifier, string or behavior changes: the Go token streams and the parsed Nix ASTs match origin/main. C63: Taskfile.yml, module.nix and flake.nix say "the host's flake", not the private repository's name. --- Dockerfile | 8 +++--- Taskfile.yml | 27 ++++++++++--------- flake.nix | 19 +++++++------ github.go | 44 ++++++++++++++++-------------- github_test.go | 3 ++- main.go | 27 +++++++++++-------- module.nix | 16 ++++++----- schedules/ctan.go | 2 +- schedules/dropbox.go | 5 ++-- schedules/github.go | 2 +- schedules/gnu.go | 2 +- schedules/nongnu.go | 2 +- schedules/schedules.go | 57 ++++++++++++++++++++++----------------- schedules/tlnet.go | 2 +- tick.go | 61 +++++++++++++++++++++++------------------- tick_test.go | 4 +-- 16 files changed, 156 insertions(+), 125 deletions(-) diff --git a/Dockerfile b/Dockerfile index 59437bc..1277b88 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,6 +1,6 @@ -# The toolbox: every check runs in here, on a laptop and in CI alike. Go 1.26 because the -# host builds with nixos-26.05's buildGoModule, which is Go 1.26; a newer toolchain here -# could accept code the host's build refuses. +# The toolbox: each check operates in this image, on a laptop and in CI. It uses Go 1.26 +# because the host builds with the buildGoModule of nixos-26.05, which is Go 1.26. A newer +# toolchain here can accept code that the build on the host rejects. FROM golang:1.26-alpine@sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7236c349a0c -# No .git inside the mount is guaranteed, and nothing reads the stamp. +# The mount does not always contain .git, and no code reads the VCS stamp. ENV GOFLAGS=-buildvcs=false diff --git a/Taskfile.yml b/Taskfile.yml index 3f94119..6fbbfc2 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -1,9 +1,10 @@ -# katoptra-dispatch: starts the katoptra mirrors' workflows on UTC slots, from a systemd -# timer on a jgrid.net host. Everything here is a local check or a read; nothing deploys. -# The host picks up a change when jshvn/jgrid.net bumps this flake in its lock. +# katoptra-dispatch starts the workflows of the katoptra mirrors at UTC slots, from a +# systemd timer on a jgrid.net host. Each task here is a local check or a read. No task +# deploys a change. The host gets a change when the host's flake moves its lock to a new +# commit of this flake. # -# Every Go command runs inside the toolbox image (Dockerfile), so the host supplies a -# container engine and go-task and nothing else. The design is CLAUDE.md. +# Each Go command operates in the toolbox image (Dockerfile). Thus, a laptop or a CI runner +# must supply only a container engine and go-task. CLAUDE.md gives the design. version: '3' vars: @@ -16,12 +17,13 @@ vars: else echo docker fi IMAGE: katoptra-dispatch-toolbox:local - # Apple container draws a progress spinner on a terminal; Docker's run has no such flag. + # Apple container shows a progress spinner on a terminal. `docker run` has no such flag. QUIET: '{{if eq .ENGINE "container"}}--progress none{{end}}' tasks: - # The menu. ponytail: hand-maintained listing -- go-task can neither group nor annotate, - # so a new task means a line below. `task --list` stays the generated view. + # The menu. ponytail: a menu that we keep manually. go-task cannot put tasks in groups or + # add notes to them. Thus, add a line below for each new task. `task --list` gives the + # view that go-task makes. default: desc: Print the menu silent: true @@ -64,8 +66,8 @@ tasks: desc: Build the toolbox image run: once silent: true - # Presence is the freshness check, so a Dockerfile edit does not rebuild on its own: - # task clean, then any task. + # If the image is there, task does not build it again. Thus, a change to the Dockerfile + # does not cause a new build. Do `task clean`, then use a task. status: - '{{.ENGINE}} image inspect {{.IMAGE}} >/dev/null 2>&1' cmds: @@ -99,8 +101,9 @@ tasks: vars: LIMIT: '{{.LIMIT | default 3}}' cmds: - # GH_PAGER=cat: on a terminal gh opens its pager for each target and waits on it. - # An unfinished run has an empty conclusion, so its status stands in. + # GH_PAGER=cat: on a terminal, gh opens its pager for each target and waits for it. + # A run in progress has an empty `conclusion`. Thus, the table shows its `status` as + # an alternative. - | task run -- go run . --list | awk 'NR > 1 { print $1 }' | while read -r id; do GH_PAGER=cat gh run list --repo "${id%/*}" --workflow "${id##*/}" --limit {{.LIMIT}} \ diff --git a/flake.nix b/flake.nix index b714148..0fdbd64 100644 --- a/flake.nix +++ b/flake.nix @@ -1,7 +1,8 @@ { description = "katoptra-dispatch: starts the katoptra mirrors' workflows on UTC slots"; - # The same release jshvn/jgrid.net builds its hosts from, so the host's lock can follow it. + # The same release that the host's flake builds its hosts from. Thus, the host's lock + # can follow it. inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05"; outputs = @@ -26,10 +27,11 @@ (lib.fileset.fileFilter (f: f.hasExt "go") ./.) ]; }; - vendorHash = null; # standard library only - # go test runs in buildGoModule's checkPhase; vet first, so either fails the host's build + vendorHash = null; # only the standard library + # go test operates in the checkPhase of buildGoModule. go vet operates first. Thus, + # an error from go vet or go test stops the build on the host. preCheck = "go vet ./..."; - # the module path's last element would name it `dispatch` + # without this step, the name is `dispatch`, the last part of the module path postInstall = "mv $out/bin/dispatch $out/bin/katoptra-dispatch"; meta.mainProgram = "katoptra-dispatch"; }; @@ -37,9 +39,9 @@ nixosModules.default = import ./module.nix self; - # The package (its tests included), and the module's two units rendered from a minimal - # system: evaluation alone, no VM. ponytail: a nixosTest would boot the timer for real; - # add one if a unit mistake ever gets past this. + # The package (with its tests), and the two units of the module, made from a small + # system. Nix makes the units, but it does not boot a VM. ponytail: a nixosTest boots + # the timer in a VM. If this check does not find a unit error, add a nixosTest. checks = forAll ( pkgs: let @@ -68,7 +70,8 @@ in { package = self.packages.${system}.default; - # the rendered units, failing if ExecStart names anything but an executable + # the two units. This check stops with an error if ExecStart is not an + # executable. module = pkgs.runCommand "katoptra-dispatch-units" { } '' service=${units."katoptra-dispatch.service".unit}/katoptra-dispatch.service exe=$(sed -n 's/^ExecStart=//p' "$service") diff --git a/github.go b/github.go index 481dca7..4702547 100644 --- a/github.go +++ b/github.go @@ -1,7 +1,7 @@ package main -// The GitHub App's side: sign a JWT, find the App's installation on the org, mint an -// installation token, dispatch a workflow. No SDK, three requests. +// The GitHub App side: sign a JWT, find the installation of the App on the org, get an +// installation token, and dispatch a workflow. It uses no SDK, only three HTTP requests. import ( "bytes" @@ -22,11 +22,11 @@ import ( "time" ) -// backoff is the wait before each retry of a transient failure: three retries, 70 s in all. -// A retry whose wait would pass the context's deadline is not made. +// backoff is the wait before each retry of a temporary error: three retries, 70 s in total. +// If a retry cannot start before the deadline of the context, call does not make it. var backoff = []time.Duration{10 * time.Second, 20 * time.Second, 40 * time.Second} -// StatusError is GitHub answering with something other than 2xx. +// StatusError is a GitHub response with a status other than 2xx. type StatusError struct { Status int Msg string @@ -34,12 +34,14 @@ type StatusError struct { func (e *StatusError) Error() string { return e.Msg } -// retryable is a failure another attempt may clear. Any other 4xx is a refusal no retry can -// change: a missing workflow file, an App without the permission, no installation. +// retryable tells if call can try again after err. With a 4xx other than 408 and 429, +// GitHub rejected the request, and a retry cannot change that. For example: a missing +// workflow file, an App without the permission, or no installation. // -// A request that starts a run (safe false) is retried only when GitHub cannot have acted on -// it: the connection never opened, or GitHub turned it away unread (408, 429). A 5xx or a -// timeout can follow a dispatch GitHub already accepted, and a retry would be a second run. +// call tries a request that starts a run (safe false) again only if GitHub did not do the +// request. This is true when the connection did not open, or when GitHub rejected the +// request before it read it (408, 429). A 5xx or a timeout can come after GitHub accepted +// the request. Then a retry starts a second run. func retryable(err error, safe bool) bool { var s *StatusError if errors.As(err, &s) { @@ -49,8 +51,8 @@ func retryable(err error, safe bool) bool { return safe || errors.As(err, &op) && op.Op == "dial" } -// ParseKey reads the App's private key. GitHub issues PKCS#1; PKCS#8 is accepted too, so a -// converted key still works. +// ParseKey reads the private key of the App. GitHub gives a PKCS#1 key. ParseKey also +// accepts PKCS#8. Thus, you can also use a converted key. func ParseKey(data []byte) (*rsa.PrivateKey, error) { block, _ := pem.Decode(data) if block == nil { @@ -70,8 +72,9 @@ func ParseKey(data []byte) (*rsa.PrivateKey, error) { return rk, nil } -// AppJWT signs a short-lived App JWT. Backdated 60 s because GitHub rejects a future iat, -// and 9 minutes long against GitHub's 10-minute ceiling. +// AppJWT signs an App JWT with a short life. Its iat is 60 s before now, because GitHub +// rejects an iat after the time on its servers. Its life is 9 minutes, less than the +// 10-minute maximum of GitHub. func AppJWT(appID string, key *rsa.PrivateKey, now time.Time) (string, error) { enc := base64.RawURLEncoding iat := now.Unix() - 60 @@ -88,12 +91,12 @@ func AppJWT(appID string, key *rsa.PrivateKey, now time.Time) (string, error) { return signed + "." + enc.EncodeToString(sig), nil } -// GitHub dispatches workflows as the App. Prepare mints one installation token and keeps it -// in memory only. +// GitHub dispatches workflows as the App. Prepare gets one installation token and keeps it +// only in memory. type GitHub struct { API string // https://api.github.com AppID string - Org string // the one installation this App has + Org string // the one installation of this App Key *rsa.PrivateKey HTTP *http.Client Sleep func(time.Duration) @@ -102,7 +105,7 @@ type GitHub struct { token string } -// Prepare mints the installation token, once per process. +// Prepare gets the installation token one time in each process. func (g *GitHub) Prepare(ctx context.Context) error { if g.token != "" { return nil @@ -148,8 +151,9 @@ func (g *GitHub) mint(ctx context.Context) (string, error) { return tok.Token, nil } -// call sends one request, retrying transient failures after each backoff step. safe says a -// repeat has no effect beyond the first; see retryable. +// call sends one request. After a temporary error, it waits for the next backoff step and +// tries again. safe is true when a second copy of the request has no more effect than the +// first copy. Refer to retryable. func (g *GitHub) call(ctx context.Context, method, path, auth string, body []byte, safe bool) ([]byte, error) { for attempt := 0; ; attempt++ { b, err := g.once(ctx, method, path, auth, body) diff --git a/github_test.go b/github_test.go index 5953c8a..48b9d7b 100644 --- a/github_test.go +++ b/github_test.go @@ -28,7 +28,8 @@ var testKey = func() *rsa.PrivateKey { return k }() -// github serves the three endpoints; dispatch answers with the statuses in order, then 204. +// github is a test server for the three endpoints. The /dispatches endpoint sends the +// statuses in dispatch, in sequence, and then 204. func github(t *testing.T, dispatch ...int) (*GitHub, *[]string, *[]time.Duration) { t.Helper() var hits []string diff --git a/main.go b/main.go index 8918de6..dfc9d3a 100644 --- a/main.go +++ b/main.go @@ -1,6 +1,7 @@ -// katoptra-dispatch is one tick of the katoptra scheduler: fire every job whose latest slot -// has not been fired yet, record it first, then ping the scheduler's healthcheck. A systemd -// timer runs it every five minutes; see module.nix. +// katoptra-dispatch is one tick of the katoptra scheduler. It finds each job with a latest +// slot that the state does not hold. It records that slot first, and then it starts the job. +// At the end, it sends a ping to the healthcheck of the scheduler. A systemd timer starts it +// at intervals of five minutes (refer to module.nix). package main import ( @@ -20,11 +21,13 @@ import ( "github.com/katoptra/dispatch/schedules" ) -// pingLimit caps the error text sent with a /fail ping. healthchecks.io keeps 100 kB. +// pingLimit is the maximum length of the error text in a /fail ping. healthchecks.io keeps +// 100 kB. const pingLimit = 10 << 10 -// githubBudget bounds everything GitHub gets in a tick, retries included, so the ping -// (at most 3 x 30 s + 4 s) still lands inside the unit's TimeoutStartSec=4min. +// githubBudget is the maximum time for all GitHub requests in a tick, with the retries +// included. Thus, the ping (a maximum of 3 x 30 s + 4 s) can complete in the +// TimeoutStartSec=4min of the unit. const githubBudget = 2 * time.Minute func main() { @@ -39,7 +42,7 @@ func main() { os.Exit(run(*dryRun, log)) } -// printJobs is the table `task targets` shows; `task runs` reads its first column. +// printJobs prints the table that `task targets` shows. `task runs` reads its first column. func printJobs(w io.Writer) { tw := tabwriter.NewWriter(w, 0, 0, 2, ' ', 0) fmt.Fprintln(tw, "JOB\tSLOT\tUTC") @@ -110,7 +113,7 @@ func newGitHub(creds string, client *http.Client) (*GitHub, error) { }, nil } -// credential reads one file systemd's LoadCredential= put in $CREDENTIALS_DIRECTORY. +// credential reads one file that systemd put in $CREDENTIALS_DIRECTORY with LoadCredential=. func credential(dir, name string) (string, error) { if dir == "" { return "", fmt.Errorf("credential %s: CREDENTIALS_DIRECTORY is not set", name) @@ -126,9 +129,11 @@ func credential(dir, name string) (string, error) { return v, nil } -// ping reports the tick to healthchecks.io: the URL itself when clean, /fail with the errors -// as the body otherwise. Three attempts; a miss is the caller's warning, not a failed tick. -// The URL is the check's credential, so no error returned here carries it. +// ping sends the result of the tick to healthchecks.io. If the tick has no errors, ping +// sends to the URL. If not, it sends to /fail, with the errors as the body. It sends the +// request a maximum of three times. If the ping does not get to healthchecks.io, the caller +// records a warning, not an error. The URL is the credential of the check. Thus, no error +// from ping contains the URL. func ping(client *http.Client, url string, errs []string, sleep func(time.Duration)) error { body := strings.Join(errs, "\n") if len(errs) > 0 { diff --git a/module.nix b/module.nix index 933bbf8..4b315ac 100644 --- a/module.nix +++ b/module.nix @@ -1,6 +1,6 @@ -# The scheduler as a NixOS service: a timer every five minutes starting one oneshot tick. -# It takes three file paths and knows nothing of where they come from; the host renders them -# (jshvn/jgrid.net: jgrid.secretTemplates into /run/secrets). +# The scheduler as a NixOS service: a timer that starts one oneshot tick at intervals of +# five minutes. It gets three file paths, and it does not know the source of the files. The +# host makes them (the host's flake: jgrid.secretTemplates into /run/secrets). self: { config, @@ -11,7 +11,8 @@ self: let cfg = config.services.katoptra-dispatch; package = self.packages.${pkgs.stdenv.hostPlatform.system}.default; - # A string, never a Nix path: a path literal would copy the secret into the world-readable store. + # A string, not a Nix path. A path literal puts a copy of the secret in the Nix store, + # which all users can read. secretPath = description: lib.mkOption { @@ -31,9 +32,10 @@ in systemd.timers.katoptra-dispatch = { description = "katoptra-dispatch tick, every five minutes"; wantedBy = [ "timers.target" ]; - # :02, :07 .. :42 .. :57, so the slot minute is always a tick. UTC whatever the host's - # zone. No Persistent=: every tick reconciles from the state file, so a missed tick is - # caught up by the next one. + # :02, :07 .. :42 .. :57. Thus, the slot minute is always a tick. The calendar is in + # UTC, and the time zone of the host has no effect. No Persistent=: each tick compares + # the slots with the state file. Thus, the next tick does the work of a tick that did + # not occur. timerConfig.OnCalendar = "*:02/5 UTC"; }; diff --git a/schedules/ctan.go b/schedules/ctan.go index c562c6e..a989a6e 100644 --- a/schedules/ctan.go +++ b/schedules/ctan.go @@ -1,4 +1,4 @@ package schedules -// katoptra/ctan mirrors CTAN into R2. +// katoptra/ctan is a mirror of CTAN in R2. var _ = register(Job{Repo: "katoptra/ctan", File: "sync.yml", Slots: Hourly}) diff --git a/schedules/dropbox.go b/schedules/dropbox.go index 8ebc349..91d9403 100644 --- a/schedules/dropbox.go +++ b/schedules/dropbox.go @@ -1,5 +1,6 @@ package schedules -// katoptra/dropbox mirrors a Dropbox account into Proton Drive. The run chains itself until -// every batch is done, so one dispatch a day starts the whole pass. +// katoptra/dropbox is a mirror of a Dropbox account in Proton Drive. Each run starts the +// next run in a chain until all batches are done. Thus, the scheduler dispatches it one +// time each day, and that starts the runs for all batches. var _ = register(Job{Repo: "katoptra/dropbox", File: "sync.yml", Slots: Overnight}) diff --git a/schedules/github.go b/schedules/github.go index 63c5e95..8e19380 100644 --- a/schedules/github.go +++ b/schedules/github.go @@ -1,4 +1,4 @@ package schedules -// katoptra/github mirrors every repository under jshvn and katoptra into Proton Drive. +// katoptra/github is a mirror of all repositories of jshvn and katoptra in Proton Drive. var _ = register(Job{Repo: "katoptra/github", File: "sync.yml", Slots: Morning}) diff --git a/schedules/gnu.go b/schedules/gnu.go index b00d783..56214f7 100644 --- a/schedules/gnu.go +++ b/schedules/gnu.go @@ -1,4 +1,4 @@ package schedules -// katoptra/gnu mirrors GNU's release tree from a secondary, twice a day. +// katoptra/gnu is a mirror of GNU's release tree from a secondary mirror, twice a day. var _ = register(Job{Repo: "katoptra/gnu", File: "sync.yml", Slots: S3 | S15}) diff --git a/schedules/nongnu.go b/schedules/nongnu.go index ac0c9b8..3d4e99c 100644 --- a/schedules/nongnu.go +++ b/schedules/nongnu.go @@ -1,4 +1,4 @@ package schedules -// katoptra/nongnu mirrors Savannah's nongnu release tree, twice a day. +// katoptra/nongnu is a mirror of Savannah's nongnu release tree, twice a day. var _ = register(Job{Repo: "katoptra/nongnu", File: "sync.yml", Slots: S3 | S15}) diff --git a/schedules/schedules.go b/schedules/schedules.go index 5c76794..791c350 100644 --- a/schedules/schedules.go +++ b/schedules/schedules.go @@ -1,14 +1,17 @@ -// Package schedules is what the scheduler starts, and when. One file per katoptra repo, -// named for the half of "owner/name" after the slash, each adding its jobs to the list -// below. Changing what runs touches only this directory. +// Package schedules tells the scheduler which jobs to start, and when. Each katoptra +// repository has one file here. The file name is the part of "owner/name" after the slash. +// Each file adds the jobs of its repository to the list below. A change to the jobs or to +// their slots is only in this directory. // -// Before adding a workflow, confirm all three. Nothing here can check them, and a workflow -// that fails any of them is dispatched into silence: +// Before you add a workflow, make sure that it has these three items. No code here can +// examine them. If a workflow does not have all three, a problem with its runs can stay +// unknown: // -// 1. It declares `workflow_dispatch:` in `on:`. -// 2. It declares a `concurrency` group with `cancel-in-progress: false`, so a dispatch -// arriving during a run queues instead of doubling up. -// 3. The workload pings its own healthcheck. This repo never learns whether a run passed. +// 1. It has `workflow_dispatch:` in `on:`. +// 2. It has a `concurrency` group with `cancel-in-progress: false`. Thus, a workflow start +// during a run waits in the queue, and two runs do not operate at the same time. +// 3. The workflow sends a ping to its healthcheck. This repository does not get the +// result of a run. package schedules import ( @@ -17,12 +20,12 @@ import ( "time" ) -// Slot is a set of hours of the UTC day. Each hour's slot fires at HH:42. Slots combine -// with |, so a job in two slots says `Morning | Evening`, and a misspelt name fails to -// compile. +// Slot is a set of hours of the UTC day. The slot of each hour is at HH:42. To put slots +// together, use |. For example, a job in two slots has `Morning | Evening`. An incorrect +// name causes a compile error. type Slot uint32 -// The 24 hourly slots, S0 at 00:42 UTC through S23 at 23:42 UTC. +// The 24 hourly slots, S0 at 00:42 UTC to S23 at 23:42 UTC. const ( S0 Slot = 1 << iota S1 @@ -50,8 +53,8 @@ const ( S23 ) -// Names for the daily slots, six hours apart. Pacific in the comments is winter time; in -// summer each is an hour later. +// Names for the daily slots, six hours apart. The Pacific times in the comments are winter +// times. In summer, each Pacific time is one hour after the time in the comment. const ( Evening = S5 // 05:42 UTC, 21:42 PST Overnight = S11 // 11:42 UTC, 03:42 PST @@ -60,12 +63,14 @@ const ( Hourly = Slot(1<<24 - 1) ) -// Minute is the minute past the hour every slot fires at: off the hour, which GitHub sheds -// first, and on a tick of the */5 timer that starts at :02. +// Minute is the minute of the hour for each slot. It is not on the hour, because GitHub +// cancels work on the hour first when its load is high. It is also on a tick of the */5 +// timer that starts at :02. const Minute = 42 -// Latest is the most recent slot time in s at or before now, in UTC. Today's and -// yesterday's slots are always enough to find it. False when s holds no hour. +// Latest gives the latest slot time in s: the last slot time at or before now, in UTC. The +// slots of the same day and of the day before are always sufficient to find it. The bool is +// false when s holds no hour. func (s Slot) Latest(now time.Time) (time.Time, bool) { now = now.UTC() for day := 0; day < 2; day++ { @@ -83,7 +88,7 @@ func (s Slot) Latest(now time.Time) (time.Time, bool) { return time.Time{}, false } -// Times is each slot's firing time, "HH:42" UTC, earliest first. +// Times gives the time of each slot in s as "HH:42" UTC, in the sequence of the day. func (s Slot) Times() []string { var out []string for hour := 0; hour < 24; hour++ { @@ -98,7 +103,8 @@ var names = map[Slot]string{ Hourly: "Hourly", Evening: "Evening", Overnight: "Overnight", Morning: "Morning", Afternoon: "Afternoon", } -// String names s as a schedule file would: "Hourly", "Morning", "Evening | Morning", "S3". +// String gives the name of s as a schedule file writes it: "Hourly", "Morning", +// "Evening | Morning", "S3". func (s Slot) String() string { if n, ok := names[s]; ok { return n @@ -118,14 +124,15 @@ func (s Slot) String() string { return strings.Join(parts, " | ") } -// Job is one workflow in one repo, dispatched on `main` in every slot it holds. +// Job is one workflow in one repository. The scheduler dispatches it on `main` in each +// slot that it holds. type Job struct { Repo string // "katoptra/" - File string // workflow file name, e.g. "sync.yml" + File string // the name of the workflow file, for example "sync.yml" Slots Slot } -// ID is the job's key in the state file: "katoptra/ctan/sync.yml". +// ID is the key of the job in the state file: "katoptra/ctan/sync.yml". func (j Job) ID() string { return j.Repo + "/" + j.File } var jobs []Job @@ -135,5 +142,5 @@ func register(j ...Job) bool { return true } -// Jobs is every registered job. +// Jobs gives all jobs that register added, in a new slice. func Jobs() []Job { return append([]Job(nil), jobs...) } diff --git a/schedules/tlnet.go b/schedules/tlnet.go index 07167ec..03dab43 100644 --- a/schedules/tlnet.go +++ b/schedules/tlnet.go @@ -1,4 +1,4 @@ package schedules -// katoptra/tlnet mirrors the TeX Live network installation tree. +// katoptra/tlnet is a mirror of the TeX Live network installation tree. var _ = register(Job{Repo: "katoptra/tlnet", File: "sync.yml", Slots: Evening}) diff --git a/tick.go b/tick.go index 1fd520e..d071988 100644 --- a/tick.go +++ b/tick.go @@ -14,15 +14,15 @@ import ( "github.com/katoptra/dispatch/schedules" ) -// stateFile holds, per job id, the last slot that job was fired for. +// stateFile holds the last dispatched slot of each job, by job id. const stateFile = "state.json" -// State maps a job id to the last slot fired for it, in UTC. +// State is a map from a job id to the last dispatched slot of that job, in UTC. type State map[string]time.Time -// LoadState reads dir's state file. A missing file is empty state, the first run; anything -// unreadable or malformed is an error, never empty state, because empty state would fire -// every job again for slots already sent. +// LoadState reads the state file in dir. A missing file is empty state: the first tick. A +// file that it cannot read, or that has an incorrect format, is an error, not empty state. +// This is because empty state starts each job again for dispatched slots. func LoadState(dir string) (State, error) { b, err := os.ReadFile(filepath.Join(dir, stateFile)) if errors.Is(err, os.ErrNotExist) { @@ -35,14 +35,14 @@ func LoadState(dir string) (State, error) { if err := json.Unmarshal(b, &s); err != nil { return nil, fmt.Errorf("state file %s is corrupt: %w", filepath.Join(dir, stateFile), err) } - if s == nil { // `null` parses; it is not a missing file, and must not act like one + if s == nil { // json.Unmarshal accepts `null`. But a `null` file is not a missing file, and it is an error. return nil, fmt.Errorf("state file %s holds no object", filepath.Join(dir, stateFile)) } return s, nil } -// SaveState replaces dir's state file atomically: a crash leaves the old file or the new -// one, never half of either. +// SaveState replaces the state file in dir atomically. After a crash, dir holds the full +// previous file or the full new file. func SaveState(dir string, s State) error { b, err := json.MarshalIndent(s, "", " ") if err != nil { @@ -52,7 +52,7 @@ func SaveState(dir string, s State) error { if err != nil { return err } - defer os.Remove(tmp.Name()) // a no-op once renamed + defer os.Remove(tmp.Name()) // a no-op after os.Rename if _, err := tmp.Write(append(b, '\n')); err != nil { tmp.Close() return err @@ -75,14 +75,15 @@ func SaveState(dir string, s State) error { return d.Sync() } -// Fire is one job due for one slot. +// Fire is one job to dispatch for one slot. type Fire struct { Job schedules.Job Slot time.Time } -// Plan returns the jobs due at now: those with no recorded slot, or whose latest slot is -// after the one recorded. However many slots a job missed, it is due once, for the latest. +// Plan gives the jobs to dispatch at now. A job is in the result if it has no recorded +// slot, or if its latest slot is after its recorded slot. If there are one or more slots +// after the recorded slot, the job is in the result one time, for the latest slot. func Plan(jobs []schedules.Job, s State, now time.Time) (fires []Fire, warnings []string) { for _, j := range jobs { due, ok := j.Slots.Latest(now) @@ -102,8 +103,9 @@ func Plan(jobs []schedules.Job, s State, now time.Time) (fires []Fire, warnings return fires, warnings } -// next is the state to write before dispatching fires: every current job's entry, with the -// fired slots recorded. Entries for jobs no longer registered are dropped. +// next is the state to write before Tick dispatches fires. It keeps the entry of each job +// in jobs, and it records the slot of each fire. It does not keep the entry of a job that +// is not in jobs. func next(jobs []schedules.Job, s State, fires []Fire) State { n := State{} for _, j := range jobs { @@ -117,20 +119,22 @@ func next(jobs []schedules.Job, s State, fires []Fire) State { return n } -// Dispatcher starts workflow runs. Prepare does everything that starts nothing (credentials), -// so its failure can leave the slots unrecorded; Dispatch starts one run. +// Dispatcher starts workflow runs. Prepare does the steps that start no run (the +// credentials). Thus, if Prepare gives an error, Tick can stop before it records the slots. +// Dispatch starts one run. type Dispatcher interface { Prepare(ctx context.Context) error Dispatch(ctx context.Context, repo, file string) error } -// Tick is one run of the timer: lock, plan, prepare, record, dispatch. Every failure goes to log; -// the caller reads log.Errors() for the exit code and the ping. +// Tick is one tick of the timer: lock, plan, prepare, record, dispatch. Each error goes to +// log. The caller reads log.Errors() for the exit code and the ping. // -// The slots are recorded before anything is dispatched, so a crash or a failed dispatch -// loses that slot rather than repeating it: at most once per slot. Prepare comes before the -// record, so a GitHub or network outage before any run could start costs five minutes, not -// the slot. +// Tick records the slots before it dispatches. Thus, after a crash or an error from +// Dispatch, that slot gets no run. A slot does not get two runs: a maximum of one run for +// each slot. Prepare comes before the record. Thus, if GitHub or the network is not +// available before a run can start, the slot gets its run at the next tick, after five +// minutes. func Tick(ctx context.Context, dir string, jobs []schedules.Job, now time.Time, d Dispatcher, dryRun bool, log *Log) { unlock, err := lock(dir) if err != nil { @@ -177,8 +181,9 @@ func Tick(ctx context.Context, dir string, jobs []schedules.Job, now time.Time, } } -// lock takes an exclusive flock on dir itself. systemd already refuses to start a oneshot -// that is still running; this covers the binary run by hand beside it. +// lock gets an exclusive flock on the directory dir, not on a file in it. systemd does not +// start a oneshot unit while that unit operates. The flock also prevents two ticks at the +// same time when you start the binary manually. func lock(dir string) (func(), error) { f, err := os.Open(dir) if err != nil { @@ -200,9 +205,9 @@ func lastFired(s State, j schedules.Job) string { return "never" } -// Log writes to the journal through stdout and stderr, and keeps the errors for the ping. -// Under systemd (JOURNAL_STREAM set) lines carry the sd-daemon priority prefix, so -// `journalctl -p err` finds the errors. +// Log writes to the journal through stdout and stderr. It keeps the errors for the ping. +// When systemd starts the binary (JOURNAL_STREAM is set), the warning and error lines have +// the sd-daemon priority prefix. Thus, `journalctl -p err` finds the errors. type Log struct { Out, Err io.Writer Journal bool @@ -225,5 +230,5 @@ func (l *Log) write(w io.Writer, prio, format string, a ...any) { fmt.Fprintf(w, prio+format+"\n", a...) } -// Errors is every error logged so far. +// Errors gives each error that Error recorded before this call. func (l *Log) Errors() []string { return l.errs } diff --git a/tick_test.go b/tick_test.go index e1d7f23..0fc5cb4 100644 --- a/tick_test.go +++ b/tick_test.go @@ -26,8 +26,8 @@ func utc(s string) time.Time { return t } -// fake records each dispatch, reads the state file as it stood at that moment, and fails -// the jobs named in fail. Prepare returns prepErr. +// fake records each job that Dispatch gets, and reads the state file as it is at that time. +// It gives an error for each job in fail. Prepare gives prepErr. type fake struct { dir string fail map[string]bool From 3a15b5804817d873410391d33aa26135d4f12d71 Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 15:00:09 -0700 Subject: [PATCH 4/7] docs(readme): fix STE review findings in README and CLAUDE.md Fix round 1 of the docs PR review. - COME, and WRITE, RECORD and WAIT as nouns: Tick is the subject, and "can follow" replaces "can come after" (review I-1, I-2). - The Tick step sequence is a numbered list; "contain" for file content; one wording for "the host gets a change" (minors 1, 3, 4). - Must knows: "each slot that had no run" (minor 2), the at-least-once note as a caution (minor 13), the retry lead names the second run (minor 14). - README: a /fail ping costs the slot only after a workflow start error; an error before the state is saved records no slot (controller ruling). - README: a fork also changes Org in main.go (controller ruling). - README: steps 2 and 5 of "Want your own?" as sub-steps (minor 6). --- CLAUDE.md | 37 ++++++++++++++++++++++--------------- README.md | 37 ++++++++++++++++++++++--------------- 2 files changed, 44 insertions(+), 30 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index f994de6..ff825d9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,7 +11,12 @@ the design. katoptra repository. Each of these files adds the jobs of its repository. A usual change is only in this directory. - `tick.go`: the state file, `Plan`, and `Tick`. `Tick` does these steps in this sequence: - lock, load, plan, prepare, record (the write before the first POST), dispatch. + 1. It locks the state directory. + 2. It loads the state. + 3. It uses `Plan` to find the jobs to dispatch. + 4. It gets the installation token (`Dispatcher.Prepare`). + 5. It records the slots in `state.json`, before the first POST. + 6. It dispatches the workflows. - `github.go`: the App JWT, the installation of the App on the organization, the installation token, `workflow_dispatch`, and the retries. It uses no SDK. - `main.go`: the flags, the credentials from `$CREDENTIALS_DIRECTORY`, the healthcheck ping, @@ -28,7 +33,7 @@ the design. `katoptra` organization. - It uses only the Go standard library (`vendorHash = null`). - It uses Go 1.26, the same version as the `buildGoModule` of nixos-26.05. The toolbox image - also uses Go 1.26: its `FROM` line holds that version and a digest. + also uses Go 1.26: its `FROM` line contains that version and a digest. - All times are in UTC. No part of the code adjusts a time for DST. - The repository, the Nix store and the state file contain no secrets. The module gets file paths, not secrets. @@ -38,21 +43,22 @@ the design. ## Must knows -- **At-most-once, because the write comes first.** Before the first POST, `Tick` records in - `state.json` the slot of each job that it will dispatch. After a crash or an error in a - workflow start, that slot gets no run. The scheduler does not try that slot again: the - next slot is the retry. If you move the write after the POST, a slot can get two runs - (at-least-once). `TestSlotIsRecordedBeforeTheDispatch` finds that error. -- **The token comes before the write.** `Dispatcher.Prepare` gets the installation token - before `Tick` records a slot, because a token request starts no run. If GitHub or the - network is not available at that step, `Tick` records no slot, and the next tick tries - again. Thus, a daily mirror gets its run for that day. +- **At-most-once, because `Tick` records the slots first.** Before the first POST, `Tick` + records in `state.json` the slot of each job that it will dispatch. After a crash or an + error in a workflow start, that slot gets no run. The scheduler does not try that slot + again: the next slot is the retry. Do not put the code that records the slots after the + POST. A slot can then get two runs (at-least-once), and + `TestSlotIsRecordedBeforeTheDispatch` finds that error. +- **`Tick` gets the token before it records a slot.** `Dispatcher.Prepare` gets the + installation token. `Tick` can do this step first because a token request starts no run. + If GitHub or the network is not available at that step, `Tick` records no slot, and the + next tick tries again. Thus, a daily mirror gets its run for that day. `TestPrepareFailureRecordsNothing` makes sure that this sequence does not change. - **`Plan`, not systemd, finds the slots that had no run.** The timer has no `Persistent=`. Each tick compares the latest slot of each job with the slot that the state records for it. Thus, a job that had no run for one or more slots gets one run. If - each slot has its timer with `Persistent=true`, systemd starts a run for each of these - slots. + each slot has its timer with `Persistent=true`, systemd starts a run for each slot that + had no run. - **A corrupted `state.json` stops all jobs.** `LoadState` gives an error for it, not empty state, because empty state starts each job again for slots that had a run. Repair or delete the file manually. If you delete it, each job starts one time. @@ -64,11 +70,12 @@ the design. - **The scheduler dispatches on `ref: main`, with empty inputs.** If the default branch of a repository is not `main`, GitHub sends a 404. The scheduler does not try again, and that slot gets no run. -- **A retry (after 10 s, 20 s, 40 s) occurs only when a second request is safe.** +- **A retry (after 10 s, 20 s, 40 s) occurs only when a second request cannot cause a + second run.** - A 4xx other than 408 and 429 gets no retry. - The two token requests try again after a 5xx, 408, 429, or network error. - A workflow start tries again only after a 408, a 429, or a connection that did not - open. A 5xx or a timeout can come after GitHub accepted the workflow start, and a retry + open. A 5xx or a timeout can follow a workflow start that GitHub accepted, and a retry then starts a second run. - **GitHub gets two minutes in each tick** (`githubBudget`), with the retries included. If a retry cannot start in the two minutes, the scheduler does not make it. The other two diff --git a/README.md b/README.md index 8ea1d45..4b5501f 100644 --- a/README.md +++ b/README.md @@ -62,8 +62,8 @@ items: - It sends a ping to its healthcheck. The scheduler starts runs, but it does not know the result of a run. -The host uses a change when the host's flake lock moves to the new commit. A new job starts -at the next tick. +The host gets a change when the host's flake moves its lock to a new commit of this flake. +A new job starts at the next tick. ## How it works @@ -78,7 +78,7 @@ flowchart LR The timer starts a tick at :02, at :07, and at intervals of five minutes after that. Each tick does these steps: -1. It loads `state.json`, which holds the last dispatched slot of each job. +1. It loads `state.json`, which contains the last dispatched slot of each job. 2. It finds the jobs to start. If a job had no run for some slots, it starts one time, for the latest slot. 3. It gets a token for the GitHub App. @@ -132,8 +132,10 @@ does not save the state, start a workflow, or send a ping. - **The healthcheck gets no pings.** The host or its timer does not operate. When the host and the timer operate again, each job that had no run for a slot gets one run. - **The healthcheck gets a `/fail` ping.** The body of the ping and - `journalctl -u katoptra-dispatch -p err` show the error. The job gets no run for that - slot. It starts again at its next slot. + `journalctl -u katoptra-dispatch -p err` show the error. If the error is from a workflow + start, that job gets no run for that slot, and it starts again at its next slot. If the + error occurs before the tick saves the state, the tick records no slot, and the next tick + tries again. - **A workflow start gets a 404.** The default branch of the repository is not `main`, or the repository does not have the workflow file. - **Each tick stops with an error about `state.json`.** The file is corrupted. No job starts @@ -144,11 +146,14 @@ does not save the state, start a workflow, or send a ping. 1. **Fork this repository.** Replace the files in `schedules/` with files for your jobs. The tests accept only `katoptra/` repositories. Change that prefix in the tests to the - name of your organization. -2. **Make a GitHub App in your organization.** Give it one permission: Actions, read and - write. A webhook is not necessary. Install the App on all repositories in the - organization. Then each new repository also gets the App automatically. Record the App - ID. Make a private key. You can use the key in the format that GitHub gives. + name of your organization. In `main.go`, change `Org: "katoptra"` to the name of your + organization. +2. **Make a GitHub App in your organization:** + 1. Give the App one permission: Actions, read and write. A webhook is not necessary. + 2. Install the App on all repositories in the organization. Then each new repository + also gets the App automatically. + 3. Record the App ID. + 4. Make a private key. You can use the key in the format that GitHub gives. 3. **Make a healthchecks.io check** with a period of 5 minutes and a grace time of 10 minutes. 4. **Add the flake to your NixOS host.** Give the module the paths of three files: @@ -170,11 +175,13 @@ does not save the state, start a workflow, or send a ping. systemd loads the files with `LoadCredential=`. Thus, the files can have the owner `root` and the mode 0400. -5. **Make sure that the scheduler operates.** On a laptop with go-task and Docker or Apple - `container`, use `task check` and `task targets`. `nix flake check` builds the package - and the systemd units. [`CLAUDE.md`](CLAUDE.md) shows how to use it without nix. Only a - tick on the host is a full test of the App. On the host, use - `sudo systemctl start katoptra-dispatch`. Then read the journal. +5. **Make sure that the scheduler operates:** + 1. On a laptop with go-task and Docker or Apple `container`, use `task check`. Then use + `task targets`. + 2. Use `nix flake check`. It builds the package and the systemd units. + [`CLAUDE.md`](CLAUDE.md) shows how to use it without nix. + 3. Only a tick on the host is a full test of the App. On the host, use + `sudo systemctl start katoptra-dispatch`. Then read the journal. ## License From b3a893fc718a45e220618c8d9ac1370dd7e5d5d8 Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 15:00:10 -0700 Subject: [PATCH 5/7] docs(comments): fix STE review findings in code comments Fix round 1 of the docs PR review. Comments only: the Go token streams and the parsed Nix ASTs still match origin/main. - COME and the verb-only nouns: "follow", Tick as the subject, and "interval" for backoff (review I-1, I-2). - retryable: no "true" (adj); says when GitHub did not start the run (review I-3). - ping: two paragraphs, the "If not" sentence in full (review I-4). - Cadence: dropbox is "daily"; gnu and nongnu say "The scheduler starts it twice a day" (fact check, minor 9). - Minute: "decreases the load", not "cancels" (fact check). - runs task: any status other than completed has an empty conclusion (fact check). - flake.nix: an unambiguous ponytail trigger; no fragments (fact check, minor 11). - "contain" for content, "in one step" for atomically, the null comment on its own line, the schedules list before its warning, the GitHub file comment, the Dockerfile and task clean cautions (minors 3, 7, 8, 10, 12, 13, 15). --- Dockerfile | 5 +++-- Taskfile.yml | 6 +++--- flake.nix | 11 ++++++----- github.go | 17 +++++++++-------- main.go | 16 +++++++++------- schedules/dropbox.go | 4 ++-- schedules/gnu.go | 3 ++- schedules/nongnu.go | 3 ++- schedules/schedules.go | 19 ++++++++++--------- tick.go | 16 +++++++++------- 10 files changed, 55 insertions(+), 45 deletions(-) diff --git a/Dockerfile b/Dockerfile index 1277b88..01e0bd4 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,6 +1,7 @@ # The toolbox: each check operates in this image, on a laptop and in CI. It uses Go 1.26 -# because the host builds with the buildGoModule of nixos-26.05, which is Go 1.26. A newer -# toolchain here can accept code that the build on the host rejects. +# because the host builds with the buildGoModule of nixos-26.05, which is Go 1.26. Do not +# use a newer Go toolchain here. A newer toolchain can accept code that the build on the +# host rejects. FROM golang:1.26-alpine@sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7236c349a0c # The mount does not always contain .git, and no code reads the VCS stamp. ENV GOFLAGS=-buildvcs=false diff --git a/Taskfile.yml b/Taskfile.yml index 6fbbfc2..170af07 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -67,7 +67,7 @@ tasks: run: once silent: true # If the image is there, task does not build it again. Thus, a change to the Dockerfile - # does not cause a new build. Do `task clean`, then use a task. + # does not cause a new build. Do `task clean`. Then the next task builds a new image. status: - '{{.ENGINE}} image inspect {{.IMAGE}} >/dev/null 2>&1' cmds: @@ -102,8 +102,8 @@ tasks: LIMIT: '{{.LIMIT | default 3}}' cmds: # GH_PAGER=cat: on a terminal, gh opens its pager for each target and waits for it. - # A run in progress has an empty `conclusion`. Thus, the table shows its `status` as - # an alternative. + # If the `status` of a run is not `completed`, its `conclusion` is empty. Thus, the + # table shows the `status` as an alternative. - | task run -- go run . --list | awk 'NR > 1 { print $1 }' | while read -r id; do GH_PAGER=cat gh run list --repo "${id%/*}" --workflow "${id##*/}" --limit {{.LIMIT}} \ diff --git a/flake.nix b/flake.nix index 0fdbd64..c9ebf20 100644 --- a/flake.nix +++ b/flake.nix @@ -1,8 +1,8 @@ { description = "katoptra-dispatch: starts the katoptra mirrors' workflows on UTC slots"; - # The same release that the host's flake builds its hosts from. Thus, the host's lock - # can follow it. + # This is the same release that the host's flake builds its hosts from. Thus, the host's + # lock can use the same release. inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05"; outputs = @@ -39,9 +39,10 @@ nixosModules.default = import ./module.nix self; - # The package (with its tests), and the two units of the module, made from a small - # system. Nix makes the units, but it does not boot a VM. ponytail: a nixosTest boots - # the timer in a VM. If this check does not find a unit error, add a nixosTest. + # These checks are the package (with its tests) and the two units of the module, made + # from a small system. Nix makes the units, but it does not boot a VM. ponytail: a + # nixosTest boots the timer in a VM. If a unit error occurs on a host and this check + # did not find it, add a nixosTest. checks = forAll ( pkgs: let diff --git a/github.go b/github.go index 4702547..2a0caf7 100644 --- a/github.go +++ b/github.go @@ -1,7 +1,8 @@ package main -// The GitHub App side: sign a JWT, find the installation of the App on the org, get an -// installation token, and dispatch a workflow. It uses no SDK, only three HTTP requests. +// This file does four steps as the GitHub App: it signs a JWT, finds the installation +// of the App on the organization, gets an installation token, and dispatches a workflow. +// It uses no SDK, only three HTTP requests. import ( "bytes" @@ -22,8 +23,8 @@ import ( "time" ) -// backoff is the wait before each retry of a temporary error: three retries, 70 s in total. -// If a retry cannot start before the deadline of the context, call does not make it. +// backoff is the interval before each retry of a temporary error: three retries, 70 s in +// total. If a retry cannot start before the deadline of the context, call does not make it. var backoff = []time.Duration{10 * time.Second, 20 * time.Second, 40 * time.Second} // StatusError is a GitHub response with a status other than 2xx. @@ -38,10 +39,10 @@ func (e *StatusError) Error() string { return e.Msg } // GitHub rejected the request, and a retry cannot change that. For example: a missing // workflow file, an App without the permission, or no installation. // -// call tries a request that starts a run (safe false) again only if GitHub did not do the -// request. This is true when the connection did not open, or when GitHub rejected the -// request before it read it (408, 429). A 5xx or a timeout can come after GitHub accepted -// the request. Then a retry starts a second run. +// call tries a request that starts a run (safe false) again only if GitHub did not start +// the run. GitHub did not start the run if the connection did not open, or if GitHub +// rejected the request before it read it (408, 429). A 5xx or a timeout can follow a +// request that GitHub accepted. Then a retry starts a second run. func retryable(err error, safe bool) bool { var s *StatusError if errors.As(err, &s) { diff --git a/main.go b/main.go index dfc9d3a..be0f99d 100644 --- a/main.go +++ b/main.go @@ -1,7 +1,7 @@ // katoptra-dispatch is one tick of the katoptra scheduler. It finds each job with a latest -// slot that the state does not hold. It records that slot first, and then it starts the job. -// At the end, it sends a ping to the healthcheck of the scheduler. A systemd timer starts it -// at intervals of five minutes (refer to module.nix). +// slot that the state does not contain. It records that slot first, and then it starts the +// job. At the end, it sends a ping to the healthcheck of the scheduler. A systemd timer +// starts it at intervals of five minutes (refer to module.nix). package main import ( @@ -130,10 +130,12 @@ func credential(dir, name string) (string, error) { } // ping sends the result of the tick to healthchecks.io. If the tick has no errors, ping -// sends to the URL. If not, it sends to /fail, with the errors as the body. It sends the -// request a maximum of three times. If the ping does not get to healthchecks.io, the caller -// records a warning, not an error. The URL is the credential of the check. Thus, no error -// from ping contains the URL. +// sends to the URL. If the tick has errors, ping sends to /fail, with the errors as the +// body. +// +// ping sends the request a maximum of three times. If the ping does not get to +// healthchecks.io, the caller records a warning, not an error. The URL is the credential of +// the check. Thus, no error from ping contains the URL. func ping(client *http.Client, url string, errs []string, sleep func(time.Duration)) error { body := strings.Join(errs, "\n") if len(errs) > 0 { diff --git a/schedules/dropbox.go b/schedules/dropbox.go index 91d9403..576e32e 100644 --- a/schedules/dropbox.go +++ b/schedules/dropbox.go @@ -1,6 +1,6 @@ package schedules // katoptra/dropbox is a mirror of a Dropbox account in Proton Drive. Each run starts the -// next run in a chain until all batches are done. Thus, the scheduler dispatches it one -// time each day, and that starts the runs for all batches. +// next run in a chain until all batches are done. Thus, the scheduler dispatches it daily, +// and that starts the runs for all batches. var _ = register(Job{Repo: "katoptra/dropbox", File: "sync.yml", Slots: Overnight}) diff --git a/schedules/gnu.go b/schedules/gnu.go index 56214f7..48d7b43 100644 --- a/schedules/gnu.go +++ b/schedules/gnu.go @@ -1,4 +1,5 @@ package schedules -// katoptra/gnu is a mirror of GNU's release tree from a secondary mirror, twice a day. +// katoptra/gnu is a mirror of GNU's release tree from a secondary mirror. The scheduler +// starts it twice a day. var _ = register(Job{Repo: "katoptra/gnu", File: "sync.yml", Slots: S3 | S15}) diff --git a/schedules/nongnu.go b/schedules/nongnu.go index 3d4e99c..052b165 100644 --- a/schedules/nongnu.go +++ b/schedules/nongnu.go @@ -1,4 +1,5 @@ package schedules -// katoptra/nongnu is a mirror of Savannah's nongnu release tree, twice a day. +// katoptra/nongnu is a mirror of Savannah's nongnu release tree. The scheduler starts it +// twice a day. var _ = register(Job{Repo: "katoptra/nongnu", File: "sync.yml", Slots: S3 | S15}) diff --git a/schedules/schedules.go b/schedules/schedules.go index 791c350..1cbe562 100644 --- a/schedules/schedules.go +++ b/schedules/schedules.go @@ -3,15 +3,16 @@ // Each file adds the jobs of its repository to the list below. A change to the jobs or to // their slots is only in this directory. // -// Before you add a workflow, make sure that it has these three items. No code here can -// examine them. If a workflow does not have all three, a problem with its runs can stay -// unknown: +// Before you add a workflow, make sure that it has these three items: // // 1. It has `workflow_dispatch:` in `on:`. // 2. It has a `concurrency` group with `cancel-in-progress: false`. Thus, a workflow start // during a run waits in the queue, and two runs do not operate at the same time. // 3. The workflow sends a ping to its healthcheck. This repository does not get the // result of a run. +// +// No code here can examine these items. If a workflow does not have all three, a problem +// with its runs can stay unknown. package schedules import ( @@ -63,14 +64,14 @@ const ( Hourly = Slot(1<<24 - 1) ) -// Minute is the minute of the hour for each slot. It is not on the hour, because GitHub -// cancels work on the hour first when its load is high. It is also on a tick of the */5 -// timer that starts at :02. +// Minute is the minute of the hour for each slot. It is not minute 0: when the load on +// GitHub is high, GitHub decreases the load at minute 0 first. It is also on a tick of the +// */5 timer that starts at :02. const Minute = 42 // Latest gives the latest slot time in s: the last slot time at or before now, in UTC. The // slots of the same day and of the day before are always sufficient to find it. The bool is -// false when s holds no hour. +// false when s contains no hour. func (s Slot) Latest(now time.Time) (time.Time, bool) { now = now.UTC() for day := 0; day < 2; day++ { @@ -124,8 +125,8 @@ func (s Slot) String() string { return strings.Join(parts, " | ") } -// Job is one workflow in one repository. The scheduler dispatches it on `main` in each -// slot that it holds. +// Job is one workflow in one repository. The scheduler dispatches it on `main` in each of +// its slots. type Job struct { Repo string // "katoptra/" File string // the name of the workflow file, for example "sync.yml" diff --git a/tick.go b/tick.go index d071988..dfbd960 100644 --- a/tick.go +++ b/tick.go @@ -14,7 +14,7 @@ import ( "github.com/katoptra/dispatch/schedules" ) -// stateFile holds the last dispatched slot of each job, by job id. +// stateFile contains the last dispatched slot of each job, by job id. const stateFile = "state.json" // State is a map from a job id to the last dispatched slot of that job, in UTC. @@ -35,14 +35,16 @@ func LoadState(dir string) (State, error) { if err := json.Unmarshal(b, &s); err != nil { return nil, fmt.Errorf("state file %s is corrupt: %w", filepath.Join(dir, stateFile), err) } - if s == nil { // json.Unmarshal accepts `null`. But a `null` file is not a missing file, and it is an error. + // json.Unmarshal accepts `null`. But a `null` file is not a missing file: LoadState + // gives an error for it, not empty state. + if s == nil { return nil, fmt.Errorf("state file %s holds no object", filepath.Join(dir, stateFile)) } return s, nil } -// SaveState replaces the state file in dir atomically. After a crash, dir holds the full -// previous file or the full new file. +// SaveState replaces the state file in dir in one step. After a crash, dir contains the +// full previous file or the full new file. func SaveState(dir string, s State) error { b, err := json.MarshalIndent(s, "", " ") if err != nil { @@ -132,9 +134,9 @@ type Dispatcher interface { // // Tick records the slots before it dispatches. Thus, after a crash or an error from // Dispatch, that slot gets no run. A slot does not get two runs: a maximum of one run for -// each slot. Prepare comes before the record. Thus, if GitHub or the network is not -// available before a run can start, the slot gets its run at the next tick, after five -// minutes. +// each slot. Tick does Prepare before it records the slots. Thus, if GitHub or the network +// is not available before a run can start, the slot gets its run at the next tick, after +// five minutes. func Tick(ctx context.Context, dir string, jobs []schedules.Job, now time.Time, d Dispatcher, dryRun bool, log *Log) { unlock, err := lock(dir) if err != nil { From 0888453eac34e2ff0dff7a09031162b13500e4fb Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 15:09:40 -0700 Subject: [PATCH 6/7] docs(comments): list the GitHub App steps, give ping an object Fix round 2 of the docs PR review. - github.go: the file comment was one 30-word sentence. The four steps are now a numbered list (review N-1). - main.go: "ping sends the request to the URL" and "to /fail" (review N-2, optional). Comments only: the Go token streams still match origin/main. --- github.go | 9 +++++++-- main.go | 4 ++-- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/github.go b/github.go index 2a0caf7..eeb3e3c 100644 --- a/github.go +++ b/github.go @@ -1,7 +1,12 @@ package main -// This file does four steps as the GitHub App: it signs a JWT, finds the installation -// of the App on the organization, gets an installation token, and dispatches a workflow. +// This file does four steps as the GitHub App: +// +// 1. It signs a JWT. +// 2. It finds the installation of the App on the organization. +// 3. It gets an installation token. +// 4. It dispatches a workflow. +// // It uses no SDK, only three HTTP requests. import ( diff --git a/main.go b/main.go index be0f99d..63ac2d5 100644 --- a/main.go +++ b/main.go @@ -130,8 +130,8 @@ func credential(dir, name string) (string, error) { } // ping sends the result of the tick to healthchecks.io. If the tick has no errors, ping -// sends to the URL. If the tick has errors, ping sends to /fail, with the errors as the -// body. +// sends the request to the URL. If the tick has errors, ping sends the request to /fail, +// with the errors as the body. // // ping sends the request a maximum of three times. If the ping does not get to // healthchecks.io, the caller records a warning, not an error. The URL is the credential of From 0ef78f228e8d1190b2bd8100cbca11902c931444 Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 15:10:18 -0700 Subject: [PATCH 7/7] docs(readme): say "saves no slot" in the /fail note Fix round 2 of the docs PR review (N-2, optional): one verb, "save", for the state write, as in the numbered steps. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4b5501f..35cc180 100644 --- a/README.md +++ b/README.md @@ -134,7 +134,7 @@ does not save the state, start a workflow, or send a ping. - **The healthcheck gets a `/fail` ping.** The body of the ping and `journalctl -u katoptra-dispatch -p err` show the error. If the error is from a workflow start, that job gets no run for that slot, and it starts again at its next slot. If the - error occurs before the tick saves the state, the tick records no slot, and the next tick + error occurs before the tick saves the state, the tick saves no slot, and the next tick tries again. - **A workflow start gets a 404.** The default branch of the repository is not `main`, or the repository does not have the workflow file.