Queue work for the desk and retire the paid API jobs

[JesperDramsch]

Fourth PR in the desk stack, now based on main (#499 is merged). GitHub Actions stops calling a paid API: the jobs that used one either collect work for the desk into a queue or keep only their deterministic part. After this PR, no workflow references ANTHROPIC_API_KEY.

The desk queue

utils/desk_queue.py reads and writes the queue on an orphan desk-queue branch, which is never merged. There is one JSON file per series under queue/, keyed by the slug of the series name, and a README.md that init writes from the module, so the format is documented in one place. Each item gets a stable id from its source, kind, URL and payload, so the same finding queued twice is one item. When the desk removes an item, its id moves to the file's handled list. A writer that finds the same thing again (the import asks the same standing questions on every run) then skips it, while a finding that has changed gets a new id. A file keeps its 50 newest items and 500 handled ids, and payload strings are cut at 20,000 characters and flagged. A broken file is moved to .bad, so it can't stop the writers. The module uses only the standard library, so any job can run it.

.github/actions/desk-queue-write is the one place that writes to the queue. It creates the branch when it is missing, adds records with desk_queue.py add --jsonl, and retries a rejected push from a fresh copy of the branch, never with force. I tested it against a local remote: creating the branch, a repeat (no-op), an append, a push race (rejected once, then landed on top), a rejected record (a workflow warning, while the valid one still lands), and a re-add after remove (reported as handled).

Writers

• check-conference-update.yml turns a changedetection.io event into one queue record, with no triage in front. The payload is read from the event file, so page text never reaches the shell, and a long diff can't hit the environment size limit. The old pipeline (Haiku triage, two agent jobs, apply-filters) goes. Over 254 events the triage had 38% precision and 58% recall.
• canonical-source-update.yml: the import stays deterministic. In CI, query_yes_no still takes the default answer. When the caller passes context, it also records the question: conference, year, field, both values and the source. With DESK_QUEUE_OUT set, the record goes to the queue, where the enricher answers it from the live site. The JSON-lines writer is shared with the link checker in tidy_conf/queue_out.py. A queue problem can't hold back the data merge (continue-on-error). The job also gets actions: write, which its existing gh workflow run call to start the enrichment always needed.
• redirect-scan.yml (new, Sundays) runs the link check in report-only mode. It queues every link that now ends on another site, the case lychee can't tell apart from a healthy 200.

Retired

• scheduled-extension-checker.yml: the worklist checks every deadline 1, 3 and 7 days after it passes.
• The full mode of utils/enrich_tba.py and its workflow input. The deterministic link extraction stays, along with the code only it uses. The confidence handling, which only the API results could reach, is removed. Review items now end in "(kept existing)".
• The remaining link extraction no longer visits cancelled editions (7d6c428), so it can't fill a sponsor or finaid link for an event that won't happen. Editions with cfp: None stay in: the job only fills Bluesky, YouTube, Mastodon, sponsor and finaid links, and an edition with no public CfP still happens. On the current data the only change is PyCon Estonia 2026.

Needs your call

• Merge order: merge the stack bottom-up. The desk only starts once you've done the Routine steps in the UI. Between merging this PR and that, changedetection events, merge questions and redirects pile up in desk-queue without being lost (at most 50 per series).
• Secret: ANTHROPIC_API_KEY can be deleted from the repository secrets once this PR is on main.

Checks

• An adversarial review found two should-fix issues: handled items came back on every run, and a broken queue file stopped the canonical merge. It also found five nits. All seven are fixed in b3705a6, and each has a test where it can.
• The routine review's two nits are fixed in 87bc32e:
  • The manual run's example name, "PyCon US 2027", was queued with year null under its own file, since a year counted only before a changedetection suffix. A trailing year from 1900 to 2099 with no suffix now counts too, so it lands in queue/pycon-us.json with year 2027. Watch titles keep their suffix and read as before.
  • A comment next to the merge prompts says each prompt's text is part of its queued question's id, so rewording one queues every standing question again, once.
• Tests: the pinned uv run gives 1175 passed, 0 failed.
• uvx ruff@0.4.4 check utils tests: clean.
• pre-commit run on every changed file: passes.
• actionlint on all workflows: clean.
• python utils/sort_yaml.py --skip_links leaves _data/ unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QzyF2ciB5jhvcTVpj4ErMr

[JesperDramsch]

Routine review

1. Request

No linked issue, so the request is the PR body. It asks to stop every paid-API call from GitHub Actions by:

• adding the desk queue (utils/desk_queue.py, .github/actions/desk-queue-write);
• turning check-conference-update.yml into a single queue writer;
• queueing the canonical import's unanswered merge questions (interactive_merge.py, utils.py, queue_out.py, both importers, canonical-source-update.yml);
• adding a weekly redirect-scan.yml;
• retiring scheduled-extension-checker.yml and the full mode of enrich_tba.py / enrich-tba-conferences.yml;
• leaving cancelled editions out of the TBA link enrichment (7d6c428).

2. Checked against

• Date: Thu Oct 15 10:49:51 UTC 2026 (date).
• No external facts are relied on. unverified: the PR body's figure for the retired triage (38% precision, 58% recall over 254 events). It comes from repository history I did not re-derive.

3. Needs your decision

1. Merge order and the gap before the desk Routine is live. Once this is on main, changedetection signals, merge questions and redirects pile up on desk-queue, with at most 50 items per series, until you set up the Routine in the UI. Recommendation: merge, then set up the Routine the same day, so no busy series hits the 50-item cap and silently drops older signals.
2. Deleting ANTHROPIC_API_KEY. On this head, no workflow, utils/ file, .claude/ file or pixi.toml refers to ANTHROPIC_API_KEY, enrichment_level, --level, apply-filters or scheduled-extension-checker (grep). Recommendation: delete the secret after merge.

4. Findings

Important: None.

Nits:

• .github/workflows/check-conference-update.yml:21: the manual input's example is "PyCon US 2027". clean_watch_title strips a year only when the title ends in - Main/- CfP, so that example becomes ("PyCon US 2027", None) and is queued to queue/pycon-us-2027.json with year: null (checked with a one-line script). Suggest the example "PyCon US 2027 - Main", or making the year part of the suffix pattern optional.
• utils/tidy_conf/utils.py record_question: the question's id hashes the full prompt text, which includes the YAML and remote values. That is correct as the PR intends: a changed value is a new finding. It also means any later rewording of a prompt in interactive_merge.py re-queues every standing question once. A comment next to the prompts would warn whoever edits them.

5. Tests

• Invariant tests: tests/test_desk_queue.py covers the queue's invariants on every source and kind it names: stable ids, a re-add is a no-op, handled ids stay handled, the 50-item and 20,000-character caps, and a broken file moved to .bad. tests/test_merge_questions.py covers the question path for the place, cfp and fuzzy-name questions, and checks that agreeing sources ask nothing. tests/test_enrich_tba.py covers every TBA word, plus None, the cancelled cases and a dated cfp.
• Fix in 7d6c428, new test against base and head:
  • Base (origin/main utils/enrich_tba.py, run with this PR's tests/test_enrich_tba.py): 2 failed, 6 passed in 0.18s. The failures are FAILED tests/test_enrich_tba.py::test_cancelled_and_dated_conferences_are_left_out[Cancelled] and FAILED tests/test_enrich_tba.py::test_cancelled_and_dated_conferences_are_left_out[cancelled], each failing with Found 1 conferences with TBA CFP.
  • Change: 8 passed in 0.06s.
• Change, full suite (REVIEW.md's pinned uv run --python 3.11 … pytest tests/ -q -x -p no:cacheprovider): 1171 passed, 5 skipped, 3 xfailed, 2 xpassed, 153 warnings in 50.29s.
• Lint (uvx ruff@0.4.4 check utils tests): All checks passed!
• Data (… python utils/sort_yaml.py --skip_links): exit 0, and git diff --stat -- _data is empty.
• JS (npm ci && npm test): Tests: 423 passed, 423 total across 15 suites. No static/js/ changes.

6. Bounds

• Route: the diff and PR came from the REST API. The head was cloned at refs/pull/500/head, and git rev-parse matched 7d6c428eaf9f280abd18645fe63f0317b61722e7.
• Read in full: utils/desk_queue.py, .github/actions/desk-queue-write/action.yml, check-conference-update.yml, redirect-scan.yml, the diffs of canonical-source-update.yml, enrich-tba-conferences.yml, .claude/routines/desk.md, interactive_merge.py, utils.py, links.py, queue_out.py, both importers and desk_worklist.py.
• Skimmed: the 884-line removal from enrich_tba.py (checked only that nothing still calls the removed code), tests/test_desk_queue.py and the deleted workflows.
• Not run:
  • the composite action and workflows. There is no Actions runner or actionlint here, so the branch-creation, push-retry and continue-on-error paths rest on reading the code and on the PR's own local-remote test;
  • pre-commit.ci - pr. I did not fetch its status;
  • pixi run links (network, and up to 150 min).
• Pre-existing: none noticed.

---

Generated by Claude Code

[JesperDramsch]

Routine review

Re-review of #500 at 87bc32e. The earlier review at 7d6c428 raised no Important findings, so this one checks only what changed in 87bc32e and, under the contract's re-review rule, leaves out Nits.

1. Request

No linked issue, so the request is the PR body. It is the same as last round:

• the desk queue (utils/desk_queue.py, .github/actions/desk-queue-write);
• check-conference-update.yml as a queue writer;
• queued merge questions from the canonical import;
• the weekly redirect-scan.yml;
• retiring scheduled-extension-checker.yml and the full mode of enrich_tba.py;
• leaving cancelled editions out of the TBA link enrichment.

This round adds 87bc32e, which addresses the earlier review's two Nits:

• clean_watch_title now reads a trailing year;
• a comment in interactive_merge.py explains what a question id hashes.

2. Checked against

• Date: Thu Oct 15 11:19:10 UTC 2026 (date).
• No external facts are relied on.

3. Needs your decision

Both questions from the earlier review still apply, and this push does not change them:

1. Merge order and the gap before the desk Routine is live. Recommendation: set up the Routine on the day you merge, so no series reaches the 50-item cap.
2. Deleting ANTHROPIC_API_KEY. Recommendation: delete it once this is on main.

4. Findings

Important: None. The earlier review had none to resolve, and 87bc32e adds none.

What I checked for the new TRAILING_YEAR rule (utils/desk_queue.py:64):

• Where it applies. clean_watch_title now also feeds record_names (desk_queue.py:208), _queue_names and _queue_year (desk_worklist.py:418,430) and the changedetection step (check-conference-update.yml:93).
• Effect on names. It strips a name only when the name ends in a year from 1900 to 2099. No conference value in _data/*.yml and no key or variation in titles.yml ends that way (grep), so no tracked series name gets cut short.
• Edge cases. "Python 3000" and "PyCon 2027 Online" keep their names, and the new tests pin both.

5. Tests

• Invariant tests:
  • tests/test_desk_worklist.py::TestCleanWatchTitle covers both new cases: a trailing year, and one with trailing whitespace. It also covers the two shapes that must not count: a number outside the range and a year in the middle of the name.
  • tests/test_desk_queue.py::test_hand_typed_year_is_not_part_of_the_series checks that "PyCon US 2027" is queued to queue/pycon-us.json.
• Change, full suite (REVIEW.md's pinned uv run --python 3.11 … pytest tests/ -q -x -p no:cacheprovider): 1175 passed, 5 skipped, 3 xfailed, 2 xpassed, 153 warnings in 58.40s.
• Lint (uvx ruff@0.4.4 check utils tests): All checks passed!
• Data (… python utils/sort_yaml.py --skip_links): exit 0, and git diff --stat -- _data is empty.
• Not rerun: the base-vs-change run. 87bc32e is a behaviour change that came from a review Nit, not a bug fix.

6. Bounds

• Route:
  • The PR came from the REST API. I cloned the head at refs/pull/500/head, and git rev-parse matched 87bc32efd1112bfe10d8bade7c43b7fb999bf6fa.
  • I diffed this round as 7d6c428..87bc32e.
• Read in full: that diff (4 files), and the call sites of clean_watch_title in desk_queue.py, desk_worklist.py and check-conference-update.yml.
• Not reread: the rest of the PR, which the earlier review covered at 7d6c428.
• Not run:
  • the composite action and workflows: there is no Actions runner here;
  • npm test: no static/js/ or tests/frontend/ changes;
  • pre-commit.ci - pr: I did not fetch its status;
  • pixi run links.
• Pre-existing: none noticed.

---

Generated by Claude Code