Skip to content

review: a fallback stays inside the reviewer's model family - #205

Merged
askalf merged 1 commit into
mainfrom
reviewer-family
Nov 11, 2026
Merged

askalf merged 1 commit into
mainfrom
reviewer-family

Conversation

@askalf

@askalf askalf commented Nov 11, 2026

Copy link
Copy Markdown
Owner

What

A review's fallback model is used only when it is of the same family as the model that stopped.

Why

Which family reads a repository is a choice. Where one vendor's coding agent writes the pull requests, another vendor's model reads them, so the reviewer does not share the author's blind spots.

The fallback could undo that choice without anyone making it. A review whose model the endpoint parked or could not route continued on REDLINE_FALLBACK_MODEL whatever that model was, and the default is claude-opus-5-5. On a host that reads on GPT models, a parked seat would have handed the review to Claude, with one log line to show for it.

How

  • modelFamily(id) in scripts/redline/review.mjs: claude for claude-*, gpt for gpt-*, chatgpt-*, codex-* and the o3 / o4-mini style ids, else the id's leading run of letters and digits (gemini-3-pro is gemini). A provider path before the id is not part of it. sameModelFamily(a, b) is false when either id has no family.
  • The fallback branch of reviewChange checks the family at the point of use, so it covers the routine, deep and pin models alike. A fallback of another family is not used: the run logs <reason>; the fallback <model> is of another model family, so it is not used and the review stops and fails with the same error as a run with no fallback.
  • A fallback of the same family behaves as before, with the same log lines.

What changes for a host

  • A host that reads on Claude: nothing.
  • A host that reads on another family and leaves REDLINE_FALLBACK_MODEL unset: a parked or unroutable model now fails the review. It gets a fallback back by naming a model of its own family.
  • Hosted mode: nothing. It already sets no fallback.

Docs

  • README: the paragraph on the pin and fallback settings.
  • docs/hosted-quickstart.md: how to choose the reviewer's family.
  • CHANGELOG.md: November entry.

Tests

node scripts/redline/review.test.mjs: ten new checks.

  • A parked GPT model with a Claude fallback fails on the one call, posts nothing and logs the refusal.
  • The same the other way round (a Claude model, a GPT fallback).
  • An unroutable GPT model with a Claude fallback fails with ModelUnroutable.
  • A GPT model falls back to another GPT model, parked or unroutable.
  • modelFamily and sameModelFamily over listed ids, unlisted ids, provider paths and empty values.

The existing Claude to Claude fallback checks pass unchanged. node test-all.mjs passes but for two path checks in review.test.mjs that fail the same way on main on a Windows checkout; CI runs them on Linux.

A review whose model the endpoint parked or could not route continued on
REDLINE_FALLBACK_MODEL whatever that model was. A host that reads on one
vendor's models because another vendor's coding agent writes its pull
requests could have a review finished by the author's own family, with
nobody choosing that.

review.mjs now uses the fallback only when it is of the stopped model's
family (modelFamily, sameModelFamily). Otherwise it logs why and fails
the run, as it does with no fallback. The default fallback,
claude-opus-5-5, now serves only a Claude reviewer: a host that reads on
another family names a fallback of that family.

The README, the hosted quickstart and the changelog say so, and the
quickstart says how to choose the reviewer's family.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve. The family check runs before switching models, preserving the original error when a fallback is refused and retaining conversation history when it is allowed. The tests cover cross-family refusals for parked and unroutable models, same-family continuation, and model ID normalization; the documentation explains the configuration change.

@askalf
askalf merged commit 19e8159 into main Nov 11, 2026
13 checks passed
@askalf
askalf deleted the reviewer-family branch November 11, 2026 00:20
askalf pushed a commit to askalf/pitlane that referenced this pull request Nov 11, 2026
Moves this repository's askalf/ci pins (its reusable workflow calls,
their redline-ref inputs and any action pins) to
askalf/ci@19e8159, the main commit that
last changed the review, the fix or an action (main 2026-11-11,
askalf/ci#205). The scripts are fetched at that same commit. The pins
are the only change.

---------

Co-authored-by: askalf-pin-bump[bot] <352119504+askalf-pin-bump[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants