Paused — archived 2026-09

Mechanism/Upshift was an experiment in detecting and repairing AI-agent regressions across model upgrades. Development is paused while I pursue a new research direction.

The repository stays up as a technical artifact: the code is MIT-licensed and complete, and every run record behind every number below is committed here. Nothing is being maintained — issues and pull requests will not be triaged, and the CLI is pinned to whatever the provider SDKs looked like in September 2026.

The strongest result. On the real gpt-5.5 → gpt-5.6-sol upgrade, a 38-case booking agent regressed on 36 of 38 cases. Three stacked, full-suite-verified repairs restored 32; 4 remained broken. The verdict was STAY PINNED — the bar was every regression repaired, and 88.9% is not every. Full accounting.

What the evidence did not support, and why the work stopped: see What it has found so far. Three controlled A/B studies found that competent engineers close these migrations quickly and cheaply without the tool. That finding is recorded here rather than buried, because it is the most useful thing this repository knows.

upshift takes a tool-calling agent, runs its eval cases on two model versions, tells you with a p-value what regressed, tries a small set of repairs, and either hands you a patch it has verified against your whole suite — or tells you to stay pinned, and why. It is a local CLI: it runs on your machine, with your own API keys, and calls no service of ours. Every claim in this README is backed by run records committed in this repository.

• Two providers: OpenAI (chat/completions, responses) and Anthropic (messages).
• Statistics, not vibes: N runs per case (default 5), pass/fail as rates, Fisher exact p on every label, contested results re-tested on 2N. Deterministic checks only — no LLM judge.
• Repairs that earn their place: prompt edits, model params (effort included), tool schemas, endpoint routing. A repair is accepted only if it restores a broken case, breaks nothing, and survives a full-suite re-verification.
• Three verdicts: SAFE, SAFE WITH PATCH, STAY PINNED. The patch is git apply-able.
• upshift adapt reads your agent's codebase (Python, notebooks) and writes the adapter with a file:line citation for everything — anything it can't trace to source is omitted and reported, never invented.

Install

Python 3.12+ and an environment with uv or pipx:

uv tool install git+https://github.com/Mechanism-world/upshift

pipx install git+https://github.com/Mechanism-world/upshift

Verify: upshift --version. Nothing is installed globally beyond the upshift command; no daemon, no account, no telemetry.

Quick start

Sixty seconds, no API key. The built-in deterministic simulator runs the whole pipeline:

upshift init my-agent
upshift upgrade --agent my-agent --provider sim \
  --baseline-model sim-5.5 --candidate-model sim-5.6-sol --tag demo

You'll watch a baseline run, a candidate run that regresses 36/38 cases, a repair loop that restores all of them, a SAFE WITH PATCH verdict, and a patch — in about a second, for $0. (Simulator results validate the machinery, never a real model: every report is tagged with its provider, and sim evidence can never produce a real verdict.)

The Anthropic-shaped simulator (sim-fable-5 → sim-fable-5-1) reproduces the documented Claude Fable 5.1 breaks the same way, for $0.

A real upgrade

Point upshift at your own agent — generated by adapt or written by hand (ADAPTER.md) — put the key in your environment or a local .env, and run:

# OpenAI
upshift upgrade --agent my-agent --flex \
  --baseline-model gpt-5.5 --candidate-model gpt-5.6-sol --tag my-upgrade

# Anthropic
upshift upgrade --agent my-agent --provider anthropic \
  --baseline-model claude-fable-5 --candidate-model claude-fable-5-1 --tag my-upgrade

--flex uses OpenAI's flex tier (~50% cheaper, stacks with prompt caching); the Anthropic path caches the prompt prefix automatically. upshift cost prints the exact recorded spend. Runs are resumable — Ctrl-C exits immediately and a rerun picks up where it stopped.

--max-cost-usd N puts a ceiling on that spend. The priced cost of everything recorded under the run id (or, for upgrade, the whole --tag family: baseline, candidate and every repair screen and verification) is checked before each rep and between phases; on reaching the ceiling the command stops before the next API call, writes a COST_STOPPED.json marker instead of a verdict, and exits 3. Rerun the same command with a higher ceiling to resume.

Onboarding your agent in minutes

upshift adapt https://github.com/you/your-agent --out my-agent --flex

adapt statically ranks the repo, uses the model as an extraction engine over cited evidence, verifies every verbatim claim against the cited file mechanically, and writes the five adapter files plus an ADAPT_REPORT.md: what it found, what it inferred, what it could not determine, and exactly which lines to review. Measured on real repos — from zero-edit on a simple agent to an honest refusal on schemas buried across files: the adapt reports.

Framework agents (capture mode)

If the failing request is built inside pydantic-ai, litellm, LangChain, the Vercel AI SDK, the Claude Agent SDK or opencode, there is nothing to lift into five adapter files. So don't read the framework — record it. upshift capture stands between your agent and api.anthropic.com and writes down the bytes it actually sends:

upshift capture --out cap                      # terminal 1: a local recorder on 127.0.0.1:8787
ANTHROPIC_BASE_URL=http://127.0.0.1:8787 ./run-my-agent    # terminal 2: your agent, unchanged
                                               # Ctrl-C the recorder when you're done
upshift adapt --from-capture cap --out my-agent
upshift upgrade --agent my-agent --provider anthropic \
  --baseline-model claude-fable-5 --candidate-model claude-fable-5-1 --tag my-upgrade

adapt --from-capture calls no model and reads no source: the prompt, the tools, the params and the cases all come out of requests your agent really made. The recorder is loopback-only by default, never writes a credential or an account identifier to disk, relays a 400 verbatim, and reassembles SSE so a streaming agent adapts like a non-streaming one.

Where to point each framework. Every row was read from that framework's own source; the citations and versions are in docs/framework-mapping.md.

framework	set this
anthropic Python / TypeScript SDK	ANTHROPIC_BASE_URL=http://127.0.0.1:8787
pydantic-ai	ANTHROPIC_BASE_URL=http://127.0.0.1:8787
litellm	ANTHROPIC_API_BASE or ANTHROPIC_BASE_URL — the origin only; litellm appends /v1/messages itself
langchain-anthropic	ANTHROPIC_API_URL (else ANTHROPIC_BASE_URL); it sends no header of its own, so add --framework langchain-anthropic
Vercel AI SDK	ANTHROPIC_BASE_URL=http://127.0.0.1:8787/v1 — with the /v1: it requests baseURL + "/messages"
claude-agent-sdk	ANTHROPIC_BASE_URL in the process env, or ClaudeAgentOptions(env=…)
opencode	provider.anthropic.options.baseURL in opencode.json — no env var exists; same /v1 rule as the AI SDK

When a repair is accepted, the report and the patch say where that repair lives in your framework — AnthropicModelSettings(anthropic_effort=…), drop_params=True, providerOptions.anthropic.effort — with the file and line each mapping was verified at. A knob a framework does not have is reported as "not mapped", never guessed.

Live smoke on a real pydantic-ai agent, claude-fable-5 → claude-fable-5-1: reports/capture-pydantic-ai-smoke.md.

Let the application run itself (native runner)

If your agent already has a test or eval command, you don't have to rebuild it as a Python backend. Add a runner block to agent.json and upshift invokes your command once per case and rep, feeding the case on stdin and reading one JSON result line (protocol v1, ADAPTER.md); the same checks, statistics and verdicts apply. Running repository code needs --allow-runner; the child gets a minimal environment, a timeout, bounded output and a fresh copy of its working directory per rep. Reference runners for Python and Node are in examples/runners/. Results from a native run are stamped native_application — the only scope that means "verified in the application". Repairs are not generated in native mode; upshift measures regressions and verifies the patch you supply.

Verify the patch you actually ship

upshift verify-patch --agent my-agent --patch runs/my-upgrade/upgrade.patch --run runs/my-upgrade-final

Applies the exact exported patch to a clean copy, rebuilds every case's first request through the same code path — from the patched files alone, so a patch that lost the endpoint routing or a param rebuilds what it really says — and compares it with the run that verified the patch. Exit 0 only when every case in the patched cases.json was compared and matched byte for byte (cache keys and seeds excluded); a case the run never recorded, or a patched configuration that disagrees with the run's, is exit 2 with the reason named. The candidate model is the one field no patch can carry (it is a command-line argument), and the block says so explicitly. This closes the gap between "the repair idea worked" and "the file we exported is what was verified".

Operational notes: --max-cost-usd bounds spend for a whole upgrade (the default --budget of 24 counts every repair candidate screened, since siblings are now screened before one is accepted); --retry-errored re-runs reps that ended in a transient provider error (rate limits, capacity, 5xx) when resuming — those never count as behaviour.

What it has found so far

Every number below is reproducible from the committed records with upshift cost and upshift diff.

OpenAI, gpt-5.5 → gpt-5.6-sol. Our 38-case booking agent regressed 36/38 (the documented function-tools 400 on chat/completions, then behavioral regressions once that was routed around). Three stacked, full-suite-verified repairs restored 32/36 with zero confirmed collateral — and the verdict was still STAY PINNED, because the bar is every regression repaired. Full accounting. Then shell_gpt (12k stars): 14/14 regressed on the same 400 with no workaround in its config; one one-line endpoint repair restored 14/14 — SAFE WITH PATCH, $0.56. Report · upstream issue.

Anthropic, Claude Fable 5 → Fable 5.1, run on release day. The cookbook SMS bot that forces tool_choice: any broke 5/5 cases on the documented 400 and was fully restored — drop the forced choice, add the documented instruction, one rung of effort — SAFE WITH PATCH, 0 broken, $0.93. A quickstarts agent built around parallel tool calls showed no regression at N=5. A third agent turned out to be broken before the migration. Report · upstream issues FACT#5, claude-cookbooks#854.

What the controlled studies showed

The runs above say the machinery works. They do not say it was worth running. Three controlled A/B studies asked that second question — a competent engineer with the official docs and the project's own tests (Arm A) against the same engineer plus upshift (Arm B), on a public repo neither arm had touched, with the protocol frozen and the patches judged by a blinded evaluator before any result existed.

Study	Target	Outcome
2026-09-08	shell_gpt → GPT-6 Astra	Break was documented in the 400 itself; upshift supplied confidence, not diagnosis
2026-09-11	lovia → GPT-6 Astra	Tie. Both arms found the same four defects and both patches were accepted at 42/42 live results. Arm B cost +6 min and ≈$0.55 and produced no run, no verdict and no flag.
2026-09-14	gptme → GPT-6 Astra	Loss. Arm A: 83 min, $0.80. Arm B: 112 min, $1.63. upshift took ~25 min and $1.42 for one finding that changed nothing, 0 accepted repairs, 2 false positives — and a STAY PINNED verdict that was wrong. Both arms passed blinded evaluation with zero regressions.

The conclusion the three agree on: these breaks are loud, documented and cheap to find. Engineers close them in about forty minutes for under a dollar without weakening a test. A detection-and-repair tool priced in engineer-minutes has to beat that, and this one did not. A confidence product that emits a confident wrong verdict has the worst failure mode available to it, and that happened once in three studies.

What remains unfalsified is the thesis this tool was never pointed at: silent behavioural drift — the upgrade that throws no error, passes the type checker, and quietly answers differently. Every study above tested a loud break, because loud breaks are what the release notes advertise. That is the open question this repository leaves behind.

Protocols, pre-registrations, sealed blinding mappings and both arms' patches are in the private ops repository; the frozen protocol and results are committed on experiment/astra-gptme-ab.

How it fits together

• The adapter (agent.json, system prompt, tool schemas, backend.py, cases/) — five files that describe your agent to upshift. The repair loop may edit only the first three; your backend and your cases are the yardstick and are never touched.
• The runner executes every case N times per model and records everything: the manifest, each API request and response verbatim, each tool execution, each check — under runs/<run_id>/. Runs resume from disk.
• The differ labels each case (stable-pass, stable-fail, regressed, improved, flaky) with a Fisher exact p, and classifies failures into signatures that drive repair.
• The repair loop screens a candidate on the broken cases, verifies it on the full suite, adjudicates contested statuses on 2N reps, and stacks accepted repairs. Rejected on confirmed evidence means never retried.
• The verdict and the patch. SAFE WITH PATCH requires every regression restored and nothing broken; anything less is STAY PINNED with the evidence attached.
• adapt and the simulators get you to a first run without hand-writing the adapter or spending money.

Design decisions, in one file: DESIGN.md. The adapter contract: ADAPTER.md.

Why single runs lie

Agents are stochastic. A borderline case passing 4/5, 3/5, 3/5, 4/5 across four runs — with or without a patch — is what we measured on our first real run, and a single-sample rule would have vetoed every behavioral repair we had. upshift never decides on one run: outcomes are rates against fixed thresholds (≥ 0.8 pass, ≤ 0.4 fail, otherwise flaky), contested statuses get 2N reps at unchanged thresholds, symmetric for restorations and vetoes, and the playbook contains no repair that dictates exact output phrasings — restoring an eval by overfitting to its assertions would make the number a lie.

Anthropic's official migration skill (/claude-api migrate) edits your code for a target model and "produces a checklist of items to verify manually." upshift is that verification: it runs your suite on both versions, proves each repair against it, and hands you the patch with the evidence — or tells you to stay pinned.

Security and privacy

What leaves your machine

upshift runs locally and sends nothing to Mechanism: there is no backend of ours, no telemetry, no analytics, no account, and no license check. What does leave your machine is what you asked upshift to send, to the provider you chose:

• Provider API calls transmit your prompts and your credentials. Every run sends your system prompt, tool schemas, eval-case messages and tool results to OpenAI or Anthropic (or to the base URL you set), authenticated with your key. That is the measurement; there is no way to test a model without talking to it.
• upshift adapt sends cited slices of your code to the extraction model. It reads the repository you point it at and puts the ranked excerpts in the prompt. Point it at a private repository and that repository's source reaches the model you configured.
• Everything else stays on disk. Run records, diffs, verdicts and patches are written to your local runs/ directory. Nothing is uploaded, committed or pushed for you.
• Your local transcripts can contain sensitive data. A run record holds your prompts, the models' outputs and your tool results verbatim — that is what makes it evidence, and it is also why you should read a run directory before you publish it. An adapt record additionally quotes the source it read.

git clone --depth 1 of a URL you pass to adapt is the only other outbound call. Details, and the guards each claim rests on: SECURITY.md.

Code that gets executed

• Backends you run are executed. The backend.py in an adapter is code, and upshift upgrade imports and calls it in your process; the shell_gpt adapter runs model-generated commands inside Docker with --network none. Read a generated backend.py before you run it.
• adapt reads; it does not execute. It never imports, builds or tests the repository it analyses.
• Vulnerability reports: see SECURITY.md.

Verification scope — what a green result actually proves

Every run, diff, verdict and report carries one of three scopes, derived from how the run executed (never declared by you):

• request_contract — upshift built the requests itself from your three patchable files and sent them, against a capture replay or a generated stub, so the result proves what the provider accepts or rejects about the request shape.
• adapted_agent — your adapter's backend.py executed real tool semantics, so the result proves the behaviour of the adapted reconstruction of your agent.
• native_application — your application's own entry point ran, with its own request-building code and the original configuration, so the result proves behaviour in the application itself.

No wording anywhere says "verified in the application" unless the scope is native_application.

Documentation

Goal	Start here
Describe your agent to upshift	ADAPTER.md
Understand every design decision and the statistics	DESIGN.md
Read the migration evidence	shell_gpt on gpt-5.6 · four Claude agents on Fable 5.1
See what adapt does on real repos	adapt reports
Capture a framework agent at the wire	docs/framework-mapping.md
What is actually implemented, and how far it is tested	docs/capabilities.md
What's out of scope, and why	ROADMAP.md · SCOPE.md
What changed	CHANGELOG.md

Honest limits

• Tested on our synthetic agent and a handful of open-source agents (one on OpenAI, four on Anthropic). That is evidence, not a benchmark suite.
• Two providers. Plain API agents on both; framework agents on Anthropic only, through upshift capture — no Google, no local models, and still no code-level framework integration (ROADMAP.md).
• Repairs are limited to prompts, params, tool schemas, and endpoint routing. Anthropic's thinking-block invalidation is detected and refused with the documented pointer, not repaired. The tool-schema repair covers one known tool shape.
• Tools that touch the world (shell, network, clusters) need a human-written deterministic backend; adapt stubs them with a TODO. Its generated eval cases are drafts.
• adapt doesn't yet chase identifiers across files, and reads notebooks as rendered cell text (cite lines accordingly).
• Capture mode has been exercised live against one framework (pydantic-ai, 3 cases at N=3 — a smoke, not evidence). The other seven mapping rows are verified from source, not from a live capture. A capture is a record of what your agent did, so the suite it produces is only as broad as the session you recorded.
• N=5 with Fisher exact tests tells you a 5/5 → 0/5 collapse is real (p ≈ 0.004). It will not resolve subtle single-case effects, and a clean result on one agent is not proof that a documented shift doesn't exist. Raise N if you need more power and can pay for it.

Development

git clone https://github.com/Mechanism-world/upshift && cd upshift
uv sync --group dev
uv run ruff check src tests agents && uv run pytest -q

macOS note: uv's editable-install .pth file sometimes gets the UF_HIDDEN flag and CPython skips it; tests self-heal via tests/conftest.py, and for the CLI entry point run chflags nohidden .venv/lib/python3.12/site-packages/*.pth. See CONTRIBUTING.md.

Status

This repository is archived and unmaintained as of September 2026. Issues, pull requests and discussions are not being triaged, and there will be no further releases. The final tagged version is the last state the work reached.

You are welcome to fork it. The parts most likely to be useful on their own are the model-comparison harness, the N-rep runner and recorder, the statistical differ, the repair loop, and the eval/verification infrastructure — see What to take from this repository below.

upshift was built by Mechanism.world and is released under the MIT License.

What to take from this repository

Everything here runs locally, holds no credentials, and is MIT-licensed. If you are building something adjacent, these are the pieces that stand alone:

Piece	Where	What it does
Runner + recorder	src/upshift/runner.py, recorder.py	Runs every eval case N times against two model versions, resumable, with every input, output, param and version written to disk so any diff stays inspectable later
Statistical differ	src/upshift/differ.py, stats.py	Fisher exact / Wilson intervals over pass rates — turns N reps into a claim about regression with a p-value, instead of a single-run anecdote
Repair loop	src/upshift/repair/	Signature-driven candidate generation, then accept-only-if: restores broken cases AND breaks nothing AND survives full-suite re-verification
Provider layer	src/upshift/providers/	OpenAI chat/completions, OpenAI responses, Anthropic messages behind one interface, with table-driven translation between the three request shapes
Capture	src/upshift/capture/	A loopback forwarding recorder that reconstructs an agent from its wire traffic — credential and account-id redaction, SSE reassembly, per-turn param derivation
Verification scope	src/upshift/verify_patch.py, verdict.py	The honest-verdict machinery: what a green result actually proves, what the patch cannot carry, and why a partial verification is never a passing one

The test suite (2,018 tests, uv run pytest) is the real documentation for all of it.

Contributors

License

MIT. This repository includes material derived from third-party projects under their own licenses and quotes provider documentation for interoperability — see THIRD_PARTY_NOTICES.md. OpenAI, GPT, Anthropic, and Claude are trademarks of their respective owners; upshift is not affiliated with or endorsed by either.