diff --git a/CHANGELOG.md b/CHANGELOG.md index cd40a09..26705b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed + +- `evalshift push` printed a traceback instead of the upgrade prompt when hosted EvalShift + refused the run with HTTP 402 right after auto-creating the project (over the monthly run + or concurrent-run limit, or a subscription that stopped paying). It now prints the + server's message and the billing link and exits 1. A 402 while auto-creating the + project — what an organization whose trial has ended gets — now shows the same prompt + instead of an access-permission hint. +- `evalshift push` printed a traceback when the connection failed while auto-creating the + project; it now prints the error and exits 1. + ### Changed - The EvalShift repositories moved from the `babaliauskas` GitHub account to @@ -19,6 +30,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `evalshift/evalshift-action` steps as well as the old `babaliauskas/evalshift-action` name, matching owner and repository names case-insensitively as GitHub does. +- Docs and the `init --ci` workflow comment describe hosted EvalShift's current plans: a + 30-day Pro trial for every new account, then Pro per organization (the hosted Free and + Team plans are gone). Local runs stay free and unlimited. ## [1.2.1] - 2026-10-01 diff --git a/DOCS.md b/DOCS.md index 9d5bc9f..824dca9 100644 --- a/DOCS.md +++ b/DOCS.md @@ -760,15 +760,15 @@ The full field-by-field data contract lives in [docs/hosted.md — Privacy model ### Plan limits -Local runs are always unlimited — plan limits apply only to what you push. When a push exceeds your org's plan (monthly runs, seats, retention) or the subscription has stopped paying, the server answers `402` and the CLI prints exactly what it said: +Local runs are always unlimited — plan limits apply only to what you push. Every new hosted account gets a 30-day Pro trial (no card) that covers every org it creates; after that each org needs its own Pro subscription. When a push exceeds your org's plan (monthly runs, seats, concurrent runs), the org's trial ended without a subscription, or the subscription has stopped paying, the server answers `402` and the CLI prints exactly what it said: ``` ✗ EvalShift: this run needs a paid plan. - Monthly run limit reached on the Free plan (50 of 50 runs used). + The free trial for this organization ended on 2026-11-02. Subscribe to Pro to push runs and keep the CI gate — existing runs stay readable. Upgrade: https://app.evalshift.dev/app/acme/settings/billing ``` -Exit code is 1 and nothing is uploaded. The CLI never decides entitlements itself and never retries a payment error — retrying would not change the answer. Your run and its `report.html` are already on disk under `.evalshift/runs/`; push it again after upgrading, or wait for the monthly reset. Transient upload failures (429, 5xx) *are* retried with backoff, which is a different thing entirely. +Exit code is 1 and nothing is uploaded. The CLI never decides entitlements itself and never retries a payment error — retrying would not change the answer. Your run and its `report.html` are already on disk under `.evalshift/runs/`; push it again once the org is subscribed (or, for the monthly quota, after the reset). An expired org stays readable — its runs, diffs and baselines still load. Transient upload failures (429, 5xx) *are* retried with backoff, which is a different thing entirely. --- @@ -777,7 +777,7 @@ Exit code is 1 and nothing is uploaded. The CLI never decides entitlements itsel `evalshift init --ci` scaffolds `.github/workflows/evalshift.yml` — a production-shaped, self-documenting workflow (the setup checklist lives in its header comment) with three jobs: - `discover` lists committed suites under `.evalshift/suites/*/golden.jsonl` — a suite added by `capture sync` is evaluated on the next run with no workflow edit, and a project with no suites yet skips green. Suites must be committed for CI to see them: keep `.evalshift/*` ignored but un-ignore `.evalshift/suites/` and `.evalshift/toolsets/`. -- `eval ` is a matrix job per suite (the action evaluates one suite per invocation) with `fail-on: policy` and `evalshift-version` pinned to the CLI that scaffolded the project. Each suite is selected with `suite-name:` — see [Selecting a suite: name, not path](#selecting-a-suite-name-not-path). `max-parallel` defaults to 1; raise it toward the hosted plan's in-flight ceiling (Free 1, Pro 5, Team 10). Only the first matrix job posts the PR comment — the comment marker is a constant, so multiple suites would overwrite one another. +- `eval ` is a matrix job per suite (the action evaluates one suite per invocation) with `fail-on: policy` and `evalshift-version` pinned to the CLI that scaffolded the project. Each suite is selected with `suite-name:` — see [Selecting a suite: name, not path](#selecting-a-suite-name-not-path). `max-parallel` defaults to 1; raise it toward the hosted plan's concurrent-run limit (Pro: 5; Enterprise: by contract). Only the first matrix job posts the PR comment — the comment marker is a constant, so multiple suites would overwrite one another. - `evalshift gate` is the single check to require in branch protection: it fails if any suite failed and passes when evaluation was skipped (fork PR, no suites, or `EVALSHIFT_TOKEN` not yet set). Don't require the per-suite jobs (dynamic names) or the `evalshift/regression` commit status (last writer wins across suites). Runs on pushes to main create the base-branch baselines PRs diff against, so the workflow cancels superseded runs on PRs only, never on main. diff --git a/docs/github-action.md b/docs/github-action.md index 7e7ff3f..24cc806 100644 --- a/docs/github-action.md +++ b/docs/github-action.md @@ -30,8 +30,8 @@ checklist) with three jobs: to the CLI that scaffolded the project: the CLI that *reads* the config in CI must be at least as new as the CLI that *wrote* it locally (`extra: forbid` rejects newer keys), and the CLI warns when the pin falls behind (or runs ahead of the local CLI) — see [Pin drift](#pin-drift). `max-parallel` defaults to 1 — - raise it toward your hosted plan's in-flight-run ceiling (Free 1, Pro 5, - Team 10). The PR comment is posted by the first matrix job only: the + raise it toward your hosted plan's concurrent-run limit (Pro: 5; + Enterprise: by contract). The PR comment is posted by the first matrix job only: the comment marker is a constant, so multiple suites would overwrite one another's summary. - **`evalshift gate`** — the join job to require in branch protection. It diff --git a/docs/hosted.md b/docs/hosted.md index 8d12f94..9792704 100644 --- a/docs/hosted.md +++ b/docs/hosted.md @@ -237,20 +237,26 @@ the wrong one fails exactly like a permissions problem. Running locally is always unlimited. Plan limits apply only to what you push. -When a push exceeds your organization's plan — monthly runs, seats, retention — -or the subscription has stopped paying, the server refuses the push with HTTP -402 and the CLI prints exactly what the server said: +Every new hosted account gets a 30-day Pro trial (no card) that covers every +org it creates; after that each org needs its own Pro subscription. + +When a push exceeds your organization's plan — monthly runs, seats, concurrent +runs — the org's trial ended without a subscription, or the subscription has +stopped paying, the server refuses the push with HTTP 402 and the CLI prints +exactly what the server said: ``` ✗ EvalShift: this run needs a paid plan. - Monthly run limit reached on the Free plan (50 of 50 runs used). + The free trial for this organization ended on 2026-11-02. Subscribe to Pro to push runs and keep the CI gate — existing runs stay readable. Upgrade: https://app.evalshift.dev/app/acme/settings/billing ``` `push` exits 1 and nothing is uploaded. The CLI does not evaluate entitlements itself and does not retry a payment error — retrying cannot change the answer. Your run and its `report.html` stay on disk under `.evalshift/runs/`, so you can -push the same run id again after upgrading, or after the monthly counter resets. +push the same run id again once the org is subscribed (or, for the monthly +quota, after the counter resets). An expired org stays readable — its runs, +diffs and baselines still load. Transient upload failures (HTTP 429 and 5xx from object storage) are a separate case and *are* retried with exponential backoff. @@ -355,5 +361,5 @@ credential file locally and repository secrets in CI. | `could not determine a valid 40-character git SHA` | `bundle` (or `push ` building a bundle) ran outside a git checkout, or in one with no commits, and `GITHUB_SHA` is unset. | Run inside a git repository with at least one commit, or set `GITHUB_SHA`. | | `project was not found` | The project does not exist and auto-create is disabled or not allowed. | Ask an owner to create it, use an org-scoped owner token, or enable auto-create. | | `cannot auto-create at ` | The message names the host it talked to and the server's status. Most often the host is not the one you meant: with no `--host` and no `EVALSHIFT_HOST`, an unset credentials file falls back to `https://api.evalshift.dev`, where your org does not exist. | Run `evalshift whoami` and check the host it prints. If it is wrong, `evalshift login --host `. If the host is right and the status is 403, the token lacks org access — see [Project auto-create](#project-auto-create). | -| `this run needs a paid plan` | The org's plan does not cover this push, or the subscription has stopped paying. | Open the upgrade URL printed with the message, or wait for the monthly reset and push the same run id again. See [Plan limits](#plan-limits). | +| `this run needs a paid plan` | The org's plan does not cover this push, its 30-day trial ended without a subscription, or the subscription has stopped paying. | Open the upgrade URL printed with the message; for the monthly quota you can also wait for the reset and push the same run id again. See [Plan limits](#plan-limits). | | `this run carries no migration policy` warning | No `migration_policy` is configured in `evalshift.yaml`, so the bundle has no `decision.policy`. | Add `migration_policy` to `evalshift.yaml` (see [Configuration](configuration.md#migration_policy)). Until then the gate has only whatever old web-app policy the project still has; with none, it reports `inconclusive` and never blocks the pull request unless the GitHub Action sets `require-policy: true`. | diff --git a/llms-full.txt b/llms-full.txt index d5dfebb..3a5b5c6 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -267,8 +267,9 @@ evalshift push [RUN_ID] [--bundle PATH] [--project] [--host] [--token] limit warning naming the server's 100 MB hard limit and uploads anyway (the hard limit is server-configurable, so the CLI quotes it rather than enforcing a stale copy). Plan limits: server answers 402 when the org's plan does not cover the push (monthly runs, - seats, retention) or the subscription stopped paying. CLI prints the server's sentence plus - `Upgrade: ` and exits 1; nothing uploaded, local run untouched. Never retried. + seats, concurrent runs), the org's 30-day trial ended without a subscription, or the + subscription stopped paying. CLI prints the server's sentence plus `Upgrade: ` and exits 1; + nothing uploaded, local run untouched. Never retried. Two policy notices, both before the success line: (1) a bundle with no decision.policy (no migration_policy configured) prints, before any HTTP call, "this run carries no migration policy; unless this project still has an old web-app policy, the hosted gate @@ -1262,7 +1263,7 @@ evalshift compare --suite-name --to gemini-3.1-pro-preview --gate critic # eval -> matrix job per suite (action evaluates ONE suite per invocation); # fail-on: policy; evalshift-version pinned to the scaffolding CLI (extra="forbid" config # from a newer CLI fails loudly on an older one -- reader >= writer; see CI pin drift); max-parallel: 1 by default — raise -# toward the hosted plan's in-flight ceiling (Free 1, Pro 5, Team 10); PR comment only +# toward the hosted plan's concurrent-run limit (Pro: 5; Enterprise: by contract); PR comment only # from the first matrix job (constant comment marker, multiple suites would overwrite); # provider key env matches init's --provider; skips on fork PRs (no secret access) and # no-ops green with a notice while EVALSHIFT_TOKEN is unset. diff --git a/src/evalshift_cli/cli/commands/_scaffold.py b/src/evalshift_cli/cli/commands/_scaffold.py index 9a62bec..a1e4794 100644 --- a/src/evalshift_cli/cli/commands/_scaffold.py +++ b/src/evalshift_cli/cli/commands/_scaffold.py @@ -231,8 +231,8 @@ strategy: # One suite regressing must not cancel the others' results. fail-fast: false - # Hosted EvalShift refuses uploads past your org plan's in-flight run - # ceiling (Free 1, Pro 5, Team 10). Raise this toward your plan's limit + # Hosted EvalShift refuses uploads past your org's concurrent-run + # limit (Pro: 5; Enterprise: by contract). Raise this toward it # to evaluate suites in parallel. max-parallel: 1 matrix: diff --git a/src/evalshift_cli/hosted/bundle_manifest.schema.json b/src/evalshift_cli/hosted/bundle_manifest.schema.json index 5686c95..d27e986 100644 --- a/src/evalshift_cli/hosted/bundle_manifest.schema.json +++ b/src/evalshift_cli/hosted/bundle_manifest.schema.json @@ -1018,7 +1018,7 @@ }, "GuardrailEvent": { "additionalProperties": false, - "description": "A guardrail or policy check. Imported agent traces only.", + "description": "A guardrail or policy check.\n\nReserved for a bring-your-own-agent trace; no current producer emits it \u2014\n``evalshift traces import`` writes such traces only to a local artifact,\nnot into a pushed bundle.", "properties": { "name": { "minLength": 1, @@ -1405,7 +1405,7 @@ }, "RetrievalEvent": { "additionalProperties": false, - "description": "A retrieval step. Imported agent traces only.", + "description": "A retrieval step.\n\nReserved for a bring-your-own-agent trace; no current producer emits it \u2014\n``evalshift traces import`` writes such traces only to a local artifact,\nnot into a pushed bundle.", "properties": { "documents": { "items": { diff --git a/src/evalshift_cli/hosted/push.py b/src/evalshift_cli/hosted/push.py index a5afb05..70e2730 100644 --- a/src/evalshift_cli/hosted/push.py +++ b/src/evalshift_cli/hosted/push.py @@ -108,6 +108,39 @@ def _upgrade_prompt(exc: HostedHTTPError) -> PushError: return PushError("\n".join(lines)) +def _initiate_after_create( + client: HostedClient, manifest: dict[str, Any], *, size_bytes: int +) -> dict[str, Any]: + """The ``POST /runs`` retried after auto-creating the project, mapped like the first. + + It runs inside the first attempt's ``except`` clause, so its own errors would skip the + sibling handlers and surface as a traceback. ``POST /runs`` answers 404 for a missing + project before any entitlement check, so a 402 for the monthly run or concurrent-run + limit, seat overage, or a subscription that stopped paying arrives on this retry, not + on the first call. + + Args: + client: The hosted client the first attempt used. + manifest: The bundle manifest sent as the run's create payload. + size_bytes: Size of the bundle file that will be uploaded. + + Returns: + The server's ``POST /runs`` response. + + Raises: + PushError: On any hosted error — the upgrade prompt for a 402, the server's + message otherwise. + """ + try: + return client.initiate_run(manifest, size_bytes=size_bytes) + except HostedHTTPError as exc: + if exc.status_code == _PAYMENT_REQUIRED: + raise _upgrade_prompt(exc) from exc + raise PushError(str(exc)) from exc + except (HostedNetworkError, HostedError) as exc: + raise PushError(str(exc)) from exc + + @dataclass(frozen=True, slots=True) class PushResult: """Outcome of a hosted push. @@ -239,7 +272,7 @@ def push_bundle( raise PushError(str(exc)) from exc _auto_create_project(client, project_slug=str(manifest["project_slug"])) project_created = True - response = client.initiate_run(manifest, size_bytes=upload_size_bytes) + response = _initiate_after_create(client, manifest, size_bytes=upload_size_bytes) except (HostedNetworkError, HostedError) as exc: raise PushError(str(exc)) from exc @@ -489,17 +522,25 @@ def _auto_create_project(client: HostedClient, *, project_slug: str) -> None: f"permission — check `evalshift whoami` reports this host and an org " f"named {org_slug!r}", ) from exc + except (HostedNetworkError, HostedError) as exc: + raise PushError(str(exc)) from exc if any(item.get("slug") == project for item in projects): return try: client.create_project(org_slug, slug=project, name=_name_from_slug(project)) except HostedHTTPError as exc: + if exc.status_code == _PAYMENT_REQUIRED: + # An expired org is read-only: creating a project is refused with the same 402 a + # push gets, and it deserves the same prompt rather than an access-permission hint. + raise _upgrade_prompt(exc) from exc raise PushError( f"cannot auto-create {project_slug!r} at {client.host}: " f"{_server_said(exc)}. Creating a project needs owner access to " f"{org_slug!r}; a scoped service-account key cannot do it — create the " f"project by hand, or push with an owner token", ) from exc + except (HostedNetworkError, HostedError) as exc: + raise PushError(str(exc)) from exc def _server_said(exc: HostedHTTPError) -> str: diff --git a/tests/conftest.py b/tests/conftest.py index 347eaaf..1af7f9e 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -40,7 +40,7 @@ from evalshift_cli.evaluators.base import EvalRecord from evalshift_cli.hosted.bundle import BundleBuildResult, build_bundle -from evalshift_cli.hosted.client import HostedHTTPError +from evalshift_cli.hosted.client import HostedError, HostedHTTPError from evalshift_cli.runner.checkpoint import append_call, write_state from evalshift_cli.runner.models import Call, RunModels, RunState @@ -264,8 +264,8 @@ class FakeHostedClient: projects: list[dict[str, Any]] | None = None created_project: dict[str, Any] | None = None finalize_error: HostedHTTPError | None = None - list_projects_error: HostedHTTPError | None = None - create_project_error: HostedHTTPError | None = None + list_projects_error: HostedError | None = None + create_project_error: HostedError | None = None host: str = "https://api.evalshift.test" on_finalize: Callable[[], None] | None = None initiate_calls: int = 0 diff --git a/tests/unit/test_docs_currency.py b/tests/unit/test_docs_currency.py index 0f13ccc..4222893 100644 --- a/tests/unit/test_docs_currency.py +++ b/tests/unit/test_docs_currency.py @@ -199,3 +199,31 @@ def test_no_doc_shows_a_removed_top_level_key(name: str) -> None: hits = [m.group(0) for m in pattern.finditer(text)] assert not hits, f"{name} shows removed top-level key(s) {hits}; that config would not load" + + +#: Exact substrings that described the v5 hosted plans. v6 (2026-10) has no hosted Free or +#: Team plan; the CLI itself is still free. Retiring another plan claim? Append it here. +RETIRED_PLAN_TERMS: tuple[str, ...] = ("Free 1, Pro 5, Team 10", "on the Free plan") + +_PLAN_COPY_FILES: tuple[str, ...] = tuple( + dict.fromkeys( + ( + *PROSE_FILES, + "docs/hosted.md", + "docs/github-action.md", + "src/evalshift_cli/cli/commands/_scaffold.py", + ) + ) +) + + +def _whitespace_normalized(name: str) -> str: + """The file's text with every whitespace run collapsed to one space, so a retired + phrase hard-wrapped across lines (as Markdown and code comments are) still matches.""" + return " ".join((REPO_ROOT / name).read_text(encoding="utf-8").split()) + + +@pytest.mark.parametrize("term", RETIRED_PLAN_TERMS) +def test_copy_does_not_describe_the_retired_hosted_plans(term: str) -> None: + offenders = [name for name in _PLAN_COPY_FILES if term in _whitespace_normalized(name)] + assert offenders == [], f"{term!r} still appears in {offenders}" diff --git a/tests/unit/test_hosted_cli.py b/tests/unit/test_hosted_cli.py index 5a48059..3fccf7b 100644 --- a/tests/unit/test_hosted_cli.py +++ b/tests/unit/test_hosted_cli.py @@ -1200,6 +1200,141 @@ def test_push_command_exits_non_zero_and_prints_the_upgrade_prompt_on_402( assert "Traceback" not in result.output +_TRIAL_ENDED = ( + "The free trial for this organization ended on 2026-11-02. Subscribe to Pro to push " + "runs and keep the CI gate — existing runs stay readable." +) +_RUN_LIMIT = "Monthly run limit reached on the Pro plan (4,000 of 4,000 runs used)." + + +def test_a_402_on_the_retry_after_auto_create_renders_the_upgrade_prompt( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """``POST /runs`` for a missing project answers 404 before any entitlement check, so a + 402 for being over the monthly run or concurrent-run limit, or for a subscription that + stopped paying, arrives on the *retried* call after auto-create. That retry runs inside + the first attempt's ``except`` clause, so its 402 used to skip every handler and reach + the user as a traceback.""" + bundle_path = _build_bundle_for_push(tmp_path, monkeypatch) + fake = _fake_client( + responses=[{"raise_404": True}, {"raise_error": _payment_required(_RUN_LIMIT)}], + projects=[], + ) + monkeypatch.setenv("EVALSHIFT_HOST", "https://api.evalshift.test") + monkeypatch.setenv("EVALSHIFT_TOKEN", "es_secret") + monkeypatch.setattr("evalshift_cli.hosted.push.HostedClient", lambda **_: fake) + + with pytest.raises(PushError) as excinfo: + _push(bundle_path, tmp_path, create_project=True) + + text = str(excinfo.value) + assert "this run needs a paid plan" in text + assert _RUN_LIMIT in text + assert "Upgrade: https://app.test/app/acme/settings/billing" in text + assert fake.created_project is not None + assert fake.initiate_calls == 2 + + +def test_any_other_error_on_the_retry_is_a_push_error_not_a_traceback( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + bundle_path = _build_bundle_for_push(tmp_path, monkeypatch) + fake = _fake_client( + responses=[ + {"raise_404": True}, + {"raise_error": HostedHTTPError(500, "database is down", code="internal_error")}, + ], + projects=[], + ) + monkeypatch.setenv("EVALSHIFT_HOST", "https://api.evalshift.test") + monkeypatch.setenv("EVALSHIFT_TOKEN", "es_secret") + monkeypatch.setattr("evalshift_cli.hosted.push.HostedClient", lambda **_: fake) + + with pytest.raises(PushError) as excinfo: + _push(bundle_path, tmp_path, create_project=True) + + assert "database is down" in str(excinfo.value) + + +def test_a_402_while_auto_creating_the_project_renders_the_upgrade_prompt( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """v6: an organization whose trial has ended is refused at ``POST /orgs/{slug}/projects`` + with a 402. That is a plan problem, not the "needs owner access" hint the generic create + failure prints.""" + bundle_path = _build_bundle_for_push(tmp_path, monkeypatch) + fake = _fake_client( + responses=[{"raise_404": True}], + projects=[], + create_project_error=_payment_required(_TRIAL_ENDED), + ) + monkeypatch.setenv("EVALSHIFT_HOST", "https://api.evalshift.test") + monkeypatch.setenv("EVALSHIFT_TOKEN", "es_secret") + monkeypatch.setattr("evalshift_cli.hosted.push.HostedClient", lambda **_: fake) + + with pytest.raises(PushError) as excinfo: + _push(bundle_path, tmp_path, create_project=True) + + text = str(excinfo.value) + assert "this run needs a paid plan" in text + assert _TRIAL_ENDED in text + assert "owner access" not in text + + +@pytest.mark.parametrize("failing_call", ["list_projects", "create_project"]) +def test_a_network_error_while_auto_creating_the_project_is_a_push_error( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + failing_call: str, +) -> None: + """Auto-create runs inside the first attempt's ``except`` clause, so a connection + failure there skips the sibling handlers unless it is mapped where it happens.""" + bundle_path = _build_bundle_for_push(tmp_path, monkeypatch) + fake = _fake_client( + responses=[{"raise_404": True}], + projects=[], + **{f"{failing_call}_error": HostedNetworkError("could not reach api.evalshift.test")}, + ) + monkeypatch.setenv("EVALSHIFT_HOST", "https://api.evalshift.test") + monkeypatch.setenv("EVALSHIFT_TOKEN", "es_secret") + monkeypatch.setattr("evalshift_cli.hosted.push.HostedClient", lambda **_: fake) + + with pytest.raises(PushError) as excinfo: + _push(bundle_path, tmp_path, create_project=True) + + assert "could not reach api.evalshift.test" in str(excinfo.value) + + +def test_push_command_prints_the_upgrade_prompt_when_the_retry_after_auto_create_is_refused( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """End to end: exit 1, the server's sentence, and no escaped exception.""" + bundle_path = _build_bundle_for_push(tmp_path, monkeypatch) + fake = _fake_client( + responses=[{"raise_404": True}, {"raise_error": _payment_required(_RUN_LIMIT)}], + projects=[], + ) + monkeypatch.setenv("EVALSHIFT_HOST", "https://api.evalshift.test") + monkeypatch.setenv("EVALSHIFT_TOKEN", "es_secret") + monkeypatch.setattr("evalshift_cli.hosted.push.HostedClient", lambda **_: fake) + + result = runner.invoke( + app, + ["push", "--bundle", str(bundle_path), "--config", str(tmp_path / "evalshift.yaml")], + ) + + # CliRunner catches an escaped exception into ``result.exception`` (exit 1, no printed + # traceback), so the exit code alone cannot tell a handled refusal from a crash. + assert result.exception is None or isinstance(result.exception, SystemExit) + assert result.exit_code == 1 + assert "needs a paid plan" in result.output + assert _RUN_LIMIT in result.output + + def test_transient_retry_statuses_never_include_payment_required() -> None: """Guard: a future edit to the retry set must not start retrying a payment error.""" assert 402 not in _TRANSIENT_STATUSES