Skip to content

docs(agents): second opinions through the Gemini API, no agy from Claude - #12226

Merged
MarkusNeusinger merged 9 commits into
mainfrom
docs/gemini-api-second-opinion
Oct 22, 2026
Merged

MarkusNeusinger merged 9 commits into
mainfrom
docs/gemini-api-second-opinion

Conversation

@MarkusNeusinger

@MarkusNeusinger MarkusNeusinger commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Google's Antigravity FAQ ("Wrapping agy inside third-party coding agents or applications … is not supported under consumer plans") and a staff reply on 2026-10-16 (forum topic 189905: "No under consumer plans", even for the unmodified binary as a local subprocess) rule out the read-only agy -p second opinions docs(agents): delegation lanes and Gemini tasks through issues #12223 introduced. Claude stopped running agy on 2026-10-21.
  • Second opinions and first PR reviews go through the new zweitmeinung project skill: .claude/skills/zweitmeinung/gemini_review.py calls Gemini through Vertex AI in project anyplot, location global (the site's agents stay on eu), with read-only tools over tracked repository files, prefetched data, and the public API; no shell, no credentials. The owner's split (2026-10-21/22): Gemini 3.8 Flash on Flex PayGo (half price) for routine first reviews, on probation against Copilot's findings; Gemini 4 Argon, thinking high, for plans and second opinions.
  • Cost work after a comparison run on three merged PRs (feat(review): ordered colours, no redundant labels, readable titles #12203, feat(agent-chat): the data box shows what it waits for #12221, fix(deploy): one deploy per service at a time, oldest build first #12222): every tool turn re-sends the conversation, so --diff-base inlines the diff and changed files and read_files reads several files per call. Two loop bugs found there are fixed and pinned by tests: a text part beside the last function responses made Vertex answer 400, and a last turn without tools lost the implicit cache while Flash kept calling tools. The budget is now enforced through the tool responses with one tool configuration for all turns. Measured: Flash $0.07–0.28 per review, Argon $0.57–1.16.
  • Flex gets 5 minutes per run (and per request); after that, or after a timeout or a 429/5xx, the run goes online (flex_fallback, per_turn_seconds). The SDK retries timeouts too: the first Flash review of this PR waited over an hour on 15-minute attempts, and with a 5-minute cap per request the next one still ran for over 25 minutes.
  • Hardening after Copilot's reviews: a symlink is visible only when its target is itself a visible entry (a tracked review.txt -> .env exposed the untracked .env before), secret-name patterns apply to every path component of --data files, lock files stay out of the inlined diff and the diff is capped at 120,000 characters, a backtracking search pattern is stopped after 10 seconds, a line longer than the tool limit is cut instead of looping, and an unencoded URL is refused instead of crashing the run.
  • gemini-task: Gemini pushes its gemini/* branch and opens a draft PR labelled gemini. CI runs on drafts, Copilot does not (review_draft_pull_requests: false in the ruleset), and Claude marks the draft ready after its own review. The owner may let Gemini run tests and builds unattended in his own agy session; only he starts agy.
  • CLAUDE.md and .claude/guardrails.md describe the lanes and costs; the guardrail test anchors pin the obligations (Flash/Argon split, draft PRs only, Claude never runs agy).

Plan

N/A. The owner chose the Vertex route on 2026-10-21 after reading a research report on the policy, then the Flash/Argon split, the draft-PR hand-over, and moving the skill into the repository on 2026-10-22.

Test plan

  • uv run pytest tests/unit tests/integration -n 8 (new tests/unit/test_gemini_review.py: confinement, http_get refusals, inlining caps, and the tool loop against a fake Vertex client)
  • uv run ruff check . && uv run ruff format --check .
  • uv run python -m tools.changelog check --base origin/main
  • Flash smoke run with --max-turns 1 on Vertex: forced last turn answers (exit 0), traffic_type ON_DEMAND_FLEX
  • Gemini 3.8 Flash first review of this PR through the repository script: 3 of 5 findings applied (owner-only PR comments for Gemini, git diff -z, *api_key*); 2 skipped (callers already resolve check_attachment's root, and the SDK returns finish_reason as an enum). Flex gave no answer on the first turn, so the run went online: 9 turns, $0.86.
  • Copilot: three rounds on the rework, 9 findings, all applied

🤖 Generated with Claude Code

https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

…ns agy

Google's Antigravity FAQ says wrapping agy inside third-party coding
agents is not supported under consumer plans, and a staff reply of
2026-10-16 (Google AI Developers Forum, topic 189905) answered a setup
like ours with "No under consumer plans", even for the unmodified binary
run read-only as a local subprocess. Google recommends an AI Studio or
Gemini Enterprise API key instead.

- CLAUDE.md: first reviews and second opinions go to Gemini 4 Argon
  through the Gemini API with the owner's AI Studio key; Claude never
  runs agy itself.
- guardrails.md: the lanes table and a dated paragraph with the reason.
- gemini-task skill: only the owner starts agy; Claude's second opinions
  use the API.
- Instruction tests anchor the two new rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
Copilot AI balanced review requested due to automatic review settings October 21, 2026 19:03
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The documentation and test anchors consistently implement the intended policy change, with no unresolved findings.

No open issues 🎉

What changed in this PR

Updates agent delegation guidance to use the Gemini API for second opinions, replacing the read-only agy route introduced in #12223.

Changes:

  • Routes first reviews and second opinions through the Gemini API.
  • Restricts agy use to owner-operated sessions.
  • Updates instruction-test anchors and adds a changelog fragment.
File Description
tests/​unit/​test_agent_instructions.py Anchors the API route and prohibition on Claude running agy.
CLAUDE.md Updates the delegation rule.
changelog.d/​gemini-api-second-opinion.md Records the policy change.
.claude/​skills/​gemini-task/​SKILL.md Clarifies owner-only agy use.
.claude/​guardrails.md Updates delegation guidance and explains the policy rationale.

Review effort: Balanced

@codecov

codecov Bot commented Oct 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…io key

The owner chose Vertex AI (project anyplot, location global) over an AI
Studio key: it runs on the existing Google Cloud login, bills to the same
account and its Ultra credit, and needs no new key. The site's agents
stay on eu. Gemini gets read-only tools over tracked repository files,
prefetched data and the public API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The documentation and matching test-anchor changes are consistent, with no blocking issues identified.

No open issues 🎉

Review effort: Balanced

MarkusNeusinger and others added 2 commits October 22, 2026 07:52
…opinion skill in the repo

The owner found the first Argon review ($1.01) too expensive at about eight
merged PRs a day and decided on 2026-10-21 and 2026-10-22:

- Routine first reviews run on Gemini 3.8 Flash with Flex PayGo (half price,
  global only); plans and second opinions stay on Gemini 4 Argon with
  thinking high. Flash is on probation, measured against Copilot's findings
  on real PRs.
- Gemini hands its work over as a draft PR labelled `gemini`: CI runs on the
  draft, Copilot does not (the ruleset skips drafts), and Claude marks it
  ready after its review. The owner may let Gemini run tests and builds
  unattended in his own agy session.
- The `zweitmeinung` skill and its `gemini_review.py` move from the user's
  home into the repository, with unit tests.

The script changes come from a comparison run on three merged PRs: every tool
turn re-sends the whole conversation, so the diff and the changed files are
inlined (`--diff-base`) and `read_files` reads several files per call. A text
part next to the last function responses made Vertex answer 400, and a last
turn without tools lost the implicit cache while Flash kept calling tools, so
the tool configuration now stays the same and the budget is enforced through
the tool responses, with at most two refused calls before the run ends.

The changelog bullet no longer says the gemini-task skill sends reviews to
Vertex, and the guardrail test anchors now pin the obligations rather than
the "Vertex AI in project anyplot" subject (both found by the Flash review).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
The first Flash review of this PR waited over an hour on one open request.
The SDK retries timeouts as well as 429 and 5xx, so five 15-minute attempts
per turn could pile up. A Flex request is now tried once with a five-minute
timeout; after a timeout or a 429/500/503 the turn and the rest of the run
use the online client, and the summary reports `flex_fallback`. Token
counting always goes online.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Symlink aliases can expose excluded files, and review completion, prompt budgeting, and repeated hand-overs have unresolved failure cases.

5 open issues

Review effort: Balanced

Comment thread .claude/skills/zweitmeinung/gemini_review.py
Comment thread .claude/skills/gemini-task/SKILL.md
Comment thread .claude/skills/gemini-task/SKILL.md
Comment thread .claude/skills/zweitmeinung/gemini_review.py Outdated
Comment thread .claude/skills/zweitmeinung/gemini_review.py
From the Copilot review of the reworked PR:

- A tracked symlink such as `review.txt -> .env` resolved to a regular file
  inside the root and exposed the untracked `.env`; a data link
  `notes.txt -> server.pem` passed the name filter on its own name. A link
  is now visible only when its target is itself a visible entry: a tracked
  file, or a data file that passes the filter.
- The inlined diff had no size limit. Lock files are left out of it, and
  it stops at 120,000 characters with a note to read the files instead.
- A last response that held text beside pending tool calls after the nudge
  limit counted as a review; it now exits 3.
- gemini-task: a later hand-over updates the existing worktree with
  `pull --ff-only` instead of repeating the fetch into a checked-out
  branch, and a clone set up with the old `no-push` URL gets a one-line
  restore for the owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Model-supplied regular expressions can block synchronous searches and prevent the review’s turn and token safeguards from taking effect.

1 open issue
5 resolved since last review

Review effort: Balanced

Comment thread .claude/skills/zweitmeinung/gemini_review.py Outdated
The search tool compiles a pattern the model supplies, and Python's matcher
can backtrack for a very long time on a short line, for example `(a+)+$`
against forty `a` and a `!`. The review could then never reach its turn or
token limits. A SIGALRM timer now interrupts the search after ten seconds,
and the tool returns the hits so far with a note to simplify the pattern.

Found by the Copilot review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The review script still has data-filtering and tool-error handling defects that need correction before approval.

No open issues 🎉

3 previously missed issues

In code that hasn't changed since last review

Medium severity Apply secret-name checks to every path component

.claude/​skills/​zweitmeinung/​gemini_review.py:120

[P2] Apply the secret-name patterns to directory components too. credentials/service-account.json and secrets/payload.json both pass this check, so files under those paths in --data become visible to Gemini; check_attachment accepts them as well. This bypasses the filename safeguard when sensitive material is accidentally included in a clearly named directory. Match every relative path component and add regression cases for nested directories.

Medium severity Prevent non-advancing continuation on oversized lines

.claude/​skills/​zweitmeinung/​gemini_review.py:201

[P2] Avoid returning a continuation that cannot advance. If a line exceeds MAX_TOOL_CHARS, such as a single-line JSON file in prefetched data, this branch returns none of that line and tells Gemini to retry at the same start_line. Retrying produces the identical response and can waste the remaining tool budget; search also skips these lines. Return a clearly marked truncated excerpt and advance past that line, or support character-offset pagination. Add a regression case for an oversized first line followed by a normal line.

Medium severity Handle malformed HTTP URLs as tool errors

.claude/​skills/​zweitmeinung/​gemini_review.py:280

[P2] Return a tool error for malformed HTTP URLs instead of aborting the review. A call with https://api.anyplot.ai/specs?search=bar plot passes the host/path checks but raises http.client.InvalidURL before connecting. Neither this handler nor run_tool catches it, so the paid review exits without writing its output file or usage summary. Catch HTTPException and add a regression case for an unescaped query.

1 resolved since last review

Review effort: Balanced

MarkusNeusinger and others added 2 commits October 22, 2026 09:51
- Flex gets five minutes per run, not only per request: with the per-request
  cap alone, a Flash review still ran for over 25 minutes because each turn
  can queue again. Once the Flex turns add up to five minutes, the rest of
  the run goes online. The summary now reports `per_turn_seconds`.
- read_file cuts a line longer than the tool limit and moves on; before, a
  one-line JSON file returned nothing and the same start_line forever.
- The secret-name patterns apply to every path component of a --data file
  or attachment, so secrets/payload.json is filtered as well.
- http_get refuses a URL with spaces or control characters, and an
  http.client error becomes a tool error instead of ending the paid run.

The last three come from Copilot's "previously missed" list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
From the Gemini 3.8 Flash first review of this PR:

- gemini-task: when Gemini continues a task, it reads only the owner's PR
  comments (Claude comments through his account). On a public repository,
  anyone else's text is data, never an instruction.
- inline_changes lists the changed files with `git diff -z`, so a path with
  spaces or umlauts is inlined instead of skipped as a C-quoted name.
- The secret-name patterns catch `*api-key*` and `*api_key*`, not only a
  file named exactly `api-key`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
@MarkusNeusinger
MarkusNeusinger merged commit df3440a into main Oct 22, 2026
15 checks passed
@MarkusNeusinger
MarkusNeusinger deleted the docs/gemini-api-second-opinion branch October 22, 2026 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants