Skip to content

[Improve] Move Auto tool approvals to Nightly Experiments - #3207

Merged
daniel-lxs merged 3 commits into
developfrom
feature/auto-tool-approvals-nightly-2654xeceytjnx
Sep 25, 2026
Merged

daniel-lxs merged 3 commits into
developfrom
feature/auto-tool-approvals-nightly-2654xeceytjnx

Conversation

@roomote-roomote

@roomote-roomote roomote-roomote Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

​Opened on behalf of @daniel-lxs. Follow up by mentioning @roomote-roomote, in the web UI, or in Slack.

Related issue

Internal Roomote follow-up to #3180 and #3198; no separate issue is linked.

Why this PR exists

  • A maintainer explicitly invited this PR in the linked issue or discussion
  • I am a maintainer / this is internal Roomote work

What changed

  • Move the Auto tool approvals experiment switch from customer-preview Experimental settings to the hidden, admin-only Settings → Nightly Experiments page. Runtime reads are available only when the deployment opts into nightly experiments.
  • Use a fresh nightly metadata key so previously stored customer-preview opt-ins stay off. Legacy Auto mode settings now start off until a fresh choice, while their saved guidance is preserved.
  • Keep Jev-only Auto eligibility and leave manual per-tool approvals available on every deployment.
  • [Feat] Add opt-in launch criteria to custom automations #3159 separately adds the custom-automation launch-criteria switch to Nightly; this PR only adds Auto tool approvals and keeps the two experiments independent.

How it was tested

  • Focused Vitest: web client (19 tests), web server (10), deployment-experiment hooks (8), database approvals (29), cloud-agent Auto evaluation (20), and feature flags (18).
  • GitHub Test job 107708255392 failed on prior head 44413017 because eager @roomote/env initialization hit an incomplete mock in personal-policies.test.ts. Deferring the Env import until Auto state resolution fixed the test; rerun 107797325284 passed on head a6132ec5.
  • pnpm lint, pnpm check-types, pnpm lint:pre-push, and pnpm changeset status passed.
  • Live smoke on exact head a6132ec527cc0ba7cfa9c0951f472d7f48103f51: with the Nightly deployment opt-in and Auto enabled, DeepWiki read_wiki_structure on public RooCodeInc/Roomote ran without a human prompt and returned the wiki section titles. Its approval audit row was auto_approved with no human decider. The primary judgment usage record confirms OpenRouter Jev (typesafe/jev-1.13-20260917) succeeded. A second ask_wiki_question was assessed ask (guidanceFlagsRisk 0.85), showed an approval card, and was denied before the tool ran.
  • Eligibility was checked at this head: the stored openrouter selection resolved to a hosted judgment model with roomoteModel=false; a probe selecting roomote with no Roomote upstream returned no eligible model, as required by the [Chore] Auto tool approvals: Jev only, via a per-decision excludeRoomoteModel requirement #3198 Jev-only gate. The earlier disabled control occurred because the fresh test worktree was missing its configured local test environment and resolved only the intentionally invalid dev-login placeholder; loading the preserved test environment enabled the stored OpenRouter Jev selection. The Auto control remains disabled when an eligible Jev backend is absent.
  • Both smoke Sessions, their two approval rows, and all eight smoke-created inference usage rows were removed. The complete deployment_settings row was restored byte-for-byte; the [Improve] Make Auto tool judgments session-aware #3195 baseline stack is healthy, and no pending approvals remain.
  • The refreshed visual-proof capture timed out after the later test and reconciliation changes. The screenshots below are earlier visually verified captures from before the Auto card was rendered directly by NightlyExperimentsPage; they show the same card and placement, but do not prove the final render path, backend authorization, or runtime behavior.

Screenshots

Auto tool approvals on Nightly Experiments

Earlier verified capture from the local Admin UI with the nightly deployment opt-in enabled and Auto itself off; captured before Auto was rendered directly by NightlyExperimentsPage.

Experimental settings without the Auto tool approvals switch

Earlier verified capture of ordinary Experimental settings; this is not refreshed proof for the final branch state.

Checklist

  • The PR title follows the repo convention: [Fix], [Feat], [Improve], [Refactor], [Docs], or [Chore] followed by a user-facing description
  • This PR is small and scoped to one change
  • pnpm lint and pnpm check-types pass locally
  • I added tests or included a clear manual validation note above
  • I removed secrets, tokens, private keys, and customer data from code, logs, and screenshots
  • If this change should appear in the changelog, I ran pnpm changeset

@roomote-community

roomote-community Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

No code issues found. See task

Reviewed a6132ec

@roomote-roomote
roomote-roomote Bot marked this pull request as ready for review September 24, 2026 15:42
@roomote-roomote
roomote-roomote Bot marked this pull request as draft September 24, 2026 19:42
@daniel-lxs
daniel-lxs marked this pull request as ready for review September 25, 2026 01:44
@daniel-lxs
daniel-lxs merged commit 21bebc5 into develop Sep 25, 2026
17 checks passed
@daniel-lxs
daniel-lxs deleted the feature/auto-tool-approvals-nightly-2654xeceytjnx branch September 25, 2026 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant