Skip to content

fix(agents): platform-started turns run on the agent's own model (#4442) - #4466

Merged
vybe merged 13 commits into
devfrom
feature/4442-wake-model
Nov 11, 2026
Merged

vybe merged 13 commits into
devfrom
feature/4442-wake-model

Conversation

@vybe

@vybe vybe commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Turns the platform starts on an agent's behalf (caller wakes, operator answers, push /chat turns) resolve their model through the agent's stored default model. The open-source build never wrote that value, so every such turn fell through to the platform default instead of the model the agent was configured with.

This change fills it:

  • Creation: a new agent's default model is seeded from its template (both model: at the top level and the runtime block, including github: templates).
  • Settings save: saving a model in Settings writes it; a failed save puts the previous model back and names the failure (model_not_saved).
  • Boot fill: on startup, agents with no stored default model get one, read from the agent's live model first and its template file only as a fallback.
  • The push /chat path looks the model up the same way.

Fixes #4442

Tests

  • tests/unit/test_4442_wake_model.py (59 passed) plus the 11 suites the plan pins: default-model resolution, set-model errors, caller wake, operator resume, reminders, human-only routes, auth wiring. All green, run file by file on eca31da90.
  • A wider -k "model or ent752" sweep: 974 passed, 3 failed. All three fail because numpy is missing from the local test venv. They are memory-embedding tests that this diff does not touch.

Review

Reading review + security pass: MERGEABLE AFTER FIXES, both important findings fixed in eca31da90 (boot fill prefers the live model; restore on a failed save). No security findings.

Draft until the operator's merge call.

🤖 Generated with Claude Code

Trinity Agent (trinity) and others added 7 commits November 11, 2026 01:20
… boot (#4442)

Adds services/agent_model_seed: declared_model (runtime.model, else a
top-level model:), the fill-blank seed, and a one-shot boot pass that
writes a NULL default_model from the container's AGENT_RUNTIME_MODEL,
else from the running agent's template.yaml. Writes go through
set_default_model_if(name, None, ...), so a value another writer set
is never overwritten and concurrent workers make one write; only a
value unrunnable_reason passes is written. docker_service gains
read_runtime_model, which returns that one env key and logs failures
by type only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…4442)

_register_agent writes default_model from the request's runtime_model or
the model the template declares (runtime.model, else a top-level model:).
Both template paths feed it: the github: path never set runtime_model, so
its agents' AGENT_RUNTIME_MODEL stayed empty and their wakes ran on the
platform default. Fill-blank and catalog-gated; never fails a creation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

PUT /api/agents/{name}/model kept the value in the container's memory
only, so the turns the platform starts never saw it and a restart lost
it. On a Claude Code agent the accepted value is now stored as
default_model and audited (update_agent_model). The agent stays the
judge of the value (#3677); a failed store after the agent accepted
answers 500 model_not_saved instead of a silent split. Route tier
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sync /chat that names no model left the choice to the container's
in-memory value, which falls back to its env after a restart, and left
model_used empty. build_chat_payload now resolves it through
resolve_default_model (fail-open: on an error the agent decides, as
before) and stamps it on the row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…4442)

53 unit tests: the declared-model parse; the create seed for a template
declaring a top-level model:; the boot fill (env, then the live
template.yaml; one write across racing workers; the env never logged);
the PUT store, refusal, failed-save and non-Claude cases; push /chat
resolution and stamping; per-kind execute_task resolution; pinned
models win; an AST guard that the wake producers pass no model.

Mutations shown red: dropping the PUT store, the boot fill's template
fallback, the create seed's declared model, the push /chat resolve, or
making execute_task resolve unconditionally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed Settings save puts the old model back (#4442)

Review fixes I1 and I2 (chain-hub-p1-4442-fix).

I1: a Settings pick made before #4442 lived only in agent memory, and
push /chat now always sends the resolved model, so a fill from the env
or template would have overridden that pick. fill_blank_default_models
now asks a running agent's GET /api/model first, then the container env,
then template.yaml. The allowlist (unrunnable_reason) and fill-only-NULL
(set_default_model_if) guards are unchanged.

I2: model_not_saved said the agent "now runs" the new model, but backend
turns send the stored value. set_agent_model reads the agent's model
before the PUT, and if the store fails it puts that model back, so the
agent and the column agree. The message names the model the agent keeps
running; `restored` says whether the put-back held, and the message
changes when it did not.

Mutation: with both fixes reverted, five new tests went red:
test_boot_fill_takes_the_running_agents_live_model_over_its_env,
..._falls_back_to_env_then_template_..., ..._keeps_the_allowlist_...,
test_a_failed_save_puts_the_agents_previous_model_back,
test_a_failed_save_that_cannot_be_put_back_says_so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@trinity-ability trinity-ability left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Validation Report: #4466

Lane B (no security-path or schema trigger) · head eca31da90 · 10 files (+1015/−14) · targets dev ✅ · Fixes #4442 ✅ (P1 bug, status-ready)

Summary

Category Status Notes
Commits / base / size ✅ conventional prefixes; dev; 10 files
Status auto-promotion ✅ Fixes #4442 is in the body
Journey Impact ✅ check_journey_impact.py exit 0. The body has no declaration, which is allowed, but the template asks for one
Architecture doc ✅ architecture/execution.md names every new writer and drops "There is no HTTP setter"
Security §4.1–4.10 ✅ the only hits are identifiers in context lines, a test fixture SECRET, and the agent-container path /home/developer/template.yaml
Build / config packaging ➖ no new top-level backend directory and no new env var
Audit rows §4.11 ✅ PUT set_agent_model: audit via chat.py::set_agent_model (update_agent_model)
Cross-surface drift §5.5 ✅ no agent-server twin touched. The router header says the model routes are UI-only; chat.ts sends nothing to /model
Tests (run file by file here) ✅ test_4442_wake_model.py 59 passed · test_3677_set_model_agent_error.py 15 passed
Execution coverage / mutation ✅ reverting chat_execution_service.py turns 2 tests red; removing the Settings store and the create seed turns 8 red; main.py boot wiring is pinned by AST
Structural review ⚠️ no /review is posted on the PR (the body describes one), so this report does not cover races or SQL
Merge gate (Tier 1) ⚠️ pytest (head, seed 12345) and both CodeQL analyses are still pending; dev-ci is success. The PR is a draft
Product fit ✅ approved bug; all 3 acceptance criteria are met for a stored model the resolver accepts

Critical: push /chat now overrides a Settings pick that the agent accepts but the resolver refuses

build_chat_payload now always sends resolve_default_model(...).model when the request names no model. When the stored value fails unrunnable_reason, that is the platform default. The agent server prefers request.model over its own current_model (agent_server/routers/chat.py:106), so the platform default replaces the agent's pick. Before this PR, the payload carried no model and chat ran on the pick.

The agent's PUT /api/model accepts any claude-* string, and ModelSelector.vue allows free text: its comment names claude-sonnet-4-6[1m] as the example. The catalog allowlist is narrower:

claude-sonnet-4-6[1m]     not_in_catalog
claude-opus-5-5[1m]       not_in_catalog
claude-sonnet-4-20250514  not_in_catalog

A scratch test using this PR's own _payload helper confirms it. With the stored value claude-sonnet-4-6[1m], the payload sent claude-sonnet-4-6 (the platform default). The Settings save returned 200 and stored the value, so the user gets no sign of the change. The same happens on an agent whose template runtime.model is a claude-* id outside the catalog: the boot fill refuses the value, the column stays NULL, and chat now runs the platform default instead of the template's model.

So for these agents the PR moves chat onto the platform default rather than moving the platform-started turns onto the agent's model. That is the reverse of #4442.

Warnings

  • No test covers push /chat when the stored value is refused or the column is NULL. The three push-chat tests use only a runnable stored value.
  • A Settings save that stores a value the resolver will skip returns 200 with no indication. If saving is meant to accept it, the response could carry the refused reason. GET /model already reports default_model_refused.

Suggestions

  • Add a Journey Impact: line to the body.

Recommendation: REQUEST CHANGES

- [ ] Push /chat: send the resolved model only when it comes from the agent's own stored value
      (source == "agent"); otherwise leave the model to the agent, as before this PR.
      Alternatively, refuse a value the catalog does not pass at Settings save (a named 4xx).
- [ ] Add a regression test: stored `claude-sonnet-4-6[1m]` (or NULL) -> push /chat must not send the platform default.
- [ ] Let Tier 1 (`pytest (head)`, CodeQL) finish green.

Report-only validation: nothing was pushed, patched or labelled.

#4442)

When the request names no model, push /chat now sends the resolved model
only when it is the agent's own stored value the catalog accepts. With
nothing stored, or a stored value outside the catalog (Settings accepts
any claude-* id the agent takes, e.g. claude-sonnet-4-6[1m]), it sends
none and the agent stays on its own model, as before #4442, instead of
being moved to the platform default. What Settings accepts is unchanged.

Red before the fix: test_push_chat_leaves_a_stored_model_outside_the_catalog_to_the_agent
and test_push_chat_with_no_stored_model_leaves_the_model_to_the_agent
(payload carried the platform default claude-sonnet-4-6).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@trinity-ability trinity-ability left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Validation Report: #4466 (re-validation after f0e07d834)

Lane B · head f0e07d834 · 10 files (+1033/−14) · targets dev ✅ · Fixes #4442 ✅

The previous critical finding is fixed ✅

build_chat_payload now sends the resolved model only when resolved.source == SOURCE_AGENT. If nothing is stored, or the stored value is outside the catalog, the payload carries no model and the agent stays on its own model, as it did before this PR. The architecture note (architecture/execution.md) says the same.

  • Tests, run one file per command: test_4442_wake_model.py 61 passed. test_3677_set_model_agent_error.py 15 passed.
  • Mutation: I reverted the one line to model = resolved.model. These two tests went red, and the other 59 stayed green:
    • test_push_chat_leaves_a_stored_model_outside_the_catalog_to_the_agent
    • test_push_chat_with_no_stored_model_leaves_the_model_to_the_agent
  • Live consumer: chat_execution_service.py:1218 calls build_chat_payload.

My other earlier rows are unchanged: security greps clean, no packaging or env changes, the audit row on PUT set_agent_model, and no cross-surface drift. The fix commit touches only the service, its test and the doc.

Critical: Tier 1 regression diff is red, and the failure is caused by this PR

❌ New failures introduced by HEAD (1)
- [F] test_ent872_fleet_reach::test_every_fleet_selection_consults_the_reach
  offenders: ['services/agent_model_seed.py::_live_agents (roster-read)']

I reproduced it locally on the PR head. The guard is already on dev, and the cached dev baseline passes it. It flags any function that reads the raw agent roster without consulting a principal's reach. The new boot fill's _live_agents() calls list_all_agents_fast(), so it trips the guard.

This is a background pass that runs at startup. It acts for no principal, writes only a NULL column, and returns counts. So the expected fix is an _ALLOW entry in tests/unit/test_ent872_fleet_reach.py that states that reason, rather than adding a reach check. (My previous review saw pytest (head) as still pending, so it did not catch this.)

Merge gate

  • Tier 1: red (regression diff). The rest passes: pytest (head, seed 12345), both CodeQL analyses, schema-parity, verify-non-root, journey-impact, gitleaks.
  • dev-ci: success, and no open dev-ci red issue.
  • The PR is still a draft.

Warnings

  • /review has not been posted on the PR, so this report does not cover races or SQL.

Recommendation: REQUEST CHANGES

- [ ] Add services/agent_model_seed.py::_live_agents to _ALLOW in tests/unit/test_ent872_fleet_reach.py,
      with the reason (a boot-time pass with no principal: it fills a NULL column only and returns counts),
      or route the roster read through an accepted accessor.
- [ ] Re-run Tier 1 to green (`regression diff`).

Report-only validation: nothing was pushed or patched.

Trinity Agent (trinity) and others added 2 commits November 11, 2026 04:36
… no principal (#4442)

The startup model seed reads the live roster on nobody's behalf and only
fills blank model values, so it selects no fleet for a caller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@trinity-ability trinity-ability left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Validation Report: #4466 (re-validation after fix 3, head eac204e7e)

Lane B · 11 files (+1035/−14) · targets dev ✅ · Fixes #4442 ✅ · merges cleanly onto the current dev tip 4448f54ee

The previous critical finding is fixed ✅

Last round, regression diff was red on test_ent872_fleet_reach::test_every_fleet_selection_consults_the_reach, with services/agent_model_seed.py::_live_agents (roster-read) named as the offender.

  • Fix 3 (eac204e7e) adds one _ALLOW entry: ("services/agent_model_seed.py", "_live_agents"), with the reason "startup: no principal; only fills blank model values". It is the only change in that commit (+2/−0). e04ca795f is a plain merge of origin/dev.
  • The reason checks out. _live_agents is called only from fill_blank_default_models. That is called only from schedule_boot_fill, and the only caller of that is the startup block in main.py:749-750. No route or principal can reach it. This is the same kind of entry as the existing background sweeps (sweep_fleet_git_remote_tokens, "background sweep: no principal").
  • Mutation: I deleted the new entry and re-ran the guard. It went red with the same offenders assertion. I restored the file, and the worktree was clean afterwards.

Tests, one file per command (prlimit --as=2500000000 -- pytest, head eac204e7e)

file result
tests/unit/test_ent872_fleet_reach.py 78 passed
tests/unit/test_4442_wake_model.py 61 passed
tests/unit/test_3677_set_model_agent_error.py 15 passed

CI on head eac204e7e: all finished, none pending

19 success, 7 skipped, 0 failed, 0 pending.

  • Tier 1: Analyze (python), Analyze (javascript-typescript), schema-parity, verify-non-root, gitleaks, lint (sys.modules pollution check), pytest (head, seed 12345) and regression diff are all green. Also green: journey-impact, prod-image-smoke, backend boots without enterprise submodule, cached dev baseline, verify-compose-file-sets and verify-docker-gid-detection.
  • Skipped as expected for Lane B: journey-smoke, e2e, pytest (base, inline fallback), pytest (push, absolute failures), auto-merge, clear-needs-fix and report.
  • dev: dev-ci is success ("all Tier 2 workflows green"), and there is no open dev-ci red issue.

Rows that have not changed since the last report

Security greps §4.1–4.10 are clean. There are no packaging or env changes. The audit row is on PUT set_agent_model. There is no cross-surface drift (§5.5). Execution coverage and the mutations from the earlier rounds are unchanged: build_chat_payload sends only an accepted stored model, and the boot fill reads the live model first. Product fit is ✅.

Warnings

  • ⚠️ No structural review. There is still no /review posted on the PR, so this report does not cover races or SQL safety.
  • ⚠️ The PR is still a draft. mergeStateStatus is BLOCKED only because of that. The body says it waits for the operator's merge call.

Recommendation

APPROVE. The one blocker is resolved, and every check on the head is green. Merging still needs the operator to take it out of draft.

PR #4466 — APPROVE — merge gate: ✅ (Tier 1 green on eac204e7e, dev-ci success)

trinity-ability and others added 2 commits November 11, 2026 09:18
…4442) — mechanical

build_chat_payload imports services.ask_service and
services.operator_queue_service.deps lazily; each binds `from database import db`
at import time, so first-importing them under patch("database.db", mock) left the
mock bound for the session and broke test_ent872_fleet_reach's operator-queue
reach test whenever 4442 ran first (seed-dependent red).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… — Gate 1 decision

The running agent's GET /api/model is whatever the last /api/chat turn sent,
which before #4442 could be the platform default; trusting it first would store
the cheaper model as the agent's own permanently. Order is now env, then
template.yaml, then the live model only when neither declares one. Tests pin the
order (both go red on the previous order).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vybe

vybe commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

merge-train pushed two commits to this branch (Gate 1, approved by the operator):

  1. 74dbb4cc0 — mechanical. _payload() patched database.db while build_chat_payload first-imported services.ask_service and services.operator_queue_service.deps; both bind from database import db at import, kept the mock for the session, and turned test_ent872_fleet_reach::test_the_operator_queue_reads_the_same_reach red whenever this file ran first (green on CI only by seed luck). The helper now imports them before patching.
  2. e76622c1a — the train's decision (needs your acknowledgement post-merge). The boot fill now reads env → template.yaml → live GET /api/model (live was first). The live value is whatever the last /api/chat turn sent — before this PR possibly the platform default — so trusting it first could store the cheaper model as the agent's own permanently. Trade-off: a Settings pick made pre-bug: platform-started turns (wakes, reminders, operator answers) run on a different model than the agent's configured one #4442 that lives only in agent memory is now used only when neither env nor template declares a model. Tests pin the order (both new tests go red on the previous order); execution.md updated.

test_4442_wake_model + test_3677: 76 passed with -p no:randomly.

@vybe
vybe marked this pull request as ready for review November 11, 2026 17:18

@trinity-ability trinity-ability left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merge-train: batch validated on train/20261111-1719 (#4480)

@vybe
vybe merged commit 4032752 into dev Nov 11, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants