Repository navigation
fix(agents): platform-started turns run on the agent's own model (#4442) - #4466
Conversation
… boot (#4442) Adds services/agent_model_seed: declared_model (runtime.model, else a top-level model:), the fill-blank seed, and a one-shot boot pass that writes a NULL default_model from the container's AGENT_RUNTIME_MODEL, else from the running agent's template.yaml. Writes go through set_default_model_if(name, None, ...), so a value another writer set is never overwritten and concurrent workers make one write; only a value unrunnable_reason passes is written. docker_service gains read_runtime_model, which returns that one env key and logs failures by type only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…4442) _register_agent writes default_model from the request's runtime_model or the model the template declares (runtime.model, else a top-level model:). Both template paths feed it: the github: path never set runtime_model, so its agents' AGENT_RUNTIME_MODEL stayed empty and their wakes ran on the platform default. Fill-blank and catalog-gated; never fails a creation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) PUT /api/agents/{name}/model kept the value in the container's memory only, so the turns the platform starts never saw it and a restart lost it. On a Claude Code agent the accepted value is now stored as default_model and audited (update_agent_model). The agent stays the judge of the value (#3677); a failed store after the agent accepted answers 500 model_not_saved instead of a silent split. Route tier unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sync /chat that names no model left the choice to the container's in-memory value, which falls back to its env after a restart, and left model_used empty. build_chat_payload now resolves it through resolve_default_model (fail-open: on an error the agent decides, as before) and stamps it on the row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…4442) 53 unit tests: the declared-model parse; the create seed for a template declaring a top-level model:; the boot fill (env, then the live template.yaml; one write across racing workers; the env never logged); the PUT store, refusal, failed-save and non-Claude cases; push /chat resolution and stamping; per-kind execute_task resolution; pinned models win; an AST guard that the wake producers pass no model. Mutations shown red: dropping the PUT store, the boot fill's template fallback, the create seed's declared model, the push /chat resolve, or making execute_task resolve unconditionally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed Settings save puts the old model back (#4442) Review fixes I1 and I2 (chain-hub-p1-4442-fix). I1: a Settings pick made before #4442 lived only in agent memory, and push /chat now always sends the resolved model, so a fill from the env or template would have overridden that pick. fill_blank_default_models now asks a running agent's GET /api/model first, then the container env, then template.yaml. The allowlist (unrunnable_reason) and fill-only-NULL (set_default_model_if) guards are unchanged. I2: model_not_saved said the agent "now runs" the new model, but backend turns send the stored value. set_agent_model reads the agent's model before the PUT, and if the store fails it puts that model back, so the agent and the column agree. The message names the model the agent keeps running; `restored` says whether the put-back held, and the message changes when it did not. Mutation: with both fixes reverted, five new tests went red: test_boot_fill_takes_the_running_agents_live_model_over_its_env, ..._falls_back_to_env_then_template_..., ..._keeps_the_allowlist_..., test_a_failed_save_puts_the_agents_previous_model_back, test_a_failed_save_that_cannot_be_put_back_says_so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trinity-ability
left a comment
There was a problem hiding this comment.
PR Validation Report: #4466
Lane B (no security-path or schema trigger) · head eca31da90 · 10 files (+1015/−14) · targets dev ✅ · Fixes #4442 ✅ (P1 bug, status-ready)
Summary
| Category | Status | Notes |
|---|---|---|
| Commits / base / size | ✅ | conventional prefixes; dev; 10 files |
| Status auto-promotion | ✅ | Fixes #4442 is in the body |
| Journey Impact | ✅ | check_journey_impact.py exit 0. The body has no declaration, which is allowed, but the template asks for one |
| Architecture doc | ✅ | architecture/execution.md names every new writer and drops "There is no HTTP setter" |
| Security §4.1–4.10 | ✅ | the only hits are identifiers in context lines, a test fixture SECRET, and the agent-container path /home/developer/template.yaml |
| Build / config packaging | ➖ | no new top-level backend directory and no new env var |
| Audit rows §4.11 | ✅ | PUT set_agent_model: audit via chat.py::set_agent_model (update_agent_model) |
| Cross-surface drift §5.5 | ✅ | no agent-server twin touched. The router header says the model routes are UI-only; chat.ts sends nothing to /model |
| Tests (run file by file here) | ✅ | test_4442_wake_model.py 59 passed · test_3677_set_model_agent_error.py 15 passed |
| Execution coverage / mutation | ✅ | reverting chat_execution_service.py turns 2 tests red; removing the Settings store and the create seed turns 8 red; main.py boot wiring is pinned by AST |
| Structural review | no /review is posted on the PR (the body describes one), so this report does not cover races or SQL |
|
| Merge gate (Tier 1) | pytest (head, seed 12345) and both CodeQL analyses are still pending; dev-ci is success. The PR is a draft |
|
| Product fit | ✅ | approved bug; all 3 acceptance criteria are met for a stored model the resolver accepts |
Critical: push /chat now overrides a Settings pick that the agent accepts but the resolver refuses
build_chat_payload now always sends resolve_default_model(...).model when the request names no model. When the stored value fails unrunnable_reason, that is the platform default. The agent server prefers request.model over its own current_model (agent_server/routers/chat.py:106), so the platform default replaces the agent's pick. Before this PR, the payload carried no model and chat ran on the pick.
The agent's PUT /api/model accepts any claude-* string, and ModelSelector.vue allows free text: its comment names claude-sonnet-4-6[1m] as the example. The catalog allowlist is narrower:
claude-sonnet-4-6[1m] not_in_catalog
claude-opus-5-5[1m] not_in_catalog
claude-sonnet-4-20250514 not_in_catalog
A scratch test using this PR's own _payload helper confirms it. With the stored value claude-sonnet-4-6[1m], the payload sent claude-sonnet-4-6 (the platform default). The Settings save returned 200 and stored the value, so the user gets no sign of the change. The same happens on an agent whose template runtime.model is a claude-* id outside the catalog: the boot fill refuses the value, the column stays NULL, and chat now runs the platform default instead of the template's model.
So for these agents the PR moves chat onto the platform default rather than moving the platform-started turns onto the agent's model. That is the reverse of #4442.
Warnings
- No test covers push
/chatwhen the stored value is refused or the column is NULL. The three push-chat tests use only a runnable stored value. - A Settings save that stores a value the resolver will skip returns 200 with no indication. If saving is meant to accept it, the response could carry the refused reason.
GET /modelalready reportsdefault_model_refused.
Suggestions
- Add a
Journey Impact:line to the body.
Recommendation: REQUEST CHANGES
- [ ] Push /chat: send the resolved model only when it comes from the agent's own stored value
(source == "agent"); otherwise leave the model to the agent, as before this PR.
Alternatively, refuse a value the catalog does not pass at Settings save (a named 4xx).
- [ ] Add a regression test: stored `claude-sonnet-4-6[1m]` (or NULL) -> push /chat must not send the platform default.
- [ ] Let Tier 1 (`pytest (head)`, CodeQL) finish green.
Report-only validation: nothing was pushed, patched or labelled.
#4442) When the request names no model, push /chat now sends the resolved model only when it is the agent's own stored value the catalog accepts. With nothing stored, or a stored value outside the catalog (Settings accepts any claude-* id the agent takes, e.g. claude-sonnet-4-6[1m]), it sends none and the agent stays on its own model, as before #4442, instead of being moved to the platform default. What Settings accepts is unchanged. Red before the fix: test_push_chat_leaves_a_stored_model_outside_the_catalog_to_the_agent and test_push_chat_with_no_stored_model_leaves_the_model_to_the_agent (payload carried the platform default claude-sonnet-4-6). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trinity-ability
left a comment
There was a problem hiding this comment.
PR Validation Report: #4466 (re-validation after f0e07d834)
Lane B · head f0e07d834 · 10 files (+1033/−14) · targets dev ✅ · Fixes #4442 ✅
The previous critical finding is fixed ✅
build_chat_payload now sends the resolved model only when resolved.source == SOURCE_AGENT. If nothing is stored, or the stored value is outside the catalog, the payload carries no model and the agent stays on its own model, as it did before this PR. The architecture note (architecture/execution.md) says the same.
- Tests, run one file per command:
test_4442_wake_model.py61 passed.test_3677_set_model_agent_error.py15 passed. - Mutation: I reverted the one line to
model = resolved.model. These two tests went red, and the other 59 stayed green:test_push_chat_leaves_a_stored_model_outside_the_catalog_to_the_agenttest_push_chat_with_no_stored_model_leaves_the_model_to_the_agent
- Live consumer:
chat_execution_service.py:1218callsbuild_chat_payload.
My other earlier rows are unchanged: security greps clean, no packaging or env changes, the audit row on PUT set_agent_model, and no cross-surface drift. The fix commit touches only the service, its test and the doc.
Critical: Tier 1 regression diff is red, and the failure is caused by this PR
❌ New failures introduced by HEAD (1)
- [F] test_ent872_fleet_reach::test_every_fleet_selection_consults_the_reach
offenders: ['services/agent_model_seed.py::_live_agents (roster-read)']
I reproduced it locally on the PR head. The guard is already on dev, and the cached dev baseline passes it. It flags any function that reads the raw agent roster without consulting a principal's reach. The new boot fill's _live_agents() calls list_all_agents_fast(), so it trips the guard.
This is a background pass that runs at startup. It acts for no principal, writes only a NULL column, and returns counts. So the expected fix is an _ALLOW entry in tests/unit/test_ent872_fleet_reach.py that states that reason, rather than adding a reach check. (My previous review saw pytest (head) as still pending, so it did not catch this.)
Merge gate
- Tier 1: red (
regression diff). The rest passes:pytest (head, seed 12345), both CodeQL analyses,schema-parity,verify-non-root,journey-impact, gitleaks. dev-ci: success, and no opendev-ci redissue.- The PR is still a draft.
Warnings
/reviewhas not been posted on the PR, so this report does not cover races or SQL.
Recommendation: REQUEST CHANGES
- [ ] Add services/agent_model_seed.py::_live_agents to _ALLOW in tests/unit/test_ent872_fleet_reach.py,
with the reason (a boot-time pass with no principal: it fills a NULL column only and returns counts),
or route the roster read through an accepted accessor.
- [ ] Re-run Tier 1 to green (`regression diff`).
Report-only validation: nothing was pushed or patched.
… no principal (#4442) The startup model seed reads the live roster on nobody's behalf and only fills blank model values, so it selects no fleet for a caller. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trinity-ability
left a comment
There was a problem hiding this comment.
PR Validation Report: #4466 (re-validation after fix 3, head eac204e7e)
Lane B · 11 files (+1035/−14) · targets dev ✅ · Fixes #4442 ✅ · merges cleanly onto the current dev tip 4448f54ee
The previous critical finding is fixed ✅
Last round, regression diff was red on test_ent872_fleet_reach::test_every_fleet_selection_consults_the_reach, with services/agent_model_seed.py::_live_agents (roster-read) named as the offender.
- Fix 3 (
eac204e7e) adds one_ALLOWentry:("services/agent_model_seed.py", "_live_agents"), with the reason "startup: no principal; only fills blank model values". It is the only change in that commit (+2/−0).e04ca795fis a plain merge oforigin/dev. - The reason checks out.
_live_agentsis called only fromfill_blank_default_models. That is called only fromschedule_boot_fill, and the only caller of that is the startup block inmain.py:749-750. No route or principal can reach it. This is the same kind of entry as the existing background sweeps (sweep_fleet_git_remote_tokens, "background sweep: no principal"). - Mutation: I deleted the new entry and re-ran the guard. It went red with the same
offendersassertion. I restored the file, and the worktree was clean afterwards.
Tests, one file per command (prlimit --as=2500000000 -- pytest, head eac204e7e)
| file | result |
|---|---|
tests/unit/test_ent872_fleet_reach.py |
78 passed |
tests/unit/test_4442_wake_model.py |
61 passed |
tests/unit/test_3677_set_model_agent_error.py |
15 passed |
CI on head eac204e7e: all finished, none pending
19 success, 7 skipped, 0 failed, 0 pending.
- Tier 1:
Analyze (python),Analyze (javascript-typescript),schema-parity,verify-non-root,gitleaks,lint (sys.modules pollution check),pytest (head, seed 12345)andregression diffare all green. Also green:journey-impact,prod-image-smoke,backend boots without enterprise submodule,cached dev baseline,verify-compose-file-setsandverify-docker-gid-detection. - Skipped as expected for Lane B:
journey-smoke,e2e,pytest (base, inline fallback),pytest (push, absolute failures),auto-merge,clear-needs-fixandreport. dev:dev-ciis success ("all Tier 2 workflows green"), and there is no opendev-ci redissue.
Rows that have not changed since the last report
Security greps §4.1–4.10 are clean. There are no packaging or env changes. The audit row is on PUT set_agent_model. There is no cross-surface drift (§5.5). Execution coverage and the mutations from the earlier rounds are unchanged: build_chat_payload sends only an accepted stored model, and the boot fill reads the live model first. Product fit is ✅.
Warnings
⚠️ No structural review. There is still no/reviewposted on the PR, so this report does not cover races or SQL safety.⚠️ The PR is still a draft.mergeStateStatusisBLOCKEDonly because of that. The body says it waits for the operator's merge call.
Recommendation
APPROVE. The one blocker is resolved, and every check on the head is green. Merging still needs the operator to take it out of draft.
PR #4466 — APPROVE — merge gate: ✅ (Tier 1 green on eac204e7e, dev-ci success)
…4442) — mechanical build_chat_payload imports services.ask_service and services.operator_queue_service.deps lazily; each binds `from database import db` at import time, so first-importing them under patch("database.db", mock) left the mock bound for the session and broke test_ent872_fleet_reach's operator-queue reach test whenever 4442 ran first (seed-dependent red). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… — Gate 1 decision The running agent's GET /api/model is whatever the last /api/chat turn sent, which before #4442 could be the platform default; trusting it first would store the cheaper model as the agent's own permanently. Order is now env, then template.yaml, then the live model only when neither declares one. Tests pin the order (both go red on the previous order). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
merge-train pushed two commits to this branch (Gate 1, approved by the operator):
|
trinity-ability
left a comment
There was a problem hiding this comment.
merge-train: batch validated on train/20261111-1719 (#4480)
# Conflicts: # tests/registry.json
Summary
Turns the platform starts on an agent's behalf (caller wakes, operator answers, push
/chatturns) resolve their model through the agent's stored default model. The open-source build never wrote that value, so every such turn fell through to the platform default instead of the model the agent was configured with.This change fills it:
model:at the top level and the runtime block, includinggithub:templates).model_not_saved)./chatpath looks the model up the same way.Fixes #4442
Tests
tests/unit/test_4442_wake_model.py(59 passed) plus the 11 suites the plan pins: default-model resolution, set-model errors, caller wake, operator resume, reminders, human-only routes, auth wiring. All green, run file by file oneca31da90.-k "model or ent752"sweep: 974 passed, 3 failed. All three fail becausenumpyis missing from the local test venv. They are memory-embedding tests that this diff does not touch.Review
Reading review + security pass: MERGEABLE AFTER FIXES, both important findings fixed in
eca31da90(boot fill prefers the live model; restore on a failed save). No security findings.Draft until the operator's merge call.
🤖 Generated with Claude Code