Repository navigation
fix(systems): apply's dry run predicts template schedules and leftover workspace volumes - #4272
Conversation
…r workspace volumes Two places where apply's dry run said something different from what the real run then did: - A member apply creates from a `local:` template: the dry run marked every declared schedule `pending_template_resolution`, while the run updated a same-named template row (which keeps its own, often disabled, state) or reported the template's own rows. The snapshot now reads the template's schedules the way create does, and the plan maps them through the same function the executor uses on its post-create read, so both runs report the same changes. A predicted update error is a blocker before any write. `github:` templates are not probed and stay pending. The executor re-reads whenever the template has (or may have) schedules, even if the manifest declares none, and the real-run report lists each schedule once. - A free name whose `agent-<name>-workspace` volume is a leftover (#1667): the dry run planned `create` and the real create answered 409. The snapshot now probes it through create's own predicate (`crud.workspace_volume_exists`) and refuses with the non-retryable `member_workspace_volume_leftover`; a probe error is the retryable `member_reads_unavailable`. Create's own behaviour (409, fail-open on a probe error, the same text) is unchanged. Create mode's dry run also warns when a manifest schedule shares its name with one its `local:` template declares, since deploy keeps the template's row. A preview without a collision is unchanged. Part of Abilityai/trinity-enterprise#984
… them back - Both runs list a created member's schedules over rows in one fixed order (the post-create read is newest first, the prediction is template order). - A `github:` member's preview says its template's own schedules are known only after create, since the run now reports them. - When the executor never read a created member's schedules back (the create or the re-read failed), the real run reports them as `pending_template_resolution` again, beside the failure, never as the prediction. - Pins the stand-in row's `enabled`, the executor's own refusals of an invalid row and an ambiguous name, the planner's refusal of a repeated template name, the warning's de-duplication and the create 409 text. Part of Abilityai/trinity-enterprise#984
|
Resolve by running |
| out.failed.append(SystemApplyFailure( | ||
| name=name, short_name=short, step="schedules", | ||
| reason=f"schedule '{entry.get('name')}': the post-create read failed ({reason})")) | ||
| if not op.args["entries"]: # the template's rows went unreported |
There was a problem hiding this comment.
A failed post-create read for a re-read-only created_schedules op (no manifest schedules, github: template) now adds a failure and puts the member in member_failed, so a create that worked is reported applied: false because a read that only reports failed.
The manifest has a github: member with no schedules:. Before this PR it had no created_schedules op. Now re_read is True, so the op runs. If db.list_agent_schedules raises after a successful create, the report adds 'the template's schedules: the post-create read failed', sets applied=false and makes the run partial. Nothing was planned to be written.
There was a problem hiding this comment.
Agreed. With no manifest entry the op is a re-read only, so a failed read now adds a warnings[] line and returns before member_failed: the member stays applied: true, the run is not partial, and its schedules are listed as pending_template_resolution. With manifest entries a failed read is still one failed[] line per entry, because those writes did not happen.
Fixed in ef55cdd. test_a_failed_re_read_with_nothing_to_write_is_a_warning_not_a_failure fails without it.
| mp.changes.append(change) | ||
| # The executor re-reads whenever the template brings rows (or may), so | ||
| # a template-only row is reported by both runs even with no manifest entry. | ||
| re_read = template_schedules is None or bool(template_schedules) |
There was a problem hiding this comment.
re_read is True for every github: member, so each one now gets a created_schedules op and a schedules/pending_template_resolution line, even when the manifest declares no schedules. That changes the dry run and the real report for manifests that have nothing to do with schedules.
A github: member with no schedules: used to preview with no schedule lines. Now its dry run shows a 'template's own schedules are known once the member is created' line, and the real run adds a DB read. If the create fails, that line still sits next to 'skipped_member_not_created'.
There was a problem hiding this comment.
Part fixed, part kept on purpose.
Fixed (ef55cdd): the failed-create case. The "known once the member is created" line is now added only for a member that exists (_unread_schedule_changes). A member whose create failed keeps only its manifest entries as pending beside the failure, which is what it showed before this PR. Covered by test_a_failed_create_reports_the_schedules_pending_not_predicted and test_a_failed_create_of_an_unprobed_template_has_no_schedule_line, both red without the fix.
Kept: the re-read and the one dry-run line for a github: member with no schedules:. Create materializes the template's schedules whatever the manifest says (ent#89), so those rows exist and may be armed on the new member. Existing members already report every row the manifest does not name as schedule_undeclared (_plan_existing_member resolves against the live rows even with no entries), so the next apply on this member lists them anyway. Without the re-read, the run that created them would be the only one that says nothing. The dry run cannot read a github: template, so a single pending line is the accurate preview. The cost is one DB read per created github: member.
| f"agent's home directory. Refusing to adopt another agent's " | ||
| f"leftover data. Remove it (docker volume rm " | ||
| f"{volume}) or choose a different name — unclaimed " | ||
| f"agent volumes are also reclaimed automatically." |
There was a problem hiding this comment.
The shared refusal text says 'unclaimed agent volumes are also reclaimed automatically'. Since #3214, volumes created before the label existed are never reclaimed automatically. This PR reuses the text as the apply blocker detail, so the wrong remedy now shows in a second place.
An operator hits member_workspace_volume_leftover on an upgraded install where the leftover volume has no trinity.instance label. They wait for the automatic sweep, but the sweep only logs the volume as 'NOT reclaimed automatically', and apply stays refused.
There was a problem hiding this comment.
Right, the sweep never reclaims an unlabelled volume since #3214 (_report_unlabelled_orphan_volumes only logs it). The shared text now ends: "Remove it (docker volume rm agent--workspace) or choose a different name. Only volumes this install labelled as its own are reclaimed automatically; one created before that label existed stays until it is removed by hand." Create's 409 and apply's blocker both use it.
Fixed in ef55cdd. test_the_refusal_text_promises_only_the_reclaim_that_happens pins the new text.
| entry_name = entry.get("name") if isinstance(entry, dict) else None | ||
| if isinstance(entry_name, str) and entry_name in by_name and entry_name not in kept: | ||
| kept.append(entry_name) | ||
| if kept: |
There was a problem hiding this comment.
The collision warning fires for every manifest entry whose name the template also declares, even when the entry's fields are identical to the template row's. In that case 'skips the manifest's fields' describes nothing that changes.
The manifest restates the template's daily schedule with the same cron, message and enabled. The create-mode dry run still warns that deploy skips the manifest's fields, which is noise that teaches operators to ignore this warning.
There was a problem hiding this comment.
Agreed. The warning now fires only when the entry differs from the template's row of that name in a field deploy would have written: cron, message, enabled, timezone, description or model (_template_row_equals_entry, comparing _build_schedule_create(entry) with the row as reconcile_declared_schedules stores it). A restated entry warns nothing.
Fixed in ef55cdd. test_an_entry_equal_to_the_template_row_adds_no_warning fails without it, and test_any_field_that_differs_is_warned keeps each of the six fields warning.
| state.blocked = True | ||
| continue | ||
| try: | ||
| state.template_schedules = _template_declared_schedules( |
There was a problem hiding this comment.
_template_declared_schedules resolves template.yaml again on a throwaway config, which _preflight_template already does for the same member in the same plan. Create mode then reads it a third time in _template_schedule_collision_warnings. All of these are synchronous file reads inside the async snapshot and deploy paths.
An N-member local: manifest parses each template.yaml 2–3 times per preview and blocks the event loop each time. Returning the template data or schedules from one shared resolve (for example from _preflight_template) would remove the repeats.
There was a problem hiding this comment.
Agreed. _preflight_template now also returns the template's declared schedules from its one resolve, and _template_declared_schedules is gone.
- Apply: the snapshot calls
_preflight_templateonce per member to create and stores the verdict and the schedules onApplyMemberState. The plan reads them from there and no longer calls the preflight itself. - Create mode: the dry run keeps the per-member preflight results and passes the schedules to
_template_schedule_collision_warnings.
template.yaml is now read once per member in both dry runs. TestOneTemplateReadPerMember counts the calls to crud._resolve_local_template: 2 before, 1 now. The read itself is still synchronous, like the preflight was before this PR. It is one small file per member, so I did not move it to a thread.
One effect: an unexpected resolver error is now the preflight's template_unresolvable blocker, as it is on dev, and no longer a separate retryable read failure.
Fixed in ef55cdd.
| Rule: any create-time volume refusal must also be probed by apply's | ||
| snapshot (`system_service.read_apply_snapshot`) through this predicate, or | ||
| apply's dry run says `create` for a name the real create refuses. | ||
| TODO(#3723): wire the `agent-<volume_base>-config` refusal in here when |
There was a problem hiding this comment.
Keeping create and apply in agreement depends on a docstring rule and a TODO ('any create-time volume refusal must also be probed by apply's snapshot'). Nothing enforces it, so the next refusal added to create (#3723 config volume) can bring back the dry-run/real-run mismatch.
#3723 adds an agent-<base>-config refusal to create but nobody touches read_apply_snapshot. Apply's dry run says create and the real create returns 409. One create_volume_refusal(name) function called by both _guard_leftover_workspace_volume and the snapshot would make the two agree by construction.
There was a problem hiding this comment.
Agreed. There is now one function, crud.create_volume_refusal(name) -> Optional[CreateVolumeRefusal] (kind, volume, detail). Create's _guard_leftover_workspace_volume raises its 409 from it, and apply's snapshot turns the same result into the member_<kind> blocker, with the remedy naming the refusal's own volume. A refusal added to that function, such as #3723's config volume, reaches both without anyone touching the snapshot. The docstring rule and the TODO are removed. Each caller keeps its own policy for a probe that cannot answer: create proceeds, apply refuses as retryable.
Fixed in ef55cdd. test_a_refusal_create_adds_is_refused_by_the_dry_run_too patches only that function to refuse a config volume and asserts both the dry-run blocker and create's 409. It fails if either caller probes on its own.
| continue | ||
| error = _schedule_entry_error(action.entry) | ||
| if error: | ||
| elif action.kind == "ambiguous": # the template's normalizer keeps one row per name |
There was a problem hiding this comment.
The planner's ambiguous branch can never run: the stand-in rows come from normalize_declared_schedules, which keeps one row per name, as the inline comment itself says.
This is dead code with a blocker code and message no test path can reach without monkeypatching. It makes the planner harder to read and suggests a template can produce this blocker when it cannot.
There was a problem hiding this comment.
Agreed, it could not run. The branch and its direct-call test are removed, and a comment says why ambiguous cannot occur there. The executor still refuses ambiguous over its real post-create read, where two rows with one name are possible (test_an_ambiguous_name_is_a_failure). test_a_template_repeating_a_name_is_predicted_once checks the real path: a template that repeats a name gives a valid dry run with one prediction.
Fixed in ef55cdd.
| # trinity-enterprise#984: the plan's schedule lines are a | ||
| # prediction. The executor's own lines replace them when it read | ||
| # the schedules back; otherwise they go back to pending. | ||
| changes = [c for c in changes if not _is_schedule_field(c.field)] |
There was a problem hiding this comment.
The report reconciles schedule lines in two passes. Pass one removes every schedule field and re-adds pending_schedule_changes. Pass two removes pending_template_resolution except for schedule fields. The extra pending_schedule_changes plan field duplicates what mp.changes already held for github: members.
Anyone later changing what a created member reports has to keep mp.changes, pending_schedule_changes, schedules_reported and both filters in step. Computing the pending lines in _apply_report from the entries, instead of storing them on the plan, would remove one piece of state.
There was a problem hiding this comment.
Agreed. ApplyMemberPlan.pending_schedule_changes is gone. The pending lines are built where they are needed by one function, _pending_schedule_changes(entries, re_read). The planner calls it for a github: dry run. _apply_report calls it through _unread_schedule_changes, using the created_schedules op's entries and a new re_read flag in its args. The report now filters a member's lines in one pass, then adds the pending schedule lines (only for schedules never read back), then the executor's lines. Apart from the failed-create change in the thread on the re_read line, the report's order and content are unchanged, and the parity tests in test_ent984_dry_run_fidelity.py and test_ent695_* pass.
Fixed in ef55cdd.
|
Resolve by merging |
…y-enterprise#984 PR1 - A failed post-create schedule read with no manifest schedule is a warning, not a failure: nothing was planned to be written, so the member stays applied and its schedules are listed as pending. - A member whose create failed gets no line for its template's own schedules (there are none); its manifest entries stay pending. - The leftover-volume refusal text no longer promises an automatic reclaim the #3214 sweep does not make for unlabelled volumes. - Create mode warns about a template collision only when the manifest entry differs from the template's row in a field deploy would write. - template.yaml is read once per member per preview: _preflight_template returns the template's declared schedules, and apply's snapshot stores its verdict for the plan. - Create's guard and apply's snapshot ask one function, crud.create_volume_refusal, so a new create-time volume refusal is refused by the dry run too. - The planner's unreachable `ambiguous` branch is removed. - _apply_report reconciles a member's lines in one pass and builds the pending schedule lines from the op's entries; the plan no longer keeps a copy of them. Part of Abilityai/trinity-enterprise#984
Part of abilityai/trinity-enterprise#984
What
Apply mode's dry run must say what the real run will do. Two cases broke that:
local:template's schedules, on a member apply creates. The dry run marked every declared schedulepending_template_resolution. The real run then updated a same-named template row in place (the row keeps its ownenabled, which defaults to off) or reported the template's own rows. Neither was visible beforehand.agent-<name>-workspacevolume (bug: agent creation silently reuses a pre-existing Docker volume of the same name (get-then-create, no adopt/refuse decision) #1667). The dry run plannedcreate. The real create then answered 409, which landed infailed[]as apartialrun.How it behaves now
Template schedules
local:template'sschedules:the way create does, throughcrud._resolve_local_templateandnormalize_declared_schedules, in the same_preflight_templatecall that checks the template, sotemplate.yamlis read once per member. It only reads, and makes no network call._created_schedule_changes. The executor runs the same function over its post-create read, so both runs report the sameschedule_create,schedule_update,enabled_differsandschedule_undeclaredlines, in the same order.github:template is not probed. Its schedules staypending_template_resolution, plus oneschedulesline saying the template's own schedules are known only after create.pending_template_resolution, never as the prediction. The line for the template's own schedules appears only when the member was created.Leftover workspace volume
crud.create_volume_refusal, the one function create's guard raises its 409 from, so a refusal added there reaches both. It is not asked for a taken name, adeploy: falsemember, or while Docker is unreadable.member_workspace_volume_leftover, with create's 409 text: the dry run isinvalid, and the real run isrefused(409) with no write and no lock key.member_reads_unavailable(503), never "free".Create mode (
/deploy)warnings[]line perlocal:member whose manifest schedule shares a name with one its template declares and differs from it, because deploy keeps the template's row and itsenabledstate.schedules_previewis unchanged. A preview with no collision is byte-identical (the create-mode golden passes unmodified).Out of scope
resourcesstaypending_template_resolution.github:templates are not probed in either dry run.schedule_createwhere the preview saidschedule_update. It always reports what it actually did.crud.create_volume_refusal, apply's dry run refuses it too.Tests
tests/unit/test_ent984_dry_run_fidelity.py(49 cases). Dry-run/real-run parity, field for field, for collisions, creates, template-only rows (one and several, armed and not),github:, a failed create and a failed re-read. The leftover-volume blocker, the probe error, and when the probe runs, including over HTTP: 409 and 503. Create's guard keeps its policy and its exact 409 text. The create-mode warning.tests/unit/_ent695_world.py: the fake Docker gains workspace volumes, and the world patches the new probe seam, so apply tests never ask a real daemon.test_ent695_apply_engine.pychange on purpose. They pinnedpending_template_resolutionforlocal:sched'sdaily, which is now predicted asschedule_update+enabled_differs. Resources stay pending.-p no:randomlyafter merging the currentdev:test_ent984_*,test_ent695_*(create-mode golden included),test_ent126_*,test_ent89_template_schedules.py,test_ent1030_*,test_ent1174_*,test_1484_create_agent_characterization.py,test_1664_renamed_agent_volume_safety.py,test_2373_system_endpoints.py,test_4284_no_exception_text_in_5xx.py. 1069 passed, 1 skipped.test_ent984_*,test_ent695_apply_engine.py, the create-mode golden andtest_ent126_dry_run_preview.pyalso pass under a random seed.Validation: the targeted suites listed above pass locally (1069 tests with
-p no:randomly; the apply and preview suites also green under a random seed). The create-mode golden (test_ent695_create_mode_golden.py) is unmodified and passes. CI is the full run.Mutation: 21 guards on the first version, plus 6 on the review fixes, removed one at a time, each red on a named test. Examples: the snapshot volume probe, a probe error read as "free", create's fail-open, the planner's prediction, the fixed row order, the
github:pending line, the real-run fallback to pending, the stand-in row'senabled, the executor's own refusals and the create-mode warning; and, from review, the shared volume refusal on each side, the warning (not failure) for a re-read only, the created-only template line, and the equal-row and differing-field checks of the create-mode warning.Unverified:
github:clone was exercised. The volume probe and create are faked at their seams, and the realvolume_get/docker.errors.NotFoundmapping is covered only through a patchedvolume_get.Docs
Requirements §16.5.4 and the §16.5 dry-run preview entry,
feature-flows/system-manifest.md,user-docs/collaboration/system-manifest.md, andarchitecture/backend.md.