Skip to content

fix(systems): apply's dry run predicts template schedules and leftover workspace volumes - #4272

Merged
vybe merged 7 commits into
devfrom
AndriiPasternak31/ent984
Nov 11, 2026
Merged

vybe merged 7 commits into
devfrom
AndriiPasternak31/ent984

Conversation

@AndriiPasternak31

@AndriiPasternak31 AndriiPasternak31 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Part of abilityai/trinity-enterprise#984

What

Apply mode's dry run must say what the real run will do. Two cases broke that:

  1. A local: template's schedules, on a member apply creates. The dry run marked every declared schedule pending_template_resolution. The real run then updated a same-named template row in place (the row keeps its own enabled, which defaults to off) or reported the template's own rows. Neither was visible beforehand.
  2. A leftover agent-<name>-workspace volume (bug: agent creation silently reuses a pre-existing Docker volume of the same name (get-then-create, no adopt/refuse decision) #1667). The dry run planned create. The real create then answered 409, which landed in failed[] as a partial run.

How it behaves now

Template schedules

  • The snapshot reads a local: template's schedules: the way create does, through crud._resolve_local_template and normalize_declared_schedules, in the same _preflight_template call that checks the template, so template.yaml is read once per member. It only reads, and makes no network call.
  • The plan builds stand-in rows from them and maps actions to changes through _created_schedule_changes. The executor runs the same function over its post-create read, so both runs report the same schedule_create, schedule_update, enabled_differs and schedule_undeclared lines, in the same order.
  • An update that would be invalid over the merged cron/timezone is a blocker before any write.
  • A github: template is not probed. Its schedules stay pending_template_resolution, plus one schedules line saying the template's own schedules are known only after create.
  • The executor re-reads whenever the template has schedules, or may have them, even when the manifest declares none. Both runs then report a template-only row. With no manifest entry nothing is written, so a failed re-read is a warning and the member stays applied.
  • If the executor never read the schedules back (the create failed, or the re-read failed), the real-run report shows them as pending_template_resolution, never as the prediction. The line for the template's own schedules appears only when the member was created.

Leftover workspace volume

Create mode (/deploy)

  • The dry run adds one warnings[] line per local: member whose manifest schedule shares a name with one its template declares and differs from it, because deploy keeps the template's row and its enabled state.
  • schedules_preview is unchanged. A preview with no collision is byte-identical (the create-mode golden passes unmodified).

Out of scope

  • A created member's resources stay pending_template_resolution.
  • github: templates are not probed in either dry run.
  • Stated residual: template materialization is non-fatal, so a template row the dry run expected can be missing after create. The real run then reports schedule_create where the preview said schedule_update. It always reports what it actually did.
  • If fix: preserve persistent agent configuration across Docker namespaces #3723's config-volume refusal lands in crud.create_volume_refusal, apply's dry run refuses it too.

Tests

  • New: tests/unit/test_ent984_dry_run_fidelity.py (49 cases). Dry-run/real-run parity, field for field, for collisions, creates, template-only rows (one and several, armed and not), github:, a failed create and a failed re-read. The leftover-volume blocker, the probe error, and when the probe runs, including over HTTP: 409 and 503. Create's guard keeps its policy and its exact 409 text. The create-mode warning.
  • tests/unit/_ent695_world.py: the fake Docker gains workspace volumes, and the world patches the new probe seam, so apply tests never ask a real daemon.
  • Two existing assertions in test_ent695_apply_engine.py change on purpose. They pinned pending_template_resolution for local:sched's daily, which is now predicted as schedule_update + enabled_differs. Resources stay pending.
  • Every new guard was removed one at a time, and each turned a named test red: 21 mutations on the first version, 6 more on the review fixes (the 21 were not re-run after them).
  • Run locally with -p no:randomly after merging the current dev: test_ent984_*, test_ent695_* (create-mode golden included), test_ent126_*, test_ent89_template_schedules.py, test_ent1030_*, test_ent1174_*, test_1484_create_agent_characterization.py, test_1664_renamed_agent_volume_safety.py, test_2373_system_endpoints.py, test_4284_no_exception_text_in_5xx.py. 1069 passed, 1 skipped. test_ent984_*, test_ent695_apply_engine.py, the create-mode golden and test_ent126_dry_run_preview.py also pass under a random seed.

Validation: the targeted suites listed above pass locally (1069 tests with -p no:randomly; the apply and preview suites also green under a random seed). The create-mode golden (test_ent695_create_mode_golden.py) is unmodified and passes. CI is the full run.

Mutation: 21 guards on the first version, plus 6 on the review fixes, removed one at a time, each red on a named test. Examples: the snapshot volume probe, a probe error read as "free", create's fail-open, the planner's prediction, the fixed row order, the github: pending line, the real-run fallback to pending, the stand-in row's enabled, the executor's own refusals and the create-mode warning; and, from review, the shared volume refusal on each side, the warning (not failure) for a re-read only, the created-only template line, and the equal-row and differing-field checks of the create-mode warning.

Unverified:

  • No real Docker daemon or real github: clone was exercised. The volume probe and create are faked at their seams, and the real volume_get/docker.errors.NotFound mapping is covered only through a patched volume_get.
  • A predicted update error is not reachable with today's parser (the template normalizer validates with the same one), so that test reaches it by patching the check.

Docs

Requirements §16.5.4 and the §16.5 dry-run preview entry, feature-flows/system-manifest.md, user-docs/collaboration/system-manifest.md, and architecture/backend.md.

…r workspace volumes

Two places where apply's dry run said something different from what the
real run then did:

- A member apply creates from a `local:` template: the dry run marked every
  declared schedule `pending_template_resolution`, while the run updated a
  same-named template row (which keeps its own, often disabled, state) or
  reported the template's own rows. The snapshot now reads the template's
  schedules the way create does, and the plan maps them through the same
  function the executor uses on its post-create read, so both runs report
  the same changes. A predicted update error is a blocker before any write.
  `github:` templates are not probed and stay pending. The executor re-reads
  whenever the template has (or may have) schedules, even if the manifest
  declares none, and the real-run report lists each schedule once.
- A free name whose `agent-<name>-workspace` volume is a leftover (#1667):
  the dry run planned `create` and the real create answered 409. The
  snapshot now probes it through create's own predicate
  (`crud.workspace_volume_exists`) and refuses with the non-retryable
  `member_workspace_volume_leftover`; a probe error is the retryable
  `member_reads_unavailable`. Create's own behaviour (409, fail-open on a
  probe error, the same text) is unchanged.

Create mode's dry run also warns when a manifest schedule shares its name
with one its `local:` template declares, since deploy keeps the template's
row. A preview without a collision is unchanged.

Part of Abilityai/trinity-enterprise#984
… them back

- Both runs list a created member's schedules over rows in one fixed order
  (the post-create read is newest first, the prediction is template order).
- A `github:` member's preview says its template's own schedules are known
  only after create, since the run now reports them.
- When the executor never read a created member's schedules back (the create
  or the re-read failed), the real run reports them as
  `pending_template_resolution` again, beside the failure, never as the
  prediction.
- Pins the stand-in row's `enabled`, the executor's own refusals of an
  invalid row and an ambiguous name, the planner's refusal of a repeated
  template name, the warning's de-duplication and the create 409 text.

Part of Abilityai/trinity-enterprise#984
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

⚠️ Nightly unit-suite check skipped — merge conflict against dev.

Resolve by running git merge dev locally and pushing the result. The next nightly run will re-test once the conflict is gone.

@AndriiPasternak31
AndriiPasternak31 marked this pull request as ready for review November 7, 2026 19:16
Comment thread src/backend/services/system_service.py Outdated
out.failed.append(SystemApplyFailure(
name=name, short_name=short, step="schedules",
reason=f"schedule '{entry.get('name')}': the post-create read failed ({reason})"))
if not op.args["entries"]: # the template's rows went unreported

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A failed post-create read for a re-read-only created_schedules op (no manifest schedules, github: template) now adds a failure and puts the member in member_failed, so a create that worked is reported applied: false because a read that only reports failed.

The manifest has a github: member with no schedules:. Before this PR it had no created_schedules op. Now re_read is True, so the op runs. If db.list_agent_schedules raises after a successful create, the report adds 'the template's schedules: the post-create read failed', sets applied=false and makes the run partial. Nothing was planned to be written.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. With no manifest entry the op is a re-read only, so a failed read now adds a warnings[] line and returns before member_failed: the member stays applied: true, the run is not partial, and its schedules are listed as pending_template_resolution. With manifest entries a failed read is still one failed[] line per entry, because those writes did not happen.

Fixed in ef55cdd. test_a_failed_re_read_with_nothing_to_write_is_a_warning_not_a_failure fails without it.

mp.changes.append(change)
# The executor re-reads whenever the template brings rows (or may), so
# a template-only row is reported by both runs even with no manifest entry.
re_read = template_schedules is None or bool(template_schedules)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re_read is True for every github: member, so each one now gets a created_schedules op and a schedules/pending_template_resolution line, even when the manifest declares no schedules. That changes the dry run and the real report for manifests that have nothing to do with schedules.

A github: member with no schedules: used to preview with no schedule lines. Now its dry run shows a 'template's own schedules are known once the member is created' line, and the real run adds a DB read. If the create fails, that line still sits next to 'skipped_member_not_created'.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Part fixed, part kept on purpose.

Fixed (ef55cdd): the failed-create case. The "known once the member is created" line is now added only for a member that exists (_unread_schedule_changes). A member whose create failed keeps only its manifest entries as pending beside the failure, which is what it showed before this PR. Covered by test_a_failed_create_reports_the_schedules_pending_not_predicted and test_a_failed_create_of_an_unprobed_template_has_no_schedule_line, both red without the fix.

Kept: the re-read and the one dry-run line for a github: member with no schedules:. Create materializes the template's schedules whatever the manifest says (ent#89), so those rows exist and may be armed on the new member. Existing members already report every row the manifest does not name as schedule_undeclared (_plan_existing_member resolves against the live rows even with no entries), so the next apply on this member lists them anyway. Without the re-read, the run that created them would be the only one that says nothing. The dry run cannot read a github: template, so a single pending line is the accurate preview. The cost is one DB read per created github: member.

f"agent's home directory. Refusing to adopt another agent's "
f"leftover data. Remove it (docker volume rm "
f"{volume}) or choose a different name — unclaimed "
f"agent volumes are also reclaimed automatically."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shared refusal text says 'unclaimed agent volumes are also reclaimed automatically'. Since #3214, volumes created before the label existed are never reclaimed automatically. This PR reuses the text as the apply blocker detail, so the wrong remedy now shows in a second place.

An operator hits member_workspace_volume_leftover on an upgraded install where the leftover volume has no trinity.instance label. They wait for the automatic sweep, but the sweep only logs the volume as 'NOT reclaimed automatically', and apply stays refused.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, the sweep never reclaims an unlabelled volume since #3214 (_report_unlabelled_orphan_volumes only logs it). The shared text now ends: "Remove it (docker volume rm agent--workspace) or choose a different name. Only volumes this install labelled as its own are reclaimed automatically; one created before that label existed stays until it is removed by hand." Create's 409 and apply's blocker both use it.

Fixed in ef55cdd. test_the_refusal_text_promises_only_the_reclaim_that_happens pins the new text.

entry_name = entry.get("name") if isinstance(entry, dict) else None
if isinstance(entry_name, str) and entry_name in by_name and entry_name not in kept:
kept.append(entry_name)
if kept:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The collision warning fires for every manifest entry whose name the template also declares, even when the entry's fields are identical to the template row's. In that case 'skips the manifest's fields' describes nothing that changes.

The manifest restates the template's daily schedule with the same cron, message and enabled. The create-mode dry run still warns that deploy skips the manifest's fields, which is noise that teaches operators to ignore this warning.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The warning now fires only when the entry differs from the template's row of that name in a field deploy would have written: cron, message, enabled, timezone, description or model (_template_row_equals_entry, comparing _build_schedule_create(entry) with the row as reconcile_declared_schedules stores it). A restated entry warns nothing.

Fixed in ef55cdd. test_an_entry_equal_to_the_template_row_adds_no_warning fails without it, and test_any_field_that_differs_is_warned keeps each of the six fields warning.

Comment thread src/backend/services/system_service.py Outdated
state.blocked = True
continue
try:
state.template_schedules = _template_declared_schedules(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_template_declared_schedules resolves template.yaml again on a throwaway config, which _preflight_template already does for the same member in the same plan. Create mode then reads it a third time in _template_schedule_collision_warnings. All of these are synchronous file reads inside the async snapshot and deploy paths.

An N-member local: manifest parses each template.yaml 2–3 times per preview and blocks the event loop each time. Returning the template data or schedules from one shared resolve (for example from _preflight_template) would remove the repeats.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. _preflight_template now also returns the template's declared schedules from its one resolve, and _template_declared_schedules is gone.

  • Apply: the snapshot calls _preflight_template once per member to create and stores the verdict and the schedules on ApplyMemberState. The plan reads them from there and no longer calls the preflight itself.
  • Create mode: the dry run keeps the per-member preflight results and passes the schedules to _template_schedule_collision_warnings.

template.yaml is now read once per member in both dry runs. TestOneTemplateReadPerMember counts the calls to crud._resolve_local_template: 2 before, 1 now. The read itself is still synchronous, like the preflight was before this PR. It is one small file per member, so I did not move it to a thread.

One effect: an unexpected resolver error is now the preflight's template_unresolvable blocker, as it is on dev, and no longer a separate retryable read failure.

Fixed in ef55cdd.

Rule: any create-time volume refusal must also be probed by apply's
snapshot (`system_service.read_apply_snapshot`) through this predicate, or
apply's dry run says `create` for a name the real create refuses.
TODO(#3723): wire the `agent-<volume_base>-config` refusal in here when

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping create and apply in agreement depends on a docstring rule and a TODO ('any create-time volume refusal must also be probed by apply's snapshot'). Nothing enforces it, so the next refusal added to create (#3723 config volume) can bring back the dry-run/real-run mismatch.

#3723 adds an agent-<base>-config refusal to create but nobody touches read_apply_snapshot. Apply's dry run says create and the real create returns 409. One create_volume_refusal(name) function called by both _guard_leftover_workspace_volume and the snapshot would make the two agree by construction.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. There is now one function, crud.create_volume_refusal(name) -> Optional[CreateVolumeRefusal] (kind, volume, detail). Create's _guard_leftover_workspace_volume raises its 409 from it, and apply's snapshot turns the same result into the member_<kind> blocker, with the remedy naming the refusal's own volume. A refusal added to that function, such as #3723's config volume, reaches both without anyone touching the snapshot. The docstring rule and the TODO are removed. Each caller keeps its own policy for a probe that cannot answer: create proceeds, apply refuses as retryable.

Fixed in ef55cdd. test_a_refusal_create_adds_is_refused_by_the_dry_run_too patches only that function to refuse a config volume and asserts both the dry-run blocker and create's 409. It fails if either caller probes on its own.

Comment thread src/backend/services/system_service.py Outdated
continue
error = _schedule_entry_error(action.entry)
if error:
elif action.kind == "ambiguous": # the template's normalizer keeps one row per name

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The planner's ambiguous branch can never run: the stand-in rows come from normalize_declared_schedules, which keeps one row per name, as the inline comment itself says.

This is dead code with a blocker code and message no test path can reach without monkeypatching. It makes the planner harder to read and suggests a template can produce this blocker when it cannot.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, it could not run. The branch and its direct-call test are removed, and a comment says why ambiguous cannot occur there. The executor still refuses ambiguous over its real post-create read, where two rows with one name are possible (test_an_ambiguous_name_is_a_failure). test_a_template_repeating_a_name_is_predicted_once checks the real path: a template that repeats a name gives a valid dry run with one prediction.

Fixed in ef55cdd.

Comment thread src/backend/services/system_service.py Outdated
# trinity-enterprise#984: the plan's schedule lines are a
# prediction. The executor's own lines replace them when it read
# the schedules back; otherwise they go back to pending.
changes = [c for c in changes if not _is_schedule_field(c.field)]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The report reconciles schedule lines in two passes. Pass one removes every schedule field and re-adds pending_schedule_changes. Pass two removes pending_template_resolution except for schedule fields. The extra pending_schedule_changes plan field duplicates what mp.changes already held for github: members.

Anyone later changing what a created member reports has to keep mp.changes, pending_schedule_changes, schedules_reported and both filters in step. Computing the pending lines in _apply_report from the entries, instead of storing them on the plan, would remove one piece of state.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. ApplyMemberPlan.pending_schedule_changes is gone. The pending lines are built where they are needed by one function, _pending_schedule_changes(entries, re_read). The planner calls it for a github: dry run. _apply_report calls it through _unread_schedule_changes, using the created_schedules op's entries and a new re_read flag in its args. The report now filters a member's lines in one pass, then adds the pending schedule lines (only for schedules never read back), then the executor's lines. Apart from the failed-create change in the thread on the re_read line, the report's order and content are unchanged, and the parity tests in test_ent984_dry_run_fidelity.py and test_ent695_* pass.

Fixed in ef55cdd.

@github-actions

Copy link
Copy Markdown

⚠️ Live-instance suite skipped — merge conflict against dev.

Resolve by merging dev locally and pushing the result; the next nightly re-tests.

…y-enterprise#984 PR1

- A failed post-create schedule read with no manifest schedule is a
  warning, not a failure: nothing was planned to be written, so the
  member stays applied and its schedules are listed as pending.
- A member whose create failed gets no line for its template's own
  schedules (there are none); its manifest entries stay pending.
- The leftover-volume refusal text no longer promises an automatic
  reclaim the #3214 sweep does not make for unlabelled volumes.
- Create mode warns about a template collision only when the manifest
  entry differs from the template's row in a field deploy would write.
- template.yaml is read once per member per preview: _preflight_template
  returns the template's declared schedules, and apply's snapshot stores
  its verdict for the plan.
- Create's guard and apply's snapshot ask one function,
  crud.create_volume_refusal, so a new create-time volume refusal is
  refused by the dry run too.
- The planner's unreachable `ambiguous` branch is removed.
- _apply_report reconciles a member's lines in one pass and builds the
  pending schedule lines from the op's entries; the plan no longer keeps
  a copy of them.

Part of Abilityai/trinity-enterprise#984

@vybe vybe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merge-train: batch validated on train/20261111-0613 (#4463)

@vybe
vybe merged commit b44dc6d into dev Nov 11, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants