Skip to content

fix(scheduler): write last_run_at only after backend accepts the run - #4659

Open
trinity-ability wants to merge 1 commit into
devfrom
feature/4655-last-run-at-after-acceptance
Open

trinity-ability wants to merge 1 commit into
devfrom
feature/4655-last-run-at-after-acceptance

Conversation

@trinity-ability

Copy link
Copy Markdown
Contributor

Summary

  • Moves last_run_at write from fire time to inside _dispatch_and_record_outcome, after the backend returns any response
  • A dispatch exception (network error, container unavailable, queue full) no longer advances last_run_at, so liveness monitors can detect schedules that silently fail on every tick
  • next_run_at is still advanced at fire time to mark the scheduling window consumed (bug: enabled schedule shows "Next: 1d ago" — stale next_run_at, cron fires silently skipped with no execution record #1472 invariant preserved)
  • In the exception path: last_run_at is advanced only when the backend demonstrably ran the execution (already-finalized row or pull-queue ownership); an unclaimed RUNNING row (backend never received the request) leaves it unchanged
  • Updates test_2514_scheduler_pull_gaps._make_db to include agent_schedules, now required by the exception path

Closes #4655

Test plan

  • TestPhantomFireDoesNotAdvanceLastRunAt::test_dispatch_exception_leaves_last_run_at_unchanged — core defect: exception must not advance last_run_at
  • TestPhantomFireDoesNotAdvanceLastRunAt::test_dispatch_exception_unclaimed_running_is_marked_failed — sanity: execution is still marked failed
  • TestSuccessfulDispatchAdvancesLastRunAt::test_success_response_advances_last_run_at
  • TestSuccessfulDispatchAdvancesLastRunAt::test_dispatched_response_advances_last_run_at
  • TestSuccessfulDispatchAdvancesLastRunAt::test_failed_response_advances_last_run_at — backend ran it and failed → still advances
  • TestExceptionWithFinalizedRow::test_already_finalized_row_advances_last_run_at — SCHED-ASYNC-001: scheduler timeout, backend already finished → advance
  • All 15 existing test_2514_scheduler_pull_gaps tests still pass

🤖 Generated with Claude Code

…4655)

Previously last_run_at was set at fire time, before the HTTP call to the
backend. A dispatch exception (container unavailable, queue full, network
error) would leave last_run_at freshly updated — a schedule that never
reached the model looked healthy to any liveness monitor reading that field.

Move the last_run_at write into _dispatch_and_record_outcome, after the
backend returns any response. The exception path only advances last_run_at
when the backend demonstrably ran the execution (already-finalized row or
pull-queue ownership); a genuinely failed dispatch (unclaimed RUNNING row,
backend never received the request) leaves last_run_at unchanged.

next_run_at is still advanced at fire time so the scheduling window is
consumed even on dispatch failures (existing #1472 invariant preserved).

Also updates test_2514_scheduler_pull_gaps._make_db to include the
agent_schedules table, which is now required by the exception path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@vybe

vybe commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

merge-train (2026-11-19): not on this train. Validated at b10260d96. Tier 1 is green, but it runs only tests/unit/, and the problems are outside it.

  1. An existing test goes red. tests/scheduler_tests/test_service.py:360 (test_execute_schedule_dispatched_updates_runtimes) asserts one update_schedule_run_times call; this PR makes two (src/scheduler/service.py:971 and :1125). It passes on dev. No workflow runs scheduler_tests/; only tests/run-full.sh does.
  2. The regression test does not detect the bug. Restoring the fire-time write at service.py:971 and keeping the rest of the PR leaves every new test green, and test_dispatch_exception_leaves_last_run_at_unchanged stays green with service.py fully reverted. The new tests call _dispatch_and_record_outcome directly, so the line in _execute_schedule_with_lock that carries the fix never runs. One test driving _execute_schedule_with_lock with a raising backend closes it.
  3. last_run_at still advances on runs that never reach the model, so three of bug(scheduler): last_run_at is written at fire time, not when the run reaches the model #4655's four criteria are unmet:
    • a 422 refusal: routers/internal.py:493-495 fails the row, the scheduler sees a terminal row, and service.py:1186-1190 advances it (the same for cancelled and skipped);
    • accepted, then failed: async mode returns accepted at internal.py:529-542 before any capacity, slot or container check, so "Agent at capacity", container unavailable and breaker-open all happen after :1125 has written it;
    • a queued row: service.py:1213 advances it explicitly.
      In production the PR changes the outcome only for backend-unreachable, 503 warming-up and dispatch timeout.

#4655 says "set only once the run is accepted by the model". The scheduler cannot know that; only the backend can, at the point the run actually starts on the agent. So the fix probably belongs on the backend side, with the scheduler no longer writing the column on the dispatch path at all. Say on the issue if you read the criterion differently.

Smaller: a dispatch timeout now leaves last_run_at stale even when the model did run; docs/memory/feature-flows/scheduler-service.md:213 still says "update last_run_at immediately"; test_async_dispatch.py:620 and test_1472_next_run_repair.py::test_a2_success_advances_once describe a single fire-time write; and client_portal/suggestions/service.py:274 can now report "has never run" for a schedule whose every dispatch fails.

status-needs-fix is set; the next push clears it.

@vybe vybe added the status-needs-fix PR has an unaddressed review/validation finding; cleared by the author's next push (#2815) label Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status-needs-fix PR has an unaddressed review/validation finding; cleared by the author's next push (#2815)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants