Skip to content

fix(monitor): count a heartbeat a few seconds early as the expected firing (closes #3564) - #3590

Merged
clabonte merged 1 commit into
mainfrom
fix/issue-3564-heartbeat-early-firing
Nov 15, 2026
Merged

clabonte merged 1 commit into
mainfrom
fix/issue-3564-heartbeat-early-firing

Conversation

@clabonte

Copy link
Copy Markdown
Contributor

Root cause

HeartbeatAdapter treated a schedule as silent whenever its last heartbeat was before the most recent expected firing (observed >= expected, no tolerance). Timer hosts can fire slightly early — an Azure Functions 0 */5 * * * * timer logged 03:59:59.999238Z for a 04:00:00Z firing, about 1.4% of heartbeats over a week — so healthy schedules got silent-schedule findings that persist for a whole period on daily or weekly schedules.

Fix

A heartbeat counts as the expected firing when it lands within HEARTBEAT_EARLY_FIRING_TOLERANCE (5 seconds) before it.

The issue suggested one minute. 5 seconds covers the observed early firing (sub-millisecond) with wide margin, while a missed firing on a one-minute or seconds-dialect schedule is still caught; a one-minute tolerance would mask a missed run on a * * * * * schedule.

Ingestion delay (the issue's related note) is not addressed here: it only matters when a run lands within seconds of a firing, and a tolerance on the comparison would not fix it anyway.

Tests

  • tests/test_heartbeat_early_firing.py — sub-millisecond and 5-second-early heartbeats are not silent; 6 seconds early and the previous firing's heartbeat still are. Fails on unchanged code.

Verification

  • scripts/smoke-test.sh: full suite OK, smoke test OK.

Closes #3564.

🤖 Generated with Claude Code

…iring

Closes #3564. `HeartbeatAdapter` treated a schedule as silent whenever its
last heartbeat was before the most recent expected firing, with no
tolerance. Timer hosts can fire slightly early (an Azure Functions
`0 */5 * * * *` timer logged 03:59:59.999238Z for a 04:00:00Z firing, about
1.4% of heartbeats over a week), so healthy schedules got `silent-schedule`
findings that persist for a whole period on daily or weekly schedules.

A heartbeat now counts as the expected firing when it lands within
`HEARTBEAT_EARLY_FIRING_TOLERANCE` (5 seconds) before it. The issue
suggested one minute; 5 seconds covers the observed early firing while a
missed firing on a one-minute or seconds-dialect schedule is still caught.

Tests:
- tests/test_heartbeat_early_firing.py — sub-millisecond and 5-second-early
  heartbeats are not silent; 6 seconds early and the previous firing's
  heartbeat still are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@clabonte
clabonte merged commit 1251ebc into main Nov 15, 2026
7 checks passed
@clabonte
clabonte deleted the fix/issue-3564-heartbeat-early-firing branch November 15, 2026 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

monitor: heartbeat files silent-schedule when a timer fires milliseconds before the expected instant

1 participant