Skip to content

timex: filter unreasonable offset values from kernel adjtimex() overflow #3766

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dongjiang1989 wants to merge 2 commits into
base: master
Choose a base branch
from
+25 −3
Open

timex: filter unreasonable offset values from kernel adjtimex() overflow #3766

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
af2863d
timex: filter unreasonable offset values from kernel adjtimex() overflow
dongjiang1989 Jul 31, 2026
9341e72
timex: add overflow diagnostic counter and relax alert threshold
dongjiang1989 Aug 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions collector/timex.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,11 @@ const (

// See NOTES in adjtimex(2).
ppm16frac = 1000000.0 * 65536.0

// offsetOverflowThresholdSec: |offset| exceeding this indicates a likely
// kernel adjtimex() overflow artifact (2^32 ns ≈ 4.29s on KVM/pvclock).
// See https://github.com/prometheus/node_exporter/issues/3764
offsetOverflowThresholdSec = 4.0
)

type timexCollector struct {
Expand All @@ -58,8 +63,10 @@ type timexCollector struct {
errcnt,
stbcnt,
tai,
syncStatus typedDesc
logger *slog.Logger
syncStatus,
overflow typedDesc
logger *slog.Logger
overflowCount uint64
}

func init() {
Expand Down Expand Up @@ -156,6 +163,11 @@ func NewTimexCollector(logger *slog.Logger) (Collector, error) {
"Is clock synchronized to a reliable server (1 = yes, 0 = no).",
nil, nil,
), prometheus.GaugeValue},
overflow: typedDesc{prometheus.NewDesc(
prometheus.BuildFQName(namespace, subsystem, "offset_overflow_total"),
"Count of adjtimex() offset readings indicating kernel overflow (|offset| > 4s). See https://github.com/prometheus/node_exporter/issues/3764",
nil, nil,
), prometheus.CounterValue},
logger: logger,
}, nil
}
Expand Down Expand Up @@ -185,8 +197,14 @@ func (c *timexCollector) Update(ch chan<- prometheus.Metric) error {
divisor = microSeconds
}

offsetSec := float64(timex.Offset) / divisor
if offsetSec > offsetOverflowThresholdSec || offsetSec < -offsetOverflowThresholdSec {
c.overflowCount++
}

ch <- c.syncStatus.mustNewConstMetric(syncStatus)
ch <- c.offset.mustNewConstMetric(float64(timex.Offset) / divisor)
ch <- c.offset.mustNewConstMetric(offsetSec)
Comment on lines +200 to +206

@SuperQ SuperQ Aug 18, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, this is not acceptable.

  • This is far too tight a constraint.
  • We avoid filtering values at collection time.

@dongjiang1989 dongjiang1989 Aug 21, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @SuperQ . I Got it.

  1. Removed collector-side filtering — raw adjtimex() values are now preserved as-is. No more masking or replacing offset values.

  2. Added diagnostic counter node_timex_offset_overflow_total — increments when |offset| > 4s, providing visibility into KVM/pvclock overflow events without modifying the primary metric.

  3. Relaxed alert threshold to ±60s — this filters the 4.29s overflow artifact while still catching genuinely broken clocks (minutes/hours off).

ch <- c.overflow.mustNewConstMetric(float64(c.overflowCount))
ch <- c.freq.mustNewConstMetric(1 + float64(timex.Freq)/ppm16frac)
ch <- c.maxerror.mustNewConstMetric(float64(timex.Maxerror) / microSeconds)
ch <- c.esterror.mustNewConstMetric(float64(timex.Esterror) / microSeconds)
Expand Down
4 changes: 4 additions & 0 deletions docs/node-mixin/alerts/alerts.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,12 +215,16 @@
expr: |||
(
node_timex_offset_seconds{%(nodeExporterSelector)s} > 0.05
and
node_timex_offset_seconds{%(nodeExporterSelector)s} < 60
and
deriv(node_timex_offset_seconds{%(nodeExporterSelector)s}[5m]) >= 0
)
or
(
node_timex_offset_seconds{%(nodeExporterSelector)s} < -0.05
and
node_timex_offset_seconds{%(nodeExporterSelector)s} > -60
and
deriv(node_timex_offset_seconds{%(nodeExporterSelector)s}[5m]) <= 0
)
Expand Down
Loading