Skip to content

Report unreachable OpenMetrics endpoints to Agent Health - #25248

Draft
mwdd146980 wants to merge 14 commits into
masterfrom
mwdd146980/openmetrics-endpoint-unreachable
Draft

mwdd146980 wants to merge 14 commits into
masterfrom
mwdd146980/openmetrics-endpoint-unreachable

Conversation

@mwdd146980

@mwdd146980 mwdd146980 commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Reports an OpenMetrics Endpoint Unreachable Agent Health issue when an OpenMetrics v1 or v2 scrape fails with EHOSTUNREACH (No route to host). Envoy is covered automatically through OpenMetricsBaseCheckV2.

The issue includes a sanitized endpoint and network-focused remediation. It uses a stable per-endpoint ID and resolves when the endpoint becomes reachable, is removed during a dynamic refresh, or its configuration is unscheduled. Existing check errors and health service checks remain unchanged. Other connectivity failures are not reported. Reporting is best-effort and uses the existing AgentCheck.report_issue bridge, so no Agent or rtloader changes are required.

Motivation

This error occurs frequently in staging for OpenMetrics and Envoy (link to FA, link to logs):

image image

There was an error scraping endpoint <endpoint>: HTTPConnectionPool(...): Failed to establish a new connection: [Errno 113] No route to host

I figured fixing this for own our org would be a good motivation for reporting from integrations to Agent Health.

mwdd146980 and others added 3 commits September 16, 2026 14:12
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@mwdd146980 mwdd146980 added the qa/required QA is required for this PR and will generate a QA card label Sep 16, 2026
@dd-octo-sts

dd-octo-sts Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

✅ Dispatcher tests · passed

Dispatcher beta: informational only

Dispatcher is running alongside existing CI while we validate it. You can ignore this report and its statuses. Existing CI remains the merge signal.

  853/853 jobs
✅ 853 passed · ❌ 0 failed

Batches

BatchStateJobsWorkflow
batch-01✅ passed240/240run 35272838541
batch-02✅ passed215/215run 35272839166
batch-03✅ passed220/220run 35272838707
batch-04✅ passed178/178run 35272838619
Dispatcher finished on 3aa766cGitHub Run.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Sep 16, 2026

Copy link
Copy Markdown

evalya-impact-summary

evalya impact analysis
Impact analysis: 0 selected, 0 skipped (of 0 test tasks)
Publish tasks:   2 (always emitted)
Diff (8 files):
  datadog_checks_base/changelog.d/25248.added
  datadog_checks_base/datadog_checks/base/checks/openmetrics/base_check.py
  datadog_checks_base/datadog_checks/base/checks/openmetrics/endpoint_unreachable_issue.py
  datadog_checks_base/datadog_checks/base/checks/openmetrics/mixins.py
  datadog_checks_base/datadog_checks/base/checks/openmetrics/v2/base.py
  datadog_checks_base/datadog_checks/base/checks/openmetrics/v2/scraper/base_scraper.py
  datadog_checks_base/datadog_checks/base/stubs/datadog_agent.py
  datadog_checks_base/tests/base/checks/openmetrics/test_endpoint_unreachable_issue.py

Debug a specific task: evalya plan impact --path <path> --task <task>

Learn more about CI impact filtering

@dd-octo-sts

dd-octo-sts Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Disk usage change

Commit c7e2073 compared against b79af8c.

Uncompressed

linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
🔺 +17.18 KiB 🔺 +17.18 KiB 🔺 +17.18 KiB 🔺 +17.18 KiB 🔺 +17.18 KiB
Details
Name Version Type linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
datadog_checks_base 38.3.2 Integration 🔺 +17.18 KiB 🔺 +17.18 KiB 🔺 +17.18 KiB 🔺 +17.18 KiB 🔺 +17.18 KiB

Compressed

linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
🔺 +4.62 KiB 🔺 +4.62 KiB 🔺 +4.62 KiB 🔺 +4.62 KiB 🔺 +4.62 KiB
Details
Name Version Type linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
datadog_checks_base 38.3.2 Integration 🔺 +4.62 KiB 🔺 +4.62 KiB 🔺 +4.62 KiB 🔺 +4.62 KiB 🔺 +4.62 KiB

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Tests  Code Coverage

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog retried 1 test - 1 passed on retry View in Datadog

🎯 Code Coverage (details)
Patch Coverage: 96.80%
Overall Coverage: 89.26% (+0.15%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c7e2073 | Docs | View more details | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Validation Report

All 21 validations passed.

Show details
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file
ci Validate CI configuration and code coverage settings
codeowners Validate every integration has a CODEOWNERS entry
config Validate default configuration files against spec.yaml
dep Verify dependency pins are consistent and Agent-compatible
http Validate integrations use the HTTP wrapper correctly
imports Validate check imports do not use deprecated modules
integration-style Validate check code style conventions
jmx-metrics Validate JMX metrics definition files and config
labeler Validate PR labeler config matches integration directories
legacy-signature Validate no integration uses the legacy Agent check signature
license-headers Validate Python files have proper license headers
licenses Validate third-party license attribution list
metadata Validate metadata.csv metric definitions
models Validate configuration data models match spec.yaml
openmetrics Validate OpenMetrics integrations disable the metric limit
package Validate Python package metadata and naming
qa-label Validate the pull request declares whether it needs QA for the next Agent release
readmes Validate README files have required sections
saved-views Validate saved view JSON file structure and fields
version Validate version consistency between package and changelog

View full run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

base_package qa/required QA is required for this PR and will generate a QA card

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant