AI-6485 Add buffered Datadog metrics delivery by AAraKKe · Pull Request #25274 · DataDog/integrations-core

AAraKKe · 2026-09-18T10:57:48Z

What does this PR do?

Adds a centralized, buffered Datadog metrics sink to ddev.monitoring.

Treats count() as an increment, aggregating one total per series over a 10-second collection window based on sink acceptance time. Clock capture and enqueueing share a lock so concurrent producers cannot reorder collection times. Counts and gauges use Datadog v2; distributions use v1. Gauge and distribution points retain their emission timestamps.
Shares count-bounded request batching and HTTP 413 bisection between metrics and logs. Rejected batches are split until accepted; an indivisible rejected item is diagnosed and dropped without losing its siblings.
Synchronizes enqueueing, idle count-window expiry, and shutdown decisions. Both exporters recheck the queue before exiting after an empty poll; draining remains deadline-bounded. Synchronous and worker metric failures share rate-limited diagnostics through the datadog-metrics component logger.
Disables metric delivery for --dry-run and --resolve-only.

MonitoringRuntime owns and drains its metrics sink before detaching caller-owned log handlers. The Dispatcher then drains the Datadog log handler it created. The application display remains the diagnostic fallback before runtime initialization.

This adds infrastructure only, with no Dispatcher execution metric call sites. Count aggregation is local to one runtime; cross-process counter identity remains out of scope.

Motivation

AI-6485 needs a failure-isolated metrics transport before Dispatcher metric call sites are added.

Review checklist (to be filled by reviewers)

Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

dd-octo-sts · 2026-09-18T10:59:22Z

✅ Dispatcher tests: passed

Dispatcher beta: informational only
Existing CI remains the merge signal.

855/855 jobs

✅ 855 passed · nothing failed

Batches · ✅ batch-01 240/240 · ✅ batch-02 217/217 · ✅ batch-03 220/220 · ✅ batch-04 178/178

_{Dispatcher finished on a68d266 — GitHub Run.}

cit-pr-commenter-54b7da · 2026-09-18T11:02:28Z

evalya-impact-summary

evalya impact analysis

Impact analysis: RUN-ALL — every test task will run
Trigger:         empty diff (default branch, scheduled run, or shallow-clone fallback)
Test tasks:      0 (all selected)
Publish tasks:   2 (always emitted)
Diff:            empty (no diff information)

Learn more about CI impact filtering

datadog-official · 2026-09-18T11:16:48Z

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 97.25%
• Overall Coverage: 89.34% (+0.15%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 2efcfad | Docs | View more details | Give us feedback!}

AAraKKe · 2026-09-21T08:46:35Z

@codex review

chatgpt-codex-connector · 2026-09-21T08:47:02Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-21T15:20:18.158109Z	`695f39f`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 98dbbb45e8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

AAraKKe · 2026-09-21T10:36:05Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0a654aced1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-21T10:41:09Z

+    def record(self, record: MetricRecord) -> None:
+        if self._closed:
+            return
+        try:
+            self._queue.put_nowait(record)


Serialize record enqueueing with shutdown

When record() races with close(), it can observe _closed == False, pause, and then enqueue after close() has inserted the sentinel. The worker exits when it reaches that sentinel, leaving the accepted record behind it permanently and reporting neither delivery nor a drop. This can lose an in-flight metric whenever runtime shutdown overlaps an emitter; protect the closed check and enqueue with the same shutdown synchronization or otherwise ensure records cannot be placed after the sentinel.

Useful? React with 👍 / 👎.

Fixed in f801294e4d. The closed check, enqueue, and sentinel insertion share a lock in both exporters.

The workers also recheck the queue before exiting after an empty poll. Without that check, a record accepted between the poll and shutdown could still be skipped even though it was ahead of the sentinel. Coverage includes both interleavings and draining when a full queue prevents sentinel insertion.

chatgpt-codex-connector · 2026-09-21T10:41:10Z

+    assert len(submitter.series) == 5
+    assert len(submitter.metric_requests[0]) == 5


Verify the intake options this test claims to preserve

These assertions only prove that five series were emitted in one request; a regression that discards or overwrites every tag, interval, unit, or metric kind would still pass. Assert the distinct serialized options for each series, or remove this redundant test, so it verifies the observable preservation behavior named by the test.

AGENTS.md reference: AGENTS.md:L202-L204

Useful? React with 👍 / 👎.

Confirmed. The test only counted series and did not verify the behavior in its name. Removed it rather than duplicate existing coverage: the v2 payload test checks type, interval, and unit; the queued-record test checks timestamp and tags; the v1 payload test checks distribution serialization.

AAraKKe · 2026-09-21T14:13:38Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 74f5795b75

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-21T14:20:03Z

+            if record.kind is MetricKind.DISTRIBUTION:
+                distributions.add(self._distribution_series(record))
+            elif record.kind is MetricKind.COUNT:
+                rejected = counts.add(record, time.monotonic(), time.time())


Bucket counts by emission time

When the worker is blocked in a synchronous intake request, count records can remain queued for longer than COUNT_WINDOW_SECONDS; once the request returns, this line timestamps and groups every queued increment using the worker's current time rather than record.timestamp. Counts emitted across multiple ten-second periods can therefore be collapsed into one later window, distorting their timing and rate. Base count windows on emission time (or capture an emission-time monotonic value) instead of dequeue time.

Useful? React with 👍 / 👎.

Fixed in f801294e4d. Count windows use sink acceptance time, captured under the same lock as enqueueing. Monotonic time determines window membership; the paired wall time timestamps the count point. A producer paused before acceptance joins the window that accepts it, and slow intake cannot move an accepted increment into a later window.

Idle expiry uses that same lock so it cannot close a window while a producer holds a captured timestamp but has not yet enqueued. Tests cover producer overtaking, that idle-expiry race, and a backlog spanning three windows behind blocked intake. Gauges and distributions keep their original emission timestamps.

dd-octo-sts · 2026-09-22T11:23:49Z

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
`agent-reqs`	Verify check versions match the Agent requirements file	✅
`ci`	Validate CI configuration and code coverage settings	✅
`codeowners`	Validate every integration has a CODEOWNERS entry	✅
`config`	Validate default configuration files against spec.yaml	✅
`dep`	Verify dependency pins are consistent and Agent-compatible	✅
`http`	Validate integrations use the HTTP wrapper correctly	✅
`imports`	Validate check imports do not use deprecated modules	✅
`integration-style`	Validate check code style conventions	✅
`jmx-metrics`	Validate JMX metrics definition files and config	✅
`labeler`	Validate PR labeler config matches integration directories	✅
`legacy-signature`	Validate no integration uses the legacy Agent check signature	✅
`license-headers`	Validate Python files have proper license headers	✅
`licenses`	Validate third-party license attribution list	✅
`metadata`	Validate metadata.csv metric definitions	✅
`models`	Validate configuration data models match spec.yaml	✅
`openmetrics`	Validate OpenMetrics integrations disable the metric limit	✅
`package`	Validate Python package metadata and naming	✅
`qa-label`	Validate the pull request declares whether it needs QA for the next Agent release	✅
`readmes`	Validate README files have required sections	✅
`saved-views`	Validate saved view JSON file structure and fields	✅
`version`	Validate version consistency between package and changelog	✅

View full run

AAraKKe added the qa/skip-qa Automatically skip this PR for the next QA label Sep 18, 2026

dd-octo-sts Bot added the ddev label Sep 18, 2026

AAraKKe force-pushed the aarakke/AI-6485-datadog-metrics-sink branch from e2bbe08 to 35c9290 Compare September 18, 2026 10:58

AAraKKe force-pushed the aarakke/AI-6485-datadog-metrics-sink branch from 35c9290 to 98dbbb4 Compare September 21, 2026 08:34

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread ddev/src/ddev/monitoring/runtime.py Outdated

AAraKKe force-pushed the aarakke/AI-6485-datadog-metrics-sink branch 2 times, most recently from 7c1ca11 to 0a654ac Compare September 21, 2026 10:30

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

AAraKKe force-pushed the aarakke/AI-6485-datadog-metrics-sink branch 4 times, most recently from 8996bd0 to 74f5795 Compare September 21, 2026 13:51

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

AAraKKe force-pushed the aarakke/AI-6485-datadog-metrics-sink branch 2 times, most recently from f3931a7 to 695f39f Compare September 21, 2026 15:11

AAraKKe marked this pull request as ready for review September 21, 2026 15:12

AAraKKe requested a review from a team as a code owner September 21, 2026 15:12

dd-octo-sts Bot added the team/agent-integrations label Sep 21, 2026

AI-6485 Add buffered Datadog metrics delivery

f801294

AAraKKe force-pushed the aarakke/AI-6485-datadog-metrics-sink branch from 695f39f to f801294 Compare September 22, 2026 07:35

Merge branch 'master' into aarakke/AI-6485-datadog-metrics-sink

2efcfad

HadhemiDD approved these changes Sep 22, 2026

View reviewed changes

AAraKKe added this pull request to the merge queue Sep 22, 2026

Merged via the queue into master with commit c8e788c Sep 22, 2026
390 checks passed

AAraKKe deleted the aarakke/AI-6485-datadog-metrics-sink branch September 22, 2026 12:22

dd-octo-sts Bot added this to the 7.85.0 milestone Sep 22, 2026

AAraKKe mentioned this pull request Sep 23, 2026

AI-6485 Emit Dispatcher execution and operation metrics #25317

Merged

2 of 3 tasks

		assert len(submitter.series) == 5
		assert len(submitter.metric_requests[0]) == 5

Conversation

AAraKKe commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Review checklist (to be filled by reviewers)

Uh oh!

dd-octo-sts Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Dispatcher tests: passed

Uh oh!

cit-pr-commenter-54b7da Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

evalya-impact-summary

Uh oh!

datadog-official Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AAraKKe commented Sep 21, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

AAraKKe commented Sep 21, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

AAraKKe Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

AAraKKe Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

AAraKKe commented Sep 21, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

AAraKKe Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dd-octo-sts Bot commented Sep 22, 2026

Validation Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AAraKKe commented Sep 18, 2026 •

edited

Loading

dd-octo-sts Bot commented Sep 18, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Sep 18, 2026 •

edited

Loading

datadog-official Bot commented Sep 18, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 21, 2026 •

edited

Loading

AAraKKe Sep 21, 2026 •

edited

Loading

AAraKKe Sep 21, 2026 •

edited

Loading