kafka_consumer: prefilter consumer groups before fetching offsets

[namtran1812]

What does this PR do?

Filters discovered Kafka consumer groups against consumer_groups_regex before fetching their offsets.

Previously, when consumer_groups_regex was configured, the check discovered all consumer groups and passed all of them to list_consumer_group_offsets. Regex filtering happened only afterward at the consumer group/topic/partition level.

This keeps the existing topic/partition filtering behavior while adding a group-level prefilter before offset retrieval.

Motivation

On clusters with many consumer groups, a narrow consumer_groups_regex could still result in offset requests for every discovered group.

For example, a regression test with 1,000 discovered groups and 10 matching groups verifies that only the 10 matching groups are submitted for offset retrieval, reducing the offset-fetch fan-out by 99% in that scenario.

monitor_unlisted_consumer_groups continues to bypass the prefilter and monitor all discovered groups.

Fixes #24911

Testing

• Added coverage verifying regex-matched groups are filtered before offset retrieval.
• Added coverage for configurations combining explicit consumer_groups and consumer_groups_regex.
• Added coverage ensuring monitor_unlisted_consumer_groups continues to fetch all groups.
• Added a 1,000-group regression case verifying only 10 matching groups are submitted for offset retrieval.
• Full Kafka Consumer unit suite passes.
• Ruff formatting and lint checks pass.
• git diff --check passes.

[dd-octo-sts]

✅ Dispatcher tests · passed

Dispatcher beta: informational only

Dispatcher is running alongside existing CI while we validate it. You can ignore this report and its statuses. Existing CI remains the merge signal.

  6/6 jobs
✅ 6 passed · ❌ 0 failed

Batches

Batch	State	Jobs	Workflow
batch-01	✅ passed	6/6	run 35042796981

_{Dispatcher finished on 18e7520 — GitHub Run.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 59ac06f530

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Require the prefilter to match the complete group ID

re.Pattern.match() accepts prefix matches, while the existing combined expression requires the group pattern to reach the comma separating the group from its topic. For example, consumer_groups_regex: {'orders': {}} should select only orders, but this prefilter also fetches offsets for orders-v2, orders-archive, and every other prefixed group before the later tuple-level filter discards them. On clusters with many similarly named groups—especially when a literal group is paired with a topic regex—this can preserve the large offset-fetch fan-out that the change is intended to eliminate; use a whole-ID or delimiter-aware match.

Useful? React with 👍 / 👎.

[datadog-prod-us1-4]

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 92.28% (+3.18%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 59ac06f | Docs | View more details | Give us feedback!}