Skip to content

Issue #8508 : Add a maximum consume duration to the Kafka Consumer - #8518

Merged
hansva merged 1 commit into
apache:mainfrom
mattcasters:issue-8508
Sep 22, 2026
Merged

hansva merged 1 commit into
apache:mainfrom
mattcasters:issue-8508

Conversation

@mattcasters

Copy link
Copy Markdown
Contributor

Fixes #8508

Adds a Max consume duration (ms) option on the Kafka Consumer Batch tab so a pipeline can poll a topic for a fixed wall-clock window and then finish gracefully — even when messages keep arriving slowly enough that Stop when idle never fires.

  • 0 (default) means no limit, so existing pipelines are unchanged.
  • The clock starts when the transform starts. Stop when idle can still finish earlier if the topic goes quiet first.
  • When a duration is set, the consumer uses short polls and wakes poll() at the deadline so an idle topic cannot block past the limit.
  • GUI field, user-manual + CDC how-to updates, unit tests, and kafka IT main-0005-kafka-test-max-duration.

@hansva
hansva merged commit db91e00 into apache:main Sep 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: Kafka consumer needs a maximum consumation duration

2 participants