Skip to content

Fix the consumer race in spring-kafka-1.3.x and 2.2.x scenarios - #844

Open
wu-sheng wants to merge 1 commit into
mainfrom
fix-spring-kafka-scenario-race
Open

wu-sheng wants to merge 1 commit into
mainfrom
fix-spring-kafka-scenario-race

Conversation

@wu-sheng

Copy link
Copy Markdown
Member

Fix the consumer race in spring-kafka-1.3.x and 2.2.x scenarios

Follow-up to #841, which fixed this race in spring-kafka-2.3.x-scenario. The 1.3.x and 2.2.x scenarios have the same code, so they have the same latent race. They still pass in CI only because they usually win it.

Cause

  • The consumer has no committed offset and starts from the latest offset.
  • healthCheck returned 200 as soon as Tomcat started.
  • If the entry request sent helloWorld before the consumer got its partition, the consumer skipped the message. The request then waited forever on the latch, so its segment was never reported (SegmentNotFoundException for GET:/case/spring-kafka-case).

Fix

  • healthCheck reports healthy only after a ConsumerRebalanceListener sees a non-empty assignment, the same as the 2.3.x fix.
  • This leaves no window. The scenarios disable auto commit, so the container calls consumer.position() and commits for every assigned partition before it calls the user listener. I checked this in the bytecode of 1.3.9, 1.3.10, 1.3.11 and 2.2.14.

Verification

  • I ran every version in both support-version.list files locally against a fresh Kafka 2.4.1 broker, with the mock collector, the agent and the validator.
  • The health check is polled every 0.2s, and the entry is called right after it passes.
  • group.initial.rebalance.delay.ms stands in for a slow runner.
Version main's controller, 15s delay This PR, 3s delay This PR, 15s delay
1.3.9.RELEASE FAILED (entry segment missing) PASSED PASSED
1.3.10.RELEASE FAILED (entry segment missing) PASSED PASSED
1.3.11.RELEASE FAILED (entry segment missing) PASSED PASSED
2.2.14.RELEASE FAILED (entry segment missing) PASSED PASSED

At the broker's default 3s delay, main's controller failed once in two 1.3.11 runs. The health check now returns 500 until the assignment, about 3s or 15s.

This changes only test scenarios; there is no plugin or CHANGES.md change.

Both scenarios have the race fixed for spring-kafka-2.3.x-scenario in #841.
The consumer has no committed offset and starts from the latest offset, while
healthCheck reported success as soon as Tomcat started. When the entry request
sent "helloWorld" before the consumer got its partition, the consumer skipped
it, the request waited forever on the latch and its segment was never
reported (SegmentNotFoundException for GET:/case/spring-kafka-case).

Report healthy only after partitions are assigned. With auto commit disabled,
the container resolves and commits the position of every assigned partition
before it calls this listener, so a message sent after the health check
passes is always consumed.
@wu-sheng wu-sheng added this to the 9.9.0 milestone Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant