Skip to content

[improve][broker] Change bookkeeperClientSeparatedIoThreadsEnabled to True by default - #25812

Closed
zhanglistar wants to merge 1 commit into
apache:masterfrom
bigo-sg:impr/bookkeeperClientSeparatedIoThreadsEnabled
Closed

zhanglistar wants to merge 1 commit into
apache:masterfrom
bigo-sg:impr/bookkeeperClientSeparatedIoThreadsEnabled

Conversation

@zhanglistar

@zhanglistar zhanglistar commented May 19, 2026

Copy link
Copy Markdown

Motivation

Apache Pulsar PR #16333 introduced bookkeeperClientSeparatedIoThreadsEnabled so the BookKeeper client can use a dedicated IO thread pool instead of sharing the broker's Pulsar EventLoopGroup. The upstream default remained false for backward compatibility.

In our deployment, sharing IO threads between the broker and the BookKeeper client can cause contention under storage-heavy workloads. Enabling separated IO threads by default isolates BookKeeper network I/O from broker I/O and better matches our production expectations.

Modifications

  • Set bookkeeperClientSeparatedIoThreadsEnabled=true in conf/broker.conf and conf/standalone.conf, and update related comments.
  • Change the Java default in ServiceConfiguration from false to true, and update @FieldContext documentation.
  • Update unit tests to reflect the new default:
    • ServiceConfigurationTest#testBookKeeperClientIoThreads
    • BookKeeperClientFactoryImplTest#testBookKeeperIoThreadsConfiguration

Operators who want the previous behavior can set:

bookkeeperClientSeparatedIoThreadsEnabled=false

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

@zhanglistar

Copy link
Copy Markdown
Author

#19061 Not related flaky test.

@lhotari lhotari added this to the 5.0.0-M1 milestone May 19, 2026
@lhotari lhotari modified the milestones: 5.0.0-M1, 5.0.0 Sep 12, 2026

@lhotari lhotari left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be required after #26599 and #26620 since they resolve the problems caused by contention (thread or lock contention itself won't disappear). Performance increases 26x when there's a constant load in a test case included in #26620. This change will ship in Pulsar 5.0.0.

PR 26620 improvements in scenario of 500 isolated V4 producer clients/connections, 20 isolated consumer clients on one Key_Shared subscription, shared client resources, random keys, unbatched 128-byte messages.
25x consume improvement for the time that all producers are producing
99.84% lower maximum backlog (11,3M -> 17,6k)

Metric Master baseline PR Improvement
Steady ingress (produce) 139,464 msg/s 152,971 msg/s +9.7%
Steady egress (consume) 5,738 msg/s 153,002 msg/s +2566.6%
Whole-run producer throughput 132,828 msg/s 132,798 msg/s -0.02%
Whole-run consumer throughput 76,983 msg/s 143,732 msg/s +86.7%
Sampled maximum backlog 11,366,885 messages 17,693 messages 99.84% lower

@lhotari lhotari closed this Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants