cdc: add kafka max-message-bytes related explain - #21832
Conversation
ti-chi-bot
Bot
added
missing-translation-status
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthrough更新 TiCDC Kafka 文档，明确 ChangesKafka 消息大小语义与处理
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: c7f382b1-4b18-4ca0-89d0-42629e7ba666
📒 Files selected for processing (3)
ticdc/ticdc-faq.mdticdc/ticdc-sink-to-kafka.mdticdc/troubleshoot-ticdc.md
| ## TiCDC 把数据同步到 Kafka 时，能在 TiDB 中控制单条消息大小的上限吗？ | ||
|
|
||
| 对于 Avro 和 Canal-JSON 格式，消息是以行变更为单位发送的，一条 Kafka Message 仅包含一条行变更。一般情况下，消息的大小不会超过 Kafka 单条消息上限，因此，一般不需要限制单条消息大小。如果单条 Kafka 消息大小确实超过 Kafka 上限，请参考[为什么 TiCDC 到 Kafka 的同步任务延时越来越大](/ticdc/ticdc-faq.md#为什么-ticdc-到-kafka-的同步任务延时越来越大)。 | ||
| `max-message-bytes` 控制 TiCDC 将多条行变更合并为一条 Kafka message 时的大小，默认值为 `10 MB`。该参数不限制单条行变更编码后的消息大小。Kafka 能接收的消息大小由 Topic 的 `max.message.bytes` 或 broker 的 `message.max.bytes` 决定。 |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
补充 max-message-bytes 默认值的版本范围。
Line 243 将 10 MB 写成无条件默认值，但参数表说明该默认值仅从 v5.0.6 和 v4.0.6 开始生效；FAQ 对旧版本用户会产生误导。
建议替换
| `max-message-bytes` 控制 TiCDC 将多条行变更合并为一条 Kafka message 时的大小，默认值为 `10 MB`。该参数不限制单条行变更编码后的消息大小。Kafka 能接收的消息大小由 Topic 的 `max.message.bytes` 或 broker 的 `message.max.bytes` 决定。 | |
| `max-message-bytes` 控制 TiCDC 将多条行变更合并为一条 Kafka message 时的大小，默认值为 `10 MB`。从 v5.0.6 和 v4.0.6 开始，该参数的默认值分别从 `64 MB` 和 `256 MB` 调整至 `10 MB`。该参数不限制单条行变更编码后的消息大小。Kafka 能接收的消息大小由 Topic 的 `max.message.bytes` 或 broker 的 `message.max.bytes` 决定。 |
根据路径说明：可安全连续替换的修复必须提供 GitHub committable suggestion。
Source: Path instructions
There was a problem hiding this comment.
Actionable comments posted: 3
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 1a5037df-581d-4431-a64d-c0d5aacb9a66
📒 Files selected for processing (4)
ticdc/ticdc-faq.mdticdc/ticdc-open-protocol.mdticdc/ticdc-sink-to-kafka.mdticdc/troubleshoot-ticdc.md
💤 Files with no reviewable changes (1)
- ticdc/ticdc-faq.md
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 1c98e889-9bb5-47fb-9067-abae08ecd22a
📒 Files selected for processing (3)
ticdc/ticdc-open-protocol.mdticdc/ticdc-sink-to-kafka.mdticdc/troubleshoot-ticdc.md
ti-chi-bot
Bot
added
size/L
qiancai
added
translation/doing
| | Kafka Topic [`max.message.bytes`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_max.message.bytes) | 为指定 Topic 设置消息大小限制，覆盖 broker 的默认值。 | | ||
| | Kafka broker [`message.max.bytes`](https://kafka.apache.org/43/configuration/broker-configs/#brokerconfigs_message.max.bytes) | Topic 未设置 `max.message.bytes` 时使用的默认值。 | | ||
|
|
||
| Kafka Sink 启动时会读取目标 Topic 当前生效的消息大小限制，用于在发送前判断消息是否过大。如果编码后的单条消息超过该限制且未配置大消息处理，Kafka Sink 会返回 `ErrMessageTooLarge`。处理方法参见[Kafka Sink 返回 `ErrMessageTooLarge` 时，如何处理？](/ticdc/troubleshoot-ticdc.md#kafka-sink-返回-errmessagetoolarge-时如何处理)。 |
There was a problem hiding this comment.
| Kafka Sink 启动时会读取目标 Topic 当前生效的消息大小限制，用于在发送前判断消息是否过大。如果编码后的单条消息超过该限制且未配置大消息处理，Kafka Sink 会返回 `ErrMessageTooLarge`。处理方法参见[Kafka Sink 返回 `ErrMessageTooLarge` 时，如何处理？](/ticdc/troubleshoot-ticdc.md#kafka-sink-返回-errmessagetoolarge-时如何处理)。 | |
| Kafka Sink 每次启动时都会读取目标 Topic 当前生效的 Kafka 消息大小限制，并根据该限制判断编码后的单条消息是否过大。如果消息超过该限制，Kafka Sink 会根据大消息处理配置进行处理；如果未配置大消息处理，或者处理后的消息仍然超过限制，则返回 `ErrMessageTooLarge`。处理方法参见 [Kafka Sink 返回 `ErrMessageTooLarge` 时，如何处理？](/ticdc/troubleshoot-ticdc.md#kafka-sink-返回-errmessagetoolarge-时如何处理)。 | |
| 。 |
Co-authored-by: Grace Cai <qqzczy@126.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Co-authored-by: Grace Cai <qqzczy@126.com>
Co-authored-by: Grace Cai <qqzczy@126.com>
Co-authored-by: Grace Cai <qqzczy@126.com>
Co-authored-by: Grace Cai <qqzczy@126.com>
Co-authored-by: Grace Cai <qqzczy@126.com>
Co-authored-by: Grace Cai <qqzczy@126.com>
Co-authored-by: Grace Cai <qqzczy@126.com>
Co-authored-by: Grace Cai <qqzczy@126.com>
Co-authored-by: Grace Cai <qqzczy@126.com>
Co-authored-by: Grace Cai <qqzczy@126.com>
Co-authored-by: Grace Cai <qqzczy@126.com>
Co-authored-by: Grace Cai <qqzczy@126.com>
Co-authored-by: Grace Cai <qqzczy@126.com>
|
@3AceShowHand: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
3 participants
First-time contributors' checklist
What is changed, added or deleted? (Required)
Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions (in Chinese).
What is the related PR or file link(s)?
This PR is translated from:
Other reference link(s):
AI agent involvement
Do your changes match any of the following descriptions?
Summary by CodeRabbit
max-message-bytes仅用于多条行变更合并为同一 Kafka message 的大小阈值（默认 10MB，最大 100MB），不限制单条行变更编码后的大小；补充默认值调整与 Open Protocol 相关规则。max-batch-size/max-message-bytes，超阈按新 message 切分。ErrMessageTooLarge/Message was too large故障处理：按版本优先调 Topic/Broker 限制，并补充large-message-handle-option等可选方案。