DRAFT: Change Data Capture documentation#11464
Conversation
|
|
|
@stefanus-heath Just to confirm, can I take a look at this first draft or are you still adding content? |
|
|
||
| ### Service Name {#service-name} | ||
|
|
||
| The service name uniquely identifies the CDC service within the app. It is used as part of the Kafka topic name. |
There was a problem hiding this comment.
It is used as part of the Kafka topic name.
We should remove this, the topic does not include the service name
There was a problem hiding this comment.
The service name must be unique within the app, however, it is not user as part of the topic name.
Please add that we use App name to make the topic name unique, and that a topic name must be unique within the app.
|
|
||
| ## How It Works | ||
|
|
||
| A CDC service document in Studio Pro defines which entities the runtime should track. On deployment, each tracked entity gets its own Kafka topic. Every time a committed object change occurs — create, update, or delete — the runtime publishes an event to the corresponding topic. |
There was a problem hiding this comment.
Mention about snapshot too to complete the CRUD. In this case snapshot would be [R]ead.
It would be a good idea to mention Leader Selection and its configuration.
|
|
||
| A CDC service document in Studio Pro defines which entities the runtime should track. On deployment, each tracked entity gets its own Kafka topic. Every time a committed object change occurs — create, update, or delete — the runtime publishes an event to the corresponding topic. | ||
|
|
||
| The Kafka broker is either the [Mendix Event Broker](/appstore/services/event-broker/) or a Bring Your Own Kafka (BYOK) cluster. For details on configuring BYOK, see the [Mendix Event Broker](/appstore/services/event-broker/) page. |
There was a problem hiding this comment.
In case they want to BYOK, then they need a schema to parse the message. This is available in the Async API.
Mention that our Kafka message is Debezium compatible. This would help BYOK to easily integrate with consumer that accept Debezium format without any transformation in between.
| | --- | --- | --- | | ||
| | `Kafka.BootstrapServers` | The address of the Kafka broker, in the format `host:port`. | | | ||
| | `Kafka.Username` | The username for SASL/SCRAM-SHA-512 authentication with the Kafka cluster. | | | ||
| | `Kafka.Password` | The password for SASL/SCRAM-SHA-512 authentication with the Kafka cluster. | | |
There was a problem hiding this comment.
These are the complete list of our Runtime configuration, added for CDC document:
{
"Kafka.BootstrapServers",
"Kafka.Username",
"Kafka.Password",
"EventBroker.Space",
// Snapshot
"Datastorage.StartSnapshotMinInterval",
"EventBroker.CdcForceSnapshotEntities", // only available in 11.12.1
// Leader election for multi instances
"EventBroker.CdcActiveProducerElectionTopic"
"EventBroker.CdcActiveProducerElectionGroupId"
// Kafka transaction, helps also in denying non-leader
"EventBroker.CdcProducerTransactionId"
}
The EventBroker.Space default value is "local", so it is recommended to give a good name on the cloud because this is part of the Kafka topic name. Or is this part of the EventBroker documentation?
The leader election is optional but available for multi instances. I think it worth to mention it incase the user want to configure it.
|
|
||
| ### Service Name {#service-name} | ||
|
|
||
| The service name uniquely identifies the CDC service within the app. It is used as part of the Kafka topic name. |
There was a problem hiding this comment.
The service name must be unique within the app, however, it is not user as part of the topic name.
Please add that we use App name to make the topic name unique, and that a topic name must be unique within the app.
| | **Exposed name** | The name used for this entity in the Kafka topic and event payload. Defaults to the entity name. | | ||
| | **Modification** | The pending change state: **Added**, **Changed**, or **Removed**. Blank if the entity is unchanged since the last accepted revision. | | ||
| | **Revision** | The schema version of the entity's event payload. See [Revisions](#revisions). | | ||
| | **Topic** | The Kafka topic name for this entity, in the format `cdc.<app-name>.<ExposedName>.<revision>.{space}`, where `{space}` is replaced at runtime by the Event Broker space name. | |
There was a problem hiding this comment.
where
{space}is replaced at runtime by the Event Broker space name.
You can link to the actual runtime configuration here.
|
|
||
| Expand an entity row to see each attribute and association with a checkbox. Uncheck an item to exclude it from the event payload. The **Exposed name** column lets you rename individual attributes in the payload independently of their domain model names. | ||
|
|
||
| Associations appear as nested data within the parent entity's event payload. They do not produce separate Kafka topics and show no **Revision** or **Topic** values of their own. |
There was a problem hiding this comment.
- Associations appear as list of identifiers actually, not as nested data.
- "They" here can be confusing, it might refer to only associations, since we just explained about Associations. Rephrase it so "they" applies to all entity properties (attributes and associations).
|
|
||
| ## Revisions {#revisions} | ||
|
|
||
| Each tracked entity has a **Revision** number that identifies the schema version of its event payload. Downstream consumers use the revision to detect and respond to schema changes. The revision is also embedded in the Kafka topic name, so each schema version has its own isolated topic. |
There was a problem hiding this comment.
so each schema version has its own isolated topic.
we only use the major version of the schema version in topic name. If the change is minor, then you don't get a new topic name.
| Studio Pro manages revisions automatically. When you modify the tracked configuration of an entity, Studio Pro marks it as **Changed** and calculates the new revision based on whether the change is breaking or non-breaking: | ||
|
|
||
| * **Minor revision** (for example, `1.0` → `1.1`) — a non-breaking change such as adding a new attribute. Existing consumers can continue reading the topic without modification. | ||
| * **Major revision** (for example, `1.0` → `2.0`) — a breaking change such as removing an attribute, renaming an entity's exposed name, or removing an entity from tracking. Consumers must be updated to use the new topic. |
There was a problem hiding this comment.
Here I assume we are talking about revision for entity. We do have revision for document but that is hidden.
The explanation here seems to be leading toward the revision for document, especially the following description "renaming an entity's exposed name, or removing an entity from tracking."
I am going to check in Studio Pro if this is true. It is possible that when you rename exposed name then you would get v1 again since the topic name is unique enough e.g. cdc.appname.newExposedName.1.{space} -> this would be a new topic in Kafka.
There was a problem hiding this comment.
I checked the Studio Pro and we do increase major version if the exposed name get renamed. I think we can leave this description as-is.
| Accepting changes confirms the new revision numbers and clears the modification states, leaving the document in a clean state ready for deployment. | ||
|
|
||
| {{% alert color="warning" %}} | ||
| A major revision creates a new Kafka topic. Consumers subscribed to the previous topic version will no longer receive events after deployment. Ensure downstream systems are updated before deploying a major revision change. |
There was a problem hiding this comment.
Ensure downstream systems are updated before deploying a major revision change.
How is this possible? I think the ordering is for provider side to deploy first, then the consumer side can get the updated topic name. The consumer are free to consume the old topic name but it won't be updated.
5 participants
No description provided.