Skip to content

Decode oplog times from connector offsets - #2883

Open
delthas wants to merge 1 commit into
improvement/BB-906/hashed-pipelinesfrom
improvement/BB-906/resume-token-times
Open

delthas wants to merge 1 commit into
improvement/BB-906/hashed-pipelinesfrom
improvement/BB-906/resume-token-times

Conversation

@delthas

@delthas delthas commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Reads the cluster time of mongo-kafka's committed resume tokens and of configured startup times.

Part of the hashed oplog ingestion stack, overview in #2869.

Issue: BB-906

@delthas
delthas added this pull request to stack #2885 October 14, 2026 08:48
Comment thread extensions/oplogPopulator/modules/resumeToken.js Outdated
@delthas
delthas force-pushed the improvement/BB-906/resume-token-times branch from 1bf8d43 to 8d6624b Compare October 14, 2026 09:02
@delthas
delthas removed this pull request from stack #2885 October 14, 2026 10:06
@delthas
delthas force-pushed the improvement/BB-906/resume-token-times branch from 8d6624b to 616f3b5 Compare October 14, 2026 10:07
@delthas
delthas added this pull request to stack #2886 October 14, 2026 10:07
@delthas
delthas force-pushed the improvement/BB-906/resume-token-times branch from 616f3b5 to 41516b3 Compare October 14, 2026 12:22
@delthas
delthas force-pushed the improvement/BB-906/resume-token-times branch from 41516b3 to 90b7f88 Compare October 14, 2026 12:49
@delthas
delthas force-pushed the improvement/BB-906/resume-token-times branch from 90b7f88 to 48f5e2a Compare October 14, 2026 12:58
@delthas
delthas force-pushed the improvement/BB-906/resume-token-times branch from 48f5e2a to 25b11b4 Compare October 14, 2026 13:04
@delthas
delthas requested review from a team, DarkIsDude and benzekrimaha October 14, 2026 13:15
Comment thread extensions/oplogPopulator/modules/resumeToken.js Outdated
Comment thread extensions/oplogPopulator/modules/resumeToken.js Outdated
Comment thread extensions/oplogPopulator/modules/resumeToken.js Outdated
Comment thread extensions/oplogPopulator/modules/resumeToken.js Outdated
Comment thread extensions/oplogPopulator/modules/resumeToken.js Outdated
Comment thread tests/utils/resumeTokens.js Outdated
@delthas
delthas force-pushed the improvement/BB-906/resume-token-times branch from 25b11b4 to 448dd6e Compare October 15, 2026 08:58

@DarkIsDude DarkIsDude left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing more than Francois

@delthas
delthas force-pushed the improvement/BB-906/resume-token-times branch from 448dd6e to 5ecb97c Compare October 15, 2026 15:34
Comment thread extensions/oplogPopulator/modules/resumeToken.js Outdated
Comment thread extensions/oplogPopulator/modules/resumeToken.js Outdated
@delthas
delthas force-pushed the improvement/BB-906/resume-token-times branch from 5ecb97c to 9b96977 Compare October 15, 2026 17:40
@delthas
delthas force-pushed the improvement/BB-906/resume-token-times branch from 9b96977 to db6a357 Compare October 16, 2026 10:47
@delthas
delthas force-pushed the improvement/BB-906/resume-token-times branch from db6a357 to 953285e Compare October 20, 2026 14:14
@delthas
delthas force-pushed the improvement/BB-906/resume-token-times branch from 953285e to 12ea157 Compare October 20, 2026 14:44
@delthas
delthas removed this pull request from stack #2886 October 20, 2026 14:45
@delthas
delthas added this pull request to stack #2895 October 20, 2026 14:45
mongo-kafka commits the change stream resume token as a JSON string in
the `_id` of its source offsets. A resume token starts with the cluster
time of its event (KeyString timestamp type 0x82, then t and i as
big-endian 32-bit integers), heartbeat tokens included.

Also parse the configured startup time of hashed connectors: an extended
JSON timestamp, which keeps the increment.

Issue: BB-906
@delthas
delthas force-pushed the improvement/BB-906/resume-token-times branch from 12ea157 to 21f1ae1 Compare October 20, 2026 16:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants