Skip to content

feat: allow source and destination clusters to overlap - #1473

Draft
levkk wants to merge 2 commits into
mainfrom
levkk-omni-only-sync
Draft

feat: allow source and destination clusters to overlap#1473
levkk wants to merge 2 commits into
mainfrom
levkk-omni-only-sync

Conversation

@levkk

@levkk levkk commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Allow source and destination clusters to overlap. Matching databases in the destination cluster are ignored and data is only routed to shards not specified in the source cluster. This little hack allows us to:

  1. Sync omni tables to new shards added to an existing cluster
  2. Copy rows from existing nodes to new nodes with new sharding map

Example

[[databases]]
name = "source"
host = "10.0.0.0"

[[databases]]
name = "destination"
host = "10.0.0.0" # Same host as source!
shard = 0

[[databases]]
name = "destination"
host = "10.0.0.1"
shard = 1

[[sharded_tables]]
database = "destination"
column = "tenant_id"

[[sharded_tables.mapping]]
values = [1, 2, 3]
shard = 0

[[sharded_tables.mapping]]
values = [4]
shard = 1

Using data-sync, we can copy:

  1. all omni tables
  2. all rows matching tenant_id = 4

from source to destination, without touching the original node.

@levkk
levkk marked this pull request as draft September 1, 2026 21:08
@codecov

codecov Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.66667% with 14 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...src/backend/replication/logical/subscriber/copy.rs 69.69% 10 Missing ⚠️
...c/backend/replication/logical/subscriber/stream.rs 93.47% 3 Missing ⚠️
.../replication/logical/subscriber/duplicate_check.rs 90.90% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant