[paradox1307]

Motivation

The current design implicitly assumes that the target side of replication knows the source schema. This limits how far the replication mechanism can go through external plugins: the metadata available to an external plugin is still incomplete, because it does not carry the metadata needed to reconstruct the table structure on the target side on its own.

If external plugins can obtain information about which fields make up the primary and unique keys, the target side can use it to:

• generate DDL/queries and recreate the source table structure as closely as possible to the original;

• identify a record when it changes, even if the target schema does not match the source;

• create indexes without accessing the source database.

This PR specifically targets external replication plugins. With this change, there will be more replication scenarios that will help integrate independent consumers, rather than using replication only for database mirroring.

Changes

The actual change is limited to the publisher, specifically IReplicatedField, which is the interface through which external replication plugins access record metadata:

• adds isPrimary(uint* position) and isUnique() to IReplicatedField. The first checks whether the column is part of the primary key. The second checks whether it is part of a unique key. If so, both methods return FB_TRUE (otherwise, FB_FALSE), and isPrimary sets the column position in the key. The returned primary key position matters for recreating the original table more accurately, as well as the indexes;

• implements these methods in ReplicatedRecordImpl by looking at the relation's index descriptions.

Impact on internal replication:

• internal replication continues to work as before;

• it does not change the internal replication log format;

• the only internal effect is that the publisher implementation can now answer whether a field is part of the primary or unique key when external plugins call these methods.

So it’s an API extension to the publisher interface for external plugins, and it doesn’t affect how internal replication works.

[aafemt]

Loading of only relevant constraints can speed up the linear lookup.

[aafemt]

BTW, what is the point to return the position within key? Building WHERE clause doesn't need it because comparison order is irrelevant.

Especially this is odd for the (strange) cases when one field is participating in different keys in different positions.

[aafemt]

If nullptr is passed - BOOM!

[paradox1307]

If nullptr is passed - BOOM!

I added a check to ensure the parameter exists. Passing nullptr is safe now.

[hvlad]

This PR adds the ability to check whether a column is a primary key or a unique key

For what ?

[paradox1307]

For what ?

The current architecture of replication plugin is not designed to execute queries to system tables from the plugin. This can cause the server to hang (at least, we encountered a hang when accessing MON$ tables while executing queries from the plugin). Retrieving PK/Unique from replication field allows to make objects/queries for client that are unaware of the current database structure or do not have access to it.

[aafemt]

Retrieving PK/Unique from replication field allows to make objects/queries for client that are unaware of the current database structure or do not have access to it.

Consider the situation when target database has a little different structure than source. You must retrieve PK/UK from target, not source. And as I said - position of the field in the key is not needed.

[sim1984]

Debezium-like replication plugins require such information (PK/UK). Therefore, it is useful in any case.

It’s not entirely clear to me why the position of fields within keys is necessary.

[EPluribusUnum]

Debezium-like replication plugins require such information (PK/UK). Therefore, it is useful in any case.

It’s not entirely clear to me why the position of fields within keys is necessary.

Maybe they don't just use is for equality condition/check.

[aafemt]

Debezium-like replication plugins require such information (PK/UK).

The only meaningful usage of this information is to cut off non-key fields from old record for update/delete but it is risky as I wrote above.

[paradox1307]

Consider the situation when target database has a little different structure than source. You must retrieve PK/UK from target, not source. And as I said - position of the field in the key is not needed.

It is bad practice to connect (from replication plugin) to target just to find out its structure. The plugin should return control to the server as quickly as possible, and external connection can significantly slow down performance. Also, access to the target is not always possible. Or it might not even exist.

In general, the plugin may have no external connections at all. It may only make messages and store them for subsequent processing by some other means.

[paradox1307]

Debezium-like replication plugins require such information (PK/UK). Therefore, it is useful in any case.

Correct.

It’s not entirely clear to me why the position of fields within keys is necessary.

Some sink-connectors can use field order in message primary keys to create schema in the target database. Adding information about keys to a message makes it more deterministic.

[aafemt]

The plugin should return control to the server as quickly as possible, and external connection can significantly slow down performance.

If the plugin sends data to the target immediately - it have to connect to it anyway. The only available optimization - connection may be established in background beforehand, not in record processing routine(s).

In general, the plugin may have no external connections at all. It may only make messages and store them for subsequent processing by some other means.

In this case it stores all data and leave processing to "apply" part that do have access to target database.

Some sink-connectors can use field order in message primary keys to create schema in the target database. Adding information about keys to a message makes it more deterministic.

For this case your implementation is wrong. As I already said, it doesn't work for more than one UK.

[hvlad]

Any PR must have clear description of both what it does and why it is necessary.
I still see no why part in this case.
Even if it is looks obvious for you, it doesn't means it is obvious (or clear) for others.

[hvlad]

The current architecture of replication plugin is not designed to execute queries to system tables from the plugin

I have no idea why the replication plugin on source side could want to execute queries.
Looks like its designer had no such idea too.

[aafemt]

I still see no why part in this case.

Consider replication of DELETE query deleting a lot of wide records from a table. The plugin can save a lot of space/traffic storing value(s) for the key only. But it requires knowledge of keys. Without isKey method, the only source of such information is a query to system tables.

PS: I have no idea why the author wanted to query any monitoring table which are famous with it total lock-up during snapshot build. Deadlock is expected in this case.

[paradox1307]

Any PR must have clear description of both what it does and why it is necessary. I still see no why part in this case. Even if it is looks obvious for you, it doesn't means it is obvious (or clear) for others.

I added motivation for PR to the first comment. Please check.

[aafemt]

Still it is not clear what is position of field is good for in the case of several unique keys.

Say for field A position 1 is returned and for field B position 1 is returned. What plugins are supposed to do with these numbers? Especially if the table has these two keys defined as B and A,B.

[hvlad]

Still it is not clear what is position of field is good for in the case of several unique keys.

Say for field A position 1 is returned and for field B position 1 is returned. What plugins are supposed to do with these numbers? Especially if the table has these two keys defined as B and A,B.

+1

[paradox1307]

Still it is not clear what is position of field is good for in the case of several unique keys.

As with the primary key, the returned column positions in the unique key can also give you the column order when rebuilding the structure on the consumer side. That’s handy for recreating indexes. The positions can also be useful when the table has no primary key and only unique keys.

I agree that with multiple unique keys, not all the positions for a column across those keys are returned. It might be worth extending the interfaces to get this info — just not within IField.

I removed the position return from the uniqueness check.

[hvlad]

I'm sorry, but... description is completely wrong (for me, at least).
I says mostly about log and that with this change log will include some metadata.
While actual change is about publisher interface and it doesn't touch a log in any point.
Also, last phrase become wrong after recent commit.

Clear description is required as it is the only source of documentation.

[aafemt]

And what about not wasting time and space on loading of non-key indexes which are used to outnumber keys?..

[sim1984]

In my view, it would be much better to provide a way to request table metadata whenever it changes, via DDL. In theory, this metadata should be published (though it’s not yet clear at what exact point), and then republished whenever a change occurs.

[paradox1307]

Clear description is required as it is the only source of documentation.

I updated the description and clarified the motivation, the changes, and the impact on internal replication. Also cleaned up the confusion around the logs. Hopefully it’s clearer now.

[paradox1307]

And what about not wasting time and space on loading of non-key indexes which are used to outnumber keys?..

I’ll try to speed that part up.

[hvlad]

Clear description is required as it is the only source of documentation.

I updated the description and clarified the motivation, the changes, and the impact on internal replication. Also cleaned up the confusion around the logs. Hopefully it’s clearer now.

Thanks