Map points, ids and vectors onto Qdrant's protobuf messages - #60
Merged
Conversation
The second mapping the gRPC transport needs, and the one where the two models stop lining up. A point id is a uint64 on the wire, and Java has no unsigned long. ULong.MAX_VALUE therefore goes out as -1 and comes back as ULong.MAX_VALUE, which is correct and looks wrong, so there is a test that asserts both halves rather than only that the round trip closes. A named vector map is map<string, Vector> and each entry carries its own dense, sparse or multi variant. Kdrant's VectorData.Named is recursive and can hold another Named, which has no wire form at all. Flattening it would store a different collection than the caller described, so it is refused and the message names the offending key. VectorData.Raw is refused too. It exists so the REST engine can carry a shape this client does not model yet, which is a JSON escape hatch and not something protobuf has an equivalent for. Reads come back as VectorOutput, a separate message from Vector with the inference variants dropped, since a server never echoes back a request to embed a document. The round-trip test goes out as one and back as the other on purpose, because assuming they are interchangeable is exactly the mistake to catch here. An empty payload map reads back as no payload rather than as an empty object. Absence and emptiness are different answers to "what is stored on this point".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third slice of #52, after #54 and #58. Points, ids and vectors — the mapping where the two models stop
lining up.
The three places they disagree
uint64and Java has no unsigned long.ULong.MAX_VALUEgoes out as-1andcomes back as
ULong.MAX_VALUE. That is correct and looks wrong, so the test asserts both halvesrather than only that the round trip closes — otherwise a symmetric bug would pass.
VectorData.Namedis recursive; the wire form is not.map<string, Vector>holds one dense,sparse or multi variant per entry, with no way to nest another map. Flattening it would store a
different collection than the caller described, so it is refused and the message names the offending
key.
VectorData.Rawhas no equivalent. It exists so the REST engine can carry a JSON shape thisclient does not model yet, which is an escape hatch protobuf does not have.
A trap worth naming
Writes send
Vector; reads come back asVectorOutput, a different message with the inferencevariants (
Document,Image,InferenceObject) dropped, because a server never echoes back a requestto embed something. The round-trip test deliberately goes out as one and back as the other, since
assuming they are interchangeable is exactly the mistake to catch at this layer rather than in the
transport.
Smaller decisions
different answers to "what is stored on this point".
VectorOutputmaps tonullrather than an empty dense vector — normal for anamed-vector collection where only some vectors were written.
would attach the payload to the wrong point.
Thirteen tests. Still no
QdrantTransportimplementation: filters are the last mapping before theoperations become mechanical.