Skip to content

Map payloads between JSON and Qdrant's protobuf Value - #58

Merged
tonytonycoder11 merged 1 commit into
mainfrom
feat/m31-grpc-payload-mapping
Jul 31, 2026
Merged

Map payloads between JSON and Qdrant's protobuf Value#58
tonytonycoder11 merged 1 commit into
mainfrom
feat/m31-grpc-payload-mapping

Conversation

@tonytonycoder11

Copy link
Copy Markdown
Contributor

Second slice of #52, after #54. The payload mapping every other gRPC operation needs, and the one place
the two wire formats genuinely disagree.

JSON has one number type. Qdrant forked google.protobuf.Value specifically to carry integer_value
separately from double_value, because a vector database that rounds an id to a double loses it.

The bug the tests found

Checking longOrNull first looks correct and is not:

"123456789012345678901234567890".toLongOrNull()    // null
"123456789012345678901234567890".toDoubleOrNull()  // 1.2345678901234568E29 — finite, so not null

A thirty-digit id is too large for Long but parses as a Double perfectly happily, so the check falls
through to the double branch, rounds, and the write succeeds having quietly changed the number. The
mapping now decides integral or fractional from the text rather than from what the text parses to, and
there is a test asserting that the huge value really does parse as a finite double, so the reason the
naive check fails is written down rather than rediscovered.

A behavioural difference between the engines, on purpose

A value that fits neither int64 nor double is rejected. Rounding it or turning it into a string
both make the write succeed and the data mean something else, found later and far from the cause. The
message names the way out: the REST engine has no such limit, because JSON has no int64 ceiling.

This is the first place the two engines will not accept the same input. Worth knowing now rather than
discovering after someone has switched.

In the other direction a Value with no variant set decodes to null. The proto comment calls that an
error, but it is one field of a response that otherwise arrived intact, and failing the whole response
over it helps nobody.

Scope

Ten tests, covering both directions, nesting at depth, empty containers, explicit nulls, strings that
look like numbers, and Long.MAX_VALUE at the boundary. Still no QdrantTransport implementation —
that needs the point, vector and filter mappings on top of this one.

The first piece of the gRPC transport that everything else needs, and the one
place where the two wire formats genuinely disagree.

JSON has one number type. Qdrant forked google.protobuf.Value specifically to
carry integer_value separately from double_value, because a vector database that
rounds an id to a double loses it. So the mapping decides integral or fractional
from the text rather than from what the text parses to.

That distinction is not academic, and the test found it. Checking longOrNull
first looks correct and is not: a thirty digit id is too large for Long but
parses as a Double perfectly happily, so the check falls through to the double
branch, rounds, and the write succeeds having quietly changed the number.

A value that fits neither is rejected. Rounding it or turning it into a string
both make the write succeed and the data mean something else, which is then found
a long way from here. The message says the REST engine has no such limit, because
JSON has no int64 ceiling, so there is somewhere to go.

In the other direction a Value with no variant set decodes to null rather than
failing. The proto calls that an error, but it is one field of a response that
otherwise arrived intact, and losing the whole response over it helps nobody.
@tonytonycoder11
tonytonycoder11 requested a review from a team as a code owner July 31, 2026 11:08
@tonytonycoder11
tonytonycoder11 merged commit 0fc81ac into main Jul 31, 2026
6 checks passed
@tonytonycoder11
tonytonycoder11 deleted the feat/m31-grpc-payload-mapping branch July 31, 2026 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant