Repository navigation
[#642] [#648] Carry arrays in a ValueArray, with nulls recorded by index - #649
Conversation
Per the spec team decision. All 16 extension ranges are removed, along with the TypedValueExtension message, which existed only to host one, and the extension_value arm of the TypedValue oneof. They had never been used. There is no extend statement anywhere in the specification, Tahu 3.x, Tahu develop or the TCK, and no hand-written call to the extension API in any of them, after three major versions of being reserved. They were not free. Six changes in v4 development moved a floor - typed Commands, the metadata separation, the topic namespace change, the header/content split, TemplateList, and the ValueArray in eclipse-sparkplug#649 - each a theoretical break for any third party occupying the numbers being vacated, for no actual third party. They also did not work in the JSON encoding. The Protocol Buffers JSON Mapping is specified for proto3, which has no extensions, so eclipse-sparkplug#645 had to invent three statements defining the bracketed key form, and an extension still survived a binary round trip but not a JSON one. Those three statements are removed with the mechanism, and the rationales that rested on extension ranges are reworded to rest on the ordinary unknown-field rule instead. array_value takes field number 16, which the extension arm freed, rather than leaving a gap at 16 with the array at 17. Not a wire change for anything that exists: a payload exercising metrics, a DataSet and a null cell serializes byte-identically under both schemas, and each decodes the other's output.
nathandavenport
left a comment
There was a problem hiding this comment.
Review: [#642] [#648] Carry arrays in a ValueArray (develop 10bba93..f5353a3)
The design is sound: fixing #642 and #648 together with one null mechanism is the right call, the size tables are honest about the DateTime cost, and zigzag for the signed arrays is correct. There are four required changes before merge, three of them inline:
- Unrelated files are included in the commit.
- The PR conflicts with #643, which is now merged.
- The TCK DataSet encoder puts unsigned and DateTime values, and null placeholders, into the wrong typed arrays.
Required, outside the diff:
- The TCK still encodes array metrics as
bytes_value. InSparkplugBPayloadEncoder.java,Int8ArraythroughDateTimeArray(from line 442) still pack little-endian bytes intobytes_value. Only DataSet columns were moved toValueArray. Array metrics built by the TCK therefore break the new tck-id-payloads-array-value-array. Fix: encode them intoarray_valuewith the same typed arrays the Column uses.
Should fix, outside the diff: the spec still describes the old layout in three places in Sparkplug_6_Payloads.adoc:
- The DataSet example (line 1126) still shows
"int_values": [1, 2]. - The DateTime note (line 1753) says a DateTime column "shares the long_values array with UInt64"; it now has its own
int64_values. - The JSON DataSet section (line 3842) still says "one of six parallel arrays".
Minor, already there before this PR: Monitor.metricsEqual (Monitor.java:1408) doesn't compare array_value. It also compares message fields with ==, so it was already unreliable for bytes_value.
Not executed: no build or TCK run. Findings are from reading the diff and the files at f5353a3.
3 optional suggestion(s) are included. They are not required for approval.
…ueArray, with nulls recorded by index Closes eclipse-sparkplug#642 and eclipse-sparkplug#648, which are the same defect seen from two directions: a packed array cannot express an absent element. A DataSet.Column carried its values in a parallel array indexed by row, so a null cell had nowhere to go - omitting it shifts every later row, and writing a zero is indistinguishable from a real zero. Sparkplug 3 could express this, because its row-major DataSetValue was present with its oneof unset. The array DataTypes had the same problem and were additionally opaque in JSON, where an element array appeared as base64. The schema also held two conventions for an array of scalars: DataSet.Column used native repeated fields while the array DataTypes used packed bytes. One ValueArray message now serves both, with null_indexes recording the elements that are absent. The signed arrays use zigzag. A negative value in a plain varint field occupies ten bytes, so sint32 is about three times smaller than int32 on signed data; on 100 signed values in -500..500 the difference is 193 bytes against 607. DateTime keeps a plain varint array, since its values are large positive nanosecond counts for which zigzag only adds a bit. The trade is deliberate rather than free. Packed varints beat the packed bytes they replace on small magnitudes - 185 bytes against 403 for 100 values in 0..1000 - but DateTime arrays grow about 12%, because a nanosecond count needs a nine byte varint. Accepting that buys null support in both places, one convention instead of two, and array values that are readable in the JSON encoding rather than base64, which is the reason the JSON encoding exists. null_indexes costs nothing when an array has no nulls and a few bytes per null otherwise: on a 100 element array, 0 nulls adds 0 bytes and 20 nulls adds 22. In JSON a ValueArray is a plain array and an absent element is a JSON null, so null_indexes has no JSON counterpart. The TCK encoder is updated to the new shape, and a null DataSet cell now records its row index rather than silently writing a zero.
…d arms, placeholders, array metrics Addresses nathandavenport's review on eclipse-sparkplug#649. - UInt32/UInt64 fallback branches wrote to the signed arrays (sint32_values/sint64_values). They now write uint32_values and uint64_values. - addDataSetPlaceholder wrote its zero into a fixed array regardless of the column's datatype, so a null cell in an unsigned or DateTime column landed in the wrong array. It now picks the array that matches the datatype. - The 13 array metric cases still encoded packed little-endian bytes_value. They now build a ValueArray and record null elements in null_indexes, as payloads-array-value-array requires. - null_indexes MUST be in ascending order. - "at most one" typed array rather than "exactly one": an empty array populates none of them. - Three stale descriptions in the chapter: the DataSet example now shows "values": { "uint32_values": [...] }, the DateTime note points at int64_values rather than claiming it shares UInt64's array, and the JSON DataSet deviation is rewritten for the column shape.
nathandavenport
left a comment
There was a problem hiding this comment.
Re-review at fbefbee
All four required findings and the three optional ones are fixed, and so are the out-of-diff ones from the first review: array metrics now use array_value, and the stale int_values / long_values / "six parallel arrays" text is gone. I've resolved those threads.
Three small leftovers, all optional: the schema comment's "Exactly one", the Boolean column in the DataSet example, and the stale comment above addDataSetPlaceholder.
3 optional suggestion(s) are included. They are not required for approval.
…nts and one example column - The ValueArray schema comment still said "exactly one" of the typed arrays is populated, contradicting both normative statements. - The Enabled column of the DataSet example kept the old column-level "boolean_values" key while the other two columns moved to "values", so the example showed two different Column shapes. - The comment above addDataSetPlaceholder still said a Column cannot express an absent cell, which null_indexes made untrue.
Closes #642. Closes #648.
These are the same defect seen from two directions: a packed array cannot express an absent
element. Resolved together, because deciding them separately risks two incompatible answers.
The problem
DataSet.Columncarries its values in a parallel array indexed by row, so a null cell hasnowhere to go. Omitting it shortens the column and shifts every later row, which
payloads-dataset-row-correlationforbids; writing a zero makes null indistinguishable from areal zero. Sparkplug 3 could express this — its row-major
DataSetValuewas present with theoneof unset — so this is a capability regression.
bytes_valueas packed little-endian bytes: same no-absence problem,and opaque in JSON, where a consumer sees
"Fc1bB7Fo3jo=".The schema also held two conventions for an array of scalars —
DataSet.Columnused nativerepeated fields while the array DataTypes used packed bytes.
The change
TypedValuegainsarray_value = 16andDataSet.Columnreplaces its six typed arrays with asingle
ValueArray. One convention, one null mechanism.Zigzag is not optional
A negative value in a plain varint field occupies ten bytes, so the signed arrays use
sint32andsint64. 100 elements:bytes_valueint32sint32Getting this wrong would have made signed arrays three times larger than the packed bytes they
replace. DateTime is the exception and keeps a plain varint array: its values are large positive
nanosecond counts, for which zigzag only adds a bit.
The trade, stated plainly
This is not a free win. Packed varints beat packed bytes on small magnitudes, but DateTime arrays
grow about 12%, because a nanosecond count needs a nine-byte varint:
bytes_valueint64Accepting that buys null support in both places, one convention instead of two, and array values
that are readable in the JSON encoding rather than base64 — which is the reason the JSON encoding
exists at all. My framing in #642, that native packed is smaller for realistic process data, holds
for small-magnitude integers and is wrong for DateTime; the corrected numbers are above.
null_indexesis pay-as-you-go on a 100-element array: 0 nulls adds 0 bytes, 1 adds 3, 20 adds 22.That is why indexes rather than a parallel array of flags.
JSON
A
ValueArrayis a plain JSON array and an absent element is a JSON null, sonull_indexeshas noJSON counterpart. The schema gains an
arrayElementdefinition so arrays admit null, and the twostructural deviations are reworded.
TCK
The encoder is updated to the new shape, and a null DataSet cell now records its row index instead
of silently writing a zero. Verified end to end:
Verification
Build passes; schema compiles under
protoc. JSON Schema valid draft 2020-12, 22/22 examplepayloads validate, 15/15 negative tests plus 4 new array/null cases behave as specified.
Broken-anchor (11) and tck-id-anomaly (7) baselines unchanged, measured against develop.
Cost
A breaking wire change to spCv4.0 —
bytesand a packed repeated field are different wire types,and
DataSet.Columnchanges shape. v4 is unreleased, so this is the free moment.Rebased on develop now that #643 has merged (#650). With the extension ranges gone there are no
extension floors to move, and
array_valuetakes field 16 — the number #650 freed when it removedextension_value— rather than opening a new gap.