Skip to content

[#642] [#648] Carry arrays in a ValueArray, with nulls recorded by index - #649

Merged
wes-johnson merged 3 commits into
eclipse-sparkplug:developfrom
wes-johnson:642-648
Nov 5, 2026
Merged

wes-johnson merged 3 commits into
eclipse-sparkplug:developfrom
wes-johnson:642-648

Conversation

@wes-johnson

@wes-johnson wes-johnson commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Closes #642. Closes #648.

These are the same defect seen from two directions: a packed array cannot express an absent
element.
Resolved together, because deciding them separately risks two incompatible answers.

The problem

  • A DataSet.Column carries its values in a parallel array indexed by row, so a null cell has
    nowhere to go. Omitting it shortens the column and shifts every later row, which
    payloads-dataset-row-correlation forbids; writing a zero makes null indistinguishable from a
    real zero. Sparkplug 3 could express this — its row-major DataSetValue was present with the
    oneof unset — so this is a capability regression.
  • The array DataTypes ride in bytes_value as packed little-endian bytes: same no-absence problem,
    and opaque in JSON, where a consumer sees "Fc1bB7Fo3jo=".

The schema also held two conventions for an array of scalars — DataSet.Column used native
repeated fields while the array DataTypes used packed bytes.

The change

message ValueArray {
  repeated sint32 sint32_values  = 1;   // Int8, Int16, Int32
  repeated sint64 sint64_values  = 2;   // Int64
  repeated uint32 uint32_values  = 3;   // UInt8, UInt16, UInt32
  repeated uint64 uint64_values  = 4;   // UInt64
  repeated int64  int64_values   = 5;   // DateTime; not zigzag - pre-1970 costs 10 bytes
  repeated float  float_values   = 6;
  repeated double double_values  = 7;
  repeated bool   boolean_values = 8;
  repeated string string_values  = 9;
  repeated uint32 null_indexes   = 10;  // omitted entirely when nothing is null
}

TypedValue gains array_value = 16 and DataSet.Column replaces its six typed arrays with a
single ValueArray. One convention, one null mechanism.

Zigzag is not optional

A negative value in a plain varint field occupies ten bytes, so the signed arrays use sint32 and
sint64. 100 elements:

data bytes_value int32 sint32
small, non-negative (0..1000) 403 185 194
signed small (-500..500) 403 607 193
full-range random int32 403 736 495

Getting this wrong would have made signed arrays three times larger than the packed bytes they
replace. DateTime is the exception and keeps a plain varint array: its values are large positive
nanosecond counts, for which zigzag only adds a bit.

The trade, stated plainly

This is not a free win. Packed varints beat packed bytes on small magnitudes, but DateTime arrays
grow about 12%
, because a nanosecond count needs a nine-byte varint:

data bytes_value int64
DateTime (ns since epoch), n=100 803 903

Accepting that buys null support in both places, one convention instead of two, and array values
that are readable in the JSON encoding rather than base64 — which is the reason the JSON encoding
exists at all. My framing in #642, that native packed is smaller for realistic process data, holds
for small-magnitude integers and is wrong for DateTime; the corrected numbers are above.

null_indexes is pay-as-you-go on a 100-element array: 0 nulls adds 0 bytes, 1 adds 3, 20 adds 22.
That is why indexes rather than a parallel array of flags.

JSON

A ValueArray is a plain JSON array and an absent element is a JSON null, so null_indexes has no
JSON counterpart. The schema gains an arrayElement definition so arrays admit null, and the two
structural deviations are reworded.

TCK

The encoder is updated to the new shape, and a null DataSet cell now records its row index instead
of silently writing a zero. Verified end to end:

sint32_values : [10, 0, 30]
null_indexes  : [1]
3 entries, row 1 marked null - distinguishable from a real zero

Verification

Build passes; schema compiles under protoc. JSON Schema valid draft 2020-12, 22/22 example
payloads validate, 15/15 negative tests plus 4 new array/null cases behave as specified.
Broken-anchor (11) and tck-id-anomaly (7) baselines unchanged, measured against develop.

Cost

A breaking wire change to spCv4.0 — bytes and a packed repeated field are different wire types,
and DataSet.Column changes shape. v4 is unreleased, so this is the free moment.

Rebased on develop now that #643 has merged (#650). With the extension ranges gone there are no
extension floors to move, and array_value takes field 16 — the number #650 freed when it removed
extension_value — rather than opening a new gap.

wes-johnson added a commit to wes-johnson/Sparkplug that referenced this pull request Nov 3, 2026
Per the spec team decision. All 16 extension ranges are removed, along with the
TypedValueExtension message, which existed only to host one, and the
extension_value arm of the TypedValue oneof.

They had never been used. There is no extend statement anywhere in the
specification, Tahu 3.x, Tahu develop or the TCK, and no hand-written call to the
extension API in any of them, after three major versions of being reserved.

They were not free. Six changes in v4 development moved a floor - typed Commands,
the metadata separation, the topic namespace change, the header/content split,
TemplateList, and the ValueArray in eclipse-sparkplug#649 - each a theoretical break for any third
party occupying the numbers being vacated, for no actual third party.

They also did not work in the JSON encoding. The Protocol Buffers JSON Mapping is
specified for proto3, which has no extensions, so eclipse-sparkplug#645 had to invent three
statements defining the bracketed key form, and an extension still survived a
binary round trip but not a JSON one. Those three statements are removed with the
mechanism, and the rationales that rested on extension ranges are reworded to
rest on the ordinary unknown-field rule instead.

array_value takes field number 16, which the extension arm freed, rather than
leaving a gap at 16 with the array at 17.

Not a wire change for anything that exists: a payload exercising metrics, a
DataSet and a null cell serializes byte-identically under both schemas, and each
decodes the other's output.

@nathandavenport nathandavenport left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: [#642] [#648] Carry arrays in a ValueArray (develop 10bba93..f5353a3)

The design is sound: fixing #642 and #648 together with one null mechanism is the right call, the size tables are honest about the DateTime cost, and zigzag for the signed arrays is correct. There are four required changes before merge, three of them inline:

  • Unrelated files are included in the commit.
  • The PR conflicts with #643, which is now merged.
  • The TCK DataSet encoder puts unsigned and DateTime values, and null placeholders, into the wrong typed arrays.

Required, outside the diff:

  • The TCK still encodes array metrics as bytes_value. In SparkplugBPayloadEncoder.java, Int8Array through DateTimeArray (from line 442) still pack little-endian bytes into bytes_value. Only DataSet columns were moved to ValueArray. Array metrics built by the TCK therefore break the new tck-id-payloads-array-value-array. Fix: encode them into array_value with the same typed arrays the Column uses.

Should fix, outside the diff: the spec still describes the old layout in three places in Sparkplug_6_Payloads.adoc:

  • The DataSet example (line 1126) still shows "int_values": [1, 2].
  • The DateTime note (line 1753) says a DateTime column "shares the long_values array with UInt64"; it now has its own int64_values.
  • The JSON DataSet section (line 3842) still says "one of six parallel arrays".

Minor, already there before this PR: Monitor.metricsEqual (Monitor.java:1408) doesn't compare array_value. It also compares message fields with ==, so it was already unreliable for bytes_value.

Not executed: no build or TCK run. Findings are from reading the diff and the files at f5353a3.
3 optional suggestion(s) are included. They are not required for approval.

Comment thread WES_NOTES.txt Resolved
…ueArray, with nulls recorded by index

Closes eclipse-sparkplug#642 and eclipse-sparkplug#648, which are the same defect seen from two directions: a
packed array cannot express an absent element.

A DataSet.Column carried its values in a parallel array indexed by row, so a null
cell had nowhere to go - omitting it shifts every later row, and writing a zero is
indistinguishable from a real zero. Sparkplug 3 could express this, because its
row-major DataSetValue was present with its oneof unset. The array DataTypes had
the same problem and were additionally opaque in JSON, where an element array
appeared as base64.

The schema also held two conventions for an array of scalars: DataSet.Column used
native repeated fields while the array DataTypes used packed bytes. One ValueArray
message now serves both, with null_indexes recording the elements that are absent.

The signed arrays use zigzag. A negative value in a plain varint field occupies
ten bytes, so sint32 is about three times smaller than int32 on signed data; on
100 signed values in -500..500 the difference is 193 bytes against 607. DateTime
keeps a plain varint array, since its values are large positive nanosecond counts
for which zigzag only adds a bit.

The trade is deliberate rather than free. Packed varints beat the packed bytes
they replace on small magnitudes - 185 bytes against 403 for 100 values in
0..1000 - but DateTime arrays grow about 12%, because a nanosecond count needs a
nine byte varint. Accepting that buys null support in both places, one convention
instead of two, and array values that are readable in the JSON encoding rather
than base64, which is the reason the JSON encoding exists.

null_indexes costs nothing when an array has no nulls and a few bytes per null
otherwise: on a 100 element array, 0 nulls adds 0 bytes and 20 nulls adds 22.

In JSON a ValueArray is a plain array and an absent element is a JSON null, so
null_indexes has no JSON counterpart.

The TCK encoder is updated to the new shape, and a null DataSet cell now records
its row index rather than silently writing a zero.
…d arms, placeholders, array metrics

Addresses nathandavenport's review on eclipse-sparkplug#649.

- UInt32/UInt64 fallback branches wrote to the signed arrays
  (sint32_values/sint64_values). They now write uint32_values and
  uint64_values.
- addDataSetPlaceholder wrote its zero into a fixed array regardless of
  the column's datatype, so a null cell in an unsigned or DateTime
  column landed in the wrong array. It now picks the array that matches
  the datatype.
- The 13 array metric cases still encoded packed little-endian
  bytes_value. They now build a ValueArray and record null elements in
  null_indexes, as payloads-array-value-array requires.
- null_indexes MUST be in ascending order.
- "at most one" typed array rather than "exactly one": an empty array
  populates none of them.
- Three stale descriptions in the chapter: the DataSet example now
  shows "values": { "uint32_values": [...] }, the DateTime note points
  at int64_values rather than claiming it shares UInt64's array, and
  the JSON DataSet deviation is rewritten for the column shape.

@nathandavenport nathandavenport left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at fbefbee

All four required findings and the three optional ones are fixed, and so are the out-of-diff ones from the first review: array metrics now use array_value, and the stale int_values / long_values / "six parallel arrays" text is gone. I've resolved those threads.

Three small leftovers, all optional: the schema comment's "Exactly one", the Boolean column in the DataSet example, and the stale comment above addDataSetPlaceholder.
3 optional suggestion(s) are included. They are not required for approval.

…nts and one example column

- The ValueArray schema comment still said "exactly one" of the typed
  arrays is populated, contradicting both normative statements.
- The Enabled column of the DataSet example kept the old column-level
  "boolean_values" key while the other two columns moved to "values",
  so the example showed two different Column shapes.
- The comment above addDataSetPlaceholder still said a Column cannot
  express an absent cell, which null_indexes made untrue.
@wes-johnson
wes-johnson merged commit 83bf202 into eclipse-sparkplug:develop Nov 5, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants