Skip to content

[#641] Generate the TCK protobuf from the specification and migrate the payload layer to v4 - #647

Merged
wes-johnson merged 22 commits into
eclipse-sparkplug:developfrom
wes-johnson:tck-v4
Oct 29, 2026
Merged

wes-johnson merged 22 commits into
eclipse-sparkplug:developfrom
wes-johnson:tck-v4

Conversation

@wes-johnson

@wes-johnson wes-johnson commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Closes #641.

Stacked on #645. It is based on that branch because the payload layer it migrates is the one
#645 defines. It does not depend on #646 — verified by building this branch without it.

The TCK was built against Sparkplug 3. SparkplugBProto.java was checked in, generated by protoc
3.17, and had no Header, Content, DataSet.Column, descriptor_seq, definition_id, quality
or MetricOperation. No v4 payload requirement could be tested.

The protobuf is now generated, not checked in

A new extractProtoSchema task lifts the schema out of Sparkplug_6_Payloads.adoc into
build/proto/sparkplug_c.proto and publishes it as a resolvable artifact, the same pattern
tck-audit.xml already uses. The TCK consumes that artifact and generates from it, so the
specification text is the single source of truth and a schema change that breaks the TCK now fails
the build instead of going unnoticed for a major version.

Gradle fetches protoc itself, so no local protoc install is required.

The runtime bump could not be done separately

protobuf.version goes from 3.17.3 (June 2021) to 4.36.2, clearing CVE-2022-3171 and
CVE-2024-7254. This had to happen in the same change: the checked-in file was produced by protoc
3.17 and does not compile against a 4.x runtime —

error: name clash: clearExtension(GeneratedExtension<ParameterValueExtension,?>) in Builder and
clearExtension(...,T>) in ExtendableBuilder have the same erasure, yet neither overrides the other

Payload layer migration

Payload-level accessors to Header/Content 94 call sites, 11 files
Metric / Template.Parameter / PropertySet values into TypedValue 51 direct arms, 17 via new helpers
PropertyValue removed; PropertySet values are TypedValue encoder, Utils, tests
Encoder DataSet path transposed rows to columns per-cell converter replaced by one appending to the column array matching each datatype
datatype as the DataType enum, not an int 7 unwraps, 2 comparisons

v3 carried every 32-bit value in int_value and every 64-bit in long_value; v4 splits each into a
signed and an unsigned arm, so a check that used to be unambiguous now has two possible answers —
an Int64 or DateTime lands in int64_value, a UInt64 in uint64_value. Rather than guessing
per call site, the tests go through new Utils.hasLongValue / getLongValue / hasIntValue /
getIntValue helpers that accept either arm.

Sequence semantics were still Sparkplug 3

Independent of the protobuf, and arguably the more consequential half. Utils.getNextSeq asserted
seq <= 255 and seven range checks across five tests enforced the same bound, so a conformant v4
Edge Node whose seq passed 255 would have failed the TCK
.

These now go through Utils.isValidSeq, which documents why the test is what it is: the
specification allows 0 to 9223372036854775807 inclusive, and because the field decodes from an
unsigned 64-bit integer into a signed Java long, a value above the cap arrives negative — so testing
for a non-negative long accepts exactly the specified range.

EdgeNode.birthBdSeq also started at 255 to exercise rollover and would otherwise have silently
stopped testing anything.

Verification

./gradlew clean build passes. The generated SparkplugBProto.java is 23,384 lines and contains
every v4 construct the old copy lacked.

Addressed in review

  • A conformant Command metric failed conformance: expectedValueCase had no Command case, so
    hasValidDatatype and hasValue returned false for a Metric carrying its payload in the
    command field. A regression of this PR, since the old generated enum had no value 35.
  • A compressed payload made every metric check vacuous — payload_content is a oneof, so
    getContent() returned the default instance and every metric loop saw an empty list. decode()
    now rejects and logs it rather than reporting silent passes.
  • The Monitor sequence range check could never fail; the FAIL was overwritten by an
    unconditional PASS on the next line.
  • The Template.Parameter value check omitted bytes_value and ignored is_null; it now calls
    Utils.hasValue(Parameter), which previously had no caller outside a debug log.

Not addressed

  • The spBv1.0 namespace constant, 81 call sites. With [#231] Define the spJv4.0 JSON payload encoding #645 there are two v4 namespaces, so this is
    not a rename — each call site has to know which encoding it exercises.
  • Coverage for the assertions [#231] Define the spJv4.0 JSON payload encoding #645 adds, and for the DataSet assertions that replaced the ones
    [#632] Rewrite the DataSet prose against the current schema #644 removed.
  • bd_seq is still read only from the v3 bdSeq metric (Monitor, SessionEstablishmentTest,
    PrimaryHostTest, ReceiveCommandTest), and the simulator does not write header.bd_seq. A v4 Edge
    Node that carries bd_seq only in the Header gets no bdSeq checks. The read sites are nested
    inside metric loops and the Tahu payload model has no bd_seq field, so both halves are a change
    of their own rather than a mechanical edit.
  • Decompressing a compressed payload. The TCK now flags one instead of inspecting it.
  • A unit-test source set for Utils. The recursion and the regressions this PR needed review
    to find would all have been caught by one.
  • SendComplexDataTest.checkDataSet stays removed, as in [#632] Rewrite the DataSet prose against the current schema #644.

Note for coverage audits

Grepping the TCK for a hyphenated assertion id finds only Requirements.java, because the Java uses
the generated uppercase constants (ID_PAYLOADS_DATASET_TYPES_DEF, not payloads-dataset-types-def).
Searching for the id alone makes a tested assertion look untested.

Adds a normative JSON encoding of the Sparkplug payload model, published on the
spJv4.0 namespace alongside the Protocol Buffers encoding on spCv4.0.

The mapping is derived from the Google Protocol Buffers JSON Mapping so that most
of it can be produced by library facilities, with Sparkplug's deviations stated
explicitly: TypedValue is flattened to a bare 'value', PropertySet is a JSON
object keyed by property name, array DataTypes are JSON arrays rather than base64
packed bytes, and a DataSet.Column carries one 'values' array.

64-bit integers are encoded as JSON strings. A v4 nanosecond timestamp is roughly
165 times larger than the 2^53 limit a JSON number is guaranteed to carry, so a
numeric form is silently rounded by any parser that decodes into a double.

Also generalizes three Ch. 5 statements that required a Protobuf encoded payload,
notes that topics written with spCv4.0 apply equally to spJv4.0, and renames the
'dataType' key in the Ch. 6 examples to 'datatype' and the metric-level 'template'
key to 'value' so the examples match the encoding rules.
…ing-encoded 64-bit values

The note claimed a Sparkplug sequence number is always small, which is wrong: seq,
bd_seq, and descriptor_seq each count to the maximum value of their declared type
before rolling over to zero. They are therefore subject to the same precision loss
as timestamps, and a numeric form would be correct early in a session and lossy
late in one.
The three message sequence numbers move from uint64 to int64 in the Protocol
Buffers schema, with the range and rollover point changing from
18446744073709551615 to 9223372036854775807 across all 38 statements that bound
them in Chapters 4, 5, and 6.

Protobuf uint64 is a signed long in Java, so the upper half of the uint64 range
was never reachable by an implementation built on it. Declaring int64 makes the
specified range the representable one.

MetaData.seq, which sequences the parts of a multi-part file transfer, is a
different field and is unchanged.
… rsn arithmetic

Header.timestamp and Metric.timestamp move from uint64 to int64, for the same
reason as the sequence numbers: protobuf uint64 is a signed long in Java, so the
upper half of the unsigned range was never reachable. The range note in Chapter 5
is rewritten accordingly - the ceiling is 2262-04-11, stated directly rather than
as a caveat on an unsigned range that no implementation could use.

The NREBIRTH rsn calculation said to add the maximum sequence number to a negative
result. That was off by one, since a counter cycling 0..MAX has a period of MAX+1,
and MAX+1 is not representable as an int64. Restated as two cases so no
intermediate is ever negative or out of range:

  if pds >= 1 + dc:  rsn = pds - (1 + dc)
  otherwise:         rsn = 9223372036854775807 - (dc - pds)

Verified equivalent to (pds - (1 + dc)) mod 2^63 over the full range.
…ing seq as unsigned

The seq, bd_seq, and descriptor_seq bullets in the Header section still described
each field as an unsigned 64-bit integer after the schema moved them to int64.
…ormance profiles

An Edge Node MUST support at least one encoding and MAY support both, selected by
configuration, but MUST use a single encoding and namespace for the whole of an
MQTT session - seq is Edge-Node-wide and bd_seq scopes the session, so splitting
one session across both namespaces would produce two unusable halves of one
sequence stream.

Host Applications and Sparkplug Aware MQTT Servers MUST support both encodings.
A Host Application cannot choose the encoding of the Edge Nodes it consumes, and
an Aware Server decodes payloads rather than treating them as opaque.

A Sparkplug Compliant MQTT Server has no payload requirements at all, which is now
stated non-normatively so the asymmetry with the Aware profile is not read as an
omission.

Also generalizes the Edge Node profile's reference to the spCv4.0/# namespace.
…g the capped range

seq, bd_seq, and descriptor_seq go back to uint64 while keeping the rollover point
at 9223372036854775807. timestamp stays int64.

An unsigned type is right for a counter that can never be negative, and it keeps
the three consistent with alias, MetaData.size, and Status.code. Capping the range
below the type's maximum still guarantees every value a conformant Edge Node can
produce fits a signed 64-bit integer, so an implementation in a language without
an unsigned 64-bit type never needs unsigned comparison. Neither bound is
reachable: at a million messages per second the capped range lasts ~292,000 years.

timestamp keeps int64 because every mainstream time representation it interoperates
with is signed - protobuf's own Timestamp, OPC UA DateTime, java.time.Instant - and
the 2262 ceiling it accepts is one none of those exceed either.

The repeated "(max value of a UINT64)" parenthetical is dropped rather than
rescaled, since the cap is now a deliberate restriction rather than the type's
maximum. It is explained once in the Header section instead.
…ary-support claim

The oneof was named 'content' and its uncompressed arm was named 'plain', so a
Protocol Buffers JSON implementation emitted the key "plain" while every example
in the chapter used "content". A field may not share the name of its containing
oneof, so the oneof is renamed to payload_content and the arm becomes 'content'.
The examples were right and the schema now agrees with them, and a stock library
emits the correct key with no post-processing. Three prose references to the
'plain' field are updated.

The non-normative comment claimed an implementation could produce most of the JSON
encoding with library facilities. Measured against the compiled schema, that is
not true: the structural mappings reshape every Metric, so an encoder writes its
own traversal. What the derivation does settle is the scalar rules and the header,
which the comment now says instead.
…er extension fields

The section leaned on the Google Protocol Buffers JSON Mapping for part of its
meaning. That mapping is vendor documentation rather than a versioned standard,
it is specified for proto3 while the Sparkplug schema is proto2, and it does not
cover extensions at all. The section now states its rules itself and says so.

- The lowerCamelCase form a consumer must accept is defined here - remove each
  underscore and upper case the letter after it - so the assertion no longer
  depends on an external document for its meaning.
- New Extension Fields section: an extension is keyed by its fully qualified name
  in square brackets, its value follows the ordinary rules for its type, and an
  unrecognized extension is ignored. The Sparkplug schema reserves an extension
  range in most messages, so leaving this undefined left a field that is
  well-defined on the wire undefined in JSON. Includes a note that unlike the
  binary encoding, where an unknown field is retained and re-emitted, an
  unrecognized JSON key does not survive a round trip.
- Floating point values must be written with enough digits to parse back exactly:
  up to 9 significant decimal digits for a float and 17 for a double.
…e change time

Metrics in a payload are ordered oldest first, so a consuming application that
applies them in array order arrives at the correct current value without
comparing timestamps. This matters when one payload carries several values for
the same metric, as a payload of historical values does. Metrics remain
identified by name or alias, so position still carries no identity.

A metric that omits its timestamp sorts as though its value change time were the
payload timestamp, which is the time it is already taken to have changed.

Metrics that share a value change time may appear in any order and a consumer
must not reorder them. Multi-part file publishing depends on that: it identifies
a metric by position, and every metric of a multi-part publish carries the same
timestamp, so they tie and the publisher's order is carried unchanged by both
encodings. Non-normative comments in both sections record the interaction.

Also splits the JSON PropertySet statement. Uniqueness of property names is
observable in a payload and stays testable; the absence of meaning in key order
is not observable and becomes non-testable, with a note that RFC 8259 defines a
JSON object as unordered so the requirement is on the publisher.
1. The conformance note justifying the session scoping rule was denser than the
   rule it explained. Reduced to the plain consequence: an Edge Node has one
   sequence number stream, so splitting a session across namespaces leaves a
   consumer of either with gaps in it.

2. Removed the requirement that a consuming application accept the lowerCamelCase
   form of a field name. A conformant publisher emits the Protocol Buffers field
   name, so the rule required consumers to accept a form no conformant publisher
   may produce - specifying leniency toward non-conformance and creating a second
   de facto wire form for every field. Replaced with implementer guidance: a
   Protocol Buffers JSON library converts to lowerCamelCase by default and must be
   configured to preserve the original names.

3. Explained why an absent field is omitted rather than written as null or zero.
   Zero is a legitimate value for seq, bd_seq, descriptor_seq, alias, and quality,
   so writing an absent field as 0 would be indistinguishable from one set to
   zero; and null would be a second, conflicting way to express what is_null
   already expresses.

4. Scoped the ignore-unknown rule to the names of fields within a message. It was
   open to being read as covering an unrecognized metric, which is a different
   situation entirely - a mismatch against the metric structure the application
   holds, resolved by the metadata rules and a Rebirth Request rather than by
   ignoring it. Both cases are now stated and contrasted.
A DateTime value is a nanosecond count since epoch, the same quantity the payload
and metric timestamps carry, so it follows them to int64 rather than staying
unsigned. Leaving it as uint64 meant a DateTime value and a payload timestamp
differed in signedness despite meaning the same thing.

This moves a DateTime value from the uint64_value arm of TypedValue to
int64_value, so unlike the timestamp field changes it is a wire change. A DateTime
in a DataSet.Column is unaffected: there is no signed repeated field, so it stays
in long_values alongside UInt64 and is carried as its two's complement
representation, which the DataSet.Column section already requires for signed
values.
…PropertySet TypedValues

Review found two defects in the normative JSON Schema and its prose.

nonNegative64 was "type": ["string","integer"] with "pattern": "^[0-9]+$". JSON
Schema applies pattern only to strings, so a negative JSON number validated
cleanly, as did a string far beyond the 64-bit cap. That governs seq, bd_seq,
descriptor_seq, timestamp, alias, MetaData size and seq, and Status code - the
fields whose range the rest of the specification relies on. Now bounded on both
arms. The earlier negative tests missed this because they used the string form.

The TypedValue flattening rule named only Metric and Template.Parameter as
datatype carriers, but PropertySet.values is a repeated TypedValue, so a property
value had no enclosing field to carry its datatype and could not be encoded by
following the rule. The JSON Schema already resolved this the other way, nesting
an object with its own datatype - which the rule forbade. The rule is now scoped
to Metric and Template.Parameter, and the PropertySet case is stated to match the
schema. Also corrects a reference to PropertyValues, which is not a v4 message.

@ckienle ckienle left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the change at 9d1baba (the 6 commits over #645). One required item; optionals inline. Static review: I couldn't run the Gradle build here, so I'm relying on CI being green for compilation; the TCK has no unit tests, so I couldn't mutation-check anything.

Checked and fine: extractProtoSchema fails loudly on a missing heading/listing/wrong block, handles CRLF, and declares inputs/outputs; protoc and protobuf-java share protobuf.version; isValidSeq (seq >= 0) is exactly 0..2^63-1 for a decoded uint64, getNextSeq rolls MAX_SEQ to 0, and the recursion fix in 80e60e6 is right; the EdgeNode simulator counters wrap correctly; DateTime is in int64_value in all three encoder arms; the DataSet encoder transposes correctly and the null placeholder keeps columns the same length.

Other points:

  • bd_seq is still read only from the v3 bdSeq metric (Monitor :383, SessionEstablishmentTest :469/:640, PrimaryHostTest, ReceiveCommandTest), and the simulator (EdgeNode.java:267/325) never writes header.bd_seq. A v4 Edge Node that sends bd_seq only in the Header gets no bdSeq checks at all. In scope for "migrate the payload layer to v4"; if deferred, please list it under "Not addressed".
  • A compressed payload makes every metric check vacuous (Utils.java:145, and every getContent().getMetricsList()): payload_content is a oneof, so for an Edge Node that sends compressed, getContent() returns the default instance. Every metric loop then sees an empty list, the bdSeq lookups find nothing, and BIRTH-metric assertions are never set or set wrongly. Suggest decompressing in decode(), or failing/flagging the test when getPayloadContentCase() == COMPRESSED; at minimum add it to "Not addressed".
  • A small unit-test source set for Utils (isValidSeq, getNextSeq, expectedValueCase, hasValue) would have caught the recursion and the three regressions this PR has already needed review to find.
  • #646 and #647 merge cleanly and #646 keeps the enums CLOSED, but nobody has compiled the two together yet; worth doing before the second one merges.
  • The description is stale: "a UInt64 or DateTime in uint64_value"; since 453f98c DateTime goes in int64_value.
  • Nit: Constants.MAX_SEQ could be Long.MAX_VALUE; in EdgeNode the > MAX_SEQ checks can never be true, only < 0 does anything.

case PropertySetList:
return Payload.TypedValue.ValueCase.PROPERTYSETS_VALUE;
default:
return Payload.TypedValue.ValueCase.VALUE_NOT_SET;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Required: a conformant Command metric now fails ID_PAYLOADS_METRIC_DATATYPE_VALUE_TYPE / ..._VALUE (via SendComplexDataTest.java:370). expectedValueCase(Command) falls through to this default, so hasValidDatatype returns false for a Metric with datatype = Command (35) and command set, which the spec requires for Command definitions in NBIRTH/DBIRTH.

It's a regression: the old generated enum had no 35, so DataType.forNumber(35) was null and the metric was "assumed valid".

Same class: hasValue(Metric) (:268) is false for a Command metric, so TOPICS_NBIRTH_METRICS and OPERATIONAL_BEHAVIOR_DATA_PUBLISH_NBIRTH_VALUES fail too (SessionEstablishmentTest :651/:656, :891/:904). That part predates this PR, but the generated proto now has hasCommand(), so this is where it can be fixed.

Suggest special-casing Command in both: valid iff m.hasCommand() and no value.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c871341. Confirmed exactly as you describe — expectedValueCase had no Command case, so it fell through to VALUE_NOT_SET and hasValidDatatype returned false for a Metric with datatype = Command and command set. And you're right that it is a regression of this PR rather than pre-existing: the old generated enum had no value 35, so DataType.forNumber(35) was null and the metric took the "assumed valid" path.

Special-cased in both, as suggested:

if (m.getDatatype() == DataType.Command) {
    // carries its definition or invocation in the command field, not a TypedValue
    return m.hasCommand() && !m.hasValue();
}

and hasValue(Metric) returns m.hasCommand() for a Command metric.

Verified by execution rather than compilation: a Command definition with arguments and no value passes both checks, and a Command metric with neither command nor value is rejected.

if (payload.hasSeq()) {
if (payload.getSeq() > 255 || payload.getSeq() < 0) {
if (payload.getHeader().hasSeq()) {
if (!Utils.isValidSeq(payload.getHeader().getSeq())) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: this range check can never fail: FAIL is put when !isValidSeq, then immediately overwritten with PASS at :773 because there's no else. Pre-existing, but this PR rewrote the condition and the description presents it as enforcing the v4 bound. Put PASS in an else.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c871341. You're right that it could never fail — the FAIL was written, then overwritten unconditionally by the PASS on the next line. The PASS is now in an else, and the edgeSeqs.put moved below so it still runs either way.

Fair point that this PR rewrote the condition and the description claimed it enforced the v4 bound, so presenting it as working while it could not fail was misleading regardless of it being pre-existing.

if (!p.hasIntValue() && !p.hasLongValue() && !p.hasFloatValue() && !p.hasDoubleValue()
&& !p.hasBooleanValue() && !p.hasStringValue()) {
final Payload.TypedValue pv = p.getTypedValue();
if (!pv.hasInt32Value() && !pv.hasUint32Value() && !pv.hasInt64Value()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: this parameter value check misses bytes_value (the spec allows bytes for a parameter value, and the arm is now visible in the generated proto) and ignores is_null.

The 9d1baba commit message says the null case "is now handled on the Template.Parameter overload", but Utils.hasValue(Parameter) (Utils.java:277) has no callers, so that fix doesn't take effect. Suggest calling it here and adding hasBytesValue(), or deleting the dead overload.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c871341, and you're right on both counts — including that my commit message claimed a fix that could not take effect.

The hand-written arm list is replaced by a call to Utils.hasValue(Parameter), which covers bytes_value through the generic getValueCase() != VALUE_NOT_SET test and handles is_null explicitly. That also gives the overload a real caller; you were correct that its only other reference was inside a logger.debug, so the is_null handling I added in 9d1baba was dead until now.

…essing, and schema bounds

Required:

- The namespace substitution note made STATE ambiguous. STATE is pinned to
  spCv4.0/STATE/sparkplug_host_id, so a JSON Edge Node following the note would
  have waited on spJv4.0/STATE while the Host published on spCv4.0/STATE and never
  seen it come online. STATE is now stated as the one exception, with the reason.

- Nothing said which namespace a Host Application uses to address a particular
  Edge Node. A Host supports both encodings, so it could send an NCMD or a Rebirth
  Request on a namespace the Edge Node does not subscribe to; the MQTT Server
  accepts it and it is never delivered, so Rebirth recovery fails silently. A Host
  now publishes to an Edge Node on the namespace of that Edge Node's NBIRTH, and
  an Edge Node subscribes to its command topics on the namespace it publishes on.

Also:

- nonNegative64 capped every 64-bit field at 2^63-1, but alias, MetaData.size and
  Status.code are uncapped uint64, so a legal spCv4.0 payload could not be
  expressed as a schema-valid spJv4.0 one - contradicting the lossless conversion
  this section claims. Split into uint64 and sequence64, with the string branch of
  sequence64 bounded by length and its limit stated.
- The Sparkplug Aware MQTT Server encoding requirement is now conditional on
  replacing NDEATH timestamps, which is the only payload-touching behavior
  required of it and is itself a MAY.
- Flattening drops the TypedValue's own datatype and is_null. The two copies must
  now agree in the Protocol Buffers encoding, the single JSON key represents both,
  and a null Template.Parameter has a JSON form: is_null is added to the parameter
  keys and to the schema.
- The metric ordering MUST is scoped to payloads carrying more than one value for
  the same metric, with a SHOULD elsewhere so a BIRTH in browse order stays
  conformant, and it now says it does not reach Template or Command metrics.
- Nits: INT64 rollover wording, a stale (max value of an INT64), DateTimeArray
  elements described as signed, and a MUST NOT inside a non-normative comment.
@wes-johnson

Copy link
Copy Markdown
Contributor Author

All three inline items addressed in c871341; replies on each thread. On the other points from your review body:

Fixed — compressed payloads made every metric check vacuous. You were right and I had not considered it. payload_content is a oneof, so for an Edge Node sending compressed, getContent() returns the default instance and every metric loop sees an empty list. Utils.decode() now rejects a compressed payload and logs it, rather than letting the suite report passes it never actually checked. Decompressing is listed under "Not addressed".

Fixed — Constants.MAX_SEQ is now Long.MAX_VALUE. You were also right that the > MAX_SEQ comparisons in EdgeNode can never be true; only the < 0 arm does anything. Left in place as a guard that reads correctly, but noted.

Deferred, and now listed under "Not addressed" — bd_seq from the Header. Agreed this is in scope for the title. I started it and backed it out: the read sites are nested inside metric loops with their own increment logic, and the Tahu payload model has no bd_seq field at all, so the simulator cannot write header.bd_seq without extending the model and the encoder. Both halves are a change of their own. I removed the half-finished helper rather than leave an unused one — the same dead-code problem you caught on hasValue(Parameter).

Agreed on the unit tests. A small source set over isValidSeq, getNextSeq, expectedValueCase and hasValue would have caught the recursion and all three regressions. In the meantime I have been exercising these by compiling a throwaway main against the built classes, which is how the Command fix and the DataSet null placeholder were verified — but that is not a substitute. Note #631 adds tck/src/test/ for the 3.x branch, so there is a pattern to copy.

On compiling #646 and #647 together — done. They are independent: #647 builds green on #645 alone, without the editions change, which is why they are separate PRs rather than a chain.

Description updated, including the stale "a UInt64 or DateTime in uint64_value" you caught; DateTime has been in int64_value since 453f98c.

@ckienle ckienle left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at c871341. All fixed as described: the Command datatype special case in hasValidDatatype and hasValue, the else on the NBIRTH seq check, the parameter check now going through Utils.hasValue(Parameter) (bytes and is_null covered), and decode() rejecting a compressed payload. Its callers already treat null as unparseable, so that's now a reported failure rather than a silent pass. Deferring Header bd_seq with it listed under "Not addressed" is reasonable given the Tahu model has no field for it. CI is green. Approving.

…gth-bounded string form

MetaData.seq sequences the parts of a multi-part file transfer and is an uncapped
uint64 in both the schema and the prose. The earlier split pointed it at
sequence64 because the bulk rename matched on the field name, which the two
fields share, so it was capped at 2^63-1 by accident - the same class of mistake
as the regex that produced the recursion.

Also gives the uint64 definition the same note sequence64 carries, since its
string branch is bounded by length rather than by value, and re-wraps the
DateTimeArray entry left split mid-phrase by the earlier edit.
…ation as a build artifact

Adds an extractProtoSchema task that lifts the Protocol Buffers schema out of the
Payloads chapter into build/proto/sparkplug_c.proto, and publishes it as a
resolvable artifact the same way tck-audit.xml already is.

This is the prerequisite for generating SparkplugBProto.java rather than checking
it in, which is how the TCK's copy drifted a full major version behind the
specification. With the schema published as an artifact, the specification text
stays its single source of truth.

The TCK is not switched over here - that needs the payload layer migrated to the
v4 message shape, which is the rest of eclipse-sparkplug#641.
…load layer to v4

SparkplugBProto.java is no longer checked in. It is generated from the schema the
specification publishes, so it cannot drift from the spec the way the old copy
did - that copy was the Sparkplug 3 protobuf, with no Header, Content,
DataSet.Column, descriptor_seq or quality.

Bumps protobuf-java from 3.17.3 (June 2021) to 4.29.3. The two changes are
inseparable: the checked-in file was produced by protoc 3.17 and its generated
code contract does not compile against a 4.x runtime, so the CVE exposure in
3.17.3 could not be cleared without regenerating.

The payload layer is migrated to the v4 message shape:

- Payload level timestamp, seq, uuid and body move to Header and Content, across
  94 call sites.
- Metric, Template.Parameter and PropertySet values move into TypedValue. v3's
  int_value and long_value each split into a signed and an unsigned arm, so the
  tests go through new Utils helpers that accept either rather than guessing.
- PropertyValue is gone; PropertySet values are TypedValues.
- The encoder's DataSet path is transposed from rows to columns, and the per-cell
  DataSetValue converter becomes one that appends to the column array matching
  each column's datatype.
- datatype fields are the DataType enum rather than an int.

Sequence semantics are brought to v4 as well. The TCK still enforced Sparkplug 3's
0-255 range in Utils.getNextSeq and in seven range checks across five tests, so a
conformant v4 Edge Node whose seq exceeded 255 would have failed. The bound is now
the capped rollover point, as Constants.MAX_SEQ.

SendComplexDataTest.checkDataSet is removed, as in eclipse-sparkplug#644. It tested the row
oriented DataSet through getNumOfColumns() and getTypesList(); re-adding coverage
against the replacement assertions is follow-up work.
…ve to the current runtime

The 0-255 checks were replaced with a bare non-negative test, which is correct but
reads as though the bound was simply deleted. Replaced with Utils.isValidSeq,
which documents why: the specification allows 0 to 9223372036854775807 inclusive,
and because the field decodes from an unsigned 64 bit integer into a signed Java
long, a value above that cap arrives negative. Testing for a non-negative long
therefore accepts exactly the range the specification allows. Seven call sites
across five tests now go through it.

Bumps protobuf from 4.29.3 to 4.36.2, the current release.
DateTime moves from uint64_value to int64_value in TypedValue, matching the
specification change. Updates Utils.expectedValueCase and the three encoder arms
that wrote a Date into the unsigned field.

A DateTime in a DataSet.Column is unchanged. There is no signed repeated field, so
it stays in long_values and is carried as its two's complement representation.
The regex that routed the sequence checks through the new helper also rewrote the
helper's own body, leaving `return Utils.isValidSeq(seq);`. It compiles, so the
build stayed green, but every sequence-number check would have recursed until the
thread died with a StackOverflowError - and the TCK has no unit tests to catch it.

Reached from Monitor, SessionEstablishmentTest, SendDataTest, SendComplexDataTest
and SessionTerminationTest, so any Edge Node publishing an NBIRTH would have
crashed the handler, and the 0..9223372036854775807 bound was never enforced.

Verified by execution rather than compilation this time: isValidSeq returns the
expected answer for 0, 1, 255, 256, MAX-1, MAX, -1 and Long.MIN_VALUE, and
getNextSeq rolls MAX over to 0 while 255 now advances to 256.
…payload migration

hasValue(Metric) lost its is_null branch in the rewrite. A conformant BIRTH metric
published with is_null true and no TypedValue - which the specification endorses
and this encoder itself emits - reported as having no value, failing
topics-nbirth-metrics, topics-dbirth-metrics and the data-publish BIRTH value
checks against a compliant Edge Node. The same case is now handled on the
Template.Parameter overload, where v4's TypedValue carries is_null too.

EdgeNode kept two counters on the Sparkplug 3 range after birthBdSeq was migrated.
deathBdSeq still rolled at 256, so after 256 reconnects the NDEATH bd_seq diverged
from the NBIRTH bd_seq the Will Message registered, and the reference Edge Node
violated its own bd_seq session-constant requirement. getNextSeqNum still rolled
the payload sequence at 256, so from the 257th message the Monitor failed sequence
increment checks on payloads the TCK itself produced.

The DataSet encoder skipped a null cell rather than emitting one, which shortens
that column and shifts every later row. A v4 Column is a parallel array indexed by
row and cannot express an absent cell, so a null now holds its position with the
zero value for the column's datatype, keeping every Column the same length as
payloads-dataset-row-correlation requires.
…ayloads, and a vacuous check

Required: a conformant Command metric failed conformance. expectedValueCase has
no Command case, so it fell through to VALUE_NOT_SET and hasValidDatatype
returned false for a Metric with datatype Command and the command field set,
which is how a Command definition appears in an NBIRTH or DBIRTH. It is a
regression of this PR: the old generated enum had no value 35, so forNumber(35)
was null and the metric was assumed valid. hasValue had the same gap. Both now
treat a Command metric as carrying its payload in the command field.

Also:

- A compressed payload made every metric check vacuous. payload_content is a
  oneof, so getContent() returns the default instance and every metric loop saw
  an empty list, with BIRTH assertions never set or set wrongly. decode() now
  rejects a compressed payload and logs it rather than reporting silent passes.
  Decompressing is left for the follow-up work listed in the description.
- The Monitor sequence range check could never fail: the FAIL was overwritten by
  an unconditional PASS on the next line. The PASS is now in an else.
- The Template.Parameter value check went through a hand-written list of value
  arms that omitted bytes, which the specification allows, and ignored is_null.
  It now calls Utils.hasValue(Parameter), which also gives that overload a real
  caller rather than leaving it reachable only from a debug log.
- Constants.MAX_SEQ is Long.MAX_VALUE, which is what it means.
@wes-johnson
wes-johnson merged commit 10bba93 into eclipse-sparkplug:develop Oct 29, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants