Repository navigation
[#641] Generate the TCK protobuf from the specification and migrate the payload layer to v4 - #647
Conversation
Adds a normative JSON encoding of the Sparkplug payload model, published on the spJv4.0 namespace alongside the Protocol Buffers encoding on spCv4.0. The mapping is derived from the Google Protocol Buffers JSON Mapping so that most of it can be produced by library facilities, with Sparkplug's deviations stated explicitly: TypedValue is flattened to a bare 'value', PropertySet is a JSON object keyed by property name, array DataTypes are JSON arrays rather than base64 packed bytes, and a DataSet.Column carries one 'values' array. 64-bit integers are encoded as JSON strings. A v4 nanosecond timestamp is roughly 165 times larger than the 2^53 limit a JSON number is guaranteed to carry, so a numeric form is silently rounded by any parser that decodes into a double. Also generalizes three Ch. 5 statements that required a Protobuf encoded payload, notes that topics written with spCv4.0 apply equally to spJv4.0, and renames the 'dataType' key in the Ch. 6 examples to 'datatype' and the metric-level 'template' key to 'value' so the examples match the encoding rules.
…ing-encoded 64-bit values The note claimed a Sparkplug sequence number is always small, which is wrong: seq, bd_seq, and descriptor_seq each count to the maximum value of their declared type before rolling over to zero. They are therefore subject to the same precision loss as timestamps, and a numeric form would be correct early in a session and lossy late in one.
The three message sequence numbers move from uint64 to int64 in the Protocol Buffers schema, with the range and rollover point changing from 18446744073709551615 to 9223372036854775807 across all 38 statements that bound them in Chapters 4, 5, and 6. Protobuf uint64 is a signed long in Java, so the upper half of the uint64 range was never reachable by an implementation built on it. Declaring int64 makes the specified range the representable one. MetaData.seq, which sequences the parts of a multi-part file transfer, is a different field and is unchanged.
… rsn arithmetic Header.timestamp and Metric.timestamp move from uint64 to int64, for the same reason as the sequence numbers: protobuf uint64 is a signed long in Java, so the upper half of the unsigned range was never reachable. The range note in Chapter 5 is rewritten accordingly - the ceiling is 2262-04-11, stated directly rather than as a caveat on an unsigned range that no implementation could use. The NREBIRTH rsn calculation said to add the maximum sequence number to a negative result. That was off by one, since a counter cycling 0..MAX has a period of MAX+1, and MAX+1 is not representable as an int64. Restated as two cases so no intermediate is ever negative or out of range: if pds >= 1 + dc: rsn = pds - (1 + dc) otherwise: rsn = 9223372036854775807 - (dc - pds) Verified equivalent to (pds - (1 + dc)) mod 2^63 over the full range.
…ing seq as unsigned The seq, bd_seq, and descriptor_seq bullets in the Header section still described each field as an unsigned 64-bit integer after the schema moved them to int64.
…ormance profiles An Edge Node MUST support at least one encoding and MAY support both, selected by configuration, but MUST use a single encoding and namespace for the whole of an MQTT session - seq is Edge-Node-wide and bd_seq scopes the session, so splitting one session across both namespaces would produce two unusable halves of one sequence stream. Host Applications and Sparkplug Aware MQTT Servers MUST support both encodings. A Host Application cannot choose the encoding of the Edge Nodes it consumes, and an Aware Server decodes payloads rather than treating them as opaque. A Sparkplug Compliant MQTT Server has no payload requirements at all, which is now stated non-normatively so the asymmetry with the Aware profile is not read as an omission. Also generalizes the Edge Node profile's reference to the spCv4.0/# namespace.
…g the capped range seq, bd_seq, and descriptor_seq go back to uint64 while keeping the rollover point at 9223372036854775807. timestamp stays int64. An unsigned type is right for a counter that can never be negative, and it keeps the three consistent with alias, MetaData.size, and Status.code. Capping the range below the type's maximum still guarantees every value a conformant Edge Node can produce fits a signed 64-bit integer, so an implementation in a language without an unsigned 64-bit type never needs unsigned comparison. Neither bound is reachable: at a million messages per second the capped range lasts ~292,000 years. timestamp keeps int64 because every mainstream time representation it interoperates with is signed - protobuf's own Timestamp, OPC UA DateTime, java.time.Instant - and the 2262 ceiling it accepts is one none of those exceed either. The repeated "(max value of a UINT64)" parenthetical is dropped rather than rescaled, since the cap is now a deliberate restriction rather than the type's maximum. It is explained once in the Header section instead.
…ary-support claim The oneof was named 'content' and its uncompressed arm was named 'plain', so a Protocol Buffers JSON implementation emitted the key "plain" while every example in the chapter used "content". A field may not share the name of its containing oneof, so the oneof is renamed to payload_content and the arm becomes 'content'. The examples were right and the schema now agrees with them, and a stock library emits the correct key with no post-processing. Three prose references to the 'plain' field are updated. The non-normative comment claimed an implementation could produce most of the JSON encoding with library facilities. Measured against the compiled schema, that is not true: the structural mappings reshape every Metric, so an encoder writes its own traversal. What the derivation does settle is the scalar rules and the header, which the comment now says instead.
…er extension fields The section leaned on the Google Protocol Buffers JSON Mapping for part of its meaning. That mapping is vendor documentation rather than a versioned standard, it is specified for proto3 while the Sparkplug schema is proto2, and it does not cover extensions at all. The section now states its rules itself and says so. - The lowerCamelCase form a consumer must accept is defined here - remove each underscore and upper case the letter after it - so the assertion no longer depends on an external document for its meaning. - New Extension Fields section: an extension is keyed by its fully qualified name in square brackets, its value follows the ordinary rules for its type, and an unrecognized extension is ignored. The Sparkplug schema reserves an extension range in most messages, so leaving this undefined left a field that is well-defined on the wire undefined in JSON. Includes a note that unlike the binary encoding, where an unknown field is retained and re-emitted, an unrecognized JSON key does not survive a round trip. - Floating point values must be written with enough digits to parse back exactly: up to 9 significant decimal digits for a float and 17 for a double.
…e change time Metrics in a payload are ordered oldest first, so a consuming application that applies them in array order arrives at the correct current value without comparing timestamps. This matters when one payload carries several values for the same metric, as a payload of historical values does. Metrics remain identified by name or alias, so position still carries no identity. A metric that omits its timestamp sorts as though its value change time were the payload timestamp, which is the time it is already taken to have changed. Metrics that share a value change time may appear in any order and a consumer must not reorder them. Multi-part file publishing depends on that: it identifies a metric by position, and every metric of a multi-part publish carries the same timestamp, so they tie and the publisher's order is carried unchanged by both encodings. Non-normative comments in both sections record the interaction. Also splits the JSON PropertySet statement. Uniqueness of property names is observable in a payload and stays testable; the absence of meaning in key order is not observable and becomes non-testable, with a note that RFC 8259 defines a JSON object as unordered so the requirement is on the publisher.
1. The conformance note justifying the session scoping rule was denser than the rule it explained. Reduced to the plain consequence: an Edge Node has one sequence number stream, so splitting a session across namespaces leaves a consumer of either with gaps in it. 2. Removed the requirement that a consuming application accept the lowerCamelCase form of a field name. A conformant publisher emits the Protocol Buffers field name, so the rule required consumers to accept a form no conformant publisher may produce - specifying leniency toward non-conformance and creating a second de facto wire form for every field. Replaced with implementer guidance: a Protocol Buffers JSON library converts to lowerCamelCase by default and must be configured to preserve the original names. 3. Explained why an absent field is omitted rather than written as null or zero. Zero is a legitimate value for seq, bd_seq, descriptor_seq, alias, and quality, so writing an absent field as 0 would be indistinguishable from one set to zero; and null would be a second, conflicting way to express what is_null already expresses. 4. Scoped the ignore-unknown rule to the names of fields within a message. It was open to being read as covering an unrecognized metric, which is a different situation entirely - a mismatch against the metric structure the application holds, resolved by the metadata rules and a Rebirth Request rather than by ignoring it. Both cases are now stated and contrasted.
A DateTime value is a nanosecond count since epoch, the same quantity the payload and metric timestamps carry, so it follows them to int64 rather than staying unsigned. Leaving it as uint64 meant a DateTime value and a payload timestamp differed in signedness despite meaning the same thing. This moves a DateTime value from the uint64_value arm of TypedValue to int64_value, so unlike the timestamp field changes it is a wire change. A DateTime in a DataSet.Column is unaffected: there is no signed repeated field, so it stays in long_values alongside UInt64 and is carried as its two's complement representation, which the DataSet.Column section already requires for signed values.
…PropertySet TypedValues Review found two defects in the normative JSON Schema and its prose. nonNegative64 was "type": ["string","integer"] with "pattern": "^[0-9]+$". JSON Schema applies pattern only to strings, so a negative JSON number validated cleanly, as did a string far beyond the 64-bit cap. That governs seq, bd_seq, descriptor_seq, timestamp, alias, MetaData size and seq, and Status code - the fields whose range the rest of the specification relies on. Now bounded on both arms. The earlier negative tests missed this because they used the string form. The TypedValue flattening rule named only Metric and Template.Parameter as datatype carriers, but PropertySet.values is a repeated TypedValue, so a property value had no enclosing field to carry its datatype and could not be encoded by following the rule. The JSON Schema already resolved this the other way, nesting an object with its own datatype - which the rule forbade. The rule is now scoped to Metric and Template.Parameter, and the PropertySet case is stated to match the schema. Also corrects a reference to PropertyValues, which is not a v4 message.
ckienle
left a comment
There was a problem hiding this comment.
Reviewed the change at 9d1baba (the 6 commits over #645). One required item; optionals inline. Static review: I couldn't run the Gradle build here, so I'm relying on CI being green for compilation; the TCK has no unit tests, so I couldn't mutation-check anything.
Checked and fine: extractProtoSchema fails loudly on a missing heading/listing/wrong block, handles CRLF, and declares inputs/outputs; protoc and protobuf-java share protobuf.version; isValidSeq (seq >= 0) is exactly 0..2^63-1 for a decoded uint64, getNextSeq rolls MAX_SEQ to 0, and the recursion fix in 80e60e6 is right; the EdgeNode simulator counters wrap correctly; DateTime is in int64_value in all three encoder arms; the DataSet encoder transposes correctly and the null placeholder keeps columns the same length.
Other points:
- bd_seq is still read only from the v3
bdSeqmetric (Monitor:383, SessionEstablishmentTest:469/:640, PrimaryHostTest, ReceiveCommandTest), and the simulator (EdgeNode.java:267/325) never writesheader.bd_seq. A v4 Edge Node that sends bd_seq only in the Header gets no bdSeq checks at all. In scope for "migrate the payload layer to v4"; if deferred, please list it under "Not addressed". - A compressed payload makes every metric check vacuous (
Utils.java:145, and everygetContent().getMetricsList()):payload_contentis a oneof, so for an Edge Node that sendscompressed,getContent()returns the default instance. Every metric loop then sees an empty list, the bdSeq lookups find nothing, and BIRTH-metric assertions are never set or set wrongly. Suggest decompressing indecode(), or failing/flagging the test whengetPayloadContentCase() == COMPRESSED; at minimum add it to "Not addressed". - A small unit-test source set for
Utils(isValidSeq,getNextSeq,expectedValueCase,hasValue) would have caught the recursion and the three regressions this PR has already needed review to find. - #646 and #647 merge cleanly and #646 keeps the enums CLOSED, but nobody has compiled the two together yet; worth doing before the second one merges.
- The description is stale: "a UInt64 or DateTime in uint64_value"; since 453f98c DateTime goes in
int64_value. - Nit:
Constants.MAX_SEQcould beLong.MAX_VALUE; inEdgeNodethe> MAX_SEQchecks can never be true, only< 0does anything.
| case PropertySetList: | ||
| return Payload.TypedValue.ValueCase.PROPERTYSETS_VALUE; | ||
| default: | ||
| return Payload.TypedValue.ValueCase.VALUE_NOT_SET; |
There was a problem hiding this comment.
Required: a conformant Command metric now fails ID_PAYLOADS_METRIC_DATATYPE_VALUE_TYPE / ..._VALUE (via SendComplexDataTest.java:370). expectedValueCase(Command) falls through to this default, so hasValidDatatype returns false for a Metric with datatype = Command (35) and command set, which the spec requires for Command definitions in NBIRTH/DBIRTH.
It's a regression: the old generated enum had no 35, so DataType.forNumber(35) was null and the metric was "assumed valid".
Same class: hasValue(Metric) (:268) is false for a Command metric, so TOPICS_NBIRTH_METRICS and OPERATIONAL_BEHAVIOR_DATA_PUBLISH_NBIRTH_VALUES fail too (SessionEstablishmentTest :651/:656, :891/:904). That part predates this PR, but the generated proto now has hasCommand(), so this is where it can be fixed.
Suggest special-casing Command in both: valid iff m.hasCommand() and no value.
There was a problem hiding this comment.
Fixed in c871341. Confirmed exactly as you describe — expectedValueCase had no Command case, so it fell through to VALUE_NOT_SET and hasValidDatatype returned false for a Metric with datatype = Command and command set. And you're right that it is a regression of this PR rather than pre-existing: the old generated enum had no value 35, so DataType.forNumber(35) was null and the metric took the "assumed valid" path.
Special-cased in both, as suggested:
if (m.getDatatype() == DataType.Command) {
// carries its definition or invocation in the command field, not a TypedValue
return m.hasCommand() && !m.hasValue();
}and hasValue(Metric) returns m.hasCommand() for a Command metric.
Verified by execution rather than compilation: a Command definition with arguments and no value passes both checks, and a Command metric with neither command nor value is rejected.
| if (payload.hasSeq()) { | ||
| if (payload.getSeq() > 255 || payload.getSeq() < 0) { | ||
| if (payload.getHeader().hasSeq()) { | ||
| if (!Utils.isValidSeq(payload.getHeader().getSeq())) { |
There was a problem hiding this comment.
Optional: this range check can never fail: FAIL is put when !isValidSeq, then immediately overwritten with PASS at :773 because there's no else. Pre-existing, but this PR rewrote the condition and the description presents it as enforcing the v4 bound. Put PASS in an else.
There was a problem hiding this comment.
Fixed in c871341. You're right that it could never fail — the FAIL was written, then overwritten unconditionally by the PASS on the next line. The PASS is now in an else, and the edgeSeqs.put moved below so it still runs either way.
Fair point that this PR rewrote the condition and the description claimed it enforced the v4 bound, so presenting it as working while it could not fail was misleading regardless of it being pre-existing.
| if (!p.hasIntValue() && !p.hasLongValue() && !p.hasFloatValue() && !p.hasDoubleValue() | ||
| && !p.hasBooleanValue() && !p.hasStringValue()) { | ||
| final Payload.TypedValue pv = p.getTypedValue(); | ||
| if (!pv.hasInt32Value() && !pv.hasUint32Value() && !pv.hasInt64Value() |
There was a problem hiding this comment.
Optional: this parameter value check misses bytes_value (the spec allows bytes for a parameter value, and the arm is now visible in the generated proto) and ignores is_null.
The 9d1baba commit message says the null case "is now handled on the Template.Parameter overload", but Utils.hasValue(Parameter) (Utils.java:277) has no callers, so that fix doesn't take effect. Suggest calling it here and adding hasBytesValue(), or deleting the dead overload.
There was a problem hiding this comment.
Fixed in c871341, and you're right on both counts — including that my commit message claimed a fix that could not take effect.
The hand-written arm list is replaced by a call to Utils.hasValue(Parameter), which covers bytes_value through the generic getValueCase() != VALUE_NOT_SET test and handles is_null explicitly. That also gives the overload a real caller; you were correct that its only other reference was inside a logger.debug, so the is_null handling I added in 9d1baba was dead until now.
…essing, and schema bounds Required: - The namespace substitution note made STATE ambiguous. STATE is pinned to spCv4.0/STATE/sparkplug_host_id, so a JSON Edge Node following the note would have waited on spJv4.0/STATE while the Host published on spCv4.0/STATE and never seen it come online. STATE is now stated as the one exception, with the reason. - Nothing said which namespace a Host Application uses to address a particular Edge Node. A Host supports both encodings, so it could send an NCMD or a Rebirth Request on a namespace the Edge Node does not subscribe to; the MQTT Server accepts it and it is never delivered, so Rebirth recovery fails silently. A Host now publishes to an Edge Node on the namespace of that Edge Node's NBIRTH, and an Edge Node subscribes to its command topics on the namespace it publishes on. Also: - nonNegative64 capped every 64-bit field at 2^63-1, but alias, MetaData.size and Status.code are uncapped uint64, so a legal spCv4.0 payload could not be expressed as a schema-valid spJv4.0 one - contradicting the lossless conversion this section claims. Split into uint64 and sequence64, with the string branch of sequence64 bounded by length and its limit stated. - The Sparkplug Aware MQTT Server encoding requirement is now conditional on replacing NDEATH timestamps, which is the only payload-touching behavior required of it and is itself a MAY. - Flattening drops the TypedValue's own datatype and is_null. The two copies must now agree in the Protocol Buffers encoding, the single JSON key represents both, and a null Template.Parameter has a JSON form: is_null is added to the parameter keys and to the schema. - The metric ordering MUST is scoped to payloads carrying more than one value for the same metric, with a SHOULD elsewhere so a BIRTH in browse order stays conformant, and it now says it does not reach Template or Command metrics. - Nits: INT64 rollover wording, a stale (max value of an INT64), DateTimeArray elements described as signed, and a MUST NOT inside a non-normative comment.
|
All three inline items addressed in c871341; replies on each thread. On the other points from your review body: Fixed — compressed payloads made every metric check vacuous. You were right and I had not considered it. Fixed — Deferred, and now listed under "Not addressed" — bd_seq from the Header. Agreed this is in scope for the title. I started it and backed it out: the read sites are nested inside metric loops with their own increment logic, and the Tahu payload model has no bd_seq field at all, so the simulator cannot write Agreed on the unit tests. A small source set over On compiling #646 and #647 together — done. They are independent: #647 builds green on #645 alone, without the editions change, which is why they are separate PRs rather than a chain. Description updated, including the stale "a UInt64 or DateTime in uint64_value" you caught; DateTime has been in |
ckienle
left a comment
There was a problem hiding this comment.
Re-reviewed at c871341. All fixed as described: the Command datatype special case in hasValidDatatype and hasValue, the else on the NBIRTH seq check, the parameter check now going through Utils.hasValue(Parameter) (bytes and is_null covered), and decode() rejecting a compressed payload. Its callers already treat null as unparseable, so that's now a reported failure rather than a silent pass. Deferring Header bd_seq with it listed under "Not addressed" is reasonable given the Tahu model has no field for it. CI is green. Approving.
…gth-bounded string form MetaData.seq sequences the parts of a multi-part file transfer and is an uncapped uint64 in both the schema and the prose. The earlier split pointed it at sequence64 because the bulk rename matched on the field name, which the two fields share, so it was capped at 2^63-1 by accident - the same class of mistake as the regex that produced the recursion. Also gives the uint64 definition the same note sequence64 carries, since its string branch is bounded by length rather than by value, and re-wraps the DateTimeArray entry left split mid-phrase by the earlier edit.
…ation as a build artifact Adds an extractProtoSchema task that lifts the Protocol Buffers schema out of the Payloads chapter into build/proto/sparkplug_c.proto, and publishes it as a resolvable artifact the same way tck-audit.xml already is. This is the prerequisite for generating SparkplugBProto.java rather than checking it in, which is how the TCK's copy drifted a full major version behind the specification. With the schema published as an artifact, the specification text stays its single source of truth. The TCK is not switched over here - that needs the payload layer migrated to the v4 message shape, which is the rest of eclipse-sparkplug#641.
…load layer to v4 SparkplugBProto.java is no longer checked in. It is generated from the schema the specification publishes, so it cannot drift from the spec the way the old copy did - that copy was the Sparkplug 3 protobuf, with no Header, Content, DataSet.Column, descriptor_seq or quality. Bumps protobuf-java from 3.17.3 (June 2021) to 4.29.3. The two changes are inseparable: the checked-in file was produced by protoc 3.17 and its generated code contract does not compile against a 4.x runtime, so the CVE exposure in 3.17.3 could not be cleared without regenerating. The payload layer is migrated to the v4 message shape: - Payload level timestamp, seq, uuid and body move to Header and Content, across 94 call sites. - Metric, Template.Parameter and PropertySet values move into TypedValue. v3's int_value and long_value each split into a signed and an unsigned arm, so the tests go through new Utils helpers that accept either rather than guessing. - PropertyValue is gone; PropertySet values are TypedValues. - The encoder's DataSet path is transposed from rows to columns, and the per-cell DataSetValue converter becomes one that appends to the column array matching each column's datatype. - datatype fields are the DataType enum rather than an int. Sequence semantics are brought to v4 as well. The TCK still enforced Sparkplug 3's 0-255 range in Utils.getNextSeq and in seven range checks across five tests, so a conformant v4 Edge Node whose seq exceeded 255 would have failed. The bound is now the capped rollover point, as Constants.MAX_SEQ. SendComplexDataTest.checkDataSet is removed, as in eclipse-sparkplug#644. It tested the row oriented DataSet through getNumOfColumns() and getTypesList(); re-adding coverage against the replacement assertions is follow-up work.
…ve to the current runtime The 0-255 checks were replaced with a bare non-negative test, which is correct but reads as though the bound was simply deleted. Replaced with Utils.isValidSeq, which documents why: the specification allows 0 to 9223372036854775807 inclusive, and because the field decodes from an unsigned 64 bit integer into a signed Java long, a value above that cap arrives negative. Testing for a non-negative long therefore accepts exactly the range the specification allows. Seven call sites across five tests now go through it. Bumps protobuf from 4.29.3 to 4.36.2, the current release.
DateTime moves from uint64_value to int64_value in TypedValue, matching the specification change. Updates Utils.expectedValueCase and the three encoder arms that wrote a Date into the unsigned field. A DateTime in a DataSet.Column is unchanged. There is no signed repeated field, so it stays in long_values and is carried as its two's complement representation.
The regex that routed the sequence checks through the new helper also rewrote the helper's own body, leaving `return Utils.isValidSeq(seq);`. It compiles, so the build stayed green, but every sequence-number check would have recursed until the thread died with a StackOverflowError - and the TCK has no unit tests to catch it. Reached from Monitor, SessionEstablishmentTest, SendDataTest, SendComplexDataTest and SessionTerminationTest, so any Edge Node publishing an NBIRTH would have crashed the handler, and the 0..9223372036854775807 bound was never enforced. Verified by execution rather than compilation this time: isValidSeq returns the expected answer for 0, 1, 255, 256, MAX-1, MAX, -1 and Long.MIN_VALUE, and getNextSeq rolls MAX over to 0 while 255 now advances to 256.
…payload migration hasValue(Metric) lost its is_null branch in the rewrite. A conformant BIRTH metric published with is_null true and no TypedValue - which the specification endorses and this encoder itself emits - reported as having no value, failing topics-nbirth-metrics, topics-dbirth-metrics and the data-publish BIRTH value checks against a compliant Edge Node. The same case is now handled on the Template.Parameter overload, where v4's TypedValue carries is_null too. EdgeNode kept two counters on the Sparkplug 3 range after birthBdSeq was migrated. deathBdSeq still rolled at 256, so after 256 reconnects the NDEATH bd_seq diverged from the NBIRTH bd_seq the Will Message registered, and the reference Edge Node violated its own bd_seq session-constant requirement. getNextSeqNum still rolled the payload sequence at 256, so from the 257th message the Monitor failed sequence increment checks on payloads the TCK itself produced. The DataSet encoder skipped a null cell rather than emitting one, which shortens that column and shifts every later row. A v4 Column is a parallel array indexed by row and cannot express an absent cell, so a null now holds its position with the zero value for the column's datatype, keeping every Column the same length as payloads-dataset-row-correlation requires.
…ayloads, and a vacuous check Required: a conformant Command metric failed conformance. expectedValueCase has no Command case, so it fell through to VALUE_NOT_SET and hasValidDatatype returned false for a Metric with datatype Command and the command field set, which is how a Command definition appears in an NBIRTH or DBIRTH. It is a regression of this PR: the old generated enum had no value 35, so forNumber(35) was null and the metric was assumed valid. hasValue had the same gap. Both now treat a Command metric as carrying its payload in the command field. Also: - A compressed payload made every metric check vacuous. payload_content is a oneof, so getContent() returns the default instance and every metric loop saw an empty list, with BIRTH assertions never set or set wrongly. decode() now rejects a compressed payload and logs it rather than reporting silent passes. Decompressing is left for the follow-up work listed in the description. - The Monitor sequence range check could never fail: the FAIL was overwritten by an unconditional PASS on the next line. The PASS is now in an else. - The Template.Parameter value check went through a hand-written list of value arms that omitted bytes, which the specification allows, and ignored is_null. It now calls Utils.hasValue(Parameter), which also gives that overload a real caller rather than leaving it reachable only from a debug log. - Constants.MAX_SEQ is Long.MAX_VALUE, which is what it means.
Closes #641.
Stacked on #645. It is based on that branch because the payload layer it migrates is the one
#645 defines. It does not depend on #646 — verified by building this branch without it.
The TCK was built against Sparkplug 3.
SparkplugBProto.javawas checked in, generated by protoc3.17, and had no
Header,Content,DataSet.Column,descriptor_seq,definition_id,qualityor
MetricOperation. No v4 payload requirement could be tested.The protobuf is now generated, not checked in
A new
extractProtoSchematask lifts the schema out ofSparkplug_6_Payloads.adocintobuild/proto/sparkplug_c.protoand publishes it as a resolvable artifact, the same patterntck-audit.xmlalready uses. The TCK consumes that artifact and generates from it, so thespecification text is the single source of truth and a schema change that breaks the TCK now fails
the build instead of going unnoticed for a major version.
Gradle fetches protoc itself, so no local protoc install is required.
The runtime bump could not be done separately
protobuf.versiongoes from 3.17.3 (June 2021) to 4.36.2, clearing CVE-2022-3171 andCVE-2024-7254. This had to happen in the same change: the checked-in file was produced by protoc
3.17 and does not compile against a 4.x runtime —
Payload layer migration
Header/ContentTypedValuePropertyValueremoved; PropertySet values areTypedValueUtils, testsdatatypeas theDataTypeenum, not an intv3 carried every 32-bit value in
int_valueand every 64-bit inlong_value; v4 splits each into asigned and an unsigned arm, so a check that used to be unambiguous now has two possible answers —
an
Int64orDateTimelands inint64_value, aUInt64inuint64_value. Rather than guessingper call site, the tests go through new
Utils.hasLongValue/getLongValue/hasIntValue/getIntValuehelpers that accept either arm.Sequence semantics were still Sparkplug 3
Independent of the protobuf, and arguably the more consequential half.
Utils.getNextSeqassertedseq <= 255and seven range checks across five tests enforced the same bound, so a conformant v4Edge Node whose seq passed 255 would have failed the TCK.
These now go through
Utils.isValidSeq, which documents why the test is what it is: thespecification allows 0 to 9223372036854775807 inclusive, and because the field decodes from an
unsigned 64-bit integer into a signed Java long, a value above the cap arrives negative — so testing
for a non-negative long accepts exactly the specified range.
EdgeNode.birthBdSeqalso started at 255 to exercise rollover and would otherwise have silentlystopped testing anything.
Verification
./gradlew clean buildpasses. The generatedSparkplugBProto.javais 23,384 lines and containsevery v4 construct the old copy lacked.
Addressed in review
expectedValueCasehad no Command case, sohasValidDatatypeandhasValuereturned false for a Metric carrying its payload in thecommandfield. A regression of this PR, since the old generated enum had no value 35.payload_contentis a oneof, sogetContent()returned the default instance and every metric loop saw an empty list.decode()now rejects and logs it rather than reporting silent passes.
unconditional PASS on the next line.
bytes_valueand ignoredis_null; it now callsUtils.hasValue(Parameter), which previously had no caller outside a debug log.Not addressed
spBv1.0namespace constant, 81 call sites. With [#231] Define the spJv4.0 JSON payload encoding #645 there are two v4 namespaces, so this isnot a rename — each call site has to know which encoding it exercises.
[#632] Rewrite the DataSet prose against the current schema #644 removed.
bd_seqis still read only from the v3bdSeqmetric (Monitor, SessionEstablishmentTest,PrimaryHostTest, ReceiveCommandTest), and the simulator does not write
header.bd_seq. A v4 EdgeNode that carries bd_seq only in the Header gets no bdSeq checks. The read sites are nested
inside metric loops and the Tahu payload model has no bd_seq field, so both halves are a change
of their own rather than a mechanical edit.
Utils. The recursion and the regressions this PR needed reviewto find would all have been caught by one.
SendComplexDataTest.checkDataSetstays removed, as in [#632] Rewrite the DataSet prose against the current schema #644.Note for coverage audits
Grepping the TCK for a hyphenated assertion id finds only
Requirements.java, because the Java usesthe generated uppercase constants (
ID_PAYLOADS_DATASET_TYPES_DEF, notpayloads-dataset-types-def).Searching for the id alone makes a tested assertion look untested.