Repository navigation
[#231] Define the spJv4.0 JSON payload encoding - #645
Conversation
Adds a normative JSON encoding of the Sparkplug payload model, published on the spJv4.0 namespace alongside the Protocol Buffers encoding on spCv4.0. The mapping is derived from the Google Protocol Buffers JSON Mapping so that most of it can be produced by library facilities, with Sparkplug's deviations stated explicitly: TypedValue is flattened to a bare 'value', PropertySet is a JSON object keyed by property name, array DataTypes are JSON arrays rather than base64 packed bytes, and a DataSet.Column carries one 'values' array. 64-bit integers are encoded as JSON strings. A v4 nanosecond timestamp is roughly 165 times larger than the 2^53 limit a JSON number is guaranteed to carry, so a numeric form is silently rounded by any parser that decodes into a double. Also generalizes three Ch. 5 statements that required a Protobuf encoded payload, notes that topics written with spCv4.0 apply equally to spJv4.0, and renames the 'dataType' key in the Ch. 6 examples to 'datatype' and the metric-level 'template' key to 'value' so the examples match the encoding rules.
…ing-encoded 64-bit values The note claimed a Sparkplug sequence number is always small, which is wrong: seq, bd_seq, and descriptor_seq each count to the maximum value of their declared type before rolling over to zero. They are therefore subject to the same precision loss as timestamps, and a numeric form would be correct early in a session and lossy late in one.
The three message sequence numbers move from uint64 to int64 in the Protocol Buffers schema, with the range and rollover point changing from 18446744073709551615 to 9223372036854775807 across all 38 statements that bound them in Chapters 4, 5, and 6. Protobuf uint64 is a signed long in Java, so the upper half of the uint64 range was never reachable by an implementation built on it. Declaring int64 makes the specified range the representable one. MetaData.seq, which sequences the parts of a multi-part file transfer, is a different field and is unchanged.
… rsn arithmetic Header.timestamp and Metric.timestamp move from uint64 to int64, for the same reason as the sequence numbers: protobuf uint64 is a signed long in Java, so the upper half of the unsigned range was never reachable. The range note in Chapter 5 is rewritten accordingly - the ceiling is 2262-04-11, stated directly rather than as a caveat on an unsigned range that no implementation could use. The NREBIRTH rsn calculation said to add the maximum sequence number to a negative result. That was off by one, since a counter cycling 0..MAX has a period of MAX+1, and MAX+1 is not representable as an int64. Restated as two cases so no intermediate is ever negative or out of range: if pds >= 1 + dc: rsn = pds - (1 + dc) otherwise: rsn = 9223372036854775807 - (dc - pds) Verified equivalent to (pds - (1 + dc)) mod 2^63 over the full range.
…ing seq as unsigned The seq, bd_seq, and descriptor_seq bullets in the Header section still described each field as an unsigned 64-bit integer after the schema moved them to int64.
…ormance profiles An Edge Node MUST support at least one encoding and MAY support both, selected by configuration, but MUST use a single encoding and namespace for the whole of an MQTT session - seq is Edge-Node-wide and bd_seq scopes the session, so splitting one session across both namespaces would produce two unusable halves of one sequence stream. Host Applications and Sparkplug Aware MQTT Servers MUST support both encodings. A Host Application cannot choose the encoding of the Edge Nodes it consumes, and an Aware Server decodes payloads rather than treating them as opaque. A Sparkplug Compliant MQTT Server has no payload requirements at all, which is now stated non-normatively so the asymmetry with the Aware profile is not read as an omission. Also generalizes the Edge Node profile's reference to the spCv4.0/# namespace.
…g the capped range seq, bd_seq, and descriptor_seq go back to uint64 while keeping the rollover point at 9223372036854775807. timestamp stays int64. An unsigned type is right for a counter that can never be negative, and it keeps the three consistent with alias, MetaData.size, and Status.code. Capping the range below the type's maximum still guarantees every value a conformant Edge Node can produce fits a signed 64-bit integer, so an implementation in a language without an unsigned 64-bit type never needs unsigned comparison. Neither bound is reachable: at a million messages per second the capped range lasts ~292,000 years. timestamp keeps int64 because every mainstream time representation it interoperates with is signed - protobuf's own Timestamp, OPC UA DateTime, java.time.Instant - and the 2262 ceiling it accepts is one none of those exceed either. The repeated "(max value of a UINT64)" parenthetical is dropped rather than rescaled, since the cap is now a deliberate restriction rather than the type's maximum. It is explained once in the Header section instead.
…ary-support claim The oneof was named 'content' and its uncompressed arm was named 'plain', so a Protocol Buffers JSON implementation emitted the key "plain" while every example in the chapter used "content". A field may not share the name of its containing oneof, so the oneof is renamed to payload_content and the arm becomes 'content'. The examples were right and the schema now agrees with them, and a stock library emits the correct key with no post-processing. Three prose references to the 'plain' field are updated. The non-normative comment claimed an implementation could produce most of the JSON encoding with library facilities. Measured against the compiled schema, that is not true: the structural mappings reshape every Metric, so an encoder writes its own traversal. What the derivation does settle is the scalar rules and the header, which the comment now says instead.
…er extension fields The section leaned on the Google Protocol Buffers JSON Mapping for part of its meaning. That mapping is vendor documentation rather than a versioned standard, it is specified for proto3 while the Sparkplug schema is proto2, and it does not cover extensions at all. The section now states its rules itself and says so. - The lowerCamelCase form a consumer must accept is defined here - remove each underscore and upper case the letter after it - so the assertion no longer depends on an external document for its meaning. - New Extension Fields section: an extension is keyed by its fully qualified name in square brackets, its value follows the ordinary rules for its type, and an unrecognized extension is ignored. The Sparkplug schema reserves an extension range in most messages, so leaving this undefined left a field that is well-defined on the wire undefined in JSON. Includes a note that unlike the binary encoding, where an unknown field is retained and re-emitted, an unrecognized JSON key does not survive a round trip. - Floating point values must be written with enough digits to parse back exactly: up to 9 significant decimal digits for a float and 17 for a double.
…e change time Metrics in a payload are ordered oldest first, so a consuming application that applies them in array order arrives at the correct current value without comparing timestamps. This matters when one payload carries several values for the same metric, as a payload of historical values does. Metrics remain identified by name or alias, so position still carries no identity. A metric that omits its timestamp sorts as though its value change time were the payload timestamp, which is the time it is already taken to have changed. Metrics that share a value change time may appear in any order and a consumer must not reorder them. Multi-part file publishing depends on that: it identifies a metric by position, and every metric of a multi-part publish carries the same timestamp, so they tie and the publisher's order is carried unchanged by both encodings. Non-normative comments in both sections record the interaction. Also splits the JSON PropertySet statement. Uniqueness of property names is observable in a payload and stays testable; the absence of meaning in key order is not observable and becomes non-testable, with a note that RFC 8259 defines a JSON object as unordered so the requirement is on the publisher.
1. The conformance note justifying the session scoping rule was denser than the rule it explained. Reduced to the plain consequence: an Edge Node has one sequence number stream, so splitting a session across namespaces leaves a consumer of either with gaps in it. 2. Removed the requirement that a consuming application accept the lowerCamelCase form of a field name. A conformant publisher emits the Protocol Buffers field name, so the rule required consumers to accept a form no conformant publisher may produce - specifying leniency toward non-conformance and creating a second de facto wire form for every field. Replaced with implementer guidance: a Protocol Buffers JSON library converts to lowerCamelCase by default and must be configured to preserve the original names. 3. Explained why an absent field is omitted rather than written as null or zero. Zero is a legitimate value for seq, bd_seq, descriptor_seq, alias, and quality, so writing an absent field as 0 would be indistinguishable from one set to zero; and null would be a second, conflicting way to express what is_null already expresses. 4. Scoped the ignore-unknown rule to the names of fields within a message. It was open to being read as covering an unrecognized metric, which is a different situation entirely - a mismatch against the metric structure the application holds, resolved by the metadata rules and a Rebirth Request rather than by ignoring it. Both cases are now stated and contrasted.
A DateTime value is a nanosecond count since epoch, the same quantity the payload and metric timestamps carry, so it follows them to int64 rather than staying unsigned. Leaving it as uint64 meant a DateTime value and a payload timestamp differed in signedness despite meaning the same thing. This moves a DateTime value from the uint64_value arm of TypedValue to int64_value, so unlike the timestamp field changes it is a wire change. A DateTime in a DataSet.Column is unaffected: there is no signed repeated field, so it stays in long_values alongside UInt64 and is carried as its two's complement representation, which the DataSet.Column section already requires for signed values.
…PropertySet TypedValues Review found two defects in the normative JSON Schema and its prose. nonNegative64 was "type": ["string","integer"] with "pattern": "^[0-9]+$". JSON Schema applies pattern only to strings, so a negative JSON number validated cleanly, as did a string far beyond the 64-bit cap. That governs seq, bd_seq, descriptor_seq, timestamp, alias, MetaData size and seq, and Status code - the fields whose range the rest of the specification relies on. Now bounded on both arms. The earlier negative tests missed this because they used the string form. The TypedValue flattening rule named only Metric and Template.Parameter as datatype carriers, but PropertySet.values is a repeated TypedValue, so a property value had no enclosing field to carry its datatype and could not be encoded by following the rule. The JSON Schema already resolved this the other way, nesting an object with its own datatype - which the rule forbade. The rule is now scoped to Metric and Template.Parameter, and the PropertySet case is stated to match the schema. Also corrects a reference to PropertyValues, which is not a v4 message.
ckienle
left a comment
There was a problem hiding this comment.
Reviewed at bcf75ba. Two required interop gaps, both small text fixes, plus some optionals inline.
What I checked and found right: the rsn fix (worked both branches by hand: pds=1/dc=2 gives MAX-1 and matches the Ch.5 example, pds=0/dc=0 gives MAX, nothing goes negative), the sequence bound rescale (no 2^64-1 left anywhere), the payload_content rename across schema/prose/normative text, the scalar mapping rules, the JSON Schema field names against the head proto, and no duplicate tck-ids among the 38 new ones.
Nits / unanchored:
Sparkplug_5_Operational_Behavior.adoc:1268still says "INT64 rollover";Sparkplug_6_Payloads.adoc:3106-3107still has "(max value of an INT64)".Sparkplug_6_Payloads.adoc:1849: DateTimeArray elements are now signed after c0781dc, but the text still just says "8-byte value".Sparkplug_6_Payloads.adoc:3316: a "MUST NOT reorder" inside a non-normative comment;metrics-ordered-tiesalready says it.- The PR description is stale: "Known gaps: DateTime is still uint64" was undone by c0781dc, and that move (uint64_value -> int64_value) is a field-number change, harmless only because spCv4.0 is unreleased. The existing approval also predates c0781dc/bcf75ba.
- Pre-existing, not this PR: the PropertyValue prose (
:942) still names the fieldtypeand the Metric value list (:774) still lists the v3 arms. Worth a separate cleanup since JSON readers will take prose field names literally.
| 'Sparkplug B' topic namespace definitions and concepts, see the earlier versions of the Sparkplug | ||
| Specification. | ||
|
|
||
| Non-normative comment: Topics are written with the spCv4.0 namespace element throughout the |
There was a problem hiding this comment.
Required: this note contradicts the STATE rule at :1172 ("the STATE topic is never prefixed and always remains at spCv4.0/STATE/sparkplug_host_id").
A JSON-only Edge Node configured to wait for a Primary Host follows this note and watches spJv4.0/STATE/host, while the Host publishes only on spCv4.0/STATE/host. The Edge Node never sees the Host come online and never births. A Host that reads it the other way publishes two STATE streams with two Wills.
Suggest excluding STATE explicitly, e.g. "STATE is not a Sparkplug payload and is published only at spCv4.0/STATE/sparkplug_host_id regardless of the encoding an Edge Node uses.
There was a problem hiding this comment.
Fixed in 1fcab1d. You're right, and the failure mode is exactly as you describe — a JSON Edge Node would have waited on a spJv4.0/STATE topic nobody publishes, and never birthed.
Added tck-id-topics-state-single-namespace: a STATE message MUST be published on spCv4.0/STATE/sparkplug_host_id whatever encoding the infrastructure uses, and an Edge Node waiting on a Primary Host MUST subscribe to that topic rather than a spJv4.0 equivalent. A non-normative comment gives the reasons: STATE carries a JSON document rather than a Sparkplug payload, so there is nothing for a second namespace to encode differently, and one topic keeps one Will Message per Host Application.
| * [tck-testable tck-id-conformance-host-encoding-both]#[yellow-background]*[tck-id-conformance-host-encoding-both] A | ||
| Sparkplug Host Application MUST support both Sparkplug payload encodings, and MUST be able to | ||
| consume messages on both the spCv4.0 and spJv4.0 namespaces.*# | ||
| * [tck-testable tck-id-conformance-host-encoding-publish]#[yellow-background]*[tck-id-conformance-host-encoding-publish] When |
There was a problem hiding this comment.
Required: nothing says which namespace a Host uses to command an Edge Node. This only ties the encoding to the namespace the Host publishes on; the Edge Node subscribes to <ns>/NCMD|DCMD/... and REBIRTH on its own namespace.
A Host that supports both encodings (as it now must) and defaults to spCv4.0 sends a rebirth request or NCMD to an Edge Node that births on spJv4.0, and it's never delivered. Rebirth recovery silently fails for every JSON Edge Node.
Suggest: a Host MUST publish NCMD/DCMD/rebirth to an Edge Node on the namespace of that Edge Node's current NBIRTH, and an Edge Node MUST subscribe to its command topics on the namespace it publishes on.
There was a problem hiding this comment.
Fixed in 1fcab1d. Agreed this was a real gap — the rule only covered which encoding a Host uses, not which namespace it addresses.
Two statements, one for each direction, close to your suggested wording:
conformance-host-encoding-address: when a Host Application publishes a message addressed to a particular Edge Node or Device, including an NCMD, a DCMD, and a Rebirth Request, it MUST publish on the namespace of that Edge Node's current NBIRTH.conformance-edge-node-encoding-subscribe: an Edge Node MUST subscribe to its command topics on the same namespace it publishes on.
The non-normative comment records why it matters: the MQTT Server accepts the misaddressed message and it is simply never delivered, so a Rebirth Request fails silently and the Host cannot recover its view of that Edge Node.
|
|
||
| * [tck-testable tck-id-conformance-mqtt-aware-basic]#[yellow-background]*[tck-id-conformance-mqtt-aware-basic] A | ||
| Sparkplug Aware MQTT Server MUST support all aspects of a Sparkplug Compliant MQTT Server*# | ||
| * [tck-testable tck-id-conformance-mqtt-aware-encoding-both]#[yellow-background]*[tck-id-conformance-mqtt-aware-encoding-both] A |
There was a problem hiding this comment.
Optional (but it affects every broker vendor): the premise here doesn't hold. The only payload-touching behaviour required of an Aware server below is mqtt-aware-ndeath-timestamp, which is a MAY, and this section also says the Aware server no longer stores or republishes BIRTHs.
As written, every Aware server has to implement JSON decode/encode even if it never rewrites a timestamp. Suggest making it conditional: "If a Sparkplug Aware MQTT Server replaces NDEATH timestamps, it MUST do so for both encodings." and fixing the comment.
There was a problem hiding this comment.
Fixed in 1fcab1d. You're right that the premise didn't hold — I had asserted the Aware server decodes payloads, when the only payload-touching behaviour required of it is the NDEATH timestamp replacement, which is itself a MAY.
Now conditional: a Sparkplug Aware MQTT Server that replaces NDEATH timestamps MUST do so for payloads in both encodings. A server that does not never inspects a payload and needs neither encoding. The comment is rewritten to say that, and to note the requirement exists so that offering the timestamp behaviour for one encoding only cannot leave Edge Nodes on the other namespace silently unserved.
| }, | ||
| "not": { "required": [ "content", "compressed" ] }, | ||
| "$defs": { | ||
| "nonNegative64": { |
There was a problem hiding this comment.
Optional: nonNegative64 is applied to alias, MetaData.size and Status.code, which are uncapped uint64 (:682, :1601), so it rejects valid values >= 2^63. It's also inconsistent with itself: the number branch caps at 2^63-1 but the string pattern ^(0|[1-9][0-9]{0,18})$ accepts up to 9999999999999999999 and rejects "18446744073709551615".
So an spCv4.0 payload with a large alias or status code can't be converted to a schema-valid spJv4.0 payload, which contradicts "converted from one to the other without loss" and json-schema-valid.
Suggest two defs: a uint64 one (0..2^64-1 or a <=20-digit string) for alias/size/MetaData.seq/code, and a bounded one for the sequence numbers and timestamps whose string pattern actually enforces 2^63-1 (or say in its comment that the string branch isn't bounded).
There was a problem hiding this comment.
Fixed in 1fcab1d, and thank you — this was a bug I introduced while fixing a different one. I had conflated "the sequence numbers are capped at 2^63-1" with "every 64-bit field is", which is wrong for alias, MetaData.size and Status.code.
Split into two definitions as you suggested:
uint64— 0 to 18446744073709551615, used foralias,MetaData.size,MetaData.seqandStatus.codesequence64— 0 to 9223372036854775807, used forseq,bd_seq,descriptor_seqandtimestamp
On the string branch: an exact "<= 9223372036854775807" pattern needs about fifteen alternatives and is unreadable in a specification, so I took your second option. It is bounded by length and the comment says so explicitly: "The string form is bounded by length only; the exact upper bound is normative prose, not expressible as a readable pattern."
Verified: alias and Status.code at uint64 max are accepted again, seq at 20 digits and a negative number are rejected.
There was a problem hiding this comment.
Thanks — all three leftovers fixed in 40728fa.
MetaData.seq — you were right, and my reply was wrong. It still pointed at sequence64. The bulk rename that split the definitions matched on the field name, which Header.seq and MetaData.seq share, so the multi-part file index got capped at 2^63-1 by accident. It is an uncapped uint64 in both the schema and the prose, and now points at uint64. Same class of mistake as the regex that produced the isValidSeq recursion, which is a pattern I should watch for.
Verified: MetaData.seq at uint64 max is accepted again, while a Header.seq at that value is still rejected.
uint64 comment — now carries the same note sequence64 has, that the string form is bounded by length only and the exact upper bound is normative prose.
DateTimeArray — re-wrapped. It was worse than you flagged: my earlier edit had split it across three lines, breaking after both "signed" and "8-byte".
| the datatype and selects one of several value fields. In JSON the datatype is already carried by | ||
| the enclosing object and the value field is not needed to disambiguate. | ||
|
|
||
| [tck-testable tck-id-payloads-json-typedvalue-flat]#[yellow-background]*[tck-id-payloads-json-typedvalue-flat] A |
There was a problem hiding this comment.
Optional: flattening drops the TypedValue's own datatype/is_null. Metric has both (fields 4/10) and so does its value TypedValue (fields 1/2), but JSON keeps one of each, and nothing says the pairs must agree in spCv4.0 or which one a JSON-to-proto converter sets. So the conversion isn't lossless as claimed.
Related: typedvalue-parameter-keys lists only name/type/value and the schema's parameter has no is_null, but json-null requires is_null: true to express null, so a null Template.Parameter has no JSON form.
Suggest requiring the two to agree in spCv4.0, saying the JSON datatype/is_null carry both, and adding is_null to the Parameter keys and schema.
There was a problem hiding this comment.
Fixed in 1fcab1d. Both halves were real.
On the duplicated datatype/is_null: added payloads-typedvalue-datatype-agree (where a Metric carries both its own and its TypedValue's, the two MUST be equal, and likewise for is_null) and payloads-json-typedvalue-carries-both (the single JSON key represents both, so a JSON-to-protobuf converter sets both fields from it). That makes the conversion lossless as claimed rather than leaving it to the implementer.
On the null Template.Parameter: is_null is added to typedvalue-parameter-keys and to the schema's parameter definition, and payloads-json-parameter-is-null states that a null parameter carries is_null true and omits value, as any other null does.
| ** _metrics_ | ||
| *** This is an array of metrics representing key/value/datatype values. Metrics are further defined | ||
| link:#payloads_c_metric[here]. | ||
| *** [tck-testable tck-id-payloads-metrics-ordered]#[yellow-background]*[tck-id-payloads-metrics-ordered] The |
There was a problem hiding this comment.
Optional: this is a broad new MUST for every Edge Node. It applies to every payload, including NBIRTH/DBIRTH, which existing stacks (Tahu, Ignition Transmission) emit in browse order, not by change time. The motivating case is several values of the same metric in one payload.
It's also unclear whether "the metrics of a payload" includes Template member metrics and Command arguments/results.
Suggest scoping the MUST to payloads carrying more than one value for the same metric (or DATA messages) with a SHOULD elsewhere, and saying explicitly whether nested metrics are covered. If the broad version is a deliberate working-group decision, at least call it out as a migration item.
There was a problem hiding this comment.
Fixed in 1fcab1d, taking your suggestion.
payloads-metrics-ordered is now scoped: a payload that carries more than one value for the same metric MUST order by value change time. payloads-metrics-ordered-should makes it a SHOULD everywhere else, so an Edge Node emitting a BIRTH in browse order — which is what Tahu and Ignition Transmission do — stays conformant.
On nested metrics, payloads-metrics-ordered-scope now says it outright: the ordering applies to the metrics array of the payload's content, and does not reach the metrics of a Template or the arguments or results of a Command, whose order is defined by the Template Definition or the Command definition.
The rationale note is reworded to match, so it explains why the requirement is narrow rather than implying the broad version.
…essing, and schema bounds Required: - The namespace substitution note made STATE ambiguous. STATE is pinned to spCv4.0/STATE/sparkplug_host_id, so a JSON Edge Node following the note would have waited on spJv4.0/STATE while the Host published on spCv4.0/STATE and never seen it come online. STATE is now stated as the one exception, with the reason. - Nothing said which namespace a Host Application uses to address a particular Edge Node. A Host supports both encodings, so it could send an NCMD or a Rebirth Request on a namespace the Edge Node does not subscribe to; the MQTT Server accepts it and it is never delivered, so Rebirth recovery fails silently. A Host now publishes to an Edge Node on the namespace of that Edge Node's NBIRTH, and an Edge Node subscribes to its command topics on the namespace it publishes on. Also: - nonNegative64 capped every 64-bit field at 2^63-1, but alias, MetaData.size and Status.code are uncapped uint64, so a legal spCv4.0 payload could not be expressed as a schema-valid spJv4.0 one - contradicting the lossless conversion this section claims. Split into uint64 and sequence64, with the string branch of sequence64 bounded by length and its limit stated. - The Sparkplug Aware MQTT Server encoding requirement is now conditional on replacing NDEATH timestamps, which is the only payload-touching behavior required of it and is itself a MAY. - Flattening drops the TypedValue's own datatype and is_null. The two copies must now agree in the Protocol Buffers encoding, the single JSON key represents both, and a null Template.Parameter has a JSON form: is_null is added to the parameter keys and to the schema. - The metric ordering MUST is scoped to payloads carrying more than one value for the same metric, with a SHOULD elsewhere so a BIRTH in browse order stays conformant, and it now says it does not reach Template or Command metrics. - Nits: INT64 rollover wording, a stale (max value of an INT64), DateTimeArray elements described as signed, and a MUST NOT inside a non-normative comment.
|
All six inline items addressed in 1fcab1d; replies on each thread. The unanchored nits from your review body:
PR description updated. You were right that it was stale in both directions: the "DateTime is still uint64" gap was undone by c0781dc, and that move is a wire change — Also noted: the existing approval predates c0781dc and bcf75ba, and now this change too. On the two pre-existing items you flagged as out of scope — the PropertyValue prose at Thanks for working the rsn arithmetic through by hand — that and the sequence rescale were the parts I was least able to check mechanically. |
ckienle
left a comment
There was a problem hiding this comment.
Re-reviewed at 1fcab1d. Everything from my review is fixed as described: the STATE single-namespace rule, the two command-addressing rules, the conditional Aware-server requirement, the uint64/sequence64 split, the TypedValue agreement rules and the null Template.Parameter, the scoped metric-ordering MUST, and all the body nits. No duplicate tck-ids. Approving.
Three small leftovers, none blocking:
- Your reply says
MetaData.seqmoved touint64, but the schema still points it atsequence64(Sparkplug_6_Payloads.adoc:3976). It's an uncapped uint64 in the proto and the prose, so I thinkuint64is what you meant. - The
uint64string pattern^(0|[1-9][0-9]{0,19})$accepts 20-digit values above 18446744073709551615. Fine to leave length-bounded, but its comment should say so, as thesequence64one does. - The DateTimeArray line at
:1858wraps after "signed" and again after "8-byte"; worth re-wrapping.
…gth-bounded string form MetaData.seq sequences the parts of a multi-part file transfer and is an uncapped uint64 in both the schema and the prose. The earlier split pointed it at sequence64 because the bulk rename matched on the field name, which the two fields share, so it was capped at 2^63-1 by accident - the same class of mistake as the regex that produced the recursion. Also gives the uint64 definition the same note sequence64 carries, since its string branch is bounded by length rather than by value, and re-wraps the DateTimeArray entry left split mid-phrase by the earlier edit.
Per the spec team decision. All 16 extension ranges are removed, along with the TypedValueExtension message, which existed only to host one, and the extension_value arm of the TypedValue oneof. They had never been used. There is no extend statement anywhere in the specification, Tahu 3.x, Tahu develop or the TCK, and no hand-written call to the extension API in any of them, after three major versions of being reserved. They were not free. Six changes in v4 development moved a floor - typed Commands, the metadata separation, the topic namespace change, the header/content split, TemplateList, and the ValueArray in eclipse-sparkplug#649 - each a theoretical break for any third party occupying the numbers being vacated, for no actual third party. They also did not work in the JSON encoding. The Protocol Buffers JSON Mapping is specified for proto3, which has no extensions, so eclipse-sparkplug#645 had to invent three statements defining the bracketed key form, and an extension still survived a binary round trip but not a JSON one. Those three statements are removed with the mechanism, and the rationales that rested on extension ranges are reworded to rest on the ordinary unknown-field rule instead. array_value takes field number 16, which the extension arm freed, rather than leaving a gap at 16 with the array at 17. Not a wire change for anything that exists: a payload exercising metrics, a DataSet and a null cell serializes byte-identically under both schemas, and each decodes the other's output.
Closes #231.
Defines the JSON payload encoding and binds it to the spJv4.0 namespace, so an Edge Node may publish
either encoding on the wire. Topic structure, operational behavior, sequence numbering and session
management are unchanged and remain encoding independent — only the payload representation differs.
38 new assertions, none removed.
Changes to spCv4.0
These are independent of the JSON work. They are here because the JSON encoding forced the question
of what a 64-bit field actually holds, and surfaced two existing defects.
None of the type changes is a wire break. protobuf encodes
int64anduint64as the samevarint and declares them wire-compatible. For every value both types can hold the bytes are
identical and either schema decodes the other's output — verified at
0, a realistic nanosecondtimestamp, and
2^63-1. What changes is generated source: accessors change type in languages thatdistinguish the two.
The three sequence numbers stay
uint64, but their rollover point moves to9223372036854775807.This is the one genuinely normative change of the three — it rescales the bound in 38 statements
across Chapters 4, 5 and 6, halving the range. An unsigned type is correct for a counter that can
never be negative and keeps
seq,bd_seq, anddescriptor_seqconsistent withalias,MetaData.size, andStatus.code. Capping below the type maximum guarantees every value aconformant Edge Node can produce also fits a signed 64-bit integer, so an implementation in a
language with no unsigned 64-bit type — Java among them — never needs unsigned comparison. Neither
bound is reachable: at a million messages per second the capped range lasts ~292,000 years. The
repeated
(max value of a UINT64)parenthetical is dropped rather than rescaled, since the numberis no longer the type's maximum; the cap is explained once under
seqin the Header section.timestampmoves toint64, in both the Header and the Metric. Every mainstream timerepresentation it interoperates with is signed — protobuf's own
Timestamp, OPC UADateTime,java.time.Instant— so an unsigned timestamp is narrowed at nearly every boundary. The ceilingmoves from 2555 to 2262-04-11; the note from #232 states that date directly rather than
describing a range no mainstream time library exposes.
The
contentoneof is renamedpayload_content, so its uncompressed arm can be namedcontentinstead of
plain. The oneof was namedcontentwhile its arm wasplain, so a Protocol BuffersJSON implementation emitted the key
"plain"while every example in the chapter used"content"—a mismatch dating to #622. A field may not share its oneof's name, so the oneof moves instead. The
examples were right; the schema now agrees with them, and a stock library emits the correct key with
no post-processing. Three prose references to "the plain field" are updated, including the normative
payloads-compression-content-only.Payload metrics MUST now be ordered by value change time, oldest first. A consumer applying
metrics in array order then arrives at the correct current value without comparing timestamps, which
matters when one payload carries several values for the same metric, as a payload of historical
values does. A metric omitting its timestamp sorts at the payload timestamp, which is the time it is
already taken to have changed. Metrics sharing a timestamp may appear in any order and a consumer
must not reorder them — multi-part file publishing depends on that, since it identifies a metric by
position and all of its metrics carry the same timestamp, so they tie. Non-normative comments in
both sections record the interaction.
The NREBIRTH
rsncalculation was off by one. It said:A counter cycling
0..MAXhas periodMAX + 1, so addingMAXlands one short — and with therollover point at
2^63-1,MAX + 1no longer fits a signed 64-bit integer, so the old form cannotbe corrected in place. Restated as two cases, so no intermediate is ever negative or out of range:
Verified equivalent to
(pds - (1 + dc)) mod 2^63over 20,070 cases including every smallcombination and both rollover boundaries.
MetaData.seq, which sequences the parts of a multi-part file transfer, is a different field and isunchanged.
The JSON encoding
It is self-contained
The section defines the encoding completely; conformance does not depend on any document outside
this specification. The scalar rules were chosen to agree with the Protocol Buffers JSON Mapping,
so the forms are familiar and a library produces them directly, but that agreement is a convenience
and not a normative dependency — the mapping is vendor documentation rather than a versioned
standard, it is specified for proto3 while the Sparkplug schema is proto2, and it does not define
extensions at all.
Tahu 3.x's
payload.jsonwas taken as input but not as the source of truth: it disagrees with thecode it validates (
datatype/metadatain the schema vsdataType/metaDatain the Jacksonannotations), names
Template.referencewhere the schema saystemplate_ref, leavesSparkplugBPayload,PropertyValue,PropertySet,MetaDataandRowunannotated so their keysare whatever Jackson's default bean naming produces, never states that
bodyis base64, and has nois_nullor array types.64-bit integers are encoded as JSON strings
RFC 8259 does not require a JSON number to carry more precision than an IEEE 754 double, and most
parsers — including every browser's — decode into a double unconditionally. Since #232 made every
Sparkplug timestamp a nanosecond count, timestamps sit far above the 2^53 limit:
The rule is driven by declared field type, not magnitude. That matters for the sequence numbers:
each counts to the rollover point, so one is small early in a session and large late in it — a
numeric form would be correct in testing and lossy in production.
Deviations from canonical JSON
Each exists so the JSON reads as JSON rather than as a transliteration of a binary format:
valuekeys[]/values[]bytes_valuevaluesarrayThe array one is different in kind. A stock library emits
"value": "Fc1bB7Fo3jo="for anInt32Array, which a JSON consumer cannot read at all without Sparkplug-specific unpacking —defeating the purpose of offering JSON. #642 proposes fixing this at the schema level with native
packed repeated fields, which would be smaller for realistic process data and remove the deviation.
Contents
payloads_json— overview, encoding selection, field naming, scalar mappingtable, extension fields, the structural deviations, a formal JSON Schema (draft 2020-12), a
worked DDATA in both encodings, and non-normative guidance on choosing an encoding.
the ordinary rules for their type, unrecognized extensions ignored. The schema reserves extension
ranges in most messages, so leaving this undefined left something well-defined on the wire
undefined in JSON. Noted: unlike the binary encoding, an unrecognized JSON key does not survive a
round trip. (See also Decide the future of the protobuf extension ranges - unused since Sparkplug B, and undefined in JSON #643 — the ranges have never been used.)
to 9 significant decimal digits for a float, 17 for a double.
NBIRTH, DBIRTH) now require a payload encoded as defined for the namespace it is published on.
Same ids, same requirement, generalized.
spCv4.0apply equally tospJv4.0— one sentence ratherthan editing 122 topic strings.
one encoding and MAY support both selected by configuration, but MUST use a single encoding and
namespace for a whole MQTT session; Host Applications and Sparkplug Aware MQTT Servers MUST support
both. A Sparkplug Compliant MQTT Server is unchanged — it treats payloads as opaque bytes, now
stated non-normatively so the asymmetry does not read as an omission.
Example churn, flagged for review
Two renames so the examples match the encoding rules rather than contradicting them, since readers
will now reasonably read them as JSON documents:
"dataType"→"datatype"(26 occurrences). The protobuf field isdatatype;dataTypewasnever a field name — the same class of error as the
dataType/typefix on parameters in [#602] Add non-normative guidance for Templates #629."template": {...}→"value": {...}(6 occurrences). A Template value is a valuelike any other, and
datatype: "Template"already identifies it.A non-normative comment states that the remaining examples illustrate structure and write 64-bit
values as JSON numbers, which a published spJv4.0 payload does not.
On the "protobuf is smaller and cheaper" claim
The issue asked whether this is actually true. Measured by compiling the spec's own schema with
protocand benchmarking against equivalent JSON (protobufupbC backend vs CPython's Cjson):Both halves hold, and the CPU gap is wider than the size gap. But the size advantage nearly
vanishes under compression — deflate eats exactly the repeated field names that make JSON large:
The non-normative guidance reflects this rather than the folklore: on a constrained link that
already compresses, size is not a strong reason to prefer one encoding; encode/decode cost still is.
Verification
protocat every commit, and theint64/uint64wire equivalence was verified by encoding the same values under both declarations and diffing.
negative-tested against 15 cases (9 malformed, 6 valid) — all behaved as specified. Those tests
caught two real bugs while drafting: a
oneOfthat rejected"NaN"because it matched both thestring and non-finite-number branches, and the
template/valuekey conflict above.rsnformula was checked exhaustively against modular arithmetic.Known gaps — decisions for the working group
DateTimemoved toint64too, in c0781dc, so a DateTime value and a payload timestamp nolonger differ in signedness. Unlike the field type changes above this one is a wire change: a
DateTime value moves from the
uint64_valuearm of theTypedValueoneof toint64_value, adifferent field number. It is harmless only because spCv4.0 has not shipped.
STATE carries a nanosecond timestamp as a JSON number. STATE is JSON but explicitly not a
Sparkplug payload, so the string rule does not reach it. The example in the spec is safe only by
luck —
1486144502122000000ends in six zeros and is exactly representable — but a real value isnot:
Every Host Application publishing STATE with genuine sub-millisecond precision has the defect this
PR documents, and
bd_seqacquires it as it grows. Fixing it changes STATE's wire format andtouches every Host Application, so it is not folded in here.
Addressed in review
longer waits on a
spJv4.0/STATEtopic that is never published.Edge Node subscribes to its command topics on the namespace it publishes on. Without this a
Rebirth Request to a JSON Edge Node was accepted by the Server and never delivered.
uint64foralias,MetaData.sizeandStatus.code, which are uncapped, andsequence64for the sequence numbers and timestamps. Thesingle capped definition rejected legal spCv4.0 values and contradicted the lossless conversion
claim.
timestamps, the only payload-touching behavior required of it.
datatypeandis_nullmust agree, the single JSON key representsboth, and a null Template.Parameter now has a JSON form.
with a SHOULD elsewhere, and does not reach Template or Command metrics.
Not addressed
spBv1.0namespace, sono v4 payload requirement can be tested at all until
SparkplugBProto.javais regenerated. These38 assertions have no coverage, and dual-encoding roughly doubles the payload test surface once
that is done.
simplify this section if resolved.
JSON mapping for
DataSet.Columnhere was written against the schema, which is the authoritativewire format. Fixed separately on the
632branch; this PR does not depend on it.