diff --git a/specification/src/main/asciidoc/chapters/Sparkplug_10_Conformance.adoc b/specification/src/main/asciidoc/chapters/Sparkplug_10_Conformance.adoc index 5c3475f9..fa9475c4 100644 --- a/specification/src/main/asciidoc/chapters/Sparkplug_10_Conformance.adoc +++ b/specification/src/main/asciidoc/chapters/Sparkplug_10_Conformance.adoc @@ -35,8 +35,28 @@ typically be both an Edge Node and a Host Application. ==== Sparkplug Edge Node A Sparkplug Edge Node is typically an 'Edge Gateway'. It sends and receives data to an MQTT Server -using the spCv4.0/# namespace. Edge Nodes typically interact with physical devices to gather data -and also write to device outputs. +using a Sparkplug namespace, either spCv4.0/# or spJv4.0/#. Edge Nodes typically interact with +physical devices to gather data and also write to device outputs. + +Sparkplug defines two payload encodings, as described in the +link:#payloads_json_encoding_selection[Encoding Selection] section. An Edge Node is not required to +implement both. + +* [tck-testable tck-id-conformance-edge-node-encoding-one]#[yellow-background]*[tck-id-conformance-edge-node-encoding-one] A +Sparkplug Edge Node MUST support at least one of the two Sparkplug payload encodings, and MUST +publish on the namespace that corresponds to the encoding it uses.*# +* [tck-not-testable tck-id-conformance-edge-node-encoding-both]#[yellow-background]*[tck-id-conformance-edge-node-encoding-both] A +Sparkplug Edge Node MAY support both payload encodings, in which case the encoding it uses is +selected by configuration.*# +* [tck-testable tck-id-conformance-edge-node-encoding-session]#[yellow-background]*[tck-id-conformance-edge-node-encoding-session] A +Sparkplug Edge Node MUST publish every message of an MQTT session using a single payload encoding on +the corresponding single namespace.*# +* [tck-testable tck-id-conformance-edge-node-encoding-subscribe]#[yellow-background]*[tck-id-conformance-edge-node-encoding-subscribe] A +Sparkplug Edge Node MUST subscribe to its command topics on the same namespace it publishes on, so +that a Host Application addressing it on that namespace reaches it.*# +** Non-normative comment: an Edge Node has one sequence number stream, so publishing part of a +session on one namespace and part on the other would leave a consumer of either namespace with gaps +in that stream. Support for strongly typed commands is optional. An Edge Node that exposes commands must do so in a conformant manner. @@ -74,6 +94,28 @@ NREBIRTH, and NDEATH of every Edge Node reporting a Device of interest, which ca state that encapsulates the state of its Devices) and its use of the descriptor sequence number in place of the Edge-Node-wide sequence number. +A Sparkplug Host Application consumes data produced by Edge Nodes that it does not control, and so +cannot assume which payload encoding those Edge Nodes use. + +* [tck-testable tck-id-conformance-host-encoding-both]#[yellow-background]*[tck-id-conformance-host-encoding-both] A +Sparkplug Host Application MUST support both Sparkplug payload encodings, and MUST be able to +consume messages on both the spCv4.0 and spJv4.0 namespaces.*# +* [tck-testable tck-id-conformance-host-encoding-publish]#[yellow-background]*[tck-id-conformance-host-encoding-publish] When +a Sparkplug Host Application publishes a message, it MUST encode that message using the encoding +that corresponds to the namespace it publishes on.*# +* [tck-testable tck-id-conformance-host-encoding-address]#[yellow-background]*[tck-id-conformance-host-encoding-address] When +a Sparkplug Host Application publishes a message addressed to a particular Edge Node or Device, +including an NCMD, a DCMD, and a Rebirth Request, it MUST publish on the namespace of that Edge +Node's current NBIRTH.*# +** Non-normative comment: a Host Application supports both encodings, so without this it could +address an Edge Node on a namespace that Edge Node does not subscribe to. The message would be +accepted by the MQTT Server and never delivered, and a Rebirth Request in particular would fail +silently, leaving the Host unable to recover its view of that Edge Node. +** Non-normative comment: which namespaces a Host Application subscribes to remains a deployment +decision. The requirement is that it is capable of decoding either, so that an Edge Node's choice of +encoding does not determine which Host Applications can consume it. This is the property that makes +the two encodings interoperable rather than two separate ecosystems. + [tck-testable tck-id-conformance-primary-host]#[yellow-background]*[tck-id-conformance-primary-host] Sparkplug Host Applications MUST publish 'STATE' messages that represent its Birth and Death Certificates.*# @@ -116,6 +158,10 @@ Sparkplug conformant MQTT Server MUST support all aspects of Will Messages inclu * [tck-testable tck-id-conformance-mqtt-retained]#[yellow-background]*[tck-id-conformance-mqtt-retained] A Sparkplug conformant MQTT Server MUST support all aspects of the 'retain flag'*# +Non-normative comment: a Sparkplug Compliant MQTT Server has no payload requirements of any kind. It +treats a Sparkplug payload as an opaque sequence of bytes, so both payload encodings are carried by +the same server with no additional support. + [[conformance_sparkplug_aware_mqtt_server]] ==== Sparkplug Aware MQTT Server @@ -132,6 +178,14 @@ continue to be published with the MQTT retain flag set to false and are not reta * [tck-testable tck-id-conformance-mqtt-aware-basic]#[yellow-background]*[tck-id-conformance-mqtt-aware-basic] A Sparkplug Aware MQTT Server MUST support all aspects of a Sparkplug Compliant MQTT Server*# +* [tck-testable tck-id-conformance-mqtt-aware-encoding-both]#[yellow-background]*[tck-id-conformance-mqtt-aware-encoding-both] A +Sparkplug Aware MQTT Server that replaces NDEATH timestamps MUST do so for payloads in both +encodings, on both the spCv4.0 and spJv4.0 namespaces.*# +** Non-normative comment: replacing an NDEATH timestamp is the only behavior required of a +Sparkplug Aware MQTT Server that decodes a payload at all, and it is itself optional. A Server that +does not do it never inspects a payload and needs neither encoding. The requirement is therefore +conditional, so that offering the timestamp behavior for one encoding only cannot leave Edge Nodes +on the other namespace silently unserved. * [tck-testable tck-id-conformance-mqtt-aware-ndeath-timestamp]#[yellow-background]*[tck-id-conformance-mqtt-aware-ndeath-timestamp] A Sparkplug Aware MQTT Server MAY replace the timestmap of NDEATH messages. If it does, it MUST set the timestamp to the UTC time at which it attempts to deliver the NDEATH to subscribed clients*# diff --git a/specification/src/main/asciidoc/chapters/Sparkplug_4_Topics.adoc b/specification/src/main/asciidoc/chapters/Sparkplug_4_Topics.adoc index 1593157c..6bad2e4a 100644 --- a/specification/src/main/asciidoc/chapters/Sparkplug_4_Topics.adoc +++ b/specification/src/main/asciidoc/chapters/Sparkplug_4_Topics.adoc @@ -72,6 +72,24 @@ and spJv4.0 namespaces, which differ only in payload encoding. For earlier 'Spar 'Sparkplug B' topic namespace definitions and concepts, see the earlier versions of the Sparkplug Specification. +Non-normative comment: Topics are written with the spCv4.0 namespace element throughout the +remainder of this specification. Every such topic applies equally to the spJv4.0 namespace with +that element substituted, because the two namespaces share one topic structure. The encoding of the +payload carried on a topic is determined by the namespace element of that topic, as required in the +link:#payloads_json_encoding_selection[Encoding Selection] section. + +[tck-testable tck-id-topics-state-single-namespace]#[yellow-background]*[tck-id-topics-state-single-namespace] The +STATE topic is the one exception. A STATE message MUST be published on +'spCv4.0/STATE/sparkplug_host_id' whatever payload encoding the Edge Nodes and Host Applications in +that infrastructure use, and an Edge Node that waits on a Primary Host Application MUST subscribe +to that topic rather than to a spJv4.0 equivalent.*# + +Non-normative comment: STATE carries a JSON document rather than a Sparkplug payload, so there is +nothing for a second namespace to encode differently. Publishing it on one topic also keeps a +single Will Message per Host Application. Without this rule a JSON Edge Node would wait on +spJv4.0/STATE while the Host published on spCv4.0/STATE, and would never see the Host come +online. + [[topics_topic_prefix_element]] ==== topic_prefix Element @@ -324,7 +342,8 @@ definitions MUST be published in the NMETA.*# include a sequence number. When the NMETA is published at the start of a BIRTH sequence it is the first message and its sequence number MUST be 0. When the NMETA is published as an update (accompanying an NMMETA), its sequence number MUST be one greater than the sequence number of the -previous message from this Edge Node, unless that message contained a value of 18446744073709551615 (max value of a UINT64), in which case the +previous message from this Edge Node, unless that message contained a value of 9223372036854775807, +in which case the sequence number MUST be 0.*# * [tck-testable tck-id-topics-nmeta-bdseq]#[yellow-background]*[tck-id-topics-nmeta-bdseq] The NMETA MUST include a bd_seq number matching the bd_seq of the current MQTT session.*# @@ -368,7 +387,8 @@ NMMETA MUST include only the metrics affected by the change, and each metric MUS of ADD, UPDATE, or DELETE.*# * [tck-testable tck-id-topics-nmmeta-seq]#[yellow-background]*[tck-id-topics-nmmeta-seq] The NMMETA MUST include a sequence number that is one greater than the sequence number of the previous message -published by this Edge Node, unless that message contained a value of 18446744073709551615 (max value of a UINT64), in which case the sequence +published by this Edge Node, unless that message contained a value of 9223372036854775807, in which +case the sequence number MUST be 0.*# * [tck-testable tck-id-topics-nmmeta-bdseq]#[yellow-background]*[tck-id-topics-nmmeta-bdseq] The NMMETA MUST include a bd_seq number matching the bd_seq of the current MQTT session.*# @@ -433,7 +453,7 @@ immediately prior MQTT CONNECT packet's Will Message.*# matching bd_seq number allows Host Applications to correlate NBIRTHs to NDEATHs.*# * [tck-testable tck-id-topics-nbirth-bdseq-increment]#[yellow-background]*[tck-id-topics-nbirth-bdseq-increment] The bd_seq number MUST start at zero and increment by one on every new MQTT CONNECT packet until it -reaches a maximum of 18446744073709551615 (max value of a UINT64). At this point, the following +reaches a maximum of 9223372036854775807. At this point, the following bd_seq number MUST be zero.*# The NBIRTH message can also include additional Node Control payload components. These are used by a @@ -528,7 +548,7 @@ payload.*# ** This allows Host Applications to correlate NREBIRTHs to NDEATHs. * [tck-testable tck-id-topics-nrebirth-bdseq-increment]#[yellow-background]*[tck-id-topics-nrebirth-bdseq-increment] The bd_seq number MUST start at zero and increment by one on every new MQTT CONNECT packet until it -reaches a maximum of 18446744073709551615 (max value of a UINT64). At this point, the following +reaches a maximum of 9223372036854775807. At this point, the following bd_seq number MUST be zero.*# The NREBIRTH message can also include additional Node Control payload components. These are used by @@ -588,7 +608,7 @@ messages MUST be published with MQTT QoS equal to 0 and retain equal to false.*# * [tck-testable tck-id-topics-ndata-seq-num]#[yellow-background]*[tck-id-topics-ndata-seq-num] The NDATA MUST include a sequence number in the payload and it MUST have a value of one greater than the previous MQTT message from the Edge Node contained unless the previous MQTT message contained a -value of 18446744073709551615 (max value of a UINT64). In this case the sequence number MUST be 0.*# +value of 9223372036854775807. In this case the sequence number MUST be 0.*# * [tck-testable tck-id-topics-ndata-timestamp-1]#[yellow-background]*[tck-id-topics-ndata-timestamp-1] The NDATA MUST include the overall payload timestamp denoting the date and time the message was sent from the Edge Node.*# @@ -750,7 +770,8 @@ metadata MAY also be included. Aliases are optional.*# DMETA MUST NOT be used to convey current metric values.*# * [tck-testable tck-id-topics-dmeta-seq]#[yellow-background]*[tck-id-topics-dmeta-seq] The DMETA MUST include a sequence number that is one greater than the sequence number of the previous message from -this Edge Node, unless that message contained a value of 18446744073709551615 (max value of a UINT64), in which case the sequence number MUST +this Edge Node, unless that message contained a value of 9223372036854775807, in which case the +sequence number MUST be 0. A DMETA is never the first message of a BIRTH sequence, as it follows the NMETA and NBIRTH.*# * [tck-testable tck-id-topics-dmeta-bdseq]#[yellow-background]*[tck-id-topics-dmeta-bdseq] The DMETA MUST include a bd_seq number matching the bd_seq of the current MQTT session.*# @@ -792,7 +813,8 @@ DMMETA MUST include only the metrics affected by the change, and each metric MUS of ADD, UPDATE, or DELETE.*# * [tck-testable tck-id-topics-dmmeta-seq]#[yellow-background]*[tck-id-topics-dmmeta-seq] The DMMETA MUST include a sequence number that is one greater than the sequence number of the previous message -published by this Edge Node, unless that message contained a value of 18446744073709551615 (max value of a UINT64), in which case the sequence +published by this Edge Node, unless that message contained a value of 9223372036854775807, in which +case the sequence number MUST be 0.*# * [tck-testable tck-id-topics-dmmeta-bdseq]#[yellow-background]*[tck-id-topics-dmmeta-bdseq] The DMMETA MUST include a bd_seq number matching the bd_seq of the current MQTT session.*# @@ -834,7 +856,7 @@ messages MUST be published with MQTT QoS equal to 0 and retain equal to false.*# * [tck-testable tck-id-topics-dbirth-seq]#[yellow-background]*[tck-id-topics-dbirth-seq] The DBIRTH MUST include a sequence number in the payload and it MUST have a value of one greater than the previous MQTT message from the Edge Node contained unless the previous MQTT message contained a -value of 18446744073709551615 (max value of a UINT64). In this case the sequence number MUST be 0.*# +value of 9223372036854775807. In this case the sequence number MUST be 0.*# * [tck-testable tck-id-topics-dbirth-timestamp-1]#[yellow-background]*[tck-id-topics-dbirth-timestamp-1] The DBIRTH MUST include the overall payload timestamp denoting the date and time the message was sent from the Edge Node.*# @@ -922,7 +944,7 @@ messages MUST be published with MQTT QoS equal to 0 and retain equal to false.*# * [tck-testable tck-id-topics-drebirth-seq]#[yellow-background]*[tck-id-topics-drebirth-seq] The DREBIRTH MUST include a sequence number in the payload and it MUST have a value of one greater than the previous MQTT message from the Edge Node contained unless the previous MQTT message contained a -value of 18446744073709551615 (max value of a UINT64). In this case the sequence number MUST be 0.*# +value of 9223372036854775807. In this case the sequence number MUST be 0.*# * [tck-testable tck-id-topics-drebirth-timestamp-1]#[yellow-background]*[tck-id-topics-drebirth-timestamp-1] The DREBIRTH MUST include the overall payload timestamp denoting the date and time the message was sent from the Edge Node.*# @@ -1000,7 +1022,7 @@ messages MUST be published with MQTT QoS equal to 0 and retain equal to false.*# * [tck-testable tck-id-topics-ddata-seq-num]#[yellow-background]*[tck-id-topics-ddata-seq-num] The DDATA MUST include a sequence number in the payload and it MUST have a value of one greater than the previous MQTT message from the Edge Node contained unless the previous MQTT message contained a -value of 18446744073709551615 (max value of a UINT64). In this case the sequence number MUST be 0.*# +value of 9223372036854775807. In this case the sequence number MUST be 0.*# * [tck-testable tck-id-topics-ddata-timestamp-1]#[yellow-background]*[tck-id-topics-ddata-timestamp-1] The DDATA MUST include the overall payload timestamp denoting the date and time the message was sent from the Edge Node.*# @@ -1046,7 +1068,7 @@ messages MUST be published with MQTT QoS equal to 0 and retain equal to false.*# * [tck-testable tck-id-topics-ddeath-seq-num]#[yellow-background]*[tck-id-topics-ddeath-seq-num] The DDEATH MUST include a sequence number in the payload and it MUST have a value of one greater than the previous MQTT message from the Edge Node contained unless the previous MQTT message contained a -value of 18446744073709551615 (max value of a UINT64). In this case the sequence number MUST be 0.*# +value of 9223372036854775807. In this case the sequence number MUST be 0.*# * [tck-testable tck-id-topics-ddeath-bdseq]#[yellow-background]*[tck-id-topics-ddeath-bdseq] The DDEATH MUST include a bd_seq number matching the bd_seq of the current MQTT session.*# diff --git a/specification/src/main/asciidoc/chapters/Sparkplug_5_Operational_Behavior.adoc b/specification/src/main/asciidoc/chapters/Sparkplug_5_Operational_Behavior.adoc index c5a6d539..2ab161a2 100644 --- a/specification/src/main/asciidoc/chapters/Sparkplug_5_Operational_Behavior.adoc +++ b/specification/src/main/asciidoc/chapters/Sparkplug_5_Operational_Behavior.adoc @@ -66,12 +66,12 @@ an implementation to have a nanosecond resolution time source. An implementation is coarser than one nanosecond expresses its timestamps in nanoseconds all the same, scaled to that denomination. For example, an implementation whose time source has a resolution of one millisecond expresses a timestamp as the number of milliseconds since epoch multiplied by 1000000. -** Non-normative comment: an unsigned 64-bit integer of nanoseconds since epoch covers a range of -approximately 584 years, reaching the year 2554. Implementers should note that this range is only -available to a genuinely unsigned representation. The Google Protocol Buffers uint64 type is -represented as a signed 64-bit long in Java, and several widely used nanosecond time representations -are signed 64-bit as well, which reduces the reachable range to approximately 292 years, ending on -2262-04-11. This is the practical ceiling for such implementations. +** Non-normative comment: a signed 64-bit integer of nanoseconds since epoch covers a range of +approximately 292 years, ending on 2262-04-11, which is the ceiling for a Sparkplug timestamp. A +signed type is specified rather than an unsigned one because the Google Protocol Buffers uint64 +type is represented as a signed 64-bit long in Java, and several widely used nanosecond time +representations are signed 64-bit as well, so the upper half of an unsigned range was never +reachable in practice. Non-normative comment: the resolution of a timestamp is not the same as its accuracy. Denominating a timestamp in nanoseconds says how finely a value can be expressed, not how closely it corresponds to @@ -470,12 +470,13 @@ Edge Node's MQTT Will Message's topic MUST be of the form '[topic_prefix/]spCv4. where the optional topic_prefix is the Sparkplug Topic Prefix and the edge_node_id is the Sparkplug Edge Node ID for this Edge Node*# * [tck-testable tck-id-message-flow-edge-node-birth-publish-will-message-payload]#[yellow-background]*[tck-id-message-flow-edge-node-birth-publish-will-message-payload] The -Edge Node's MQTT Will Message's payload MUST be a Sparkplug Google Protobuf encoded payload.*# +Edge Node's MQTT Will Message's payload MUST be a Sparkplug payload encoded as defined for the +namespace on which it is published.*# * [tck-testable tck-id-message-flow-edge-node-birth-publish-will-message-payload-bdSeq]#[yellow-background]*[tck-id-message-flow-edge-node-birth-publish-will-message-payload-bdSeq] The Edge Node's MQTT Will Message's payload MUST include a bd_seq number in the payload and the value MUST be incremented by one from the value in the previous MQTT CONNECT packet unless the value would -be greater than 18446744073709551615 (max value of a UINT64). If in the previous NBIRTH a value of -18446744073709551615 was sent, the next MQTT Connect packet Will Message payload bd_seq number value +be greater than 9223372036854775807. If in the previous NBIRTH a value of +9223372036854775807 was sent, the next MQTT Connect packet Will Message payload bd_seq number value MUST have a value of 0.*# * [tck-testable tck-id-message-flow-edge-node-birth-publish-will-message-qos]#[yellow-background]*[tck-id-message-flow-edge-node-birth-publish-will-message-qos] The Edge Node's MQTT Will Message's MQTT QoS MUST be 1.*# @@ -566,7 +567,8 @@ Edge Node's NBIRTH MQTT topic MUST be of the form '[topic_prefix/]spCv4.0/NBIRTH/edge_node_id' where the optional topic_prefix is the Sparkplug Topic Prefix and the edge_node_id is the Sparkplug Edge Node ID for this Edge Node*# * [tck-testable tck-id-message-flow-edge-node-birth-publish-nbirth-payload]#[yellow-background]*[tck-id-message-flow-edge-node-birth-publish-nbirth-payload] The -Edge Node's NBIRTH payload MUST be a Sparkplug Google Protobuf encoded payload.*# +Edge Node's NBIRTH payload MUST be a Sparkplug payload encoded as defined for the namespace on +which it is published.*# * [tck-testable tck-id-message-flow-edge-node-birth-publish-nbirth-payload-bdSeq]#[yellow-background]*[tck-id-message-flow-edge-node-birth-publish-nbirth-payload-bdSeq] The Edge Node's NBIRTH payload MUST include a bd_seq number in the payload and the value MUST be the same as the bd_seq number value specified in the immediately previous MQTT CONNECT packet.*# @@ -575,11 +577,11 @@ Edge Node's NBIRTH MQTT QoS MUST be 0.*# * [tck-testable tck-id-message-flow-edge-node-birth-publish-nbirth-retained]#[yellow-background]*[tck-id-message-flow-edge-node-birth-publish-nbirth-retained] The Edge Node's NBIRTH retained flag MUST be set to false.*# * [tck-testable tck-id-message-flow-edge-node-birth-publish-nbirth-payload-seq]#[yellow-background]*[tck-id-message-flow-edge-node-birth-publish-nbirth-payload-seq] The -Edge Node's NBIRTH payload MUST include a 'seq' number that is between 0 and 18446744073709551615 +Edge Node's NBIRTH payload MUST include a 'seq' number that is between 0 and 9223372036854775807 (inclusive).*# ** This will become the starting sequence number which all following messages will include a -sequence number that is one more than the previous up to 18446744073709551615 (max value of a -UINT64) where it wraps back to zero. +sequence number that is one more than the previous up to 9223372036854775807 where it wraps back to +zero. * [tck-testable tck-id-message-flow-edge-node-birth-publish-phid-offline]#[yellow-background]*[tck-id-message-flow-edge-node-birth-publish-phid-offline] If the Edge Node is configured to wait for a Primary Host Application, it is connected to the MQTT Server, and receives a STATE message on its configured Primary Host, the timestamp value in the @@ -851,13 +853,14 @@ identify the Sparkplug Device.*# Device's DBIRTH MQTT topic topic_prefix and edge_node_id MUST match the topic_prefix and edge_node_id that were sent in the prior NBIRTH message for the Edge Node this Device is associated with.*# * [tck-testable tck-id-message-flow-device-birth-publish-dbirth-payload]#[yellow-background]*[tck-id-message-flow-device-birth-publish-dbirth-payload] The -Device's DBIRTH payload MUST be a Sparkplug Google Protobuf encoded payload.*# +Device's DBIRTH payload MUST be a Sparkplug payload encoded as defined for the namespace on which +it is published.*# * [tck-testable tck-id-message-flow-device-birth-publish-dbirth-qos]#[yellow-background]*[tck-id-message-flow-device-birth-publish-dbirth-qos] The Device's DBIRTH MQTT QoS MUST be 0.*# * [tck-testable tck-id-message-flow-device-birth-publish-dbirth-retained]#[yellow-background]*[tck-id-message-flow-device-birth-publish-dbirth-retained] The Device's DBIRTH retained flag MUST be set to false.*# * [tck-testable tck-id-message-flow-device-birth-publish-dbirth-payload-seq]#[yellow-background]*[tck-id-message-flow-device-birth-publish-dbirth-payload-seq] The -Device's DBIRTH payload MUST include a 'seq' number that is between 0 and 18446744073709551615 +Device's DBIRTH payload MUST include a 'seq' number that is between 0 and 9223372036854775807 (inclusive) and be one more than was included in the prior Sparkplug message sent from the Edge Node associated with this Device.*# @@ -1262,7 +1265,7 @@ values stop making sense. A Host Application also has to establish where in a descriptor sequence stream it is starting from. A BIRTH begins a stream and carries a descriptor_seq of 0. A REBIRTH does not begin a stream; it slip-streams the Host Application into a stream that is already running and carries the next -incrementing value, which is only 0 in the case of a UINT64 rollover. +incrementing value, which is only 0 in the case of a sequence number rollover. * [tck-testable tck-id-operational-behavior-host-partial-descriptor-seq-anchor]#[yellow-background]*[tck-id-operational-behavior-host-partial-descriptor-seq-anchor] A Host Application MUST anchor its expected descriptor_seq for a Sparkplug Descriptor on the @@ -1743,9 +1746,10 @@ Application getting in sync with the data stream. NREBIRTH payload MUST account for sequence number rollover if required based on the next pending DATA message and the number of devices under the scope of the Edge Node.*# ** For example, if the next pending DATA message sequence number will be 1 and the Edge Node has -two devices, the NBIRTHs sequence number value must be 18446744073709551614. This means the sequence -numbers will be: NBIRTH: 18446744073709551614, DBIRTH 1: 18446744073709551615, DBIRTH 2: 0, DATA: 1 -** The above example is based on the fact that the max value of a UINT64 is 18446744073709551615 +two devices, the NBIRTHs sequence number value must be 9223372036854775806. This means the sequence +numbers will be: NBIRTH: 9223372036854775806, DBIRTH 1: 9223372036854775807, DBIRTH 2: 0, DATA: 1 +** The above example is based on the fact that the maximum Sparkplug sequence number is +9223372036854775807 * [tck-testable tck-id-operational-behavior-rebirth-action-6]#[yellow-background]*[tck-id-operational-behavior-rebirth-action-6] Each DREBIRTH payload MUST use a seq number that is incremented by one from the previous message.*# * [tck-testable tck-id-operational-behavior-rebirth-action-7]#[yellow-background]*[tck-id-operational-behavior-rebirth-action-7] Each diff --git a/specification/src/main/asciidoc/chapters/Sparkplug_6_Payloads.adoc b/specification/src/main/asciidoc/chapters/Sparkplug_6_Payloads.adoc index c52650ac..de882fcd 100644 --- a/specification/src/main/asciidoc/chapters/Sparkplug_6_Payloads.adoc +++ b/specification/src/main/asciidoc/chapters/Sparkplug_6_Payloads.adoc @@ -122,16 +122,25 @@ being requested. The entire Google Protocol Buffers definition is below. // To compile: // cd client_libraries/java // protoc --proto_path=../../ --java_out=src/main/java ../../sparkplug_c.proto +// Requires protoc 27 or later and a Protocol Buffers runtime that supports edition 2023 +// (protobuf-java 4.27 or later, Python protobuf 5.27 or later, and equivalents). // -syntax = "proto2"; +edition = "2023"; package org.eclipse.tahu.protobuf; option java_package = "org.eclipse.tahu.protobuf"; option java_outer_classname = "SparkplugBProto"; +// Sparkplug string fields are UTF-8, so a decoder validates them. This is the edition 2023 +// default, stated explicitly because it differs from proto2, which did not validate. The Java +// runtime's own utf8_validation feature defaults to respecting this setting, so every language +// accepts or rejects the same payloads without a Java specific option here. +option features.utf8_validation = VERIFY; + enum DataType { + option features.enum_type = CLOSED; // Indexes of Data Types // Unknown placeholder for future expansion. @@ -186,6 +195,7 @@ enum DataType { // Per-metric operation, used only within NMMETA/DMMETA (modified metadata) messages to denote a // create/update/delete of a metric definition. Absent in NMETA/DMETA/NBIRTH/DBIRTH messages. enum MetricOperation { + option features.enum_type = CLOSED; ADD = 1; UPDATE = 2; DELETE = 3; @@ -194,24 +204,24 @@ enum MetricOperation { message Payload { message TypedValue { - optional DataType datatype = 1; - optional bool is_null = 2; + DataType datatype = 1; + bool is_null = 2; oneof value { - int32 int32_value = 3; - int64 int64_value = 4; - uint32 uint32_value = 5; - uint64 uint64_value = 6; - float float_value = 7; - double double_value = 8; - bool boolean_value = 9; - string string_value = 10; - bytes bytes_value = 11; - DataSet dataset_value = 12; - Template template_value = 13; - PropertySet propertyset_value = 14; - PropertySetList propertysets_value = 15; - TypedValueExtension extension_value = 16; + int32 int32_value = 3; + int64 int64_value = 4; + uint32 uint32_value = 5; + uint64 uint64_value = 6; + float float_value = 7; + double double_value = 8; + bool boolean_value = 9; + string string_value = 10; + bytes bytes_value = 11; + DataSet dataset_value = 12; + Template template_value = 13; + PropertySet propertyset_value = 14; + PropertySetList propertysets_value = 15; + TypedValueExtension extension_value = 16; } message TypedValueExtension { @@ -224,57 +234,57 @@ message Payload { message Template { message Parameter { - optional string name = 1; - optional TypedValue typed_value = 2; + string name = 1; + TypedValue typed_value = 2; extensions 3 to max; } - optional string version = 1; // The version of the Template to prevent mismatches - repeated Metric metrics = 2; // Each metric includes a name, datatype, and optionally a value - repeated Parameter parameters = 3; - optional string template_ref = 4; // MUST be a reference to a template definition if this is an instance (i.e. the name of the template definition) - MUST be omitted for template definitions - optional bool is_definition = 5; + string version = 1; // The version of the Template to prevent mismatches + repeated Metric metrics = 2; // Each metric includes a name, datatype, and optionally a value + repeated Parameter parameters = 3; + string template_ref = 4; // MUST be a reference to a template definition if this is an instance (i.e. the name of the template definition) - MUST be omitted for template definitions + bool is_definition = 5; extensions 6 to max; } message Status { - optional uint64 code = 1; // Result code of a command invocation - 0 indicates success, any non-zero value indicates an error - optional string message = 2; // Optional human readable status/error description + uint64 code = 1; // Result code of a command invocation - 0 indicates success, any non-zero value indicates an error + string message = 2; // Optional human readable status/error description extensions 3 to max; } message Command { - repeated Metric arguments = 1; // Input argument definitions (NBIRTH/DBIRTH) or supplied argument values (NCMD/DCMD) - omitted for commands that accept no inputs - repeated Metric results = 2; // Result/output argument definitions (NBIRTH/DBIRTH) or returned result values (NRESP/DRESP) - omitted for commands that return no outputs - optional string correlation_id = 3; // Set on an NCMD/DCMD request and echoed unchanged in the corresponding NRESP/DRESP response - optional Status status = 4; // Execution status - present in NRESP/DRESP responses, omitted in BIRTH definitions and CMD requests + repeated Metric arguments = 1; // Input argument definitions (NBIRTH/DBIRTH) or supplied argument values (NCMD/DCMD) - omitted for commands that accept no inputs + repeated Metric results = 2; // Result/output argument definitions (NBIRTH/DBIRTH) or returned result values (NRESP/DRESP) - omitted for commands that return no outputs + string correlation_id = 3; // Set on an NCMD/DCMD request and echoed unchanged in the corresponding NRESP/DRESP response + Status status = 4; // Execution status - present in NRESP/DRESP responses, omitted in BIRTH definitions and CMD requests extensions 5 to max; } message DataSet { message Column { - optional string name = 1; - optional DataType datatype = 2; + string name = 1; + DataType datatype = 2; // Only one type of repeated value shall be used per Column - repeated uint32 int_values = 3 [packed=true]; - repeated uint64 long_values = 4 [packed=true]; - repeated float float_values = 5 [packed=true]; - repeated double double_values = 6 [packed=true]; - repeated bool boolean_values = 7 [packed=true]; - repeated string string_values = 8 [packed=false]; + repeated uint32 int_values = 3; + repeated uint64 long_values = 4; + repeated float float_values = 5; + repeated double double_values = 6; + repeated bool boolean_values = 7; + repeated string string_values = 8; extensions 9 to max; } - repeated Column columns = 1; + repeated Column columns = 1; extensions 2 to max; } message PropertySet { - repeated string keys = 1; // Names of the properties - repeated TypedValue values = 2; // The Property Values + repeated string keys = 1; // Names of the properties + repeated TypedValue values = 2; // The Property Values extensions 3 to max; } @@ -285,21 +295,21 @@ message Payload { message MetaData { // Bytes specific metadata - optional bool is_multi_part = 1; + bool is_multi_part = 1; // General metadata - optional string content_type = 2; // Content/Media type - optional uint64 size = 3; // File size, String size, Multi-part size, etc - optional uint64 seq = 4; // Sequence number for multi-part messages + string content_type = 2; // Content/Media type + uint64 size = 3; // File size, String size, Multi-part size, etc + uint64 seq = 4; // Sequence number for multi-part messages // File metadata - optional string file_name = 5; // File name - optional string file_type = 6; // File type (i.e. xml, json, txt, cpp, etc) - optional string md5 = 7; // md5 of data + string file_name = 5; // File name + string file_type = 6; // File type (i.e. xml, json, txt, cpp, etc) + string md5 = 7; // md5 of data // Others - optional string description = 8; // Could be anything such as json or xml of custom properties - optional string engUnit = 9; // Engineering units + string description = 8; // Could be anything such as json or xml of custom properties + string engUnit = 9; // Engineering units // Future expansion extensions 10 to max; @@ -307,48 +317,48 @@ message Payload { message Metric { - optional string name = 1; // Metric name - Required in BIRTH messages, optional in DATA messages if using aliases - optional uint64 alias = 2; // Metric alias - tied to name on BIRTH and included in all later DATA messages - optional uint64 timestamp = 3; // Metric value change time - the time this metric's value last changed - if omitted, the overall payload timestamp is assumed as the metric value change time - optional DataType datatype = 4; // DataType of the metric/tag value - optional uint32 quality = 5; // Quality code of the metric/tag value - severity in bits 30-31 - optional string quality_context = 6; // The optional quality context of the metric/tag value - optional string source = 7; // Source for CMD messages if unique to the Metric (typically for forwarded writes in CMD messages) - optional bool is_historical = 8; // If this is historical data and should not update real time tag - optional bool is_transient = 9; // Tells consuming clients such as MQTT Engine to not store this as a tag - optional bool is_null = 10; // If this is null - explicitly say so rather than using -1, false, etc for some datatypes. - optional MetaData metadata = 11; // Metadata for the Metric - optional PropertySet properties = 12; // Properties of the Metric - optional TypedValue value = 13; - optional Command command = 14; // Command definition (BIRTH), invocation (CMD), or response (RESP) - present when datatype == Command instead of 'value' - optional MetricOperation operation = 15; // Only in NMMETA/DMMETA: ADD, UPDATE, or DELETE + string name = 1; // Metric name - Required in BIRTH messages, optional in DATA messages if using aliases + uint64 alias = 2; // Metric alias - tied to name on BIRTH and included in all later DATA messages + int64 timestamp = 3; // Metric value change time - the time this metric's value last changed - if omitted, the overall payload timestamp is assumed as the metric value change time + DataType datatype = 4; // DataType of the metric/tag value + uint32 quality = 5; // Quality code of the metric/tag value - severity in bits 30-31 + string quality_context = 6; // The optional quality context of the metric/tag value + string source = 7; // Source for CMD messages if unique to the Metric (typically for forwarded writes in CMD messages) + bool is_historical = 8; // If this is historical data and should not update real time tag + bool is_transient = 9; // Tells consuming clients such as MQTT Engine to not store this as a tag + bool is_null = 10; // If this is null - explicitly say so rather than using -1, false, etc for some datatypes. + MetaData metadata = 11; // Metadata for the Metric + PropertySet properties = 12; // Properties of the Metric + TypedValue value = 13; + Command command = 14; // Command definition (BIRTH), invocation (CMD), or response (RESP) - present when datatype == Command instead of 'value' + MetricOperation operation = 15; // Only in NMMETA/DMMETA: ADD, UPDATE, or DELETE extensions 16 to max; } message Header { - optional uint64 timestamp = 1; // Timestamp at message sending time - optional uint64 seq = 2; // Message sequence number - ordered stream of all messages from the Edge Node - optional uint64 bd_seq = 3; // The Birth/Death sequence number - scopes this message to the MQTT session it was published in - optional uint64 descriptor_seq = 4; // Descriptor sequence number - ordered stream per Sparkplug Descriptor (the Edge Node Descriptor for Edge Node messages, or a Device Descriptor for Device messages) - optional string definition_id = 5; // Definition identifier: content hash of the metric definitions this message reports against - optional string previous_definition_id = 6; // NMMETA/DMMETA only: the definition_id in effect before the change - optional string source = 7; // Source for CMD and REBIRTH messages - the Host Application ID that sent the message - optional string uuid = 8; // UUID to track message type in terms of schema definitions - optional string compression_algorithm = 9; // Present only when the content section is compressed - names the algorithm used + int64 timestamp = 1; // Timestamp at message sending time + uint64 seq = 2; // Message sequence number - ordered stream of all messages from the Edge Node + uint64 bd_seq = 3; // The Birth/Death sequence number - scopes this message to the MQTT session it was published in + uint64 descriptor_seq = 4; // Descriptor sequence number - ordered stream per Sparkplug Descriptor (the Edge Node Descriptor for Edge Node messages, or a Device Descriptor for Device messages) + string definition_id = 5; // Definition identifier: content hash of the metric definitions this message reports against + string previous_definition_id = 6; // NMMETA/DMMETA only: the definition_id in effect before the change + string source = 7; // Source for CMD and REBIRTH messages - the Host Application ID that sent the message + string uuid = 8; // UUID to track message type in terms of schema definitions + string compression_algorithm = 9; // Present only when the content section is compressed - names the algorithm used extensions 10 to max; // For third party extensions } message Content { - repeated Metric metrics = 1; // Repeated forever - no limit in Google Protobufs - optional bytes body = 2; // To optionally bypass the whole definition above + repeated Metric metrics = 1; // Repeated forever - no limit in Google Protobufs + bytes body = 2; // To optionally bypass the whole definition above extensions 3 to max; // For third party extensions } - optional Header header = 1; // Always present and always uncompressed + Header header = 1; // Always present and always uncompressed - oneof content { - Content plain = 2; // The Content of this message, uncompressed - bytes compressed = 3; // The compressed bytes of a serialized Content + oneof payload_content { + Content content = 2; // The Content of this message, uncompressed + bytes compressed = 3; // The compressed bytes of a serialized Content } extensions 4 to max; // For third party extensions @@ -378,7 +388,7 @@ JSON as follows: "name": , "alias": , "timestamp": , - "dataType": , + "datatype": , "value": }] } @@ -401,7 +411,7 @@ A simple Sparkplug B payload with values would be represented as follows: "name": "My Metric", "alias": 1, "timestamp": 1479123452194000000, - "dataType": "String", + "datatype": "String", "value": "Test" }] } @@ -445,6 +455,26 @@ and the definition identifier. The _content_ carries the metrics themselves. Bot length delimited, so a consuming application that only requires the header can decode it without decoding the content. +[tck-testable tck-id-payloads-string-utf8]#[yellow-background]*[tck-id-payloads-string-utf8] Every +string field of a Sparkplug payload MUST be valid UTF-8.*# + +[tck-testable tck-id-payloads-string-utf8-reject]#[yellow-background]*[tck-id-payloads-string-utf8-reject] A +consuming application MUST reject a payload in which any string field is not valid UTF-8, and MUST +NOT process any part of it.*# + +Non-normative comment: this is stated here rather than left to the encoding, because the Google +Protocol Buffers schema enforces it through a feature whose default changed between proto2 and +edition 2023, and because the JSON encoding inherits it from RFC 8259 instead. Rejecting the whole +payload rather than the offending field is deliberate: a payload is the unit a sequence number +orders, so accepting part of one would leave a consuming application unable to say whether it had +applied that message. + +Non-normative comment: a rejected payload is a non-conformant one, and republishing it will not +make it valid. A Host Application that rejects an NBIRTH or DBIRTH on this basis therefore gains +nothing by issuing a Rebirth Request, because the Edge Node will send the same payload again. It +should treat the Edge Node as unusable and report the condition, rather than requesting Rebirths +until the Edge Node is corrected. + * [tck-testable tck-id-payloads-header-present]#[yellow-background]*[tck-id-payloads-header-present] Every Sparkplug payload MUST include a header.*# ** Non-normative comment: this applies to every message that carries a Sparkplug payload, which is @@ -489,7 +519,7 @@ The header section includes the following components. * *header* ** _timestamp_ -*** This is the timestamp in the form of an unsigned 64-bit integer representing the number of +*** This is the timestamp in the form of a signed 64-bit integer representing the number of nanoseconds since epoch (Jan 1, 1970). *** [tck-testable tck-id-payloads-timestamp-in-UTC]#[yellow-background]*[tck-id-payloads-timestamp-in-UTC] This timestamp MUST be in UTC.*# @@ -503,6 +533,15 @@ is the time that metric's value last changed. A consuming application needs both value is and how long it has held that value. ** _seq_ *** This is the sequence number which is an unsigned 64-bit integer. +*** Non-normative comment: the Sparkplug sequence numbers - seq, bd_seq, and descriptor_seq - are +declared as unsigned 64-bit integers but roll over at 9223372036854775807, which is half of what +that type can hold. The restriction is deliberate. An unsigned type is used because a sequence +number is a counter and can never be negative, and the range is capped so that every value a +conformant Edge Node can produce also fits a signed 64-bit integer. Several widely used languages, +including Java, have no unsigned 64-bit type, so an implementation in one of them would otherwise +have to compare sequence numbers with unsigned arithmetic to remain correct near the top of the +range. Neither bound is reachable in practice: at one million messages per second the capped range +takes roughly 292,000 years to exhaust. **** [tck-testable tck-id-payloads-sequence-num-always-included]#[yellow-background]*[tck-id-payloads-sequence-num-always-included] A sequence number MUST be included in the payload of every Sparkplug MQTT message from an Edge Node except an NDEATH message that is delivered as the Edge Node's registered MQTT Will Message.*# @@ -517,11 +556,11 @@ Node does. This allows Host Applications to easily distinguish an NDEATH deliver from an NDEATH published by the Edge Node itself. **** [tck-testable tck-id-payloads-sequence-num-req-nbirth]#[yellow-background]*[tck-id-payloads-sequence-num-req-nbirth] A NBIRTH message from an Edge Node MUST always contain a sequence number between 0 and -18446744073709551615 (inclusive).*# +9223372036854775807 (inclusive).*# **** [tck-testable tck-id-payloads-sequence-num-incrementing]#[yellow-background]*[tck-id-payloads-sequence-num-incrementing] All subsequent messages after an NBIRTH from an Edge Node MUST contain a sequence number that is continually increasing by one in each message from that Edge Node until a value of -18446744073709551615 (max value of a UINT64) is reached. At that point, the sequence number of the +9223372036854775807 is reached. At that point, the sequence number of the following message MUST be zero.*# ** _bd_seq_ *** This is the birth/death sequence number which is an unsigned 64-bit integer. It scopes a message to @@ -545,7 +584,7 @@ that Host Application and is unrelated to the bd_seq number of any Edge Node. Se link:#payloads_c_state_bd_seq[STATE bd_seq Number] section. **** [tck-testable tck-id-payloads-bd-sequence-num-req-nbirth]#[yellow-background]*[tck-id-payloads-bd-sequence-num-req-nbirth] Every birth/death sequence number included in the payload from an Edge Node MUST always be between 0 and -18446744073709551615 (inclusive).*# +9223372036854775807 (inclusive).*# **** [tck-testable tck-id-payloads-bd-sequence-num-session-constant]#[yellow-background]*[tck-id-payloads-bd-sequence-num-session-constant] Every message an Edge Node publishes within a single MQTT session MUST carry the same birth/death sequence number, and that value MUST be the one registered in the Will Message payload of that session's MQTT @@ -555,7 +594,7 @@ message from an Edge Node can be attributed to the session that produced it, and discard a message belonging to a session it has already replaced. **** [tck-testable tck-id-payloads-bd-sequence-num-incrementing]#[yellow-background]*[tck-id-payloads-bd-sequence-num-incrementing] The birth/death sequence number MUST start at zero and MUST increment by one on every new MQTT CONNECT -packet sent by the Edge Node, until a value of 18446744073709551615 (max value of a UINT64) is reached. +packet sent by the Edge Node, until a value of 9223372036854775807 is reached. At that point the birth/death sequence number of the following MQTT CONNECT packet MUST be zero.*# ***** Non-normative comment: the bd_seq number is not incremented from message to message within a session, and it is not incremented by an NREBIRTH, because an NREBIRTH does not establish a new MQTT @@ -584,7 +623,7 @@ sequence (see the Reconnect Optimization), the NBIRTH or DBIRTH is the first mes **** [tck-testable tck-id-payloads-descriptor-seq-incrementing]#[yellow-background]*[tck-id-payloads-descriptor-seq-incrementing] Each subsequent message in a descriptor sequence stream MUST contain a descriptor sequence number that is one greater than the descriptor sequence number of the previous message in that same stream, until a -value of 18446744073709551615 (max value of a UINT64) is reached, after which the next descriptor +value of 9223372036854775807 is reached, after which the next descriptor sequence number MUST be 0.*# **** [tck-testable tck-id-payloads-descriptor-seq-rebirth]#[yellow-background]*[tck-id-payloads-descriptor-seq-rebirth] An NREBIRTH or DREBIRTH slip-streams a Host Application into the existing data stream and MUST NOT reset @@ -626,6 +665,34 @@ link:#payloads_c_compression[Compression] section. ** _metrics_ *** This is an array of metrics representing key/value/datatype values. Metrics are further defined link:#payloads_c_metric[here]. +*** [tck-testable tck-id-payloads-metrics-ordered]#[yellow-background]*[tck-id-payloads-metrics-ordered] A +payload that carries more than one value for the same metric MUST order its metrics by value change +time, oldest first.*# +*** [tck-testable tck-id-payloads-metrics-ordered-should]#[yellow-background]*[tck-id-payloads-metrics-ordered-should] Every +other payload SHOULD order its metrics the same way.*# +*** [tck-not-testable tck-id-payloads-metrics-ordered-scope]#[yellow-background]*[tck-id-payloads-metrics-ordered-scope] This +ordering applies to the metrics array of the payload's content. It does not apply to the metrics of +a Template, or to the arguments or results of a Command, whose order is defined by the Template +Definition or the Command definition.*# +*** [tck-testable tck-id-payloads-metrics-ordered-no-timestamp]#[yellow-background]*[tck-id-payloads-metrics-ordered-no-timestamp] A +metric that omits its timestamp MUST be ordered as though its value change time were the payload +timestamp, which is the time that metric is taken to have changed as described in the +link:#payloads_c_metric[Metric] section.*# +*** [tck-not-testable tck-id-payloads-metrics-ordered-ties]#[yellow-background]*[tck-id-payloads-metrics-ordered-ties] Metrics +that share a value change time MAY appear in any order relative to one another, and a consuming +application MUST NOT reorder them.*# +*** Non-normative comment: Ordering by value change time means a consuming application that applies +the metrics in array order arrives at the correct current value without comparing timestamps. That +is why it is required only where a payload carries several values for the same metric, as a payload +of historical values does; elsewhere it is a recommendation, so that an Edge Node emitting a BIRTH +in browse order remains conformant. Metrics remain identified by name or alias, so position still +carries no identity. +*** Non-normative comment: The order among metrics that share a value change time is the publisher's +and is preserved by both encodings, because a JSON array and a Google Protocol Buffers repeated +field are both ordered. Multi-part file publishing depends on this: every metric of a multi-part +publish carries the same timestamp, as required in the +link:#payloads_c_file_publishing[File Publishing] section, so they tie, and the rules that +distinguish the first metric of the publish from the rest continue to apply. ** _body_ *** This is an array of bytes which can be used for any custom binary encoded data. @@ -666,7 +733,7 @@ and DBIRTH messages MUST include both a metric name and alias.*# *** [tck-testable tck-id-payloads-alias-data-cmd-requirement]#[yellow-background]*[tck-id-payloads-alias-data-cmd-requirement] NDATA, DDATA, NCMD, and DCMD messages MUST only include an alias and the metric name MUST be excluded.*# * *timestamp* -** This is the timestamp in the form of an unsigned 64-bit integer representing the number of +** This is the timestamp in the form of a signed 64-bit integer representing the number of nanoseconds since epoch (Jan 1, 1970). ** [tck-not-testable]#[yellow-background]*In an NBIRTH, NREBIRTH, DBIRTH, DREBIRTH, NDATA, or DDATA message, the timestamp of a Metric MUST represent the metric value change time, meaning the time at @@ -1255,8 +1322,8 @@ Acme/Dublin/Packaging/spCv4.0/NMETA/EdgeNode1 "content": { "metrics": [{ "name": "MotorType", - "dataType": "Template", - "template": { + "datatype": "Template", + "value": { "version": "1.0", "is_definition": true, "parameters": [{ @@ -1266,13 +1333,13 @@ Acme/Dublin/Packaging/spCv4.0/NMETA/EdgeNode1 }], "metrics": [{ "name": "RPM", - "dataType": "Int32" + "datatype": "Int32" }, { "name": "AmpDraw", - "dataType": "Float" + "datatype": "Float" }, { "name": "Running", - "dataType": "Boolean" + "datatype": "Boolean" }] } }] @@ -1300,16 +1367,16 @@ Acme/Dublin/Packaging/spCv4.0/DMETA/EdgeNode1/Line3 "content": { "metrics": [{ "name": "Motors/Conveyor", - "dataType": "Template", - "template": { + "datatype": "Template", + "value": { "version": "1.0", "is_definition": false, "template_ref": "MotorType" } }, { "name": "Motors/Filler", - "dataType": "Template", - "template": { + "datatype": "Template", + "value": { "version": "1.0", "is_definition": false, "template_ref": "MotorType" @@ -1339,8 +1406,8 @@ Acme/Dublin/Packaging/spCv4.0/DBIRTH/EdgeNode1/Line3 "content": { "metrics": [{ "name": "Motors/Conveyor", - "dataType": "Template", - "template": { + "datatype": "Template", + "value": { "version": "1.0", "is_definition": false, "template_ref": "MotorType", @@ -1351,36 +1418,36 @@ Acme/Dublin/Packaging/spCv4.0/DBIRTH/EdgeNode1/Line3 }], "metrics": [{ "name": "RPM", - "dataType": "Int32", + "datatype": "Int32", "value": 1780 }, { "name": "AmpDraw", - "dataType": "Float", + "datatype": "Float", "value": 12.4 }, { "name": "Running", - "dataType": "Boolean", + "datatype": "Boolean", "value": true }] } }, { "name": "Motors/Filler", - "dataType": "Template", - "template": { + "datatype": "Template", + "value": { "version": "1.0", "is_definition": false, "template_ref": "MotorType", "metrics": [{ "name": "RPM", - "dataType": "Int32", + "datatype": "Int32", "value": 900 }, { "name": "AmpDraw", - "dataType": "Float", + "datatype": "Float", "value": 7.1 }, { "name": "Running", - "dataType": "Boolean", + "datatype": "Boolean", "value": false }] } @@ -1408,13 +1475,13 @@ Acme/Dublin/Packaging/spCv4.0/DDATA/EdgeNode1/Line3 "content": { "metrics": [{ "name": "Motors/Conveyor", - "dataType": "Template", - "template": { + "datatype": "Template", + "value": { "is_definition": false, "template_ref": "MotorType", "metrics": [{ "name": "RPM", - "dataType": "Int32", + "datatype": "Int32", "value": 1802 }] } @@ -1586,6 +1653,7 @@ represented as an enum in Google Protobufs as shown below. ---- enum DataType { + option features.enum_type = CLOSED; // Indexes of Data Types // Unknown placeholder for future expansion. @@ -1694,9 +1762,15 @@ enum DataType { *** Google Protocol Buffer Type: string *** Sparkplug enum value: 12 * _DateTime_ -** Date time value as uint64 value representing nanoseconds since epoch (Jan 1, 1970) -** Google Protocol Buffer Type: uint64 +** Date time value as int64 value representing nanoseconds since epoch (Jan 1, 1970) +** Google Protocol Buffer Type: int64 ** Sparkplug enum value: 13 +** Non-normative comment: a DateTime value is a nanosecond count since epoch, the same quantity the +payload and metric timestamps carry, and is signed for the same reason. Every mainstream time +representation it interoperates with is signed, so the ceiling is 2262-04-11 as described in the +link:#operational_behavior_timestamps[Timestamps] section. A DateTime carried in a DataSet.Column +shares the long_values array with UInt64, as the +link:#payloads_c_dataset_column[DataSet.Column] section describes. * _Text_ ** String value (UTF-8) ** Google Protocol Buffer Type: string @@ -1812,8 +1886,8 @@ ends on a byte boundary. *** Sparkplug enum value: 33 *** Example (string array to Metric bytes_value): [ABC, hello] -> [0x41, 0x42, 0x43, 0x00, 0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x00] ** _DateTimeArray_ -*** DateTimeArray as an array of packed little endian bytes where each Datetime value is an 8-byte -value representing the number of nanoseconds since epoch in UTC +*** DateTimeArray as an array of packed little endian bytes where each DateTime value is a signed +8-byte value representing the number of nanoseconds since epoch in UTC *** Google Protocol Buffer Type: bytes *** Sparkplug enum value: 34 *** Example (DateTime array -> nanoseconds since epoch -> Metric bytes_value): ['Wednesday, October 21, 2009 5:27:55.335 AM', 'Friday, June 24, 2022 9:57:55 PM'] -> [1256102875335000000, 1656107875000000000] -> [0xC0, 0x7F, 0xB0, 0xF5, 0xE8, 0x92, 0x6E, 0x11, 0x00, 0x1E, 0x1A, 0x7F, 0x56, 0xAD, 0xFB, 0x16] @@ -2109,11 +2183,12 @@ alias was assigned to it in the referenced NMETA, or otherwise by its metric nam NOT be used to convey metric datatype, properties, or metadata.*# * [tck-testable tck-id-payloads-nrebirth-seq-1]#[yellow-background]*[tck-id-payloads-nrebirth-seq-1] Every NREBIRTH message MUST include a sequence number and it MUST have a value between 0 and -18446744073709551615 (inclusive).*# +9223372036854775807 (inclusive).*# * [tck-testable tck-id-payloads-nrebirth-seq-2]#[yellow-background]*[tck-id-payloads-nrebirth-seq-2] Every NREBIRTH message MUST include a sequence number that is the following value:*# -** rsn = pds - (1 + dc) -** If the rsn results in a negative number, add 18446744073709551615 (max value of a UINT64) to it. +** If pds is greater than or equal to (1 + dc), then rsn = pds - (1 + dc) +** Otherwise the sequence number has rolled over since that pending DATA publish, and +rsn = 9223372036854775807 - (dc - pds) *** rsn = Rebirth Sequence Number value *** pds = Next pending DATA publish seq number *** dc = Device count (number of Sparkplug Devices under the scope of the Edge Node) @@ -2230,8 +2305,8 @@ NOT be used to convey metric datatype, properties, or metadata.*# DBIRTH message MUST include a sequence number.*# * [tck-testable tck-id-payloads-dbirth-seq-inc]#[yellow-background]*[tck-id-payloads-dbirth-seq-inc] Every DBIRTH message MUST include a sequence number value that is one greater than the previous sequence -number sent by the Edge Node. This value MUST never exceed 18446744073709551615 (max value of a -UINT64). If in the previous sequence number sent by the Edge Node was 18446744073709551615, the next +number sent by the Edge Node. This value MUST never exceed 9223372036854775807. If in the previous +sequence number sent by the Edge Node was 9223372036854775807, the next sequence number sent MUST have a value of 0.*# * [tck-testable tck-id-payloads-dbirth-order]#[yellow-background]*[tck-id-payloads-dbirth-order] All DBIRTH messages sent by an Edge Node MUST be sent immediately after the NBIRTH and before any NDATA @@ -2333,8 +2408,8 @@ NOT be used to convey metric datatype, properties, or metadata.*# DREBIRTH message MUST include a sequence number.*# * [tck-testable tck-id-payloads-drebirth-seq-inc]#[yellow-background]*[tck-id-payloads-drebirth-seq-inc] Every DREBIRTH message MUST include a sequence number value that is one greater than the previous sequence -number sent by the Edge Node. This value MUST never exceed 18446744073709551615 (max value of a -UINT64). If in the previous sequence number sent by the Edge Node was 18446744073709551615, the next +number sent by the Edge Node. This value MUST never exceed 9223372036854775807. If in the previous +sequence number sent by the Edge Node was 9223372036854775807, the next sequence number sent MUST be a value of 0.*# * [tck-testable tck-id-payloads-drebirth-order]#[yellow-background]*[tck-id-payloads-drebirth-order] All DREBIRTH messages sent by an Edge Node MUST be sent immediately after the NREBIRTH and before any @@ -2433,8 +2508,8 @@ published.*# NDATA message MUST include a sequence number.*# * [tck-testable tck-id-payloads-ndata-seq-inc]#[yellow-background]*[tck-id-payloads-ndata-seq-inc] Every NDATA message MUST include a sequence number value that is one greater than the previous sequence -number sent by the Edge Node. This value MUST never exceed 18446744073709551615 (max value of a -UINT64). If in the previous sequence number sent by the Edge Node was 18446744073709551615, the next +number sent by the Edge Node. This value MUST never exceed 9223372036854775807. If in the previous +sequence number sent by the Edge Node was 9223372036854775807, the next sequence number sent MUST have a value of 0.*# * [tck-testable tck-id-payloads-ndata-order]#[yellow-background]*[tck-id-payloads-ndata-order] All NDATA messages sent by an Edge Node MUST NOT be sent until all the NBIRTH and all DBIRTH messages @@ -2500,8 +2575,8 @@ published.*# DDATA message MUST include a sequence number.*# * [tck-testable tck-id-payloads-ddata-seq-inc]#[yellow-background]*[tck-id-payloads-ddata-seq-inc] Every DDATA message MUST include a sequence number value that is one greater than the previous sequence -number sent by the Edge Node. This value MUST never exceed 18446744073709551615 (max value of a -UINT64). If in the previous sequence number sent by the Edge Node was 18446744073709551615, the next +number sent by the Edge Node. This value MUST never exceed 9223372036854775807. If in the previous +sequence number sent by the Edge Node was 9223372036854775807, the next sequence number sent MUST have a value of 0.*# * [tck-testable tck-id-payloads-ddata-order]#[yellow-background]*[tck-id-payloads-ddata-order] All DDATA messages sent by an Edge Node MUST NOT be sent until all the NBIRTH and all DBIRTH messages @@ -2640,7 +2715,7 @@ Consider the following Sparkplug B payload in the NCMD message shown above: "metrics": [{ "name": "DIGITAL_OUT_1", "timestamp": 1486144502122000000, - "dataType": "Boolean", + "datatype": "Boolean", "value": true }] } @@ -2671,12 +2746,12 @@ The following is a representation of an NCMD payload that invokes the 'Calibrate "metrics": [{ "name": "Calibrate", "timestamp": 1486144502122000000, - "dataType": "Command", + "datatype": "Command", "command": { "correlation_id": "5d7c1f6e-9c2a-4f1b-8f3a-1c9d2e6b4a01", "arguments": [{ "name": "channel", - "dataType": "Int32", + "datatype": "Int32", "value": 1 }] } @@ -2735,12 +2810,12 @@ Consider the following Sparkplug B payload in the DCMD message shown above: "metrics": [{ "name": "Motor/Run", "timestamp": 1486144502122000000, - "dataType": "Boolean", + "datatype": "Boolean", "value": true }, { "name": "Motor/Speed Setpoint", "timestamp": 1486144502122000000, - "dataType": "Int32", + "datatype": "Int32", "value": 120 }] } @@ -2799,7 +2874,7 @@ Consider the following Sparkplug B payload in the NRESP message shown above: "metrics": [{ "name": "Calibrate", "timestamp": 1486144502122000000, - "dataType": "Command", + "datatype": "Command", "command": { "correlation_id": "5d7c1f6e-9c2a-4f1b-8f3a-1c9d2e6b4a01", "status": { @@ -2808,7 +2883,7 @@ Consider the following Sparkplug B payload in the NRESP message shown above: }, "results": [{ "name": "offset", - "dataType": "Double", + "datatype": "Double", "value": 0.125 }] } @@ -2863,7 +2938,7 @@ Consider the following Sparkplug B payload in the DRESP message shown above: "metrics": [{ "name": "Reset", "timestamp": 1486144502122000000, - "dataType": "Command", + "datatype": "Command", "command": { "correlation_id": "a1b2c3d4-0001", "status": { @@ -2903,7 +2978,7 @@ message. payload compared to what was in the previous CONNECT packet. This ensures that any Host Applications will be able to distinguish between current and old bd_seq numbers in the event that messages are delivered out of order. When incrementing the bd_seq number, if the previous value was -18446744073709551615 (max value of a UINT64), the next value must be zero. +9223372036854775807, the next value must be zero. * [tck-testable tck-id-payloads-ndeath-will-message-publisher]#[yellow-background]*[tck-id-payloads-ndeath-will-message-publisher] An NDEATH message SHOULD be published by the Edge Node before it intentionally disconnects from the MQTT Server.*# @@ -2971,8 +3046,8 @@ published.*# DDEATH message MUST include a sequence number.*# * [tck-testable tck-id-payloads-ddeath-seq-inc]#[yellow-background]*[tck-id-payloads-ddeath-seq-inc] Every DDEATH message MUST include a sequence number value that is one greater than the previous sequence -number sent by the Edge Node. This value MUST never exceed 18446744073709551615 (max value of a -UINT64). If in the previous sequence number sent by the Edge Node was 18446744073709551615, the next +number sent by the Edge Node. This value MUST never exceed 9223372036854775807. If in the previous +sequence number sent by the Edge Node was 9223372036854775807, the next sequence number sent MUST have a value of 0.*# The following is a representation of a simple DDEATH message on the topic: @@ -3061,15 +3136,15 @@ Certificate to be matched to the Birth Certificate of the session it terminates. bd_seq number MUST be included in every STATE message published by or on behalf of a Sparkplug Host Application, including the Death Certificate registered as the MQTT Will Message.*# * [tck-testable tck-id-payloads-state-bdseq-range]#[yellow-background]*[tck-id-payloads-state-bdseq-range] The -bd_seq number in a STATE payload MUST always be between 0 and 18446744073709551615 (inclusive).*# +bd_seq number in a STATE payload MUST always be between 0 and 9223372036854775807 (inclusive).*# * [tck-testable tck-id-payloads-state-bdseq-session-constant]#[yellow-background]*[tck-id-payloads-state-bdseq-session-constant] Every STATE message a Sparkplug Host Application publishes within a single MQTT session MUST carry the same bd_seq number, and that value MUST be the one registered in the Will Message payload of that session's MQTT CONNECT packet.*# * [tck-testable tck-id-payloads-state-bdseq-increment]#[yellow-background]*[tck-id-payloads-state-bdseq-increment] The bd_seq number MUST start at zero and MUST increment by one on every new MQTT CONNECT packet the -Sparkplug Host Application sends to a given MQTT Server, until a value of 18446744073709551615 (max -value of a UINT64) is reached. At that point the bd_seq number of the following MQTT CONNECT packet +Sparkplug Host Application sends to a given MQTT Server, until a value of 9223372036854775807 is +reached. At that point the bd_seq number of the following MQTT CONNECT packet MUST be zero.*# ** Non-normative comment: these are the same rules that apply to an Edge Node's bd_seq number. As with an Edge Node, the bd_seq number is not incremented from message to message within a session. @@ -3142,13 +3217,14 @@ Compression applies to the content section of a payload only. The header section compressed, so a consuming application can read the header of a compressed payload without decompressing anything. -A compressed payload carries its content in the compressed field of the content oneof rather than in -the plain field. Which of the two is present is therefore evident from the payload itself, and no +A compressed payload carries its content in the compressed field of the payload_content oneof +rather than in the content field. Which of the two is present is therefore evident from the payload +itself, and no separate marker is required. * [tck-testable tck-id-payloads-compression-content-only]#[yellow-background]*[tck-id-payloads-compression-content-only] When using Sparkplug compression, the compressed field of a payload MUST contain the compressed bytes of -a serialized Content message, and the plain field MUST NOT be present.*# +a serialized Content message, and the content field MUST NOT be present.*# * [tck-testable tck-id-payloads-compression-header-uncompressed]#[yellow-background]*[tck-id-payloads-compression-header-uncompressed] The header section of a payload MUST NOT be compressed.*# ** Non-normative comment: this is what allows a Host Application to evaluate the sequence numbers @@ -3165,7 +3241,7 @@ before decompressing it. In earlier versions of this specification the algorithm metric inside the payload being compressed, which required the payload to be decompressed in order to discover how to decompress it. ** Non-normative comment: the value names the algorithm only. Whether the content is compressed at -all is already carried by which field of the content oneof is present, so no marker value is +all is already carried by which field of the payload_content oneof is present, so no marker value is required in this field. * [tck-testable tck-id-payloads-compression-algorithm-support]#[yellow-background]*[tck-id-payloads-compression-algorithm-support] A consuming application that is delivered a compressed payload whose algorithm it does not support @@ -3185,7 +3261,8 @@ the original, and the header and framing are not compressible in any case. The following is the NDATA message shown earlier in this chapter, with its content section compressed. The header is identical to the uncompressed form apart from the addition of the -compression_algorithm, the plain field is absent, and the compressed field holds the deflated bytes +compression_algorithm, the content field is absent, and the compressed field holds the deflated +bytes of the serialized Content: ---- @@ -3270,6 +3347,14 @@ metric MetaData file_name MUST be included and set to the String value of the fi When publishing in multi-part form, the following rules apply regarding the first metric associated with the file publish. +Non-normative comment: The rules below identify a metric by its position in the payload, which is +the one place in this specification where the position of a metric carries meaning. This does not +conflict with the requirement that metrics be ordered by value change time, stated in the +link:#payloads_c_payload_content[Content] section, because every metric of a multi-part file +publish carries the same timestamp. They therefore tie, the ordering requirement does not constrain +their relative order, and the order the publishing Edge Node or Device chooses is carried unchanged +by both payload encodings. + * [tck-testable tck-id-payloads-file-multi-metadata-size-first]#[yellow-background]*[tck-id-payloads-file-multi-metadata-size-first] The first metric regarding the file publish's metric MetaData size MUST be included and set to the uint64 value of the total number of separate metrics that make up the full multi-part file publish.*# @@ -3444,3 +3529,636 @@ on the publishing (transmission) side, and 5xx indicate an error on the receivin |Receiving Side Error |Error Renaming Partial File |=== + +[[payloads_json]] +=== Sparkplug JSON Payload Definition + +[[payloads_json_overview]] +==== Overview + +Sparkplug defines two payload encodings of the same data model. The Google Protocol Buffers +encoding is defined in the link:#payloads_sparkplug_c_mqtt_payload_definition[Sparkplug B MQTT +Payload Definition] section above and is published on the spCv4.0 namespace. The JSON encoding is +defined in this section and is published on the spJv4.0 namespace. + +The two encodings carry identical information. Every component defined in the sections above - +Header, Content, Metric, Quality, MetaData, PropertySet, DataSet, Template, Command, and the +DataTypes - applies unchanged to the JSON encoding, and every normative statement made about those +components applies to both encodings. This section defines only how those components are +represented as JSON text. All operational behavior, topic structure, sequence numbering, and +session management defined elsewhere in this specification is encoding independent. + +Non-normative comment: Because the two encodings are two representations of one model, a message +may be converted from one to the other without loss and without consulting anything outside the +message itself. This is what allows a bridging application to republish an spCv4.0 namespace onto +spJv4.0, or the reverse, as a mechanical transformation. + +This section defines the JSON encoding completely. Conformance to it does not depend on any +document outside this specification. + +Non-normative comment: The scalar rules below were chosen to agree with the JSON Mapping that +Google Protocol Buffers defines for proto3, so the representation of 64-bit integers, enumerated +values, bytes, and non-finite floating point values will be familiar, and a Protocol Buffers JSON +implementation produces those forms directly. That agreement is a convenience and not a normative +dependency. The Sparkplug schema uses Protocol Buffers edition 2023, that mapping is specified for +proto3, it does not define how extension fields are represented, and it is not a versioned +standard, so this specification states the rules itself rather than incorporating them by +reference. + +Non-normative comment: Agreement on the scalar forms does not mean a conformant encoder can be +assembled from library calls. The structural mappings reshape every Metric, so an implementation +writes its own traversal and uses a library, if at all, for the scalar formatting within it. + +[[payloads_json_encoding_selection]] +==== Encoding Selection + +[tck-testable tck-id-payloads-json-namespace-encoding]#[yellow-background]*[tck-id-payloads-json-namespace-encoding] The +payload of any Sparkplug message published on the spJv4.0 namespace MUST be encoded as defined in +this section.*# + +[tck-testable tck-id-payloads-protobuf-namespace-encoding]#[yellow-background]*[tck-id-payloads-protobuf-namespace-encoding] The +payload of any Sparkplug message published on the spCv4.0 namespace MUST be encoded using the +Google Protocol Buffers schema defined in the +link:#payloads_c_google_protocol_buffer_schema[Google Protocol Buffer Schema] section.*# + +[tck-testable tck-id-payloads-json-utf8]#[yellow-background]*[tck-id-payloads-json-utf8] A Sparkplug +JSON payload MUST be a JSON text as defined by RFC 8259, encoded as UTF-8 with no byte order +mark.*# + +[tck-testable tck-id-payloads-json-object]#[yellow-background]*[tck-id-payloads-json-object] The +top level JSON value of a Sparkplug JSON payload MUST be a JSON object representing the Payload +message.*# + +Non-normative comment: An Edge Node publishes on one namespace, not both. An Edge Node that +publishes on spJv4.0 is a complete Sparkplug Edge Node and is not required to implement the +Protocol Buffers encoding at all. The choice is a deployment decision; see the +link:#payloads_json_choosing[Choosing a Payload Encoding] section. + +[[payloads_json_field_naming]] +==== Field Naming + +[tck-testable tck-id-payloads-json-field-names]#[yellow-background]*[tck-id-payloads-json-field-names] Each +JSON object key MUST be the name of the corresponding field exactly as it appears in the Sparkplug +Google Protocol Buffers schema, except where the +link:#payloads_json_structural[Structural Mappings] section names a different key.*# + +Non-normative comment: A Google Protocol Buffers JSON implementation converts field names to +lowerCamelCase by default, emitting bdSeq for bd_seq and isMultiPart for is_multi_part. That is not +the Sparkplug form, so an implementation that generates JSON with such a library must configure it +to preserve the original field names. In the Java and Python implementations this is the +preservingProtoFieldNames printer option. + +[tck-testable tck-id-payloads-json-omit-absent]#[yellow-background]*[tck-id-payloads-json-omit-absent] A +field that is not present MUST be omitted from the JSON object rather than written with a JSON null +or a zero value.*# + +Non-normative comment: Every singular field of the Sparkplug schema has explicit presence, so a +field carries the distinction between having been set and never having been set, and the JSON +encoding has to keep that distinction. Zero is a legitimate value for seq, bd_seq, descriptor_seq, +alias, and quality, so writing an absent field as 0 would make it indistinguishable from one +deliberately set to zero. A JSON null is excluded for a different reason: a null metric value is +expressed by setting is_null to true and omitting the value key, as required in the +link:#payloads_json_typed_values[Typed Values] section, so a null appearing as a value would be a +second and conflicting way to say the same thing. + +[tck-testable tck-id-payloads-json-ignore-unknown]#[yellow-background]*[tck-id-payloads-json-ignore-unknown] A +consuming application MUST ignore a JSON object key that does not name a field it recognizes, and +MUST NOT treat the payload as malformed on that basis.*# + +[tck-not-testable tck-id-payloads-json-unknown-not-metric]#[yellow-background]*[tck-id-payloads-json-unknown-not-metric] The +requirement above applies only to the names of fields within a message. It does not apply to a +metric whose name or alias the consuming application does not recognize, which is a mismatch +against the metric structure the application holds and is resolved by the rules in the +link:#operational_behavior_modified_metadata_updates[Modified Metadata Updates] section rather than +by ignoring it.*# + +Non-normative comment: The two cases look similar in a payload and are not the same thing. An +unrecognized field name means the payload carries something this version of the application does +not know about, such as a field added by a later revision of this specification or a third party +extension; ignoring it is what allows an older application to keep working. An unrecognized metric +means the application's view of what the Edge Node reports is out of date, which is a +synchronization problem and a reason to issue a Rebirth Request, not something to pass over. + +Non-normative comment: The naming rule keeps a JSON payload visually identical to the field names +in the Protocol Buffers schema, so the two encodings can be read against one another and against +the component definitions above without a translation table. Ignoring unrecognized keys is the JSON +counterpart of the extension ranges reserved in the Protocol Buffers schema, and serves the same +purpose for third party extensions. + +[[payloads_json_scalar]] +==== Scalar Value Mapping + +The following table defines how each Protocol Buffers scalar type is represented in JSON. + +[cols="2,2,4"] +|=== +|*Protocol Buffers Type* +|*JSON Type* +|*Representation* + +|int32, uint32, sint32, fixed32, sfixed32 +|number +|A JSON number with no fractional part or exponent. + +|int64, uint64, sint64, fixed64, sfixed64 +|string +|The value written in decimal, as a JSON string. For example "1486144502122847913". + +|float, double +|number or string +|A JSON number, or one of the strings "NaN", "Infinity", or "-Infinity". + +|bool +|true or false +|A JSON literal, never the strings "true" or "false" and never 0 or 1. + +|string +|string +|The string value. + +|bytes +|string +|The bytes, base64 encoded as defined in RFC 4648 Section 4, including padding. + +|enum +|string +|The name of the enumerated value, for example "Int32", "Template", or "ADD". +|=== + +[tck-testable tck-id-payloads-json-64bit-string]#[yellow-background]*[tck-id-payloads-json-64bit-string] A +publishing application MUST represent every 64-bit integer field as a JSON string containing the +decimal value, including when the value is small enough to be represented exactly as a JSON +number.*# + +[tck-testable tck-id-payloads-json-64bit-accept]#[yellow-background]*[tck-id-payloads-json-64bit-accept] A +consuming application MUST accept a 64-bit integer field represented as either a JSON string or a +JSON number.*# + +[tck-testable tck-id-payloads-json-bytes-base64]#[yellow-background]*[tck-id-payloads-json-bytes-base64] A +field of Protocol Buffers type bytes MUST be represented as a JSON string containing the base64 +encoding of those bytes as defined in RFC 4648 Section 4, including padding.*# + +[tck-testable tck-id-payloads-json-enum-name]#[yellow-background]*[tck-id-payloads-json-enum-name] A +publishing application MUST represent an enumerated value, including a DataType and a +MetricOperation, as a JSON string containing the name of that value.*# + +[tck-testable tck-id-payloads-json-enum-accept]#[yellow-background]*[tck-id-payloads-json-enum-accept] A +consuming application MUST accept an enumerated value represented as either the name of that value +or the integer value assigned to it in the Google Protocol Buffers schema.*# + +[tck-testable tck-id-payloads-json-float-nonfinite]#[yellow-background]*[tck-id-payloads-json-float-nonfinite] A +non-finite float or double value MUST be represented as one of the JSON strings "NaN", "Infinity", +or "-Infinity", because RFC 8259 provides no numeric form for these values.*# + +[tck-testable tck-id-payloads-json-float-precision]#[yellow-background]*[tck-id-payloads-json-float-precision] A +finite float or double value MUST be written with enough decimal digits that parsing it back +reproduces the original value exactly, which requires up to 9 significant decimal digits for a +float and up to 17 for a double.*# + +Non-normative comment: 64-bit integers are represented as strings because a JSON number is not +required by RFC 8259 to carry more precision than an IEEE 754 double, which holds integers exactly +only up to 2^53^. Several widely used JSON parsers, including the one built into every web browser, +decode JSON numbers into doubles unconditionally. A Sparkplug timestamp is a nanosecond count and +is therefore roughly 165 times larger than that limit, so a timestamp written as a JSON number +would be silently rounded, typically to the nearest 256 nanoseconds. Metric values of type Int64, +UInt64, and DateTime have the same exposure. + +Non-normative comment: The sequence numbers are subject to this for the same reason. A Sparkplug +seq, bd_seq, and descriptor_seq each count up to 9223372036854775807 before rolling over to zero, as +required in the link:#payloads_c_payload_header[Header] section, so they reach magnitudes a JSON +number cannot carry exactly. A sequence number is small early in a session and large late in one, +which would make a numeric form correct in testing and lossy in production. + +Non-normative comment: The rule is driven by the declared type of the field rather than by the +magnitude of a particular value, so an encoder derives it from the schema alone and no value can +lose precision because of how large it happened to be. + +[[payloads_json_extensions]] +==== Extension Fields + +The Sparkplug Google Protocol Buffers schema reserves an extension range in most of its messages so +that a third party can add fields without colliding with future Sparkplug fields. The JSON encoding +represents those fields as well. + +[tck-testable tck-id-payloads-json-extension-key]#[yellow-background]*[tck-id-payloads-json-extension-key] A +field carried in a message's extension range MUST be represented with a JSON object key that is the +fully qualified name of that extension enclosed in square brackets, for example +'[com.example.vendor_tag]'.*# + +[tck-testable tck-id-payloads-json-extension-value]#[yellow-background]*[tck-id-payloads-json-extension-value] The +value of an extension field MUST be represented according to the same rules as any other field of +its type, as defined in the link:#payloads_json_scalar[Scalar Value Mapping] and +link:#payloads_json_structural[Structural Mappings] sections.*# + +[tck-not-testable tck-id-payloads-json-extension-ignore]#[yellow-background]*[tck-id-payloads-json-extension-ignore] A +consuming application that does not recognize an extension MUST ignore it, as it ignores any other +unrecognized key, and MUST NOT treat the payload as malformed.*# + +Non-normative comment: The bracketed form is the one Google Protocol Buffers JSON implementations +already use for extensions, so it is specified here rather than invented. It is stated normatively +because the Protocol Buffers JSON Mapping is defined for proto3, which has no extensions, and so +does not cover this case at all. + +Non-normative comment: An extension does not survive a round trip through an application that does +not know it. In the Google Protocol Buffers encoding an unknown field is retained and re-emitted, +but in JSON an unrecognized key is not. A deployment that depends on extensions reaching a +consumer through an intermediary should confirm that the intermediary understands them, rather +than assuming they are carried through. + +[[payloads_json_structural]] +==== Structural Mappings + +A Protocol Buffers message is represented as a JSON object and a repeated field as a JSON array. +The following cases are where the Sparkplug JSON encoding deviates from a direct mapping of the +Protocol Buffers schema, in each case so that the JSON encoding reads as JSON rather than as a +transliteration of a binary format. + +[[payloads_json_typed_values]] +===== Typed Values + +In the Protocol Buffers schema a Metric carries its value in a TypedValue submessage, which repeats +the datatype and selects one of several value fields. In JSON the datatype is already carried by +the enclosing object and the value field is not needed to disambiguate. + +[tck-testable tck-id-payloads-json-typedvalue-flat]#[yellow-background]*[tck-id-payloads-json-typedvalue-flat] A +TypedValue that is a member of a Metric or of a Template.Parameter MUST be represented as the value +itself, under the key 'value', rather than as a nested object, and its datatype MUST be carried by +the 'datatype' key of the enclosing Metric or the 'type' key of the enclosing Template.Parameter.*# + +[tck-testable tck-id-payloads-json-typedvalue-propertyset]#[yellow-background]*[tck-id-payloads-json-typedvalue-propertyset] A +TypedValue that is a member of a PropertySet MUST instead be represented as an object carrying its +own 'datatype' key, its 'value', and 'is_null' where applicable, because a property value has no +enclosing Metric or Template.Parameter to carry its datatype.*# + +[tck-testable tck-id-payloads-typedvalue-datatype-agree]#[yellow-background]*[tck-id-payloads-typedvalue-datatype-agree] Where +a Metric carries both its own datatype and a TypedValue that also carries one, as it does in the +Google Protocol Buffers encoding, the two MUST be equal, and the same applies to is_null.*# + +[tck-testable tck-id-payloads-json-typedvalue-carries-both]#[yellow-background]*[tck-id-payloads-json-typedvalue-carries-both] The +single 'datatype' and 'is_null' key a flattened JSON value carries represents both, so an +application converting a JSON payload to the Google Protocol Buffers encoding MUST set the Metric +field and the TypedValue field from it.*# + +[tck-testable tck-id-payloads-json-parameter-is-null]#[yellow-background]*[tck-id-payloads-json-parameter-is-null] A +Template.Parameter whose value is null MUST carry 'is_null' set to true and omit 'value', as any +other null value does.*# + +[tck-testable tck-id-payloads-json-typedvalue-parameter-keys]#[yellow-background]*[tck-id-payloads-json-typedvalue-parameter-keys] A +Template.Parameter MUST therefore be represented with the keys 'name', 'type', 'value', and +'is_null' where applicable, and MUST NOT use the key 'typed_value'.*# + +Non-normative comment: These are the two places where a JSON key is not a Protocol Buffers field +name. 'value' is the name of the oneof within TypedValue rather than of any field in it, because +flattening replaces the whole oneof with the single value it holds; a Metric's own field is also +called value, so a Metric reads the same either way. A Template.Parameter's field is typed_value, +and flattening replaces it with the 'type' and 'value' pair used for parameters throughout this +specification. + +[tck-testable tck-id-payloads-json-null]#[yellow-background]*[tck-id-payloads-json-null] A value +that is null MUST be represented by setting 'is_null' to true and omitting the 'value' key.*# + +[[payloads_json_propertyset]] +===== PropertySet + +In the Protocol Buffers schema a PropertySet is a pair of parallel arrays of keys and values. In +JSON it is represented as an object, which is both smaller and directly usable by a consuming +application. + +[tck-testable tck-id-payloads-json-propertyset-object]#[yellow-background]*[tck-id-payloads-json-propertyset-object] A +PropertySet MUST be represented as a JSON object whose keys are the property names and whose values +are the corresponding TypedValues.*# + +[tck-testable tck-id-payloads-json-propertyset-unique]#[yellow-background]*[tck-id-payloads-json-propertyset-unique] Property +names within a single PropertySet MUST be unique.*# + +[tck-not-testable tck-id-payloads-json-propertyset-unordered]#[yellow-background]*[tck-id-payloads-json-propertyset-unordered] The +order of the keys of a PropertySet MUST NOT be significant, and an application MUST NOT encode +meaning in it.*# + +Non-normative comment: RFC 8259 defines a JSON object as an unordered collection, so a parser is +free to present the keys in any order and key order cannot be relied upon in any case. The +requirement is on the publisher not to attach meaning to it. Uniqueness is stated separately +because it is observable in a payload, whereas the absence of meaning in the ordering is not. + +[[payloads_json_array_types]] +===== Array Types + +In the Protocol Buffers encoding the array DataTypes are carried as packed little-endian bytes, as +described in the link:#payloads_c_datatype_array[Array Types] section. Carrying those same packed +bytes in JSON would make an array value opaque to a JSON consumer. + +[tck-testable tck-id-payloads-json-array-values]#[yellow-background]*[tck-id-payloads-json-array-values] A +value of an array DataType MUST be represented as a JSON array of the element values, not as a +base64 encoded packed byte array.*# + +[tck-testable tck-id-payloads-json-array-element-type]#[yellow-background]*[tck-id-payloads-json-array-element-type] Each +element of that JSON array MUST be represented according to the scalar mapping for the element +type, so that the elements of an Int64Array and a DateTimeArray are JSON strings and the elements +of an Int32Array are JSON numbers.*# + +[[payloads_json_dataset]] +===== DataSet + +In the Protocol Buffers schema a DataSet.Column carries its values in one of six parallel arrays, +one per underlying Protocol Buffers type, of which exactly one is used. In JSON the column's +datatype already identifies the element type. + +[tck-testable tck-id-payloads-json-dataset-values]#[yellow-background]*[tck-id-payloads-json-dataset-values] A +DataSet.Column MUST carry its values as a single JSON array under the key 'values', rather than as +one of the typed value arrays named in the Google Protocol Buffers schema, and each element MUST be +represented according to the scalar mapping for the column's datatype.*# + +[[payloads_json_content]] +===== Content and Compression + +[tck-testable tck-id-payloads-json-content-key]#[yellow-background]*[tck-id-payloads-json-content-key] An +uncompressed payload MUST carry its content under the key 'content', and a compressed payload MUST +carry it under the key 'compressed', matching the two arms of the payload_content oneof in the +Google Protocol Buffers schema.*# + +[tck-testable tck-id-payloads-json-compressed-base64]#[yellow-background]*[tck-id-payloads-json-compressed-base64] The +value of 'compressed' MUST be a JSON string containing the base64 encoding of the compressed bytes, +and those bytes MUST be the compressed form of the UTF-8 JSON text of the Content object.*# + +Non-normative comment: The header of a compressed JSON payload is a sibling of the 'compressed' key +and remains readable without decompressing, exactly as in the Protocol Buffers encoding. + +[[payloads_json_schema]] +==== JSON Schema + +The following JSON Schema describes a Sparkplug JSON payload. It is normative for the structure and +types it expresses. Requirements stated elsewhere in this specification that a JSON Schema cannot +express, such as the relationship between a metric's datatype and the form of its value, apply in +addition. + +[tck-testable tck-id-payloads-json-schema-valid]#[yellow-background]*[tck-id-payloads-json-schema-valid] A +Sparkplug JSON payload MUST validate against the JSON Schema below.*# + +---- +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://sparkplug.eclipse.org/specification/4.0.0/sparkplug_j.json", + "title": "Sparkplug JSON Payload", + "type": "object", + "properties": { + "header": { "$ref": "#/$defs/header" }, + "content": { "$ref": "#/$defs/content" }, + "compressed": { "$ref": "#/$defs/base64" } + }, + "not": { "required": [ "content", "compressed" ] }, + "$defs": { + "uint64": { + "comment": "an unsigned 64-bit value: 0 to 18446744073709551615. The string form is bounded by length only; the exact upper bound is normative prose, not expressible as a readable pattern.", + "anyOf": [ + { "type": "integer", "minimum": 0, "maximum": 18446744073709551615 }, + { "type": "string", "pattern": "^(0|[1-9][0-9]{0,19})$" } + ] + }, + "sequence64": { + "comment": "a Sparkplug sequence number or timestamp: 0 to 9223372036854775807. The string form is bounded by length only; the exact upper bound is normative prose, not expressible as a readable pattern.", + "anyOf": [ + { "type": "integer", "minimum": 0, "maximum": 9223372036854775807 }, + { "type": "string", "pattern": "^(0|[1-9][0-9]{0,18})$" } + ] + }, + "base64": { "type": "string", "contentEncoding": "base64" }, + "number": { + "oneOf": [ + { "type": "number" }, + { "enum": [ "NaN", "Infinity", "-Infinity" ] } + ] + }, + "datatype": { + "oneOf": [ + { "type": "string" }, + { "type": "integer", "minimum": 0 } + ] + }, + "header": { + "type": "object", + "properties": { + "timestamp": { "$ref": "#/$defs/sequence64" }, + "seq": { "$ref": "#/$defs/sequence64" }, + "bd_seq": { "$ref": "#/$defs/sequence64" }, + "descriptor_seq": { "$ref": "#/$defs/sequence64" }, + "definition_id": { "type": "string" }, + "previous_definition_id": { "type": "string" }, + "source": { "type": "string" }, + "uuid": { "type": "string" }, + "compression_algorithm": { "type": "string" } + } + }, + "content": { + "type": "object", + "properties": { + "metrics": { "type": "array", "items": { "$ref": "#/$defs/metric" } }, + "body": { "$ref": "#/$defs/base64" } + } + }, + "value": { + "anyOf": [ + { "type": [ "string", "boolean" ] }, + { "$ref": "#/$defs/number" }, + { "type": "array" }, + { "$ref": "#/$defs/dataset" }, + { "$ref": "#/$defs/template" }, + { "$ref": "#/$defs/propertyset" }, + { "type": "array", "items": { "$ref": "#/$defs/propertyset" } } + ] + }, + "metric": { + "type": "object", + "properties": { + "name": { "type": "string" }, + "alias": { "$ref": "#/$defs/uint64" }, + "timestamp": { "$ref": "#/$defs/sequence64" }, + "datatype": { "$ref": "#/$defs/datatype" }, + "quality": { "type": "integer", "minimum": 0 }, + "quality_context": { "type": "string" }, + "source": { "type": "string" }, + "is_historical": { "type": "boolean" }, + "is_transient": { "type": "boolean" }, + "is_null": { "type": "boolean" }, + "metadata": { "$ref": "#/$defs/metadata" }, + "properties": { "$ref": "#/$defs/propertyset" }, + "value": { "$ref": "#/$defs/value" }, + "command": { "$ref": "#/$defs/command" }, + "operation": { "enum": [ "ADD", "UPDATE", "DELETE", 1, 2, 3 ] } + } + }, + "propertyset": { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/propertyvalue" } + }, + "propertyvalue": { + "type": "object", + "properties": { + "datatype": { "$ref": "#/$defs/datatype" }, + "is_null": { "type": "boolean" }, + "value": { "$ref": "#/$defs/value" } + } + }, + "metadata": { + "type": "object", + "properties": { + "is_multi_part": { "type": "boolean" }, + "content_type": { "type": "string" }, + "size": { "$ref": "#/$defs/uint64" }, + "seq": { "$ref": "#/$defs/uint64" }, + "file_name": { "type": "string" }, + "file_type": { "type": "string" }, + "md5": { "type": "string" }, + "description": { "type": "string" }, + "engUnit": { "type": "string" } + } + }, + "dataset": { + "type": "object", + "properties": { + "columns": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { "type": "string" }, + "datatype": { "$ref": "#/$defs/datatype" }, + "values": { "type": "array" } + } + } + } + } + }, + "template": { + "type": "object", + "properties": { + "version": { "type": "string" }, + "template_ref": { "type": "string" }, + "is_definition": { "type": "boolean" }, + "metrics": { "type": "array", "items": { "$ref": "#/$defs/metric" } }, + "parameters": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { "type": "string" }, + "type": { "$ref": "#/$defs/datatype" }, + "is_null": { "type": "boolean" }, + "value": { "$ref": "#/$defs/value" } + } + } + } + } + }, + "command": { + "type": "object", + "properties": { + "arguments": { "type": "array", "items": { "$ref": "#/$defs/metric" } }, + "results": { "type": "array", "items": { "$ref": "#/$defs/metric" } }, + "correlation_id": { "type": "string" }, + "status": { + "type": "object", + "properties": { + "code": { "$ref": "#/$defs/uint64" }, + "message": { "type": "string" } + } + } + } + } + } +} +---- + +[[payloads_json_example]] +==== JSON Payload Example + +The following is a non-normative example of one DDATA message shown in both encodings, so the two +may be compared directly. The payload reports a single aliased Float metric. + +The spCv4.0 form, shown as a structural illustration of the Protocol Buffers message: + +---- +Acme/Dublin/Packaging/spCv4.0/DDATA/EdgeNode1/Line3 +---- + +---- +{ + "header": { + "timestamp": 1486144502122847913, + "seq": 4, + "bd_seq": 0, + "descriptor_seq": 2, + "definition_id": "a1c9...77" + }, + "content": { + "metrics": [{ + "alias": 1, + "datatype": "Float", + "value": 1802.5 + }] + } +} +---- + +The same message on the spJv4.0 namespace. The payload below is the literal UTF-8 text published as +the MQTT message payload, with the 64-bit fields represented as strings: + +---- +Acme/Dublin/Packaging/spJv4.0/DDATA/EdgeNode1/Line3 +---- + +---- +{ + "header": { + "timestamp": "1486144502122847913", + "seq": "4", + "bd_seq": "0", + "descriptor_seq": "2", + "definition_id": "a1c9...77" + }, + "content": { + "metrics": [{ + "alias": "1", + "datatype": "Float", + "value": 1802.5 + }] + } +} +---- + +Non-normative comment: The examples given throughout the rest of this chapter use the same JSON +notation to illustrate the structure of Protocol Buffers messages, and predate the JSON encoding. +They show field names and nesting rather than the exact text of a published spJv4.0 payload, and in +particular they write 64-bit values as JSON numbers. A published spJv4.0 payload follows the rules +in this section. + +[[payloads_json_choosing]] +==== Choosing a Payload Encoding + +Non-normative comment: The two encodings are interchangeable in terms of the information they +carry, so the choice between them is made on operational grounds. The Protocol Buffers encoding +produces smaller messages and is cheaper to encode and decode. The JSON encoding is readable +without tooling and can be consumed by any application that has a JSON parser, which is most of +them. + +Non-normative comment: The size difference is largest on small messages and narrows as messages +grow, because a larger message amortizes the per-message overhead over more metrics while the +per-metric overhead of JSON keys stays proportional. For representative payloads the JSON encoding +is roughly two to three times the size of the Protocol Buffers encoding. The difference in +encode and decode cost is of roughly one order of magnitude, and unlike the size difference it does +not narrow with message size. + +Non-normative comment: The size difference largely disappears when the content section is +compressed, as described in the link:#payloads_c_compression[Compression] section. The repeated +field names that make JSON large are highly compressible, so for payloads of more than a few +metrics the two encodings compress to within a few percent of one another. On a bandwidth +constrained link where compression is already in use, message size is therefore not a strong reason +to prefer one encoding over the other; encode and decode cost remains one. + +Non-normative comment: Taken together, the Protocol Buffers encoding is the better fit for Edge +Nodes on constrained hardware, on metered or low bandwidth links, and wherever messages are small +and frequent, which is the case Sparkplug was originally designed around. The JSON encoding is the +better fit where Sparkplug data is consumed directly by applications that cannot readily add a +Protocol Buffers dependency, and where readability during development and troubleshooting is worth +more than the bytes. A deployment may use both, with a bridging application republishing one +namespace onto the other, so that constrained Edge Nodes publish spCv4.0 while enterprise consumers +subscribe to spJv4.0.