Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
244 changes: 115 additions & 129 deletions specification/src/main/asciidoc/chapters/Sparkplug_6_Payloads.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,7 @@ message Payload {
Template template_value = 13;
PropertySet propertyset_value = 14;
PropertySetList propertysets_value = 15;
ValueArray array_value = 16;
}

}
Expand Down Expand Up @@ -251,19 +252,31 @@ message Payload {
Status status = 4; // Execution status - present in NRESP/DRESP responses, omitted in BIRTH definitions and CMD requests
}

// An array of scalar values of one type, used both by the array DataTypes and by
// DataSet.Column. At most one of the typed arrays is populated - none for an empty array -
// chosen by the datatype of the enclosing Metric or Column. The signed arrays use zigzag
// encoding because a negative plain varint occupies ten bytes.
message ValueArray {
repeated sint32 sint32_values = 1; // Int8, Int16, Int32
repeated sint64 sint64_values = 2; // Int64
repeated uint32 uint32_values = 3; // UInt8, UInt16, UInt32
repeated uint64 uint64_values = 4; // UInt64
repeated int64 int64_values = 5; // DateTime; not zigzag - pre-1970 costs 10 bytes
repeated float float_values = 6; // Float
repeated double double_values = 7; // Double
repeated bool boolean_values = 8; // Boolean
repeated string string_values = 9; // String, Text

// The indexes of the elements that are null. Omitted entirely when none are.
repeated uint32 null_indexes = 10;
}

message DataSet {

message Column {
string name = 1;
DataType datatype = 2;

// Only one type of repeated value shall be used per Column
repeated uint32 int_values = 3;
repeated uint64 long_values = 4;
repeated float float_values = 5;
repeated double double_values = 6;
repeated bool boolean_values = 7;
repeated string string_values = 8;
string name = 1;
DataType datatype = 2;
ValueArray values = 3; // one entry per row, in row order
}

repeated Column columns = 1;
Expand Down Expand Up @@ -1057,55 +1070,21 @@ Column MUST include a datatype, which is the datatype of every value in that col
datatype MUST be one of the enumerated values shown in the
link:#payloads_c_datatype_basic[Sparkplug Basic Data Types].*#
* *values*
** The values of a Column are carried in one of six repeated fields, selected by the Column's
datatype as shown in the table below.
** [tck-testable tck-id-payloads-dataset-column-values-field]#[yellow-background]*[tck-id-payloads-dataset-column-values-field] The
values of a Column MUST be carried in the repeated field that corresponds to that Column's datatype
as shown in the DataSet Column Value Fields table.*#
** [tck-testable tck-id-payloads-dataset-column-values-one]#[yellow-background]*[tck-id-payloads-dataset-column-values-one] A
Column MUST NOT populate more than one of the six repeated value fields.*#
** [tck-testable tck-id-payloads-dataset-column-values-signed]#[yellow-background]*[tck-id-payloads-dataset-column-values-signed] A
signed value carried in int_values or long_values MUST be encoded as its two's complement
representation, and a consuming application MUST interpret it according to the Column's datatype.*#

The following table defines which repeated field carries the values of a Column for each datatype.

.DataSet Column Value Fields
[cols="2,2,4"]
|===
|*Column datatype*
|*Field*
|*Google Protocol Buffers Type*

|Int8, Int16, Int32, UInt8, UInt16, UInt32
|int_values
|repeated uint32, packed

|Int64, UInt64, DateTime
|long_values
|repeated uint64, packed

|Float
|float_values
|repeated float, packed

|Double
|double_values
|repeated double, packed

|Boolean
|boolean_values
|repeated bool, packed

|String, Text
|string_values
|repeated string, not packed
|===

Non-normative comment: There is no signed repeated field, so the signed datatypes share int_values
and long_values with their unsigned counterparts. A publishing application writes the value's two's
complement bit pattern and a consuming application narrows it back to the declared datatype, which
is the same convention the Metric value fields use for Int8 and Int16.
** The values of a Column are carried in a ValueArray, one entry per row, in row order. The typed
array within it is selected by the Column's datatype, as described in the
link:#payloads_c_datatype_array[Array Types] section, which defines the same structure for the
array DataTypes.
** [tck-testable tck-id-payloads-dataset-column-values-field]#[yellow-background]*[tck-id-payloads-dataset-column-values-field] A
DataSet.Column MUST carry its values in a ValueArray, populating only the typed array that
corresponds to that Column's datatype.*#
** [tck-testable tck-id-payloads-dataset-column-null]#[yellow-background]*[tck-id-payloads-dataset-column-null] A
null cell MUST be recorded by its row index in that ValueArray's null_indexes field, so that the
column keeps one entry per row.*#
** Non-normative comment: a Sparkplug 3 DataSet was row oriented and a null cell was an empty
DataSetValue, which held its position with no value set. A column oriented DataSet has no
equivalent, because a repeated field is dense, so null_indexes restores what the row oriented form
could express. Without it an encoder would have to omit the cell, which shortens the column and
shifts every later row, or write a zero, which is indistinguishable from a real zero.

[[payloads_c_dataset_example]]
===== DataSet Example
Expand All @@ -1121,15 +1100,15 @@ two rows. The first row is (1, 72.5, true) and the second is (2, 80.0, false).
"columns": [{
"name": "Step",
"datatype": "UInt32",
"int_values": [1, 2]
"values": { "uint32_values": [1, 2] }
}, {
"name": "Setpoint",
"datatype": "Double",
"double_values": [72.5, 80.0]
"values": { "double_values": [72.5, 80.0] }
Comment thread
nathandavenport marked this conversation as resolved.
Resolved
}, {
"name": "Enabled",
"datatype": "Boolean",
"boolean_values": [true, false]
"values": { "boolean_values": [true, false] }
}]
}
}
Expand Down Expand Up @@ -1747,9 +1726,9 @@ enum DataType {
** Non-normative comment: a DateTime value is a nanosecond count since epoch, the same quantity the
payload and metric timestamps carry, and is signed for the same reason. Every mainstream time
representation it interoperates with is signed, so the ceiling is 2262-04-11 as described in the
link:#operational_behavior_timestamps[Timestamps] section. A DateTime carried in a DataSet.Column
shares the long_values array with UInt64, as the
link:#payloads_c_dataset_column[DataSet.Column] section describes.
link:#operational_behavior_timestamps[Timestamps] section. A DateTime carried in an array or a DataSet.Column
uses the int64_values array of the ValueArray, as the
link:#payloads_c_datatype_array[Array Types] section describes.
* _Text_
** String value (UTF-8)
** Google Protocol Buffer Type: string
Expand Down Expand Up @@ -1792,84 +1771,77 @@ link:#payloads_c_dataset_column[DataSet.Column] section describes.
[[payloads_c_datatype_array]]
* *Array Types*

All array types use the bytes_value field of the Metric value field. They are simply little-endian
packed byte arrays.

For example, consider an Int32 array with two decimal values [123456789, 987654321]

Array converted to little endian hex: [0x15CD5B07, 0xB168DE3A]

The bytes_value of the Sparkplug Metric must be: [0x15, 0xCD, 0x5B, 0x07, 0xB1, 0x68, 0xDE, 0x3A]
An array DataType carries its elements in a ValueArray, in the array_value field of the Metric's
TypedValue. At most one of the ValueArray's typed arrays is populated, selected by the datatype as
listed below, and the elements appear in order; an array with no elements populates none.

** [tck-testable tck-id-payloads-array-value-array]#[yellow-background]*[tck-id-payloads-array-value-array] A
Metric whose datatype is an array type MUST carry its elements in the array_value field of its
TypedValue.*#
** [tck-testable tck-id-payloads-array-single-field]#[yellow-background]*[tck-id-payloads-array-single-field] A
ValueArray MUST populate at most one typed array, which MUST be the one that corresponds to the
datatype of the enclosing Metric or DataSet.Column. An array with no elements populates none.*#
** [tck-testable tck-id-payloads-array-null-indexes]#[yellow-background]*[tck-id-payloads-array-null-indexes] An
element that is null MUST be recorded by its index in the null_indexes field, and the typed array
MUST still hold a value at that index so that every element keeps its position.*#
** [tck-testable tck-id-payloads-array-null-indexes-range]#[yellow-background]*[tck-id-payloads-array-null-indexes-range] Every
Comment thread
nathandavenport marked this conversation as resolved.
Resolved
value in null_indexes MUST be a valid index of the populated typed array, MUST NOT appear more than
once, and the indexes MUST appear in ascending order.*#
** [tck-not-testable tck-id-payloads-array-null-value-ignored]#[yellow-background]*[tck-id-payloads-array-null-value-ignored] A
consuming application MUST treat an element whose index appears in null_indexes as null, whatever
value the typed array holds at that index.*#
** Non-normative comment: the signed arrays use zigzag encoding because a negative value in a plain
varint field occupies ten bytes, which would make a signed array substantially larger than the
packed byte array this replaces. DateTime is the exception: its values are large positive nanosecond
counts, for which zigzag only adds a bit, so it uses a plain varint array.
** Non-normative comment: null_indexes exists because a repeated field is dense and has no
per-element presence. Recording the indexes rather than carrying a parallel array of flags costs
nothing when an array has no nulls, which is the ordinary case, and a few bytes per null otherwise.
A publisher still writes some value at a null element's position, and a consuming application
ignores it.
** For example, an Int32Array of [123456789, 987654321] populates sint32_values with those two
elements. An Int32Array of [10, null, 30] populates sint32_values with three elements and sets
null_indexes to [1].

** _Int8Array_
*** Int8Array as an array of packed little endian int8 bytes
*** Google Protocol Buffer Type: bytes
*** Elements are carried in the sint32_values field of the ValueArray
*** Sparkplug enum value: 22
*** Example (Decimal to Metric bytes_value): [-23, 123] -> [0xE9, 0x7B]
** _Int16Array_
*** Int16Array as an array of packed little endian int16 bytes
*** Google Protocol Buffer Type: bytes
*** Elements are carried in the sint32_values field of the ValueArray
*** Sparkplug enum value: 23
*** Example (Decimal to Metric bytes_value): [-30000, 30000] -> [0xD0, 0x8A, 0x30, 0x75]
** _Int32Array_
*** Int8Array as an array of packed little endian int32 bytes
*** Google Protocol Buffer Type: bytes
*** Elements are carried in the sint32_values field of the ValueArray
*** Sparkplug enum value: 24
*** Example (Decimal to Metric bytes_value): [-1, 315338746] -> [0xFF, 0xFF, 0xFF, 0xFF, 0xFA, 0xAF, 0xCB, 0x12]
** _Int64Array_
*** Int8Array as an array of packed little endian int64 bytes
*** Google Protocol Buffer Type: bytes
*** Elements are carried in the sint64_values field of the ValueArray
*** Sparkplug enum value: 25
*** Example (Decimal to Metric bytes_value): [-4270929666821191986, -3601064768563266876] -> [0xCE, 0x06, 0x72, 0xAC, 0x18, 0x9C, 0xBA, 0xC4, 0xC4, 0xBA, 0x9C, 0x18, 0xAC, 0x72, 0x06, 0xCE]
** _UInt8Array_
*** UInt8Array as an array of packed little endian uint8 bytes
*** Google Protocol Buffer Type: bytes
*** Elements are carried in the uint32_values field of the ValueArray
*** Sparkplug enum value: 26
*** Example (Decimal to Metric bytes_value): [23, 250] -> [0x17, 0xFA]
** _UInt16Array_
*** UInt16Array as an array of packed little endian uint16 bytes
*** Google Protocol Buffer Type: bytes
*** Elements are carried in the uint32_values field of the ValueArray
*** Sparkplug enum value: 27
*** Example (Decimal to Metric bytes_value): [30, 52360] -> [0x1E, 0x00, 0x88, 0xCC]
** _UInt32Array_
*** UInt32Array as an array of packed little endian uint32 bytes
*** Google Protocol Buffer Type: bytes
*** Elements are carried in the uint32_values field of the ValueArray
*** Sparkplug enum value: 28
*** Example (Decimal to Metric bytes_value): [52, 3293969225] -> [0x34, 0x00, 0x00, 0x00, 0x49, 0xFB, 0x55, 0xC4]
** _UInt64Array_
*** UInt64Array as an array of packed little endian uint64 bytes
*** Google Protocol Buffer Type: bytes
*** Elements are carried in the uint64_values field of the ValueArray
*** Sparkplug enum value: 29
*** Example (Decimal to Metric bytes_value): [52, 16444743074749521625] -> [0x34, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xD9, 0x9E, 0x02, 0xD1, 0xB2, 0x76, 0x37, 0xE4]
** _FloatArray_
*** FloatArray as an array of packed little endian 32-bit float bytes
*** Google Protocol Buffer Type: bytes
*** Elements are carried in the float_values field of the ValueArray
*** Sparkplug enum value: 30
*** Example (Decimal to Metric bytes_value): [1.23, 89.341] -> [0xA4, 0x70, 0x9D, 0x3F, 0x98, 0xAE, 0xB2, 0x42]
** _DoubleArray_
*** DoubleArray as an array of packed little endian 64-bit float bytes
*** Google Protocol Buffer Type: bytes
*** Elements are carried in the double_values field of the ValueArray
*** Sparkplug enum value: 31
*** Example (Decimal to Metric bytes_value): [12.354213, 1022.9123213] -> [0xD7, 0xA2, 0x05, 0x68, 0x5B, 0xB5, 0x28, 0x40, 0x8E, 0x17, 0x1C, 0x6F, 0x4C, 0xF7, 0x8F, 0x40]
** _BooleanArray_
*** BooleanArray as an array of bit-packed bytes preceeded by a 4-byte integer that represents the
total number of boolean values
*** Google Protocol Buffer Type: bytes
*** Elements are carried in the boolean_values field of the ValueArray
*** Sparkplug enum value: 32
*** Example (boolean array to Metric bytes_value): [false, false, true, true, false, true, false, false, true, true, false, true] -> [0x0C, 0x00, 0x00, 0x00, 0x34, 0xDX]
**** Note an 'X' above is a 'do not care'. It can be either 1 or 0 but must be present so the array
ends on a byte boundary.
** _StringArray_
*** StringArray as an array of null terminated strings
*** Google Protocol Buffer Type: bytes
*** Elements are carried in the string_values field of the ValueArray
*** Sparkplug enum value: 33
*** Example (string array to Metric bytes_value): [ABC, hello] -> [0x41, 0x42, 0x43, 0x00, 0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x00]
** _DateTimeArray_
*** DateTimeArray as an array of packed little endian bytes where each DateTime value is a signed
8-byte value representing the number of nanoseconds since epoch in UTC
*** Google Protocol Buffer Type: bytes
*** Elements are carried in the int64_values field of the ValueArray
*** Sparkplug enum value: 34
*** Example (DateTime array -> nanoseconds since epoch -> Metric bytes_value): ['Wednesday, October 21, 2009 5:27:55.335 AM', 'Friday, June 24, 2022 9:57:55 PM'] -> [1256102875335000000, 1656107875000000000] -> [0xC0, 0x7F, 0xB0, 0xF5, 0xE8, 0x92, 0x6E, 0x11, 0x00, 0x1E, 0x1A, 0x7F, 0x56, 0xAD, 0xFB, 0x16]

[[payloads_c_datatype_command]]
* *Command Type*
Expand Down Expand Up @@ -3791,13 +3763,19 @@ because it is observable in a payload, whereas the absence of meaning in the ord
[[payloads_json_array_types]]
===== Array Types

In the Protocol Buffers encoding the array DataTypes are carried as packed little-endian bytes, as
described in the link:#payloads_c_datatype_array[Array Types] section. Carrying those same packed
bytes in JSON would make an array value opaque to a JSON consumer.
In the Protocol Buffers encoding the array DataTypes and a DataSet.Column carry their elements in a
ValueArray, which selects one of several typed arrays and records absent elements separately, as
described in the link:#payloads_c_datatype_array[Array Types] section. In JSON the datatype is
already known and a JSON array can hold a null directly, so a ValueArray is represented as a plain
array.

[tck-testable tck-id-payloads-json-array-values]#[yellow-background]*[tck-id-payloads-json-array-values] A
value of an array DataType MUST be represented as a JSON array of the element values, not as a
base64 encoded packed byte array.*#
ValueArray MUST be represented as a JSON array of its element values, rather than as an object
carrying one of the typed arrays named in the Google Protocol Buffers schema.*#

[tck-testable tck-id-payloads-json-array-null]#[yellow-background]*[tck-id-payloads-json-array-null] An
element whose index appears in null_indexes MUST be represented as a JSON null, and the JSON array
MUST NOT carry a separate null_indexes key.*#

[tck-testable tck-id-payloads-json-array-element-type]#[yellow-background]*[tck-id-payloads-json-array-element-type] Each
element of that JSON array MUST be represented according to the scalar mapping for the element
Expand All @@ -3807,14 +3785,14 @@ of an Int32Array are JSON numbers.*#
[[payloads_json_dataset]]
===== DataSet

In the Protocol Buffers schema a DataSet.Column carries its values in one of six parallel arrays,
one per underlying Protocol Buffers type, of which exactly one is used. In JSON the column's
datatype already identifies the element type.
In the Protocol Buffers schema a DataSet.Column carries its values in a ValueArray, which selects
a typed array by the column's datatype and records absent cells separately. In JSON the datatype is
already known and a JSON array can hold a null directly.

[tck-testable tck-id-payloads-json-dataset-values]#[yellow-background]*[tck-id-payloads-json-dataset-values] A
DataSet.Column MUST carry its values as a single JSON array under the key 'values', rather than as
one of the typed value arrays named in the Google Protocol Buffers schema, and each element MUST be
represented according to the scalar mapping for the column's datatype.*#
DataSet.Column MUST carry its values as a single JSON array under the key 'values', one entry per
row, with each element represented according to the scalar mapping for the column's datatype and a
null cell represented as a JSON null.*#

[[payloads_json_content]]
===== Content and Compression
Expand Down Expand Up @@ -3903,11 +3881,19 @@ Sparkplug JSON payload MUST validate against the JSON Schema below.*#
"body": { "$ref": "#/$defs/base64" }
}
},
"arrayElement": {
"comment": "an element of an array DataType or DataSet.Column; null marks an absent element",
"anyOf": [
{ "type": "null" },
{ "type": [ "string", "boolean" ] },
{ "$ref": "#/$defs/number" }
]
},
"value": {
"anyOf": [
{ "type": [ "string", "boolean" ] },
{ "$ref": "#/$defs/number" },
{ "type": "array" },
{ "type": "array", "items": { "$ref": "#/$defs/arrayElement" } },
{ "$ref": "#/$defs/dataset" },
{ "$ref": "#/$defs/template" },
{ "$ref": "#/$defs/propertyset" },
Expand Down Expand Up @@ -3970,7 +3956,7 @@ Sparkplug JSON payload MUST validate against the JSON Schema below.*#
"properties": {
"name": { "type": "string" },
"datatype": { "$ref": "#/$defs/datatype" },
"values": { "type": "array" }
"values": { "type": "array", "items": { "$ref": "#/$defs/arrayElement" } }
}
}
}
Expand Down
Loading
Loading