Skip to content

[#231] Define the spJv4.0 JSON payload encoding - #645

Merged
wes-johnson merged 15 commits into
eclipse-sparkplug:developfrom
wes-johnson:231
Oct 29, 2026
Merged

wes-johnson merged 15 commits into
eclipse-sparkplug:developfrom
wes-johnson:231

Conversation

@wes-johnson

@wes-johnson wes-johnson commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Closes #231.

Larger than "add a JSON encoding." It also halves the rollover point of the three sequence
numbers, changes the declared type of timestamp, renames a oneof, requires payload metrics to
be ordered, and corrects a sequence-arithmetic bug. Those apply to spCv4.0 whether or not an
implementation ever uses JSON. See Changes to spCv4.0.

Defines the JSON payload encoding and binds it to the spJv4.0 namespace, so an Edge Node may publish
either encoding on the wire. Topic structure, operational behavior, sequence numbering and session
management are unchanged and remain encoding independent — only the payload representation differs.

38 new assertions, none removed.

Changes to spCv4.0

These are independent of the JSON work. They are here because the JSON encoding forced the question
of what a 64-bit field actually holds, and surfaced two existing defects.

None of the type changes is a wire break. protobuf encodes int64 and uint64 as the same
varint and declares them wire-compatible. For every value both types can hold the bytes are
identical and either schema decodes the other's output — verified at 0, a realistic nanosecond
timestamp, and 2^63-1. What changes is generated source: accessors change type in languages that
distinguish the two.

The three sequence numbers stay uint64, but their rollover point moves to 9223372036854775807.
This is the one genuinely normative change of the three — it rescales the bound in 38 statements
across Chapters 4, 5 and 6, halving the range. An unsigned type is correct for a counter that can
never be negative and keeps seq, bd_seq, and descriptor_seq consistent with alias,
MetaData.size, and Status.code. Capping below the type maximum guarantees every value a
conformant Edge Node can produce also fits a signed 64-bit integer, so an implementation in a
language with no unsigned 64-bit type — Java among them — never needs unsigned comparison. Neither
bound is reachable: at a million messages per second the capped range lasts ~292,000 years. The
repeated (max value of a UINT64) parenthetical is dropped rather than rescaled, since the number
is no longer the type's maximum; the cap is explained once under seq in the Header section.

timestamp moves to int64, in both the Header and the Metric. Every mainstream time
representation it interoperates with is signed — protobuf's own Timestamp, OPC UA DateTime,
java.time.Instant — so an unsigned timestamp is narrowed at nearly every boundary. The ceiling
moves from 2555 to 2262-04-11; the note from #232 states that date directly rather than
describing a range no mainstream time library exposes.

The content oneof is renamed payload_content, so its uncompressed arm can be named content
instead of plain. The oneof was named content while its arm was plain, so a Protocol Buffers
JSON implementation emitted the key "plain" while every example in the chapter used "content" —
a mismatch dating to #622. A field may not share its oneof's name, so the oneof moves instead. The
examples were right; the schema now agrees with them, and a stock library emits the correct key with
no post-processing. Three prose references to "the plain field" are updated, including the normative
payloads-compression-content-only.

Payload metrics MUST now be ordered by value change time, oldest first. A consumer applying
metrics in array order then arrives at the correct current value without comparing timestamps, which
matters when one payload carries several values for the same metric, as a payload of historical
values does. A metric omitting its timestamp sorts at the payload timestamp, which is the time it is
already taken to have changed. Metrics sharing a timestamp may appear in any order and a consumer
must not reorder them — multi-part file publishing depends on that, since it identifies a metric by
position and all of its metrics carry the same timestamp, so they tie. Non-normative comments in
both sections record the interaction.

The NREBIRTH rsn calculation was off by one. It said:

rsn = pds - (1 + dc)
If the rsn results in a negative number, add <max> to it.

A counter cycling 0..MAX has period MAX + 1, so adding MAX lands one short — and with the
rollover point at 2^63-1, MAX + 1 no longer fits a signed 64-bit integer, so the old form cannot
be corrected in place. Restated as two cases, so no intermediate is ever negative or out of range:

if pds >= 1 + dc:   rsn = pds - (1 + dc)
otherwise:          rsn = 9223372036854775807 - (dc - pds)

Verified equivalent to (pds - (1 + dc)) mod 2^63 over 20,070 cases including every small
combination and both rollover boundaries.

MetaData.seq, which sequences the parts of a multi-part file transfer, is a different field and is
unchanged.

The JSON encoding

It is self-contained

The section defines the encoding completely; conformance does not depend on any document outside
this specification. The scalar rules were chosen to agree with the Protocol Buffers JSON Mapping,
so the forms are familiar and a library produces them directly, but that agreement is a convenience
and not a normative dependency — the mapping is vendor documentation rather than a versioned
standard, it is specified for proto3 while the Sparkplug schema is proto2, and it does not define
extensions at all.

Tahu 3.x's payload.json was taken as input but not as the source of truth: it disagrees with the
code it validates (datatype/metadata in the schema vs dataType/metaData in the Jackson
annotations), names Template.reference where the schema says template_ref, leaves
SparkplugBPayload, PropertyValue, PropertySet, MetaData and Row unannotated so their keys
are whatever Jackson's default bean naming produces, never states that body is base64, and has no
is_null or array types.

64-bit integers are encoded as JSON strings

RFC 8259 does not require a JSON number to carry more precision than an IEEE 754 double, and most
parsers — including every browser's — decode into a double unconditionally. Since #232 made every
Sparkplug timestamp a nanosecond count, timestamps sit far above the 2^53 limit:

on the wire : 1486144502122847913
JSON.parse  : 1486144502122848000     <- 87 ns, silently

The rule is driven by declared field type, not magnitude. That matters for the sequence numbers:
each counts to the rollover point, so one is small early in a session and large late in it — a
numeric form would be correct in testing and lossy in production.

Deviations from canonical JSON

Each exists so the JSON reads as JSON rather than as a transliteration of a binary format:

Protobuf spJv4.0
TypedValue nested submessage repeating the datatype the value itself under value
PropertySet parallel keys[] / values[] object keyed by property name
Array DataTypes packed little-endian bytes in bytes_value a JSON array of elements
DataSet.Column six parallel typed arrays one values array
Field names lowerCamelCase by default the proto field name verbatim

The array one is different in kind. A stock library emits "value": "Fc1bB7Fo3jo=" for an
Int32Array, which a JSON consumer cannot read at all without Sparkplug-specific unpacking —
defeating the purpose of offering JSON. #642 proposes fixing this at the schema level with native
packed repeated fields, which would be smaller for realistic process data and remove the deviation.

Contents

  • New Ch. 6 section payloads_json — overview, encoding selection, field naming, scalar mapping
    table, extension fields, the structural deviations, a formal JSON Schema (draft 2020-12), a
    worked DDATA in both encodings, and non-normative guidance on choosing an encoding.
  • Extension fields are defined: keyed by fully qualified name in square brackets, values follow
    the ordinary rules for their type, unrecognized extensions ignored. The schema reserves extension
    ranges in most messages, so leaving this undefined left something well-defined on the wire
    undefined in JSON. Noted: unlike the binary encoding, an unrecognized JSON key does not survive a
    round trip. (See also Decide the future of the protobuf extension ranges - unused since Sparkplug B, and undefined in JSON #643 — the ranges have never been used.)
  • Float precision: finite floats and doubles must carry enough digits to parse back exactly — up
    to 9 significant decimal digits for a float, 17 for a double.
  • Three Ch. 5 statements requiring "a Sparkplug Google Protobuf encoded payload" (Will Message,
    NBIRTH, DBIRTH) now require a payload encoded as defined for the namespace it is published on.
    Same ids, same requirement, generalized.
  • A note in Ch. 4 that topics written with spCv4.0 apply equally to spJv4.0 — one sentence rather
    than editing 122 topic strings.
  • Conformance profiles (Ch. 10) gain an encoding dimension: an Edge Node MUST support at least
    one encoding and MAY support both selected by configuration, but MUST use a single encoding and
    namespace for a whole MQTT session; Host Applications and Sparkplug Aware MQTT Servers MUST support
    both. A Sparkplug Compliant MQTT Server is unchanged — it treats payloads as opaque bytes, now
    stated non-normatively so the asymmetry does not read as an omission.

Example churn, flagged for review

Two renames so the examples match the encoding rules rather than contradicting them, since readers
will now reasonably read them as JSON documents:

  • "dataType" → "datatype" (26 occurrences). The protobuf field is datatype; dataType was
    never a field name — the same class of error as the dataType/type fix on parameters in [#602] Add non-normative guidance for Templates #629.
  • metric-level "template": {...} → "value": {...} (6 occurrences). A Template value is a value
    like any other, and datatype: "Template" already identifies it.

A non-normative comment states that the remaining examples illustrate structure and write 64-bit
values as JSON numbers, which a published spJv4.0 payload does not.

On the "protobuf is smaller and cheaper" claim

The issue asked whether this is actually true. Measured by compiling the spec's own schema with
protoc and benchmarking against equivalent JSON (protobuf upb C backend vs CPython's C json):

Payload protobuf JSON size encode decode
NDATA, 1 aliased metric 79 B 234 B 2.96x 10.8x 6.1x
NDATA, 10 aliased metrics 300 B 941 B 3.14x 10.1x 7.5x
NDATA, 100 aliased metrics 2508 B 8083 B 3.22x 9.9x 9.8x
NBIRTH, 100 named metrics 6208 B 13583 B 2.19x 10.7x 11.7x

Both halves hold, and the CPU gap is wider than the size gap. But the size advantage nearly
vanishes under compression
— deflate eats exactly the repeated field names that make JSON large:

Payload pb+deflate json+deflate ratio
NDATA, 10 metrics 209 B 299 B 1.43x
NDATA, 100 metrics 1030 B 1131 B 1.10x
NBIRTH, 100 metrics 1396 B 1475 B 1.06x

The non-normative guidance reflects this rather than the folklore: on a constrained link that
already compresses, size is not a strong reason to prefer one encoding; encode/decode cost still is.

Verification

  • Build passes. The schema compiles clean under protoc at every commit, and the int64/uint64
    wire equivalence was verified by encoding the same values under both declarations and diffing.
  • The JSON Schema is valid draft 2020-12, validates all 22 example payloads, and was
    negative-tested against 15 cases (9 malformed, 6 valid) — all behaved as specified. Those tests
    caught two real bugs while drafting: a oneOf that rejected "NaN" because it matched both the
    string and non-finite-number branches, and the template/value key conflict above.
  • The rsn formula was checked exhaustively against modular arithmetic.
  • Broken-anchor (10) and tck-id-anomaly (7) baselines unchanged; no new trailing whitespace.

Known gaps — decisions for the working group

DateTime moved to int64 too, in c0781dc, so a DateTime value and a payload timestamp no
longer differ in signedness. Unlike the field type changes above this one is a wire change: a
DateTime value moves from the uint64_value arm of the TypedValue oneof to int64_value, a
different field number. It is harmless only because spCv4.0 has not shipped.

STATE carries a nanosecond timestamp as a JSON number. STATE is JSON but explicitly not a
Sparkplug payload, so the string rule does not reach it. The example in the spec is safe only by
luck — 1486144502122000000 ends in six zeros and is exactly representable — but a real value is
not:

1486144502122847913  ->  1486144502122848000   err   -87 ns

Every Host Application publishing STATE with genuine sub-millisecond precision has the defect this
PR documents, and bd_seq acquires it as it grows. Fixing it changes STATE's wire format and
touches every Host Application, so it is not folded in here.

Addressed in review

  • STATE is stated as the one topic exempt from namespace substitution, so a JSON Edge Node no
    longer waits on a spJv4.0/STATE topic that is never published.
  • A Host Application addresses an Edge Node on the namespace of that Edge Node's NBIRTH, and an
    Edge Node subscribes to its command topics on the namespace it publishes on. Without this a
    Rebirth Request to a JSON Edge Node was accepted by the Server and never delivered.
  • The 64-bit JSON Schema definition is split: uint64 for alias, MetaData.size and
    Status.code, which are uncapped, and sequence64 for the sequence numbers and timestamps. The
    single capped definition rejected legal spCv4.0 values and contradicted the lossless conversion
    claim.
  • The Sparkplug Aware MQTT Server encoding requirement is conditional on replacing NDEATH
    timestamps, the only payload-touching behavior required of it.
  • A Metric's two copies of datatype and is_null must agree, the single JSON key represents
    both, and a null Template.Parameter now has a JSON form.
  • The metric ordering MUST is scoped to payloads carrying more than one value for the same metric,
    with a SHOULD elsewhere, and does not reach Template or Command metrics.

Not addressed

Adds a normative JSON encoding of the Sparkplug payload model, published on the
spJv4.0 namespace alongside the Protocol Buffers encoding on spCv4.0.

The mapping is derived from the Google Protocol Buffers JSON Mapping so that most
of it can be produced by library facilities, with Sparkplug's deviations stated
explicitly: TypedValue is flattened to a bare 'value', PropertySet is a JSON
object keyed by property name, array DataTypes are JSON arrays rather than base64
packed bytes, and a DataSet.Column carries one 'values' array.

64-bit integers are encoded as JSON strings. A v4 nanosecond timestamp is roughly
165 times larger than the 2^53 limit a JSON number is guaranteed to carry, so a
numeric form is silently rounded by any parser that decodes into a double.

Also generalizes three Ch. 5 statements that required a Protobuf encoded payload,
notes that topics written with spCv4.0 apply equally to spJv4.0, and renames the
'dataType' key in the Ch. 6 examples to 'datatype' and the metric-level 'template'
key to 'value' so the examples match the encoding rules.
…ing-encoded 64-bit values

The note claimed a Sparkplug sequence number is always small, which is wrong: seq,
bd_seq, and descriptor_seq each count to the maximum value of their declared type
before rolling over to zero. They are therefore subject to the same precision loss
as timestamps, and a numeric form would be correct early in a session and lossy
late in one.
The three message sequence numbers move from uint64 to int64 in the Protocol
Buffers schema, with the range and rollover point changing from
18446744073709551615 to 9223372036854775807 across all 38 statements that bound
them in Chapters 4, 5, and 6.

Protobuf uint64 is a signed long in Java, so the upper half of the uint64 range
was never reachable by an implementation built on it. Declaring int64 makes the
specified range the representable one.

MetaData.seq, which sequences the parts of a multi-part file transfer, is a
different field and is unchanged.
… rsn arithmetic

Header.timestamp and Metric.timestamp move from uint64 to int64, for the same
reason as the sequence numbers: protobuf uint64 is a signed long in Java, so the
upper half of the unsigned range was never reachable. The range note in Chapter 5
is rewritten accordingly - the ceiling is 2262-04-11, stated directly rather than
as a caveat on an unsigned range that no implementation could use.

The NREBIRTH rsn calculation said to add the maximum sequence number to a negative
result. That was off by one, since a counter cycling 0..MAX has a period of MAX+1,
and MAX+1 is not representable as an int64. Restated as two cases so no
intermediate is ever negative or out of range:

  if pds >= 1 + dc:  rsn = pds - (1 + dc)
  otherwise:         rsn = 9223372036854775807 - (dc - pds)

Verified equivalent to (pds - (1 + dc)) mod 2^63 over the full range.
…ing seq as unsigned

The seq, bd_seq, and descriptor_seq bullets in the Header section still described
each field as an unsigned 64-bit integer after the schema moved them to int64.
…ormance profiles

An Edge Node MUST support at least one encoding and MAY support both, selected by
configuration, but MUST use a single encoding and namespace for the whole of an
MQTT session - seq is Edge-Node-wide and bd_seq scopes the session, so splitting
one session across both namespaces would produce two unusable halves of one
sequence stream.

Host Applications and Sparkplug Aware MQTT Servers MUST support both encodings.
A Host Application cannot choose the encoding of the Edge Nodes it consumes, and
an Aware Server decodes payloads rather than treating them as opaque.

A Sparkplug Compliant MQTT Server has no payload requirements at all, which is now
stated non-normatively so the asymmetry with the Aware profile is not read as an
omission.

Also generalizes the Edge Node profile's reference to the spCv4.0/# namespace.
…g the capped range

seq, bd_seq, and descriptor_seq go back to uint64 while keeping the rollover point
at 9223372036854775807. timestamp stays int64.

An unsigned type is right for a counter that can never be negative, and it keeps
the three consistent with alias, MetaData.size, and Status.code. Capping the range
below the type's maximum still guarantees every value a conformant Edge Node can
produce fits a signed 64-bit integer, so an implementation in a language without
an unsigned 64-bit type never needs unsigned comparison. Neither bound is
reachable: at a million messages per second the capped range lasts ~292,000 years.

timestamp keeps int64 because every mainstream time representation it interoperates
with is signed - protobuf's own Timestamp, OPC UA DateTime, java.time.Instant - and
the 2262 ceiling it accepts is one none of those exceed either.

The repeated "(max value of a UINT64)" parenthetical is dropped rather than
rescaled, since the cap is now a deliberate restriction rather than the type's
maximum. It is explained once in the Header section instead.
…ary-support claim

The oneof was named 'content' and its uncompressed arm was named 'plain', so a
Protocol Buffers JSON implementation emitted the key "plain" while every example
in the chapter used "content". A field may not share the name of its containing
oneof, so the oneof is renamed to payload_content and the arm becomes 'content'.
The examples were right and the schema now agrees with them, and a stock library
emits the correct key with no post-processing. Three prose references to the
'plain' field are updated.

The non-normative comment claimed an implementation could produce most of the JSON
encoding with library facilities. Measured against the compiled schema, that is
not true: the structural mappings reshape every Metric, so an encoder writes its
own traversal. What the derivation does settle is the scalar rules and the header,
which the comment now says instead.
…er extension fields

The section leaned on the Google Protocol Buffers JSON Mapping for part of its
meaning. That mapping is vendor documentation rather than a versioned standard,
it is specified for proto3 while the Sparkplug schema is proto2, and it does not
cover extensions at all. The section now states its rules itself and says so.

- The lowerCamelCase form a consumer must accept is defined here - remove each
  underscore and upper case the letter after it - so the assertion no longer
  depends on an external document for its meaning.
- New Extension Fields section: an extension is keyed by its fully qualified name
  in square brackets, its value follows the ordinary rules for its type, and an
  unrecognized extension is ignored. The Sparkplug schema reserves an extension
  range in most messages, so leaving this undefined left a field that is
  well-defined on the wire undefined in JSON. Includes a note that unlike the
  binary encoding, where an unknown field is retained and re-emitted, an
  unrecognized JSON key does not survive a round trip.
- Floating point values must be written with enough digits to parse back exactly:
  up to 9 significant decimal digits for a float and 17 for a double.
…e change time

Metrics in a payload are ordered oldest first, so a consuming application that
applies them in array order arrives at the correct current value without
comparing timestamps. This matters when one payload carries several values for
the same metric, as a payload of historical values does. Metrics remain
identified by name or alias, so position still carries no identity.

A metric that omits its timestamp sorts as though its value change time were the
payload timestamp, which is the time it is already taken to have changed.

Metrics that share a value change time may appear in any order and a consumer
must not reorder them. Multi-part file publishing depends on that: it identifies
a metric by position, and every metric of a multi-part publish carries the same
timestamp, so they tie and the publisher's order is carried unchanged by both
encodings. Non-normative comments in both sections record the interaction.

Also splits the JSON PropertySet statement. Uniqueness of property names is
observable in a payload and stays testable; the absence of meaning in key order
is not observable and becomes non-testable, with a note that RFC 8259 defines a
JSON object as unordered so the requirement is on the publisher.
1. The conformance note justifying the session scoping rule was denser than the
   rule it explained. Reduced to the plain consequence: an Edge Node has one
   sequence number stream, so splitting a session across namespaces leaves a
   consumer of either with gaps in it.

2. Removed the requirement that a consuming application accept the lowerCamelCase
   form of a field name. A conformant publisher emits the Protocol Buffers field
   name, so the rule required consumers to accept a form no conformant publisher
   may produce - specifying leniency toward non-conformance and creating a second
   de facto wire form for every field. Replaced with implementer guidance: a
   Protocol Buffers JSON library converts to lowerCamelCase by default and must be
   configured to preserve the original names.

3. Explained why an absent field is omitted rather than written as null or zero.
   Zero is a legitimate value for seq, bd_seq, descriptor_seq, alias, and quality,
   so writing an absent field as 0 would be indistinguishable from one set to
   zero; and null would be a second, conflicting way to express what is_null
   already expresses.

4. Scoped the ignore-unknown rule to the names of fields within a message. It was
   open to being read as covering an unrecognized metric, which is a different
   situation entirely - a mismatch against the metric structure the application
   holds, resolved by the metadata rules and a Rebirth Request rather than by
   ignoring it. Both cases are now stated and contrasted.
A DateTime value is a nanosecond count since epoch, the same quantity the payload
and metric timestamps carry, so it follows them to int64 rather than staying
unsigned. Leaving it as uint64 meant a DateTime value and a payload timestamp
differed in signedness despite meaning the same thing.

This moves a DateTime value from the uint64_value arm of TypedValue to
int64_value, so unlike the timestamp field changes it is a wire change. A DateTime
in a DataSet.Column is unaffected: there is no signed repeated field, so it stays
in long_values alongside UInt64 and is carried as its two's complement
representation, which the DataSet.Column section already requires for signed
values.
…PropertySet TypedValues

Review found two defects in the normative JSON Schema and its prose.

nonNegative64 was "type": ["string","integer"] with "pattern": "^[0-9]+$". JSON
Schema applies pattern only to strings, so a negative JSON number validated
cleanly, as did a string far beyond the 64-bit cap. That governs seq, bd_seq,
descriptor_seq, timestamp, alias, MetaData size and seq, and Status code - the
fields whose range the rest of the specification relies on. Now bounded on both
arms. The earlier negative tests missed this because they used the string form.

The TypedValue flattening rule named only Metric and Template.Parameter as
datatype carriers, but PropertySet.values is a repeated TypedValue, so a property
value had no enclosing field to carry its datatype and could not be encoded by
following the rule. The JSON Schema already resolved this the other way, nesting
an object with its own datatype - which the rule forbade. The rule is now scoped
to Metric and Template.Parameter, and the PropertySet case is stated to match the
schema. Also corrects a reference to PropertyValues, which is not a v4 message.

@ckienle ckienle left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at bcf75ba. Two required interop gaps, both small text fixes, plus some optionals inline.

What I checked and found right: the rsn fix (worked both branches by hand: pds=1/dc=2 gives MAX-1 and matches the Ch.5 example, pds=0/dc=0 gives MAX, nothing goes negative), the sequence bound rescale (no 2^64-1 left anywhere), the payload_content rename across schema/prose/normative text, the scalar mapping rules, the JSON Schema field names against the head proto, and no duplicate tck-ids among the 38 new ones.

Nits / unanchored:

  • Sparkplug_5_Operational_Behavior.adoc:1268 still says "INT64 rollover"; Sparkplug_6_Payloads.adoc:3106-3107 still has "(max value of an INT64)".
  • Sparkplug_6_Payloads.adoc:1849: DateTimeArray elements are now signed after c0781dc, but the text still just says "8-byte value".
  • Sparkplug_6_Payloads.adoc:3316: a "MUST NOT reorder" inside a non-normative comment; metrics-ordered-ties already says it.
  • The PR description is stale: "Known gaps: DateTime is still uint64" was undone by c0781dc, and that move (uint64_value -> int64_value) is a field-number change, harmless only because spCv4.0 is unreleased. The existing approval also predates c0781dc/bcf75ba.
  • Pre-existing, not this PR: the PropertyValue prose (:942) still names the field type and the Metric value list (:774) still lists the v3 arms. Worth a separate cleanup since JSON readers will take prose field names literally.

'Sparkplug B' topic namespace definitions and concepts, see the earlier versions of the Sparkplug
Specification.

Non-normative comment: Topics are written with the spCv4.0 namespace element throughout the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Required: this note contradicts the STATE rule at :1172 ("the STATE topic is never prefixed and always remains at spCv4.0/STATE/sparkplug_host_id").

A JSON-only Edge Node configured to wait for a Primary Host follows this note and watches spJv4.0/STATE/host, while the Host publishes only on spCv4.0/STATE/host. The Edge Node never sees the Host come online and never births. A Host that reads it the other way publishes two STATE streams with two Wills.

Suggest excluding STATE explicitly, e.g. "STATE is not a Sparkplug payload and is published only at spCv4.0/STATE/sparkplug_host_id regardless of the encoding an Edge Node uses.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1fcab1d. You're right, and the failure mode is exactly as you describe — a JSON Edge Node would have waited on a spJv4.0/STATE topic nobody publishes, and never birthed.

Added tck-id-topics-state-single-namespace: a STATE message MUST be published on spCv4.0/STATE/sparkplug_host_id whatever encoding the infrastructure uses, and an Edge Node waiting on a Primary Host MUST subscribe to that topic rather than a spJv4.0 equivalent. A non-normative comment gives the reasons: STATE carries a JSON document rather than a Sparkplug payload, so there is nothing for a second namespace to encode differently, and one topic keeps one Will Message per Host Application.

* [tck-testable tck-id-conformance-host-encoding-both]#[yellow-background]*[tck-id-conformance-host-encoding-both] A
Sparkplug Host Application MUST support both Sparkplug payload encodings, and MUST be able to
consume messages on both the spCv4.0 and spJv4.0 namespaces.*#
* [tck-testable tck-id-conformance-host-encoding-publish]#[yellow-background]*[tck-id-conformance-host-encoding-publish] When

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Required: nothing says which namespace a Host uses to command an Edge Node. This only ties the encoding to the namespace the Host publishes on; the Edge Node subscribes to <ns>/NCMD|DCMD/... and REBIRTH on its own namespace.

A Host that supports both encodings (as it now must) and defaults to spCv4.0 sends a rebirth request or NCMD to an Edge Node that births on spJv4.0, and it's never delivered. Rebirth recovery silently fails for every JSON Edge Node.

Suggest: a Host MUST publish NCMD/DCMD/rebirth to an Edge Node on the namespace of that Edge Node's current NBIRTH, and an Edge Node MUST subscribe to its command topics on the namespace it publishes on.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1fcab1d. Agreed this was a real gap — the rule only covered which encoding a Host uses, not which namespace it addresses.

Two statements, one for each direction, close to your suggested wording:

  • conformance-host-encoding-address: when a Host Application publishes a message addressed to a particular Edge Node or Device, including an NCMD, a DCMD, and a Rebirth Request, it MUST publish on the namespace of that Edge Node's current NBIRTH.
  • conformance-edge-node-encoding-subscribe: an Edge Node MUST subscribe to its command topics on the same namespace it publishes on.

The non-normative comment records why it matters: the MQTT Server accepts the misaddressed message and it is simply never delivered, so a Rebirth Request fails silently and the Host cannot recover its view of that Edge Node.


* [tck-testable tck-id-conformance-mqtt-aware-basic]#[yellow-background]*[tck-id-conformance-mqtt-aware-basic] A
Sparkplug Aware MQTT Server MUST support all aspects of a Sparkplug Compliant MQTT Server*#
* [tck-testable tck-id-conformance-mqtt-aware-encoding-both]#[yellow-background]*[tck-id-conformance-mqtt-aware-encoding-both] A

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional (but it affects every broker vendor): the premise here doesn't hold. The only payload-touching behaviour required of an Aware server below is mqtt-aware-ndeath-timestamp, which is a MAY, and this section also says the Aware server no longer stores or republishes BIRTHs.

As written, every Aware server has to implement JSON decode/encode even if it never rewrites a timestamp. Suggest making it conditional: "If a Sparkplug Aware MQTT Server replaces NDEATH timestamps, it MUST do so for both encodings." and fixing the comment.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1fcab1d. You're right that the premise didn't hold — I had asserted the Aware server decodes payloads, when the only payload-touching behaviour required of it is the NDEATH timestamp replacement, which is itself a MAY.

Now conditional: a Sparkplug Aware MQTT Server that replaces NDEATH timestamps MUST do so for payloads in both encodings. A server that does not never inspects a payload and needs neither encoding. The comment is rewritten to say that, and to note the requirement exists so that offering the timestamp behaviour for one encoding only cannot leave Edge Nodes on the other namespace silently unserved.

},
"not": { "required": [ "content", "compressed" ] },
"$defs": {
"nonNegative64": {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: nonNegative64 is applied to alias, MetaData.size and Status.code, which are uncapped uint64 (:682, :1601), so it rejects valid values >= 2^63. It's also inconsistent with itself: the number branch caps at 2^63-1 but the string pattern ^(0|[1-9][0-9]{0,18})$ accepts up to 9999999999999999999 and rejects "18446744073709551615".

So an spCv4.0 payload with a large alias or status code can't be converted to a schema-valid spJv4.0 payload, which contradicts "converted from one to the other without loss" and json-schema-valid.

Suggest two defs: a uint64 one (0..2^64-1 or a <=20-digit string) for alias/size/MetaData.seq/code, and a bounded one for the sequence numbers and timestamps whose string pattern actually enforces 2^63-1 (or say in its comment that the string branch isn't bounded).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1fcab1d, and thank you — this was a bug I introduced while fixing a different one. I had conflated "the sequence numbers are capped at 2^63-1" with "every 64-bit field is", which is wrong for alias, MetaData.size and Status.code.

Split into two definitions as you suggested:

  • uint64 — 0 to 18446744073709551615, used for alias, MetaData.size, MetaData.seq and Status.code
  • sequence64 — 0 to 9223372036854775807, used for seq, bd_seq, descriptor_seq and timestamp

On the string branch: an exact "<= 9223372036854775807" pattern needs about fifteen alternatives and is unreadable in a specification, so I took your second option. It is bounded by length and the comment says so explicitly: "The string form is bounded by length only; the exact upper bound is normative prose, not expressible as a readable pattern."

Verified: alias and Status.code at uint64 max are accepted again, seq at 20 digits and a negative number are rejected.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — all three leftovers fixed in 40728fa.

MetaData.seq — you were right, and my reply was wrong. It still pointed at sequence64. The bulk rename that split the definitions matched on the field name, which Header.seq and MetaData.seq share, so the multi-part file index got capped at 2^63-1 by accident. It is an uncapped uint64 in both the schema and the prose, and now points at uint64. Same class of mistake as the regex that produced the isValidSeq recursion, which is a pattern I should watch for.

Verified: MetaData.seq at uint64 max is accepted again, while a Header.seq at that value is still rejected.

uint64 comment — now carries the same note sequence64 has, that the string form is bounded by length only and the exact upper bound is normative prose.

DateTimeArray — re-wrapped. It was worse than you flagged: my earlier edit had split it across three lines, breaking after both "signed" and "8-byte".

the datatype and selects one of several value fields. In JSON the datatype is already carried by
the enclosing object and the value field is not needed to disambiguate.

[tck-testable tck-id-payloads-json-typedvalue-flat]#[yellow-background]*[tck-id-payloads-json-typedvalue-flat] A

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: flattening drops the TypedValue's own datatype/is_null. Metric has both (fields 4/10) and so does its value TypedValue (fields 1/2), but JSON keeps one of each, and nothing says the pairs must agree in spCv4.0 or which one a JSON-to-proto converter sets. So the conversion isn't lossless as claimed.

Related: typedvalue-parameter-keys lists only name/type/value and the schema's parameter has no is_null, but json-null requires is_null: true to express null, so a null Template.Parameter has no JSON form.

Suggest requiring the two to agree in spCv4.0, saying the JSON datatype/is_null carry both, and adding is_null to the Parameter keys and schema.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1fcab1d. Both halves were real.

On the duplicated datatype/is_null: added payloads-typedvalue-datatype-agree (where a Metric carries both its own and its TypedValue's, the two MUST be equal, and likewise for is_null) and payloads-json-typedvalue-carries-both (the single JSON key represents both, so a JSON-to-protobuf converter sets both fields from it). That makes the conversion lossless as claimed rather than leaving it to the implementer.

On the null Template.Parameter: is_null is added to typedvalue-parameter-keys and to the schema's parameter definition, and payloads-json-parameter-is-null states that a null parameter carries is_null true and omits value, as any other null does.

** _metrics_
*** This is an array of metrics representing key/value/datatype values. Metrics are further defined
link:#payloads_c_metric[here].
*** [tck-testable tck-id-payloads-metrics-ordered]#[yellow-background]*[tck-id-payloads-metrics-ordered] The

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: this is a broad new MUST for every Edge Node. It applies to every payload, including NBIRTH/DBIRTH, which existing stacks (Tahu, Ignition Transmission) emit in browse order, not by change time. The motivating case is several values of the same metric in one payload.

It's also unclear whether "the metrics of a payload" includes Template member metrics and Command arguments/results.

Suggest scoping the MUST to payloads carrying more than one value for the same metric (or DATA messages) with a SHOULD elsewhere, and saying explicitly whether nested metrics are covered. If the broad version is a deliberate working-group decision, at least call it out as a migration item.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1fcab1d, taking your suggestion.

payloads-metrics-ordered is now scoped: a payload that carries more than one value for the same metric MUST order by value change time. payloads-metrics-ordered-should makes it a SHOULD everywhere else, so an Edge Node emitting a BIRTH in browse order — which is what Tahu and Ignition Transmission do — stays conformant.

On nested metrics, payloads-metrics-ordered-scope now says it outright: the ordering applies to the metrics array of the payload's content, and does not reach the metrics of a Template or the arguments or results of a Command, whose order is defined by the Template Definition or the Command definition.

The rationale note is reworded to match, so it explains why the requirement is narrow rather than implying the broad version.

…essing, and schema bounds

Required:

- The namespace substitution note made STATE ambiguous. STATE is pinned to
  spCv4.0/STATE/sparkplug_host_id, so a JSON Edge Node following the note would
  have waited on spJv4.0/STATE while the Host published on spCv4.0/STATE and never
  seen it come online. STATE is now stated as the one exception, with the reason.

- Nothing said which namespace a Host Application uses to address a particular
  Edge Node. A Host supports both encodings, so it could send an NCMD or a Rebirth
  Request on a namespace the Edge Node does not subscribe to; the MQTT Server
  accepts it and it is never delivered, so Rebirth recovery fails silently. A Host
  now publishes to an Edge Node on the namespace of that Edge Node's NBIRTH, and
  an Edge Node subscribes to its command topics on the namespace it publishes on.

Also:

- nonNegative64 capped every 64-bit field at 2^63-1, but alias, MetaData.size and
  Status.code are uncapped uint64, so a legal spCv4.0 payload could not be
  expressed as a schema-valid spJv4.0 one - contradicting the lossless conversion
  this section claims. Split into uint64 and sequence64, with the string branch of
  sequence64 bounded by length and its limit stated.
- The Sparkplug Aware MQTT Server encoding requirement is now conditional on
  replacing NDEATH timestamps, which is the only payload-touching behavior
  required of it and is itself a MAY.
- Flattening drops the TypedValue's own datatype and is_null. The two copies must
  now agree in the Protocol Buffers encoding, the single JSON key represents both,
  and a null Template.Parameter has a JSON form: is_null is added to the parameter
  keys and to the schema.
- The metric ordering MUST is scoped to payloads carrying more than one value for
  the same metric, with a SHOULD elsewhere so a BIRTH in browse order stays
  conformant, and it now says it does not reach Template or Command metrics.
- Nits: INT64 rollover wording, a stale (max value of an INT64), DateTimeArray
  elements described as signed, and a MUST NOT inside a non-normative comment.
@wes-johnson

Copy link
Copy Markdown
Contributor Author

All six inline items addressed in 1fcab1d; replies on each thread. The unanchored nits from your review body:

  • Sparkplug_5_Operational_Behavior.adoc:1268 — "INT64 rollover" now reads "a sequence number rollover".
  • Sparkplug_6_Payloads.adoc:3106-3107 — the stale "(max value of an INT64)" parenthetical is gone.
  • :1849 — DateTimeArray elements are now described as "a signed 8-byte value", matching c0781dc.
  • :3316 — the "MUST NOT reorder" is removed from the non-normative comment; metrics-ordered-ties already carries it.

PR description updated. You were right that it was stale in both directions: the "DateTime is still uint64" gap was undone by c0781dc, and that move is a wire change — uint64_value to int64_value is a different field number, harmless only because spCv4.0 has not shipped. The description now says so rather than listing it as a known gap.

Also noted: the existing approval predates c0781dc and bcf75ba, and now this change too.

On the two pre-existing items you flagged as out of scope — the PropertyValue prose at :942 still naming the field type, and the Metric value list at :774 still listing the v3 arms — I agree they want a separate cleanup, and agree on the reason: JSON readers will take prose field names literally now that there is a JSON encoding. Not touched here so this PR stays reviewable.

Thanks for working the rsn arithmetic through by hand — that and the sequence rescale were the parts I was least able to check mechanically.

@ckienle ckienle left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 1fcab1d. Everything from my review is fixed as described: the STATE single-namespace rule, the two command-addressing rules, the conditional Aware-server requirement, the uint64/sequence64 split, the TypedValue agreement rules and the null Template.Parameter, the scoped metric-ordering MUST, and all the body nits. No duplicate tck-ids. Approving.

Three small leftovers, none blocking:

  • Your reply says MetaData.seq moved to uint64, but the schema still points it at sequence64 (Sparkplug_6_Payloads.adoc:3976). It's an uncapped uint64 in the proto and the prose, so I think uint64 is what you meant.
  • The uint64 string pattern ^(0|[1-9][0-9]{0,19})$ accepts 20-digit values above 18446744073709551615. Fine to leave length-bounded, but its comment should say so, as the sequence64 one does.
  • The DateTimeArray line at :1858 wraps after "signed" and again after "8-byte"; worth re-wrapping.

…gth-bounded string form

MetaData.seq sequences the parts of a multi-part file transfer and is an uncapped
uint64 in both the schema and the prose. The earlier split pointed it at
sequence64 because the bulk rename matched on the field name, which the two
fields share, so it was capped at 2^63-1 by accident - the same class of mistake
as the regex that produced the recursion.

Also gives the uint64 definition the same note sequence64 carries, since its
string branch is bounded by length rather than by value, and re-wraps the
DateTimeArray entry left split mid-phrase by the earlier edit.
@wes-johnson
wes-johnson merged commit 825527a into eclipse-sparkplug:develop Oct 29, 2026
3 checks passed
wes-johnson added a commit to wes-johnson/Sparkplug that referenced this pull request Nov 3, 2026
Per the spec team decision. All 16 extension ranges are removed, along with the
TypedValueExtension message, which existed only to host one, and the
extension_value arm of the TypedValue oneof.

They had never been used. There is no extend statement anywhere in the
specification, Tahu 3.x, Tahu develop or the TCK, and no hand-written call to the
extension API in any of them, after three major versions of being reserved.

They were not free. Six changes in v4 development moved a floor - typed Commands,
the metadata separation, the topic namespace change, the header/content split,
TemplateList, and the ValueArray in eclipse-sparkplug#649 - each a theoretical break for any third
party occupying the numbers being vacated, for no actual third party.

They also did not work in the JSON encoding. The Protocol Buffers JSON Mapping is
specified for proto3, which has no extensions, so eclipse-sparkplug#645 had to invent three
statements defining the bracketed key form, and an extension still survived a
binary round trip but not a JSON one. Those three statements are removed with the
mechanism, and the rationales that rested on extension ranges are reworded to
rest on the ordinary unknown-field rule instead.

array_value takes field number 16, which the extension arm freed, rather than
leaving a gap at 16 with the array at 17.

Not a wire change for anything that exists: a payload exercising metrics, a
DataSet and a null cell serializes byte-identically under both schemas, and each
decodes the other's output.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants