Repository navigation
[#231] Move the schema to Protocol Buffers edition 2023 - #646
Conversation
Adds a normative JSON encoding of the Sparkplug payload model, published on the spJv4.0 namespace alongside the Protocol Buffers encoding on spCv4.0. The mapping is derived from the Google Protocol Buffers JSON Mapping so that most of it can be produced by library facilities, with Sparkplug's deviations stated explicitly: TypedValue is flattened to a bare 'value', PropertySet is a JSON object keyed by property name, array DataTypes are JSON arrays rather than base64 packed bytes, and a DataSet.Column carries one 'values' array. 64-bit integers are encoded as JSON strings. A v4 nanosecond timestamp is roughly 165 times larger than the 2^53 limit a JSON number is guaranteed to carry, so a numeric form is silently rounded by any parser that decodes into a double. Also generalizes three Ch. 5 statements that required a Protobuf encoded payload, notes that topics written with spCv4.0 apply equally to spJv4.0, and renames the 'dataType' key in the Ch. 6 examples to 'datatype' and the metric-level 'template' key to 'value' so the examples match the encoding rules.
…ing-encoded 64-bit values The note claimed a Sparkplug sequence number is always small, which is wrong: seq, bd_seq, and descriptor_seq each count to the maximum value of their declared type before rolling over to zero. They are therefore subject to the same precision loss as timestamps, and a numeric form would be correct early in a session and lossy late in one.
The three message sequence numbers move from uint64 to int64 in the Protocol Buffers schema, with the range and rollover point changing from 18446744073709551615 to 9223372036854775807 across all 38 statements that bound them in Chapters 4, 5, and 6. Protobuf uint64 is a signed long in Java, so the upper half of the uint64 range was never reachable by an implementation built on it. Declaring int64 makes the specified range the representable one. MetaData.seq, which sequences the parts of a multi-part file transfer, is a different field and is unchanged.
… rsn arithmetic Header.timestamp and Metric.timestamp move from uint64 to int64, for the same reason as the sequence numbers: protobuf uint64 is a signed long in Java, so the upper half of the unsigned range was never reachable. The range note in Chapter 5 is rewritten accordingly - the ceiling is 2262-04-11, stated directly rather than as a caveat on an unsigned range that no implementation could use. The NREBIRTH rsn calculation said to add the maximum sequence number to a negative result. That was off by one, since a counter cycling 0..MAX has a period of MAX+1, and MAX+1 is not representable as an int64. Restated as two cases so no intermediate is ever negative or out of range: if pds >= 1 + dc: rsn = pds - (1 + dc) otherwise: rsn = 9223372036854775807 - (dc - pds) Verified equivalent to (pds - (1 + dc)) mod 2^63 over the full range.
…ing seq as unsigned The seq, bd_seq, and descriptor_seq bullets in the Header section still described each field as an unsigned 64-bit integer after the schema moved them to int64.
…ormance profiles An Edge Node MUST support at least one encoding and MAY support both, selected by configuration, but MUST use a single encoding and namespace for the whole of an MQTT session - seq is Edge-Node-wide and bd_seq scopes the session, so splitting one session across both namespaces would produce two unusable halves of one sequence stream. Host Applications and Sparkplug Aware MQTT Servers MUST support both encodings. A Host Application cannot choose the encoding of the Edge Nodes it consumes, and an Aware Server decodes payloads rather than treating them as opaque. A Sparkplug Compliant MQTT Server has no payload requirements at all, which is now stated non-normatively so the asymmetry with the Aware profile is not read as an omission. Also generalizes the Edge Node profile's reference to the spCv4.0/# namespace.
…g the capped range seq, bd_seq, and descriptor_seq go back to uint64 while keeping the rollover point at 9223372036854775807. timestamp stays int64. An unsigned type is right for a counter that can never be negative, and it keeps the three consistent with alias, MetaData.size, and Status.code. Capping the range below the type's maximum still guarantees every value a conformant Edge Node can produce fits a signed 64-bit integer, so an implementation in a language without an unsigned 64-bit type never needs unsigned comparison. Neither bound is reachable: at a million messages per second the capped range lasts ~292,000 years. timestamp keeps int64 because every mainstream time representation it interoperates with is signed - protobuf's own Timestamp, OPC UA DateTime, java.time.Instant - and the 2262 ceiling it accepts is one none of those exceed either. The repeated "(max value of a UINT64)" parenthetical is dropped rather than rescaled, since the cap is now a deliberate restriction rather than the type's maximum. It is explained once in the Header section instead.
…ary-support claim The oneof was named 'content' and its uncompressed arm was named 'plain', so a Protocol Buffers JSON implementation emitted the key "plain" while every example in the chapter used "content". A field may not share the name of its containing oneof, so the oneof is renamed to payload_content and the arm becomes 'content'. The examples were right and the schema now agrees with them, and a stock library emits the correct key with no post-processing. Three prose references to the 'plain' field are updated. The non-normative comment claimed an implementation could produce most of the JSON encoding with library facilities. Measured against the compiled schema, that is not true: the structural mappings reshape every Metric, so an encoder writes its own traversal. What the derivation does settle is the scalar rules and the header, which the comment now says instead.
…er extension fields The section leaned on the Google Protocol Buffers JSON Mapping for part of its meaning. That mapping is vendor documentation rather than a versioned standard, it is specified for proto3 while the Sparkplug schema is proto2, and it does not cover extensions at all. The section now states its rules itself and says so. - The lowerCamelCase form a consumer must accept is defined here - remove each underscore and upper case the letter after it - so the assertion no longer depends on an external document for its meaning. - New Extension Fields section: an extension is keyed by its fully qualified name in square brackets, its value follows the ordinary rules for its type, and an unrecognized extension is ignored. The Sparkplug schema reserves an extension range in most messages, so leaving this undefined left a field that is well-defined on the wire undefined in JSON. Includes a note that unlike the binary encoding, where an unknown field is retained and re-emitted, an unrecognized JSON key does not survive a round trip. - Floating point values must be written with enough digits to parse back exactly: up to 9 significant decimal digits for a float and 17 for a double.
…e change time Metrics in a payload are ordered oldest first, so a consuming application that applies them in array order arrives at the correct current value without comparing timestamps. This matters when one payload carries several values for the same metric, as a payload of historical values does. Metrics remain identified by name or alias, so position still carries no identity. A metric that omits its timestamp sorts as though its value change time were the payload timestamp, which is the time it is already taken to have changed. Metrics that share a value change time may appear in any order and a consumer must not reorder them. Multi-part file publishing depends on that: it identifies a metric by position, and every metric of a multi-part publish carries the same timestamp, so they tie and the publisher's order is carried unchanged by both encodings. Non-normative comments in both sections record the interaction. Also splits the JSON PropertySet statement. Uniqueness of property names is observable in a payload and stays testable; the absence of meaning in key order is not observable and becomes non-testable, with a note that RFC 8259 defines a JSON object as unordered so the requirement is on the publisher.
1. The conformance note justifying the session scoping rule was denser than the rule it explained. Reduced to the plain consequence: an Edge Node has one sequence number stream, so splitting a session across namespaces leaves a consumer of either with gaps in it. 2. Removed the requirement that a consuming application accept the lowerCamelCase form of a field name. A conformant publisher emits the Protocol Buffers field name, so the rule required consumers to accept a form no conformant publisher may produce - specifying leniency toward non-conformance and creating a second de facto wire form for every field. Replaced with implementer guidance: a Protocol Buffers JSON library converts to lowerCamelCase by default and must be configured to preserve the original names. 3. Explained why an absent field is omitted rather than written as null or zero. Zero is a legitimate value for seq, bd_seq, descriptor_seq, alias, and quality, so writing an absent field as 0 would be indistinguishable from one set to zero; and null would be a second, conflicting way to express what is_null already expresses. 4. Scoped the ignore-unknown rule to the names of fields within a message. It was open to being read as covering an unrecognized metric, which is a different situation entirely - a mismatch against the metric structure the application holds, resolved by the metadata rules and a Rebirth Request rather than by ignoring it. Both cases are now stated and contrasted.
A DateTime value is a nanosecond count since epoch, the same quantity the payload and metric timestamps carry, so it follows them to int64 rather than staying unsigned. Leaving it as uint64 meant a DateTime value and a payload timestamp differed in signedness despite meaning the same thing. This moves a DateTime value from the uint64_value arm of TypedValue to int64_value, so unlike the timestamp field changes it is a wire change. A DateTime in a DataSet.Column is unaffected: there is no signed repeated field, so it stays in long_values alongside UInt64 and is carried as its two's complement representation, which the DataSet.Column section already requires for signed values.
922d96b to
d7834f0
Compare
…PropertySet TypedValues Review found two defects in the normative JSON Schema and its prose. nonNegative64 was "type": ["string","integer"] with "pattern": "^[0-9]+$". JSON Schema applies pattern only to strings, so a negative JSON number validated cleanly, as did a string far beyond the 64-bit cap. That governs seq, bd_seq, descriptor_seq, timestamp, alias, MetaData size and seq, and Status code - the fields whose range the rest of the specification relies on. Now bounded on both arms. The earlier negative tests missed this because they used the string form. The TypedValue flattening rule named only Metric and Template.Parameter as datatype carriers, but PropertySet.values is a repeated TypedValue, so a property value had no enclosing field to carry its datatype and could not be encoded by following the rule. The JSON Schema already resolved this the other way, nesting an object with its own datatype - which the rule forbade. The rule is now scoped to Metric and Template.Parameter, and the PropertySet case is stated to match the schema. Also corrects a reference to PropertyValues, which is not a v4 message.
d7834f0 to
b6edfc3
Compare
ckienle
left a comment
There was a problem hiding this comment.
Reviewed the change at b6edfc3 (the diff over #645). The conversion looks correct apart from one behaviour change inline. Caveat up front: I couldn't run protoc 27+ here, so this is a static comparison against the proto2 -> edition 2023 feature defaults, not a descriptor-set diff.
Checked clean: field numbers, types, oneofs and extension ranges are unchanged; presence defaults to EXPLICIT, matching proto2 optional; all five packable repeated scalars were already [packed=true], so the PACKED default doesn't change the wire format (and dropping [packed=false] on string_values is right); both enums are marked CLOSED; the duplicated DataType listing at :1619 matches the schema.
Questions:
- Edition 2023 also changes
json_formatfrom LEGACY_BEST_EFFORT to ALLOW, so JSON-name conflicts become protoc errors. The description says the schema compiles clean; which protoc version did you use? - Does a current nanopb generate from an edition-2023 file? Tahu C uses nanopb, and if it can't, C implementers will have to hand-convert back to proto2.
Nits:
:3579"Every field ... has explicit presence": repeated fields have no presence; "Every singular field".:3581is 109 characters; the reworded paragraph wasn't re-wrapped to the chapter's ~100.- Dropping
optionalleft the field names and//comments out of column alignment with therepeatedlines in the same messages (e.g.:235).
| // | ||
|
|
||
| syntax = "proto2"; | ||
| edition = "2023"; |
There was a problem hiding this comment.
Should fix (or document): this silently changes UTF-8 handling. proto2 files resolve to features.utf8_validation = NONE; edition 2023 defaults to VERIFY, and the behaviour-preserving conversion adds option features.utf8_validation = NONE; at file level.
Without it, decoders generated from this schema in C++, Python (upb), Go and other upb-based runtimes reject the whole payload if any string field holds invalid UTF-8, where today the same bytes decode. Java is unaffected (it follows its own utf8 option). The "not a wire change" equivalence test can't show this because its sample payload has only valid strings.
The spec already says payload strings are UTF-8, so VERIFY only rejects payloads that were already non-conformant, but it turns one bad field into a dropped message, so it should be a deliberate choice. Either add the option, or keep VERIFY and say so (e.g. "a decoder MAY reject a payload whose string fields are not valid UTF-8").
There was a problem hiding this comment.
Fixed in 97e5f89, but resolved the opposite way to my first attempt — worth reading, since the outcome reverses what I initially committed.
You were right that this silently changed UTF-8 handling. I first set utf8_validation = NONE to preserve the proto2 behaviour. On review that reasoning was wrong: v4 has not shipped, so there is no byte compatibility to preserve, and the specification already requires string fields to be UTF-8. The schema now keeps VERIFY, which enforces what is already normative rather than accepting payloads the specification calls non-conformant.
Two statements were added to the specification text so the rule belongs to Sparkplug rather than to a Protocol Buffers default that changed between proto2 and edition 2023 — payloads-string-utf8 and payloads-string-utf8-reject. The JSON encoding inherits the same rule from RFC 8259 instead, so stating it once in the payload section covers both.
Rejecting the whole payload rather than the offending field is deliberate and now explained: a payload is the unit a sequence number orders, so accepting part of one would leave a consuming application unable to say whether it had applied that message.
On your "Java follows its own utf8 option" note — I checked. features.(pb.java).utf8_validation defaults to respecting the file level setting, so one option gives every language the same behaviour. I tried setting the Java feature explicitly as well and reverted it: it requires importing java_features.proto, which makes generated code in every other language depend on that file, and the Python runtime does not ship it.
Verified: a payload with an invalid UTF-8 string is rejected, a valid one round-trips.
There was a problem hiding this comment.
Good catch — added in 2ba1c64.
The rebirth loop is real and worth saying out loud, because the obvious reaction to a rejected BIRTH is to ask for another one. The non-normative comment now says that republishing cannot make a non-conformant payload valid, so a Host Application that rejects an NBIRTH or DBIRTH on this basis gains nothing from a Rebirth Request and will receive the same payload back. It should report the condition and treat the Edge Node as unusable until it is corrected, rather than requesting Rebirths indefinitely.
Agreed with your framing that this is arguably just what a non-conformant Edge Node gets — the value is in telling the implementer what to expect rather than leaving them to discover it from a loop in production.
| // To compile: | ||
| // cd client_libraries/java | ||
| // protoc --proto_path=../../ --java_out=src/main/java ../../sparkplug_c.proto | ||
| // Requires protoc 27 or later for edition 2023 support. |
There was a problem hiding this comment.
Optional: worth stating the runtime floor too, not just protoc. Code generated by protoc 27+ for an edition-2023 file needs a matching runtime (protobuf-java >= 4.27, Python protobuf >= 5.27, etc.). Tahu Java is on protobuf-java 3.25.5, so implementers regenerating from the spec need a runtime upgrade as well.
Suggest: "Requires protoc 27 or later and a Protocol Buffers runtime that supports edition 2023".
There was a problem hiding this comment.
Fixed in 66d8e5f, using your wording. The compile comment now reads:
// Requires protoc 27 or later and a Protocol Buffers runtime that supports edition 2023
// (protobuf-java 4.27 or later, Python protobuf 5.27 or later, and equivalents).
Worth noting alongside it: #647 moves the TCK from protobuf-java 3.17.3 to 4.36.2, so the TCK itself clears that floor. Tahu Java on 3.25.5 will need the same bump when it regenerates from this schema, which is exactly the point you were making.
…essing, and schema bounds Required: - The namespace substitution note made STATE ambiguous. STATE is pinned to spCv4.0/STATE/sparkplug_host_id, so a JSON Edge Node following the note would have waited on spJv4.0/STATE while the Host published on spCv4.0/STATE and never seen it come online. STATE is now stated as the one exception, with the reason. - Nothing said which namespace a Host Application uses to address a particular Edge Node. A Host supports both encodings, so it could send an NCMD or a Rebirth Request on a namespace the Edge Node does not subscribe to; the MQTT Server accepts it and it is never delivered, so Rebirth recovery fails silently. A Host now publishes to an Edge Node on the namespace of that Edge Node's NBIRTH, and an Edge Node subscribes to its command topics on the namespace it publishes on. Also: - nonNegative64 capped every 64-bit field at 2^63-1, but alias, MetaData.size and Status.code are uncapped uint64, so a legal spCv4.0 payload could not be expressed as a schema-valid spJv4.0 one - contradicting the lossless conversion this section claims. Split into uint64 and sequence64, with the string branch of sequence64 bounded by length and its limit stated. - The Sparkplug Aware MQTT Server encoding requirement is now conditional on replacing NDEATH timestamps, which is the only payload-touching behavior required of it and is itself a MAY. - Flattening drops the TypedValue's own datatype and is_null. The two copies must now agree in the Protocol Buffers encoding, the single JSON key represents both, and a null Template.Parameter has a JSON form: is_null is added to the parameter keys and to the schema. - The metric ordering MUST is scoped to payloads carrying more than one value for the same metric, with a SHOULD elsewhere so a BIRTH in browse order stays conformant, and it now says it does not reach Template or Command metrics. - Nits: INT64 rollover wording, a stale (max value of an INT64), DateTimeArray elements described as signed, and a MUST NOT inside a non-normative comment.
b6edfc3 to
66d8e5f
Compare
|
Both questions answered, and the UTF-8 one is now resolved the other way. protoc version: 4.36.2, fetched by Gradle via the protobuf plugin in #647, so no local protoc nanopb: yes. I said earlier I couldn't test this; that was wrong, and I have now run it. The output carries the v4 constructs ( utf8_validation is now VERIFY, not NONE. Your point stood but the resolution went further: v4 A payload with an invalid UTF-8 string is rejected whole rather than per field, because a payload is One thing I checked because of your "Java follows its own utf8 option" note: Java's Also addressed: the runtime floor is now stated alongside the protoc one, "every singular field", |
ckienle
left a comment
There was a problem hiding this comment.
Re-reviewed at 97e5f89. Keeping VERIFY and making it normative (payloads-string-utf8 / -reject) is a better resolution than the NONE I suggested: v4 hasn't shipped, the spec already required UTF-8, and stating it in the spec rather than leaving it to a protobuf feature default is right. Thanks for confirming protoc 4.36.2 and nanopb 0.4.9.1, and for the runtime floor in the compile comment. Approving.
One optional thought: with the reject rule, a Host that rejects an NBIRTH (or DBIRTH) because of an invalid UTF-8 string will request a Rebirth and get the same BIRTH back, indefinitely. That's arguably just what a non-conformant Edge Node gets, but a sentence in the non-normative comment saying what a consumer should expect (or that it shouldn't keep requesting rebirths for a payload it rejected) might save an implementer some confusion.
…gth-bounded string form MetaData.seq sequences the parts of a multi-part file transfer and is an uncapped uint64 in both the schema and the prose. The earlier split pointed it at sequence64 because the bulk rename matched on the field name, which the two fields share, so it was capped at 2^63-1 by accident - the same class of mistake as the regex that produced the recursion. Also gives the uint64 definition the same note sequence64 carries, since its string branch is bounded by length rather than by value, and re-wraps the DateTimeArray entry left split mid-phrase by the earlier edit.
proto2 and proto3 are both frozen syntax modes; editions superseded them in protoc 27 and is where protobuf development happens. Moving to proto3 was not an option - it bans extension ranges, and its replacement, unknown field preservation, does not work in JSON, which would make the extension story worse in the encoding that most needs a good one. Edition 2024 was rejected: it enables enforce_naming_style, which produces 36 errors against this schema because every DataType value is PascalCase. Those names are the JSON wire format under this specification, so complying would rename Int32 to INT32 and UInt32 to U_INT32 in every payload and example. The alternative, features.enforce_naming_style = STYLE_LEGACY, means adopting 2024 and opting straight back out of it. The conversion is mechanical: - syntax = "proto2" becomes edition = "2023" - the optional label is dropped from 48 fields; EXPLICIT presence is the editions default - 6 [packed=...] options are dropped; packed is a feature, and the default for repeated scalars - DataType and MetricOperation are marked features.enum_type = CLOSED to keep proto2 enum semantics, so neither needs a zero value This is not a wire change. A payload exercising templates, datasets, packed arrays, and property sets serializes to byte-identical output under both schemas, and each decodes the other's bytes. The TCK's protobuf runtime pin is left alone. Edition 2023 needs protobuf-java 4.27+, but the checked-in SparkplugBProto.java was generated by protoc 3.17 and does not compile against a 4.x runtime, so the bump is inseparable from the regeneration tracked in eclipse-sparkplug#641.
…iour under edition 2023 proto2 files resolve to features.utf8_validation = NONE; edition 2023 defaults to VERIFY. Without pinning it, decoders generated from this schema on upb-based runtimes - C++, Python, Go - would reject a whole payload when any string field holds invalid UTF-8, where the Sparkplug 3 schema decoded it. The byte-identical equivalence test could not surface this because its sample carried only valid strings. NONE is set explicitly so the conversion stays behaviour preserving, and the reason is recorded in the schema. Also states the runtime floor alongside the protoc one, since generated code for an edition 2023 file needs protobuf-java 4.27 or later and equivalents. Nits: repeated fields have no presence, so "every singular field"; the reworded paragraph is re-wrapped to the chapter width; and the schema is realigned, since dropping the optional label left singular fields out of column with the repeated ones. The realignment changes whitespace only, asserted by comparing the token streams before and after.
…to2 behaviour Reverses the earlier choice of utf8_validation = NONE. That was made to preserve Sparkplug 3 decoding behaviour, which is not a goal: v4 has not shipped, so there is no byte compatibility to keep, and the specification already requires string fields to be UTF-8. VERIFY enforces what is already normative instead of accepting payloads the specification calls non-conformant. The Java runtime's own utf8_validation feature defaults to respecting the file level setting, so one option gives every language the same behaviour. Setting the Java feature explicitly was tried and reverted: it requires importing java_features.proto, which makes generated code in every other language depend on that file, and the Python runtime does not ship it. Adds the requirement to the specification text rather than leaving it implicit in the encoding, since the Protocol Buffers default changed between proto2 and edition 2023 and the JSON encoding inherits the rule from RFC 8259 instead. A non-conformant payload is rejected whole rather than per field, because a payload is the unit a sequence number orders. Verified: a payload with an invalid UTF-8 string is rejected and a valid one round trips, and nanopb still generates from the schema.
…it rejects With the UTF-8 reject rule, a Host Application that rejects an NBIRTH or DBIRTH would otherwise issue a Rebirth Request and receive the same non-conformant payload back, indefinitely. The non-normative comment now says that republishing cannot make a payload valid, so the Host should report the condition and treat the Edge Node as unusable rather than requesting Rebirths until it is corrected.
97e5f89 to
2ba1c64
Compare
Moves the Protocol Buffers schema from
syntax = "proto2"toedition = "2023".Stacked on #645. The schema lines it touches are the ones #645 changes, so it is based on that
branch and should merge after it. The diff below is only this change.
Why editions, and why not proto3
proto2 and proto3 are both frozen syntax modes. Editions superseded them in protoc 27 and are where
protobuf development happens.
proto3 was not an option: it bans extension ranges outright, and its replacement — unknown field
preservation — does not work in JSON, where an unrecognized key is dropped on output and rejected on
input. That would make the extension story worse in the encoding that most needs a good one.
Edition 2024 was rejected on evidence. It enables
enforce_naming_style, which produces 36 errorsagainst this schema because every
DataTypevalue is PascalCase. Those names are the JSON wireformat under #645, so complying would rename
Int32toINT32andUInt32toU_INT32in everypayload, example and spec sentence.
STYLE_LEGACYavoids that but means adopting 2024 and optingstraight back out of its headline change.
Edition 2024 also changes the generated Java layout:
nest_in_file_classdefaults to flat, soSparkplugBProto.Payloadbecomes a top-levelPayload, breaking every Java implementation'simports. Restoring the nested layout is not possible at file scope —
Option pb.JavaFeatures.nest_in_file_class cannot be set on an entity of type file— so it wouldrequire a Java-specific option annotated on every top-level type, inside a schema that is
language-neutral and shared by C, C++, Python and Go implementations.
The conversion
syntax = "proto2"becomesedition = "2023"optionallabel is dropped from 48 fields; EXPLICIT presence is the editions default[packed=...]options are dropped; packed is a feature now, and the default for repeated scalarsDataTypeandMetricOperationare markedfeatures.enum_type = CLOSED, keeping proto2 enumsemantics so neither needs a zero value
The two
DataTypelistings in the chapter are kept textually identical, as before.Not a wire change
A payload exercising templates, datasets, packed arrays and property sets serializes to
byte-identical output under both schemas, and each decodes the other's bytes:
On edition 2023 aging
Low risk. Today's protoc still compiles proto2, a syntax from 2001, so nothing has been dropped in
twenty years. Editions are designed so a file can stay on one indefinitely and migrate per-feature on
its own schedule. Moving to a later edition later is a one-line change plus a runtime pin.
Verification
Build passes; the schema compiles clean under
protoc. Broken-anchor (10) and tck-id-anomaly (7)baselines unchanged. No assertion changes.