Define message-size string accounting as UTF-8 byte length (with explicit string-length exceptions) by sacOO7 · Pull Request #516 · ably/specification

sacOO7 · 2026-08-12T12:38:18Z

Problem

The client-side publish-size gate (RTO15d for ObjectMessage, TO3l8/maxMessageSize for Message) sums per-field sizes and rejects an over-limit publish before it hits the wire. The spec told SDKs to measure string fields by their "length", but never said whether "length" means UTF-8 bytes or UTF-16 code units. For ASCII these coincide; for non-ASCII they diverge (你 = 3 UTF-8 bytes / 1 UTF-16 code unit; 😊 = 4 UTF-8 bytes / 2 UTF-16 code units).

The spec was also internally contradictory:

OM3d (extras): "the string length of its JSON representation" — reads as UTF-16 code units.
OD3g (json): "the byte length of its JSON-encoded string representation" — explicitly bytes.
OM3f / MCR3a1 / MST3c / MRM3a / OMP4a1 / OD3e / TM6f: bare "length" — undefined.

Because "length" was undefined, the SDKs diverged, and the same map key or clientId produced different gate results across SDKs. Concrete before-state (clientId = "émile", keys = "héllo👍", string data = "你好", extras = {"k":"你"}):

Field (spec point)	Value	cocoa (before)	java (before)	js (before)	After (all SDKs)
`clientId` (OM3f)	`émile`	6 (UTF-8)	6 (UTF-8)	5 (UTF-16)	6
MapSet key (MST3c)	`héllo👍`	10 (UTF-8)	10 (UTF-8)	7 (UTF-16)	10
map-state key (OMP4a1)	`héllo👍`	7 (UTF-16)	7 (UTF-16)	7 (UTF-16)	10
string data (OD3e)	`你好`	6 (UTF-8)	6 (UTF-8)	6 (UTF-8)	6
extras JSON (OM3d)	`{"k":"你"}`	9 (UTF-16)	9 (UTF-16)	9 (UTF-16)	9 (unchanged)

This surfaces the concrete gate-flip: a boundary ObjectMessage whose clientId/keys push it just over the limit is accepted by js (UTF-16 undercount) but rejected by cocoa/java (UTF-8) — the same publish, two verdicts. Two distinct defects:

Cross-SDK divergence. js under-counted clientId and operation keys (UTF-16) while cocoa/java counted them in UTF-8 bytes.
Intra-SDK inconsistency. cocoa/java measured the same map key two different ways depending on whether it arrived via an operation (MST3c, UTF-8) or via map state (OMP4a1, UTF-16).

The authoritative definition already exists in Ably's published billing/limits accounting — How is maximum message size measured? — verbatim:

The size is calculated as the sum of the name, clientId, data and extras properties before any compression or expansion occurs in the serialization process.

name and clientId: "calculated as the size in bytes of their UTF-8 representation"

data: "calculated as the size in bytes if it is in binary, or its UTF-8 byte length if it is a string"

extras: "calculated as the string length of its JSON representation"

binary on text transports: "the size is calculated using the actual size of the binary data, not its base64 encoded string"

arrays: "the message size limit applies to the sum of all messages in the array"

Note the deliberate asymmetry in the published contract: name/clientId/string data are UTF-8 bytes, but extras is string length of its JSON representation.

Relationship to #331

This supersedes #331, the earlier attempt at the same disambiguation. This work substantially agrees with #331 — its conclusions and these changes converge on the same unit for every field both cover (string data, name, clientId, OD3e → UTF-8), and per-point coverage is documented below. Two deltas:

extras deliberately follows the published docs (UTF-16 string length) where Message/PresenceMessage/Annotation size #331's discussion had converged on UTF-8. Measuring extras in UTF-8 would over-count (bytes ≥ units) and false-reject documented-valid messages — the worse failure direction. Message/PresenceMessage/Annotation size #331's UTF-8-for-extras was a hedged consistency preference ("technically a bug ... not one that matters much"), with no authoritative server-side statement that the server byte-counts extras. The docs are the currently-published enforced contract.
This PR additionally disambiguates the LiveObjects map/operation keys (MCR3a1/MST3c/MRM3a/OMP4a1) and OD3g, which Message/PresenceMessage/Annotation size #331 predates, and uses a single canonical umbrella at TO3l8f instead of per-type duplication (addressing Message/PresenceMessage/Annotation size #331's "don't copy-paste size clauses per type" review feedback).

#331 is now CONFLICTING (the repo migrated textile/features.textile → specifications/features.md, so its diff no longer applies). It is left open for the author to close; this PR notes the supersession.

Solution

Match the service's published accounting per field. Every plain string and map key is measured in UTF-8 bytes; extras (and the regular-Message object/array data JSON) keep the documented "string length" (UTF-16 code units).

Component	Spec point(s)	Unit	Rationale
`name` (regular Message)	TM6a + TO3l8f default	UTF-8 bytes	docs: "size in bytes of UTF-8 representation"
`clientId` (regular + object)	(TM6a) / OM3f	UTF-8 bytes	docs verbatim
string `data`	TM6f	UTF-8 bytes	docs verbatim
binary `data` / `bytes`	TM6c / OD3c	raw bytes	actual binary, not base64
`ObjectData.string`	OD3e	UTF-8 bytes	same string family as values
`ObjectData.json`	OD3g	UTF-8 bytes	already byte-length; server counts bytes
`ObjectData.number`	OD3d	8	fixed
`ObjectData.boolean`	OD3b	1	fixed
operation keys	MCR3a1 / MST3c / MRM3a	UTF-8 bytes	keys are the same kind of string as values
map-state entry key	OMP4a1	UTF-8 bytes	changed from UTF-16 — consistency with operation keys
`extras` (regular + object)	TM6d / OM3d	UTF-16 string length of JSON	docs verbatim: "string length of its JSON representation"
object/array `data` (regular Message)	TM6b	UTF-16 string length after JSON-stringify	pre-existing convention; unchanged

Per-clause edits to specifications/features.md:

→ UTF-8 byte length: TM6f (string data), OM3f (object clientId), MCR3a1, MST3c, MRM3a, OMP4a1 (all map/operation keys), OD3e (string), OD3g (json — clarified "byte" → "UTF-8 byte").
→ explicit UTF-16 string length: TM6b (object/array data), TM6d and OM3d (extras) — reworded from bare "string length" to "string length (the number of UTF-16 code units)".
Canonical umbrella at TO3l8f: one sentence establishing the default (all message-size string accounting is UTF-8 byte length unless a clause states otherwise, explicitly including the json OD3g payload) and enumerating the two UTF-16 exceptions (extras TM6d/OM3d and object/array data TM6b). Deliberately one umbrella rule rather than per-field/per-type duplication.

Wording-only change; no test additions in this repo.

SDK status

SDK	Sites	Fix	Status
ably-js	`objectmessage.ts` (clientId/OM3f, OMP4a1, MCR3a1, MST3c, MRM3a → `dataSizeBytes`); `message.ts` `getMessageSize` (name+clientId → `Utils.dataSizeBytes`)	UTF-8 for keys/clientId/name; `extras` stays `JSON.stringify(...).length`	companion PR: ably/ably-js#2289
ably-cocoa	`ObjectMessage.swift` `ObjectsMap.size` (OMP4a1: `utf16.count` → `utf8.count` — the one behavioural change); `ARTBaseMessage.m` core-Message alignment	UTF-8 keys; `extras`/object-array `data` UTF-16	changes ready on `feature/liveobjects-implementation`
ably-java	`WireObjectMessage.kt` `WireObjectsMap.size` (OMP4a1: `.length` → `.byteSize`)	UTF-8 (java was the UTF-8 reference impl); `extras` UTF-16	changes ready on `refactor/uts-objects-unit-into-liveobjects`

Notes

The extras/UTF-16 rationale. The published accounting has an explicit rule for extras ("string length of its JSON representation") distinct from the byte rule for other strings. Matching it exactly (UTF-16 code units) makes the gate neither false-accept nor false-reject extras relative to the documented contract. UTF-8 would over-count and false-reject documented-valid messages. This is a deliberate documented exception, not an inconsistency; all three SDKs already agreed on UTF-16 here.
Why keys/strings are UTF-8 (strict-dominance). For any string, UTF-8 byte length ≥ UTF-16 code-unit count (equality only for ASCII). Measuring a key in UTF-8 bytes equals the server's on-the-wire byte count, so the gate never false-accepts a key the server would reject. UTF-8 strictly dominates UTF-16 for keys.
name/clientId for regular Message have no explicit per-field sub-clause in TM6; they are covered by the TO3l8f umbrella default (UTF-8).
TP5 (PresenceMessage) needed no edit — it already delegates wholesale to TM6, and RSAN1a4 (annotations) delegates to TO3l8, so presence and annotation sizing inherit the rule with no separate edits.

…N representations as string length) Disambiguate every message-size string clause in features.md. "length"/"string length" was undefined (UTF-8 bytes vs UTF-16 code units), and the spec was internally contradictory (OM3d string-length vs OD3g byte-length), causing cross-SDK gate divergence for non-ASCII clientIds and map keys. - UTF-8 byte length: OM3f (object clientId), MCR3a1/MST3c/MRM3a/OMP4a1 (all map/operation keys), OD3e (string), OD3g (json; "byte" -> "UTF-8 byte"), TM6f (string data). - Explicit UTF-16 string length: TM6b (object/array data), TM6d and OM3d (extras) reworded from bare "string length" to "string length (the number of UTF-16 code units)". - Canonical umbrella at TO3l8f: a single default (message-size string accounting is UTF-8 byte length, including OD3g) with the two UTF-16 exceptions (extras TM6d/OM3d and object/array data TM6b), matching Ably's published message-size accounting. Supersedes #331.

Copilot

Pull request overview

Clarifies the specification’s message-size accounting rules by defining how to measure string sizes (defaulting to UTF-8 byte length) and explicitly documenting the UTF-16 code-unit exceptions, to eliminate cross-SDK divergence and internal inconsistencies.

Changes:

Defines string data, clientId, and LiveObjects map key sizing as UTF-8 byte length (e.g., TM6f, OM3f, MCR3a1, MST3c, MRM3a, OMP4a1, OD3e, OD3g).
Makes the UTF-16 code-unit basis explicit for JSON-stringified extras and object/array data sizing (TM6b, TM6d, OM3d).
Adds an umbrella rule at TO3l8f defining the default and enumerating the exceptions.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

      - `(TO3l8d)` This clause has been replaced by [TM6d](#TM6d)
      - `(TO3l8e)` This clause has been replaced by [TM6e](#TM6e)
-      - `(TO3l8f)` The size is defined as the sum of all message sizes being published, calculated based on [TM6](#TM6), [TP5](#TP5) and [OM3](#OM3)
+      - `(TO3l8f)` The size is defined as the sum of all message sizes being published, calculated based on [TM6](#TM6), [TP5](#TP5) and [OM3](#OM3). Unless a clause explicitly states otherwise, string sizes in message-size accounting are measured as their UTF-8 byte length (this includes the `json` `ObjectData` payload, [OD3g](#OD3g)). The only exceptions are the `extras` property ([TM6d](#TM6d)/[OM3d](#OM3d)) and an `Object` or `Array` `data` property ([TM6b](#TM6b)), whose JSON representations are instead measured by string length (the number of UTF-16 code units), matching the service's published accounting for `extras`


…a1); align regular-Message extras sizing with the published accounting Message-size accounting now matches Ably's published per-field rule: every plain string field and map key is measured as its UTF-8 byte length, while extras (and JSON-stringified object/array data) keep the documented "string length of its JSON representation" (UTF-16 code units). Sites changed: - ObjectMessage.swift: ObjectsMap.size (OMP4a1) key measurement utf16.count -> utf8.count, so map-state entry keys now match the MapCreate/MapSet/MapRemove operation keys; extras stays utf16.count; the string-measurement comment block is rewritten for the resolved convention. - ARTBaseMessage.m#messageSize (core regular Message): extras changed from UTF-8 byte length to NSString.length (UTF-16 string length) and object/array data from jsonData length (bytes) to jsonString length (UTF-16); clientId and string data stay UTF-8. Tests: +4 non-ASCII ObjectMessage tests (clientIdIsUTF8ByteLength OM3f, mapSetKeyIsUTF8ByteLength MST3c, objectsMapEntryKeyIsUTF8ByteLength OMP4a1, extrasIsUTF16StringLengthOfJSON OM3d); +1 core test test__027__ in UtilitiesTests covering name/clientId/string-data UTF-8 and extras/object-data UTF-16. Spec: ably/specification#516

…(OMP4a1) Message-size accounting matches Ably's published per-field rule: every plain string field and map key is measured as its UTF-8 byte length, while extras keeps the documented "string length of its JSON representation" (UTF-16 code units). Sites changed: - WireObjectMessage.kt: WireObjectsMap.size (OMP4a1) key measurement it.key.length -> it.key.byteSize, so map-state entry keys now match the MapCreate/MapSet/MapRemove operation keys; fixed a duplicated-// comment typo; corrected the WireObjectData json branch comment from OD3e to OD3g; extras keeps gson.toJson(it).length (UTF-16) now with an explanatory comment. Tests: +1 non-ASCII test testObjectMapStateEntryKeyUnicodeSizeIsUtf8 (OMP4a1). Spec: ably/specification#516

ttypic · 2026-08-20T14:18:39Z

 - `(TM6)` The size of the `Message` for [TO3l8](#TO3l8) is calculated as follows:
  - `(TM6a)` The size is the sum of the sizes of the `name`, `data`, `clientId`, and `extras` properties
-  - `(TM6b)` The size of an `Object` or `Array` `data` property is its string length after being JSON-stringified
+  - `(TM6b)` The size of an `Object` or `Array` `data` property is its string length (the number of UTF-16 code units) after being JSON-stringified


Shouldn’t this be all measured in bytes? I’m not sure I understand the logic behind choosing UTF-16/UTF-8 encoding

It's updated as per https://ably.com/docs/platform/pricing/faqs#concurrent-channels

PS. More information included in PR description

github-actions Bot deployed to staging/pull/516 August 12, 2026 12:38 View deployment

sacOO7 requested a lite review from Copilot August 12, 2026 12:39

Copilot started reviewing on behalf of sacOO7 August 12, 2026 12:40 View session

sacOO7 mentioned this pull request Aug 12, 2026

Measure size of string fields in UTF-8 bytes for maxMessageSize accounting ably/ably-js#2289

Open

Copilot AI reviewed Aug 12, 2026

View reviewed changes

sacOO7 mentioned this pull request Aug 12, 2026

Message/PresenceMessage/Annotation size #331

Closed

ttypic reviewed Aug 20, 2026

View reviewed changes

VeskeR self-requested a review August 20, 2026 14:37

Navigation Menu

Uh oh!

Define message-size string accounting as UTF-8 byte length (with explicit string-length exceptions) - #516

Define message-size string accounting as UTF-8 byte length (with explicit string-length exceptions)#516
sacOO7 wants to merge 1 commit into
mainfrom
size-accounting-utf8-byte-length

sacOO7 commented Aug 12, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

ttypic Aug 20, 2026

Uh oh!

sacOO7 Aug 20, 2026 •

edited

Loading

Uh oh!

sacOO7 Aug 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

sacOO7 commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Relationship to #331

Solution

SDK status

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

ttypic Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

sacOO7 Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sacOO7 Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

sacOO7 commented Aug 12, 2026 •

edited

Loading

sacOO7 Aug 20, 2026 •

edited

Loading