fix(wire): BLOB columns no longer corrupt to {} — normalize ArrayBuffer to Uint8Array in both codecs

CHANGELOG.md

@@ -22,6 +22,17 @@ While pre-1.0, the public API may change between 0.x releases.
   sync over a plain `DurableObject` base — which Actors' own examples document.
   Supersedes the 0.5.0 "host defect" phrasing.
 
+### Fixed
+
+- **BLOB columns no longer corrupt to `{}` over the wire (ADR-0017,
+  [#27](https://github.com/grrowl/tanstack-durable-object-sync/issues/27)).**
+  workerd's `SqlStorage` returns BLOB values as bare `ArrayBuffer`, which the
+  msgpack encoder fell through to `encodeMap` on (and the JSON debug codec
+  stringified as `{}`) — clients silently received an empty object, for
+  snapshots and deltas alike. Both codecs now normalize `ArrayBuffer` at
+  emission, so BLOB columns arrive on the client as a `Uint8Array` with the
+  exact bytes.
+
 ## [0.5.1] — 2026-07-03
 
 ### Fixed

docs/adr/0017-blob-wire-normalization.md

@@ -0,0 +1,84 @@
+# ADR-0017: BLOB wire normalization — bare ArrayBuffer becomes Uint8Array at emission
+
+**Status**: Accepted
+**Date**: 2026-07-22
+**Issue**: [#27](https://github.com/grrowl/tanstack-durable-object-sync/issues/27)
+
+## Context
+
+workerd's `SqlStorage` returns BLOB column values as **bare `ArrayBuffer`**.
+Both row-reading paths (`SELECT *` in the sql-compiler's snapshot/fetch reader
+and `hydrateRows` for deltas) hand that value straight to the frame codec.
+
+Neither codec handled it:
+
+- **Binary (msgpack)**: `@msgpack/msgpack`'s `encodeObject` only special-cases
+  `ArrayBuffer.isView`, so a bare `ArrayBuffer` fell through to `encodeMap`
+  and encoded as an **empty map**. The client received `{}` — silently, for
+  snapshots and deltas alike.
+- **JSON debug codec**: `classify()` tagged `Uint8Array` (`u8`) but not
+  `ArrayBuffer`, and `JSON.stringify(ArrayBuffer)` is `{}` — same corruption.
+
+Silent data loss is the worst failure mode this repo recognises ("fail loud"),
+so *something* had to change. Two candidate shapes:
+
+1. **Normalize at emission**: teach both codecs that a bare `ArrayBuffer` is
+   bytes, delivered to the decoder as `Uint8Array`.
+2. **Reject in `assertSyncCompatible`**: refuse BLOB-affinity columns at
+   `registerSync` so authors fail loud instead of losing data.
+
+## Decision
+
+**Normalize at emission, in both codecs.** Rejection is only the right tool
+when the data genuinely cannot be made safe; here it can be, losslessly:
+
+- A BLOB is bytes. `Uint8Array` is the codecs' existing byte type (msgpack
+  `bin`; tagged-codec `u8`), and the decode side already produces `Uint8Array`
+  for it. Normalizing `ArrayBuffer → Uint8Array` at encode time is a pure
+  view change over the same bytes — nothing to lose.
+- The round-trip is stable: a client writing a `Uint8Array` back through a
+  mutation stores the same bytes in the BLOB column; the next read emits the
+  same `Uint8Array`. Optimistic and confirmed values agree.
+
+Implementation:
+
+- **Binary codec** (`wire/frame-codec.ts`): an `ExtensionCodec` entry
+  (custom ext type `0`) — `@msgpack/msgpack` consults the extension codec
+  *before* the `encodeMap` fallthrough, so it intercepts exactly the bare
+  `ArrayBuffer` case while `Uint8Array` keeps using the native `bin` format.
+  Decode returns the bytes as `Uint8Array`, never reconstructing an
+  `ArrayBuffer`.
+- **JSON debug codec** (`wire/codec.ts`): `classify()` tags `ArrayBuffer` as
+  `u8`; `toPlaceholder` reads its bytes through a `Uint8Array` view. Decode
+  was already symmetric (`u8 → Uint8Array`) and is unchanged.
+
+The normalization is one-way by design: **the wire has exactly one byte type,
+`Uint8Array`**. Clients never see a bare `ArrayBuffer`, regardless of which
+codec or which server runtime produced the row.
+
+### Why `assertSyncCompatible` does not also warn
+
+Once normalized, a BLOB column round-trips as faithfully as TEXT or INTEGER —
+there is nothing for an author to act on, so a warning would be noise
+attached to a working feature. Rejection/warning remains the tool for
+genuinely unsafe shapes (e.g. the rowid and pk-affinity rules of ADR-0015 /
+0001 D9), not for this one.
+
+### Out of scope
+
+INTEGER values above 2^53 still lose precision — workerd's `SqlStorage`
+returns JS numbers, so the damage happens before the wire is involved. That
+is issue [#10](https://github.com/grrowl/tanstack-durable-object-sync/issues/10)
+(wontfix), unaffected by this decision; `tests/row-shape.test.ts` pins the
+rounding so a behaviour change is noticed.
+
+## Consequences
+
+- BLOB columns are now syncable: clients receive `Uint8Array` with the exact
+  bytes, for cold snapshots, fetches, and live deltas alike.
+- Binary wire format: bare `ArrayBuffer` values encode as msgpack ext type
+  `0` rather than (corrupt) map. Both ends of the wire ship in this package,
+  so no cross-version concern exists — but a foreign msgpack decoder would
+  need the same extension registered to read BLOB values.
+- `tests/row-shape.test.ts` pins the end-to-end guarantee (snapshot + delta);
+  `tests/codec.test.ts` / `tests/frame-codec.test.ts` pin it per codec.

docs/adr/README.md

@@ -22,3 +22,4 @@ explains the displacement.
 | [0013](./0013-predicate-floor-one-evaluator.md) | Filtered-subscription membership: one evaluator is the source of truth; the floor is the verified-agreeing set | Accepted |
 | [0014](./0014-object-sync-schema.md) | `defineSync`: one schema value, mutations on the collection, commands on the connection | Accepted (supersedes 0001 D11 builder; closes 0010 manifest) |
 | [0015](./0015-syncable-mixin.md) | `Syncable` mixin: the sync core as a mixin over any DO base | Accepted (reframes 0001 D13; Actor unsupported) |
+| [0017](./0017-blob-wire-normalization.md) | BLOB wire normalization: bare ArrayBuffer becomes Uint8Array at emission | Accepted |

src/wire/codec.ts

@@ -1,8 +1,9 @@
 // Tagged value codec (ADR-0001 D17).
 //
 // Lossless transport + at-rest encoding for values JSON cannot represent on its
-// own: bigint, Date, NaN, ±Infinity, -0, undefined, and Uint8Array. Naive
-// `JSON.stringify` silently corrupts every one of these.
+// own: bigint, Date, NaN, ±Infinity, -0, undefined, and Uint8Array (plus bare
+// ArrayBuffer, normalized to Uint8Array — ADR-0017). Naive `JSON.stringify`
+// silently corrupts every one of these.
 //
 // Collision-proof by construction: special values are replaced by neutral
 // placeholders in the payload, and their (path, type) is recorded in a separate
@@ -38,6 +39,10 @@ function classify(v: unknown): TypeTag | null {
       if (v === null) return null
       if (v instanceof Date) return "date"
       if (v instanceof Uint8Array) return "u8"
+      // workerd's SqlStorage returns BLOB columns as bare ArrayBuffer; bare
+      // JSON stringifies it as {}. Normalize to bytes at emission — decodes as
+      // Uint8Array, same as a Uint8Array input (ADR-0017).
+      if (v instanceof ArrayBuffer) return "u8"
       return null
     case "function":
     case "symbol":
@@ -54,7 +59,7 @@ function toPlaceholder(tag: TypeTag, v: unknown): unknown {
     case "date":
       return (v as Date).getTime()
     case "u8":
-      return Array.from(v as Uint8Array)
+      return Array.from(v instanceof ArrayBuffer ? new Uint8Array(v) : (v as Uint8Array))
     case "-0":
       return 0
     case "nan":

src/wire/frame-codec.ts

@@ -13,7 +13,7 @@
 // saves ~9 bytes/frame, dwarfed by row payloads, so it is deferred to M9 and
 // measured rather than assumed. Flagged here rather than silently dropped.
 
-import { decode as mpDecode, encode as mpEncode } from "@msgpack/msgpack"
+import { decode as mpDecode, encode as mpEncode, ExtensionCodec } from "@msgpack/msgpack"
 import { decode as valueDecode, encode as valueEncode } from "./codec.ts"
 import type { ClientFrame, ServerFrame } from "./frames.ts"
 
@@ -36,7 +36,23 @@ function toBytes(d: WireIn): Uint8Array {
   return new TextEncoder().encode(d)
 }
 
-const MSGPACK_OPTS = { useBigInt64: true } as const
+// workerd's SqlStorage returns BLOB columns as bare ArrayBuffer. @msgpack/msgpack
+// only special-cases ArrayBuffer.isView, so a bare ArrayBuffer would fall through
+// to encodeMap and arrive as {} — silently (issue #27, ADR-0017). The extension
+// codec is consulted before that fallthrough: encode the raw bytes, decode to
+// Uint8Array so BLOBs land on the client exactly like a Uint8Array column value.
+const ARRAY_BUFFER_EXT_TYPE = 0
+const extensionCodec = new ExtensionCodec()
+extensionCodec.register({
+  type: ARRAY_BUFFER_EXT_TYPE,
+  encode: (v) => (v instanceof ArrayBuffer ? new Uint8Array(v) : null),
+  // Copy out of the decode buffer: `bytes` is a subarray VIEW of the whole
+  // incoming frame — returning it as-is would alias the wire buffer (mutable
+  // by the caller) and keep the full frame alive behind a small BLOB.
+  decode: (bytes) => bytes.slice(), // Uint8Array — normalized, never back to ArrayBuffer
+})
+
+const MSGPACK_OPTS = { useBigInt64: true, extensionCodec } as const
 
 const binaryCodec: FrameCodec = {
   binary: true,

tests/codec.test.ts

@@ -57,6 +57,17 @@ describe("tagged value codec", () => {
     expect(Array.from(r as Uint8Array)).toEqual([0, 1, 2, 254, 255])
   })
 
+  it("normalizes a bare ArrayBuffer (workerd BLOB) to Uint8Array", () => {
+    // workerd's SqlStorage returns BLOB columns as bare ArrayBuffer, which
+    // bare JSON stringifies to {} — silent corruption (issue #27). Normalized
+    // at emission: the decoder yields a Uint8Array with the exact bytes.
+    const r = roundtrip({ payload: new Uint8Array([1, 2, 255]).buffer }) as {
+      payload: unknown
+    }
+    expect(r.payload).toBeInstanceOf(Uint8Array)
+    expect(Array.from(r.payload as Uint8Array)).toEqual([1, 2, 255])
+  })
+
   it("does not misinterpret user data shaped like an internal tag", () => {
     // Collision-proofing: this object must survive as plain data.
     const v = { $type: "bigint", value: "not-a-bigint", m: [[[], "date"]] }

tests/frame-codec.test.ts

@@ -66,6 +66,30 @@ for (const binary of [true, false]) {
       expect(row.nested).toEqual({ a: [1, 2, 3], b: "z" })
     })
 
+    it("normalizes a bare ArrayBuffer row value (workerd BLOB) to Uint8Array", () => {
+      // workerd's SqlStorage returns BLOB columns as bare ArrayBuffer. msgpack
+      // only special-cases ArrayBuffer.isView, so without normalization a bare
+      // ArrayBuffer fell through to encodeMap and arrived as {} — silently
+      // (issue #27). Both codecs must deliver the bytes as a Uint8Array.
+      const f: ServerFrame = {
+        t: "snap",
+        sub: "s",
+        key: "k",
+        row: { payload: new Uint8Array([1, 2, 254]).buffer },
+        seq: "1",
+      }
+      const encoded = codec.encode(f)
+      const r = codec.decode(encoded) as Extract<ServerFrame, { t: "snap" }>
+      const payload = (r.row as Record<string, unknown>).payload
+      expect(payload).toBeInstanceOf(Uint8Array)
+      expect(Array.from(payload as Uint8Array)).toEqual([1, 2, 254])
+      // The decoded bytes must be a COPY, not a view aliasing the wire buffer:
+      // an aliased view is mutable through the transport's buffer and pins the
+      // whole frame allocation behind a small BLOB.
+      if (encoded instanceof Uint8Array) encoded.fill(0)
+      expect(Array.from(payload as Uint8Array)).toEqual([1, 2, 254])
+    })
+
     it(`encodes as ${binary ? "Uint8Array" : "string"}`, () => {
       const out = codec.encode({ t: "uptodate", seq: "1" })
       if (binary) expect(out).toBeInstanceOf(Uint8Array)