diff --git a/SPEC.md b/SPEC.md index e115dab..d1bb06d 100644 --- a/SPEC.md +++ b/SPEC.md @@ -115,10 +115,26 @@ HTTP status: `200`. ``` - `columns` (REQUIRED, array of strings) — the column names of the result, in the order produced by the SQL engine. Empty array for statements that produce no result set (INSERT, UPDATE, DELETE, DDL). -- `rows` (REQUIRED, array of arrays) — each inner array has the same length as `columns`, with values in column order. Values use the same JSON / tagged-value encoding as section 5. Empty array if no rows. +- `rows` (REQUIRED, array of arrays) — each inner array has the same length as `columns`, with values in column order. Values use the same JSON / tagged-value encoding as section 5, subject to the emission rules below. Empty array if no rows. - `rowsAffected` (REQUIRED, integer) — the number of rows changed by the statement. `0` for SELECT. - `lastInsertId` (OPTIONAL, string, number, or null) — the identifier of the most recently inserted row when the server can determine it (typically the auto-increment id). `null` when not applicable or not available. +#### Response value encoding + +These rules are keyed on the **stored** value the statement produced, not on whatever runtime type the server's database driver handed back for it. + +The **JSON-safe integer range** is -(2^53 - 1) through 2^53 - 1 inclusive. + +- A server MUST emit an integer outside the JSON-safe integer range as `{"$type": "bigint", "$value": ""}`. +- A server MUST emit a binary value as `{"$type": "blob", "$value": ""}`. +- A server MUST NOT substitute a lossy representation — a rounded number, a truncated integer, a re-formatted string that does not decode to the stored value — for the tagged form. +- A server MAY emit the tagged form for any value it could also emit as a JSON primitive, including integers inside the JSON-safe range. +- A server SHOULD emit values that are exactly representable as JSON primitives as JSON primitives. + +The same rules apply to the `rows` of every `results` entry in section 6.2. + +Note, non-normative: this is a constraint on the whole server, not only on its encoding layer. A driver that returns a 64-bit integer as a double has already destroyed the value before any encoding step runs, so conformance here is decided by how the database is queried, not by how the result is serialized. + The arrays-of-arrays shape (not arrays-of-objects) is normative. It keeps payloads compact, makes column order explicit, and supports duplicate column names from joins. ### 6.2 Batch diff --git a/conformance/README.md b/conformance/README.md index d4f1811..4df72a8 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -61,6 +61,23 @@ Conformance is self-asserted. The community can call out failures via issues. | P-5 | Roundtrip a `bigint` tagged value | Returned value is `{"$type":"bigint","$value":""}` | | P-6 | Send an unknown tagged type `{"$type":"unknown","$value":"..."}` | 400, `error.code` = `bad_request` | +P-1 through P-5 only prove that a server can hand back what the client just gave it. They cannot detect a server that rounds values it reads out of storage, so the cases below read values the client never sent as parameters. + +### Response value encoding + +These cases exercise section 6.1's emission rules. Each writes the value as **SQL literal text**, so the value reaches storage without ever passing through the request's `params` array, then reads it back. + +| ID | Description | Expected response | +|-------|------------------------------------------------------------------|---------------------------------------------| +| V-1 | `INSERT INTO http_sql_conformance_notes (id, big_value) VALUES ('v1', 9007199254740993)` then `SELECT big_value FROM http_sql_conformance_notes WHERE id = 'v1'` | 200, value is `{"$type":"bigint","$value":"9007199254740993"}`. A bare JSON number FAILS this case, including `9007199254740992` — the rounded form. | +| V-2 | Same as V-1 with the negative bound `-9007199254740993` | 200, value is `{"$type":"bigint","$value":"-9007199254740993"}` | +| V-3 | `INSERT INTO http_sql_conformance_notes (id, blob_value) VALUES ('v3', x'48656c6c6f')` then SELECT it back | 200, value is `{"$type":"blob","$value":"SGVsbG8="}` | +| V-4 | Insert `42` into `big_value` as SQL literal text, then SELECT it | 200, value is the JSON number `42` (the tagged form `{"$type":"bigint","$value":"42"}` also passes -- section 6.1 permits it) | + +V-1 is the case that a server passes only if its database driver surfaces 64-bit integers without loss. An encoding layer that branches on the runtime type it was handed cannot pass V-1 by itself: once the driver returns a rounded double, the stored value is unrecoverable. + +Servers on a non-SQLite backend substitute their dialect's literal syntax for the binary literal in V-3 (for example `'\x48656c6c6f'::bytea` on PostgreSQL). The assertions on the response are unchanged. + ### Response headers | ID | Description | Expected response | @@ -82,10 +99,14 @@ The runner provisions a test schema before exercising the cases above. The fixtu ```sql CREATE TABLE http_sql_conformance_notes ( id TEXT PRIMARY KEY, - body TEXT + body TEXT, + big_value BIGINT, + blob_value BLOB ); ``` +`big_value` and `blob_value` exist for the V cases. Their declared types matter: on SQLite `BIGINT` carries INTEGER affinity and `BLOB` carries none, so neither column coerces the literal on the way in. Storing the V-1 literal in a `TEXT` column would convert it to a string and the case would prove nothing. On other backends use the nearest equivalents (PostgreSQL: `bigint` and `bytea`). + Servers SHOULD allow the test runner to issue this `CREATE TABLE` as a normal http-sql request, or provide an out-of-band setup hook. The runner cleans up rows it inserts but does not drop the table. ## Status diff --git a/examples/cloudflare-durable-object/README.md b/examples/cloudflare-durable-object/README.md index bfd4e68..959d5ac 100644 --- a/examples/cloudflare-durable-object/README.md +++ b/examples/cloudflare-durable-object/README.md @@ -89,6 +89,7 @@ This is the point: Bob isn't filtered out of Alice's table -- the table genuinel - **WebSocket fan-out for live sync.** The DO already holds the perfect spot for it: after a successful write, broadcast a `{type:"changed"}` message to every connected websocket for the same tenant. Connected browsers wake up and pull. That's how you get "tab A's INSERT shows up in tab B without polling." Skipped in v1 to keep the example focused; ~30 lines to add. - **JWT verification.** Hono has `hono/jwt` and works with JWKS-based verification too. The demo uses a hardcoded token map for clarity. - **Multi-database per tenant.** This example assumes one SQLite per tenant. If you want multiple logical databases per tenant, route on `/sql/:db` or include `db` in the token claims. +- **Return 64-bit integers without loss.** This one is a known non-conformance with spec section 6.1 and conformance case V-1, not a deferred feature. `SqlStorageValue` is `ArrayBuffer | string | number | null`; BigInt is not in the union and no option adds it, and the [storage docs](https://developers.cloudflare.com/durable-objects/api/storage-api/) state that a very large `int64` "may be less precise than your original number" when retrieved. The rounding happens inside the driver, before the DO sees the row. Tracked upstream at [workerd#4195](https://github.com/cloudflare/workerd/issues/4195). Workaround today: `SELECT CAST(col AS TEXT)` and parse the digits client-side. - **Migrations across DOs.** Schema changes need to fan out across every DO instance. You can do this lazily (first request after a deploy runs `CREATE TABLE IF NOT EXISTS` etc.) or eagerly (a job iterates the tenant directory). ## See also diff --git a/examples/cloudflare-durable-object/src/index.ts b/examples/cloudflare-durable-object/src/index.ts index 482a3b4..8efa592 100644 --- a/examples/cloudflare-durable-object/src/index.ts +++ b/examples/cloudflare-durable-object/src/index.ts @@ -155,6 +155,23 @@ function decodeParam(value: unknown): unknown { return value; } +// Response encoding per SPEC.md section 6.1. This branches on the runtime type +// ctx.storage.sql returned, which is only sufficient because the value survived +// the driver. +// +// KNOWN NON-CONFORMANCE (spec 6.1, conformance case V-1): SqlStorage has no +// lossless integer mode. The Durable Objects storage docs state that "any +// numeric value in a column is affected by JavaScript's 52-bit precision for +// numbers. If you store a very large number (in int64), then retrieve the same +// value, the returned value may be less precise than your original number." +// SqlStorageValue is ArrayBuffer | string | number | null -- BigInt is not in +// the union and no option adds it. So an integer above 2^53 arrives here +// already rounded and this branch cannot recover it. Tracked upstream at +// https://github.com/cloudflare/workerd/issues/4195 (open, covers DO SQLite +// mode as well as D1). Until that lands, the workaround is at the SQL layer: +// `SELECT CAST(col AS TEXT)` keeps the digits intact. Applying that +// automatically would require parsing the caller's SQL, which this example +// deliberately does not do. function encodeValue(value: unknown): unknown { if (value instanceof ArrayBuffer) return { $type: "blob", $value: base64Encode(new Uint8Array(value)) }; if (typeof value === "bigint") return { $type: "bigint", $value: value.toString() }; diff --git a/examples/cloudflare-worker-to-d1/README.md b/examples/cloudflare-worker-to-d1/README.md index 12dd466..efeea01 100644 --- a/examples/cloudflare-worker-to-d1/README.md +++ b/examples/cloudflare-worker-to-d1/README.md @@ -80,6 +80,7 @@ Response to the SELECT: - **No tenancy enforcement.** Anyone with the bearer token can run any SQL against the bound D1. Add row-level scoping (e.g. inject `WHERE tenant_id = ?` derived from the auth token) if you need multi-tenancy. - **No statement allowlisting.** A compromised token grants `DROP TABLE`. Production setups should restrict the statement surface based on the authenticated principal. - **No rate limiting.** Use Cloudflare's built-in rate limiting or a service binding to enforce it. +- **No lossless 64-bit integers on the way out.** This is a known non-conformance with spec section 6.1 and conformance case V-1, not a design choice. D1 stores 64-bit INTEGERs, but the Workers binding has no mode that returns them as `BigInt` -- an integer above 2^53 is rounded to a double inside the driver, before the Worker sees it. Tracked upstream at [workerd#4195](https://github.com/cloudflare/workerd/issues/4195). If you need those values today, `SELECT CAST(col AS TEXT)` in your SQL and parse the digits client-side. - **No pagination.** Per spec section 8, large result sets should be capped via `LIMIT` / `OFFSET` in the SQL. A future http-sql revision may add cursor pagination. ## Variants worth building yourself diff --git a/examples/cloudflare-worker-to-d1/src/index.ts b/examples/cloudflare-worker-to-d1/src/index.ts index 2918988..4650bf0 100644 --- a/examples/cloudflare-worker-to-d1/src/index.ts +++ b/examples/cloudflare-worker-to-d1/src/index.ts @@ -126,6 +126,19 @@ function decodeParam(value: unknown): unknown { return value; } +// Response encoding per SPEC.md section 6.1. This branches on the runtime type +// D1 returned, which is only sufficient because the value survived the driver. +// +// KNOWN NON-CONFORMANCE (spec 6.1, conformance case V-1): D1 stores 64-bit +// INTEGERs but its Workers binding has no lossless mode -- there is no option, +// method, or compatibility flag that makes it return a BigInt, so an integer +// above 2^53 comes back as an already-rounded double and reaches this function +// as a `number`. The original value is gone before any encoding step runs and +// this branch cannot recover it. Tracked upstream at +// https://github.com/cloudflare/workerd/issues/4195 (open). Until that lands, +// the workaround is at the SQL layer: `SELECT CAST(col AS TEXT)` keeps the +// digits intact. Applying that automatically would require parsing the caller's +// SQL and inferring column types, which this example deliberately does not do. function encodeValue(value: unknown): unknown { if (value instanceof ArrayBuffer) return { $type: "blob", $value: base64Encode(new Uint8Array(value)) }; if (typeof value === "bigint") return { $type: "bigint", $value: value.toString() }; diff --git a/implementations.md b/implementations.md index f2e5551..beccede 100644 --- a/implementations.md +++ b/implementations.md @@ -7,8 +7,8 @@ A directory of known servers and clients speaking the [http-sql v0.1 spec](./SPE | Implementation | Form | Backend | Notes | |----------------|------|---------|-------| | [examples/reference-server.ts](./examples/reference-server.ts) | TypeScript handler | _swap in your own DB_ | Dependency-free reference; the wire format with nothing else attached. | -| [examples/cloudflare-worker-to-d1](./examples/cloudflare-worker-to-d1) | Cloudflare Worker (Hono) | Cloudflare D1 | Drop-in http-sql endpoint for an existing D1 database. Auth via bearer token. | -| [examples/cloudflare-durable-object](./examples/cloudflare-durable-object) | Cloudflare Worker + Durable Object (Hono) | SQLite-backed DO storage | Each tenant is its own real SQLite at the edge. The flagship dogfood for "SQLite on both sides." | +| [examples/cloudflare-worker-to-d1](./examples/cloudflare-worker-to-d1) | Cloudflare Worker (Hono) | Cloudflare D1 | Drop-in http-sql endpoint for an existing D1 database. Auth via bearer token. Fails conformance case V-1: the D1 binding has no lossless 64-bit integer mode ([workerd#4195](https://github.com/cloudflare/workerd/issues/4195)). | +| [examples/cloudflare-durable-object](./examples/cloudflare-durable-object) | Cloudflare Worker + Durable Object (Hono) | SQLite-backed DO storage | Each tenant is its own real SQLite at the edge. The flagship dogfood for "SQLite on both sides." Fails conformance case V-1 for the same reason as the D1 example: `SqlStorageValue` has no BigInt. | ## Clients