From e95339fcaa050412e357af5447f2e04a454be708 Mon Sep 17 00:00:00 2001 From: dryrun Date: Sat, 24 Oct 2026 20:31:57 +0000 Subject: [PATCH 1/2] Ask for a new signature when a request hits its deadline The 503 body for a request that reached its deadline now says to retry with a new signature. The SQL error text and the route constant stay equal. Details: TC-1997 --- CHANGELOG.md | 3 ++- tinycloud-core/src/sql/types.rs | 2 +- tinycloud-node-server/src/routes/mod.rs | 3 ++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fb211081..4486fff9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ - SQL local durability and reservation before commit (`docs/sql-local-durability.md`). Postgres stays the master copy. File-backed SQL databases run `PRAGMA synchronous = FULL` (`[global.storage.sql] synchronous = "full"|"normal"`, env `TINYCLOUD_STORAGE__SQL__SYNCHRONOUS`; `normal` selects the earlier fsync behaviour), and the directories and renames that make a database file or a hydrated cache visible are fsynced. A write is checked before COMMIT and rolled back whole if it would pass the owner's remaining storage budget (`402`), and, when configured, the per-database cap (`max_database_bytes`, `507 sql_database_size_limit`) or the disk headroom (`min_free_bytes`, bounded by the WAL and checkpoint growth the transaction can cause and shared by all databases on the node, `507 sql_disk_headroom`). `max_database_bytes` and `min_free_bytes` default to `0` (off) and are set by config. A COMMIT that fails on disk answers `500 sql_commit_failed` and the node re-hydrates the database from Postgres. A write that fails because the disk is full answers `507 sql_disk_headroom`, whatever `min_free_bytes` is, not the owner's quota or the size limit. Every write request runs in one transaction (a single statement, a batch, a nested write (`WITH … INSERT`) and a request that carries a `PRAGMA` included), so such a request counts two more native SQL calls in request telemetry, and all of them get the same owner budget, size cap and disk gate. A request that wrote nothing is not gated and reserves nothing. The growth of the main file that the next checkpoint will cause stays held in the disk reservation from the commit until a checkpoint runs or the database's connection closes. No migration. Details: TC-1491. - `MemorySink` caps its size hint at 64 MiB and treats a failed reservation as no hint. Details: TC-1491. - A local SQL commit failure (answered 500) records the same diagnostic failure note as other SQL 500 responses. Details: TC-1491. +- The 503 body of a request that reached its deadline reads "request_timeout: no change was applied; retry the request with a new signature". Details: TC-1997 ## [1.22.0] - 2026-10-23 @@ -197,7 +198,7 @@ - SQL requests no longer wait without limit, and `/invoke` has a deadline (TC-1274). The cause of the original stall is still unknown; this change bounds the waits and reports them. - **Bounded waits.** A SQL request waits at most `storage.sql.lock_wait_timeout_ms` (30 s) for its database's operation lock, `hydration_wait_timeout_ms` (30 s) for the database to be loaded into the node's cache, and `artifact_transfer_timeout_ms` (45 s) for a durable artifact load or save. It then answers `503 sql_busy: : …` with `Retry-After: 5`. A save that times out after its write statement was sent is `500 outcome_unknown`: PostgreSQL may still commit that statement. - **Hydration keeps going.** Concurrent requests share one hydration, and it continues after its callers give up (up to `hydration_max_ms`, 10 min), so the next request finds the database warm. - - **Request deadline.** `/invoke` has a deadline, `invocation.deadline_ms` (55 s, below the ingress's 60 s; `0` turns it off). A request that hasn't started a data write by then is stopped and answered `503 request_timeout: no change was applied; retry the request` with `Retry-After: 1`. A request that has started one (a KV write transaction, a SQL statement, a DuckDB statement or import, a meeting publication, a decrypt) completes and reports its real result, and the node logs `Request completed after its deadline`. The invocation's replay row is kept, so a retry must be a fresh invocation, which the SDK already sends for every call. + - **Request deadline.** `/invoke` has a deadline, `invocation.deadline_ms` (55 s, below the ingress's 60 s; `0` turns it off). A request that hasn't started a data write by then is stopped and answered `503 request_timeout` (its body says that no change was applied and asks for a retry) with `Retry-After: 1`. A request that has started one (a KV write transaction, a SQL statement, a DuckDB statement or import, a meeting publication, a decrypt) completes and reports its real result, and the node logs `Request completed after its deadline`. The invocation's replay row is kept, so a retry must be a fresh invocation, which the SDK already sends for every call. - **Write safety.** A SQL write and its durable save always finish together, even if the request is stopped while waiting. - **Logging.** Typed warnings `SQL busy: lock wait|hydration|artifact transfer` (fields `error_class`, `stage`, `bound_ms`, `waited_ms`), `Database hydration gave up loading its artifact` and `Request deadline exceeded` (`stage`, `deadline_ms`) are new. The log shipper maps them to `sql_busy_*`, `sql_hydration_timeout`, `request_timeout` and `deadline_overrun`. The embedded Vector configuration is regenerated, so the reviewed N0 compose template changes to `00938226…` (raw `b25b8414…`; `docs/releasing.md`). - **Not covered.** DuckDB's lock and hydration waits are still unbounded, and a DuckDB request is not cut by the deadline once it reaches the DuckDB service; this is unchanged from before (TC-1412). Disconnect detection is TC-1400, bounding SQLite statement execution is TC-1401, and bounding identity resolution's artifact read is TC-1403 (until then only the deadline bounds it). diff --git a/tinycloud-core/src/sql/types.rs b/tinycloud-core/src/sql/types.rs index 731e0c12..496caf13 100644 --- a/tinycloud-core/src/sql/types.rs +++ b/tinycloud-core/src/sql/types.rs @@ -327,7 +327,7 @@ pub enum SqlError { Busy(SqlBusyStage), /// The request deadline was reached before the request's write began, so /// nothing was applied (TC-1274). - #[error("request_timeout: no change was applied; retry the request")] + #[error("request_timeout: no change was applied; retry the request with a new signature")] RequestTimeout, } diff --git a/tinycloud-node-server/src/routes/mod.rs b/tinycloud-node-server/src/routes/mod.rs index da081fa4..bf37f90e 100644 --- a/tinycloud-node-server/src/routes/mod.rs +++ b/tinycloud-node-server/src/routes/mod.rs @@ -142,7 +142,8 @@ pub const SQL_BUSY_RETRY_AFTER_SECS: u32 = 5; /// write began (TC-1274). pub const REQUEST_TIMEOUT: &str = "request_timeout"; /// The whole body of a [`REQUEST_TIMEOUT`] response. -pub const REQUEST_TIMEOUT_BODY: &str = "request_timeout: no change was applied; retry the request"; +pub const REQUEST_TIMEOUT_BODY: &str = + "request_timeout: no change was applied; retry the request with a new signature"; /// Body prefix of a `503` whose read's audit record was not committed within /// the read-audit wait bound (TC-1317); `Retry-After` is /// [`SQL_BUSY_RETRY_AFTER_SECS`], since the audit queue is backlogged. From 489ede7aa6a1e007f57c2a7cbb195d83dbef1ccc Mon Sep 17 00:00:00 2001 From: dryrun Date: Sat, 24 Oct 2026 20:48:23 +0000 Subject: [PATCH 2/2] Keep the released 1.21.1 changelog text as published Details: TC-1997 --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4486fff9..bf34e37a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ - SQL local durability and reservation before commit (`docs/sql-local-durability.md`). Postgres stays the master copy. File-backed SQL databases run `PRAGMA synchronous = FULL` (`[global.storage.sql] synchronous = "full"|"normal"`, env `TINYCLOUD_STORAGE__SQL__SYNCHRONOUS`; `normal` selects the earlier fsync behaviour), and the directories and renames that make a database file or a hydrated cache visible are fsynced. A write is checked before COMMIT and rolled back whole if it would pass the owner's remaining storage budget (`402`), and, when configured, the per-database cap (`max_database_bytes`, `507 sql_database_size_limit`) or the disk headroom (`min_free_bytes`, bounded by the WAL and checkpoint growth the transaction can cause and shared by all databases on the node, `507 sql_disk_headroom`). `max_database_bytes` and `min_free_bytes` default to `0` (off) and are set by config. A COMMIT that fails on disk answers `500 sql_commit_failed` and the node re-hydrates the database from Postgres. A write that fails because the disk is full answers `507 sql_disk_headroom`, whatever `min_free_bytes` is, not the owner's quota or the size limit. Every write request runs in one transaction (a single statement, a batch, a nested write (`WITH … INSERT`) and a request that carries a `PRAGMA` included), so such a request counts two more native SQL calls in request telemetry, and all of them get the same owner budget, size cap and disk gate. A request that wrote nothing is not gated and reserves nothing. The growth of the main file that the next checkpoint will cause stays held in the disk reservation from the commit until a checkpoint runs or the database's connection closes. No migration. Details: TC-1491. - `MemorySink` caps its size hint at 64 MiB and treats a failed reservation as no hint. Details: TC-1491. - A local SQL commit failure (answered 500) records the same diagnostic failure note as other SQL 500 responses. Details: TC-1491. -- The 503 body of a request that reached its deadline reads "request_timeout: no change was applied; retry the request with a new signature". Details: TC-1997 +- The 503 body of a request that reached its deadline reads "request_timeout: no change was applied; retry the request with a new signature". Details: TC-1997. ## [1.22.0] - 2026-10-23 @@ -198,7 +198,7 @@ - SQL requests no longer wait without limit, and `/invoke` has a deadline (TC-1274). The cause of the original stall is still unknown; this change bounds the waits and reports them. - **Bounded waits.** A SQL request waits at most `storage.sql.lock_wait_timeout_ms` (30 s) for its database's operation lock, `hydration_wait_timeout_ms` (30 s) for the database to be loaded into the node's cache, and `artifact_transfer_timeout_ms` (45 s) for a durable artifact load or save. It then answers `503 sql_busy: : …` with `Retry-After: 5`. A save that times out after its write statement was sent is `500 outcome_unknown`: PostgreSQL may still commit that statement. - **Hydration keeps going.** Concurrent requests share one hydration, and it continues after its callers give up (up to `hydration_max_ms`, 10 min), so the next request finds the database warm. - - **Request deadline.** `/invoke` has a deadline, `invocation.deadline_ms` (55 s, below the ingress's 60 s; `0` turns it off). A request that hasn't started a data write by then is stopped and answered `503 request_timeout` (its body says that no change was applied and asks for a retry) with `Retry-After: 1`. A request that has started one (a KV write transaction, a SQL statement, a DuckDB statement or import, a meeting publication, a decrypt) completes and reports its real result, and the node logs `Request completed after its deadline`. The invocation's replay row is kept, so a retry must be a fresh invocation, which the SDK already sends for every call. + - **Request deadline.** `/invoke` has a deadline, `invocation.deadline_ms` (55 s, below the ingress's 60 s; `0` turns it off). A request that hasn't started a data write by then is stopped and answered `503 request_timeout: no change was applied; retry the request` with `Retry-After: 1`. A request that has started one (a KV write transaction, a SQL statement, a DuckDB statement or import, a meeting publication, a decrypt) completes and reports its real result, and the node logs `Request completed after its deadline`. The invocation's replay row is kept, so a retry must be a fresh invocation, which the SDK already sends for every call. - **Write safety.** A SQL write and its durable save always finish together, even if the request is stopped while waiting. - **Logging.** Typed warnings `SQL busy: lock wait|hydration|artifact transfer` (fields `error_class`, `stage`, `bound_ms`, `waited_ms`), `Database hydration gave up loading its artifact` and `Request deadline exceeded` (`stage`, `deadline_ms`) are new. The log shipper maps them to `sql_busy_*`, `sql_hydration_timeout`, `request_timeout` and `deadline_overrun`. The embedded Vector configuration is regenerated, so the reviewed N0 compose template changes to `00938226…` (raw `b25b8414…`; `docs/releasing.md`). - **Not covered.** DuckDB's lock and hydration waits are still unbounded, and a DuckDB request is not cut by the deadline once it reaches the DuckDB service; this is unchanged from before (TC-1412). Disconnect detection is TC-1400, bounding SQLite statement execution is TC-1401, and bounding identity resolution's artifact read is TC-1403 (until then only the deadline bounds it).