sql_store: S3 and filesystem object stores (slice 1b PR B)

.github/workflows/rust.yml

@@ -332,6 +332,12 @@ jobs:
         (set -a; . "$TC887_RUN_DIR/harness.env"; set +a
          cargo test --locked -p tinycloud-node --test s3_block_store -- --ignored)
 
+    - name: SQL object store (against the harness MinIO)
+      timeout-minutes: 10
+      run: |
+        (set -a; . "$TC887_RUN_DIR/harness.env"; set +a
+         cargo test --locked -p tinycloud-node --test sql_object_store -- --ignored)
+
     - name: Harness logs
       if: failure()
       run: |

CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## [Unreleased]
 
+- Added the SQL object store backends for S3 (conditional create with a startup probe and a check-then-create fallback) and the filesystem (atomic no-clobber create), with the contract rules in `docs/sql-block-store-contract.md` section 12. The S3 block store's start-up size scan lists the `tinycloud:` key prefix only. Details: TC-1491.
+
 ## [1.24.0] - 2026-10-25
 
 - KV puts persist content-addressed blobs before taking the per-space sequence lock, and committed `kv/sync` cursors stay ordered. A preparatory transaction authorizes the write and claims every distinct blob under a durable attempt, committing all claims or none; on PostgreSQL each claim statement also checks that the invocation's spaces are hosted. Uploads hold no pooled database connection, transaction or lock. Protected legacy meeting writes keep their pre-upload freeze barrier, which waits for commit or abort and fails after 30 seconds if stalled (TC-1260).

docs/sql-block-store-contract.md

@@ -698,3 +698,29 @@ These keys sit under `[sql_store]`. They are checked at boot only when `enabled
 
 PR A tests: `object_format_must_match_its_kind`, `segment_from_must_follow_base_seq` and `object_state_flip_checks_the_format` (11.1); `created_at_is_stored_canonically`, `created_at_out_of_range_is_invalid_not_a_panic` and `revive_restarts_the_orphan_clock` (11.2); `sql_store_refuses_enable_mysql_and_non_canonical_spaces` (11.3); `sql_store_defaults_match_the_contract`, `sql_store_new_setting_defaults_and_pg_concurrency_floor`, `sql_store_new_settings_parse_and_are_bounded`, `sql_store_settings_are_checked_only_when_enabled` and `sql_store_new_settings_are_bounded_when_enabled` (11.4). The manifest tests run against SQLite and PostgreSQL.
 
+## 12. Object store backends
+
+Slice 1b PR B adds the two backends of the object store trait in `tinycloud-node-server/src/storage/sql_object.rs`. Neither is started by the node yet.
+
+### 12.1 S3
+
+- The store uses the block store's bucket and client settings. The object key is the key of section 3.1, unchanged. The object carries its lowercase hex SHA-256 as user metadata `sha256`.
+- A body that does not hash to the key is refused with `BodyMismatch` before any request is sent.
+- In conditional mode an object is created with one PUT carrying `If-None-Match: *`. A 412 answer is accepted only after a HEAD shows the stored object has the same length and the same `sha256` metadata (`AlreadyPresent`); any other stored object gives `Conflict`. A stored object without the metadata is a backend error, never a match.
+- The startup probe writes a fixed small body to `sql/v1/_probe/v1` twice with `If-None-Match: *`. A 412 on the second request selects conditional mode. A second success, or a 501 or NotImplemented answer for the header, selects check-then-create mode. Any other failure, any other status included, is an error and the store does not start. The result is logged once at info level (target `tinycloud::sql_store`) with the mode and the status and code of the answer that decided it. The probe object is not deleted.
+- In check-then-create mode the store sends a HEAD first. An absent object is created with an unconditional PUT; an existing one is compared as above. Creators of one key in one process run one at a time in both modes; creators in different processes can each report `Created` for the same key in check-then-create mode, and each stores the bytes the key names.
+- In conditional mode a 409 `ConditionalRequestConflict` answer (another conditional write of the key is in flight) is retried up to 5 times; any other status is a backend error with its status and code.
+
+### 12.2 Filesystem
+
+- The root is `{datadir}/sql-store`. A relative root or block store root is made absolute against the current directory first (a lexical step that reads nothing from the filesystem; `..` components are refused). Boot then refuses a root that is equal to, inside, or a parent of the block store root. The filesystem store is not supported on Windows.
+- The path of an object is the root joined with its key. Every key component must be an ordinary file name, and the resulting path must stay under the root.
+- A create writes the body to a `.tmp-` file in the target directory, fsyncs it, and publishes it with a hard link, which fails instead of replacing an existing file. An existing file with the same length and SHA-256 gives `AlreadyPresent`, any other gives `Conflict`. The directory is fsynced and the temporary file is removed in every outcome. Each new directory and its parent are fsynced when created, and the parent of a directory that already exists is fsynced too. A leftover temporary file does not affect a later create.
+
+### 12.3 Ranges and head
+
+Both backends answer a range as the in-memory store does: the range is resolved against the object size and clamped to it, and a range that selects no byte (start at or beyond the end, an inverted range, a zero-length suffix, any range of an empty object) and a missing object both give `None`. `head` returns the length and SHA-256; the filesystem store hashes the file in bounded chunks.
+
+### 12.4 KV size scan
+
+The S3 block store's start-up size scan lists objects with the prefix `tinycloud:`. Space ids start with `tinycloud:`, and only keys whose part before the last `/` is a space id count towards a space, so the computed sizes are the same as for a scan of the whole bucket and `sql/v1/` objects are not listed.

tinycloud-auth/src/resource.rs

@@ -281,9 +281,12 @@ pub enum ResourceCheckError {
     DoesNotExtendPath,
 }
 
+/// The text every space id string starts with.
+pub const SPACE_ID_PREFIX: &str = "tinycloud:";
+
 impl fmt::Display for SpaceId {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        write!(f, "tinycloud:{}:{}", self.suffix(), self.name)
+        write!(f, "{}{}:{}", SPACE_ID_PREFIX, self.suffix(), self.name)
     }
 }
 

tinycloud-node-server/src/storage/mod.rs

@@ -1,6 +1,7 @@
 pub mod file_system;
 pub mod s3;
 pub mod size;
+pub mod sql_object;
 
 use tinycloud_auth::resource::SpaceId;
 use tinycloud_core::hash::Hash;

tinycloud-node-server/src/storage/s3.rs

@@ -26,7 +26,7 @@ use std::{
     sync::Arc,
     time::Instant,
 };
-use tinycloud_auth::resource::SpaceId;
+use tinycloud_auth::resource::{SpaceId, SPACE_ID_PREFIX};
 use tinycloud_core::{hash::Hash, storage::*};
 use tokio_util::bytes::Bytes;
 
@@ -37,6 +37,9 @@ use tokio::sync::Mutex;
 #[cfg(test)]
 use tokio::sync::Notify;
 
+/// Prefix of the boot scan's listing: every KV object key starts with it.
+pub(crate) const KV_SCAN_PREFIX: &str = SPACE_ID_PREFIX;
+
 /// Pinned so an SDK bump cannot change client defaults unnoticed.
 pub fn behavior_version() -> BehaviorVersion {
     BehaviorVersion::v2026_01_12()
@@ -115,7 +118,7 @@ impl StorageSetup for S3BlockStore {
     }
 }
 
-async fn new_client(config: &S3BlockConfig) -> Client {
+pub(crate) async fn new_client(config: &S3BlockConfig) -> Client {
     let general_config = aws_config().await;
     // Checksums only when an operation requires them: the SDK 1.x defaults
     // (CRC32 trailers, aws-chunked uploads, checksum-mode GETs) are rejected
@@ -163,6 +166,9 @@ impl S3BlockStore {
         let mut pages = client
             .list_objects_v2()
             .bucket(&config.bucket)
+            // Every KV object key is `{space}/{hash}` and a space id string
+            // starts with `SPACE_ID_PREFIX`; only such keys count towards a space.
+            .prefix(KV_SCAN_PREFIX)
             .into_paginator()
             .send();
         // get the sum of all objects per space across all pages, and the
@@ -202,7 +208,7 @@ impl S3BlockStore {
         })
     }
 
-    fn key_for(space: &SpaceId, id: &Hash, generation: i64) -> String {
+    pub(crate) fn key_for(space: &SpaceId, id: &Hash, generation: i64) -> String {
         let mut key = format!("{space}/");
         // Hash digests fit in 64 bytes; encode on the stack so generated
         // reads allocate only their storage-key String.