From 304b3c5ddcea4ed4182714e8705b8dbadc98ef79 Mon Sep 17 00:00:00 2001
From: Anthony Ettinger
Date: Mon, 9 Nov 2026 12:38:31 +0000
Subject: [PATCH 1/8] feat(hosting): media-size sites: striped placement, Range
streaming, fast multi-source replication
- Sites over 1 GiB are striped: chunks bucketed by hash into one group per
64 MiB, each group placed on 3 nodes by rendezvous hashing on (site, group).
Healing, copy counting, GC, challenges, budgets and the free-disk floor all
work per group; gossip carries a group bitmap and each node's room.
- Caps: 256 GiB paid / 64 GiB free per site, 128 GiB per file; small sites
other features publish keep 1 GiB. Parted manifests also split by chunk
count so every part stays one chunk fetch.
- Node budget follows the disk by default (held + 25% of free above the
floor); per-owner limit (16 GiB) applies to free copies only. Placement
skips nodes that announced too little room, whole sites too.
- Gateways answer Range requests (206/416, If-Range, 304) on /s// and
custom domains, fetching only the chunks a range covers, streaming large
files with read-ahead; video content types (mkv, m4v, mov, ...).
- Fetching: many chunks in flight, each from the holder with the lowest
expected wait, AIMD per-peer limits, hedged slow requests, failover, DHT
last. Chunk fetch protocol 2.0.0 sends byte strings (v1 sent integer
arrays, ~1.9x the bytes); v1 kept for older peers. Chunk reads no longer
block the swarm loop.
- Hosting nodes and publishers dial hosting peers that announced a public
address, so peers behind NAT can be fetched from.
- publish scans in place (no copy; served from the files), caches hashes,
resumes on re-run, shows hashing/replication progress, and says which
limit was hit. --disks auto asks for disks with room for their share.
- New: c0mpute hosting get (download with MB/s, --sequential to compare),
c0mpute hosting rm (publish an empty version).
Tests: stripe placement determinism and refill, room/widening, Range parsing
and slicing, chunk-bounded parts, scanner and cache, AIMD limits, and
hosting_striped_e2e: a 260 MiB site striped over 5 nodes, ranges via a
gateway, refill after a holder leaves.
Co-Authored-By: Claude Opus 5.5
---
Cargo.lock | 1 +
apps/web/src/app/getting-started/page.tsx | 7 +-
docs/hosting.md | 119 +-
node/crates/c0mpute-cli/src/main.rs | 78 +-
node/crates/c0mpute-core/Cargo.toml | 1 +
node/crates/c0mpute-core/src/ads/manage.rs | 4 +-
node/crates/c0mpute-core/src/apps/deploy.rs | 4 +-
node/crates/c0mpute-core/src/config.rs | 4 +-
node/crates/c0mpute-core/src/hosting/fetch.rs | 349 +++
node/crates/c0mpute-core/src/hosting/mod.rs | 2063 +++++++++++++++--
node/crates/c0mpute-core/src/hosting/proof.rs | 17 +
.../c0mpute-core/src/hosting/replicas.rs | 128 +-
node/crates/c0mpute-core/src/hosting/seed.rs | 89 +
.../crates/c0mpute-core/src/hosting/settle.rs | 16 +-
node/crates/c0mpute-core/src/hosting/site.rs | 433 +++-
.../crates/c0mpute-core/src/hosting/stripe.rs | 567 +++++
node/crates/c0mpute-core/src/opendisk.rs | 12 +-
node/crates/c0mpute-core/src/podcast/show.rs | 2 +-
node/crates/c0mpute-core/src/supervisor.rs | 6 +-
.../c0mpute-core/tests/ads_billing_e2e.rs | 2 +
node/crates/c0mpute-core/tests/ads_e2e.rs | 2 +
node/crates/c0mpute-core/tests/hosting_e2e.rs | 2 +
.../c0mpute-core/tests/hosting_market_e2e.rs | 2 +
.../c0mpute-core/tests/hosting_striped_e2e.rs | 370 +++
node/crates/c0mpute-core/tests/podcast_e2e.rs | 2 +
node/crates/c0mpute-net/src/swarm.rs | 130 +-
26 files changed, 4123 insertions(+), 287 deletions(-)
create mode 100644 node/crates/c0mpute-core/src/hosting/fetch.rs
create mode 100644 node/crates/c0mpute-core/src/hosting/seed.rs
create mode 100644 node/crates/c0mpute-core/src/hosting/stripe.rs
create mode 100644 node/crates/c0mpute-core/tests/hosting_striped_e2e.rs
diff --git a/Cargo.lock b/Cargo.lock
index fe06723f..bb3ff2ab 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -568,6 +568,7 @@ dependencies = [
"c0mpute-verify",
"directories",
"flate2",
+ "futures",
"hex",
"hkdf",
"hmac",
diff --git a/apps/web/src/app/getting-started/page.tsx b/apps/web/src/app/getting-started/page.tsx
index 8bdeda43..9f743f8b 100644
--- a/apps/web/src/app/getting-started/page.tsx
+++ b/apps/web/src/app/getting-started/page.tsx
@@ -92,8 +92,11 @@ $ c0mpute hosting ls # sites a gateway knows, with live copy counts`}
can publish new versions: run the same command with the same{" "}
--name to update it. Any gateway serves it at{" "}
/s/<site_id>/, for example{" "}
- https://gateway.c0mpute.com/s/<site_id>/. Limits:
- 256 MiB and 10,000 files per site.
+ https://gateway.c0mpute.com/s/<site_id>/, with
+ byte ranges so video seeks. Limits: 64 GiB per free site, 256 GiB
+ with --disks auto. Sites over 1 GiB are striped over the
+ network in 64 MiB groups; run the same command again to resume an
+ interrupted publish.
point a Moshpit name at it (free to register):
diff --git a/docs/hosting.md b/docs/hosting.md
index 997a4e1d..afa75c90 100644
--- a/docs/hosting.md
+++ b/docs/hosting.md
@@ -9,11 +9,21 @@ c0mpute hosting publish ./public --name my-site # first version
c0mpute hosting publish ./public --name my-site # same name again = new version
c0mpute hosting publish ./public --name my-site --domain blog.example.org
c0mpute hosting ls # sites a gateway knows, with copy counts
+c0mpute hosting get movies/a.mp4 # download one file, shows MB/s
+c0mpute hosting rm --name my-site # take it down (publishes an empty version)
```
`publish` signs the site with this machine's identity (`/buyer/identity.key`),
-seeds it, and stays up until three hosting nodes hold copies (default wait 10 min).
-Only the machine holding that key can publish new versions of the site.
+seeds it, and stays up until three hosting nodes hold copies (default wait 10 min; for a
+large site, 10 min without progress). Only the machine holding that key can publish new
+versions of the site.
+
+The folder is scanned in place: nothing is copied, and the publisher serves chunks
+straight from the files while the network takes its copies, so publishing 50 GiB of
+video does not need another 50 GiB of local disk. Hashes of unchanged files are kept in
+`/hosting/scan-cache.json`, so running the same command again is quick. That
+is also how to resume: if `publish` stops (Ctrl-C, laptop asleep), run it again. The
+record already on the network is reused and holders keep every group they have.
The site is then served by every gateway at `http://:7777/s//`.
Paths resolve like a static host: `/` and directories map to `index.html`, an
@@ -30,18 +40,111 @@ is used for misses.
| Healing | A site with fewer than 3 live copies: each hosting node ranks the candidates (rendezvous hashing on site id + peer id). The top-ranked ones fetch a copy. A node that stops announcing stops counting after ~200 s and another node takes over. Above 5 copies the lowest-ranked holder lets go. A publisher seeding its own site is a source, not a copy. |
| Serving | Any gateway: local chunks when it holds them, otherwise fetched over libp2p from the fastest live holder (measured round-trip, round-robin among equally fast ones), failing over to the next and then to the DHT. Every chunk is checked against its hash. |
-Limits per site: 1 GiB, 10,000,000 files. Up to 5,000 files the manifest is one chunk;
-above that it is a root listing parts of 4,000 files each (by first path), so a
-gateway fetches only the part a request needs. Nodes before v0.3.0 cannot read
-parted manifests and skip those sites. A node holds at most `hosting.max_bytes`
-(default 5 GiB) of other people's sites, and at most 1 GiB per owner key.
+## Limits
+
+| | |
+|---|---|
+| Site, paid (`--disks`, a v2 record) | 256 GiB |
+| Site, free (v1 record) | 64 GiB |
+| Files per site | 10,000,000 |
+| One file | 128 GiB |
+| Small sites other features publish (ad creatives, app images, podcast shows, git repos) | 1 GiB, built in memory |
+| Node budget | `[hosting] max_bytes`; `0` (default) = what it holds plus 25% of the free space above the free-disk floor |
+| Free-disk floor | a copy must leave max(10 GiB, 5% of the disk), capped at 50 GiB, free |
+| Per owner, per node | 16 GiB of free sites; sites the node is a preferred (paid) host for do not count |
```toml
# ~/.config/c0mpute/config.toml
[hosting]
-max_bytes = 5368709120
+max_bytes = 0 # 0 = follow the disk; or a fixed number of bytes
```
+The budget advertised to d1sks (`max_gib`) is the same number, so `--disks auto` only
+picks disks with room for their share of the site.
+
+Errors say which limit was hit and what to do (`--disks auto` for a free site over
+64 GiB, split the folder above 256 GiB). A hosting node logs the limit that made it
+refuse a copy (`over this node's hosting budget`, `disk nearly full`, `per-owner limit`).
+
+## Large sites: striped placement
+
+A site up to 1 GiB is replicated whole, exactly as before. Above that it is
+**striped**: its chunks are bucketed by hash into groups, one group per 64 MiB of the
+site (the count comes from the size in the signed record), and every group is placed on
+its own:
+
+* The hosting nodes that rank highest for `(site, group)` (rendezvous hashing, the same
+ answer on every node) take that group until it has three copies. No node needs the
+ whole site; a 50 GiB folder spreads over the whole network.
+* Hosting nodes gossip which groups they hold (a bitmap per site) and how many bytes
+ they can still take (`room`). Only nodes that announce `room` are counted on for a
+ group, and only when they have room for it. A group that stays short for 3 minutes
+ brings in the next-ranked node, so a chosen node that cannot take it never stalls it.
+* Healing, copy counting, garbage collection, proof-of-storage challenges (a chunk of a
+ group the holder says it holds), budgets and the free-disk floor all work per group.
+ A node holds each group as soon as it is complete, so a restart resumes group by group.
+* Paid sites: preferred hosts take every group when there are at most three of them (a
+ paid host holds the whole site), else the three that rank highest per group; each is
+ paid for its share (`size x 3 / preferred`).
+* `c0mpute hosting ls` and `/hosting/v1/sites` show, for a striped site, `copies` = the
+ fewest copies of any group, plus `groups`, `short_groups` and `missing_groups`.
+
+Manifests: up to 5,000 files and 1 GiB the manifest is one chunk, byte for byte what
+older nodes read. Above either, it is a root listing parts (by first path) of at most
+4,000 files and 65,536 chunk hashes each, so every part is one chunk fetch even for a
+256 GiB site; a gateway fetches only the part a request needs. Every holder of any group
+keeps the root and all parts.
+
+Groups are defined over chunk hashes only (the group of a chunk is its hash modulo the
+group count), never file contents. A gateway finds the holders of any chunk from the
+record alone, and the same layout works for a site whose chunks are ciphertext.
+
+**Older nodes** refuse records over their own cap (256 MiB before v0.3.0, 1 GiB in
+v0.3.0 to v0.3.2), so they never see a striped site: they neither hold nor serve it (a
+gateway that old answers 404 for it), and they are never counted on for a group because
+they do not announce `room`. Sites up to 1 GiB are unchanged and those nodes keep holding
+and serving them.
+
+## Streaming and Range requests
+
+Gateways answer `Range` requests on `/s//...` and on custom domains: `206 Partial
+Content` with `Content-Range`, only the chunks the range covers are fetched, and
+`Accept-Ranges: bytes` is on every file, so `