From 304b3c5ddcea4ed4182714e8705b8dbadc98ef79 Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Mon, 9 Nov 2026 12:38:31 +0000 Subject: [PATCH 1/8] feat(hosting): media-size sites: striped placement, Range streaming, fast multi-source replication - Sites over 1 GiB are striped: chunks bucketed by hash into one group per 64 MiB, each group placed on 3 nodes by rendezvous hashing on (site, group). Healing, copy counting, GC, challenges, budgets and the free-disk floor all work per group; gossip carries a group bitmap and each node's room. - Caps: 256 GiB paid / 64 GiB free per site, 128 GiB per file; small sites other features publish keep 1 GiB. Parted manifests also split by chunk count so every part stays one chunk fetch. - Node budget follows the disk by default (held + 25% of free above the floor); per-owner limit (16 GiB) applies to free copies only. Placement skips nodes that announced too little room, whole sites too. - Gateways answer Range requests (206/416, If-Range, 304) on /s// and custom domains, fetching only the chunks a range covers, streaming large files with read-ahead; video content types (mkv, m4v, mov, ...). - Fetching: many chunks in flight, each from the holder with the lowest expected wait, AIMD per-peer limits, hedged slow requests, failover, DHT last. Chunk fetch protocol 2.0.0 sends byte strings (v1 sent integer arrays, ~1.9x the bytes); v1 kept for older peers. Chunk reads no longer block the swarm loop. - Hosting nodes and publishers dial hosting peers that announced a public address, so peers behind NAT can be fetched from. - publish scans in place (no copy; served from the files), caches hashes, resumes on re-run, shows hashing/replication progress, and says which limit was hit. --disks auto asks for disks with room for their share. - New: c0mpute hosting get (download with MB/s, --sequential to compare), c0mpute hosting rm (publish an empty version). Tests: stripe placement determinism and refill, room/widening, Range parsing and slicing, chunk-bounded parts, scanner and cache, AIMD limits, and hosting_striped_e2e: a 260 MiB site striped over 5 nodes, ranges via a gateway, refill after a holder leaves. Co-Authored-By: Claude Opus 5.5 --- Cargo.lock | 1 + apps/web/src/app/getting-started/page.tsx | 7 +- docs/hosting.md | 119 +- node/crates/c0mpute-cli/src/main.rs | 78 +- node/crates/c0mpute-core/Cargo.toml | 1 + node/crates/c0mpute-core/src/ads/manage.rs | 4 +- node/crates/c0mpute-core/src/apps/deploy.rs | 4 +- node/crates/c0mpute-core/src/config.rs | 4 +- node/crates/c0mpute-core/src/hosting/fetch.rs | 349 +++ node/crates/c0mpute-core/src/hosting/mod.rs | 2063 +++++++++++++++-- node/crates/c0mpute-core/src/hosting/proof.rs | 17 + .../c0mpute-core/src/hosting/replicas.rs | 128 +- node/crates/c0mpute-core/src/hosting/seed.rs | 89 + .../crates/c0mpute-core/src/hosting/settle.rs | 16 +- node/crates/c0mpute-core/src/hosting/site.rs | 433 +++- .../crates/c0mpute-core/src/hosting/stripe.rs | 567 +++++ node/crates/c0mpute-core/src/opendisk.rs | 12 +- node/crates/c0mpute-core/src/podcast/show.rs | 2 +- node/crates/c0mpute-core/src/supervisor.rs | 6 +- .../c0mpute-core/tests/ads_billing_e2e.rs | 2 + node/crates/c0mpute-core/tests/ads_e2e.rs | 2 + node/crates/c0mpute-core/tests/hosting_e2e.rs | 2 + .../c0mpute-core/tests/hosting_market_e2e.rs | 2 + .../c0mpute-core/tests/hosting_striped_e2e.rs | 370 +++ node/crates/c0mpute-core/tests/podcast_e2e.rs | 2 + node/crates/c0mpute-net/src/swarm.rs | 130 +- 26 files changed, 4123 insertions(+), 287 deletions(-) create mode 100644 node/crates/c0mpute-core/src/hosting/fetch.rs create mode 100644 node/crates/c0mpute-core/src/hosting/seed.rs create mode 100644 node/crates/c0mpute-core/src/hosting/stripe.rs create mode 100644 node/crates/c0mpute-core/tests/hosting_striped_e2e.rs diff --git a/Cargo.lock b/Cargo.lock index fe06723f..bb3ff2ab 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -568,6 +568,7 @@ dependencies = [ "c0mpute-verify", "directories", "flate2", + "futures", "hex", "hkdf", "hmac", diff --git a/apps/web/src/app/getting-started/page.tsx b/apps/web/src/app/getting-started/page.tsx index 8bdeda43..9f743f8b 100644 --- a/apps/web/src/app/getting-started/page.tsx +++ b/apps/web/src/app/getting-started/page.tsx @@ -92,8 +92,11 @@ $ c0mpute hosting ls # sites a gateway knows, with live copy counts`}--name to update it. Any gateway serves it at{" "} /s/<site_id>/, for example{" "} - https://gateway.c0mpute.com/s/<site_id>/. Limits: - 256 MiB and 10,000 files per site. + https://gateway.c0mpute.com/s/<site_id>/, with + byte ranges so video seeks. Limits: 64 GiB per free site, 256 GiB + with --disks auto. Sites over 1 GiB are striped over the + network in 64 MiB groups; run the same command again to resume an + interrupted publish.

point a Moshpit name at it (free to register):

diff --git a/docs/hosting.md b/docs/hosting.md index 997a4e1d..afa75c90 100644 --- a/docs/hosting.md +++ b/docs/hosting.md @@ -9,11 +9,21 @@ c0mpute hosting publish ./public --name my-site # first version c0mpute hosting publish ./public --name my-site # same name again = new version c0mpute hosting publish ./public --name my-site --domain blog.example.org c0mpute hosting ls # sites a gateway knows, with copy counts +c0mpute hosting get movies/a.mp4 # download one file, shows MB/s +c0mpute hosting rm --name my-site # take it down (publishes an empty version) ``` `publish` signs the site with this machine's identity (`/buyer/identity.key`), -seeds it, and stays up until three hosting nodes hold copies (default wait 10 min). -Only the machine holding that key can publish new versions of the site. +seeds it, and stays up until three hosting nodes hold copies (default wait 10 min; for a +large site, 10 min without progress). Only the machine holding that key can publish new +versions of the site. + +The folder is scanned in place: nothing is copied, and the publisher serves chunks +straight from the files while the network takes its copies, so publishing 50 GiB of +video does not need another 50 GiB of local disk. Hashes of unchanged files are kept in +`/hosting/scan-cache.json`, so running the same command again is quick. That +is also how to resume: if `publish` stops (Ctrl-C, laptop asleep), run it again. The +record already on the network is reused and holders keep every group they have. The site is then served by every gateway at `http://:7777/s//`. Paths resolve like a static host: `/` and directories map to `index.html`, an @@ -30,18 +40,111 @@ is used for misses. | Healing | A site with fewer than 3 live copies: each hosting node ranks the candidates (rendezvous hashing on site id + peer id). The top-ranked ones fetch a copy. A node that stops announcing stops counting after ~200 s and another node takes over. Above 5 copies the lowest-ranked holder lets go. A publisher seeding its own site is a source, not a copy. | | Serving | Any gateway: local chunks when it holds them, otherwise fetched over libp2p from the fastest live holder (measured round-trip, round-robin among equally fast ones), failing over to the next and then to the DHT. Every chunk is checked against its hash. | -Limits per site: 1 GiB, 10,000,000 files. Up to 5,000 files the manifest is one chunk; -above that it is a root listing parts of 4,000 files each (by first path), so a -gateway fetches only the part a request needs. Nodes before v0.3.0 cannot read -parted manifests and skip those sites. A node holds at most `hosting.max_bytes` -(default 5 GiB) of other people's sites, and at most 1 GiB per owner key. +## Limits + +| | | +|---|---| +| Site, paid (`--disks`, a v2 record) | 256 GiB | +| Site, free (v1 record) | 64 GiB | +| Files per site | 10,000,000 | +| One file | 128 GiB | +| Small sites other features publish (ad creatives, app images, podcast shows, git repos) | 1 GiB, built in memory | +| Node budget | `[hosting] max_bytes`; `0` (default) = what it holds plus 25% of the free space above the free-disk floor | +| Free-disk floor | a copy must leave max(10 GiB, 5% of the disk), capped at 50 GiB, free | +| Per owner, per node | 16 GiB of free sites; sites the node is a preferred (paid) host for do not count | ```toml # ~/.config/c0mpute/config.toml [hosting] -max_bytes = 5368709120 +max_bytes = 0 # 0 = follow the disk; or a fixed number of bytes ``` +The budget advertised to d1sks (`max_gib`) is the same number, so `--disks auto` only +picks disks with room for their share of the site. + +Errors say which limit was hit and what to do (`--disks auto` for a free site over +64 GiB, split the folder above 256 GiB). A hosting node logs the limit that made it +refuse a copy (`over this node's hosting budget`, `disk nearly full`, `per-owner limit`). + +## Large sites: striped placement + +A site up to 1 GiB is replicated whole, exactly as before. Above that it is +**striped**: its chunks are bucketed by hash into groups, one group per 64 MiB of the +site (the count comes from the size in the signed record), and every group is placed on +its own: + +* The hosting nodes that rank highest for `(site, group)` (rendezvous hashing, the same + answer on every node) take that group until it has three copies. No node needs the + whole site; a 50 GiB folder spreads over the whole network. +* Hosting nodes gossip which groups they hold (a bitmap per site) and how many bytes + they can still take (`room`). Only nodes that announce `room` are counted on for a + group, and only when they have room for it. A group that stays short for 3 minutes + brings in the next-ranked node, so a chosen node that cannot take it never stalls it. +* Healing, copy counting, garbage collection, proof-of-storage challenges (a chunk of a + group the holder says it holds), budgets and the free-disk floor all work per group. + A node holds each group as soon as it is complete, so a restart resumes group by group. +* Paid sites: preferred hosts take every group when there are at most three of them (a + paid host holds the whole site), else the three that rank highest per group; each is + paid for its share (`size x 3 / preferred`). +* `c0mpute hosting ls` and `/hosting/v1/sites` show, for a striped site, `copies` = the + fewest copies of any group, plus `groups`, `short_groups` and `missing_groups`. + +Manifests: up to 5,000 files and 1 GiB the manifest is one chunk, byte for byte what +older nodes read. Above either, it is a root listing parts (by first path) of at most +4,000 files and 65,536 chunk hashes each, so every part is one chunk fetch even for a +256 GiB site; a gateway fetches only the part a request needs. Every holder of any group +keeps the root and all parts. + +Groups are defined over chunk hashes only (the group of a chunk is its hash modulo the +group count), never file contents. A gateway finds the holders of any chunk from the +record alone, and the same layout works for a site whose chunks are ciphertext. + +**Older nodes** refuse records over their own cap (256 MiB before v0.3.0, 1 GiB in +v0.3.0 to v0.3.2), so they never see a striped site: they neither hold nor serve it (a +gateway that old answers 404 for it), and they are never counted on for a group because +they do not announce `room`. Sites up to 1 GiB are unchanged and those nodes keep holding +and serving them. + +## Streaming and Range requests + +Gateways answer `Range` requests on `/s//...` and on custom domains: `206 Partial +Content` with `Content-Range`, only the chunks the range covers are fetched, and +`Accept-Ranges: bytes` is on every file, so `