[ralyodio]

c0mpute hosting publish ./movies --name movies --disks auto failed with Error: site is over 256 MiB (1 GiB since v0.3.0). This makes folders of movies publish, replicate across the network without any node holding the whole site, and stream with seeking from any gateway.

Striped placement (sites over 1 GiB)

• Chunks are bucketed by hash into one group per 64 MiB of the site (the count comes from the size in the signed record). Each group goes to the 3 nodes that rank highest for (site, group) (rendezvous hashing, same answer on every node). No node needs the whole site.
• Healing, copy counting, GC, proof-of-storage challenges, budgets and the free-disk floor work per group. A node holds each group as soon as it is complete, so restarts resume group by group. A group short for 3 minutes brings in the next-ranked node.
• Gossip: Held.groups (a bitmap) and Holding.room (bytes a node can still take). Only nodes that announce room are counted on for a group, so older nodes are never relied on.
• Paid sites: up to 3 preferred hosts take every group; with more, the 3 that rank highest per group take it, and settlement pays each its share.
• hosting ls and /hosting/v1/sites: copies = the fewest copies of any group, plus groups, short_groups, missing_groups.
• Older nodes: they refuse records over their own cap (256 MiB before v0.3.0, 1 GiB in v0.3.0 to v0.3.2), so they never see a striped site (no holding, no serving). Sites up to 1 GiB replicate whole exactly as before. Groups are defined over chunk hashes only, so the same layout works for encrypted (ciphertext) sites.

Limits

	before	now
Site, paid (--disks)	1 GiB	256 GiB
Site, free	1 GiB	64 GiB
One file	n/a	128 GiB (one manifest part stays one chunk fetch)
Small sites other features publish (ads, apps, podcasts, git)	1 GiB	1 GiB (separate constant)
Node budget	5 GiB fixed	[hosting] max_bytes = 0 (default): what it holds + 25% of free space above the floor; a fixed value still wins
Per owner, per node	1 GiB	16 GiB, free copies only (preferred hosts are not limited)

The d1sks descriptor advertises the same budget, and --disks auto asks for disks with room for their share. Parted manifests (#211) now also close a part at 65,536 chunks, so a 256 GiB site's chunk lists are fetched a part at a time. Placement skips nodes that announced too little room for a copy, whole sites too (before, a full node ranked first left the site short forever).

Streaming

Gateways answer Range on /s/<id>/... and custom domains: 206 with Content-Range, 416 past the end, Accept-Ranges: bytes, If-Range, If-None-Match (304). Only the chunks a range covers are fetched; large bodies stream with 8 chunks of read-ahead instead of being loaded whole. Added mkv, m4v, mov, ogv, m3u8, srt types; media caches for an hour, HTML for a minute.

Faster replication

• Many chunks in flight (32 per copy, 8 of read-ahead per stream), each to the source with the lowest expected wait (its measured time per chunk times what is already queued on it). Per-peer limits adapt (AIMD) so a thin uplink is not asked for more than it can send before timeouts; slow requests are hedged to the next source; failures retry elsewhere; the DHT is last. A new holder pulls a group from all of its holders and the publisher at once.
• /c0mpute/chunk-fetch/2.0.0 sends chunks as CBOR byte strings (v1 sent integer arrays, about 1.9x the bytes); v1 is kept for older peers (fallback on UnsupportedProtocols). Chunk reads moved off the swarm event loop.
• No rsync/diskpush between nodes: chunks are already content-addressed, so only missing chunks move.
• Hosting nodes and publishers dial hosting peers that announced a public address, so a peer behind NAT can still be fetched from over the connection it opened.

Measured: dev2 -> bonita (home, behind NAT), 200 MiB file, one source

c0mpute hosting get <site> big.bin --from <holder> on bonita, the holder a publisher on dev2 (reached through an ssh tunnel, since dev2's random p2p ports are firewalled):

holder	client	MB/s
v0.3.x release (chunk-fetch v1)	sequential, as every release fetched	1.4
v0.3.x release (chunk-fetch v1)	this PR (window + AIMD)	5.0
this PR (chunk-fetch v2)	sequential	1.7
this PR (chunk-fetch v2)	this PR	10.7

7.6x on one source; with several holders a copy pulls from all of them at once. Over loopback the e2e test places 3 copies of 260 MiB in 2.6 s and a gateway streams the whole file in 0.7 s.

Finding, not fixed here: most nodes cannot dial each other at all. dev2 and dev3 drop inbound on random p2p ports and home nodes are behind NAT, and c0mpute has no relay, so cross-network copies only happen where one side has a reachable fixed port. Before this PR a 200 MB site published from bonita got 0 copies and the gateway answered 502 (Failed to dial the requested peer). The dial-out in this PR helps whenever one side is reachable; a fixed open p2p port on the gateway (or a relay) is still needed.

Publish UX

• Scans in place: nothing is copied; the publisher serves chunks from the files (hosting::seed), so 50 GiB of video does not need 50 GiB more disk. Hashes are cached per file (size + mtime), so a re-run is quick.
• Resumable: re-running reuses the record already on the network when the content is the same, and holders keep their groups.
• Progress bars for hashing and replication (bytes of 3 copies placed, groups complete, groups with no copy, MB/s). For a striped site --wait-secs is a no-progress timeout.
• Errors name the limit and the fix (free over 64 GiB: add --disks auto; over 256 GiB or a file over 128 GiB: split).
• New c0mpute hosting get <site> <path> (download with MB/s, --sequential to compare, --from to pin sources) and c0mpute hosting rm --name <site> (publishes an empty version; holders drop the files).

Tests

• Unit: stripe placement determinism, every group gets exactly 3 takers and no node takes everything, refill when a holder leaves touches only its groups, room and widening, group bitmaps, holding messages under 64 KiB, Range parsing (start/middle/end/suffix/open/past-end/multi), chunk slicing, chunk-bounded parts (including a 128 GiB file), scanner + cache, size caps, AIMD limits, auto budget.
• hosting_striped_e2e: a 260 MiB site (threshold lowered for the test) striped over 5 local nodes, every group at 3 copies with no node holding all, a gateway serving ranges across chunk boundaries plus 416/304/If-Range, and refill after a holder leaves.
• hosting_e2e, hosting_parts_e2e, podcast_e2e, ads_e2e, ads_billing_e2e pass locally (test nodes now pin a pretend disk above the free-disk floor, so they run on full boxes too).

🤖 Generated with Claude Code

[github-actions]

ThreatCrush Security Scan

13 finding(s)

HIGH/CRITICAL: 1 | MEDIUM: 8 | LOW: 4

Severity	Rule	Location
HIGH	js-ssrf-outbound-request	apps/web/public/sw.js:24
MEDIUM	js-unescaped-html-sink	apps/web/src/app/blog/[slug]/page.tsx:53
MEDIUM	js-unescaped-html-sink	apps/web/src/app/layout.tsx:89
MEDIUM	sh-remote-script-execution	scripts/dev-setup.sh:25
MEDIUM	sh-eval-expansion	scripts/dev-setup.sh:35
MEDIUM	sh-remote-script-execution	scripts/install.sh:181
MEDIUM	sh-remote-script-execution	scripts/install.sh:206
MEDIUM	sh-remote-script-execution	scripts/install.sh:399
MEDIUM	sh-remote-script-execution	scripts/install.sh:441
LOW	py-ssrf-outbound-request	examples/business-strategist-agent/chat.py:37
LOW	py-ssrf-outbound-request	examples/business-strategist-agent/train.py:54
LOW	secret-generic-credential	node/crates/c0mpute-core/src/webhook.rs:1171
LOW	py-path-traversal	node/crates/c0mpute-core/tests/fixtures/fake-infernet-text:29

Snippets are redacted; ThreatCrush never prints matched credential material.

[ralyodio]

Heads-up from a parallel session: this branch's build + test will fail on two tests in opendisk.rs, descriptor_is_valid_opendisk and the_descriptor_parses_as_a_marketplace_hosting_disk. Both still assert the old fixed 5 GiB budget, but Config::default() now means the automatic budget of the runner's real disk (CI saw 16.11 GiB). Fixed in #224, which is stacked on this branch and also raises the paid site cap to 4 TiB, as Anthony asked ("make it 4tb max"). Merging #224 into this branch, or cherry-picking 8ada74c, carries both.