Repository navigation
feat(hosting): 4 TiB site cap (stacked on #222) - #224
Merged
ralyodio merged 5 commits intoNov 9, 2026
Merged
Conversation
…fast multi-source replication - Sites over 1 GiB are striped: chunks bucketed by hash into one group per 64 MiB, each group placed on 3 nodes by rendezvous hashing on (site, group). Healing, copy counting, GC, challenges, budgets and the free-disk floor all work per group; gossip carries a group bitmap and each node's room. - Caps: 256 GiB paid / 64 GiB free per site, 128 GiB per file; small sites other features publish keep 1 GiB. Parted manifests also split by chunk count so every part stays one chunk fetch. - Node budget follows the disk by default (held + 25% of free above the floor); per-owner limit (16 GiB) applies to free copies only. Placement skips nodes that announced too little room, whole sites too. - Gateways answer Range requests (206/416, If-Range, 304) on /s/<id>/ and custom domains, fetching only the chunks a range covers, streaming large files with read-ahead; video content types (mkv, m4v, mov, ...). - Fetching: many chunks in flight, each from the holder with the lowest expected wait, AIMD per-peer limits, hedged slow requests, failover, DHT last. Chunk fetch protocol 2.0.0 sends byte strings (v1 sent integer arrays, ~1.9x the bytes); v1 kept for older peers. Chunk reads no longer block the swarm loop. - Hosting nodes and publishers dial hosting peers that announced a public address, so peers behind NAT can be fetched from. - publish scans in place (no copy; served from the files), caches hashes, resumes on re-run, shows hashing/replication progress, and says which limit was hit. --disks auto asks for disks with room for their share. - New: c0mpute hosting get (download with MB/s, --sequential to compare), c0mpute hosting rm (publish an empty version). Tests: stripe placement determinism and refill, room/widening, Range parsing and slicing, chunk-bounded parts, scanner and cache, AIMD limits, and hosting_striped_e2e: a 260 MiB site striped over 5 nodes, ranges via a gateway, refill after a holder leaves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g, fast replication) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paid (--disks) sites go from 256 GiB to 4 TiB. Striping already caps a site at MAX_GROUPS (8,192) groups, so past 512 GiB the group count stays put and each group grows (512 MiB at 4 TiB): the gossip bitmap stays at most 1 KiB and replication still streams chunks through the fetch window. Free sites stay at 64 GiB, one file at 128 GiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ThreatCrush Security Scan13 finding(s) HIGH/CRITICAL: 1 | MEDIUM: 8 | LOW: 4
Snippets are redacted; ThreatCrush never prints matched credential material. |
sh1pt-actions-fleet
Bot
force-pushed
the
feat/hosting-striped-media
branch
from
November 9, 2026 12:48
80dd655 to
0e9081c
Compare
…k budget size_caps_split_free_and_paid: 300 GiB is now a valid paid site; the 'Split' error moves to MAX_SITE_BYTES + 1 GiB. opendisk descriptor tests asserted the old fixed 5 GiB budget while Config::default() now means the automatic budget of the real disk (#222), so they failed on CI runners; pin max_bytes = 5 GiB in both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… feat/hosting-4tib-on-striped # Conflicts: # node/crates/c0mpute-core/src/opendisk.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #222. Merge #222 first; this PR then retargets to master on its own.
Anthony hit
c0mpute hosting publish ./done --name seed2 --disks auto->Error: site is over 1024 MiBon v0.3.1 and asked for 4 TB max. #222 fixes the error but caps paid sites at 256 GiB. This raises the cap to 4 TiB.Change
site::MAX_SITE_BYTES: 256 GiB -> 4 TiB (paid /--disks, v2 records). Free sites stay at 64 GiB, one file at 128 GiB, small feature sites (ads, apps, podcasts, git) at 1 GiB.stripe::MAX_GROUPS= 8,192 groups. Past 512 GiB the count stays at 8,192 and each group gets bigger (512 MiB at 4 TiB), so:HOLDING_MESSAGE_BYTESfetch_into_store, so a bigger group does not mean more memoryMAX_PARTSgrows by 128 (4 Mi chunks / 65,536 per part, times 2), and the root stays one chunk.docs/hosting.mdand the getting-started page say 4 TiB. The docs note that a multi-TiB paid site needs disks with room for their share (--copies 5= 60% each).group_countat 256 GiB (4,096), 512 GiB and 4 TiB (MAX_GROUPS), and 512 MiB groups at the cap.Older nodes refuse records over their own cap (1 GiB before #222, 256 GiB with #222 alone), so they never see a multi-TiB site.
Not compiled locally: dev2's disk is full and other sessions are running docker builds. CI's Rust job runs on this PR.
🤖 Generated with Claude Code