Skip to content

c0mpute edge: verify every byte from an untrusted gateway against the owner's signature #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ralyodio merged 1 commit into from
Nov 9, 2026
Merged

c0mpute edge: verify every byte from an untrusted gateway against the owner's signature #35

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,19 @@ browser --TLS (Moshpit CA leaf)--> edge --HTTP--> any healthy c0mpute gateway --
that announce a gateway URL), re-checked every minute; the last good list is
kept when discovery fails. No c0mpute server is on the path: any gateway can
serve any site.
- Only our own gateways (`C0MPUTE_GATEWAYS`, `C0MPUTE_TRUSTED_GATEWAYS`) are
proxied as they answer. Any c0mpute node can announce a gateway, so a
discovered one is a stranger's and gives only verified reads: the edge asks it
for `/hosting/v1/sites/<site>/resolve?path=...` (the owner-signed record, the
manifest and the file the path resolves to) and checks the ed25519 signature,
the site id, the manifest and parts by blake3, the path resolution (no
skipped candidate) and every chunk (`/hosting/v1/sites/<site>/chunks/<hash>`,
from any gateway) against its hash, the same rules as c0mpute's
`hosting/site.rs`. A record older than the highest seq already verified for
the site is refused (rollback). A gateway without those routes (c0mpute
before 0.3.5) or that fails a check sits out verified reads; the edge never
falls back to its unverified bytes. Verified responses carry
`x-c0mpute-verified: 1`; verified chunks are cached on disk by hash.
- When every gateway fails, the edge retries the round on a Fibonacci schedule
(0.5, 0.5, 1, 1.5, 2.5, 4 s), long enough to ride out a gateway restart or
self-update. If they are still down it serves the last good copy of the page
Expand All @@ -295,6 +308,9 @@ browser --TLS (Moshpit CA leaf)--> edge --HTTP--> any healthy c0mpute gateway --
| `MOSHPIT_API_KEY` | | the edge account's key; required to issue leaves |
| `MOSHPIT_REGISTRY_BASE` | `https://pit.moshcode.sh` | |
| `C0MPUTE_GATEWAYS` | | seed gateways, comma-separated (`http://host:7777`) |
| `C0MPUTE_TRUSTED_GATEWAYS` | | more gateways of ours, served without verification |
| `C0MPUTE_VERIFY_ALL` | off | `1`: verify our own gateways too |
| `EDGE_CHUNK_DIR` | `$EDGE_CERT_DIR/chunks` | verified chunk cache (1 GiB); `off` disables |
| `C0MPUTE_NODES_URL` | `https://c0mpute.com/api/nodes` | gateway discovery; empty disables |
| `EDGE_CERT_DIR` | `/var/lib/moshpit/c0mpute-edge` | |
| `EDGE_HTTPS_PORT` / `EDGE_HTTP_PORT` | `443` / `80` | `EDGE_HTTP_PORT=0` disables http |
Expand Down
18 changes: 16 additions & 2 deletions bin/moshpit-c0mpute-edge.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
import { realpathSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { CertStore, GatewayPool, LastGood, SiteLookup, startEdge } from "../lib/c0mpute-edge.ts";
import { ChunkCache, SeqGuard } from "../lib/c0mpute-verify.ts";

/** `host:port` or `[v6]:port`, or null. */
export function hostPort(v: string | undefined): { host: string; port: number } | null {
Expand All @@ -24,7 +25,12 @@ export function edgeConfig(env: Record<string, string | undefined> = process.env
registryBase: env.MOSHPIT_REGISTRY_BASE || "https://pit.moshcode.sh",
apiKey: env.MOSHPIT_API_KEY || "",
certDir: env.EDGE_CERT_DIR || "/var/lib/moshpit/c0mpute-edge",
// Our own gateways: served from as they answer. Every gateway discovered
// from the node list is a stranger's and only ever gives verified reads.
seeds: (env.C0MPUTE_GATEWAYS || "").split(/[\s,]+/).filter(Boolean),
trusted: (env.C0MPUTE_TRUSTED_GATEWAYS || "").split(/[\s,]+/).filter(Boolean),
// Verify every gateway, ours included.
verifyAll: /^(1|true|yes|on)$/i.test(env.C0MPUTE_VERIFY_ALL || ""),
// The public node list, for gateways that announce a URL. Empty disables
// discovery and the edge uses the seeds alone.
nodesUrl: env.C0MPUTE_NODES_URL === undefined ? "https://c0mpute.com/api/nodes" : env.C0MPUTE_NODES_URL || null,
Expand All @@ -36,6 +42,10 @@ export function edgeConfig(env: Record<string, string | undefined> = process.env
// Last good copy of every page, served stale when every gateway is down.
// On disk so it survives a restart of the edge; "off" keeps it in memory only.
staleDir: env.EDGE_STALE_DIR === "off" ? null : env.EDGE_STALE_DIR || `${env.EDGE_CERT_DIR || "/var/lib/moshpit/c0mpute-edge"}/stale`,
// Verified chunks (immutable, by hash) and the highest verified seq per
// site, so a restart neither refetches everything nor forgets the rollback floor.
chunkDir: env.EDGE_CHUNK_DIR === "off" ? null : env.EDGE_CHUNK_DIR || `${env.EDGE_CERT_DIR || "/var/lib/moshpit/c0mpute-edge"}/chunks`,
seqFile: `${env.EDGE_CERT_DIR || "/var/lib/moshpit/c0mpute-edge"}/site-seqs.json`,
};
}

Expand All @@ -44,18 +54,22 @@ async function main() {
const log = (line: string) => console.log(line);
if (!cfg.apiKey) log("[edge] warning: MOSHPIT_API_KEY is not set; only certificates already on disk can be served");

const gateways = new GatewayPool({ seeds: cfg.seeds, nodesUrl: cfg.nodesUrl });
const gateways = new GatewayPool({ seeds: cfg.seeds, trusted: cfg.trusted, nodesUrl: cfg.nodesUrl });
await gateways.refresh();
if (!gateways.size) log("[edge] warning: no c0mpute gateways known yet (set C0MPUTE_GATEWAYS)");
const timer = setInterval(() => { void gateways.refresh(); }, cfg.refreshMs);
timer.unref();

const lastGood = new LastGood({ dir: cfg.staleDir });
await lastGood.load();
const chunks = new ChunkCache({ dir: cfg.chunkDir });
await chunks.load();
const seqs = new SeqGuard({ file: cfg.seqFile });
await seqs.load();
const sites = new SiteLookup({ registryBase: cfg.registryBase });
const certs = new CertStore({ dir: cfg.certDir, registryBase: cfg.registryBase, apiKey: cfg.apiKey, log });
await startEdge({ sites, gateways, certs, host: cfg.host, httpsPort: cfg.httpsPort, httpPort: cfg.httpPort,
proxyProtocol: cfg.proxyProtocol, fallback: cfg.fallback, lastGood, log });
proxyProtocol: cfg.proxyProtocol, fallback: cfg.fallback, lastGood, chunks, seqs, verifyAll: cfg.verifyAll, log });
log(`[edge] serving c0mpute sites on https :${cfg.httpsPort}${cfg.httpPort ? ` and http :${cfg.httpPort}` : ""}; ${gateways.size} gateway(s) known${cfg.fallback ? `; other names to ${cfg.fallback.host}:${cfg.fallback.port}` : ""}`);
}

Expand Down
189 changes: 186 additions & 3 deletions lib/c0mpute-edge.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,12 @@
// discovered from seeds and from the public node list (nodes that announce a
// gateway URL), and the last good list is kept when discovery fails.
//
// Any c0mpute node may announce a gateway, so only our own (the seeds and
// C0MPUTE_TRUSTED_GATEWAYS) are proxied as they answer. A discovered gateway is
// a stranger's: from it the edge serves only what it has checked against the
// site owner's signature (lib/c0mpute-verify.ts), and a gateway that cannot
// give a verified read is skipped, never served from unverified.
//
// When every gateway fails a request the edge does not give up at once: it
// retries the round a few times on a Fibonacci schedule (a gateway restarting
// or re-executing for an update is back within seconds), then serves the last
Expand All @@ -40,6 +46,7 @@ import type { SecureContext, TLSSocket } from "node:tls";
import { join } from "node:path";
import type { Server as NetServer } from "node:net";
import { createSniFront } from "./sni-front.ts";
import { ChunkCache, SeqGuard, VerifyError, chunkLength, entryEtag, verifyChunk, verifyResolve } from "./c0mpute-verify.ts";
import { promisify } from "node:util";

const run = promisify(execFile);
Expand Down Expand Up @@ -107,7 +114,12 @@ export function gatewaysFromNodes(payload: unknown): string[] {
type GatewayState = { url: string; healthy: boolean; rttMs: number; failures: number; downUntil: number };

export interface GatewayPoolOptions {
/** Gateways to start from. Trusted (served from without verification) unless `trustSeeds` is false. */
seeds?: string[];
/** More gateways that are ours and served from without verification. */
trusted?: string[];
/** Default true: the seeds are our own gateways. */
trustSeeds?: boolean;
/** A node list to discover gateways from; null disables discovery. */
nodesUrl?: string | null;
fetchImpl?: typeof fetch;
Expand All @@ -126,14 +138,18 @@ export interface GatewayPoolOptions {
export class GatewayPool {
private states = new Map<string, GatewayState>();
private seeds: string[];
private trusted: Set<string>;
private nodesUrl: string | null;
private fetchImpl: typeof fetch;
private timeoutMs: number;
private now: () => number;
private turn = 0;

constructor(opts: GatewayPoolOptions = {}) {
this.seeds = (opts.seeds ?? []).map(normalizeGateway).filter((u): u is string => Boolean(u));
const norm = (list: string[] | undefined) => (list ?? []).map(normalizeGateway).filter((u): u is string => Boolean(u));
const extra = norm(opts.trusted);
this.seeds = [...new Set([...norm(opts.seeds), ...extra])];
this.trusted = new Set([...(opts.trustSeeds === false ? [] : norm(opts.seeds)), ...extra]);
this.nodesUrl = opts.nodesUrl ?? null;
this.fetchImpl = opts.fetchImpl ?? fetch;
this.timeoutMs = opts.timeoutMs ?? 3000;
Expand All @@ -152,6 +168,13 @@ export class GatewayPool {

get size(): number { return this.states.size; }

/**
* True for our own gateways (seeds and C0MPUTE_TRUSTED_GATEWAYS). Anything
* discovered from the node list is a stranger's, and the edge serves its
* bytes only after checking them against the site owner's signature.
*/
isTrusted(url: string): boolean { return this.trusted.has(url); }

/** Rediscover and re-check every gateway. Never throws; a failed discovery keeps the old list. */
async refresh(): Promise<void> {
if (this.nodesUrl) {
Expand Down Expand Up @@ -390,7 +413,18 @@ const PASS_HEADERS = ["content-type", "content-length", "etag", "last-modified",

export interface EdgeHandlerOptions {
sites: Pick<SiteLookup, "site">;
gateways: Pick<GatewayPool, "pick" | "succeeded" | "failed">;
/**
* `isTrusted` says which gateways are ours; a pool without it is taken to be
* all ours (the edge before any node could be a gateway).
*/
gateways: Pick<GatewayPool, "pick" | "succeeded" | "failed"> & Partial<Pick<GatewayPool, "isTrusted">>;
/** Verify trusted gateways too (C0MPUTE_VERIFY_ALL). */
verifyAll?: boolean;
/** Highest verified seq per site (rollback guard). In memory by default. */
seqs?: SeqGuard;
/** Verified chunks by hash. None (no cache) by default. */
chunks?: ChunkCache;
now?: () => number;
fetchImpl?: typeof fetch;
timeoutMs?: number;
attempts?: number;
Expand Down Expand Up @@ -532,6 +566,19 @@ export function createEdgeHandler(opts: EdgeHandlerOptions) {
const sleep = opts.sleep ?? ((ms: number) => new Promise<void>((r) => setTimeout(r, ms)));
const lastGood = opts.lastGood === undefined ? new LastGood() : opts.lastGood;
const log = opts.log ?? (() => {});
const now = opts.now ?? Date.now;
const verifyAll = Boolean(opts.verifyAll);
const seqs = opts.seqs ?? new SeqGuard();
const chunks = opts.chunks ?? new ChunkCache();
// Gateways that cannot do verified reads just now (no resolve route yet, or
// caught lying), and until when they sit those out.
const unverifiable = new Map<string, number>();
const trusted = (gw: string) => !verifyAll && (opts.gateways.isTrusted ? opts.gateways.isTrusted(gw) : true);
const sittingOut = (gw: string) => (unverifiable.get(gw) ?? 0) > now();
const sitOut = (gw: string, ms: number, why: string) => {
if (!sittingOut(gw)) log(`[edge] ${gw}: no verified reads for ${Math.round(ms / 1000)} s: ${why}`);
unverifiable.set(gw, now() + ms);
};

return async function handle(req: IncomingMessage, res: ServerResponse): Promise<void> {
const send = (status: number, text: string) => {
Expand Down Expand Up @@ -564,7 +611,14 @@ export function createEdgeHandler(opts: EdgeHandlerOptions) {

// One pass over the gateways; true when a response was sent.
const tryRound = async (): Promise<boolean> => {
for (const gw of opts.gateways.pick(attempts)) {
// Gateways that cannot serve a verified read just now are passed over,
// not counted against `attempts`.
const order = opts.gateways.pick(64).filter((gw) => trusted(gw) || !sittingOut(gw)).slice(0, attempts);
for (const gw of order) {
if (!trusted(gw)) {
if (await serveVerified(gw, order)) return true;
continue;
}
let upstream: Response;
try {
upstream = await fetchImpl(upstreamUrl(gw, site, path), {
Expand Down Expand Up @@ -624,6 +678,135 @@ export function createEdgeHandler(opts: EdgeHandlerOptions) {
return false;
};

// A stranger's gateway: take nothing from it on its word. Ask for the
// owner-signed evidence, check it (lib/c0mpute-verify.ts) and send only
// bytes whose hashes the signed manifest names. False when nothing was
// sent and the next gateway should be tried.
async function serveVerified(gw: string, others: string[]): Promise<boolean> {
const reqPath = path.split(/[?#]/)[0] || "/";
let answer: unknown;
try {
const r = await fetchImpl(`${gw}/hosting/v1/sites/${site}/resolve?path=${encodeURIComponent(reqPath)}`, {
redirect: "manual", signal: AbortSignal.timeout(timeoutMs),
});
if (r.status >= 500) {
await r.body?.cancel();
opts.gateways.failed(gw);
log(`[edge] ${host} via ${gw} (verified): ${r.status}`);
return false;
}
if (r.status !== 200) {
// 404: a gateway older than the resolve route (c0mpute < 0.3.5), or
// one that does not know the site. Either way, no verified read.
await r.body?.cancel();
sitOut(gw, 300_000, `resolve answered ${r.status}`);
return false;
}
answer = await r.json();
} catch (e) {
if (e instanceof SyntaxError) { sitOut(gw, 300_000, "resolve answer is not JSON"); return false; }
opts.gateways.failed(gw);
log(`[edge] ${host} via ${gw} (verified): ${(e as Error).message}`);
return false;
}
let resolved: ReturnType<typeof verifyResolve>;
try {
resolved = verifyResolve(site!, reqPath, answer, seqs.min(site!));
} catch (e) {
const msg = e instanceof VerifyError ? e.message : String(e);
log(`[edge] ${host}${reqPath} via ${gw}: REJECTED: ${msg}`);
sitOut(gw, 600_000, `failed verification (${msg})`);
return false;
}
opts.gateways.succeeded(gw);
seqs.note(site!, resolved.record.seq);

const base: Record<string, string> = {
"x-c0mpute-gateway": new URL(gw).host,
"x-c0mpute-verified": "1",
"x-content-type-options": "nosniff",
};
const entry = resolved.entry;
if (!entry) {
res.writeHead(404, { ...base, "content-type": "text/plain; charset=utf-8", "cache-control": "public, max-age=60" });
res.end(req.method === "HEAD" ? undefined : "not found");
return true;
}
const etag = entryEtag(entry);
const headers: Record<string, string> = {
...base,
"content-type": /^[\x20-\x7e]{1,255}$/.test(entry.content_type) ? entry.content_type : "application/octet-stream",
"content-length": String(entry.size),
etag,
"cache-control": "public, max-age=60",
};
if (resolved.status === 200 && conditional && String(req.headers["if-none-match"]).split(/\s*,\s*/).includes(etag)) {
delete headers["content-length"];
res.writeHead(304, headers);
res.end();
return true;
}
if (req.method === "HEAD") {
res.writeHead(resolved.status, headers);
res.end();
return true;
}

// Chunks from anywhere: the hash makes the source irrelevant. The
// gateway that resolved first, then the others still in the running.
const sources = [gw, ...others, ...opts.gateways.pick(64)]
.filter((o, i, all) => all.indexOf(o) === i && !sittingOut(o));
const chunk = async (i: number): Promise<Buffer | null> => {
const hash = entry.chunks[i];
const want = chunkLength(entry.size, i);
const cached = await chunks.get(hash);
if (cached && cached.length === want) return cached;
for (const src of sources) {
if (sittingOut(src)) continue;
try {
const r = await fetchImpl(`${src}/hosting/v1/sites/${site}/chunks/${hash}`, { redirect: "manual", signal: AbortSignal.timeout(timeoutMs) });
if (r.status !== 200) { await r.body?.cancel(); continue; }
const bytes = Buffer.from(await r.arrayBuffer());
verifyChunk(bytes, hash, want);
chunks.put(hash, bytes);
return bytes;
} catch (e) {
if (e instanceof VerifyError) {
log(`[edge] ${host}${reqPath} chunk from ${src}: REJECTED: ${e.message}`);
sitOut(src, 600_000, `served a bad chunk (${e.message})`);
}
}
}
return null;
};

// The first chunk before the status line, so a file whose chunks cannot
// be had falls through to the next gateway (and then the stale copy).
const first = await chunk(0);
if (!first) {
log(`[edge] ${host}${reqPath}: no gateway had a good copy of chunk 0 of ${resolved.path}`);
return false;
}
res.writeHead(resolved.status, headers);
const keep = lastGood && resolved.status === 200 && entry.size <= lastGood.maxEntryBytes ? [first] : null;
if (!res.write(first)) await new Promise((r) => res.once("drain", r));
for (let i = 1; i < entry.chunks.length; i++) {
const next = await chunk(i);
if (!next) {
// Every byte sent so far was checked; the rest cannot be had. Cut
// the response short rather than send anything unverified.
log(`[edge] ${host}${reqPath}: chunk ${i} of ${resolved.path} unavailable mid-file`);
res.destroy();
return true;
}
keep?.push(next);
if (!res.write(next)) await new Promise((r) => res.once("drain", r));
}
res.end();
if (keep) lastGood!.put(key, 200, headers, Buffer.concat(keep));
return true;
}

if (await tryRound()) return;
for (const ms of delays) {
if (res.destroyed || req.socket.destroyed) return;
Expand Down
Loading
Loading