Many machines. One model.
CrowdInfer is an early-stage, open-source project to run a single large language model across the combined memory and compute of multiple computers on a local network.
The goal is to make models that do not fit within one machine's available memory usable on ordinary hardware: a home server and a few laptops, or a larger pool of volunteer workstations. CrowdInfer aims to provide one local, OpenAI-compatible inference endpoint while placing parts of the model on different machines.
Status: concept / feasibility stage. Distributed inference is not implemented yet. This README describes project goals, not currently available functionality. See the roadmap for planned milestones.
A home might have a GPU workstation, an Apple Silicon MacBook Pro, and a MacBook Air. Each has useful memory and compute, but the largest model any one machine can run is constrained by its own resources.
CrowdInfer's central idea is to combine those resources at the model-execution level. Instead of merely routing separate requests to separate machines, participating nodes would execute different portions of the same inference request.
The idea is inspired by volunteer computing projects such as SETI@home, but interactive LLM inference is more tightly coupled: nodes must exchange intermediate activations and preserve inference state while generating tokens.
- A stable home server hosting the coordinator and inference API.
- Two or more optional worker machines.
- Ordinary wired 1 Gbit/s Ethernet as the baseline; no Thunderbolt, RDMA, or specialized network required.
- Initially Apple Silicon macOS workers; ultimately mixed macOS, Windows, and Linux.
- Resource-conscious participation: a laptop owner can limit contributed memory/compute and withdraw when the machine is needed.
A future enterprise deployment could encompass many workstations across subnets, but it is not the scope of the first proof of concept.
- Run larger models: Partition model weights and execution across nodes when the model cannot fit inside a single node's configured memory budget.
- Stay local: Keep model execution and prompts within a user-controlled LAN, without requiring a hosted inference service.
- Work over 1 GbE: Favor communication-efficient execution and measure end-to-end performance on commodity Ethernet.
- Be backend-independent: Separate cluster control and worker protocols from engines such as MLX or llama.cpp.
- Support heterogeneous hardware: Evolve toward Apple Silicon, NVIDIA-equipped Windows/Linux machines, and suitable CPU workers.
- Be elastic: Allow workers to join, gracefully withdraw, and eventually recover from failures.
- Be honest about performance: Compare against a single-machine baseline; pooling memory does not imply linear throughput scaling.
flowchart TB
Client["Local clients"] --> API["OpenAI-compatible API"]
API --> Coordinator["Coordinator and planner"]
Coordinator --> Registry["Worker and shard registry"]
Coordinator --> Pipeline["Distributed execution pipeline"]
Pipeline --> W1["Worker A: model layers"]
W1 --> W2["Worker B: model layers"]
W2 --> W3["Worker C: model layers"]
| Component | Responsibility |
|---|---|
| API gateway | Authentication, requests, token streaming, and model availability |
| Coordinator | Cluster membership, admission control, scheduling, and execution plans |
| Model planner | Map model layers and state to workers according to memory, compute, and network constraints |
| Worker agent | Advertise capabilities, enforce resource budgets, cache/load shards, execute assigned operations |
| Backend adapter | Connect the worker protocol to a native inference engine |
| Transport | Transfer intermediate activations and control messages across the LAN |
The stable coordinator does not need to host most model layers. A powerful laptop may be the best execution node while it is available.
Pipeline parallelism comes first. A worker hosts a contiguous range of transformer layers. During inference it receives intermediate activations, runs its layers, and forwards the result. This is a plausible starting point for 1 GbE because it avoids frequent all-to-all communication, but prompt processing, network latency, uneven stage speeds, and KV-cache placement still require measurement.
Tensor parallelism and mixture-of-experts-aware placement are research topics for later stages, not commitments for the initial release.
A worker's memory budget is a hard planning constraint; CPU/GPU percentage budgets require monitoring and best-effort throttling, not an assumption that arbitrary GPU workloads can be preempted precisely.
The first milestone is deliberately narrow:
- Two computers connected by 1 GbE.
- One supported transformer architecture and quantization format.
- A model that exceeds either worker's permitted memory budget, but fits within their combined budgets plus runtime overhead.
- Model layers executed on both machines for one inference request.
- Output checked against single-node reference inference within an appropriate numerical tolerance.
- Measurements for time to first token, prefill tokens/s, decoding tokens/s, per-stage execution time, and network traffic.
A successful first milestone proves distributed model execution, not automatic acceleration or seamless failover.
- Model-aware sharding, not remote swap: Place executable layers near compute rather than pretending that remote RAM is local GPU memory.
- Performance-aware placement: Account for bandwidth, latency, available accelerator memory, and per-stage execution time.
- Stable core, optional workers: Preserve the service endpoint when volunteers disconnect; an oversized model may become temporarily unavailable unless an alternative execution plan exists.
- Explicit compatibility: Cross-backend pipelines require agreed tensor formats, model semantics, and KV-cache behavior.
- Security by default: Authenticated, encrypted node-to-node communication; no anonymous access to worker execution endpoints. A trusted home LAN is still not a substitute for authentication.
- No cloud dependency: Model downloads may require internet access, but inference should not.
- Training or distributed fine-tuning.
- Combining unrelated small models into a stronger model.
- Guaranteeing that adding a worker increases tokens/s.
- Seamless migration of an in-progress request after sudden node failure.
- Support for arbitrary model architectures or arbitrary GPU backends.
- Production readiness or unattended execution on corporate endpoints.
CrowdInfer is a separate project, not an extension of JMLX. JMLX may eventually become one backend adapter for MLX execution, while CrowdInfer owns discovery, scheduling, network transport, sharding, and failure handling. Other inference engines should be usable without depending on JMLX.
CrowdInfer is in the feasibility and architecture-definition stage. There is no installation procedure or usable release yet.
The best early contributions are experiments and reproducible benchmarks for:
- Layer-partition execution in existing inference backends.
- Cross-node activation transfer over 1 GbE.
- Correct KV-cache ownership and recovery behavior.
- The performance of asymmetric workers.
- Secure and portable worker communication.
See ROADMAP.md for the proposed development sequence. Design discussion and benchmark results are welcome through GitHub issues.
CrowdInfer is licensed under the MIT License.