LLM infrastructure engineer. I keep a model router in production and spend most of my time on the boring half of that job: which provider is healthy right now, which model is cheap enough for this particular call, and how much of the prompt never needed to be sent.
One OpenAI-compatible /v1 endpoint in front of ten or more providers. Every request is scored before it goes anywhere: how well a model fits the task, whether the provider is healthy, how stable it has been, and how much quota is left. The call goes to the cheapest model that clears the bar.
Cost comes down in four places, none of them clever:
- Role-based routing. A planner gets the strongest model, an executor the cheapest one that can finish the job, a reviewer something in between. Most requests are executor requests.
- Context compression before the call. The prompt is cut down before it reaches a provider, not after the invoice arrives.
- Cache-hit stability. Prompts are kept in a shape that keeps hitting the prefix cache, which takes out both cost and first-token latency.
- Per-provider rate limits with health-based fallback. A degraded provider drops out of scoring instead of failing requests.
Runs on Proxmox with Docker and LXC. Node and Python, state in SQLite, plain logs and cost metrics.
wheel scores candidate repositories on five deterministic maturity flags, measures three separate gaps between what a project gives and what you asked for, and returns an adoption mode instead of a diff. Writing from scratch is not one of the possible outcomes.
awesome-bc250 is the other half of the same habit: 125k messages of community chat compressed into a handbook that answers the questions in the order a person actually hits them, with the tuning trade-offs drawn as curves so nobody flashes a board before seeing what it costs.
Multi-agent orchestration: sub-agent fan-out, reviewer and judge roles, swarm patterns. Python services and mitm proxies when an API needs to be understood rather than trusted. Networking and DevOps from the hardware up: Proxmox, WireGuard, mesh, VLESS.
Engineering degree. Based in Kazakhstan, working remotely.
Open to LLM engineering roles — @maxim_kukuruza on Telegram.