Radix trie-based prompt prefix cache optimizer matching longest common prompt token prefixes to eliminate redundant prefill computation and reduce TTFT.
Radix trie-based prompt prefix cache optimizer matching longest common prompt token prefixes to eliminate redundant prefill computation and reduce TTFT.
Hi, I'm IncidentIQ!! A Slack-based SRE assistant built with Google ADK, FastAPI, and multi-agent orchestration. Works via Slack, API, or ADK Web UI; supports incident briefs, reliability reviews, AWS cost/core ops, Kubernetes checks, RAG over runbooks/past incidents, alert triage/dedupe, various LLM modes, eval benchmarks, and Docker deployment.
A production-style LLM evaluation pipeline spanning vLLM serving, lm-eval-harness integration, performance metrics (TTFT/TPOT/p95), deterministic guardrails, and statistically significant benchmark improvements.
Algorithmic context manager and execution engine for multi-agent workflows. Reduces TTFT and isolates attention by routing the minimal topological R/W subset to parallel LLMs.
LLM inference system with adaptive routing, multi-precision quantization, and speculative decoding (2.54× speedup). Achieves <200ms TTFT at 18.9 tok/s on 2.29GB VRAM via vLLM-PagedAttention with semantic caching and reproducible benchmarks.
To associate your repository with the ttft-optimization topic, visit your repo's landing page and select "manage topics."