kvcache

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL weight transfer), and EP (e.g., GPU-driven)

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

[Neurips 2025] R-KV: Redundancy-aware KV Cache Compression for Reasoning Models

Inference-native Tokenmaxxing Agent Harness for Loop Engineering

Persist and reuse KV Cache to speedup your LLM.

CacheRoute is an innovative LLM scheduling scheme dedicated to enabling flexible KV cache reuse across LLM systems, improving task performance and system efficiency.

Alibaba Cloud's high-performance KVCache system for LLM inference, with components for global cache management, inference simulation(HiSim), and more.

Sardeenz is a proof-of-concept application that allows you to load more than one model on a given GPU. It allows you to add more and more models onto a GPU, until it is fully utilized.

PiKV: KV Cache Management System for Mixture of Experts [Efficient ML System]

Recomputation-free, context-independent KV caching for large language models

(ACL2025 oral) SCOPE: Optimizing KV Cache Compression in Long-context Generation

🔥 [ICML'26] ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs

Official implementation of "Shadow in the Cache: Unveiling and Mitigating Privacy Risks of KV-cache in LLM Inference" (NDSS 2026)

Span Queries: What if we had a way to plan and optimize GenAI like we do for SQL?

A TurboQuant implementation with Llama.cpp for AMD with Vulkan runtime

inference cookbook / inference 框架原理解析

C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

KV Cache with PagedAttention vs PagedAttention + TurboQuant - experiments across token sizes comparing memory, latency, and accuracy.

RestoreKV: Recovering full-cache behavior under aggressive query-agnostic KV cache eviction.