low-bit-quantization

llama.cpp fork with TurboQuant WHT-rotated KV cache & weight compression + Gemma 4 MTP and Qwen 3.6 NextN speculative decoding (+30-50% throughput).

[MobiSys 2026] On-device ultra-low-bit LLM inference with LUT.

BGEMM-CUDA is a CUDA-based low-bit GEMM kernel library for efficient neural network inference. It implements optimized binary and ternary matrix multiplication primitives, including binary-weight and ternary-activation computation, with PyTorch extension support for model-level integration.

Official implementation of P4Q: Learning to Prompt for Quantization in Low-Bit CLIP (ACM TOMM).

Low-bit LLM inference engine in Rust + CUDA. Converts models to a 4-bit .wstone format (3.75x smaller) and runs them on consumer GPUs. Built for memory-bandwidth-bound decode on Turing, where bytes per weight — not TOPS — sets token throughput.

RWKV-7 FP8 quantized inference - 6.4x decode speedup on Blackwell GPUs with <0.3% accuracy loss. Full FP8 E4M3 weight quantization with fused Triton kernels.

MP-Sparse-Attn provides Triton kernels for Diagonal-Tiled Mixed-Precision Attention, targeting efficient low-bit MXFP inference for Transformer models. It combines tile-level mixed-precision computation and kernel fusion to accelerate attention on modern GPUs.

TurboQuant KV-cache compression (2/3/4-bit WHT + PolarQuant) and TQ weight quantization for llama.cpp, kept in sync with upstream.

experimenting with binary brain(C++ framework for training Binary Neural Networks)

TaQuants protects you from model collapse.

Final Year Research & Engineering Project — A hardware-aware compression pipeline that profiles layer-by-layer CPU execution metrics. It balances model perplexity against physical latency, eliminates microarchitectural overhead, and compiles optimized mixed-precision graphs into deployable .gguf format.

Deploying logic gate networks on FPGAs for faster Edge AI Inference[Implementation of Differentiable Logic Gate Networks (NeurIPS 2024)]