KVQuant: Run 70B LLMs on 8GB RAM with Real-Time KV Cache Compression

I built KVQuant because I wanted to run 70B parameter models on my gaming laptop. The problem? Even with 4-bit quantization, a 128K context window needs 256GB RAM just for the KV cache. When you run an LLM, the memory bottleneck is not the model weights - it is the KV cache. Model Weights (4-bit) KV