KVQuant: Run 70B LLMs on 8GB RAM with 4-bit KV Cache Quantization

I compressed GPT-2 to run on an Arduino! Here's how I did it with KVQuant. The Problem: LLMs need huge memory for key-value caches during inference. The Solution: 4-bit KV cache quantization that reduces memory 4x with <1% accuracy loss. Results: GPT-2: 512MB → 128MB (4x reduction) LLaMA-7B: 8GB → 2