KV Cache Quantization: I Stretched Qwen 35B's Context 8 on 12GB VRAM

<!-- SC_OFF --><div class="md"><p>Last year researchers affiliated with NVIDIA, University of Warsaw, and University of Edinburgh published <a

<blockquote> <p><em>Originally published at <a href="https://llmkube.com/blog/turboquant-m5-max-quality-and-asymmetric" rel="noopener

<p> </p> <p>Four VRAM numbers that looked wrong on an 8GB laptop, none of them a bug: what dxdiag actually<br /> reports, what the desktop holds back before a model even loads,

<!-- SC_OFF --><div class="md"><p>I applied video compression to LLM inference and got <strong>10,000x less quantization error at the same storage cost</strong></p> <p><a

<!-- SC_OFF --><div class="md"><p>Every output token the model generates requires reading the entire KV cache from High Bandwidth Memory.</p> <p>Here's what that actually looks

<p>Ask how much memory it takes to fine-tune a 7B model and the instinct is "the model's 14 GB in fp16, so a bit more than that". The real figure is about 112 GB, before you've

<ul> <li> <strong>Book:</strong> <a href="https://www.amazon.com/dp/B0GX2SQ594" rel="noopener noreferrer">System Design Pocket Guide: Interviews</a> </li> <li> <strong>Also by

<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1s5cggh/google_turboquant_blew_up_for_kv_cache_heres/"> <img alt="Google TurboQuant blew up for KV cache.