at what point does quantization stop being a tradeoff and start being actual qua

<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1s5cggh/google_turboquant_blew_up_for_kv_cache_heres/"> <img alt="Google TurboQuant blew up for KV cache.

<!-- SC_OFF --><div class="md"><p>I applied video compression to LLM inference and got <strong>10,000x less quantization error at the same storage cost</strong></p> <p><a

<p>Building a production-ready, fault-tolerant Retrieval-Augmented Generation system is an exercise in managing harsh tradeoffs. You want massive context, lightning-fast hybrid

<!-- SC_OFF --><div class="md"><p>Running Llama-3-70B with long contexts and hitting HBM limits constantly. 128K context burns approx 40GB just for KV cache — half the H100 gone

<!-- SC_OFF --><div class="md"><p>What's the difference between FP8 or INT8 ? For nvidia you would go FP8 but on ampere you would rely on INT8. On the other side new intel gpu

<!-- SC_OFF --><div class="md"><p>which model quantization technique is best suitable for smartphones at this point...specially if the model is finetuned as that tends to amplify

<!-- SC_OFF --><div class="md"><blockquote> <p>Wait, I need to be careful about the &quot;no_think&quot; tag in the system prompt. The system prompt says <code>/no_think</code>.

<!-- SC_OFF --><div class="md"><p>I’m planning to deploy the 9B and 27B parameter models using vLLM and was wondering if anyone has done some thorough testing on the non-GGUF