Quantization Guidance

<!-- SC_OFF --><div class="md"><p>which model quantization technique is best suitable for smartphones at this point...specially if the model is finetuned as that tends to amplify

<!-- SC_OFF --><div class="md"><p>What's the difference between FP8 or INT8 ? For nvidia you would go FP8 but on ampere you would rely on INT8. On the other side new intel gpu

<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1s5cggh/google_turboquant_blew_up_for_kv_cache_heres/"> <img alt="Google TurboQuant blew up for KV cache.

<!-- SC_OFF --><div class="md"><p>Been running a few models locally at different quant levels and honestly the jump from Q5 to Q4 sometimes feels like nothing and other times it

<a href="https://news.ycombinator.com/item?id=47519295">Comments</a>

<!-- SC_OFF --><div class="md"><p>I applied video compression to LLM inference and got <strong>10,000x less quantization error at the same storage cost</strong></p> <p><a

<p>Ask how much memory it takes to fine-tune a 7B model and the instinct is "the model's 14 GB in fp16, so a bit more than that". The real figure is about 112 GB, before you've

<!-- SC_OFF --><div class="md"><p>I’m planning to deploy the 9B and 27B parameter models using vLLM and was wondering if anyone has done some thorough testing on the non-GGUF