Best quantization techniques for smartphones

<!-- SC_OFF --><div class="md"><p>What's the difference between FP8 or INT8 ? For nvidia you would go FP8 but on ampere you would rely on INT8. On the other side new intel gpu

<!-- SC_OFF --><div class="md"><p>Can anyone guide me generally on how to make your own quantized versions of models? </p> </div><!-- SC_ON --> &#32; submitted by &#32; <a

<!-- SC_OFF --><div class="md"><p>Been running a few models locally at different quant levels and honestly the jump from Q5 to Q4 sometimes feels like nothing and other times it

<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1s5cggh/google_turboquant_blew_up_for_kv_cache_heres/"> <img alt="Google TurboQuant blew up for KV cache.

<!-- SC_OFF --><div class="md"><p>I applied video compression to LLM inference and got <strong>10,000x less quantization error at the same storage cost</strong></p> <p><a

<!-- SC_OFF --><div class="md"><p>I’m planning to deploy the 9B and 27B parameter models using vLLM and was wondering if anyone has done some thorough testing on the non-GGUF

<!-- SC_OFF --><div class="md"><p>Can someone ELI5? We've been using the same methods on both model and cache for a while (Q4_0/1, etc). </p> </div><!-- SC_ON --> &#32; submitted

<p>Ask how much memory it takes to fine-tune a 7B model and the instinct is "the model's 14 GB in fp16, so a bit more than that". The real figure is about 112 GB, before you've