INT8 vs FP8 quantization

<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1s5cggh/google_turboquant_blew_up_for_kv_cache_heres/"> <img alt="Google TurboQuant blew up for KV cache.

<!-- SC_OFF --><div class="md"><p>which model quantization technique is best suitable for smartphones at this point...specially if the model is finetuned as that tends to amplify

<!-- SC_OFF --><div class="md"><p>I applied video compression to LLM inference and got <strong>10,000x less quantization error at the same storage cost</strong></p> <p><a

<!-- SC_OFF --><div class="md"><p>Been running a few models locally at different quant levels and honestly the jump from Q5 to Q4 sometimes feels like nothing and other times it

<!-- SC_OFF --><div class="md"><p>Can anyone guide me generally on how to make your own quantized versions of models? </p> </div><!-- SC_ON --> &#32; submitted by &#32; <a

<!-- SC_OFF --><div class="md"><p>I’m planning to deploy the 9B and 27B parameter models using vLLM and was wondering if anyone has done some thorough testing on the non-GGUF

<!-- SC_OFF --><div class="md"><p>I see this kind of approach quite often with local AI models. For example, when optimizing models for an RTX 3090, people sometimes convert them

<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1s5qdf6/qwopus_v2_nvfp4_quantization/"> <img alt="Qwopus v2 nvfp4 quantization"