I solved kv-cache

<!-- SC_OFF --><div class="md"><p>Apparently you can use a SmartSSD which is basically an FPGA + Flash memory (or in theory cheap ram sticks) to offload/evict your KV cache off

<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1v5wviz/dkv_opensource_kvcache_compression_framework_for/"> <img alt="DKV: Open-source KV-cache compression

<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1s1f8sq/designed_a_photonic_chip_for_o1_kv_cache_block/"> <img alt="Designed a photonic chip for O(1) KV

<!-- SC_OFF --><div class="md"><p>Running Llama-3-70B with long contexts and hitting HBM limits constantly. 128K context burns approx 40GB just for KV cache — half the H100 gone

<!-- SC_OFF --><div class="md"><p>Every output token the model generates requires reading the entire KV cache from High Bandwidth Memory.</p> <p>Here's what that actually looks

<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1s5cggh/google_turboquant_blew_up_for_kv_cache_heres/"> <img alt="Google TurboQuant blew up for KV cache.

arXiv:2604.05887v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) have advanced unified reasoning over text, images, and videos, but their inference is

<!-- SC_OFF --><div class="md"><p><a href="https://x.com/i/status/2036533564158910740">https://x.com/i/status/2036533564158910740</a></p> </div><!-- SC_ON --> &#32; submitted by