FPGAs for speculative decoding

<!-- SC_OFF --><div class="md"><p>I’ve been working on an educational implementation repo for speculative decoding:</p> <p><a

<!-- SC_OFF --><div class="md"><p>Running Llama-3-70B with long contexts and hitting HBM limits constantly. 128K context burns approx 40GB just for KV cache — half the H100 gone

<!-- SC_OFF --><div class="md"><p>Got Qwen 3.5 27B to 1,103,941 tok/s on 12 nodes with 96 B200 GPUs.</p> <p>At that rate you process 50,000 insurance policy documents in hours

<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1t4uj9h/mtp_on_strix_halo_with_llamacpp_pr_22673/"> <img alt="MTP on strix halo with llama.cpp (PR #22673)"

<!-- SC_OFF --><div class="md"><p>Every output token the model generates requires reading the entire KV cache from High Bandwidth Memory.</p> <p>Here's what that actually looks

<!-- SC_OFF --><div class="md"><p>So i recently updated LMstudio after a long pause and updated my llama.cpp runtimes too.. i was shocked.. i thought maybe something like

<!-- SC_OFF --><div class="md"><p><a

<!-- SC_OFF --><div class="md"><p>Wrote a deep dive on <strong>FlashAttention-4 (03/05/2026)</strong> that's relevant for anyone thinking about inference performance.</p>