MTP Inference Performance Optimization
Something is moving quietly in the inference optimization space, and it started on May 5th. A project called MTPLX surfaced on LocalLLaMA with a specific, verifiable claim: 2.24x faster tokens per second on Apple Silicon, built around a native Multi-Token Prediction inference engine. No marketing copy, no product launch — just a benchmark figure and a GitHub link. The kind of post that either gets buried or gets noticed by exactly the right people. It got noticed. The following day, a second signal appeared: Qwen3.6-27B running with MTP grafted onto Unsloth's UD XL format, pulling 2.5x throughput improvement through an unmerged pull request against llama.cpp. Two posts in two days, from different angles, converging on the same underlying idea — that Multi-Token Prediction isn't just a training-time trick but a practical inference accelerator that works on hardware people actually own.
If confirmed, here is what this means. The numbers being discussed — 2.24x and 2.5x throughput — are not incremental. They represent the difference between a model that fits usably into a workflow and one that doesn't. For anyone running local inference on Apple Silicon, or building on top of llama.cpp, this is the kind of performance headroom that changes what's practically deployable. The second-order effect matters more: if MTP can be "grafted" onto existing models without retraining, as the Unsloth signal suggests, then this becomes a retrofit rather than a rebuild. That shifts the timeline considerably. Developers don't need to wait for new model releases — they can potentially apply this to Qwen3, and by implication other architectures, right now. The strategic weight falls on llama.cpp, which remains the connective tissue of the local inference ecosystem. An unmerged PR carrying this kind of gain will not stay unmerged quietly.
Watch for the llama.cpp pull request to either land or generate a formal review thread — that outcome will tell you whether this is a reproducible, generalizable technique or an edge case that doesn't survive contact with the broader codebase.
<!-- SC_OFF --><div class="md"><p>Every output token the model generates requires reading the entire KV cache from High Bandwidth Memory.</p> <p>Here's what that actually looks
<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1t4uj9h/mtp_on_strix_halo_with_llamacpp_pr_22673/"> <img alt="MTP on strix halo with llama.cpp (PR #22673)"
<!-- SC_OFF --><div class="md"><p>That foodtruck bench post showing deepseek v4 matching gpt-5.2 at 17x cheaper got me thinking. if frontier cloud models are that overpriced for
<!-- SC_OFF --><div class="md"><p>PR #22673 (commit 4f13cb7) landed MTP speculative decoding in mainline llama.cpp on May 16. I tested it on two separate rigs.</p> <p>Qwen3.6 27B,
<!-- SC_OFF --><div class="md"><p>Last year researchers affiliated with NVIDIA, University of Warsaw, and University of Edinburgh published <a
<!-- SC_OFF --><div class="md"><p>Got Qwen 3.5 27B to 1,103,941 tok/s on 12 nodes with 96 B200 GPUs.</p> <p>At that rate you process 50,000 insurance policy documents in hours
<!-- SC_OFF --><div class="md"><p>DeepSeek dropped the full V4 paper this week. preview from april was 58 pages, this version adds a lot of technical depth.</p> <p>What stood out
<!-- SC_OFF --><div class="md"><p>I tested two llama.cpp builds on the same <strong>Qwen3.6-27B-NVFP4</strong> model.</p> <p><code>llama-bench</code> reports the model label as
NewsHive monitors these sources continuously. All signal titles above link to the original reporting.
Intelligence by NewsHive. Need help navigating what this means for your business? Contact GeekyBee →