Smaller gguf getting way less tokens per second?? So confused!

On May 5th, someone running local models hit a wall that shouldn't exist. The complaint was specific: a smaller GGUF quantisation — a more compressed, lighter version of an underlying model, designed to trade a little quality for lower memory overhead and faster throughput — was generating tokens more slowly than the larger quantisation of the same model. That is a direct contradiction of the premise. The whole reason you reach for a smaller quant is speed and accessibility. You accept the quality compromise precisely because you expect the performance gain. When the smaller file runs slower, the implicit contract of quantisation breaks, and the person at the keyboard is left staring at numbers that refuse to make sense. The post captured that specific bewilderment well enough to gather a modest audience, which suggests this isn't an isolated configuration error on one machine — it's a scenario recognisable enough that others stopped scrolling to look.

If confirmed, here is what this means. Users optimising local inference pipelines around quantisation size as a reliable proxy for speed are working from a flawed assumption, at least in some hardware and software configurations. The second-order effect is real: anyone who has automated model selection, built latency targets around quant tiers, or advised others to drop to a smaller GGUF for faster responses may be giving subtly wrong guidance. The cause is almost certainly a layer-count or context-length interaction — smaller quants can alter how inference engines batch and schedule operations, particularly if the reduction tips computation across a hardware threshold like a GPU VRAM boundary, forcing CPU offloading. That one transition can cost far more than the compression saves.

Watch for follow-up posts on LocalLLaMA that either reproduce the behaviour across different hardware configurations or identify the specific runtime version and model family where this appears. A reproducible test case from a second user would transform this from a single frustrating data point into a genuine inference tooling issue worth tracking.

NewsHive monitors these sources continuously. All signal titles above link to the original reporting.

Intelligence by NewsHive. Need help navigating what this means for your business? Contact GeekyBee →