tierKV: A Distributed KV Cache That Makes Evicted Blocks Faster to Restore Than

<h1> Complete Guide to llm-d CNCF Sandbox — Kubernetes-Native Distributed LLM Inference Framework </h1> <p>At KubeCon Europe 2026 in Amsterdam, IBM Research, Red Hat, and Google

arXiv:2604.05887v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) have advanced unified reasoning over text, images, and videos, but their inference is

<p>Prefix caching is one of the biggest cost levers in LLM serving. vLLM, SGLang, TGI, and most hosted providers all do some version of it: during prefill they compute a key-value

<ul> <li> <strong>Book:</strong> <a href="https://www.amazon.com/dp/B0GX2SQ594" rel="noopener noreferrer">System Design Pocket Guide: Interviews</a> </li> <li> <strong>Also by

<blockquote> <p><em>Originally published at <a href="https://llmkube.com/blog/turboquant-m5-max-quality-and-asymmetric" rel="noopener

Running large language model inference at scale forces a KV cache trade-off: oversized GPU instances or slow time-to-first-token. This post builds a tiered KV cache on Amazon

<h2> 600 MiB of headroom </h2> <p>My RTX 4070 was running Qwen 35B beautifully after the <code>--cpu-moe</code> trick from a previous run. The tokens/sec were where I wanted them.

arXiv:2608.23834v1 Announce Type: new Abstract: The key-value (KV) cache is a primary capacity and bandwidth bottleneck in long-context LLM serving. We present Minima-KV, a