Kimi Linear: An Expressive, Efficient Attention Architecture

<a href="https://news.ycombinator.com/item?id=47674749">Comments</a>

<h1> Kimi Linear: How Moonshot AI Built a Hybrid Attention Architecture That Beats Full Attention </h1> <p>The standard Transformer attention mechanism has a well-known problem:

arXiv:2603.22300v1 Announce Type: cross Abstract: Scaling Transformers to ultra-long contexts is bottlenecked by the $O(n^2 d)$ cost of self-attention. Existing methods reduce

<a href="https://news.ycombinator.com/item?id=49101852">Comments</a>

<a href="https://news.ycombinator.com/item?id=48981703">Comments</a>

<!-- SC_OFF --><div class="md"><p><em>Scaling language models to handle longer input sequences typically necessitates large key-value (KV) caches, resulting in substantial memory

<p>Every transformer you've ever used stacks layers with a dead-simple formula: take the input, add the layer's output, move on. <code>x + layer(x)</code>. Fixed weight of 1. No