Run a 1T parameter model on a 32gb Mac by streaming tensors from NVMe

<a href="https://news.ycombinator.com/item?id=47476422">Comments</a>

<a href="https://news.ycombinator.com/item?id=49166984">Comments</a>

<a href="https://news.ycombinator.com/item?id=47689174">Comments</a>

<!-- SC_OFF --><div class="md"><p>Every output token the model generates requires reading the entire KV cache from High Bandwidth Memory.</p> <p>Here's what that actually looks

<!-- SC_OFF --><div class="md"><p>Hey everyone, I’m currently an ML Engineer looking to pick up iOS development, and I’m upgrading my hardware to handle both. I’m moving away from

<!-- SC_OFF --><div class="md"><p>I could fit the Q3 model not sure if it's worth over 27B ? </p> </div><!-- SC_ON --> &#32; submitted by &#32; <a