Gemma-4-E2B-it on iPhone (memory bottleneck)

<!-- SC_OFF --><div class="md"><p>I’ve been experimenting with running LLMs fully on-device, and managed to get Gemma 4 running locally on an iPhone 13 Pro.</p> <p>This is built

<!-- SC_OFF --><div class="md"><p><a href="https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF">https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF</a></p> <p><a

<!-- SC_OFF --><div class="md"><p>question, I'm using it for text translations and on each large prompt (20K+) it stops with a remark 'now I'm going to put that to the file' or

<!-- SC_OFF --><div class="md"><p>Seems like ive seen some people get okay enough token generation speeds on just system memory at ~ 10 t/s with some larger models i am interested

<!-- SC_OFF --><div class="md"><p>Running Llama-3-70B with long contexts and hitting HBM limits constantly. 128K context burns approx 40GB just for KV cache — half the H100 gone

<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1s3i83m/fully_local_voice_ai_on_iphone/"> <img alt="Fully local voice AI on iPhone"