Anyone else tired of deploying models just to test ideas?

<!-- SC_OFF --><div class="md"><p>I’ve been exploring how to build a fully on-prem document AI pipeline for handling confidential data — no cloud APIs, no external processing.</p>

<!-- SC_OFF --><div class="md"><p>Most of my LLM cost was on the wrong tier of work.</p> <p>Classification, extraction, JSON formatting, summarization I'm going to review anyway.

<!-- SC_OFF --><div class="md"><p>One thing we’ve been noticing lately is that a surprisingly large percentage of day-to-day AI workflows no longer seem to require frontier-scale

<!-- SC_OFF --><div class="md"><p>doing infrastructure evaluation for inference workloads and running into the same problem everywhere: every platform publishes p50 cold start

<!-- SC_OFF --><div class="md"><p>I’m trying to understand the current open-source LLM landscape beyond surface-level hype.</p> <p>We all got used to the nerfed products of

<!-- SC_OFF --><div class="md"><p>Just putting together a quick list of the new <strong>open-source physical AI / robotics models</strong> from <strong>NVIDIA GTC

<!-- SC_OFF --><div class="md"><p>Running Llama-3-70B with long contexts and hitting HBM limits constantly. 128K context burns approx 40GB just for KV cache — half the H100 gone

<!-- SC_OFF --><div class="md"><p>I have experience using llama.cpp on Windows/Linux with 8GB NVIDIA card (384 GB/s bandwidth) and offloading to CPU to run MoE models. I typically