LLM Performance on Coding Tasks
On 12 May, Dev.to published results from testing 10 previously untested LLMs on agent coding tasks. On 26 May, a second 10-model test reported scores ranging from 81.6% to 10% and described the free tier as “a full-on gamble.” By 30 September, testing had moved to a deceptively awkward edge case: 1:30 AM occurs twice on 1 November, and most AI models picked one instance and moved on.
Related Dev.to coverage compared Claude Opus 4.7 with 4.6. It reported another test in which five new model families debuted and none scored below 75%, and examined how data alignment limits LLM generalization through a “Benchmark Shadows” study. Dev.to also reported that open-weight models have caught up with GPT, Gemini and Claude. Separate coverage addressed domain-specific language models.
The wider evaluation picture remains untidy. LocalLLaMA said its LoCoMo audit found that 6.4% of the answer key was wrong and that the judge accepted up to 63% of intentionally wrong answers. LocalLLaMA also discussed Qwen3.8 27B running on one R9700 with 262K context, half a million tokens of reusable cache and roughly 180 tokens per second, including questions about its “3-bit” description. ArtificialInteligence offered an opinion on TurboQuant in practice.
Taking this as factual, impact is 10%. Developers choosing coding agents, free-tier models and open-weight systems face inconsistent performance across tests. Edge cases such as duplicated clock times expose failures that aggregate scores can miss. Benchmark designers also have reason to inspect answer keys and automated judges, since a faulty evaluation can make a model look clever for agreeing with the paperwork.
Watch for independent reproductions that publish prompts, scoring methods, model versions and test code. Further results on agent coding and ambiguous time handling could show whether the reported spread reflects durable model differences or problems with the benchmarks.
<!-- SC_OFF --><div class="md"><p>Last year researchers affiliated with NVIDIA, University of Warsaw, and University of Edinburgh published <a
<blockquote> <p>A controlled study finds that data distribution, not just volume, dictates LLM capability. Benchmark-aligned training inflates scores but creates narrow, brittle
<p>The price page hasn't moved. Five dollars per million input. Twenty-five per million output. Identical to Opus 4.6.</p> <p>Run the same Claude Code call on both, though, and
<!-- SC_OFF --><div class="md"><p>I tested two llama.cpp builds on the same <strong>Qwen3.6-27B-NVFP4</strong> model.</p> <p><code>llama-bench</code> reports the model label as
<!-- SC_OFF --><div class="md"><p>Projects are <a href="https://github.com/snap-research/locomo/issues/31">still submitting new scores on LoCoMo as of March 2026.</a> but the
<!-- SC_OFF --><div class="md"><p>There are plenty of "bro trust me, this model is better for coding" discussions out there. I wanted to replace the vibes with actual
<p>In the first eight weeks of 2026, ten major open-weight LLM architectures were released.</p> <p>GLM-5 matched GPT-5.2 and Claude Opus 4.6 on benchmarks. Step 3.5 Flash
<ul> <li> <strong>Book:</strong> <a href="https://www.amazon.com/dp/B0GX2SQ594" rel="noopener noreferrer">System Design Pocket Guide: Interviews</a> </li> <li> <strong>Also by
NewsHive monitors these sources continuously. All signal titles above link to the original reporting.
Intelligence by NewsHive. Need help navigating what this means for your business? Contact GeekyBee →