ARC AGI Scoring Methodology Changes

In late March, a quiet observation started circulating: ARC-AGI-3, the latest iteration of the benchmark designed to measure progress toward general intelligence, was apparently not scoring results the same way as its predecessors. Not a minor methodological tweak, but a fundamental change in how performance gets calculated — which matters enormously when the entire point of the benchmark is to track progress over time. If the ruler changes shape between measurements, the measurements stop being comparable. Then, in mid-April, a more pointed detail emerged. Anthropic, announcing their Claude model through what was described as a "Mythos" release, quietly omitted any mention of ARC-AGI-3 performance. The suggested reason: a score of 0.3% is not exactly the kind of number you lead with when you are trying to convey that your model is powerful enough to require careful handling before public release. That combination — a changed scoring methodology and a conspicuously absent benchmark result — is the kind of detail that tends to mean something, even if we cannot yet say precisely what.

If confirmed, here is what this means. ARC-AGI was always a fragile instrument, valuable precisely because it resisted the benchmark-gaming that inflated scores on every other test. Changing the methodology without loud, transparent disclosure would undermine the one thing that gave the benchmark credibility: continuity. Scores from frontier labs would become incomparable across versions, making it easier for any model to claim progress without actually demonstrating it. The 0.3% figure for what is presumably a state-of-the-art Anthropic model also suggests ARC-AGI-3 is genuinely harder — which is worth knowing — but only if the scoring change is properly documented. The deeper strategic implication is this: if labs can selectively disclose benchmark results and the benchmarks themselves keep shifting, the entire apparatus of external evaluation starts to feel like theatre.

Watch for ARC Prize's official documentation comparing the scoring methodology across all three versions — any silence or vagueness there would itself be telling. Also watch for whether other frontier labs publish ARC-AGI-3 scores, and how loudly they do or do not.

NewsHive monitors these sources continuously. All signal titles above link to the original reporting.

Intelligence by NewsHive. Need help navigating what this means for your business? Contact GeekyBee →