Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderbo

The question arrived quietly, as the best ones usually do. On April 25th, a research paper surfaced on ArXiv asking something the AI benchmarking world has been carefully stepping around for years: who actually decides what "best" means when we rank large language models? The paper — "Who Defines 'Best'? Towards Interactive, User-Defined Evaluation of LLM Leaderboards" — trains its attention not on the models themselves but on the infrastructure of credibility surrounding them. Chatbot Arena rankings, MMLU scores, and similar leaderboards have quietly accumulated enormous power over which models get adopted, funded, and trusted. The researchers argue that this power is being exercised by a surprisingly narrow group — benchmark designers, platform operators, a handful of academic consensus-setters — while the actual users of these models, with their wildly different needs and contexts, have almost no meaningful input. The proposed alternative is an interactive, user-defined evaluation framework: one where individuals can weight criteria according to their own priorities rather than accept someone else's definition of excellence. A lawyer evaluating contract drafting speed cares about nothing the same things a novelist cares about, which makes a single ranked list not just imprecise but quietly misleading.

If confirmed, here is what this means. The entire scaffolding of AI credibility — the leaderboards that influence enterprise procurement decisions, the benchmarks that shape research funding, the rankings that determine which startups attract capital — rests on assumptions that nobody voted for. A shift toward user-defined evaluation frameworks would not merely change how we measure models; it would redistribute authority over what AI is supposed to be good at. Enterprise buyers gain leverage. Niche use cases gain visibility. The models optimised for benchmark performance rather than real-world utility lose their artificial advantage. The second-order effect is more interesting still: if evaluation becomes personalised, so does accountability. It becomes harder for a lab to claim its model is "best" without specifying best for whom, under what conditions, weighted by whose priorities. That discipline, applied consistently, would force a more honest conversation across the whole industry.

Watch for whether this framework attracts implementation interest from any major benchmarking organisations — Hugging Face, LMSYS, or similar — or whether it stays an academic proposal. Adoption signals would change this from a paper to a movement.

NewsHive monitors these sources continuously. All signal titles above link to the original reporting.

Intelligence by NewsHive. Need help navigating what this means for your business? Contact GeekyBee →