Rating the Raters: Rasch Measurement Theory for LLM Evaluation

arXiv:2605.04454v1 Announce Type: new Abstract: Alignment evaluation in machine learning has largely become evaluation of models. Influential benchmarks score model outputs under

arXiv:2603.25133v1 Announce Type: new Abstract: Rubric-based evaluation has become a prevailing paradigm for evaluating instruction following in large language models (LLMs).

arXiv:2608.24314v1 Announce Type: new Abstract: Evaluating conversational voice agents at scale re- quires reliable assessment methods that capture both observ- able interaction

arXiv:2608.06609v1 Announce Type: new Abstract: Automated item evaluation (AIE) refers to the use of computational methods to assess item quality without requiring manual expert

arXiv:2510.08663v3 Announce Type: replace-cross Abstract: Psychological assessments commonly rely on rating-scale items, which require respondents to condense complex experiences

arXiv:2604.00477v1 Announce Type: new Abstract: LLM-based agent judges are an emerging approach to evaluating conversational AI, yet a fundamental uncertainty remains: can we

arXiv:2605.08445v1 Announce Type: new Abstract: AI models are increasingly deployed in live clinical environments where they must perform reliably across complex, high-stakes

arXiv:2605.11496v1 Announce Type: new Abstract: Recent published evidence from frontier laboratories shows that contemporary AI models can recognise evaluation contexts, latently