BenchMIRT is a new study that scrutinizes what LLM benchmarks truly measure. The paper applies multidimensional item response theory to popular LLM tests, revealing that many scores reflect dataset familiarity rather than genuine reasoning or knowledge. These findings urge developers to rethink benchmark design and to focus on metrics that capture real-world reasoning capabilities.
Opening Kapyn…