Benchmark^2: Systematic Evaluation of LLM Benchmarks
The rapid proliferation of benchmarks for evaluating large language models (LLMs) has created an urgent need for systematic methods to assess benchmark quality itself. We propose Benchmark^2, a comprehensive framework comprising three complementary metrics: (1) Cross-Benchmark Ranking Consistency, measuring whether a benchmark produces model rankings aligned with peer benchmarks; (2) Discriminability Score, quantifying a benchmark's ability to differentiate between models; and (3) Capability Alignment Deviation, identifying problematic instances where stronger models fail but weaker models succeed within the same model family. We conduct extensive experiments across 15 benchmarks spanning mathematics, reasoning, and knowledge domains, evaluating 11 LLMs across four model families. Our analysis reveals significant quality variations among existing benchmarks and demonstrates that selective benchmark construction based on our metrics can achieve comparable evaluation performance with substantially reduced test sets.
695f290d5fa3847525c41d7d
Qi Qian
695f290d5fa3847525c41d7e
Chengsong Huang
claimed_verified
2026-01-08T08:31:58.524Z
62ea79dd01ed9b0e8f61ccd3
/avatars/70af83e0e267be39fcd5f23b85e2dafa.svg
Chengsong Huang
ChengsongHuang
user
ChengsongHuang
695f290d5fa3847525c41d7f
Jingwen Xu
695f290d5fa3847525c41d80
Changze Lv
claimed_verified
2026-01-29T13:56:45.203Z
60efa4da8432bc401cd0abc6
/avatars/3c8d8db9aaa5bd5fd1f870ac0a6b655a.svg
Changze Lv
fdu-lcz
user
fdu-lcz
695f290d5fa3847525c41d81
Muling Wu
695f290d5fa3847525c41d82
Wenhao Liu
695f290d5fa3847525c41d83
Xiaohua Wang
695f290d5fa3847525c41d84
Zhenghua Wang
695f290d5fa3847525c41d85
Zisu Huang
claimed_verified
2026-01-12T10:36:25.406Z
662b6a8f0b7f23f3c000559e
/avatars/0a5b4e09ac9a8e40342131319ff32b29.svg
Zisu Huang
zisuh
user
zisuh
695f290d5fa3847525c41d86
Muzhao Tian
695f290d5fa3847525c41d87
Jianhan Xu
695f290d5fa3847525c41d88
Kun Hu
695f290d5fa3847525c41d89
He-Da Wang
695f290d5fa3847525c41d8a
Yao Hu
695f290d5fa3847525c41d8b
Xuanjing Huang
695f290d5fa3847525c41d8c
Xiaoqing Zheng
Benchmark^2: Systematic Evaluation of LLM Benchmarks
The rapid proliferation of benchmarks for evaluating large language models (LLMs) has created an urgent need for systematic methods to assess benchmark quality itself. We propose Benchmark^2, a comprehensive framework comprising three complementary metrics: (1) Cross-Benchmark Ranking Consistency, measuring whether a benchmark produces model rankings aligned with peer benchmarks; (2) Discriminability Score, quantifying a benchmark's ability to differentiate between models; and (3) Capability Alignment Deviation, identifying problematic instances where stronger models fail but weaker models succeed within the same model family. We conduct extensive experiments across 15 benchmarks spanning mathematics, reasoning, and knowledge domains, evaluating 11 LLMs across four model families. Our analysis reveals significant quality variations among existing benchmarks and demonstrates that selective benchmark construction based on our metrics can achieve comparable evaluation performance with substantially reduced test sets.
Researchers developed Benchmark^2, a framework with three metrics to evaluate benchmark quality for large language models, revealing significant variations in existing benchmarks and enabling more efficient evaluation through selective benchmark construction.
Benchmark^2 cross-benchmark ranking consistency discriminability score capability alignment deviation large language models benchmarks model rankings evaluation performance test sets