RubricBench: Aligning Model-Generated Rubrics with Human Standards
As Large Language Model (LLM) alignment evolves from simple completions to complex, highly sophisticated generation, Reward Models are increasingly shifting toward rubric-guided evaluation to mitigate surface-level biases. However, the community lacks a unified benchmark to assess this evaluation paradigm, as existing benchmarks lack both the discriminative complexity and the ground-truth rubric annotations required for rigorous analysis. To bridge this gap, we introduce RubricBench, a curated benchmark with 1,147 pairwise comparisons specifically designed to assess the reliability of rubric-based evaluation. Our construction employs a multi-dimensional filtration pipeline to target hard samples featuring nuanced input complexity and misleading surface bias, augmenting each with expert-annotated, atomic rubrics derived strictly from instructions. Comprehensive experiments reveal a substantial capability gap between human-annotated and model-generated rubrics, indicating that even state-of-the-art models struggle to autonomously specify valid evaluation criteria, lagging considerably behind human-guided performance.
69a65c9e663a122a7ce66716
Qiyuan Zhang
claimed_verified
2026-03-03T10:38:38.663Z
62a42f22c683d02f5b63320c
/avatars/bc611abe9c4ef8d378123cb8ac9fdbf2.svg
Qiyuan Zhang
DonJoey
user
DonJoey
69a65c9e663a122a7ce66717
Junyi Zhou
69a65c9e663a122a7ce66718
Yufei Wang
69a65c9e663a122a7ce66719
Fuyuan Lyu
69a65c9e663a122a7ce6671a
Yidong Ming
69a65c9e663a122a7ce6671b
Can Xu
69a65c9e663a122a7ce6671c
Qingfeng Sun
69a65c9e663a122a7ce6671d
Kai Zheng
69a65c9e663a122a7ce6671e
Peng Kang
69a65c9e663a122a7ce6671f
Xue Liu
69a65c9e663a122a7ce66720
Chen Ma
RubricBench: Aligning Model-Generated Rubrics with Human Standards
As Large Language Model (LLM) alignment evolves from simple completions to complex, highly sophisticated generation, Reward Models are increasingly shifting toward rubric-guided evaluation to mitigate surface-level biases. However, the community lacks a unified benchmark to assess this evaluation paradigm, as existing benchmarks lack both the discriminative complexity and the ground-truth rubric annotations required for rigorous analysis. To bridge this gap, we introduce RubricBench, a curated benchmark with 1,147 pairwise comparisons specifically designed to assess the reliability of rubric-based evaluation. Our construction employs a multi-dimensional filtration pipeline to target hard samples featuring nuanced input complexity and misleading surface bias, augmenting each with expert-annotated, atomic rubrics derived strictly from instructions. Comprehensive experiments reveal a substantial capability gap between human-annotated and model-generated rubrics, indicating that even state-of-the-art models struggle to autonomously specify valid evaluation criteria, lagging considerably behind human-guided performance.
RubricBench is introduced as a benchmark for evaluating rubric-guided reward models in large language model alignment, addressing the lack of discriminative complexity and ground-truth annotations in existing benchmarks.
Reward Models rubric-guided evaluation Large Language Models benchmark pairwise comparisons atomic rubrics multi-dimensional filtration pipeline surface-level biases discriminative complexity