Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation
As Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks, a pivotal challenge persists: evaluating their proficiency in complex, open-ended tasks characterizing genuine expert-level cognition. Existing frameworks suffer from narrow domain coverage, reliance on generalist tasks, or self-evaluation biases. To bridge this gap, we present XpertBench, a high-fidelity benchmark engineered to assess LLMs across authentic professional domains. XpertBench consists of 1,346 meticulously curated tasks across 80 categories, spanning finance, healthcare, legal services, education, and dual-track research (STEM and Humanities). These tasks are derived from over 1,000 submissions by domain experts--including researchers from elite institutions and practitioners with extensive clinical or industrial experience--ensuring superior ecological validity. Each task uses detailed rubrics with mostly 15-40 weighted checkpoints to assess professional rigor. To facilitate scalable yet human-aligned assessment, we introduce ShotJudge, a novel evaluation paradigm that employs LLM judges calibrated with expert few-shot exemplars to mitigate self-rewarding biases. Our empirical evaluation of state-of-the-art LLMs reveals a pronounced performance ceiling: even leading models achieve a peak success rate of only ~66%, with a mean score around 55%. Models also exhibit domain-specific divergence, showing non-overlapping strengths in quantitative reasoning versus linguistic synthesis.. These findings underscore a significant "expert-gap" in current AI systems and establish XpertBench as a critical instrument for navigating the transition from general-purpose assistants to specialized professional collaborators.
69d311062c523d36431facff
Xue Liu
69d311062c523d36431fad00
Xin Ma
69d311062c523d36431fad01
Yuxin Ma
69d311062c523d36431fad02
Yongchang Peng
69d311062c523d36431fad03
Duo Wang
69d311062c523d36431fad04
Zhoufutu Wen
69d311062c523d36431fad05
Ge Zhang
69d311062c523d36431fad06
Kaiyuan Zhang
69d311062c523d36431fad07
Xinyu Chen
69d311062c523d36431fad08
Tianci He
69d311062c523d36431fad09
Jiani Hou
69d311062c523d36431fad0a
Liang Hu
69d311062c523d36431fad0b
Ziyun Huang
69d311062c523d36431fad0c
Yongzhe Hui
69d311062c523d36431fad0d
Jianpeng Jiao
69d311062c523d36431fad0e
Chennan Ju
69d311062c523d36431fad0f
Yingru Kong
69d311062c523d36431fad10
Yiran Li
69d311062c523d36431fad11
Mengyun Liu
69d311062c523d36431fad12
Luyao Ma
69d311062c523d36431fad13
Fei Ni
69d311062c523d36431fad14
Yiqing Ni
69d311062c523d36431fad15
Yueyan Qiu
69d311062c523d36431fad16
Yanle Ren
69d311062c523d36431fad17
Zilin Shi
69d311062c523d36431fad18
Zaiyuan Wang
69d311062c523d36431fad19
Wenjie Yue
69d311062c523d36431fad1a
Shiyu Zhang
69d311062c523d36431fad1b
Xinyi Zhang
69d311062c523d36431fad1c
Kaiwen Zhao
69d311062c523d36431fad1d
Zhenwei Zhu
Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation
As Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks, a pivotal challenge persists: evaluating their proficiency in complex, open-ended tasks characterizing genuine expert-level cognition. Existing frameworks suffer from narrow domain coverage, reliance on generalist tasks, or self-evaluation biases. To bridge this gap, we present XpertBench, a high-fidelity benchmark engineered to assess LLMs across authentic professional domains. XpertBench consists of 1,346 meticulously curated tasks across 80 categories, spanning finance, healthcare, legal services, education, and dual-track research (STEM and Humanities). These tasks are derived from over 1,000 submissions by domain experts--including researchers from elite institutions and practitioners with extensive clinical or industrial experience--ensuring superior ecological validity. Each task uses detailed rubrics with mostly 15-40 weighted checkpoints to assess professional rigor. To facilitate scalable yet human-aligned assessment, we introduce ShotJudge, a novel evaluation paradigm that employs LLM judges calibrated with expert few-shot exemplars to mitigate self-rewarding biases. Our empirical evaluation of state-of-the-art LLMs reveals a pronounced performance ceiling: even leading models achieve a peak success rate of only ~66%, with a mean score around 55%. Models also exhibit domain-specific divergence, showing non-overlapping strengths in quantitative reasoning versus linguistic synthesis.. These findings underscore a significant "expert-gap" in current AI systems and establish XpertBench as a critical instrument for navigating the transition from general-purpose assistants to specialized professional collaborators.
XpertBench presents a comprehensive benchmark for evaluating large language models across professional domains using expert-curated tasks and a novel LLM-based evaluation approach called ShotJudge.
Large Language Models XpertBench ShotJudge expert-level cognition domain expertise few-shot exemplars self-rewarding biases professional domains ecological validity