QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs
As Large Language Models (LLMs) saturate elementary benchmarks, the research frontier has shifted from generation to the reliability of automated evaluation. We demonstrate that standard "LLM-as-a-Judge" protocols suffer from a systematic Alignment Gap when applied to upper-undergraduate to early graduate level mathematics. To quantify this, we introduce QEDBench, the first large-scale dual-rubric alignment benchmark to systematically measure alignment with human experts on university-level math proofs by contrasting course-specific rubrics against expert common knowledge criteria. By deploying a dual-evaluation matrix (7 judges x 5 solvers) against 1,000+ hours of human evaluation, we reveal that certain frontier evaluators like Claude Opus 4.5, DeepSeek-V3, Qwen 2.5 Max, and Llama 4 Maverick exhibit significant positive bias (up to +0.18, +0.20, +0.30, +0.36 mean score inflation, respectively). Furthermore, we uncover a critical reasoning gap in the discrete domain: while Gemini 3.0 Pro achieves state-of-the-art performance (0.91 average human evaluation score), other reasoning models like GPT-5 Pro and Claude Sonnet 4.5 see their performance significantly degrade in discrete domains. Specifically, their average human evaluation scores drop to 0.72 and 0.63 in Discrete Math, and to 0.74 and 0.50 in Graph Theory. In addition to these research results, we also release QEDBench as a public benchmark for evaluating and improving AI judges. Our benchmark is publicly published at https://github.com/qqliu/Yale-QEDBench.
69a5cb79663a122a7ce6657f
Santiago Gonzalez
69a5cb79663a122a7ce66580
Alireza Amiri Bavandpour
69a5cb79663a122a7ce66581
Peter Ye
69a5cb79663a122a7ce66582
Edward Zhang
69a5cb79663a122a7ce66583
Ruslans Aleksejevs
69a5cb79663a122a7ce66584
Todor Antić
69a5cb79663a122a7ce66585
Polina Baron
69a5cb79663a122a7ce66586
Sujeet Bhalerao
claimed_verified
2026-03-16T10:44:59.030Z
65ad9ea7b68db4f26ef641b8
/avatars/f807e2d324603c9927a55d295432439a.svg
Sujeet Bhalerao
sujeetbhalerao
user
sujeetbhalerao
69a5cb79663a122a7ce66587
Shubhrajit Bhattacharya
69a5cb79663a122a7ce66588
Zachary Burton
69a5cb79663a122a7ce66589
John Byrne
69a5cb79663a122a7ce6658a
Hyungjun Choi
69a5cb79663a122a7ce6658b
Nujhat Ahmed Disha
69a5cb79663a122a7ce6658c
Koppany István Encz
69a5cb79663a122a7ce6658d
Yuchen Fang
69a5cb79663a122a7ce6658e
Robert Joseph George
69a5cb79663a122a7ce6658f
Ebrahim Ghorbani
69a5cb79663a122a7ce66590
Alan Goldfarb
69a5cb79663a122a7ce66591
Jing Guo
claimed_verified
2026-03-03T10:40:33.656Z
6423475423b98d5ee6e95220
https://cdn-avatars.huggingface.co/v1/production/uploads/6423475423b98d5ee6e95220/92jO3vOYpdk5WtQ04n67q.jpeg
Jing Guo
hedgehog0
user
hedgehog0
69a5cb79663a122a7ce66592
Meghal Gupta
69a5cb79663a122a7ce66593
Stefano Huber
69a5cb79663a122a7ce66594
Annika Kanckos
69a5cb79663a122a7ce66595
Minjung Kang
69a5cb79663a122a7ce66596
Hyun Jong Kim
69a5cb79663a122a7ce66597
Dino Lorenzini
69a5cb79663a122a7ce66598
Levi Lorenzo
69a5cb79663a122a7ce66599
Tianyi Mao
69a5cb79663a122a7ce6659a
Giovanni Marzenta
69a5cb79663a122a7ce6659b
Ariane M. Masuda
69a5cb79663a122a7ce6659c
Lukas Mauth
69a5cb79663a122a7ce6659d
Ana Mickovic
69a5cb79663a122a7ce6659e
Andres Miniguano-Trujillo
69a5cb79663a122a7ce6659f
Antoine Moulin
69a5cb79663a122a7ce665a0
Wenqi Ni
69a5cb79663a122a7ce665a1
Tomos Parry
69a5cb79663a122a7ce665a2
Kevin Ren
69a5cb79663a122a7ce665a3
Hossein Roodbarani
69a5cb79663a122a7ce665a4
Mathieu Rundström
69a5cb79663a122a7ce665a5
Manjil Saikia
69a5cb79663a122a7ce665a6
Detchat Samart
69a5cb79663a122a7ce665a7
Rebecca Steiner
69a5cb79663a122a7ce665a8
Connor Stewart
69a5cb79663a122a7ce665a9
Dhara Thakkar
69a5cb79663a122a7ce665aa
Jeffrey Tse
69a5cb79663a122a7ce665ab
Vasiliki Velona
69a5cb79663a122a7ce665ac
Yunhai Xiang
69a5cb79663a122a7ce665ad
Sibel Yalçın
69a5cb79663a122a7ce665ae
Jun Yan
69a5cb79663a122a7ce665af
Ji Zeng
69a5cb79663a122a7ce665b0
Arman Cohan
69a5cb79663a122a7ce665b1
Quanquan C. Liu
claimed_verified
2026-03-03T10:40:39.291Z
693b269f4cb1a4ac8acf35ee
/avatars/1751718579ef88f281c8eb009b2ebe4a.svg
Quanquan Liu
qqggez
user
qqggez
QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs
As Large Language Models (LLMs) saturate elementary benchmarks, the research frontier has shifted from generation to the reliability of automated evaluation. We demonstrate that standard "LLM-as-a-Judge" protocols suffer from a systematic Alignment Gap when applied to upper-undergraduate to early graduate level mathematics. To quantify this, we introduce QEDBench, the first large-scale dual-rubric alignment benchmark to systematically measure alignment with human experts on university-level math proofs by contrasting course-specific rubrics against expert common knowledge criteria. By deploying a dual-evaluation matrix (7 judges x 5 solvers) against 1,000+ hours of human evaluation, we reveal that certain frontier evaluators like Claude Opus 4.5, DeepSeek-V3, Qwen 2.5 Max, and Llama 4 Maverick exhibit significant positive bias (up to +0.18, +0.20, +0.30, +0.36 mean score inflation, respectively). Furthermore, we uncover a critical reasoning gap in the discrete domain: while Gemini 3.0 Pro achieves state-of-the-art performance (0.91 average human evaluation score), other reasoning models like GPT-5 Pro and Claude Sonnet 4.5 see their performance significantly degrade in discrete domains. Specifically, their average human evaluation scores drop to 0.72 and 0.63 in Discrete Math, and to 0.74 and 0.50 in Graph Theory. In addition to these research results, we also release QEDBench as a public benchmark for evaluating and improving AI judges. Our benchmark is publicly published at https://github.com/qqliu/Yale-QEDBench.
Standard LLM-based evaluation protocols show systematic alignment gaps in mathematical proof assessment, with some models exhibiting significant score inflation and others demonstrating degraded performance in discrete mathematics domains.
Large Language Models LLM-as-a-Judge alignment gap mathematical proofs dual-rubric alignment QEDBench human evaluation score inflation discrete mathematics graph theory state-of-the-art performance