Towards Robust Mathematical Reasoning
Finding the right north-star metrics is highly critical for advancing the
mathematical reasoning capabilities of foundation models, especially given that
existing evaluations are either too easy or only focus on getting correct short
answers. To address these issues, we present IMO-Bench, a suite of advanced
reasoning benchmarks, vetted by a panel of top specialists and that
specifically targets the level of the International Mathematical Olympiad
(IMO), the most prestigious venue for young mathematicians. IMO-AnswerBench
first tests models on 400 diverse Olympiad problems with verifiable short
answers. IMO-Proof Bench is the next-level evaluation for proof-writing
capabilities, which includes both basic and advanced IMO level problems as well
as detailed grading guidelines to facilitate automatic grading. These
benchmarks played a crucial role in our historic achievement of the gold-level
performance at IMO 2025 with Gemini Deep Think (Luong and Lockhart, 2025). Our
model achieved 80.0% on IMO-AnswerBench and 65.7% on the advanced IMO-Proof
Bench, surpassing the best non-Gemini models by large margins of 6.9% and 42.4%
respectively. We also showed that autograders built with Gemini reasoning
correlate well with human evaluations and construct IMO-GradingBench, with 1000
human gradings on proofs, to enable further progress in automatic evaluation of
long-form answers. We hope that IMO-Bench will help the community towards
advancing robust mathematical reasoning and release it at
https://imobench.github.io/.
6909790d812eca10f9cc6262
Thang Luong
6909790d812eca10f9cc6263
Dawsen Hwang
6909790d812eca10f9cc6264
Hoang H. Nguyen
6909790d812eca10f9cc6265
Golnaz Ghiasi
6909790d812eca10f9cc6266
Yuri Chervonyi
6909790d812eca10f9cc6267
Insuk Seo
6909790d812eca10f9cc6268
Junsu Kim
6909790d812eca10f9cc6269
Garrett Bingham
6909790d812eca10f9cc626a
Jonathan Lee
6909790d812eca10f9cc626b
Swaroop Mishra
6909790d812eca10f9cc626c
Alex Zhai
6909790d812eca10f9cc626d
Clara Huiyi Hu
6909790d812eca10f9cc626e
Henryk Michalewski
6909790d812eca10f9cc626f
Jimin Kim
6909790d812eca10f9cc6270
Jeonghyun Ahn
6909790d812eca10f9cc6271
Junhwi Bae
6909790d812eca10f9cc6272
Xingyou Song
6909790d812eca10f9cc6273
Trieu H. Trinh
6909790d812eca10f9cc6274
Quoc V. Le
6909790d812eca10f9cc6275
Junehyuk Jung
Towards Robust Mathematical Reasoning
Finding the right north-star metrics is highly critical for advancing the
mathematical reasoning capabilities of foundation models, especially given that
existing evaluations are either too easy or only focus on getting correct short
answers. To address these issues, we present IMO-Bench, a suite of advanced
reasoning benchmarks, vetted by a panel of top specialists and that
specifically targets the level of the International Mathematical Olympiad
(IMO), the most prestigious venue for young mathematicians. IMO-AnswerBench
first tests models on 400 diverse Olympiad problems with verifiable short
answers. IMO-Proof Bench is the next-level evaluation for proof-writing
capabilities, which includes both basic and advanced IMO level problems as well
as detailed grading guidelines to facilitate automatic grading. These
benchmarks played a crucial role in our historic achievement of the gold-level
performance at IMO 2025 with Gemini Deep Think (Luong and Lockhart, 2025). Our
model achieved 80.0% on IMO-AnswerBench and 65.7% on the advanced IMO-Proof
Bench, surpassing the best non-Gemini models by large margins of 6.9% and 42.4%
respectively. We also showed that autograders built with Gemini reasoning
correlate well with human evaluations and construct IMO-GradingBench, with 1000
human gradings on proofs, to enable further progress in automatic evaluation of
long-form answers. We hope that IMO-Bench will help the community towards
advancing robust mathematical reasoning and release it at
https://imobench.github.io/.
IMO-Bench, a suite of advanced reasoning benchmarks, evaluates mathematical reasoning capabilities of foundation models using IMO-level problems and detailed grading guidelines, achieving gold-level performance with Gemini Deep Think.
IMO-Bench IMO-AnswerBench IMO-Proof Bench IMO-GradingBench mathematical reasoning foundation models autograders human evaluations