Reasoning over mathematical objects: on-policy reward modeling and test time aggregation
The ability to precisely derive mathematical objects is a core requirement for downstream STEM applications, including mathematics, physics, and chemistry, where reasoning must culminate in formally structured expressions. Yet, current LM evaluations of mathematical and scientific reasoning rely heavily on simplified answer formats such as numerical values or multiple choice options due to the convenience of automated assessment. In this paper we provide three contributions for improving reasoning over mathematical objects: (i) we build and release training data and benchmarks for deriving mathematical objects, the Principia suite; (ii) we provide training recipes with strong LLM-judges and verifiers, where we show that on-policy judge training boosts performance; (iii) we show how on-policy training can also be used to scale test-time compute via aggregation. We find that strong LMs such as Qwen3-235B and o3 struggle on Principia, while our training recipes can bring significant improvements over different LLM backbones, while simultaneously improving results on existing numerical and MCQA tasks, demonstrating cross-format generalization of reasoning abilities.
69bca45a081b9ad49af57189
Pranjal Aggarwal
69bca45a081b9ad49af5718a
Marjan Ghazvininejad
69bca45a081b9ad49af5718b
Seungone Kim
69bca45a081b9ad49af5718c
Ilia Kulikov
69bca45a081b9ad49af5718d
Jack Lanchantin
69bca45a081b9ad49af5718e
Xian Li
69bca45a081b9ad49af5718f
Tianjian Li
69bca45a081b9ad49af57190
Bo Liu
claimed_verified
2026-03-23T10:55:24.839Z
635e3a76106f984574c36409
https://cdn-avatars.huggingface.co/v1/production/uploads/1667120725800-635e3a76106f984574c36409.png
Bo Liu
Benjamin-eecs
user
Benjamin-eecs
69bca45a081b9ad49af57191
Graham Neubig
69bca45a081b9ad49af57192
Anaelia Ovalle
69bca45a081b9ad49af57193
Swarnadeep Saha
69bca45a081b9ad49af57194
Sainbayar Sukhbaatar
69bca45a081b9ad49af57195
Sean Welleck
69bca45a081b9ad49af57196
Jason Weston
69bca45a081b9ad49af57197
Chenxi Whitehouse
69bca45a081b9ad49af57198
Adina Williams
69bca45a081b9ad49af57199
Jing Xu
69bca45a081b9ad49af5719a
Ping Yu
69bca45a081b9ad49af5719b
Weizhe Yuan
69bca45a081b9ad49af5719c
Jingyu Zhang
claimed_verified
2026-04-15T12:09:31.520Z
62fb40b59af1d16bc0ac60f4
/avatars/03ff66a419db8f2bc8e89a3b47aaaeac.svg
Jack Zhang
jackzhang
user
jackzhang
69bca45a081b9ad49af5719d
Wenting Zhao
Reasoning over mathematical objects: on-policy reward modeling and test time aggregation
The ability to precisely derive mathematical objects is a core requirement for downstream STEM applications, including mathematics, physics, and chemistry, where reasoning must culminate in formally structured expressions. Yet, current LM evaluations of mathematical and scientific reasoning rely heavily on simplified answer formats such as numerical values or multiple choice options due to the convenience of automated assessment. In this paper we provide three contributions for improving reasoning over mathematical objects: (i) we build and release training data and benchmarks for deriving mathematical objects, the Principia suite; (ii) we provide training recipes with strong LLM-judges and verifiers, where we show that on-policy judge training boosts performance; (iii) we show how on-policy training can also be used to scale test-time compute via aggregation. We find that strong LMs such as Qwen3-235B and o3 struggle on Principia, while our training recipes can bring significant improvements over different LLM backbones, while simultaneously improving results on existing numerical and MCQA tasks, demonstrating cross-format generalization of reasoning abilities.
Training recipes using LLM judges and verifiers improve mathematical object derivation capabilities, showing cross-format generalization in reasoning tasks.
mathematical objects LLM-judges verifiers on-policy training test-time compute cross-format generalization