Long-horizon Reasoning Agent for Olympiad-Level Mathematical Problem Solving
Large language models (LLMs) have achieved significant progress in solving complex reasoning tasks by Reinforcement Learning with Verifiable Rewards (RLVR). This advancement is also inseparable from the oversight automated by reliable verifiers. However, current outcome-based verifiers (OVs) are unable to inspect the unreliable intermediate steps in the long reasoning chains of thought (CoTs). Meanwhile, current process-based verifiers (PVs) have difficulties in reliably detecting errors in the complex long CoTs, limited by the scarcity of high-quality annotations due to the prohibitive costs of human annotations. Therefore, we propose the Outcome-based Process Verifier (OPV), which verifies the rationale process of summarized outcomes from long CoTs to achieve both accurate and efficient verification and enable large-scale annotation. To empower the proposed verifier, we adopt an iterative active learning framework with expert annotations to progressively improve the verification capability of OPV with fewer annotation costs. Specifically, in each iteration, the most uncertain cases of the current best OPV are annotated and then subsequently used to train a new OPV through Rejection Fine-Tuning (RFT) and RLVR for the next round. Extensive experiments demonstrate OPV's superior performance and broad applicability. It achieves new state-of-the-art results on our held-out \thisbench, outperforming much larger open-source models such as Qwen3-Max-Preview with an F1 score of 83.1 compared to 76.3. Furthermore, OPV effectively detects false positives within synthetic dataset, closely align with expert assessment. When collaborating with policy models, OPV consistently yields performance gains, e.g., raising the accuracy of DeepSeek-R1-Distill-Qwen-32B from 55.2\% to 73.3\% on AIME2025 as the compute budget scales.
693b91d89874a2a5e4ffb329
Songyang Gao
693b91d89874a2a5e4ffb32a
Yuzhe Gu
claimed_verified
2025-12-12T09:14:56.632Z
6601196cc91ba4c08ad6e270
https://cdn-avatars.huggingface.co/v1/production/uploads/6601196cc91ba4c08ad6e270/venywO3WPi2fNi5WUJTH0.jpeg
Yuzhe Gu
vanilla1116
user
vanilla1116
693b91d89874a2a5e4ffb32b
Zijian Wu
693b91d89874a2a5e4ffb32c
Lingkai Kong
693b91d89874a2a5e4ffb32d
Wenwei Zhang
admin_assigned
2025-12-12T13:03:21.987Z
64e8505321540e1da3226b54
/avatars/18958b8406d1ce492b54c1c839f18c54.svg
Wenwei Zhang
ZwwWayne
user
ZwwWayne
693b91d89874a2a5e4ffb32e
Zhongrui Cai
693b91d89874a2a5e4ffb32f
Fan Zheng
693b91d89874a2a5e4ffb330
Tianyou Ma
admin_assigned
2025-12-12T13:03:57.205Z
670f8df2005a358fdc6c2fb6
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/qw2yocepl2vhC5T2ae49b.png
tianyou
matianyou
user
matianyou
693b91d89874a2a5e4ffb331
Junhao Shen
claimed_verified
2025-12-12T09:15:00.496Z
687f853bb39262ba84f3eeff
/avatars/cdfc44fde8237f08f10192553fe5a075.svg
Junhao Shen
shenjunhao
user
shenjunhao
693b91d89874a2a5e4ffb332
Haiteng Zhao
693b91d89874a2a5e4ffb333
Duanyang Zhang
admin_assigned
2025-12-12T13:04:12.911Z
6454b1073aaeff9f3d330ef6
/avatars/331fcbbf18a72b0dc8419ca3a77299bb.svg
Duanyang Zhang
KKKDaniel
user
KKKDaniel
693b91d89874a2a5e4ffb334
Huilun Zhang
693b91d89874a2a5e4ffb335
Kuikun Liu
claimed_verified
2025-12-12T09:14:58.650Z
63fd691794cc8f815d50c112
/avatars/87305d1cbfcc717e910ccdfaf0568f80.svg
liu
Harold-lkk
user
Harold-lkk
693b91d89874a2a5e4ffb336
Chengqi Lyu
693b91d89874a2a5e4ffb337
Yanhui Duan
693b91d89874a2a5e4ffb338
Chiyu Chen
693b91d89874a2a5e4ffb339
Ningsheng Ma
693b91d89874a2a5e4ffb33a
Jianfei Gao
693b91d89874a2a5e4ffb33b
Han Lyu
693b91d89874a2a5e4ffb33c
Dahua Lin
admin_assigned
2025-12-12T13:04:20.346Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
lindahua
user
lindahua
693b91d89874a2a5e4ffb33d
Kai Chen
Long-horizon Reasoning Agent for Olympiad-Level Mathematical Problem Solving
Large language models (LLMs) have achieved significant progress in solving complex reasoning tasks by Reinforcement Learning with Verifiable Rewards (RLVR). This advancement is also inseparable from the oversight automated by reliable verifiers. However, current outcome-based verifiers (OVs) are unable to inspect the unreliable intermediate steps in the long reasoning chains of thought (CoTs). Meanwhile, current process-based verifiers (PVs) have difficulties in reliably detecting errors in the complex long CoTs, limited by the scarcity of high-quality annotations due to the prohibitive costs of human annotations. Therefore, we propose the Outcome-based Process Verifier (OPV), which verifies the rationale process of summarized outcomes from long CoTs to achieve both accurate and efficient verification and enable large-scale annotation. To empower the proposed verifier, we adopt an iterative active learning framework with expert annotations to progressively improve the verification capability of OPV with fewer annotation costs. Specifically, in each iteration, the most uncertain cases of the current best OPV are annotated and then subsequently used to train a new OPV through Rejection Fine-Tuning (RFT) and RLVR for the next round. Extensive experiments demonstrate OPV's superior performance and broad applicability. It achieves new state-of-the-art results on our held-out \thisbench, outperforming much larger open-source models such as Qwen3-Max-Preview with an F1 score of 83.1 compared to 76.3. Furthermore, OPV effectively detects false positives within synthetic dataset, closely align with expert assessment. When collaborating with policy models, OPV consistently yields performance gains, e.g., raising the accuracy of DeepSeek-R1-Distill-Qwen-32B from 55.2\% to 73.3\% on AIME2025 as the compute budget scales.
OPV, an iterative active learning framework with Rejection Fine-Tuning, enhances verification of long reasoning chains in large language models, achieving state-of-the-art results and improving accuracy in collaborative tasks.
Reinforcement Learning with Verifiable Rewards (RLVR) outcome-based verifiers (OVs) process-based verifiers (PVs) long reasoning chains of thought (CoTs) iterative active learning Rejection Fine-Tuning (RFT) F1 score accuracy DeepSeek-R1-Distill-Qwen-32B AIME2025