ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement
We introduce ThinkTwice, a simple two-phase framework that jointly optimizes LLMs to solve reasoning problems and refine the answers, based on Group Relative Policy Optimization (GRPO). In each pair of training steps, ThinkTwice first optimizes the model on solving reasoning problems, then optimizes it on refining its own solutions to the same problems, using the same binary correctness reward in both phases without correctness signals or critique annotations. Across five mathematical reasoning benchmarks and two model families including Qwen3-4B and Olmo3-7B, ThinkTwice substantially improves both reasoning and refinement performance over competitive online policy optimization baselines. Specifically, on Qwen3-4B, ThinkTwice outperforms GRPO on AIME by 5 percentage points before refinement and by 11.5 points after one self-refinement step, measured by pass@4. Analysis of the training dynamics of ThinkTwice reveals an implicit rectify-then-fortify curriculum: refinement predominantly corrects errors early in training and naturally shifts toward preserving already-correct solutions as the model improves, yielding a more rectified reward signal. Our work establishes joint training of reasoning and self-refinement as a principled and effective methodology for RLVR.
69cf13e8e02fdd68b91b79f9
Difan Jiao
claimed_verified
2026-04-03T09:08:04.754Z
65244bd306e839c55403beeb
/avatars/2e475eaf9885bf579ca5f791704c0a3d.svg
Difan Jiao
difanjiao
user
difanjiao
69cf13e8e02fdd68b91b79fa
Qianfeng Wen
claimed_verified
2026-04-07T08:45:48.735Z
667465a45ad1ce3339324940
/avatars/08bf0f6a9ea16195c85e7f60aa807945.svg
Qianfeng Wen
wieeii
user
wieeii
69cf13e8e02fdd68b91b79fb
Blair Yang
69cf13e8e02fdd68b91b79fc
Zhenwei Tang
claimed_verified
2026-04-07T08:45:51.037Z
653964f156c9b35961f44456
/avatars/0245e7543bff797405a927df5d16aeca.svg
Joseph Tang
lilvjosephtang
user
lilvjosephtang
69cf13e8e02fdd68b91b79fd
Ashton Anderson
ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement
We introduce ThinkTwice, a simple two-phase framework that jointly optimizes LLMs to solve reasoning problems and refine the answers, based on Group Relative Policy Optimization (GRPO). In each pair of training steps, ThinkTwice first optimizes the model on solving reasoning problems, then optimizes it on refining its own solutions to the same problems, using the same binary correctness reward in both phases without correctness signals or critique annotations. Across five mathematical reasoning benchmarks and two model families including Qwen3-4B and Olmo3-7B, ThinkTwice substantially improves both reasoning and refinement performance over competitive online policy optimization baselines. Specifically, on Qwen3-4B, ThinkTwice outperforms GRPO on AIME by 5 percentage points before refinement and by 11.5 points after one self-refinement step, measured by pass@4. Analysis of the training dynamics of ThinkTwice reveals an implicit rectify-then-fortify curriculum: refinement predominantly corrects errors early in training and naturally shifts toward preserving already-correct solutions as the model improves, yielding a more rectified reward signal. Our work establishes joint training of reasoning and self-refinement as a principled and effective methodology for RLVR.
ThinkTwice is a two-phase framework that jointly optimizes large language models for reasoning and self-refinement using Group Relative Policy Optimization, demonstrating improved performance on mathematical reasoning benchmarks.
Group Relative Policy Optimization reasoning problems self-refinement binary correctness reward reinforcement learning with human feedback policy optimization training dynamics rectify-then-fortify curriculum