When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning
Recent progress in multimodal large language models has led to strong performance on reasoning tasks, but these improvements largely rely on high-quality annotated data or teacher-model distillation, both of which are costly and difficult to scale. To address this, we propose an unsupervised self-evolution training framework for multimodal reasoning that achieves stable performance improvements without using human-annotated answers or external reward models. For each input, we sample multiple reasoning trajectories and jointly model their within group structure. We use the Actor's self-consistency signal as a training prior, and introduce a bounded Judge based modulation to continuously reweight trajectories of different quality. We further model the modulated scores as a group level distribution and convert absolute scores into relative advantages within each group, enabling more robust policy updates. Trained with Group Relative Policy Optimization (GRPO) on unlabeled data, our method consistently improves reasoning performance and generalization on five mathematical reasoning benchmarks, offering a scalable path toward self-evolving multimodal models. The code are available at https://github.com/OPPO-Mente-Lab/LLM-Self-Judge.
69c3523795b48f16a30077f5
Zhengxian Wu
claimed_verified
2026-03-25T10:23:30.851Z
66ed28d311f93327d5a269a7
/avatars/ff4035365ae132f42892c80b94ea0a1f.svg
Wu Zhengxian
zx-Wu
user
zx-Wu
69c3523795b48f16a30077f6
Kai Shi
claimed_verified
2026-03-25T10:23:28.410Z
65d6ee076a36b5b354e6dfc0
/avatars/68b6468543e0c04e152629d430f02e55.svg
Kane Shi
Kane-oppo-2024
user
Kane-oppo-2024
69c3523795b48f16a30077f7
Chuanrui Zhang
claimed_verified
2026-03-27T11:38:11.704Z
6530df1de85cff4618e14d27
/avatars/515a85b8e3d1b805619cb0a70f2b3c26.svg
Chuanrui Zhang
xingyoujun
user
xingyoujun
69c3523795b48f16a30077f8
Zirui Liao
claimed_verified
2026-03-26T13:44:33.064Z
6735e993b5d2adaae6ee50fe
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/dUQ11D76fFiatNBq-iRNa.png
廖子睿
Slient-rain
user
Slient-rain
69c3523795b48f16a30077f9
Jun Yang
69c3523795b48f16a30077fa
Ni Yang
69c3523795b48f16a30077fb
Qiuying Peng
69c3523795b48f16a30077fc
Luyuan Zhang
69c3523795b48f16a30077fd
Hangrui Xu
69c3523795b48f16a30077fe
Tianhuang Su
69c3523795b48f16a30077ff
Zhenyu Yang
69c3523795b48f16a3007800
Haonan Lu
69c3523795b48f16a3007801
Haoqian Wang
When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning
Recent progress in multimodal large language models has led to strong performance on reasoning tasks, but these improvements largely rely on high-quality annotated data or teacher-model distillation, both of which are costly and difficult to scale. To address this, we propose an unsupervised self-evolution training framework for multimodal reasoning that achieves stable performance improvements without using human-annotated answers or external reward models. For each input, we sample multiple reasoning trajectories and jointly model their within group structure. We use the Actor's self-consistency signal as a training prior, and introduce a bounded Judge based modulation to continuously reweight trajectories of different quality. We further model the modulated scores as a group level distribution and convert absolute scores into relative advantages within each group, enabling more robust policy updates. Trained with Group Relative Policy Optimization (GRPO) on unlabeled data, our method consistently improves reasoning performance and generalization on five mathematical reasoning benchmarks, offering a scalable path toward self-evolving multimodal models. The code are available at https://github.com/OPPO-Mente-Lab/LLM-Self-Judge.
A self-evolution training framework for multimodal reasoning uses unsupervised learning with self-consistency signals and group-relative policy optimization to improve performance without labeled data.
multimodal large language models self-evolution training self-consistency signal bounded Judge Group Relative Policy Optimization reasoning trajectories policy updates mathematical reasoning benchmarks