All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models
Recent studies have demonstrated that Reinforcement Learning (RL), notably Group Relative Policy Optimization (GRPO), can intrinsically elicit and enhance the reasoning capabilities of Vision-Language Models (VLMs). However, despite the promise, the underlying mechanisms that drive the effectiveness of RL models as well as their limitations remain underexplored. In this paper, we highlight a fundamental behavioral distinction between RL and base models, where the former engages in deeper yet narrow reasoning, while base models, despite less refined along individual path, exhibit broader and more diverse thinking patterns. Through further analysis of training dynamics, we show that GRPO is prone to diversity collapse, causing models to prematurely converge to a limited subset of reasoning strategies while discarding the majority of potential alternatives, leading to local optima and poor scalability. To address this, we propose Multi-Group Policy Optimization (MUPO), a simple yet effective approach designed to incentivize divergent thinking across multiple solutions, and demonstrate its effectiveness on established benchmarks. Project page: https://xytian1008.github.io/MUPO/
69cdc4bfacf91530b21df8c4
Xinyu Tian
claimed_verified
2026-04-03T09:08:48.412Z
630c7d9ecb1d6a2384273f1a
/avatars/8ba76de6092e5d9fcc4f23d548befe9a.svg
Xinyu Tian
xytian1008
user
xytian1008
69cdc4bfacf91530b21df8c5
Shu Zou
claimed_verified
2026-04-13T07:35:19.743Z
679de3b13d9e2d1228188899
/avatars/768e4dd418567ae066563f3d3a7217fd.svg
shuzou
treeZou
user
treeZou
69cdc4bfacf91530b21df8c6
Zhaoyuan Yang
69cdc4bfacf91530b21df8c7
Mengqi He
69cdc4bfacf91530b21df8c8
Peter Tu
69cdc4bfacf91530b21df8c9
Jing Zhang
All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models
Recent studies have demonstrated that Reinforcement Learning (RL), notably Group Relative Policy Optimization (GRPO), can intrinsically elicit and enhance the reasoning capabilities of Vision-Language Models (VLMs). However, despite the promise, the underlying mechanisms that drive the effectiveness of RL models as well as their limitations remain underexplored. In this paper, we highlight a fundamental behavioral distinction between RL and base models, where the former engages in deeper yet narrow reasoning, while base models, despite less refined along individual path, exhibit broader and more diverse thinking patterns. Through further analysis of training dynamics, we show that GRPO is prone to diversity collapse, causing models to prematurely converge to a limited subset of reasoning strategies while discarding the majority of potential alternatives, leading to local optima and poor scalability. To address this, we propose Multi-Group Policy Optimization (MUPO), a simple yet effective approach designed to incentivize divergent thinking across multiple solutions, and demonstrate its effectiveness on established benchmarks. Project page: https://xytian1008.github.io/MUPO/
Reinforcement Learning enhances Vision-Language Model reasoning but suffers from diversity collapse; a new Multi-Group Policy Optimization method is proposed to encourage diverse thinking patterns.
Reinforcement Learning Group Relative Policy Optimization Vision-Language Models reasoning capabilities training dynamics diversity collapse Multi-Group Policy Optimization convergence local optima scalability