From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoning
The cold-start initialization stage plays a pivotal role in training Multimodal Large Reasoning Models (MLRMs), yet its mechanisms remain insufficiently understood. To analyze this stage, we introduce the Visual Attention Score (VAS), an attention-based metric that quantifies how much a model attends to visual tokens. We find that reasoning performance is strongly correlated with VAS (r=0.9616): models with higher VAS achieve substantially stronger multimodal reasoning. Surprisingly, multimodal cold-start fails to elevate VAS, resulting in attention distributions close to the base model, whereas text-only cold-start leads to a clear increase. We term this counter-intuitive phenomenon Lazy Attention Localization. To validate its causal role, we design training-free interventions that directly modulate attention allocation during inference, performance gains of 1-2% without any retraining. Building on these insights, we further propose Attention-Guided Visual Anchoring and Reflection (AVAR), a comprehensive cold-start framework that integrates visual-anchored data synthesis, attention-guided objectives, and visual-anchored reward shaping. Applied to Qwen2.5-VL-7B, AVAR achieves an average gain of 7.0% across 7 multimodal reasoning benchmarks. Ablation studies further confirm that each component of AVAR contributes step-wise to the overall gains. The code, data, and models are available at https://github.com/lrlbbzl/Qwen-AVAR.
69af7f3ab3ab31dee1281bf9
Ruilin Luo
claimed_verified
2026-03-10T09:45:17.407Z
6548956fe49bd8d58e8adf0e
/avatars/24c54b14c253c87d4b7438193f16ce28.svg
Ruilin
Antimage01
user
Antimage01
69af7f3ab3ab31dee1281bfa
Chufan Shi
69af7f3ab3ab31dee1281bfb
Yizhen Zhang
69af7f3ab3ab31dee1281bfc
Cheng Yang
69af7f3ab3ab31dee1281bfd
Songtao Jiang
69af7f3ab3ab31dee1281bfe
Tongkun Guan
69af7f3ab3ab31dee1281bff
Ruizhe Chen
69af7f3ab3ab31dee1281c00
Ruihang Chu
69af7f3ab3ab31dee1281c01
Peng Wang
69af7f3ab3ab31dee1281c02
Mingkun Yang
69af7f3ab3ab31dee1281c03
Yujiu Yang
69af7f3ab3ab31dee1281c04
Junyang Lin
69af7f3ab3ab31dee1281c05
Zhibo Yang
From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoning
The cold-start initialization stage plays a pivotal role in training Multimodal Large Reasoning Models (MLRMs), yet its mechanisms remain insufficiently understood. To analyze this stage, we introduce the Visual Attention Score (VAS), an attention-based metric that quantifies how much a model attends to visual tokens. We find that reasoning performance is strongly correlated with VAS (r=0.9616): models with higher VAS achieve substantially stronger multimodal reasoning. Surprisingly, multimodal cold-start fails to elevate VAS, resulting in attention distributions close to the base model, whereas text-only cold-start leads to a clear increase. We term this counter-intuitive phenomenon Lazy Attention Localization. To validate its causal role, we design training-free interventions that directly modulate attention allocation during inference, performance gains of 1-2% without any retraining. Building on these insights, we further propose Attention-Guided Visual Anchoring and Reflection (AVAR), a comprehensive cold-start framework that integrates visual-anchored data synthesis, attention-guided objectives, and visual-anchored reward shaping. Applied to Qwen2.5-VL-7B, AVAR achieves an average gain of 7.0% across 7 multimodal reasoning benchmarks. Ablation studies further confirm that each component of AVAR contributes step-wise to the overall gains. The code, data, and models are available at https://github.com/lrlbbzl/Qwen-AVAR.
The study introduces a novel attention-based metric called Visual Attention Score to analyze cold-start initialization in multimodal large reasoning models, identifying a counter-intuitive phenomenon termed Lazy Attention Localization and proposing a framework called Attention-Guided Visual Anchoring and Reflection to improve multimodal reasoning performance.
Multimodal Large Reasoning Models Visual Attention Score attention-based metric visual tokens multimodal reasoning cold-start initialization Lazy Attention Localization Attention-Guided Visual Anchoring and Reflection visual-anchored data synthesis attention-guided objectives visual-anchored reward shaping