Actial: Activate Spatial Reasoning Ability of Multimodal Large Language
  Models
Recent advances in Multimodal Large Language Models (MLLMs) have
significantly improved 2D visual understanding, prompting interest in their
application to complex 3D reasoning tasks. However, it remains unclear whether
these models can effectively capture the detailed spatial information required
for robust real-world performance, especially cross-view consistency, a key
requirement for accurate 3D reasoning. Considering this issue, we introduce
Viewpoint Learning, a task designed to evaluate and improve the spatial
reasoning capabilities of MLLMs. We present the Viewpoint-100K dataset,
consisting of 100K object-centric image pairs with diverse viewpoints and
corresponding question-answer pairs. Our approach employs a two-stage
fine-tuning strategy: first, foundational knowledge is injected to the baseline
MLLM via Supervised Fine-Tuning (SFT) on Viewpoint-100K, resulting in
significant improvements across multiple tasks; second, generalization is
enhanced through Reinforcement Learning using the Group Relative Policy
Optimization (GRPO) algorithm on a broader set of questions. Additionally, we
introduce a hybrid cold-start initialization method designed to simultaneously
learn viewpoint representations and maintain coherent reasoning thinking.
Experimental results show that our approach significantly activates the spatial
reasoning ability of MLLM, improving performance on both in-domain and
out-of-domain reasoning tasks. Our findings highlight the value of developing
foundational spatial skills in MLLMs, supporting future progress in robotics,
autonomous systems, and 3D scene understanding.
6909a0a0812eca10f9cc63ba
Xiaoyu Zhan
6909a0a0812eca10f9cc63bb
Wenxuan Huang
claimed_verified
2025-11-04T09:25:01.262Z
67dc162ec8c00778e8689f42
https://cdn-avatars.huggingface.co/v1/production/uploads/67dc162ec8c00778e8689f42/_y_tO6W3ONOkOWbumAFXA.png
Wenxuan Huang
Osilly
user
Osilly
6909a0a0812eca10f9cc63bc
Hao Sun
6909a0a0812eca10f9cc63bd
Xinyu Fu
6909a0a0812eca10f9cc63be
Changfeng Ma
6909a0a0812eca10f9cc63bf
Shaosheng Cao
6909a0a0812eca10f9cc63c0
Bohan Jia
6909a0a0812eca10f9cc63c1
Shaohui Lin
6909a0a0812eca10f9cc63c2
Zhenfei Yin
6909a0a0812eca10f9cc63c3
Lei Bai
6909a0a0812eca10f9cc63c4
Wanli Ouyang
6909a0a0812eca10f9cc63c5
Yuanqi Li
6909a0a0812eca10f9cc63c6
Jie Guo
6909a0a0812eca10f9cc63c7
Yanwen Guo
Actial: Activate Spatial Reasoning Ability of Multimodal Large Language
  Models
Recent advances in Multimodal Large Language Models (MLLMs) have
significantly improved 2D visual understanding, prompting interest in their
application to complex 3D reasoning tasks. However, it remains unclear whether
these models can effectively capture the detailed spatial information required
for robust real-world performance, especially cross-view consistency, a key
requirement for accurate 3D reasoning. Considering this issue, we introduce
Viewpoint Learning, a task designed to evaluate and improve the spatial
reasoning capabilities of MLLMs. We present the Viewpoint-100K dataset,
consisting of 100K object-centric image pairs with diverse viewpoints and
corresponding question-answer pairs. Our approach employs a two-stage
fine-tuning strategy: first, foundational knowledge is injected to the baseline
MLLM via Supervised Fine-Tuning (SFT) on Viewpoint-100K, resulting in
significant improvements across multiple tasks; second, generalization is
enhanced through Reinforcement Learning using the Group Relative Policy
Optimization (GRPO) algorithm on a broader set of questions. Additionally, we
introduce a hybrid cold-start initialization method designed to simultaneously
learn viewpoint representations and maintain coherent reasoning thinking.
Experimental results show that our approach significantly activates the spatial
reasoning ability of MLLM, improving performance on both in-domain and
out-of-domain reasoning tasks. Our findings highlight the value of developing
foundational spatial skills in MLLMs, supporting future progress in robotics,
autonomous systems, and 3D scene understanding.
Viewpoint Learning enhances the spatial reasoning capabilities of Multimodal Large Language Models using a two-stage fine-tuning strategy and a hybrid cold-start initialization method, improving performance on both in-domain and out-of-domain 3D reasoning tasks.
Multimodal Large Language Models Viewpoint Learning Viewpoint-100K dataset Supervised Fine-Tuning Reinforcement Learning Group Relative Policy Optimization hybrid cold-start initialization spatial reasoning