Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videos
Vision-Language-Action (VLA) models provide a promising paradigm for robot learning by integrating visual perception with language-guided policy learning. However, most existing approaches rely on 2D visual inputs to perform actions in 3D physical environments, creating a significant gap between perception and action grounding. To bridge this gap, we propose a Spatial-Aware VLA Pretraining paradigm that performs explicit alignment between visual space and physical space during pretraining, enabling models to acquire 3D spatial understanding before robot policy learning. Starting from pretrained vision-language models, we leverage large-scale human demonstration videos to extract 3D visual and 3D action annotations, forming a new source of supervision that aligns 2D visual observations with 3D spatial reasoning. We instantiate this paradigm with VIPA-VLA, a dual-encoder architecture that incorporates a 3D visual encoder to augment semantic visual representations with 3D-aware features. When adapted to downstream robot tasks, VIPA-VLA achieves significantly improved grounding between 2D vision and 3D action, resulting in more robust and generalizable robotic policies.
6940e59265f1e24a1178065c
Yicheng Feng
claimed_verified
2026-01-22T08:47:50.027Z
6655b86e607894ea80d74910
/avatars/663c0135c903c9c127fe1b8d8aaf279c.svg
yicheng feng
takenpeanut
user
takenpeanut
6940e59265f1e24a1178065d
Wanpeng Zhang
claimed_verified
2025-12-16T09:37:38.330Z
640dd700fdeaae139081f598
https://cdn-avatars.huggingface.co/v1/production/uploads/640dd700fdeaae139081f598/L986zu4-iOPFs9Y3_T5Ue.jpeg
Wanpeng Zhang
zawnpn
user
zawnpn
6940e59265f1e24a1178065e
Ye Wang
6940e59265f1e24a1178065f
Hao Luo
claimed_verified
2026-01-21T10:16:01.962Z
6708cbdcf8a1d7b26732c038
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/CN1WHMPKfjQ8wmfwOe0ni.png
Hao Luo
Lightet
user
Lightet
6940e59265f1e24a11780660
Haoqi Yuan
6940e59265f1e24a11780661
Sipeng Zheng
6940e59265f1e24a11780662
Zongqing Lu
Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videos
Vision-Language-Action (VLA) models provide a promising paradigm for robot learning by integrating visual perception with language-guided policy learning. However, most existing approaches rely on 2D visual inputs to perform actions in 3D physical environments, creating a significant gap between perception and action grounding. To bridge this gap, we propose a Spatial-Aware VLA Pretraining paradigm that performs explicit alignment between visual space and physical space during pretraining, enabling models to acquire 3D spatial understanding before robot policy learning. Starting from pretrained vision-language models, we leverage large-scale human demonstration videos to extract 3D visual and 3D action annotations, forming a new source of supervision that aligns 2D visual observations with 3D spatial reasoning. We instantiate this paradigm with VIPA-VLA, a dual-encoder architecture that incorporates a 3D visual encoder to augment semantic visual representations with 3D-aware features. When adapted to downstream robot tasks, VIPA-VLA achieves significantly improved grounding between 2D vision and 3D action, resulting in more robust and generalizable robotic policies.
A Spatial-Aware VLA Pretraining paradigm improves 3D spatial understanding in robots by aligning 2D visual inputs with 3D actions using dual-encoder architecture with a 3D visual encoder.
Vision-Language-Action (VLA) models Spatial-Aware VLA Pretraining 3D spatial understanding dual-encoder architecture 3D visual encoder 3D visual and 3D action annotations