Visual Spatial Tuning
Capturing spatial relationships from visual inputs is a cornerstone of
human-like general intelligence. Several previous studies have tried to enhance
the spatial awareness of Vision-Language Models (VLMs) by adding extra expert
encoders, which brings extra overhead and usually harms general capabilities.
To enhance the spatial ability in general architectures, we introduce Visual
Spatial Tuning (VST), a comprehensive framework to cultivate VLMs with
human-like visuospatial abilities, from spatial perception to reasoning. We
first attempt to enhance spatial perception in VLMs by constructing a
large-scale dataset termed VST-P, which comprises 4.1 million samples spanning
19 skills across single views, multiple images, and videos. Then, we present
VST-R, a curated dataset with 135K samples that instruct models to reason in
space. In particular, we adopt a progressive training pipeline: supervised
fine-tuning to build foundational spatial knowledge, followed by reinforcement
learning to further improve spatial reasoning abilities. Without the
side-effect to general capabilities, the proposed VST consistently achieves
state-of-the-art results on several spatial benchmarks, including 34.8% on
MMSI-Bench and 61.2% on VSIBench. It turns out that the
Vision-Language-Action models can be significantly enhanced with the proposed
spatial tuning paradigm, paving the way for more physically grounded AI.
69115163830c57fd4c795fff
Rui Yang
claimed_verified
2025-11-10T08:44:47.523Z
641d6c27c5f150c3af0bf879
/avatars/39808793703406b4fe997e351b6a6bfb.svg
Ray Yang
rayruiyang
user
rayruiyang
69115163830c57fd4c796000
Ziyu Zhu
69115163830c57fd4c796001
Yanwei Li
claimed_verified
2025-11-10T08:44:49.860Z
643ff78cdc984afcbbbc3b1a
/avatars/eec5198ce88aaf8156840bec0d190a7f.svg
Yanwei Li
YanweiLi
user
YanweiLi
69115163830c57fd4c796002
Jingjia Huang
69115163830c57fd4c796003
Shen Yan
69115163830c57fd4c796004
Siyuan Zhou
69115163830c57fd4c796005
Zhe Liu
69115163830c57fd4c796006
Xiangtai Li
claimed_verified
2025-12-15T08:14:40.202Z
63958b4414513eaf9029ebf1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
Xiangtai Li
LXT
user
LXT
69115163830c57fd4c796007
Shuangye Li
69115163830c57fd4c796008
Wenqian Wang
69115163830c57fd4c796009
Yi Lin
69115163830c57fd4c79600a
Hengshuang Zhao
Visual Spatial Tuning
Capturing spatial relationships from visual inputs is a cornerstone of
human-like general intelligence. Several previous studies have tried to enhance
the spatial awareness of Vision-Language Models (VLMs) by adding extra expert
encoders, which brings extra overhead and usually harms general capabilities.
To enhance the spatial ability in general architectures, we introduce Visual
Spatial Tuning (VST), a comprehensive framework to cultivate VLMs with
human-like visuospatial abilities, from spatial perception to reasoning. We
first attempt to enhance spatial perception in VLMs by constructing a
large-scale dataset termed VST-P, which comprises 4.1 million samples spanning
19 skills across single views, multiple images, and videos. Then, we present
VST-R, a curated dataset with 135K samples that instruct models to reason in
space. In particular, we adopt a progressive training pipeline: supervised
fine-tuning to build foundational spatial knowledge, followed by reinforcement
learning to further improve spatial reasoning abilities. Without the
side-effect to general capabilities, the proposed VST consistently achieves
state-of-the-art results on several spatial benchmarks, including 34.8% on
MMSI-Bench and 61.2% on VSIBench. It turns out that the
Vision-Language-Action models can be significantly enhanced with the proposed
spatial tuning paradigm, paving the way for more physically grounded AI.
A framework called Visual Spatial Tuning (VST) enhances the spatial abilities of Vision-Language Models (VLMs) through progressive training with specialized datasets, achieving state-of-the-art results on spatial benchmarks.
Visual Spatial Tuning VST Vision-Language Models VLMs spatial perception spatial reasoning VST-P VST-R supervised fine-tuning reinforcement learning MMSI-Bench VSIBench Vision-Language-Action models