N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models
While current multimodal models can answer questions based on 2D images, they lack intrinsic 3D object perception, limiting their ability to comprehend spatial relationships and depth cues in 3D scenes. In this work, we propose N3D-VLM, a novel unified framework that seamlessly integrates native 3D object perception with 3D-aware visual reasoning, enabling both precise 3D grounding and interpretable spatial understanding. Unlike conventional end-to-end models that directly predict answers from RGB/RGB-D inputs, our approach equips the model with native 3D object perception capabilities, enabling it to directly localize objects in 3D space based on textual descriptions. Building upon accurate 3D object localization, the model further performs explicit reasoning in 3D, achieving more interpretable and structured spatial understanding. To support robust training for these capabilities, we develop a scalable data construction pipeline that leverages depth estimation to lift large-scale 2D annotations into 3D space, significantly increasing the diversity and coverage for 3D object grounding data, yielding over six times larger than the largest existing single-image 3D detection dataset. Moreover, the pipeline generates spatial question-answering datasets that target chain-of-thought (CoT) reasoning in 3D, facilitating joint training for both 3D object localization and 3D spatial reasoning. Experimental results demonstrate that our unified framework not only achieves state-of-the-art performance on 3D grounding tasks, but also consistently surpasses existing methods in 3D spatial reasoning in vision-language model.
6944b692fbf17e708e185f41
Yuxin Wang
claimed_verified
2025-12-19T08:58:20.875Z
6707f8800812a88fd6d87ebe
/avatars/28cd99359799ce85d5fe503144b776d8.svg
YUXIN WANG
yuxinhk
user
yuxinhk
6944b692fbf17e708e185f42
Lei Ke
6944b692fbf17e708e185f43
Boqiang Zhang
admin_assigned
2026-03-03T16:39:26.907Z
626938b16f8f86ad21deb989
/avatars/b577835cb6053c90c1e86b7376c6104b.svg
Boqiang Zhang
Cyril666
user
Cyril666
6944b692fbf17e708e185f44
Tianyuan Qu
claimed_verified
2026-03-09T10:42:15.442Z
66e79b3c1c79fc2e51dc1d60
/avatars/8706336e9e7a417505c9bb32583a662f.svg
QU Tianyuan
TainU
user
TainU
6944b692fbf17e708e185f45
Hanxun Yu
claimed_verified
2026-02-03T10:09:39.100Z
652d3023cdb2a91205709b6a
https://cdn-avatars.huggingface.co/v1/production/uploads/652d3023cdb2a91205709b6a/x-MaRkBwSdZ0xIYDJg09u.jpeg
Hanxun Yu
JonnyYu828
user
JonnyYu828
6944b692fbf17e708e185f46
Zhenpeng Huang
claimed_verified
2026-02-05T10:57:47.543Z
62bec18e7e808565cc15610f
/avatars/78e8b98120a61bf90c43bd8c8ea8d375.svg
Zhenpeng Huang
hzp
user
hzp
6944b692fbf17e708e185f47
Meng Yu
6944b692fbf17e708e185f48
Dan Xu
claimed_verified
2025-12-19T16:25:54.282Z
66feab48651e00e22f33222e
/avatars/7344377e2c796c7ec85194bb2fc78521.svg
Dan Xu
danxuhk
user
danxuhk
6944b692fbf17e708e185f49
Dong Yu
N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models
While current multimodal models can answer questions based on 2D images, they lack intrinsic 3D object perception, limiting their ability to comprehend spatial relationships and depth cues in 3D scenes. In this work, we propose N3D-VLM, a novel unified framework that seamlessly integrates native 3D object perception with 3D-aware visual reasoning, enabling both precise 3D grounding and interpretable spatial understanding. Unlike conventional end-to-end models that directly predict answers from RGB/RGB-D inputs, our approach equips the model with native 3D object perception capabilities, enabling it to directly localize objects in 3D space based on textual descriptions. Building upon accurate 3D object localization, the model further performs explicit reasoning in 3D, achieving more interpretable and structured spatial understanding. To support robust training for these capabilities, we develop a scalable data construction pipeline that leverages depth estimation to lift large-scale 2D annotations into 3D space, significantly increasing the diversity and coverage for 3D object grounding data, yielding over six times larger than the largest existing single-image 3D detection dataset. Moreover, the pipeline generates spatial question-answering datasets that target chain-of-thought (CoT) reasoning in 3D, facilitating joint training for both 3D object localization and 3D spatial reasoning. Experimental results demonstrate that our unified framework not only achieves state-of-the-art performance on 3D grounding tasks, but also consistently surpasses existing methods in 3D spatial reasoning in vision-language model.
N3D-VLM integrates native 3D perception and reasoning in vision-language models, enabling precise 3D localization and spatial understanding with a large-scale dataset.
3D object perception 3D-aware visual reasoning 3D grounding spatial understanding native 3D object perception 3D object localization explicit reasoning depth estimation chain-of-thought reasoning 3D spatial reasoning vision-language model