Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning
Existing Multimodal Large Language Models (MLLMs) struggle with 3D spatial reasoning, as they fail to construct structured abstractions of the 3D environment depicted in video inputs. To bridge this gap, drawing inspiration from cognitive theories of allocentric spatial reasoning, we investigate how to enable MLLMs to model and reason over text-based spatial representations of video. Specifically, we introduce Textual Representation of Allocentric Context from Egocentric Video (TRACE), a prompting method that induces MLLMs to generate text-based representations of 3D environments as intermediate reasoning traces for more accurate spatial question answering. TRACE encodes meta-context, camera trajectories, and detailed object entities to support structured spatial reasoning over egocentric videos. Extensive experiments on VSI-Bench and OST-Bench demonstrate that TRACE yields notable and consistent improvements over prior prompting strategies across a diverse range of MLLM backbones, spanning different parameter scales and training schemas. We further present ablation studies to validate our design choices, along with detailed analyses that probe the bottlenecks of 3D spatial reasoning in MLLMs.
69c3803695b48f16a30078e2
Jiacheng Hua
claimed_verified
2026-03-25T10:22:58.062Z
685dfe26aef7f289a3a085ba
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/uqdxT75mT22BUV6k4ZhXd.png
jiacheng
onlyfaces
user
onlyfaces
69c3803695b48f16a30078e3
Yishu Yin
69c3803695b48f16a30078e4
Yuhang Wu
69c3803695b48f16a30078e5
Tai Wang
69c3803695b48f16a30078e6
Yifei Huang
69c3803695b48f16a30078e7
Miao Liu
Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning
Existing Multimodal Large Language Models (MLLMs) struggle with 3D spatial reasoning, as they fail to construct structured abstractions of the 3D environment depicted in video inputs. To bridge this gap, drawing inspiration from cognitive theories of allocentric spatial reasoning, we investigate how to enable MLLMs to model and reason over text-based spatial representations of video. Specifically, we introduce Textual Representation of Allocentric Context from Egocentric Video (TRACE), a prompting method that induces MLLMs to generate text-based representations of 3D environments as intermediate reasoning traces for more accurate spatial question answering. TRACE encodes meta-context, camera trajectories, and detailed object entities to support structured spatial reasoning over egocentric videos. Extensive experiments on VSI-Bench and OST-Bench demonstrate that TRACE yields notable and consistent improvements over prior prompting strategies across a diverse range of MLLM backbones, spanning different parameter scales and training schemas. We further present ablation studies to validate our design choices, along with detailed analyses that probe the bottlenecks of 3D spatial reasoning in MLLMs.
TRACE is a prompting method that enables multimodal large language models to perform 3D spatial reasoning by generating text-based representations of video environments as intermediate reasoning traces.
Multimodal Large Language Models 3D spatial reasoning allocentric spatial reasoning egocentric video text-based representations spatial question answering prompting method meta-context camera trajectories object entities VSI-Bench OST-Bench