Token Warping Helps MLLMs Look from Nearby Viewpoints
Can warping tokens, rather than pixels, help multimodal large language models (MLLMs) understand how a scene appears from a nearby viewpoint? While MLLMs perform well on visual reasoning, they remain fragile to viewpoint changes, as pixel-wise warping is highly sensitive to small depth errors and often introduces geometric distortions. Drawing on theories of mental imagery that posit part-level structural representations as the basis for human perspective transformation, we examine whether image tokens in ViT-based MLLMs serve as an effective substrate for viewpoint changes. We compare forward and backward warping, finding that backward token warping, which defines a dense grid on the target view and retrieves a corresponding source-view token for each grid point, achieves greater stability and better preserves semantic coherence under viewpoint shifts. Experiments on our proposed ViewBench benchmark demonstrate that token-level warping enables MLLMs to reason reliably from nearby viewpoints, consistently outperforming all baselines including pixel-wise warping approaches, spatially fine-tuned MLLMs, and a generative warping method.
69d33c582c523d36431fad91
Phillip Y. Lee
69d33c582c523d36431fad92
Chanho Park
claimed_verified
2026-04-07T08:38:25.960Z
64b142d4f7f82164df73e9cd
https://cdn-avatars.huggingface.co/v1/production/uploads/64b142d4f7f82164df73e9cd/71jpuvYixc0RPbb-jhSjB.png
Chanho Park
charlieppark
user
charlieppark
69d33c582c523d36431fad93
Mingue Park
69d33c582c523d36431fad94
Seungwoo Yoo
69d33c582c523d36431fad95
Juil Koo
69d33c582c523d36431fad96
Minhyuk Sung
Token Warping Helps MLLMs Look from Nearby Viewpoints
Can warping tokens, rather than pixels, help multimodal large language models (MLLMs) understand how a scene appears from a nearby viewpoint? While MLLMs perform well on visual reasoning, they remain fragile to viewpoint changes, as pixel-wise warping is highly sensitive to small depth errors and often introduces geometric distortions. Drawing on theories of mental imagery that posit part-level structural representations as the basis for human perspective transformation, we examine whether image tokens in ViT-based MLLMs serve as an effective substrate for viewpoint changes. We compare forward and backward warping, finding that backward token warping, which defines a dense grid on the target view and retrieves a corresponding source-view token for each grid point, achieves greater stability and better preserves semantic coherence under viewpoint shifts. Experiments on our proposed ViewBench benchmark demonstrate that token-level warping enables MLLMs to reason reliably from nearby viewpoints, consistently outperforming all baselines including pixel-wise warping approaches, spatially fine-tuned MLLMs, and a generative warping method.
Token-level warping in vision-language models demonstrates superior stability and semantic coherence for viewpoint transformation compared to pixel-wise methods, achieving better visual reasoning performance.
warping tokens multimodal large language models viewpoint changes visual reasoning ViT-based models forward warping backward warping ViewBench benchmark semantic coherence geometric distortions