Perceptio: Perception Enhanced Vision Language Models via Spatial Token Generation
Large Vision Language Models (LVLMs) excel at semantic understanding but struggle with fine grained spatial grounding, as the model must implicitly infer complex geometry without ever producing a spatial interpretation. We present Perceptio, a perception enhanced LVLM with 2D and 3D spatial reasoning abilities, enabled via explicit semantic segmentation tokens and depth tokens generated directly within the autoregressive sequence. Concretely, we (i) distill a VQVAE depth codebook from a strong monocular teacher to tokenize dense depth into compact sequences, and (ii) integrate SAM2 based semantic segmentation tokens and VQ-VAE depth tokens inside the LLM so the model first emits spatial tokens and then answers. To stabilize depth token generation, we introduce novel composite depth-token objectives (marker, token, and count losses) and a soft-merging technique for differentiable reconstruction. We adopt a multi-task co-training strategy across diverse datasets, letting the model learn perception tokens to tackle multiple downstream tasks. Building on InternVL, Perceptio achieves state-of-the-art performance across benchmarks: improving referring expression segmentation by +0.8/+1.4/+1.1 cIoU on RefCOCO/+/g HardBLINK spatial understanding accuracy by 10.3%, and MMBench accuracy by 1.0%, demonstrating that explicit spatial chain-of-thought materially strengthens spatial grounding in LVLMs.
69c2006fc105f45cb0481e5f
Yuchen Li
claimed_verified
2026-03-25T10:51:22.362Z
6360ea87da0599dc08c09cab
https://cdn-avatars.huggingface.co/v1/production/uploads/1667295877014-noauth.png
Yuchen Li
yuchenlichuck
user
yuchenlichuck
69c2006fc105f45cb0481e60
Amanmeet Garg
69c2006fc105f45cb0481e61
Shalini Chaudhuri
claimed_verified
2026-03-24T09:21:55.409Z
69c21deb729327c2109b5a2c
/avatars/431147464026b069176095dd6a88351c.svg
Shalini Chaudhuri
shalinichaudhuri
user
shalinichaudhuri
69c2006fc105f45cb0481e62
Rui Zhao
69c2006fc105f45cb0481e63
Garin Kessler
Perceptio: Perception Enhanced Vision Language Models via Spatial Token Generation
Large Vision Language Models (LVLMs) excel at semantic understanding but struggle with fine grained spatial grounding, as the model must implicitly infer complex geometry without ever producing a spatial interpretation. We present Perceptio, a perception enhanced LVLM with 2D and 3D spatial reasoning abilities, enabled via explicit semantic segmentation tokens and depth tokens generated directly within the autoregressive sequence. Concretely, we (i) distill a VQVAE depth codebook from a strong monocular teacher to tokenize dense depth into compact sequences, and (ii) integrate SAM2 based semantic segmentation tokens and VQ-VAE depth tokens inside the LLM so the model first emits spatial tokens and then answers. To stabilize depth token generation, we introduce novel composite depth-token objectives (marker, token, and count losses) and a soft-merging technique for differentiable reconstruction. We adopt a multi-task co-training strategy across diverse datasets, letting the model learn perception tokens to tackle multiple downstream tasks. Building on InternVL, Perceptio achieves state-of-the-art performance across benchmarks: improving referring expression segmentation by +0.8/+1.4/+1.1 cIoU on RefCOCO/+/g HardBLINK spatial understanding accuracy by 10.3%, and MMBench accuracy by 1.0%, demonstrating that explicit spatial chain-of-thought materially strengthens spatial grounding in LVLMs.
Perceptio enhances vision-language models with explicit spatial reasoning through integrated semantic segmentation and depth tokens generated via VQ-VAE distillation and multi-task learning.
VQVAE depth codebook semantic segmentation tokens autoregressive sequence SAM2 multi-task co-training spatial reasoning referring expression segmentation cIoU spatial understanding accuracy MMBench