Towards Pixel-Level VLM Perception via Simple Points Prediction
We present SimpleSeg, a strikingly simple yet highly effective approach to endow Multimodal Large Language Models (MLLMs) with native pixel-level perception. Our method reframes segmentation as a simple sequence generation problem: the model directly predicts sequences of points (textual coordinates) delineating object boundaries, entirely within its language space. To achieve high fidelity, we introduce a two-stage SFtoRL training pipeline, where Reinforcement Learning with an IoU-based reward refines the point sequences to accurately match ground-truth contours. We find that the standard MLLM architecture possesses a strong, inherent capacity for low-level perception that can be unlocked without any specialized architecture. On segmentation benchmarks, SimpleSeg achieves performance that is comparable to, and often surpasses, methods relying on complex, task-specific designs. This work lays out that precise spatial understanding can emerge from simple point prediction, challenging the prevailing need for auxiliary components and paving the way for more unified and capable VLMs. Homepage: https://simpleseg.github.io/
6979b0b9df44b75fa47e486d
Tianhui Song
claimed_verified
2026-01-29T09:17:07.933Z
649e7693a83143427691769c
/avatars/d04f7b3d417423abaa053375212da21f.svg
Tianhui Song
sthui
user
sthui
6979b0b9df44b75fa47e486e
Haoyu Lu
6979b0b9df44b75fa47e486f
Hao Yang
6979b0b9df44b75fa47e4870
Lin Sui
6979b0b9df44b75fa47e4871
Haoning Wu
6979b0b9df44b75fa47e4872
Zaida Zhou
6979b0b9df44b75fa47e4873
Zhiqi Huang
6979b0b9df44b75fa47e4874
Yiping Bao
6979b0b9df44b75fa47e4875
Y. Charles
6979b0b9df44b75fa47e4876
Xinyu Zhou
6979b0b9df44b75fa47e4877
Limin Wang
Towards Pixel-Level VLM Perception via Simple Points Prediction
We present SimpleSeg, a strikingly simple yet highly effective approach to endow Multimodal Large Language Models (MLLMs) with native pixel-level perception. Our method reframes segmentation as a simple sequence generation problem: the model directly predicts sequences of points (textual coordinates) delineating object boundaries, entirely within its language space. To achieve high fidelity, we introduce a two-stage SFtoRL training pipeline, where Reinforcement Learning with an IoU-based reward refines the point sequences to accurately match ground-truth contours. We find that the standard MLLM architecture possesses a strong, inherent capacity for low-level perception that can be unlocked without any specialized architecture. On segmentation benchmarks, SimpleSeg achieves performance that is comparable to, and often surpasses, methods relying on complex, task-specific designs. This work lays out that precise spatial understanding can emerge from simple point prediction, challenging the prevailing need for auxiliary components and paving the way for more unified and capable VLMs. Homepage: https://simpleseg.github.io/
SimpleSeg enables multimodal large language models to perform pixel-level segmentation by predicting point sequences within language space, achieving competitive results without specialized architectures.
Multimodal Large Language Models segmentation sequence generation point prediction Reinforcement Learning IoU-based reward two-stage training pixel-level perception language space