PixelRefer: A Unified Framework for Spatio-Temporal Object Referring
  with Arbitrary Granularity
Multimodal large language models (MLLMs) have demonstrated strong
general-purpose capabilities in open-world visual comprehension. However, most
existing MLLMs primarily focus on holistic, scene-level understanding, often
overlooking the need for fine-grained, object-centric reasoning. In this paper,
we present PixelRefer, a unified region-level MLLM framework that enables
advanced fine-grained understanding over user-specified regions across both
images and videos. Motivated by the observation that LLM attention
predominantly focuses on object-level tokens, we propose a Scale-Adaptive
Object Tokenizer (SAOT) to generate compact and semantically rich object
representations from free-form regions. Our analysis reveals that global visual
tokens contribute mainly in early LLM layers, inspiring the design of
PixelRefer-Lite, an efficient variant that employs an Object-Centric Infusion
module to pre-fuse global context into object tokens. This yields a lightweight
Object-Only Framework that substantially reduces computational cost while
maintaining high semantic fidelity. To facilitate fine-grained instruction
tuning, we curate PixelRefer-2.2M, a high-quality object-centric instruction
dataset. Extensive experiments across a range of benchmarks validate that
PixelRefer achieves leading performance with fewer training samples, while
PixelRefer-Lite offers competitive accuracy with notable gains in efficiency.
6900320522d452aac6dd4309
Yuqian Yuan
claimed_verified
2025-10-28T15:36:36.571Z
64a3fe3dde901eb01df12398
https://cdn-avatars.huggingface.co/v1/production/uploads/64a3fe3dde901eb01df12398/Js2bEx4rxKuEKVt5z9I2D.jpeg
YuqianYuan
CircleRadon
user
CircleRadon
6900320522d452aac6dd430a
Wenqiao Zhang
6900320522d452aac6dd430b
Xin Li
6900320522d452aac6dd430c
Shihao Wang
6900320522d452aac6dd430d
Kehan Li
6900320522d452aac6dd430e
Wentong Li
claimed_verified
2025-11-18T14:10:01.026Z
64c48a78d07620bdc99777d4
https://cdn-avatars.huggingface.co/v1/production/uploads/64c48a78d07620bdc99777d4/NJC4Ot0a7YSdU5RC6dgga.jpeg
LI WENTONG
sunshine-lwt
user
sunshine-lwt
6900320522d452aac6dd430f
Jun Xiao
6900320522d452aac6dd4310
Lei Zhang
6900320522d452aac6dd4311
Beng Chin Ooi
PixelRefer: A Unified Framework for Spatio-Temporal Object Referring
  with Arbitrary Granularity
Multimodal large language models (MLLMs) have demonstrated strong
general-purpose capabilities in open-world visual comprehension. However, most
existing MLLMs primarily focus on holistic, scene-level understanding, often
overlooking the need for fine-grained, object-centric reasoning. In this paper,
we present PixelRefer, a unified region-level MLLM framework that enables
advanced fine-grained understanding over user-specified regions across both
images and videos. Motivated by the observation that LLM attention
predominantly focuses on object-level tokens, we propose a Scale-Adaptive
Object Tokenizer (SAOT) to generate compact and semantically rich object
representations from free-form regions. Our analysis reveals that global visual
tokens contribute mainly in early LLM layers, inspiring the design of
PixelRefer-Lite, an efficient variant that employs an Object-Centric Infusion
module to pre-fuse global context into object tokens. This yields a lightweight
Object-Only Framework that substantially reduces computational cost while
maintaining high semantic fidelity. To facilitate fine-grained instruction
tuning, we curate PixelRefer-2.2M, a high-quality object-centric instruction
dataset. Extensive experiments across a range of benchmarks validate that
PixelRefer achieves leading performance with fewer training samples, while
PixelRefer-Lite offers competitive accuracy with notable gains in efficiency.
PixelRefer is a unified region-level multimodal large language model framework that enhances fine-grained object-centric understanding and efficiency through a Scale-Adaptive Object Tokenizer and Object-Centric Infusion module.
multimodal large language models MLLMs PixelRefer region-level understanding object-centric reasoning Scale-Adaptive Object Tokenizer SAOT object tokens global visual tokens PixelRefer-Lite Object-Centric Infusion module object-centric instruction dataset PixelRefer-2.2M