SAMTok: Representing Any Mask with Two Words
Pixel-wise capabilities are essential for building interactive intelligent systems. However, pixel-wise multi-modal LLMs (MLLMs) remain difficult to scale due to complex region-level encoders, specialized segmentation decoders, and incompatible training objectives. To address these challenges, we present SAMTok, a discrete mask tokenizer that converts any region mask into two special tokens and reconstructs the mask using these tokens with high fidelity. By treating masks as new language tokens, SAMTok enables base MLLMs (such as the QwenVL series) to learn pixel-wise capabilities through standard next-token prediction and simple reinforcement learning, without architectural modifications and specialized loss design. SAMTok builds on SAM2 and is trained on 209M diverse masks using a mask encoder and residual vector quantizer to produce discrete, compact, and information-rich tokens. With 5M SAMTok-formatted mask understanding and generation data samples, QwenVL-SAMTok attains state-of-the-art or comparable results on region captioning, region VQA, grounded conversation, referring segmentation, scene graph parsing, and multi-round interactive segmentation. We further introduce a textual answer-matching reward that enables efficient reinforcement learning for mask generation, delivering substantial improvements on GRES and GCG benchmarks. Our results demonstrate a scalable and straightforward paradigm for equipping MLLMs with strong pixel-wise capabilities. Our code and models are available.
6972f00ffb12c92b735b74bb
Yikang Zhou
claimed_verified
2026-01-23T09:38:04.804Z
644d0b5c6dfd5f8240d95f5e
/avatars/c5bcc5691fe8ec3e24558afb80feecab.svg
zhou yikang
zhouyik
user
zhouyik
6972f00ffb12c92b735b74bc
Tao Zhang
6972f00ffb12c92b735b74bd
Dengxian Gong
claimed_verified
2026-01-23T09:38:21.767Z
66350ea032a5f38150f7a82b
https://cdn-avatars.huggingface.co/v1/production/uploads/66350ea032a5f38150f7a82b/UOrv46KJsQV9Ye_bmyIS6.jpeg
GongDengxian
godx7
user
godx7
6972f00ffb12c92b735b74be
Yuanzheng Wu
6972f00ffb12c92b735b74bf
Ye Tian
6972f00ffb12c92b735b74c0
Haochen Wang
6972f00ffb12c92b735b74c1
Haobo Yuan
6972f00ffb12c92b735b74c2
Jiacong Wang
claimed_verified
2026-03-12T09:08:18.016Z
64d201b1c2bd235422fb1d14
/avatars/e50581aa66391cedae94e116e759b9ec.svg
wang
stormthunder
user
stormthunder
6972f00ffb12c92b735b74c3
Lu Qi
6972f00ffb12c92b735b74c4
Hao Fei
claimed_verified
2026-03-09T10:40:51.003Z
647773a1168cb428e00e9a8f
https://cdn-avatars.huggingface.co/v1/production/uploads/647773a1168cb428e00e9a8f/NiRR3ScY6Plzjibfwy1hC.jpeg
Hao Fei
scofield7419
user
scofield7419
6972f00ffb12c92b735b74c5
Anran Wang
6972f00ffb12c92b735b74c6
Zhuochen Wang
6972f00ffb12c92b735b74c7
Yujing Wang
6972f00ffb12c92b735b74c8
Cheng Chen
6972f00ffb12c92b735b74c9
Shunping Ji
6972f00ffb12c92b735b74ca
Xiangtai Li
SAMTok: Representing Any Mask with Two Words
Pixel-wise capabilities are essential for building interactive intelligent systems. However, pixel-wise multi-modal LLMs (MLLMs) remain difficult to scale due to complex region-level encoders, specialized segmentation decoders, and incompatible training objectives. To address these challenges, we present SAMTok, a discrete mask tokenizer that converts any region mask into two special tokens and reconstructs the mask using these tokens with high fidelity. By treating masks as new language tokens, SAMTok enables base MLLMs (such as the QwenVL series) to learn pixel-wise capabilities through standard next-token prediction and simple reinforcement learning, without architectural modifications and specialized loss design. SAMTok builds on SAM2 and is trained on 209M diverse masks using a mask encoder and residual vector quantizer to produce discrete, compact, and information-rich tokens. With 5M SAMTok-formatted mask understanding and generation data samples, QwenVL-SAMTok attains state-of-the-art or comparable results on region captioning, region VQA, grounded conversation, referring segmentation, scene graph parsing, and multi-round interactive segmentation. We further introduce a textual answer-matching reward that enables efficient reinforcement learning for mask generation, delivering substantial improvements on GRES and GCG benchmarks. Our results demonstrate a scalable and straightforward paradigm for equipping MLLMs with strong pixel-wise capabilities. Our code and models are available.
SAMTok enables pixel-wise capabilities in multi-modal large language models through discrete mask tokenization and standard training methods, achieving state-of-the-art performance on various vision-language tasks.
multi-modal LLMs region mask discrete mask tokenizer SAM2 mask encoder residual vector quantizer next-token prediction reinforcement learning region captioning region VQA grounded conversation referring segmentation scene graph parsing multi-round interactive segmentation textual answer-matching reward GRES GCG