MolmoPoint: Better Pointing for VLMs with Grounding Tokens
Grounding has become a fundamental capability of vision-language models (VLMs). Most existing VLMs point by generating coordinates as part of their text output, which requires learning a complicated coordinate system and results in a high token count. Instead, we propose a more intuitive pointing mechanism that directly selects the visual tokens that contain the target concept. Our model generates a special pointing token that cross-attends to the input image or video tokens and selects the appropriate one. To make this model more fine-grained, we follow these pointing tokens with an additional special token that selects a fine-grained subpatch within the initially selected region, and then a third token that specifies a location within that subpatch. We further show that performance improves by generating points sequentially in a consistent order, encoding the relative position of the previously selected point, and including a special no-more-points class when selecting visual tokens. Using this method, we set a new state-of-the-art on image pointing (70.7% on PointBench), set a new state-of-the-art among fully open models on GUI pointing (61.1% on ScreenSpotPro), and improve video pointing (59.1% human preference win rate vs. a text coordinate baseline) and tracking (+6.3% gain on Molmo2Track). We additionally show that our method achieves much higher sample efficiency and discuss the qualitative differences that emerge from this design change.
69cb412922f9e9dce3fe200a
Christopher Clark
69cb412922f9e9dce3fe200b
Yue Yang
claimed_verified
2026-04-07T08:47:12.095Z
62f6c68904e5e02f82b04690
https://cdn-avatars.huggingface.co/v1/production/uploads/62f6c68904e5e02f82b04690/kK2-PkeAGzAOLhkfajswf.jpeg
Yue Yang
yyupenn
user
yyupenn
69cb412922f9e9dce3fe200c
Jae Sung Park
69cb412922f9e9dce3fe200d
Zixian Ma
69cb412922f9e9dce3fe200e
Jieyu Zhang
69cb412922f9e9dce3fe200f
Rohun Tripathi
admin_assigned
2026-03-31T09:49:29.252Z
646e5aa2cb6ea6e6b6df2a5f
/avatars/e36e83efa0a08f64e308a18b08f463da.svg
Rohun Tripathi
roTripathi
user
roTripathi
69cb412922f9e9dce3fe2010
Mohammadreza Salehi
69cb412922f9e9dce3fe2011
Sangho Lee
69cb412922f9e9dce3fe2012
Taira Anderson
admin_assigned
2026-03-31T09:49:15.950Z
65de20ad4e73a7dea7fb4f08
/avatars/f3b0ad6cc9417e8ea3f0607fa62824d1.svg
Taira Anderson
tairaa
user
tairaa
69cb412922f9e9dce3fe2013
Winson Han
69cb412922f9e9dce3fe2014
Ranjay Krishna
admin_assigned
2026-03-31T09:49:06.444Z
66429868ab89e3a3a85668b0
/avatars/170e0daa454838deee2bf946f7118651.svg
Ranjay Krishna
ranjaykrishna
user
ranjaykrishna
MolmoPoint: Better Pointing for VLMs with Grounding Tokens
Grounding has become a fundamental capability of vision-language models (VLMs). Most existing VLMs point by generating coordinates as part of their text output, which requires learning a complicated coordinate system and results in a high token count. Instead, we propose a more intuitive pointing mechanism that directly selects the visual tokens that contain the target concept. Our model generates a special pointing token that cross-attends to the input image or video tokens and selects the appropriate one. To make this model more fine-grained, we follow these pointing tokens with an additional special token that selects a fine-grained subpatch within the initially selected region, and then a third token that specifies a location within that subpatch. We further show that performance improves by generating points sequentially in a consistent order, encoding the relative position of the previously selected point, and including a special no-more-points class when selecting visual tokens. Using this method, we set a new state-of-the-art on image pointing (70.7% on PointBench), set a new state-of-the-art among fully open models on GUI pointing (61.1% on ScreenSpotPro), and improve video pointing (59.1% human preference win rate vs. a text coordinate baseline) and tracking (+6.3% gain on Molmo2Track). We additionally show that our method achieves much higher sample efficiency and discuss the qualitative differences that emerge from this design change.
A vision-language model approach for grounding that directly selects visual tokens containing target concepts through specialized pointing tokens, achieving superior performance in image, GUI, video pointing, and tracking tasks.
vision-language models visual tokens cross-attends pointing tokens fine-grained subpatch relative position encoding no-more-points class PointBench ScreenSpotPro Molmo2Track