POINTS-GUI-G: GUI-Grounding Journey
The rapid advancement of vision-language models has catalyzed the emergence of GUI agents, which hold immense potential for automating complex tasks, from online shopping to flight booking, thereby alleviating the burden of repetitive digital workflows. As a foundational capability, GUI grounding is typically established as a prerequisite for end-to-end task execution. It enables models to precisely locate interface elements, such as text and icons, to perform accurate operations like clicking and typing. Unlike prior works that fine-tune models already possessing strong spatial awareness (e.g., Qwen3-VL), we aim to master the full technical pipeline by starting from a base model with minimal grounding ability, such as POINTS-1.5. We introduce POINTS-GUI-G-8B, which achieves state-of-the-art performance with scores of 59.9 on ScreenSpot-Pro, 66.0 on OSWorld-G, 95.7 on ScreenSpot-v2, and 49.9 on UI-Vision. Our model's success is driven by three key factors: (1) Refined Data Engineering, involving the unification of diverse open-source datasets format alongside sophisticated strategies for augmentation, filtering, and difficulty grading; (2) Improved Training Strategies, including continuous fine-tuning of the vision encoder to enhance perceptual accuracy and maintaining resolution consistency between training and inference; and (3) Reinforcement Learning (RL) with Verifiable Rewards. While RL is traditionally used to bolster reasoning, we demonstrate that it significantly improves precision in the perception-intensive GUI grounding task. Furthermore, GUI grounding provides a natural advantage for RL, as rewards are easily verifiable and highly accurate.
69894bc6beecc443208d25c0
Zhongyin Zhao
69894bc6beecc443208d25c1
Yuan Liu
claimed_verified
2026-02-09T08:29:55.119Z
64d47a7a508a6313e33faedd
https://cdn-avatars.huggingface.co/v1/production/uploads/64d47a7a508a6313e33faedd/8DrbXdxA-sKZb4jgNhvY1.jpeg
Yuan Liu
YuanLiuuuuuu
user
YuanLiuuuuuu
69894bc6beecc443208d25c2
Yikun Liu
69894bc6beecc443208d25c3
Haicheng Wang
69894bc6beecc443208d25c4
Le Tian
69894bc6beecc443208d25c5
Xiao Zhou
69894bc6beecc443208d25c6
Yangxiu You
69894bc6beecc443208d25c7
Zilin Yu
69894bc6beecc443208d25c8
Yang Yu
69894bc6beecc443208d25c9
Jie Zhou
POINTS-GUI-G: GUI-Grounding Journey
The rapid advancement of vision-language models has catalyzed the emergence of GUI agents, which hold immense potential for automating complex tasks, from online shopping to flight booking, thereby alleviating the burden of repetitive digital workflows. As a foundational capability, GUI grounding is typically established as a prerequisite for end-to-end task execution. It enables models to precisely locate interface elements, such as text and icons, to perform accurate operations like clicking and typing. Unlike prior works that fine-tune models already possessing strong spatial awareness (e.g., Qwen3-VL), we aim to master the full technical pipeline by starting from a base model with minimal grounding ability, such as POINTS-1.5. We introduce POINTS-GUI-G-8B, which achieves state-of-the-art performance with scores of 59.9 on ScreenSpot-Pro, 66.0 on OSWorld-G, 95.7 on ScreenSpot-v2, and 49.9 on UI-Vision. Our model's success is driven by three key factors: (1) Refined Data Engineering, involving the unification of diverse open-source datasets format alongside sophisticated strategies for augmentation, filtering, and difficulty grading; (2) Improved Training Strategies, including continuous fine-tuning of the vision encoder to enhance perceptual accuracy and maintaining resolution consistency between training and inference; and (3) Reinforcement Learning (RL) with Verifiable Rewards. While RL is traditionally used to bolster reasoning, we demonstrate that it significantly improves precision in the perception-intensive GUI grounding task. Furthermore, GUI grounding provides a natural advantage for RL, as rewards are easily verifiable and highly accurate.
GUI agents for automated digital tasks rely on vision-language models with enhanced grounding capabilities, achieved through refined data engineering, improved training strategies, and reinforcement learning with verifiable rewards.
vision-language models GUI agents GUI grounding ScreenSpot-Pro OSWorld-G ScreenSpot-v2 UI-Vision data engineering training strategies reinforcement learning verifiable rewards