UI-Ins: Enhancing GUI Grounding with Multi-Perspective
  Instruction-as-Reasoning
GUI grounding, which maps natural-language instructions to actionable UI
elements, is a core capability of GUI agents. Prior works largely treats
instructions as a static proxy for user intent, overlooking the impact of
instruction diversity and quality on grounding performance. Through a careful
investigation of existing grounding datasets, we find a 23.3% flaw rate in
their instructions and show that inference-time exploitation of instruction
diversity yields up to a substantial 76% relative performance improvement. In
this paper, we introduce the Instruction-as-Reasoning paradigm, treating
instructions as dynamic analytical pathways that offer distinct perspectives
and enabling the model to select the most effective pathway during reasoning.
To achieve this, we propose a two-stage training framework: supervised
fine-tuning (SFT) on synthesized, diverse instructions to instill
multi-perspective reasoning, followed by reinforcement learning (RL) to
optimize pathway selection and composition. Our resulting models, UI-Ins-7B and
UI-Ins-32B, achieve state-of-the-art results on five challenging grounding
benchmarks and exhibit emergent reasoning, selectively composing and
synthesizing novel instruction pathways at inference. In particular, UI-Ins-32B
attains the best grounding accuracy, scoring 87.3% on UI-I2E-Bench, 57.0% on
ScreenSpot-Pro, and 84.9% on MMBench-GUI L2. Furthermore, our model
demonstrates strong agentic potential, achieving a 74.1% success rate on
AndroidWorld using UI-Ins-7B as the executor. Our in-depth analysis reveals
additional insights such as how reasoning can be formulated to enhance rather
than hinder grounding performance, and how our method mitigates policy collapse
in the SFT+RL framework. All code and model checkpoints will be publicly
released in https://github.com/alibaba/UI-Ins.
68fb1853f158a71c5a2f594c
Liangyu Chen
claimed_verified
2025-10-27T10:29:20.509Z
68fb2e8097199a90be333fe6
/avatars/b798b45e53d23bc6d291b6030653c8b6.svg
Liangyu Chen
CleyChen
user
CleyChen
68fb1853f158a71c5a2f594d
Hanzhang Zhou
68fb1853f158a71c5a2f594e
Chenglin Cai
68fb1853f158a71c5a2f594f
Jianan Zhang
claimed_verified
2025-10-24T16:00:57.725Z
6825ddbfcbbc52f7ad9aa4d1
/avatars/53def7ee23686a75a976a16d7533856f.svg
zhang
Jiananhello
user
Jiananhello
68fb1853f158a71c5a2f5950
Panrong Tong
68fb1853f158a71c5a2f5951
Quyu Kong
claimed_verified
2025-10-24T16:00:53.633Z
638c05b7d274cbbad284ced0
/avatars/e26f659316b4975562e006c081a795ba.svg
Quyu Kong
kongquyu
user
kongquyu
68fb1853f158a71c5a2f5952
Xu Zhang
claimed_verified
2025-12-25T20:54:43.603Z
63525c3a6cfb8f1498127a34
/avatars/7054d8116af1225996a9caf1516a002c.svg
Xu Zhang
Xu0307
user
Xu0307
68fb1853f158a71c5a2f5953
Chen Liu
68fb1853f158a71c5a2f5954
Yuqi Liu
claimed_verified
2025-10-27T10:29:22.553Z
669cefd6119595d21b55a995
/avatars/bafc2387ee70b263bf45c42159381da8.svg
Yuqi Liu
Ricky06662
user
Ricky06662
68fb1853f158a71c5a2f5955
Wenxuan Wang
68fb1853f158a71c5a2f5956
Yue Wang
68fb1853f158a71c5a2f5957
Qin Jin
68fb1853f158a71c5a2f5958
Steven Hoi
UI-Ins: Enhancing GUI Grounding with Multi-Perspective
  Instruction-as-Reasoning
GUI grounding, which maps natural-language instructions to actionable UI
elements, is a core capability of GUI agents. Prior works largely treats
instructions as a static proxy for user intent, overlooking the impact of
instruction diversity and quality on grounding performance. Through a careful
investigation of existing grounding datasets, we find a 23.3% flaw rate in
their instructions and show that inference-time exploitation of instruction
diversity yields up to a substantial 76% relative performance improvement. In
this paper, we introduce the Instruction-as-Reasoning paradigm, treating
instructions as dynamic analytical pathways that offer distinct perspectives
and enabling the model to select the most effective pathway during reasoning.
To achieve this, we propose a two-stage training framework: supervised
fine-tuning (SFT) on synthesized, diverse instructions to instill
multi-perspective reasoning, followed by reinforcement learning (RL) to
optimize pathway selection and composition. Our resulting models, UI-Ins-7B and
UI-Ins-32B, achieve state-of-the-art results on five challenging grounding
benchmarks and exhibit emergent reasoning, selectively composing and
synthesizing novel instruction pathways at inference. In particular, UI-Ins-32B
attains the best grounding accuracy, scoring 87.3% on UI-I2E-Bench, 57.0% on
ScreenSpot-Pro, and 84.9% on MMBench-GUI L2. Furthermore, our model
demonstrates strong agentic potential, achieving a 74.1% success rate on
AndroidWorld using UI-Ins-7B as the executor. Our in-depth analysis reveals
additional insights such as how reasoning can be formulated to enhance rather
than hinder grounding performance, and how our method mitigates policy collapse
in the SFT+RL framework. All code and model checkpoints will be publicly
released in https://github.com/alibaba/UI-Ins.
The Instruction-as-Reasoning paradigm enhances GUI grounding by treating instructions as dynamic pathways, improving performance through multi-perspective reasoning and reinforcement learning.
GUI grounding Instruction-as-Reasoning supervised fine-tuning reinforcement learning multi-perspective reasoning pathway selection policy collapse