HyperClick: Advancing Reliable GUI Grounding via Uncertainty Calibration
Autonomous Graphical User Interface (GUI) agents rely on accurate GUI
grounding, which maps language instructions to on-screen coordinates, to
execute user commands. However, current models, whether trained via supervised
fine-tuning (SFT) or reinforcement fine-tuning (RFT), lack self-awareness of
their capability boundaries, leading to overconfidence and unreliable
predictions. We first systematically evaluate probabilistic and verbalized
confidence in general and GUI-specific models, revealing a misalignment between
confidence and actual accuracy, which is particularly critical in dynamic GUI
automation tasks, where single errors can cause task failure. To address this,
we propose HyperClick, a novel framework that enhances reliable GUI grounding
through uncertainty calibration. HyperClick introduces a dual reward mechanism,
combining a binary reward for correct actions with a truncated Gaussian-based
spatial confidence modeling, calibrated using the Brier score. This approach
jointly optimizes grounding accuracy and confidence reliability, fostering
introspective self-criticism. Extensive experiments on seven challenge
benchmarks show that HyperClick achieves state-of-the-art performance while
providing well-calibrated confidence. By enabling explicit confidence
calibration and introspective self-criticism, HyperClick reduces overconfidence
and supports more reliable GUI automation.
69081881812eca10f9cc5e4f
Shaojie Zhang
69081881812eca10f9cc5e50
Pei Fu
69081881812eca10f9cc5e51
Ruoceng Zhang
69081881812eca10f9cc5e52
Jiahui Yang
69081881812eca10f9cc5e53
Anan Du
69081881812eca10f9cc5e54
Xiuwen Xi
69081881812eca10f9cc5e55
Shaokang Wang
69081881812eca10f9cc5e56
Ying Huang
69081881812eca10f9cc5e57
Bin Qin
69081881812eca10f9cc5e58
Zhenbo Luo
69081881812eca10f9cc5e59
Jian Luan
HyperClick: Advancing Reliable GUI Grounding via Uncertainty Calibration
Autonomous Graphical User Interface (GUI) agents rely on accurate GUI
grounding, which maps language instructions to on-screen coordinates, to
execute user commands. However, current models, whether trained via supervised
fine-tuning (SFT) or reinforcement fine-tuning (RFT), lack self-awareness of
their capability boundaries, leading to overconfidence and unreliable
predictions. We first systematically evaluate probabilistic and verbalized
confidence in general and GUI-specific models, revealing a misalignment between
confidence and actual accuracy, which is particularly critical in dynamic GUI
automation tasks, where single errors can cause task failure. To address this,
we propose HyperClick, a novel framework that enhances reliable GUI grounding
through uncertainty calibration. HyperClick introduces a dual reward mechanism,
combining a binary reward for correct actions with a truncated Gaussian-based
spatial confidence modeling, calibrated using the Brier score. This approach
jointly optimizes grounding accuracy and confidence reliability, fostering
introspective self-criticism. Extensive experiments on seven challenge
benchmarks show that HyperClick achieves state-of-the-art performance while
providing well-calibrated confidence. By enabling explicit confidence
calibration and introspective self-criticism, HyperClick reduces overconfidence
and supports more reliable GUI automation.
HyperClick enhances GUI automation by calibrating confidence and reducing overconfidence through a dual reward mechanism and spatial confidence modeling.
GUI grounding supervised fine-tuning reinforcement fine-tuning probabilistic confidence verbalized confidence uncertainty calibration dual reward mechanism truncated Gaussian Brier score introspective self-criticism