UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding
GUI grounding, which localizes interface elements from screenshots given natural language queries, remains challenging for small icons and dense layouts. Test-time zoom-in methods improve localization by cropping and re-running inference at higher resolution, but apply cropping uniformly across all instances with fixed crop sizes, ignoring whether the model is actually uncertain on each case. We propose UI-Zoomer, a training-free adaptive zoom-in framework that treats both the trigger and scale of zoom-in as a prediction uncertainty quantification problem. A confidence-aware gate fuses spatial consensus among stochastic candidates with token-level generation confidence to selectively trigger zoom-in only when localization is uncertain. When triggered, an uncertainty-driven crop sizing module decomposes prediction variance into inter-sample positional spread and intra-sample box extent, deriving a per-instance crop radius via the law of total variance. Extensive experiments on ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2 demonstrate consistent improvements over strong baselines across multiple model architectures, achieving gains of up to +13.4\%, +10.3\%, and +4.2\% respectively, with no additional training required.
69e042155b704445ba50b65b
Fei Tang
69e042155b704445ba50b65c
Bofan Chen
69e042155b704445ba50b65d
Zhengxi Lu
claimed_verified
2026-04-16T08:01:44.822Z
676127cf11b19ea602bb202a
/avatars/dfd802a24bd63e509728159ebb1769f6.svg
Zhengxi Lu
LZXzju
user
LZXzju
69e042155b704445ba50b65e
Tongbo Chen
69e042155b704445ba50b65f
Songqin Nong
69e042155b704445ba50b660
Tao Jiang
69e042155b704445ba50b661
Wenhao Xu
69e042155b704445ba50b662
Weiming Lu
69e042155b704445ba50b663
Jun Xiao
69e042155b704445ba50b664
Yueting Zhuang
69e042155b704445ba50b665
Yongliang Shen
UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding
GUI grounding, which localizes interface elements from screenshots given natural language queries, remains challenging for small icons and dense layouts. Test-time zoom-in methods improve localization by cropping and re-running inference at higher resolution, but apply cropping uniformly across all instances with fixed crop sizes, ignoring whether the model is actually uncertain on each case. We propose UI-Zoomer, a training-free adaptive zoom-in framework that treats both the trigger and scale of zoom-in as a prediction uncertainty quantification problem. A confidence-aware gate fuses spatial consensus among stochastic candidates with token-level generation confidence to selectively trigger zoom-in only when localization is uncertain. When triggered, an uncertainty-driven crop sizing module decomposes prediction variance into inter-sample positional spread and intra-sample box extent, deriving a per-instance crop radius via the law of total variance. Extensive experiments on ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2 demonstrate consistent improvements over strong baselines across multiple model architectures, achieving gains of up to +13.4\%, +10.3\%, and +4.2\% respectively, with no additional training required.
UI-Zoomer is a training-free adaptive zoom-in framework for GUI grounding that improves localization accuracy by selectively triggering zoom-in based on prediction uncertainty quantification.
GUI grounding test-time zoom-in natural language queries stochastic candidates token-level generation confidence confidence-aware gate uncertainty quantification prediction variance inter-sample positional spread intra-sample box extent law of total variance