GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor
  for GUI Grounding
Graphical user interface (GUI) grounding is a key function of computer-use
agents, which maps natural-language instructions to actionable screen regions.
Existing approaches based on Multimodal Large Language Models (MLLMs) typically
formulate it as a text-based coordinate generation task, yet directly
generating precise coordinates from visual inputs remains challenging and
computationally intensive. An intuitive way to implement GUI grounding is to
first select visual patches relevant to the instructions and then determine the
precise click location within those patches. Based on the observations that
general MLLMs have some native grounding capability, nested within their
attentions, we propose GUI-AIMA, an attention-based and coordinate-free
supervised fine-tuning framework for efficient GUI grounding. GUI-AIMA aligns
the intrinsic multimodal attention of MLLMs with patch-wise grounding signals.
These signals are calculated adaptively for diverse user instructions by
multi-head aggregation on simplified query-visual attention matrices. Besides,
its coordinate-free manner can easily integrate a plug-and-play zoom-in stage.
GUI-AIMA-3B was trained with only 85k screenshots, demonstrating exceptional
data efficiency and verifying that light training can trigger the native
grounding capability of MLLMs. It achieves state-of-the-art performance among
3B models, attaining an average accuracy of 58.6% on ScreenSpot-Pro and 62.2%
on OSWorld-G. Project page: https://github.com/sjz5202/GUI-AIMA
690a6cb3d70e173c84528f0c
Shijie Zhou
690a6cb3d70e173c84528f0d
Viet Dac Lai
690a6cb3d70e173c84528f0e
Hao Tan
690a6cb3d70e173c84528f0f
Jihyung Kil
690a6cb3d70e173c84528f10
Wanrong Zhu
690a6cb3d70e173c84528f11
Changyou Chen
690a6cb3d70e173c84528f12
Ruiyi Zhang
GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor
  for GUI Grounding
Graphical user interface (GUI) grounding is a key function of computer-use
agents, which maps natural-language instructions to actionable screen regions.
Existing approaches based on Multimodal Large Language Models (MLLMs) typically
formulate it as a text-based coordinate generation task, yet directly
generating precise coordinates from visual inputs remains challenging and
computationally intensive. An intuitive way to implement GUI grounding is to
first select visual patches relevant to the instructions and then determine the
precise click location within those patches. Based on the observations that
general MLLMs have some native grounding capability, nested within their
attentions, we propose GUI-AIMA, an attention-based and coordinate-free
supervised fine-tuning framework for efficient GUI grounding. GUI-AIMA aligns
the intrinsic multimodal attention of MLLMs with patch-wise grounding signals.
These signals are calculated adaptively for diverse user instructions by
multi-head aggregation on simplified query-visual attention matrices. Besides,
its coordinate-free manner can easily integrate a plug-and-play zoom-in stage.
GUI-AIMA-3B was trained with only 85k screenshots, demonstrating exceptional
data efficiency and verifying that light training can trigger the native
grounding capability of MLLMs. It achieves state-of-the-art performance among
3B models, attaining an average accuracy of 58.6% on ScreenSpot-Pro and 62.2%
on OSWorld-G. Project page: https://github.com/sjz5202/GUI-AIMA
GUI-AIMA, an attention-based and coordinate-free framework, enhances GUI grounding by aligning MLLM attention with patch-wise signals, achieving state-of-the-art performance with minimal training data.
Multimodal Large Language Models MLLMs GUI grounding attention-based coordinate-free supervised fine-tuning patch-wise grounding multi-head aggregation query-visual attention matrices data efficiency native grounding capability ScreenSpot-Pro OSWorld-G