Mobile GUI Agents under Real-world Threats: Are We There Yet?
Recent years have witnessed a rapid development of mobile GUI agents powered by large language models (LLMs), which can autonomously execute diverse device-control tasks based on natural language instructions. The increasing accuracy of these agents on standard benchmarks has raised expectations for large-scale real-world deployment, and there are already several commercial agents released and used by early adopters. However, are we really ready for GUI agents integrated into our daily devices as system building blocks? We argue that an important pre-deployment validation is missing to examine whether the agents can maintain their performance under real-world threats. Specifically, unlike existing common benchmarks that are based on simple static app contents (they have to do so to ensure environment consistency between different tests), real-world apps are filled with contents from untrustworthy third parties, such as advertisement emails, user-generated posts and medias, etc. ... To this end, we introduce a scalable app content instrumentation framework to enable flexible and targeted content modifications within existing applications. Leveraging this framework, we create a test suite comprising both a dynamic task execution environment and a static dataset of challenging GUI states. The dynamic environment encompasses 122 reproducible tasks, and the static dataset consists of over 3,000 scenarios constructed from commercial apps. We perform experiments on both open-source and commercial GUI agents. Our findings reveal that all examined agents can be significantly degraded due to third-party contents, with an average misleading rate of 42.0% and 36.1% in dynamic and static environments respectively. The framework and benchmark has been released at https://agenthazard.github.io.
69e03b995b704445ba50b57e
Guohong Liu
claimed_verified
2026-04-16T08:01:57.402Z
64549950c13cdb83f10ffd74
https://cdn-avatars.huggingface.co/v1/production/uploads/64549950c13cdb83f10ffd74/g4UJV_YGshdZmW9FZock4.png
Guohong Liu
Zacharyvixx
user
Zacharyvixx
69e03b995b704445ba50b57f
Jialei Ye
69e03b995b704445ba50b580
Jiacheng Liu
69e03b995b704445ba50b581
Yuanchun Li
69e03b995b704445ba50b582
Wei Liu
69e03b995b704445ba50b583
Pengzhi Gao
69e03b995b704445ba50b584
Jian Luan
69e03b995b704445ba50b585
Yunxin Liu
Mobile GUI Agents under Real-world Threats: Are We There Yet?
Recent years have witnessed a rapid development of mobile GUI agents powered by large language models (LLMs), which can autonomously execute diverse device-control tasks based on natural language instructions. The increasing accuracy of these agents on standard benchmarks has raised expectations for large-scale real-world deployment, and there are already several commercial agents released and used by early adopters. However, are we really ready for GUI agents integrated into our daily devices as system building blocks? We argue that an important pre-deployment validation is missing to examine whether the agents can maintain their performance under real-world threats. Specifically, unlike existing common benchmarks that are based on simple static app contents (they have to do so to ensure environment consistency between different tests), real-world apps are filled with contents from untrustworthy third parties, such as advertisement emails, user-generated posts and medias, etc. ... To this end, we introduce a scalable app content instrumentation framework to enable flexible and targeted content modifications within existing applications. Leveraging this framework, we create a test suite comprising both a dynamic task execution environment and a static dataset of challenging GUI states. The dynamic environment encompasses 122 reproducible tasks, and the static dataset consists of over 3,000 scenarios constructed from commercial apps. We perform experiments on both open-source and commercial GUI agents. Our findings reveal that all examined agents can be significantly degraded due to third-party contents, with an average misleading rate of 42.0% and 36.1% in dynamic and static environments respectively. The framework and benchmark has been released at https://agenthazard.github.io.
Mobile GUI agents powered by large language models show significant performance degradation when exposed to real-world third-party content in commercial applications.
mobile GUI agents large language models device-control tasks natural language instructions benchmarking app content instrumentation GUI states dynamic environment static dataset