MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive, and MCP-Augmented Environments
Among existing online mobile-use benchmarks, AndroidWorld has emerged as the dominant benchmark due to its reproducible environment and deterministic evaluation; however, recent agents achieving over 90% success rates indicate its saturation and motivate the need for a more challenging benchmark. In addition, its environment lacks key application categories, such as e-commerce and enterprise communication, and does not reflect realistic mobile-use scenarios characterized by vague user instructions and hybrid tool usage. To bridge this gap, we introduce MobileWorld, a substantially more challenging benchmark designed to better reflect real-world mobile usage, comprising 201 tasks across 20 applications, while maintaining the same level of reproducible evaluation as AndroidWorld. The difficulty of MobileWorld is twofold. First, it emphasizes long-horizon tasks with cross-application interactions: MobileWorld requires nearly twice as many task-completion steps on average (27.8 vs. 14.3) and includes far more multi-application tasks (62.2% vs. 9.5%) compared to AndroidWorld. Second, MobileWorld extends beyond standard GUI manipulation by introducing novel task categories, including agent-user interaction and MCP-augmented tasks. To ensure robust evaluation, we provide snapshot-based container environment and precise functional verifications, including backend database inspection and task callback APIs. We further develop a planner-executor agentic framework with extended action spaces to support user interactions and MCP calls. Our results reveal a sharp performance drop compared to AndroidWorld, with the best agentic framework and end-to-end model achieving 51.7% and 20.9% success rates, respectively. Our analysis shows that current models struggle significantly with user interaction and MCP calls, offering a strategic roadmap toward more robust, next-generation mobile intelligence.
694a0e4e335742716e932266
Quyu Kong
claimed_verified
2026-01-13T15:46:01.314Z
638c05b7d274cbbad284ced0
/avatars/e26f659316b4975562e006c081a795ba.svg
Quyu Kong
kongquyu
user
kongquyu
694a0e4e335742716e932267
Xu Zhang
claimed_verified
2025-12-25T20:48:44.155Z
63525c3a6cfb8f1498127a34
/avatars/7054d8116af1225996a9caf1516a002c.svg
Xu Zhang
Xu0307
user
Xu0307
694a0e4e335742716e932268
Zhenyu Yang
694a0e4e335742716e932269
Nolan Gao
694a0e4e335742716e93226a
Chen Liu
694a0e4e335742716e93226b
Panrong Tong
694a0e4e335742716e93226c
Chenglin Cai
694a0e4e335742716e93226d
Hanzhang Zhou
694a0e4e335742716e93226e
Jianan Zhang
claimed_verified
2025-12-25T20:48:46.663Z
6825ddbfcbbc52f7ad9aa4d1
/avatars/53def7ee23686a75a976a16d7533856f.svg
zhang
Jiananhello
user
Jiananhello
694a0e4e335742716e93226f
Liangyu Chen
694a0e4e335742716e932270
Zhidan Liu
694a0e4e335742716e932271
Steven Hoi
694a0e4e335742716e932272
Yue Wang
MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive, and MCP-Augmented Environments
Among existing online mobile-use benchmarks, AndroidWorld has emerged as the dominant benchmark due to its reproducible environment and deterministic evaluation; however, recent agents achieving over 90% success rates indicate its saturation and motivate the need for a more challenging benchmark. In addition, its environment lacks key application categories, such as e-commerce and enterprise communication, and does not reflect realistic mobile-use scenarios characterized by vague user instructions and hybrid tool usage. To bridge this gap, we introduce MobileWorld, a substantially more challenging benchmark designed to better reflect real-world mobile usage, comprising 201 tasks across 20 applications, while maintaining the same level of reproducible evaluation as AndroidWorld. The difficulty of MobileWorld is twofold. First, it emphasizes long-horizon tasks with cross-application interactions: MobileWorld requires nearly twice as many task-completion steps on average (27.8 vs. 14.3) and includes far more multi-application tasks (62.2% vs. 9.5%) compared to AndroidWorld. Second, MobileWorld extends beyond standard GUI manipulation by introducing novel task categories, including agent-user interaction and MCP-augmented tasks. To ensure robust evaluation, we provide snapshot-based container environment and precise functional verifications, including backend database inspection and task callback APIs. We further develop a planner-executor agentic framework with extended action spaces to support user interactions and MCP calls. Our results reveal a sharp performance drop compared to AndroidWorld, with the best agentic framework and end-to-end model achieving 51.7% and 20.9% success rates, respectively. Our analysis shows that current models struggle significantly with user interaction and MCP calls, offering a strategic roadmap toward more robust, next-generation mobile intelligence.
MobileWorld, a more challenging benchmark than AndroidWorld, includes diverse real-world mobile tasks and interactions, revealing significant gaps in current model capabilities.
long-horizon tasks cross-application interactions agent-user interaction MCP-augmented tasks snapshot-based container environment planner-executor agentic framework extended action spaces