DigiData: Training and Evaluating General-Purpose Mobile Control Agents
AI agents capable of controlling user interfaces have the potential to
transform human interaction with digital devices. To accelerate this
transformation, two fundamental building blocks are essential: high-quality
datasets that enable agents to achieve complex and human-relevant goals, and
robust evaluation methods that allow researchers and practitioners to rapidly
enhance agent performance. In this paper, we introduce DigiData, a large-scale,
high-quality, diverse, multi-modal dataset designed for training mobile control
agents. Unlike existing datasets, which derive goals from unstructured
interactions, DigiData is meticulously constructed through comprehensive
exploration of app features, resulting in greater diversity and higher goal
complexity. Additionally, we present DigiData-Bench, a benchmark for evaluating
mobile control agents on real-world complex tasks. We demonstrate that the
commonly used step-accuracy metric falls short in reliably assessing mobile
control agents and, to address this, we propose dynamic evaluation protocols
and AI-powered evaluations as rigorous alternatives for agent assessment. Our
contributions aim to significantly advance the development of mobile control
agents, paving the way for more intuitive and effective human-device
interactions.
6912bc52a644ba07c499c7b0
Yuxuan Sun
claimed_verified
2025-11-11T19:43:39.339Z
665e28741d30854dbbf62046
/avatars/78b7291107eff7180031094103bd0a8b.svg
Yuxuan Sun
yuxuans
user
yuxuans
6912bc52a644ba07c499c7b1
Manchen Wang
6912bc52a644ba07c499c7b2
Shengyi Qian
claimed_verified
2025-11-11T19:54:33.507Z
6445dc7e219dc6142a6a6861
https://cdn-avatars.huggingface.co/v1/production/uploads/6445dc7e219dc6142a6a6861/Ak_YV8V1NZpkF5AQnJVlJ.png
Shengyi Qian
shengyi-qian
user
shengyi-qian
6912bc52a644ba07c499c7b3
William R. Wong
6912bc52a644ba07c499c7b4
Eric Gan
6912bc52a644ba07c499c7b5
Pierluca D'Oro
6912bc52a644ba07c499c7b6
Alejandro Castillejo Munoz
6912bc52a644ba07c499c7b7
Sneha Silwal
6912bc52a644ba07c499c7b8
Pedro Matias
6912bc52a644ba07c499c7b9
Nitin Kamra
6912bc52a644ba07c499c7ba
Satwik Kottur
6912bc52a644ba07c499c7bb
Nick Raines
6912bc52a644ba07c499c7bc
Xuanyi Zhao
6912bc52a644ba07c499c7bd
Joy Chen
6912bc52a644ba07c499c7be
Joseph Greer
6912bc52a644ba07c499c7bf
Andrea Madotto
6912bc52a644ba07c499c7c0
Allen Bolourchi
6912bc52a644ba07c499c7c1
James Valori
6912bc52a644ba07c499c7c2
Kevin Carlberg
6912bc52a644ba07c499c7c3
Karl Ridgeway
6912bc52a644ba07c499c7c4
Joseph Tighe
DigiData: Training and Evaluating General-Purpose Mobile Control Agents
AI agents capable of controlling user interfaces have the potential to
transform human interaction with digital devices. To accelerate this
transformation, two fundamental building blocks are essential: high-quality
datasets that enable agents to achieve complex and human-relevant goals, and
robust evaluation methods that allow researchers and practitioners to rapidly
enhance agent performance. In this paper, we introduce DigiData, a large-scale,
high-quality, diverse, multi-modal dataset designed for training mobile control
agents. Unlike existing datasets, which derive goals from unstructured
interactions, DigiData is meticulously constructed through comprehensive
exploration of app features, resulting in greater diversity and higher goal
complexity. Additionally, we present DigiData-Bench, a benchmark for evaluating
mobile control agents on real-world complex tasks. We demonstrate that the
commonly used step-accuracy metric falls short in reliably assessing mobile
control agents and, to address this, we propose dynamic evaluation protocols
and AI-powered evaluations as rigorous alternatives for agent assessment. Our
contributions aim to significantly advance the development of mobile control
agents, paving the way for more intuitive and effective human-device
interactions.
DigiData and DigiData-Bench advance mobile control agents by providing a diverse, high-quality dataset and dynamic evaluation protocols, respectively.
mobile control agents DigiData DigiData-Bench step-accuracy metric dynamic evaluation protocols AI-powered evaluations