Grounding Computer Use Agents on Human Demonstrations
Building reliable computer-use agents requires grounding: accurately
connecting natural language instructions to the correct on-screen elements.
While large datasets exist for web and mobile interactions, high-quality
resources for desktop environments are limited. To address this gap, we
introduce GroundCUA, a large-scale desktop grounding dataset built from expert
human demonstrations. It covers 87 applications across 12 categories and
includes 56K screenshots, with every on-screen element carefully annotated for
a total of over 3.56M human-verified annotations. From these demonstrations, we
generate diverse instructions that capture a wide range of real-world tasks,
providing high-quality data for model training. Using GroundCUA, we develop the
GroundNext family of models that map instructions to their target UI elements.
At both 3B and 7B scales, GroundNext achieves state-of-the-art results across
five benchmarks using supervised fine-tuning, while requiring less than
one-tenth the training data of prior work. Reinforcement learning post-training
further improves performance, and when evaluated in an agentic setting on the
OSWorld benchmark using o3 as planner, GroundNext attains comparable or
superior results to models trained with substantially more data,. These results
demonstrate the critical role of high-quality, expert-driven datasets in
advancing general-purpose computer-use agents.
6912b70ea644ba07c499c752
Aarash Feizi
claimed_verified
2025-11-11T19:44:00.927Z
62e98e784fa4bc6de6c0f65b
https://cdn-avatars.huggingface.co/v1/production/uploads/62e98e784fa4bc6de6c0f65b/V8M3vr8yC4d5x4o2MfIhX.jpeg
Aarash Feizi
aarashfeizi
user
aarashfeizi
6912b70ea644ba07c499c753
Shravan Nayak
claimed_verified
2025-11-17T10:31:59.740Z
643ee7606d5be535d28034f6
https://cdn-avatars.huggingface.co/v1/production/uploads/643ee7606d5be535d28034f6/1F241mP_buW74byC2GAWH.jpeg
Shravan Nayak
BAJUKA
user
BAJUKA
6912b70ea644ba07c499c754
Xiangru Jian
claimed_verified
2025-11-11T19:44:03.406Z
636865b8cca0a0a962c21f3f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Mja7cpws4gb2Jmdj_foPA.png
Xiangru (Edward) Jian
HideOnBush
user
HideOnBush
6912b70ea644ba07c499c755
Kevin Qinghong Lin
claimed_verified
2025-11-11T19:44:05.862Z
64440be5af034cdfd69ca3a7
https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
Qinghong (Kevin) Lin
KevinQHLin
user
KevinQHLin
6912b70ea644ba07c499c756
Kaixin Li
claimed_verified
2025-11-13T13:07:09.911Z
6346be8f7fb9f11870c63998
https://cdn-avatars.huggingface.co/v1/production/uploads/6346be8f7fb9f11870c63998/tFWawSkXL6bv1zgvzFWQd.png
Kaixin Li
likaixin
user
likaixin
6912b70ea644ba07c499c757
Rabiul Awal
claimed_verified
2025-11-11T19:43:58.411Z
60edf5e03203a5daf7d3912e
https://cdn-avatars.huggingface.co/v1/production/uploads/60edf5e03203a5daf7d3912e/fq1AzGutKI_qojhevc03P.jpeg
Rabiul Awal
rabiulawal
user
rabiulawal
6912b70ea644ba07c499c758
Xing Han Lù
claimed_verified
2025-11-11T19:43:50.993Z
5fa9ff3ea13e063b8b2b60cb
https://cdn-avatars.huggingface.co/v1/production/uploads/1633380224986-5fa9ff3ea13e063b8b2b60cb.jpeg
Xing Han Lù
xhluca
user
xhluca
6912b70ea644ba07c499c759
Johan Obando-Ceron
claimed_verified
2025-11-11T19:43:55.983Z
6478f30ea68454566353ef95
https://cdn-avatars.huggingface.co/v1/production/uploads/6478f30ea68454566353ef95/hX5NiVPyQbK8TIBKnT4J1.jpeg
Johan Samir Obando Ceron
johanobandoc
user
johanobandoc
6912b70ea644ba07c499c75a
Juan A. Rodriguez
claimed_verified
2025-11-13T13:07:12.059Z
63a614d264f470027818b066
https://cdn-avatars.huggingface.co/v1/production/uploads/63a614d264f470027818b066/Q5Eih2VqD4NaHB_MkNp32.jpeg
Juan A. Rodriguez
joanrodai
user
joanrodai
6912b70ea644ba07c499c75b
Nicolas Chapados
6912b70ea644ba07c499c75c
David Vazquez
6912b70ea644ba07c499c75d
Adriana Romero-Soriano
6912b70ea644ba07c499c75e
Reihaneh Rabbany
6912b70ea644ba07c499c75f
Perouz Taslakian
6912b70ea644ba07c499c760
Christopher Pal
6912b70ea644ba07c499c761
Spandana Gella
claimed_verified
2025-11-17T10:32:02.038Z
65ca8745d64d82c92fa7c71f
https://cdn-avatars.huggingface.co/v1/production/uploads/65ca8745d64d82c92fa7c71f/xayDYjs_LoAsa8BEgDlhb.jpeg
G
spandanagella
user
spandanagella
6912b70ea644ba07c499c762
Sai Rajeswar
claimed_verified
2025-11-13T13:07:07.609Z
642f99079b2484d7d857341b
/avatars/01965cc5a5dbe9c08025a51973462a6a.svg
Sai Rajeswar
rajeswarsai
user
rajeswarsai
Grounding Computer Use Agents on Human Demonstrations
Building reliable computer-use agents requires grounding: accurately
connecting natural language instructions to the correct on-screen elements.
While large datasets exist for web and mobile interactions, high-quality
resources for desktop environments are limited. To address this gap, we
introduce GroundCUA, a large-scale desktop grounding dataset built from expert
human demonstrations. It covers 87 applications across 12 categories and
includes 56K screenshots, with every on-screen element carefully annotated for
a total of over 3.56M human-verified annotations. From these demonstrations, we
generate diverse instructions that capture a wide range of real-world tasks,
providing high-quality data for model training. Using GroundCUA, we develop the
GroundNext family of models that map instructions to their target UI elements.
At both 3B and 7B scales, GroundNext achieves state-of-the-art results across
five benchmarks using supervised fine-tuning, while requiring less than
one-tenth the training data of prior work. Reinforcement learning post-training
further improves performance, and when evaluated in an agentic setting on the
OSWorld benchmark using o3 as planner, GroundNext attains comparable or
superior results to models trained with substantially more data,. These results
demonstrate the critical role of high-quality, expert-driven datasets in
advancing general-purpose computer-use agents.
GroundCUA, a large-scale desktop grounding dataset, enables the development of GroundNext models that achieve state-of-the-art performance in mapping instructions to UI elements with less training data.
grounding natural language instructions on-screen elements desktop environments GroundCUA expert human demonstrations screenshots human-verified annotations GroundNext supervised fine-tuning reinforcement learning OSWorld benchmark o3 planner