GUI-360: A Comprehensive Dataset and Benchmark for Computer-Using Agents
We introduce GUI-360^circ, a large-scale, comprehensive dataset and
benchmark suite designed to advance computer-using agents (CUAs). CUAs present
unique challenges and is constrained by three persistent gaps: a scarcity of
real-world CUA tasks, the lack of automated collection-and-annotation pipelines
for multi-modal trajectories, and the absence of a unified benchmark that
jointly evaluates GUI grounding, screen parsing, and action prediction.
  GUI-360^circ addresses these gaps with an LLM-augmented, largely automated
pipeline for query sourcing, environment-template construction, task
instantiation, batched execution, and LLM-driven quality filtering. The
released corpus contains over 1.2M executed action steps across thousands of
trajectories in popular Windows office applications, and includes
full-resolution screenshots, accessibility metadata when available,
instantiated goals, intermediate reasoning traces, and both successful and
failed action trajectories. The dataset supports three canonical tasks, GUI
grounding, screen parsing, and action prediction, and a hybrid GUI+API action
space that reflects modern agent designs. Benchmarking state-of-the-art
vision--language models on GUI-360^circ reveals substantial out-of-the-box
shortcomings in grounding and action prediction; supervised fine-tuning and
reinforcement learning yield significant gains but do not close the gap to
human-level reliability. We release GUI-360^circ and accompanying code to
facilitate reproducible research and accelerate progress on robust desktop
CUAs.
  The full dataset has been made public on
https://huggingface.co/datasets/vyokky/GUI-360.
690d5db4ad2597bf6c464d4b
Jian Mu
690d5db4ad2597bf6c464d4c
Chaoyun Zhang
690d5db4ad2597bf6c464d4d
Chiming Ni
690d5db4ad2597bf6c464d4e
Lu Wang
690d5db4ad2597bf6c464d4f
Bo Qiao
690d5db4ad2597bf6c464d50
Kartik Mathur
690d5db4ad2597bf6c464d51
Qianhui Wu
claimed_verified
2026-03-19T13:05:53.652Z
63ef330b1e695b35aa484e11
https://cdn-avatars.huggingface.co/v1/production/uploads/63ef330b1e695b35aa484e11/bXwpGy0dl8JXeJwJ--ilr.jpeg
Qianhui WU
qianhuiwu
user
qianhuiwu
690d5db4ad2597bf6c464d52
Yuhang Xie
690d5db4ad2597bf6c464d53
Xiaojun Ma
690d5db4ad2597bf6c464d54
Mengyu Zhou
690d5db4ad2597bf6c464d55
Si Qin
690d5db4ad2597bf6c464d56
Liqun Li
690d5db4ad2597bf6c464d57
Yu Kang
690d5db4ad2597bf6c464d58
Minghua Ma
690d5db4ad2597bf6c464d59
Qingwei Lin
690d5db4ad2597bf6c464d5a
Saravan Rajmohan
690d5db4ad2597bf6c464d5b
Dongmei Zhang
GUI-360: A Comprehensive Dataset and Benchmark for Computer-Using Agents
We introduce GUI-360^circ, a large-scale, comprehensive dataset and
benchmark suite designed to advance computer-using agents (CUAs). CUAs present
unique challenges and is constrained by three persistent gaps: a scarcity of
real-world CUA tasks, the lack of automated collection-and-annotation pipelines
for multi-modal trajectories, and the absence of a unified benchmark that
jointly evaluates GUI grounding, screen parsing, and action prediction.
  GUI-360^circ addresses these gaps with an LLM-augmented, largely automated
pipeline for query sourcing, environment-template construction, task
instantiation, batched execution, and LLM-driven quality filtering. The
released corpus contains over 1.2M executed action steps across thousands of
trajectories in popular Windows office applications, and includes
full-resolution screenshots, accessibility metadata when available,
instantiated goals, intermediate reasoning traces, and both successful and
failed action trajectories. The dataset supports three canonical tasks, GUI
grounding, screen parsing, and action prediction, and a hybrid GUI+API action
space that reflects modern agent designs. Benchmarking state-of-the-art
vision--language models on GUI-360^circ reveals substantial out-of-the-box
shortcomings in grounding and action prediction; supervised fine-tuning and
reinforcement learning yield significant gains but do not close the gap to
human-level reliability. We release GUI-360^circ and accompanying code to
facilitate reproducible research and accelerate progress on robust desktop
CUAs.
  The full dataset has been made public on
https://huggingface.co/datasets/vyokky/GUI-360.
GUI-360° is a large-scale dataset and benchmark suite for computer-using agents, addressing gaps in real-world tasks, automated data collection, and unified evaluation of GUI grounding, screen parsing, and action prediction.
LLM-augmented GUI grounding screen parsing action prediction hybrid GUI+API action space vision--language models supervised fine-tuning reinforcement learning