Video-Based Reward Modeling for Computer-Use Agents
Computer-using agents (CUAs) are becoming increasingly capable; however, it remains difficult to scale evaluation of whether a trajectory truly fulfills a user instruction. In this work, we study reward modeling from execution video: a sequence of keyframes from an agent trajectory that is independent of the agent's internal reasoning or actions. Although video-execution modeling is method-agnostic, it presents key challenges, including highly redundant layouts and subtle, localized cues that determine success. We introduce Execution Video Reward 53k (ExeVR-53k), a dataset of 53k high-quality video--task--reward triplets. We further propose adversarial instruction translation to synthesize negative samples with step-level annotations. To enable learning from long, high-resolution execution videos, we design spatiotemporal token pruning, which removes homogeneous regions and persistent tokens while preserving decisive UI changes. Building on these components, we fine-tune an Execution Video Reward Model (ExeVRM) that takes only a user instruction and a video-execution sequence to predict task success. Our ExeVRM 8B achieves 84.7% accuracy and 87.7% recall on video-execution assessment, outperforming strong proprietary models such as GPT-5.2 and Gemini-3 Pro across Ubuntu, macOS, Windows, and Android, while providing more precise temporal attribution. These results show that video-execution reward modeling can serve as a scalable, model-agnostic evaluator for CUAs.
69b21678206bad330cd7ff09
Linxin Song
admin_assigned
2026-03-13T08:49:22.391Z
64d660308ebc40443813f014
/avatars/516bb2d2383be99794e366dfb41636b6.svg
Linxin Song
linxinso
user
linxinso
69b21678206bad330cd7ff0a
Jieyu Zhang
69b21678206bad330cd7ff0b
Huanxin Sheng
admin_assigned
2026-03-13T08:49:34.465Z
66edf4ed50a76f821a2fc106
/avatars/9fa3863b316391903329668053dda081.svg
Huanxin Sheng
HuanxinSheng
user
HuanxinSheng
69b21678206bad330cd7ff0c
Taiwei Shi
claimed_verified
2026-03-23T11:00:23.578Z
62e1b3cb3eb0730f621a83f6
https://cdn-avatars.huggingface.co/v1/production/uploads/1658958764563-noauth.jpeg
Taiwei Shi
MaksimSTW
user
MaksimSTW
69b21678206bad330cd7ff0d
Gupta Rahul
69b21678206bad330cd7ff0e
Yang Liu
69b21678206bad330cd7ff0f
Ranjay Krishna
admin_assigned
2026-03-13T08:49:54.144Z
66429868ab89e3a3a85668b0
/avatars/170e0daa454838deee2bf946f7118651.svg
Ranjay Krishna
ranjaykrishna
user
ranjaykrishna
69b21678206bad330cd7ff10
Jian Kang
69b21678206bad330cd7ff11
Jieyu Zhao
admin_assigned
2026-03-13T08:49:47.703Z
64ba0230c9a9894feb0cd32b
https://cdn-avatars.huggingface.co/v1/production/uploads/64ba0230c9a9894feb0cd32b/V1XWySRXAhLN4wqXZJ3Ai.jpeg
Jieyu Zhao
jieyuz
user
jieyuz
Video-Based Reward Modeling for Computer-Use Agents
Computer-using agents (CUAs) are becoming increasingly capable; however, it remains difficult to scale evaluation of whether a trajectory truly fulfills a user instruction. In this work, we study reward modeling from execution video: a sequence of keyframes from an agent trajectory that is independent of the agent's internal reasoning or actions. Although video-execution modeling is method-agnostic, it presents key challenges, including highly redundant layouts and subtle, localized cues that determine success. We introduce Execution Video Reward 53k (ExeVR-53k), a dataset of 53k high-quality video--task--reward triplets. We further propose adversarial instruction translation to synthesize negative samples with step-level annotations. To enable learning from long, high-resolution execution videos, we design spatiotemporal token pruning, which removes homogeneous regions and persistent tokens while preserving decisive UI changes. Building on these components, we fine-tune an Execution Video Reward Model (ExeVRM) that takes only a user instruction and a video-execution sequence to predict task success. Our ExeVRM 8B achieves 84.7% accuracy and 87.7% recall on video-execution assessment, outperforming strong proprietary models such as GPT-5.2 and Gemini-3 Pro across Ubuntu, macOS, Windows, and Android, while providing more precise temporal attribution. These results show that video-execution reward modeling can serve as a scalable, model-agnostic evaluator for CUAs.
Video-execution reward modeling enables scalable evaluation of computer-using agents by predicting task success from user instructions and execution videos, outperforming proprietary models across multiple operating systems.
computer-using agents reward modeling execution video video-execution modeling dataset adversarial instruction translation negative samples spatiotemporal token pruning Execution Video Reward Model temporal attribution