Computer-Using World Model
Agents operating in complex software environments benefit from reasoning about the consequences of their actions, as even a single incorrect user interface (UI) operation can derail long, artifact-preserving workflows. This challenge is particularly acute for computer-using scenarios, where real execution does not support counterfactual exploration, making large-scale trial-and-error learning and planning impractical despite the environment being fully digital and deterministic. We introduce the Computer-Using World Model (CUWM), a world model for desktop software that predicts the next UI state given the current state and a candidate action. CUWM adopts a two-stage factorization of UI dynamics: it first predicts a textual description of agent-relevant state changes, and then realizes these changes visually to synthesize the next screenshot. CUWM is trained on offline UI transitions collected from agents interacting with real Microsoft Office applications, and further refined with a lightweight reinforcement learning stage that aligns textual transition predictions with the structural requirements of computer-using environments. We evaluate CUWM via test-time action search, where a frozen agent uses the world model to simulate and compare candidate actions before execution. Across a range of Office tasks, world-model-guided test-time scaling improves decision quality and execution robustness.
6997d0587a658569d5a10173
Yiming Guan
6997d0587a658569d5a10174
Rui Yu
admin_assigned
2026-02-20T09:11:06.649Z
68e278859cfbd3ce4f4ad60b
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/FL65Bz18nprjeHchx52lV.png
Rui Yu
ruiyu0
user
ruiyu0
6997d0587a658569d5a10175
John Zhang
6997d0587a658569d5a10176
Lu Wang
6997d0587a658569d5a10177
Chaoyun Zhang
admin_assigned
2026-02-20T09:11:37.500Z
654dbac9938fbf1e696be8aa
/avatars/b3c4035c48169c1bfb04a439fce3499f.svg
Chaoyun Zhang
vyokky
user
vyokky
6997d0587a658569d5a10178
Liqun Li
6997d0587a658569d5a10179
Bo Qiao
6997d0587a658569d5a1017a
Si Qin
6997d0587a658569d5a1017b
He Huang
6997d0587a658569d5a1017c
Fangkai Yang
6997d0587a658569d5a1017d
Pu Zhao
6997d0587a658569d5a1017e
Lukas Wutschitz
admin_assigned
2026-02-20T09:11:44.590Z
6380a37a5c62156ce7dff8b9
/avatars/fbe5a20869cb55ec43759c1b5f9c4135.svg
Lukas Wutschitz
wulu
user
wulu
6997d0587a658569d5a1017f
Samuel Kessler
6997d0587a658569d5a10180
Huseyin A Inan
6997d0587a658569d5a10181
Robert Sim
6997d0587a658569d5a10182
Saravan Rajmohan
6997d0587a658569d5a10183
Qingwei Lin
6997d0587a658569d5a10184
Dongmei Zhang
Computer-Using World Model
Agents operating in complex software environments benefit from reasoning about the consequences of their actions, as even a single incorrect user interface (UI) operation can derail long, artifact-preserving workflows. This challenge is particularly acute for computer-using scenarios, where real execution does not support counterfactual exploration, making large-scale trial-and-error learning and planning impractical despite the environment being fully digital and deterministic. We introduce the Computer-Using World Model (CUWM), a world model for desktop software that predicts the next UI state given the current state and a candidate action. CUWM adopts a two-stage factorization of UI dynamics: it first predicts a textual description of agent-relevant state changes, and then realizes these changes visually to synthesize the next screenshot. CUWM is trained on offline UI transitions collected from agents interacting with real Microsoft Office applications, and further refined with a lightweight reinforcement learning stage that aligns textual transition predictions with the structural requirements of computer-using environments. We evaluate CUWM via test-time action search, where a frozen agent uses the world model to simulate and compare candidate actions before execution. Across a range of Office tasks, world-model-guided test-time scaling improves decision quality and execution robustness.
A world model for desktop software that predicts UI state changes through textual description followed by visual synthesis, improving decision quality and execution robustness in computer-using tasks.
world model user interface UI state action search reinforcement learning test-time scaling computer-using environments desktop software textual description visual synthesis