UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization
MLLM-based GUI agents have demonstrated strong capabilities in complex user interface interaction tasks. However, long-horizon scenarios remain challenging, as these agents are burdened with tasks beyond their intrinsic capabilities, suffering from memory degradation, progress confusion, and math hallucination. To address these challenges, we present UI-Copilot, a collaborative framework where the GUI agent focuses on task execution while a lightweight copilot provides on-demand assistance for memory retrieval and numerical computation. We introduce memory decoupling to separate persistent observations from transient execution context, and train the policy agent to selectively invoke the copilot as Retriever or Calculator based on task demands. To enable effective tool invocation learning, we propose Tool-Integrated Policy Optimization (TIPO), which separately optimizes tool selection through single-turn prediction and task execution through on-policy multi-turn rollouts. Experimental results show that UI-Copilot-7B achieves state-of-the-art performance on challenging MemGUI-Bench, outperforming strong 7B-scale GUI agents such as GUI-Owl-7B and UI-TARS-1.5-7B. Moreover, UI-Copilot-7B delivers a 17.1% absolute improvement on AndroidWorld over the base Qwen model, highlighting UI-Copilot's strong generalization to real-world GUI tasks.
69e047ee5b704445ba50b6f8
Zhengxi Lu
claimed_verified
2026-04-16T07:59:28.501Z
676127cf11b19ea602bb202a
/avatars/dfd802a24bd63e509728159ebb1769f6.svg
Zhengxi Lu
LZXzju
user
LZXzju
69e047ee5b704445ba50b6f9
Fei Tang
69e047ee5b704445ba50b6fa
Guangyi Liu
69e047ee5b704445ba50b6fb
Kaitao Song
69e047ee5b704445ba50b6fc
Xu Tan
69e047ee5b704445ba50b6fd
Jin Ma
69e047ee5b704445ba50b6fe
Wenqi Zhang
69e047ee5b704445ba50b6ff
Weiming Lu
69e047ee5b704445ba50b700
Jun Xiao
69e047ee5b704445ba50b701
Yueting Zhuang
69e047ee5b704445ba50b702
Yongliang Shen
UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization
MLLM-based GUI agents have demonstrated strong capabilities in complex user interface interaction tasks. However, long-horizon scenarios remain challenging, as these agents are burdened with tasks beyond their intrinsic capabilities, suffering from memory degradation, progress confusion, and math hallucination. To address these challenges, we present UI-Copilot, a collaborative framework where the GUI agent focuses on task execution while a lightweight copilot provides on-demand assistance for memory retrieval and numerical computation. We introduce memory decoupling to separate persistent observations from transient execution context, and train the policy agent to selectively invoke the copilot as Retriever or Calculator based on task demands. To enable effective tool invocation learning, we propose Tool-Integrated Policy Optimization (TIPO), which separately optimizes tool selection through single-turn prediction and task execution through on-policy multi-turn rollouts. Experimental results show that UI-Copilot-7B achieves state-of-the-art performance on challenging MemGUI-Bench, outperforming strong 7B-scale GUI agents such as GUI-Owl-7B and UI-TARS-1.5-7B. Moreover, UI-Copilot-7B delivers a 17.1% absolute improvement on AndroidWorld over the base Qwen model, highlighting UI-Copilot's strong generalization to real-world GUI tasks.
UI-Copilot is a collaborative framework that enhances GUI agents by decoupling memory management and integrating on-demand tool assistance for improved performance in complex user interface tasks.
GUI agents memory degradation progress confusion math hallucination memory decoupling tool invocation learning Tool-Integrated Policy Optimization TIPO policy agent copilot Retriever Calculator on-policy multi-turn rollouts single-turn prediction