MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments
Current mobile GUI agent benchmarks systematically fail to assess memory capabilities, with only 5.2-11.8% memory-related tasks and no cross-session learning evaluation. We introduce MemGUI-Bench, a comprehensive memory-centric benchmark with pass@k and staged LLM-as-judge evaluation. Our contributions include: (1) a systematic memory taxonomy analyzing 11 agents across 5 architectures; (2) 128 tasks across 26 applications where 89.8% challenge memory through cross-temporal and cross-spatial retention; (3) MemGUI-Eval, an automated pipeline with Progressive Scrutiny and 7 hierarchical metrics; and (4) RQ-driven assessment of 11 state-of-the-art agents. Our experiments reveal significant memory deficits across all evaluated systems, identify 5 distinct failure modes, and synthesize 5 actionable design implications. All resources including code, benchmark, and evaluation results will be \textit{fully open-sourced and continuously maintained} at https://lgy0404.github.io/MemGUI-Bench/.
698949ccbeecc443208d25a3
Guangyi Liu
claimed_verified
2026-02-09T14:32:13.554Z
64d761b98ebc40443831f82a
https://cdn-avatars.huggingface.co/v1/production/uploads/64d761b98ebc40443831f82a/DHBOtOstiFp2-lDY6b9gb.png
Guangyi Liu
lgy0404
user
lgy0404
698949ccbeecc443208d25a4
Pengxiang Zhao
698949ccbeecc443208d25a5
Yaozhen Liang
698949ccbeecc443208d25a6
Qinyi Luo
698949ccbeecc443208d25a7
Shunye Tang
698949ccbeecc443208d25a8
Yuxiang Chai
claimed_verified
2026-03-10T09:46:35.343Z
6458ce236fa580137af5aa95
/avatars/db65a7332e375eb5daad5c1b076b1e3b.svg
Yuxiang Chai
Yuxiang007
user
Yuxiang007
698949ccbeecc443208d25a9
Weifeng Lin
698949ccbeecc443208d25aa
Han Xiao
claimed_verified
2026-03-10T09:46:37.813Z
666aa99cd1652853e4f9a8b9
/avatars/7cd5a0c34b5ccb8eff5a353d88d15a93.svg
HanXiao
HanXiao1999
user
HanXiao1999
698949ccbeecc443208d25ab
WenHao Wang
claimed_verified
2026-03-17T08:31:20.008Z
64faa838da4429f25dd19471
/avatars/635f33920a05f63ba7783c3ecc75a462.svg
WENHAO WANG
wwh0411
user
wwh0411
698949ccbeecc443208d25ac
Siheng Chen
698949ccbeecc443208d25ad
Zhengxi Lu
claimed_verified
2026-02-09T08:29:59.917Z
676127cf11b19ea602bb202a
/avatars/dfd802a24bd63e509728159ebb1769f6.svg
Zhengxi Lu
LZXzju
user
LZXzju
698949ccbeecc443208d25ae
Gao Wu
698949ccbeecc443208d25af
Hao Wang
698949ccbeecc443208d25b0
Liang Liu
698949ccbeecc443208d25b1
Yong Liu
MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments
Current mobile GUI agent benchmarks systematically fail to assess memory capabilities, with only 5.2-11.8% memory-related tasks and no cross-session learning evaluation. We introduce MemGUI-Bench, a comprehensive memory-centric benchmark with pass@k and staged LLM-as-judge evaluation. Our contributions include: (1) a systematic memory taxonomy analyzing 11 agents across 5 architectures; (2) 128 tasks across 26 applications where 89.8% challenge memory through cross-temporal and cross-spatial retention; (3) MemGUI-Eval, an automated pipeline with Progressive Scrutiny and 7 hierarchical metrics; and (4) RQ-driven assessment of 11 state-of-the-art agents. Our experiments reveal significant memory deficits across all evaluated systems, identify 5 distinct failure modes, and synthesize 5 actionable design implications. All resources including code, benchmark, and evaluation results will be \textit{fully open-sourced and continuously maintained} at https://lgy0404.github.io/MemGUI-Bench/.
A comprehensive memory-focused benchmark for mobile GUI agents reveals significant memory capability gaps and provides systematic evaluation methods and design insights.
memory-centric benchmark LLM-as-judge evaluation memory taxonomy cross-temporal retention cross-spatial retention automated pipeline Progressive Scrutiny hierarchical metrics state-of-the-art agents failure modes design implications