KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation
Personalized mobile agents that infer user preferences and calibrate proactive assistance hold great promise as everyday digital assistants, yet existing benchmarks fail to capture what this requires. Prior work evaluates preference recovery from static histories or intent prediction from fixed contexts. Neither tests whether an agent can elicit missing preferences through interaction, nor whether it can decide when to intervene, seek consent, or remain silent in a live GUI environment. We introduce KnowU-Bench, an online benchmark for personalized mobile agents built on a reproducible Android emulation environment, covering 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks. Unlike prior work that treats user preferences as static context, KnowU-Bench hides the user profile from the agent and exposes only behavioral logs, forcing genuine preference inference rather than context lookup. To support multi-turn preference elicitation, it instantiates an LLM-driven user simulator grounded in structured profiles, enabling realistic clarification dialogues and proactive consent handling. Beyond personalization, KnowU-Bench provides comprehensive evaluation of the complete proactive decision chain, including grounded GUI execution, consent negotiation, and post-rejection restraint, evaluated through a hybrid protocol combining rule-based verification with LLM-as-a-Judge scoring. Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions requiring user preference inference or intervention calibration, even for frontier models like Claude Sonnet 4.6. The core bottlenecks are not GUI navigation but preference acquisition and intervention calibration, exposing a fundamental gap between competent interface operation and trustworthy personal assistance.
69d856cc79548974da946456
Tongbo Chen
69d856cc79548974da946457
Zhengxi Lu
claimed_verified
2026-04-13T07:34:06.997Z
676127cf11b19ea602bb202a
/avatars/dfd802a24bd63e509728159ebb1769f6.svg
Zhengxi Lu
LZXzju
user
LZXzju
69d856cc79548974da946458
Zhan Xu
69d856cc79548974da946459
Guocheng Shao
69d856cc79548974da94645a
Shaohan Zhao
69d856cc79548974da94645b
Fei Tang
69d856cc79548974da94645c
Yong Du
69d856cc79548974da94645d
Kaitao Song
69d856cc79548974da94645e
Yizhou Liu
69d856cc79548974da94645f
Yuchen Yan
claimed_verified
2026-04-10T07:43:59.940Z
64098738342c26884c792c93
https://cdn-avatars.huggingface.co/v1/production/uploads/64098738342c26884c792c93/SxBUd-wLrl-PjQsrVYJte.jpeg
Yuchen Yan
yanyc
user
yanyc
69d856cc79548974da946460
Wenqi Zhang
69d856cc79548974da946461
Xu Tan
69d856cc79548974da946462
Weiming Lu
69d856cc79548974da946463
Jun Xiao
69d856cc79548974da946464
Yueting Zhuang
69d856cc79548974da946465
Yongliang Shen
claimed_verified
2026-04-14T08:17:40.672Z
5e1058e9fcf41d740b69966d
/avatars/ce74839ba871f2b54313a670a233ba82.svg
Yongliang Shen
tricktreat
user
tricktreat
KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation
Personalized mobile agents that infer user preferences and calibrate proactive assistance hold great promise as everyday digital assistants, yet existing benchmarks fail to capture what this requires. Prior work evaluates preference recovery from static histories or intent prediction from fixed contexts. Neither tests whether an agent can elicit missing preferences through interaction, nor whether it can decide when to intervene, seek consent, or remain silent in a live GUI environment. We introduce KnowU-Bench, an online benchmark for personalized mobile agents built on a reproducible Android emulation environment, covering 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks. Unlike prior work that treats user preferences as static context, KnowU-Bench hides the user profile from the agent and exposes only behavioral logs, forcing genuine preference inference rather than context lookup. To support multi-turn preference elicitation, it instantiates an LLM-driven user simulator grounded in structured profiles, enabling realistic clarification dialogues and proactive consent handling. Beyond personalization, KnowU-Bench provides comprehensive evaluation of the complete proactive decision chain, including grounded GUI execution, consent negotiation, and post-rejection restraint, evaluated through a hybrid protocol combining rule-based verification with LLM-as-a-Judge scoring. Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions requiring user preference inference or intervention calibration, even for frontier models like Claude Sonnet 4.6. The core bottlenecks are not GUI navigation but preference acquisition and intervention calibration, exposing a fundamental gap between competent interface operation and trustworthy personal assistance.
KnowU-Bench presents a comprehensive benchmark for personalized mobile agents that evaluates true preference inference and proactive assistance capabilities in real-world GUI environments.
personalized mobile agents preference recovery user simulation LLM-driven user simulator proactive assistance GUI execution consent negotiation preference inference intervention calibration