Do Phone-Use Agents Respect Your Privacy?
We study whether phone-use agents respect privacy while completing benign mobile tasks. This question has remained hard to answer because privacy-compliant behavior is not operationalized for phone-use agents, and ordinary apps do not reveal exactly what data agents type into which form entries during execution. To make this question measurable, we introduce MyPhoneBench, a verifiable evaluation framework for privacy behavior in mobile agents. We operationalize privacy-respecting phone use as permissioned access, minimal disclosure, and user-controlled memory through a minimal privacy contract, iMy, and pair it with instrumented mock apps plus rule-based auditing that make unnecessary permission requests, deceptive re-disclosure, and unnecessary form filling observable and reproducible. Across five frontier models on 10 mobile apps and 300 tasks, we find that task success, privacy-compliant task completion, and later-session use of saved preferences are distinct capabilities, and no single model dominates all three. Evaluating success and privacy jointly reshuffles the model ordering relative to either metric alone. The most persistent failure mode across models is simple data minimization: agents still fill optional personal entries that the task does not require. These results show that privacy failures arise from over-helpful execution of benign tasks, and that success-only evaluation overestimates the deployment readiness of current phone-use agents. All code, mock apps, and agent trajectories are publicly available at~ https://github.com/tangzhy/MyPhoneBench.
69ce81b33dd602d72f632a68
Zhengyang Tang
claimed_verified
2026-04-07T08:46:31.308Z
64912976b95c3f0a1e6233cb
/avatars/3e338c5eef2514055ed98ae6141a5d1a.svg
Zhengyang Tang
tangzhy
user
tangzhy
69ce81b33dd602d72f632a69
Ke Ji
69ce81b33dd602d72f632a6a
Xidong Wang
69ce81b33dd602d72f632a6b
Zihan Ye
69ce81b33dd602d72f632a6c
Xinyuan Wang
69ce81b33dd602d72f632a6d
Yiduo Guo
69ce81b33dd602d72f632a6e
Ziniu Li
69ce81b33dd602d72f632a6f
Chenxin Li
69ce81b33dd602d72f632a70
Jingyuan Hu
69ce81b33dd602d72f632a71
Shunian Chen
69ce81b33dd602d72f632a72
Tongxu Luo
69ce81b33dd602d72f632a73
Jiaxi Bi
69ce81b33dd602d72f632a74
Zeyu Qin
69ce81b33dd602d72f632a75
Shaobo Wang
69ce81b33dd602d72f632a76
Xin Lai
69ce81b33dd602d72f632a77
Pengyuan Lyu
69ce81b33dd602d72f632a78
Junyi Li
69ce81b33dd602d72f632a79
Can Xu
69ce81b33dd602d72f632a7a
Chengquan Zhang
69ce81b33dd602d72f632a7b
Han Hu
69ce81b33dd602d72f632a7c
Ming Yan
69ce81b33dd602d72f632a7d
Benyou Wang
Do Phone-Use Agents Respect Your Privacy?
We study whether phone-use agents respect privacy while completing benign mobile tasks. This question has remained hard to answer because privacy-compliant behavior is not operationalized for phone-use agents, and ordinary apps do not reveal exactly what data agents type into which form entries during execution. To make this question measurable, we introduce MyPhoneBench, a verifiable evaluation framework for privacy behavior in mobile agents. We operationalize privacy-respecting phone use as permissioned access, minimal disclosure, and user-controlled memory through a minimal privacy contract, iMy, and pair it with instrumented mock apps plus rule-based auditing that make unnecessary permission requests, deceptive re-disclosure, and unnecessary form filling observable and reproducible. Across five frontier models on 10 mobile apps and 300 tasks, we find that task success, privacy-compliant task completion, and later-session use of saved preferences are distinct capabilities, and no single model dominates all three. Evaluating success and privacy jointly reshuffles the model ordering relative to either metric alone. The most persistent failure mode across models is simple data minimization: agents still fill optional personal entries that the task does not require. These results show that privacy failures arise from over-helpful execution of benign tasks, and that success-only evaluation overestimates the deployment readiness of current phone-use agents. All code, mock apps, and agent trajectories are publicly available at~ https://github.com/tangzhy/MyPhoneBench.