Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization
Mobile GUI agents powered by Multimodal Large Language Models (MLLMs) can execute complex tasks on mobile devices. Despite this progress, most existing systems still optimize task success or efficiency, neglecting users' privacy personalization. In this paper, we study the often-overlooked problem of agent personalization. We observe that personalization can induce systematic structural heterogeneity in execution trajectories. For example, privacy-first users often prefer protective actions, e.g., refusing permissions, logging out, and minimizing exposure, leading to logically different execution trajectories from utility-first users. Such variable-length and structurally different trajectories make standard preference optimization unstable and less informative. To address this issue, we propose Trajectory Induced Preference Optimization (TIPO), which uses preference-intensity weighting to emphasize key privacy-related steps and padding gating to suppress alignment noise. Results on our Privacy Preference Dataset show that TIPO improves persona alignment and distinction while preserving strong task executability, achieving 65.60% SR, 46.22 Compliance, and 66.67% PD, outperforming existing optimization methods across various GUI tasks. The code and dataset will be publicly released at https://github.com/Zhixin-L/TIPO.
69de21799d30fe1923da4a2c
Zhixin Lin
69de21799d30fe1923da4a2d
Jungang Li
claimed_verified
2026-04-15T12:06:37.972Z
64b76528fdb702b3d8641514
https://cdn-avatars.huggingface.co/v1/production/uploads/64b76528fdb702b3d8641514/Ho-uWcQCAEIURM1lhWEWJ.jpeg
Jungang Li
Jungang
user
Jungang
69de21799d30fe1923da4a2e
Dongliang Xu
69de21799d30fe1923da4a2f
Shidong Pan
69de21799d30fe1923da4a30
Yibo Shi
69de21799d30fe1923da4a31
Yuchi Liu
69de21799d30fe1923da4a32
Yuecong Min
69de21799d30fe1923da4a33
Yue Yao
Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization
Mobile GUI agents powered by Multimodal Large Language Models (MLLMs) can execute complex tasks on mobile devices. Despite this progress, most existing systems still optimize task success or efficiency, neglecting users' privacy personalization. In this paper, we study the often-overlooked problem of agent personalization. We observe that personalization can induce systematic structural heterogeneity in execution trajectories. For example, privacy-first users often prefer protective actions, e.g., refusing permissions, logging out, and minimizing exposure, leading to logically different execution trajectories from utility-first users. Such variable-length and structurally different trajectories make standard preference optimization unstable and less informative. To address this issue, we propose Trajectory Induced Preference Optimization (TIPO), which uses preference-intensity weighting to emphasize key privacy-related steps and padding gating to suppress alignment noise. Results on our Privacy Preference Dataset show that TIPO improves persona alignment and distinction while preserving strong task executability, achieving 65.60% SR, 46.22 Compliance, and 66.67% PD, outperforming existing optimization methods across various GUI tasks. The code and dataset will be publicly released at https://github.com/Zhixin-L/TIPO.
Mobile GUI agents using MLLMs can execute complex tasks while addressing privacy personalization through trajectory-induced preference optimization that improves persona alignment and task executability.
Multimodal Large Language Models GUI agents privacy personalization preference optimization trajectory induction preference-intensity weighting padding gating persona alignment task executability