Learning Personalized Agents from Human Feedback
Modern AI agents are powerful but often fail to align with the idiosyncratic, evolving preferences of individual users. Prior approaches typically rely on static datasets, either training implicit preference models on interaction history or encoding user profiles in external memory. However, these approaches struggle with new users and with preferences that change over time. We introduce Personalized Agents from Human Feedback (PAHF), a framework for continual personalization in which agents learn online from live interaction using explicit per-user memory. PAHF operationalizes a three-step loop: (1) seeking pre-action clarification to resolve ambiguity, (2) grounding actions in preferences retrieved from memory, and (3) integrating post-action feedback to update memory when preferences drift. To evaluate this capability, we develop a four-phase protocol and two benchmarks in embodied manipulation and online shopping. These benchmarks quantify an agent's ability to learn initial preferences from scratch and subsequently adapt to persona shifts. Our theoretical analysis and empirical results show that integrating explicit memory with dual feedback channels is critical: PAHF learns substantially faster and consistently outperforms both no-memory and single-channel baselines, reducing initial personalization error and enabling rapid adaptation to preference shifts.
699774bb7a658569d5a100df
Kaiqu Liang
699774bb7a658569d5a100e0
Julia Kruk
699774bb7a658569d5a100e1
Shengyi Qian
699774bb7a658569d5a100e2
Xianjun Yang
699774bb7a658569d5a100e3
Shengjie Bi
699774bb7a658569d5a100e4
Yuanshun Yao
699774bb7a658569d5a100e5
Shaoliang Nie
699774bb7a658569d5a100e6
Mingyang Zhang
699774bb7a658569d5a100e7
Lijuan Liu
699774bb7a658569d5a100e8
Jaime Fernández Fisac
699774bb7a658569d5a100e9
Shuyan Zhou
699774bb7a658569d5a100ea
Saghar Hosseini
Learning Personalized Agents from Human Feedback
Modern AI agents are powerful but often fail to align with the idiosyncratic, evolving preferences of individual users. Prior approaches typically rely on static datasets, either training implicit preference models on interaction history or encoding user profiles in external memory. However, these approaches struggle with new users and with preferences that change over time. We introduce Personalized Agents from Human Feedback (PAHF), a framework for continual personalization in which agents learn online from live interaction using explicit per-user memory. PAHF operationalizes a three-step loop: (1) seeking pre-action clarification to resolve ambiguity, (2) grounding actions in preferences retrieved from memory, and (3) integrating post-action feedback to update memory when preferences drift. To evaluate this capability, we develop a four-phase protocol and two benchmarks in embodied manipulation and online shopping. These benchmarks quantify an agent's ability to learn initial preferences from scratch and subsequently adapt to persona shifts. Our theoretical analysis and empirical results show that integrating explicit memory with dual feedback channels is critical: PAHF learns substantially faster and consistently outperforms both no-memory and single-channel baselines, reducing initial personalization error and enabling rapid adaptation to preference shifts.
PAHF framework enables continual personalization of AI agents through explicit user memory and dual feedback channels, allowing rapid adaptation to changing user preferences.
personalized agents human feedback continual personalization explicit memory dual feedback channels preference modeling user profiles embodied manipulation online shopping