Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization
Exploration remains the key bottleneck for large language model agents trained with reinforcement learning. While prior methods exploit pretrained knowledge, they fail in environments requiring the discovery of novel states. We propose Exploratory Memory-Augmented On- and Off-Policy Optimization (EMPO^2), a hybrid RL framework that leverages memory for exploration and combines on- and off-policy updates to make LLMs perform well with memory while also ensuring robustness without it. On ScienceWorld and WebShop, EMPO^2 achieves 128.6% and 11.3% improvements over GRPO, respectively. Moreover, in out-of-distribution tests, EMPO^2 demonstrates superior adaptability to new tasks, requiring only a few trials with memory and no parameter updates. These results highlight EMPO^2 as a promising framework for building more exploratory and generalizable LLM-based agents.
69a11540a13deaa449448995
Zeyuan Liu
claimed_verified
2026-02-27T16:40:23.614Z
66228a37f5c285535cc9cc83
/avatars/0982d6553a9c9001ebdca2878bcfff34.svg
Zeyuan Liu
ZeyuanLiu
user
ZeyuanLiu
69a11540a13deaa449448996
Jeonghye Kim
claimed_verified
2026-02-27T16:40:18.705Z
63e48f6d9db5da2dc1f6288e
https://cdn-avatars.huggingface.co/v1/production/uploads/1676046878664-63e48f6d9db5da2dc1f6288e.png
JeonghyeKim
beanie00
user
beanie00
69a11540a13deaa449448997
Xufang Luo
claimed_verified
2026-02-27T16:40:15.195Z
66a1f912345b3106f47ce860
/avatars/40177299c64e16703e7bfe83de0810be.svg
Xufang Luo
daixufang
user
daixufang
69a11540a13deaa449448998
Dongsheng Li
69a11540a13deaa449448999
Yuqing Yang
Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization
Exploration remains the key bottleneck for large language model agents trained with reinforcement learning. While prior methods exploit pretrained knowledge, they fail in environments requiring the discovery of novel states. We propose Exploratory Memory-Augmented On- and Off-Policy Optimization (EMPO^2), a hybrid RL framework that leverages memory for exploration and combines on- and off-policy updates to make LLMs perform well with memory while also ensuring robustness without it. On ScienceWorld and WebShop, EMPO^2 achieves 128.6% and 11.3% improvements over GRPO, respectively. Moreover, in out-of-distribution tests, EMPO^2 demonstrates superior adaptability to new tasks, requiring only a few trials with memory and no parameter updates. These results highlight EMPO^2 as a promising framework for building more exploratory and generalizable LLM-based agents.
EMPO² is a hybrid reinforcement learning framework that enhances exploration for large language model agents by integrating memory mechanisms with on- and off-policy updates, demonstrating improved performance and adaptability in complex environments.
reinforcement learning large language model agents exploration memory augmentation on-policy updates off-policy updates ScienceWorld WebShop