Meta-RL Induces Exploration in Language Agents
Reinforcement learning (RL) has enabled the training of large language model (LLM) agents to interact with the environment and to solve multi-turn long-horizon tasks. However, the RL-trained agents often struggle in tasks that require active exploration and fail to efficiently adapt from trial-and-error experiences. In this paper, we present LaMer, a general Meta-RL framework that enables LLM agents to actively explore and learn from the environment feedback at test time. LaMer consists of two key components: (i) a cross-episode training framework to encourage exploration and long-term rewards optimization; and (ii) in-context policy adaptation via reflection, allowing the agent to adapt their policy from task feedback signal without gradient update. Experiments across diverse environments show that LaMer significantly improves performance over RL baselines, with 11%, 14%, and 19% performance gains on Sokoban, MineSweeper and Webshop, respectively. Moreover, LaMer also demonstrates better generalization to more challenging or previously unseen tasks compared to the RL-trained agents. Overall, our results demonstrate that Meta-RL provides a principled approach to induce exploration in language agents, enabling more robust adaptation to novel environments through learned exploration strategies.
69492d0cd2000e944d383a72
Yulun Jiang
claimed_verified
2025-12-25T20:50:55.496Z
662a75287181150b857245fb
/avatars/6a489bf6d3d9cd26ba88f17b35c6ecb5.svg
Yulun Jiang
yljblues
user
yljblues
69492d0cd2000e944d383a73
Liangze Jiang
69492d0cd2000e944d383a74
Damien Teney
69492d0cd2000e944d383a75
Michael Moor
claimed_verified
2025-12-25T20:50:58.079Z
6438d1d843d932c462404500
https://cdn-avatars.huggingface.co/v1/production/uploads/6438d1d843d932c462404500/6UrtJbed9N5ETbImgIh-C.png
Michael Moor
mdmoor
user
mdmoor
69492d0cd2000e944d383a76
Maria Brbic
Meta-RL Induces Exploration in Language Agents
Reinforcement learning (RL) has enabled the training of large language model (LLM) agents to interact with the environment and to solve multi-turn long-horizon tasks. However, the RL-trained agents often struggle in tasks that require active exploration and fail to efficiently adapt from trial-and-error experiences. In this paper, we present LaMer, a general Meta-RL framework that enables LLM agents to actively explore and learn from the environment feedback at test time. LaMer consists of two key components: (i) a cross-episode training framework to encourage exploration and long-term rewards optimization; and (ii) in-context policy adaptation via reflection, allowing the agent to adapt their policy from task feedback signal without gradient update. Experiments across diverse environments show that LaMer significantly improves performance over RL baselines, with 11%, 14%, and 19% performance gains on Sokoban, MineSweeper and Webshop, respectively. Moreover, LaMer also demonstrates better generalization to more challenging or previously unseen tasks compared to the RL-trained agents. Overall, our results demonstrate that Meta-RL provides a principled approach to induce exploration in language agents, enabling more robust adaptation to novel environments through learned exploration strategies.
LaMer is a meta-reinforcement learning framework that enables large language model agents to actively explore and adapt through cross-episode training and in-context policy adaptation without gradient updates.
reinforcement learning large language models meta-reinforcement learning exploration long-horizon tasks cross-episode training in-context policy adaptation reflection policy adaptation environment feedback