Meta-Reinforcement Learning with Self-Reflection for Agentic Search
This paper introduces MR-Search, an in-context meta reinforcement learning (RL) formulation for agentic search with self-reflection. Instead of optimizing a policy within a single independent episode with sparse rewards, MR-Search trains a policy that conditions on past episodes and adapts its search strategy across episodes. MR-Search learns to learn a search strategy with self-reflection, allowing search agents to improve in-context exploration at test-time. Specifically, MR-Search performs cross-episode exploration by generating explicit self-reflections after each episode and leveraging them as additional context to guide subsequent attempts, thereby promoting more effective exploration during test-time. We further introduce a multi-turn RL algorithm that estimates a dense relative advantage at the turn level, enabling fine-grained credit assignment on each episode. Empirical results across various benchmarks demonstrate the advantages of MR-Search over baselines based RL, showing strong generalization and relative improvements of 9.2% to 19.3% across eight benchmarks. Our code and data are available at https://github.com/tengxiao1/MR-Search.
69b36a282ad6c5d4cf48697c
Teng Xiao
69b36a282ad6c5d4cf48697d
Yige Yuan
69b36a282ad6c5d4cf48697e
Hamish Ivison
69b36a282ad6c5d4cf48697f
Huaisheng Zhu
69b36a282ad6c5d4cf486980
Faeze Brahman
69b36a282ad6c5d4cf486981
Nathan Lambert
69b36a282ad6c5d4cf486982
Pradeep Dasigi
69b36a282ad6c5d4cf486983
Noah A. Smith
69b36a282ad6c5d4cf486984
Hannaneh Hajishirzi
Meta-Reinforcement Learning with Self-Reflection for Agentic Search
This paper introduces MR-Search, an in-context meta reinforcement learning (RL) formulation for agentic search with self-reflection. Instead of optimizing a policy within a single independent episode with sparse rewards, MR-Search trains a policy that conditions on past episodes and adapts its search strategy across episodes. MR-Search learns to learn a search strategy with self-reflection, allowing search agents to improve in-context exploration at test-time. Specifically, MR-Search performs cross-episode exploration by generating explicit self-reflections after each episode and leveraging them as additional context to guide subsequent attempts, thereby promoting more effective exploration during test-time. We further introduce a multi-turn RL algorithm that estimates a dense relative advantage at the turn level, enabling fine-grained credit assignment on each episode. Empirical results across various benchmarks demonstrate the advantages of MR-Search over baselines based RL, showing strong generalization and relative improvements of 9.2% to 19.3% across eight benchmarks. Our code and data are available at https://github.com/tengxiao1/MR-Search.
MR-Search enables adaptive agent search through meta-reinforcement learning with self-reflection, improving exploration and generalization across multiple episodes.
meta reinforcement learning in-context learning self-reflection cross-episode exploration dense relative advantage fine-grained credit assignment