Complementary Reinforcement Learning
Reinforcement Learning (RL) has emerged as a powerful paradigm for training LLM-based agents, yet remains limited by low sample efficiency, stemming not only from sparse outcome feedback but also from the agent's inability to leverage prior experience across episodes. While augmenting agents with historical experience offers a promising remedy, existing approaches suffer from a critical weakness: the experience distilled from history is either stored statically or fail to coevolve with the improving actor, causing a progressive misalignment between the experience and the actor's evolving capability that diminishes its utility over the course of training. Inspired by complementary learning systems in neuroscience, we present Complementary RL to achieve seamless co-evolution of an experience extractor and a policy actor within the RL optimization loop. Specifically, the actor is optimized via sparse outcome-based rewards, while the experience extractor is optimized according to whether its distilled experiences demonstrably contribute to the actor's success, thereby evolving its experience management strategy in lockstep with the actor's growing capabilities. Empirically, Complementary RL outperforms outcome-based agentic RL baselines that do not learn from experience, achieving 10% performance improvement in single-task scenarios and exhibits robust scalability in multi-task settings. These results establish Complementary RL as a paradigm for efficient experience-driven agent learning.
69bb7106ba2b62a5fb0b0c51
Dilxat Muhtar
69bb7106ba2b62a5fb0b0c52
Jiashun Liu
69bb7106ba2b62a5fb0b0c53
Wei Gao
69bb7106ba2b62a5fb0b0c54
Weixun Wang
69bb7106ba2b62a5fb0b0c55
Shaopan Xiong
69bb7106ba2b62a5fb0b0c56
Ju Huang
69bb7106ba2b62a5fb0b0c57
Siran Yang
69bb7106ba2b62a5fb0b0c58
Wenbo Su
69bb7106ba2b62a5fb0b0c59
Jiamang Wang
69bb7106ba2b62a5fb0b0c5a
Ling Pan
69bb7106ba2b62a5fb0b0c5b
Bo Zheng
Complementary Reinforcement Learning
Reinforcement Learning (RL) has emerged as a powerful paradigm for training LLM-based agents, yet remains limited by low sample efficiency, stemming not only from sparse outcome feedback but also from the agent's inability to leverage prior experience across episodes. While augmenting agents with historical experience offers a promising remedy, existing approaches suffer from a critical weakness: the experience distilled from history is either stored statically or fail to coevolve with the improving actor, causing a progressive misalignment between the experience and the actor's evolving capability that diminishes its utility over the course of training. Inspired by complementary learning systems in neuroscience, we present Complementary RL to achieve seamless co-evolution of an experience extractor and a policy actor within the RL optimization loop. Specifically, the actor is optimized via sparse outcome-based rewards, while the experience extractor is optimized according to whether its distilled experiences demonstrably contribute to the actor's success, thereby evolving its experience management strategy in lockstep with the actor's growing capabilities. Empirically, Complementary RL outperforms outcome-based agentic RL baselines that do not learn from experience, achieving 10% performance improvement in single-task scenarios and exhibits robust scalability in multi-task settings. These results establish Complementary RL as a paradigm for efficient experience-driven agent learning.
Complementary RL enables efficient agent learning by synchronizing experience extraction with policy optimization through dual objectives that evolve together during training.
Reinforcement Learning LLM-based agents sample efficiency sparse outcome feedback experience extraction policy actor complementary learning systems co-evolution experience distillation multi-task settings