Efficient RL Training for LLMs with Experience Replay
While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample diversity and the high computational cost of generation. We show that strict on-policy sampling is suboptimal when generation is expensive. Empirically, we show that a well-designed replay buffer can drastically reduce inference compute without degrading - and in some cases even improving - final model performance, while preserving policy entropy.
69de75fed3c84fcbf292143e
Charles Arnal
claimed_verified
2026-04-15T12:06:11.441Z
662938fe85faa365a7a59645
/avatars/0958564dffb8b2fd15da09623587d462.svg
Charles Arnal
CharlesArnal
user
CharlesArnal
69de75fed3c84fcbf292143f
Vivien Cabannes
69de75fed3c84fcbf2921440
Taco Cohen
69de75fed3c84fcbf2921441
Julia Kempe
69de75fed3c84fcbf2921442
Remi Munos
Efficient RL Training for LLMs with Experience Replay
While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample diversity and the high computational cost of generation. We show that strict on-policy sampling is suboptimal when generation is expensive. Empirically, we show that a well-designed replay buffer can drastically reduce inference compute without degrading - and in some cases even improving - final model performance, while preserving policy entropy.
Experience replay techniques for large language model post-training balance staleness variance and computational costs while maintaining performance and policy entropy.
Experience Replay rollouts reinforcement learning LLM post-training on-policy sampling replay buffers staleness-induced variance sample diversity inference compute policy entropy