The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping
Despite the success of reinforcement learning for large language models, a common failure mode is reduced sampling diversity, where the policy repeatedly generates similar erroneous behaviors. Classical entropy regularization encourages randomness under the current policy, but does not explicitly discourage recurrent failure patterns across rollouts. We propose MEDS, a Memory-Enhanced Dynamic reward Shaping framework that incorporates historical behavioral signals into reward design. By storing and leveraging intermediate model representations, we capture features of past rollouts and use density-based clustering to identify frequently recurring error patterns. Rollouts assigned to more prevalent error clusters are penalized more heavily, encouraging broader exploration while reducing repeated mistakes. Across five datasets and three base models, MEDS consistently improves average performance over existing baselines, achieving gains of up to 4.13 pass@1 points and 4.37 pass@128 points. Additional analyses using both LLM-based annotations and quantitative diversity metrics show that MEDS increases behavioral diversity during sampling.
69ddfb6aaea31e227c22c6e4
Yang Liu
admin_assigned
2026-04-16T16:32:10.458Z
68bcde6cd56d85b61916771a
/avatars/1e6ac49fea2eebc3f97a9027c73e85c7.svg
Yang Liu
LinXiii
user
LinXiii
69ddfb6aaea31e227c22c6e5
Enxi Wang
69ddfb6aaea31e227c22c6e6
Yufei Gao
69ddfb6aaea31e227c22c6e7
Weixin Zhang
69ddfb6aaea31e227c22c6e8
Bo Wang
claimed_verified
2026-04-15T12:07:10.293Z
663c23b4a4152b77b67b7c1c
/avatars/7af03530ce2b48bc77ce84296878978f.svg
Bo Wang
Musicode
user
Musicode
69ddfb6aaea31e227c22c6e9
Zhiyuan Zeng
69ddfb6aaea31e227c22c6ea
Yikai Zhang
69ddfb6aaea31e227c22c6eb
Yining Zheng
69ddfb6aaea31e227c22c6ec
Xipeng Qiu
The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping
Despite the success of reinforcement learning for large language models, a common failure mode is reduced sampling diversity, where the policy repeatedly generates similar erroneous behaviors. Classical entropy regularization encourages randomness under the current policy, but does not explicitly discourage recurrent failure patterns across rollouts. We propose MEDS, a Memory-Enhanced Dynamic reward Shaping framework that incorporates historical behavioral signals into reward design. By storing and leveraging intermediate model representations, we capture features of past rollouts and use density-based clustering to identify frequently recurring error patterns. Rollouts assigned to more prevalent error clusters are penalized more heavily, encouraging broader exploration while reducing repeated mistakes. Across five datasets and three base models, MEDS consistently improves average performance over existing baselines, achieving gains of up to 4.13 pass@1 points and 4.37 pass@128 points. Additional analyses using both LLM-based annotations and quantitative diversity metrics show that MEDS increases behavioral diversity during sampling.
MEDS is a memory-enhanced dynamic reward shaping framework that improves sampling diversity in reinforcement learning for large language models by identifying and penalizing recurrent error patterns through clustering of historical behavioral signals.
reinforcement learning large language models sampling diversity entropy regularization reward shaping memory-enhanced dynamic reward shaping historical behavioral signals intermediate model representations density-based clustering error patterns behavioral diversity