Internalizing Meta-Experience into Memory for Guided Reinforcement Learning in Large Language Models
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for enhancing the reasoning capabilities of Large Language Models (LLMs). Despite its efficacy, RLVR faces a meta-learning bottleneck: it lacks mechanisms for error attribution and experience internalization intrinsic to the human learning cycle beyond practice and verification, thereby limiting fine-grained credit assignment and reusable knowledge formation. We term such reusable knowledge representations derived from past errors as meta-experience. Based on this insight, we propose Meta-Experience Learning (MEL), a novel framework that incorporates self-distilled meta-experience into the model's parametric memory. Building upon standard RLVR, we introduce an additional design that leverages the LLM's self-verification capability to conduct contrastive analysis on paired correct and incorrect trajectories, identify the precise bifurcation points where reasoning errors arise, and summarize them into generalizable meta-experience. The meta-experience is further internalized into the LLM's parametric memory by minimizing the negative log-likelihood, which induces a language-modeled reward signal that bridges correct and incorrect reasoning trajectories and facilitates effective knowledge reuse. Experimental results demonstrate that MEL achieves consistent improvements on benchmarks, yielding 3.92%--4.73% Pass@1 gains across varying model sizes.
698d401f65c0d15a6d162015
Shiting Huang
claimed_verified
2026-02-12T13:57:00.511Z
66ae3fbf491b555fef3bac0c
/avatars/47353470d46097ce108d32792dbbf2a2.svg
Shiting Huang
chocckaka
user
chocckaka
698d401f65c0d15a6d162016
Zecheng Li
698d401f65c0d15a6d162017
Yu Zeng
claimed_verified
2026-02-12T13:57:02.566Z
665d652e0f35c005de892108
https://cdn-avatars.huggingface.co/v1/production/uploads/665d652e0f35c005de892108/OGLbgZekX-3XTBkwS8k86.jpeg
Yu Zeng
YuZeng260
user
YuZeng260
698d401f65c0d15a6d162018
Qingnan Ren
698d401f65c0d15a6d162019
Zhen Fang
claimed_verified
2026-04-08T08:07:09.182Z
64b0a5037a475fba70a7260d
https://cdn-avatars.huggingface.co/v1/production/uploads/64b0a5037a475fba70a7260d/MauBbb6raMA23yrR1Zq21.jpeg
Zhen Fang
CostaliyA
user
CostaliyA
698d401f65c0d15a6d16201a
Qisheng Su
claimed_verified
2026-03-27T11:38:43.767Z
670a3bc3ada59c956f18cc17
/avatars/1a853deca3ec206f47aec2c6abe3a146.svg
SII-sqs
groundhogLLM
user
groundhogLLM
698d401f65c0d15a6d16201b
Kou Shi
698d401f65c0d15a6d16201c
Lin Chen
698d401f65c0d15a6d16201d
Zehui Chen
698d401f65c0d15a6d16201e
Feng Zhao
Internalizing Meta-Experience into Memory for Guided Reinforcement Learning in Large Language Models
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for enhancing the reasoning capabilities of Large Language Models (LLMs). Despite its efficacy, RLVR faces a meta-learning bottleneck: it lacks mechanisms for error attribution and experience internalization intrinsic to the human learning cycle beyond practice and verification, thereby limiting fine-grained credit assignment and reusable knowledge formation. We term such reusable knowledge representations derived from past errors as meta-experience. Based on this insight, we propose Meta-Experience Learning (MEL), a novel framework that incorporates self-distilled meta-experience into the model's parametric memory. Building upon standard RLVR, we introduce an additional design that leverages the LLM's self-verification capability to conduct contrastive analysis on paired correct and incorrect trajectories, identify the precise bifurcation points where reasoning errors arise, and summarize them into generalizable meta-experience. The meta-experience is further internalized into the LLM's parametric memory by minimizing the negative log-likelihood, which induces a language-modeled reward signal that bridges correct and incorrect reasoning trajectories and facilitates effective knowledge reuse. Experimental results demonstrate that MEL achieves consistent improvements on benchmarks, yielding 3.92%--4.73% Pass@1 gains across varying model sizes.
Meta-Experience Learning enhances LLM reasoning by incorporating self-distilled error representations into parametric memory through contrastive trajectory analysis and language-modeled reward signals.
reinforcement learning large language models meta-experience self-verification contrastive analysis parametric memory negative log-likelihood language-modeled reward Pass@1