Online Experiential Learning for Language Models
The prevailing paradigm for improving large language models relies on offline training with human annotations or simulated environments, leaving the rich experience accumulated during real-world deployment entirely unexploited. We propose Online Experiential Learning (OEL), a framework that enables language models to continuously improve from their own deployment experience. OEL operates in two stages: first, transferable experiential knowledge is extracted and accumulated from interaction trajectories collected on the user side; second, this knowledge is consolidated into model parameters via on-policy context distillation, requiring no access to the user-side environment. The two stages are iterated to form an online learning loop, where the improved model collects higher-quality trajectories that yield richer experiential knowledge for subsequent rounds. We evaluate OEL on text-based game environments across multiple model scales and both thinking and non-thinking variants. OEL achieves consistent improvements over successive iterations, enhancing both task accuracy and token efficiency while preserving out-of-distribution performance. Our analysis further shows that extracted experiential knowledge is significantly more effective than raw trajectories, and that on-policy consistency between the knowledge source and the policy model is critical for effective learning.
69ba0c365d2b9c28ea09f042
Tianzhu Ye
admin_assigned
2026-03-18T10:00:38.917Z
68ef6325918a07690443c922
/avatars/6200c248f89afca2bab5b62428f3df0d.svg
Tianzhu Ye
6eo
user
6eo
69ba0c365d2b9c28ea09f043
Li Dong
claimed_verified
2026-03-18T08:45:16.817Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
unilm
user
unilm
69ba0c365d2b9c28ea09f044
Qingxiu Dong
69ba0c365d2b9c28ea09f045
Xun Wu
69ba0c365d2b9c28ea09f046
Shaohan Huang
69ba0c365d2b9c28ea09f047
Furu Wei
admin_assigned
2026-03-18T10:01:07.555Z
67ecd6178647cfa1775f75ed
/avatars/98882cc58dc0a5de94df765d523d92c9.svg
Furu Wei
frontierai
user
frontierai
Online Experiential Learning for Language Models
The prevailing paradigm for improving large language models relies on offline training with human annotations or simulated environments, leaving the rich experience accumulated during real-world deployment entirely unexploited. We propose Online Experiential Learning (OEL), a framework that enables language models to continuously improve from their own deployment experience. OEL operates in two stages: first, transferable experiential knowledge is extracted and accumulated from interaction trajectories collected on the user side; second, this knowledge is consolidated into model parameters via on-policy context distillation, requiring no access to the user-side environment. The two stages are iterated to form an online learning loop, where the improved model collects higher-quality trajectories that yield richer experiential knowledge for subsequent rounds. We evaluate OEL on text-based game environments across multiple model scales and both thinking and non-thinking variants. OEL achieves consistent improvements over successive iterations, enhancing both task accuracy and token efficiency while preserving out-of-distribution performance. Our analysis further shows that extracted experiential knowledge is significantly more effective than raw trajectories, and that on-policy consistency between the knowledge source and the policy model is critical for effective learning.
Online Experiential Learning enables continuous improvement of language models through deployment experience by extracting and consolidating experiential knowledge via on-policy distillation.
online learning experiential knowledge on-policy context distillation deployment experience interaction trajectories model parameters iterative learning loop task accuracy token efficiency out-of-distribution performance